kou commented on a change in pull request #11584: URL: https://github.com/apache/arrow/pull/11584#discussion_r740510193
########## File path: ruby/red-arrow/doc/text/examples.md ########## @@ -0,0 +1,106 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Examples + +## Create table +### From file +```ruby +table = Arrow::Table.load('data.arrow') +table = Arrow::Table.load('data.csv', format: :csv) +table = Arrow::Table.load('data.parquet', format: :parquet) # requires 'red-parquet' gem +``` +### From Ruby hash +Types will be detected automatically +```ruby +table = Arrow::Table.new('name' => ['Tom', 'Max'], 'age' => [22, 23]) +``` +### From String +Suppose you have your data available via Http. Let's connect to demo Clickhouse db. See https://play.clickhouse.com/ for details Review comment: ```suggestion Suppose you have your data available via HTTP. Let's connect to demo ClickHouse DB. See https://play.clickhouse.com/ for details ``` ########## File path: ruby/red-arrow/doc/text/examples.md ########## @@ -0,0 +1,106 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Examples + +## Create table +### From file +```ruby +table = Arrow::Table.load('data.arrow') +table = Arrow::Table.load('data.csv', format: :csv) +table = Arrow::Table.load('data.parquet', format: :parquet) # requires 'red-parquet' gem +``` +### From Ruby hash +Types will be detected automatically +```ruby +table = Arrow::Table.new('name' => ['Tom', 'Max'], 'age' => [22, 23]) +``` +### From String +Suppose you have your data available via Http. Let's connect to demo Clickhouse db. See https://play.clickhouse.com/ for details +```ruby +params = { + query: "SELECT WatchID as watch FROM hits_v1 LIMIT 10 FORMAT Arrow", + user: "playground", + password: "clickhouse", + database: "datasets" +} +uri = URI('https://play-api.clickhouse.com:8443') +uri.query = URI.encode_www_form(params) +resp = Net::HTTP.get(uri) +table = Arrow::Table.load(Arrow::Buffer.try_convert(resp)) +``` + +## Filtering +Uses concept of slicers in Arrow +```ruby +table = Arrow::Table.new( + 'name' => ['Tom', 'Max', 'Kate'], + 'age' => [22, 23, 19] +) +slicer = Arrow::Slicer.new(table) +table.slice(slicer['age'] > 19) +=> #<Arrow::Table:0x7fa38838c448 ptr=0x7fa3ad269f40> + name age +0 Tom 22 +1 Max 23 Review comment: ```suggestion # => #<Arrow::Table:0x7fa38838c448 ptr=0x7fa3ad269f40> # name age # 0 Tom 22 # 1 Max 23 ``` ########## File path: ruby/red-arrow/doc/text/examples.md ########## @@ -0,0 +1,106 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Examples + +## Create table +### From file +```ruby +table = Arrow::Table.load('data.arrow') +table = Arrow::Table.load('data.csv', format: :csv) +table = Arrow::Table.load('data.parquet', format: :parquet) # requires 'red-parquet' gem +``` +### From Ruby hash +Types will be detected automatically +```ruby +table = Arrow::Table.new('name' => ['Tom', 'Max'], 'age' => [22, 23]) +``` +### From String +Suppose you have your data available via Http. Let's connect to demo Clickhouse db. See https://play.clickhouse.com/ for details +```ruby +params = { + query: "SELECT WatchID as watch FROM hits_v1 LIMIT 10 FORMAT Arrow", + user: "playground", + password: "clickhouse", + database: "datasets" +} +uri = URI('https://play-api.clickhouse.com:8443') +uri.query = URI.encode_www_form(params) +resp = Net::HTTP.get(uri) +table = Arrow::Table.load(Arrow::Buffer.try_convert(resp)) +``` + +## Filtering +Uses concept of slicers in Arrow +```ruby +table = Arrow::Table.new( + 'name' => ['Tom', 'Max', 'Kate'], + 'age' => [22, 23, 19] +) +slicer = Arrow::Slicer.new(table) +table.slice(slicer['age'] > 19) +=> #<Arrow::Table:0x7fa38838c448 ptr=0x7fa3ad269f40> + name age +0 Tom 22 +1 Max 23 + +table.slice(slicer['age'].in?(19..22)) +=> #<Arrow::Table:0x7fa3881cf998 ptr=0x7fa3a4bb5f30> + name age +0 Tom 22 +1 Kate 19 +``` +Multiple slice conditions can be joined using and(`&`) / or (`|`) / xor(`^`) logical operations +```ruby +table.slice((slicer['age'] > 19) & (slicer['age'] < 23)) +=> #<Arrow::Table:0x7fa3882cc300 ptr=0x7fa3ad260b00> + name age +0 Tom 22 Review comment: ```suggestion # => #<Arrow::Table:0x7fa3882cc300 ptr=0x7fa3ad260b00> # name age # 0 Tom 22 ``` ########## File path: ruby/red-arrow/doc/text/examples.md ########## @@ -0,0 +1,106 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Examples + +## Create table +### From file +```ruby +table = Arrow::Table.load('data.arrow') +table = Arrow::Table.load('data.csv', format: :csv) +table = Arrow::Table.load('data.parquet', format: :parquet) # requires 'red-parquet' gem +``` +### From Ruby hash +Types will be detected automatically +```ruby +table = Arrow::Table.new('name' => ['Tom', 'Max'], 'age' => [22, 23]) +``` +### From String +Suppose you have your data available via Http. Let's connect to demo Clickhouse db. See https://play.clickhouse.com/ for details +```ruby +params = { + query: "SELECT WatchID as watch FROM hits_v1 LIMIT 10 FORMAT Arrow", + user: "playground", + password: "clickhouse", + database: "datasets" +} +uri = URI('https://play-api.clickhouse.com:8443') +uri.query = URI.encode_www_form(params) +resp = Net::HTTP.get(uri) +table = Arrow::Table.load(Arrow::Buffer.try_convert(resp)) +``` + +## Filtering +Uses concept of slicers in Arrow +```ruby +table = Arrow::Table.new( + 'name' => ['Tom', 'Max', 'Kate'], + 'age' => [22, 23, 19] +) +slicer = Arrow::Slicer.new(table) +table.slice(slicer['age'] > 19) +=> #<Arrow::Table:0x7fa38838c448 ptr=0x7fa3ad269f40> + name age +0 Tom 22 +1 Max 23 + +table.slice(slicer['age'].in?(19..22)) +=> #<Arrow::Table:0x7fa3881cf998 ptr=0x7fa3a4bb5f30> + name age +0 Tom 22 +1 Kate 19 +``` +Multiple slice conditions can be joined using and(`&`) / or (`|`) / xor(`^`) logical operations +```ruby +table.slice((slicer['age'] > 19) & (slicer['age'] < 23)) +=> #<Arrow::Table:0x7fa3882cc300 ptr=0x7fa3ad260b00> + name age +0 Tom 22 +``` + +## Operations +Arrow compute functions can be accessed through `Arrow::Function` +```ruby +add = Arrow::Function.find('add') +add.execute([table['age'].data, table['age'].data]).value +=> #<Arrow::ChunkedArray:0x7fa389b87250 ptr=0x7fa3a4bb5c40 [ + [ + 44, + 46, + 38 + ] +]> Review comment: ```suggestion # => #<Arrow::ChunkedArray:0x7fa389b87250 ptr=0x7fa3a4bb5c40 [ # [ # 44, # 46, # 38 # ] # ]> ``` ########## File path: ruby/red-arrow/doc/text/examples.md ########## @@ -0,0 +1,106 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Examples + +## Create table +### From file +```ruby +table = Arrow::Table.load('data.arrow') +table = Arrow::Table.load('data.csv', format: :csv) +table = Arrow::Table.load('data.parquet', format: :parquet) # requires 'red-parquet' gem +``` +### From Ruby hash +Types will be detected automatically +```ruby +table = Arrow::Table.new('name' => ['Tom', 'Max'], 'age' => [22, 23]) +``` +### From String +Suppose you have your data available via Http. Let's connect to demo Clickhouse db. See https://play.clickhouse.com/ for details +```ruby +params = { + query: "SELECT WatchID as watch FROM hits_v1 LIMIT 10 FORMAT Arrow", + user: "playground", + password: "clickhouse", + database: "datasets" +} +uri = URI('https://play-api.clickhouse.com:8443') +uri.query = URI.encode_www_form(params) +resp = Net::HTTP.get(uri) +table = Arrow::Table.load(Arrow::Buffer.try_convert(resp)) +``` + +## Filtering +Uses concept of slicers in Arrow +```ruby +table = Arrow::Table.new( + 'name' => ['Tom', 'Max', 'Kate'], + 'age' => [22, 23, 19] +) +slicer = Arrow::Slicer.new(table) +table.slice(slicer['age'] > 19) +=> #<Arrow::Table:0x7fa38838c448 ptr=0x7fa3ad269f40> + name age +0 Tom 22 +1 Max 23 + +table.slice(slicer['age'].in?(19..22)) Review comment: ```suggestion table.slice {|slicer| slicer['age'].in?(19..22)} ``` ########## File path: ruby/red-arrow/doc/text/examples.md ########## @@ -0,0 +1,106 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Examples + +## Create table +### From file +```ruby +table = Arrow::Table.load('data.arrow') +table = Arrow::Table.load('data.csv', format: :csv) +table = Arrow::Table.load('data.parquet', format: :parquet) # requires 'red-parquet' gem +``` +### From Ruby hash +Types will be detected automatically +```ruby +table = Arrow::Table.new('name' => ['Tom', 'Max'], 'age' => [22, 23]) +``` +### From String +Suppose you have your data available via Http. Let's connect to demo Clickhouse db. See https://play.clickhouse.com/ for details +```ruby +params = { + query: "SELECT WatchID as watch FROM hits_v1 LIMIT 10 FORMAT Arrow", + user: "playground", + password: "clickhouse", + database: "datasets" +} +uri = URI('https://play-api.clickhouse.com:8443') +uri.query = URI.encode_www_form(params) +resp = Net::HTTP.get(uri) +table = Arrow::Table.load(Arrow::Buffer.try_convert(resp)) +``` + +## Filtering +Uses concept of slicers in Arrow +```ruby +table = Arrow::Table.new( + 'name' => ['Tom', 'Max', 'Kate'], + 'age' => [22, 23, 19] +) +slicer = Arrow::Slicer.new(table) +table.slice(slicer['age'] > 19) +=> #<Arrow::Table:0x7fa38838c448 ptr=0x7fa3ad269f40> + name age +0 Tom 22 +1 Max 23 + +table.slice(slicer['age'].in?(19..22)) +=> #<Arrow::Table:0x7fa3881cf998 ptr=0x7fa3a4bb5f30> + name age +0 Tom 22 +1 Kate 19 +``` +Multiple slice conditions can be joined using and(`&`) / or (`|`) / xor(`^`) logical operations +```ruby +table.slice((slicer['age'] > 19) & (slicer['age'] < 23)) +=> #<Arrow::Table:0x7fa3882cc300 ptr=0x7fa3ad260b00> + name age +0 Tom 22 +``` + +## Operations +Arrow compute functions can be accessed through `Arrow::Function` +```ruby +add = Arrow::Function.find('add') +add.execute([table['age'].data, table['age'].data]).value Review comment: Ah, we should support `add.execute([table['age'], table['age']]).value` for this. Could you open a Jira issue for this? ########## File path: ruby/red-arrow/doc/text/examples.md ########## @@ -0,0 +1,106 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Examples + +## Create table +### From file +```ruby +table = Arrow::Table.load('data.arrow') +table = Arrow::Table.load('data.csv', format: :csv) +table = Arrow::Table.load('data.parquet', format: :parquet) # requires 'red-parquet' gem +``` +### From Ruby hash +Types will be detected automatically +```ruby +table = Arrow::Table.new('name' => ['Tom', 'Max'], 'age' => [22, 23]) +``` +### From String +Suppose you have your data available via Http. Let's connect to demo Clickhouse db. See https://play.clickhouse.com/ for details +```ruby +params = { + query: "SELECT WatchID as watch FROM hits_v1 LIMIT 10 FORMAT Arrow", + user: "playground", + password: "clickhouse", + database: "datasets" +} +uri = URI('https://play-api.clickhouse.com:8443') +uri.query = URI.encode_www_form(params) +resp = Net::HTTP.get(uri) +table = Arrow::Table.load(Arrow::Buffer.try_convert(resp)) +``` + +## Filtering +Uses concept of slicers in Arrow +```ruby +table = Arrow::Table.new( + 'name' => ['Tom', 'Max', 'Kate'], + 'age' => [22, 23, 19] +) +slicer = Arrow::Slicer.new(table) +table.slice(slicer['age'] > 19) Review comment: ```suggestion table.slice {|slicer| slicer['age'] > 19} ``` ########## File path: ruby/red-arrow/doc/text/examples.md ########## @@ -0,0 +1,106 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Examples + +## Create table +### From file +```ruby +table = Arrow::Table.load('data.arrow') +table = Arrow::Table.load('data.csv', format: :csv) +table = Arrow::Table.load('data.parquet', format: :parquet) # requires 'red-parquet' gem +``` +### From Ruby hash +Types will be detected automatically +```ruby +table = Arrow::Table.new('name' => ['Tom', 'Max'], 'age' => [22, 23]) +``` +### From String +Suppose you have your data available via Http. Let's connect to demo Clickhouse db. See https://play.clickhouse.com/ for details +```ruby +params = { + query: "SELECT WatchID as watch FROM hits_v1 LIMIT 10 FORMAT Arrow", + user: "playground", + password: "clickhouse", + database: "datasets" +} +uri = URI('https://play-api.clickhouse.com:8443') +uri.query = URI.encode_www_form(params) +resp = Net::HTTP.get(uri) +table = Arrow::Table.load(Arrow::Buffer.try_convert(resp)) Review comment: ```suggestion table = Arrow::Table.load(Arrow::Buffer.new(resp)) ``` BTW, we should add support for `Arrow::Table.load(URI("https://...";))` style. Could you open a Jira issue for it? ########## File path: ruby/red-arrow/doc/text/examples.md ########## @@ -0,0 +1,106 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Examples + +## Create table +### From file +```ruby +table = Arrow::Table.load('data.arrow') +table = Arrow::Table.load('data.csv', format: :csv) +table = Arrow::Table.load('data.parquet', format: :parquet) # requires 'red-parquet' gem +``` +### From Ruby hash +Types will be detected automatically +```ruby +table = Arrow::Table.new('name' => ['Tom', 'Max'], 'age' => [22, 23]) +``` +### From String +Suppose you have your data available via Http. Let's connect to demo Clickhouse db. See https://play.clickhouse.com/ for details +```ruby +params = { + query: "SELECT WatchID as watch FROM hits_v1 LIMIT 10 FORMAT Arrow", + user: "playground", + password: "clickhouse", + database: "datasets" +} +uri = URI('https://play-api.clickhouse.com:8443') +uri.query = URI.encode_www_form(params) +resp = Net::HTTP.get(uri) +table = Arrow::Table.load(Arrow::Buffer.try_convert(resp)) +``` + +## Filtering +Uses concept of slicers in Arrow +```ruby +table = Arrow::Table.new( + 'name' => ['Tom', 'Max', 'Kate'], + 'age' => [22, 23, 19] +) +slicer = Arrow::Slicer.new(table) +table.slice(slicer['age'] > 19) +=> #<Arrow::Table:0x7fa38838c448 ptr=0x7fa3ad269f40> + name age +0 Tom 22 +1 Max 23 + +table.slice(slicer['age'].in?(19..22)) +=> #<Arrow::Table:0x7fa3881cf998 ptr=0x7fa3a4bb5f30> + name age +0 Tom 22 +1 Kate 19 Review comment: ```suggestion # => #<Arrow::Table:0x7fa3881cf998 ptr=0x7fa3a4bb5f30> # name age # 0 Tom 22 # 1 Kate 19 ``` ########## File path: ruby/red-arrow/doc/text/examples.md ########## @@ -0,0 +1,106 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Examples + +## Create table +### From file +```ruby +table = Arrow::Table.load('data.arrow') +table = Arrow::Table.load('data.csv', format: :csv) +table = Arrow::Table.load('data.parquet', format: :parquet) # requires 'red-parquet' gem +``` +### From Ruby hash +Types will be detected automatically +```ruby +table = Arrow::Table.new('name' => ['Tom', 'Max'], 'age' => [22, 23]) +``` +### From String +Suppose you have your data available via Http. Let's connect to demo Clickhouse db. See https://play.clickhouse.com/ for details +```ruby +params = { + query: "SELECT WatchID as watch FROM hits_v1 LIMIT 10 FORMAT Arrow", + user: "playground", + password: "clickhouse", + database: "datasets" +} +uri = URI('https://play-api.clickhouse.com:8443') +uri.query = URI.encode_www_form(params) +resp = Net::HTTP.get(uri) Review comment: Could you add `require "net/http"`? ########## File path: ruby/red-arrow/doc/text/examples.md ########## @@ -0,0 +1,106 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Examples + +## Create table +### From file +```ruby +table = Arrow::Table.load('data.arrow') +table = Arrow::Table.load('data.csv', format: :csv) +table = Arrow::Table.load('data.parquet', format: :parquet) # requires 'red-parquet' gem +``` +### From Ruby hash +Types will be detected automatically +```ruby +table = Arrow::Table.new('name' => ['Tom', 'Max'], 'age' => [22, 23]) +``` +### From String +Suppose you have your data available via Http. Let's connect to demo Clickhouse db. See https://play.clickhouse.com/ for details +```ruby +params = { + query: "SELECT WatchID as watch FROM hits_v1 LIMIT 10 FORMAT Arrow", + user: "playground", + password: "clickhouse", + database: "datasets" +} +uri = URI('https://play-api.clickhouse.com:8443') +uri.query = URI.encode_www_form(params) +resp = Net::HTTP.get(uri) +table = Arrow::Table.load(Arrow::Buffer.try_convert(resp)) +``` + +## Filtering +Uses concept of slicers in Arrow +```ruby +table = Arrow::Table.new( + 'name' => ['Tom', 'Max', 'Kate'], + 'age' => [22, 23, 19] +) +slicer = Arrow::Slicer.new(table) +table.slice(slicer['age'] > 19) +=> #<Arrow::Table:0x7fa38838c448 ptr=0x7fa3ad269f40> + name age +0 Tom 22 +1 Max 23 + +table.slice(slicer['age'].in?(19..22)) +=> #<Arrow::Table:0x7fa3881cf998 ptr=0x7fa3a4bb5f30> + name age +0 Tom 22 +1 Kate 19 +``` +Multiple slice conditions can be joined using and(`&`) / or (`|`) / xor(`^`) logical operations +```ruby +table.slice((slicer['age'] > 19) & (slicer['age'] < 23)) +=> #<Arrow::Table:0x7fa3882cc300 ptr=0x7fa3ad260b00> + name age +0 Tom 22 +``` + +## Operations +Arrow compute functions can be accessed through `Arrow::Function` +```ruby +add = Arrow::Function.find('add') +add.execute([table['age'].data, table['age'].data]).value +=> #<Arrow::ChunkedArray:0x7fa389b87250 ptr=0x7fa3a4bb5c40 [ + [ + 44, + 46, + 38 + ] +]> +``` + +## Grouping +```ruby +table = Arrow::Table.new( + 'name' => ['Tom', 'Max', 'Kate', 'Tom'], + 'amount' => [10, 2, 3, 5] +) +table.group('name').sum('amount') +=> #<Arrow::Table:0x7fa389894ae8 ptr=0x7fa364141a50> + name amount +0 Kate 3 +1 Max 2 +2 Tom 15 Review comment: ```suggestion # => #<Arrow::Table:0x7fa389894ae8 ptr=0x7fa364141a50> # name amount # 0 Kate 3 # 1 Max 2 # 2 Tom 15 ``` ########## File path: ruby/red-arrow/doc/text/examples.md ########## @@ -0,0 +1,106 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Examples + +## Create table +### From file +```ruby +table = Arrow::Table.load('data.arrow') Review comment: Could you add `require "arrow"` to examples? ########## File path: ruby/red-arrow/doc/text/examples.md ########## @@ -0,0 +1,106 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Examples + +## Create table +### From file +```ruby +table = Arrow::Table.load('data.arrow') +table = Arrow::Table.load('data.csv', format: :csv) +table = Arrow::Table.load('data.parquet', format: :parquet) # requires 'red-parquet' gem +``` +### From Ruby hash +Types will be detected automatically +```ruby +table = Arrow::Table.new('name' => ['Tom', 'Max'], 'age' => [22, 23]) +``` +### From String +Suppose you have your data available via Http. Let's connect to demo Clickhouse db. See https://play.clickhouse.com/ for details +```ruby +params = { + query: "SELECT WatchID as watch FROM hits_v1 LIMIT 10 FORMAT Arrow", + user: "playground", + password: "clickhouse", + database: "datasets" +} +uri = URI('https://play-api.clickhouse.com:8443') +uri.query = URI.encode_www_form(params) +resp = Net::HTTP.get(uri) +table = Arrow::Table.load(Arrow::Buffer.try_convert(resp)) +``` + +## Filtering +Uses concept of slicers in Arrow +```ruby +table = Arrow::Table.new( + 'name' => ['Tom', 'Max', 'Kate'], + 'age' => [22, 23, 19] +) +slicer = Arrow::Slicer.new(table) +table.slice(slicer['age'] > 19) +=> #<Arrow::Table:0x7fa38838c448 ptr=0x7fa3ad269f40> + name age +0 Tom 22 +1 Max 23 + +table.slice(slicer['age'].in?(19..22)) +=> #<Arrow::Table:0x7fa3881cf998 ptr=0x7fa3a4bb5f30> + name age +0 Tom 22 +1 Kate 19 +``` +Multiple slice conditions can be joined using and(`&`) / or (`|`) / xor(`^`) logical operations +```ruby +table.slice((slicer['age'] > 19) & (slicer['age'] < 23)) Review comment: ```suggestion table.slice {|slicer| (slicer['age'] > 19) & (slicer['age'] < 23)} ``` ########## File path: ruby/red-arrow/doc/text/examples.md ########## @@ -0,0 +1,106 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Examples + +## Create table +### From file +```ruby +table = Arrow::Table.load('data.arrow') +table = Arrow::Table.load('data.csv', format: :csv) +table = Arrow::Table.load('data.parquet', format: :parquet) # requires 'red-parquet' gem +``` +### From Ruby hash +Types will be detected automatically +```ruby +table = Arrow::Table.new('name' => ['Tom', 'Max'], 'age' => [22, 23]) +``` +### From String +Suppose you have your data available via Http. Let's connect to demo Clickhouse db. See https://play.clickhouse.com/ for details +```ruby +params = { + query: "SELECT WatchID as watch FROM hits_v1 LIMIT 10 FORMAT Arrow", + user: "playground", + password: "clickhouse", + database: "datasets" +} +uri = URI('https://play-api.clickhouse.com:8443') +uri.query = URI.encode_www_form(params) +resp = Net::HTTP.get(uri) +table = Arrow::Table.load(Arrow::Buffer.try_convert(resp)) Review comment: Apache Arrow C++ provides filesystem module: https://github.com/apache/arrow/tree/master/cpp/src/arrow/filesystem It supports some HTTP based data source such as Amazon S3 and Google Cloud Storage. I think that `URI("https://...";)` is convenient for simple (and many?) use cases. How about thinking new API when we have use cases that require complex HTTP use such as POST and Authorization headers? It may be better that these complex use cases are implemented as a new filesystem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
