[jira] [Updated] (ARROW-10902) [Rust][DataFusion] Catalog abstraction

2020-12-15 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-10902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remi Dettai updated ARROW-10902: Description: This issue is intended to be a design discussion around the introduction of the

[jira] [Comment Edited] (ARROW-10902) [Rust][DataFusion] Catalog abstraction

2020-12-14 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-10902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249089#comment-17249089 ] Remi Dettai edited comment on ARROW-10902 at 12/14/20, 4:36 PM: Catalogs

[jira] [Commented] (ARROW-10902) [Rust][DataFusion] Catalog abstraction

2020-12-14 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-10902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249089#comment-17249089 ] Remi Dettai commented on ARROW-10902: - Catalogs typically associate to each partition/file an

[jira] [Commented] (ARROW-10902) [Rust][DataFusion] Catalog abstraction

2020-12-14 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-10902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249064#comment-17249064 ] Remi Dettai commented on ARROW-10902: - If anyone wants write access, feel free to ask! >

[jira] [Created] (ARROW-10902) [Rust][DataFusion] Catalog abstraction

2020-12-14 Thread Remi Dettai (Jira)
Remi Dettai created ARROW-10902: --- Summary: [Rust][DataFusion] Catalog abstraction Key: ARROW-10902 URL: https://issues.apache.org/jira/browse/ARROW-10902 Project: Apache Arrow Issue Type:

[jira] [Created] (ARROW-10900) [Rust][DataFusion] Resolve TableScan provider eagerly

2020-12-14 Thread Remi Dettai (Jira)
Remi Dettai created ARROW-10900: --- Summary: [Rust][DataFusion] Resolve TableScan provider eagerly Key: ARROW-10900 URL: https://issues.apache.org/jira/browse/ARROW-10900 Project: Apache Arrow

[jira] [Comment Edited] (ARROW-9770) [Rust] [DataFusion] Add constant folding to expressions during logically planning

2020-12-08 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-9770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17245907#comment-17245907 ] Remi Dettai edited comment on ARROW-9770 at 12/8/20, 2:11 PM: -- This could

[jira] [Commented] (ARROW-9770) [Rust] [DataFusion] Add constant folding to expressions during logically planning

2020-12-08 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-9770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17245907#comment-17245907 ] Remi Dettai commented on ARROW-9770: This could also be used to apply filter pushdown into the

[jira] [Created] (ARROW-10789) [Rust][DataFusion] Make TableProvider dynamically typed

2020-12-02 Thread Remi Dettai (Jira)
Remi Dettai created ARROW-10789: --- Summary: [Rust][DataFusion] Make TableProvider dynamically typed Key: ARROW-10789 URL: https://issues.apache.org/jira/browse/ARROW-10789 Project: Apache Arrow

[jira] [Closed] (ARROW-10387) [Rust] Avoid call for file size metadata when reading parquet footer

2020-12-02 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-10387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remi Dettai closed ARROW-10387. --- Resolution: Abandoned Closing because nobody showed interest in this feature > [Rust] Avoid call

[jira] [Commented] (ARROW-10553) [Rust] [Parquet] Panic when reading Parquet file produced with parquet-cpp

2020-11-19 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-10553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235536#comment-17235536 ] Remi Dettai commented on ARROW-10553: - [~spektom] It is confirmed that forward compability is an

[jira] [Commented] (ARROW-10553) [Rust] [Parquet] Panic when reading Parquet file produced with parquet-cpp

2020-11-17 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-10553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233466#comment-17233466 ] Remi Dettai commented on ARROW-10553: - I looked a little bit into thrift and the c++ vs the rust

[jira] [Commented] (ARROW-6256) [Rust] parquet-format should be released by Apache process

2020-11-17 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-6256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233428#comment-17233428 ] Remi Dettai commented on ARROW-6256: This issue is still pending. I did not find any discussion

[jira] [Commented] (ARROW-10553) [Rust] [Parquet] Panic when reading Parquet file produced with parquet-cpp

2020-11-17 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-10553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233426#comment-17233426 ] Remi Dettai commented on ARROW-10553: - {quote}Is it that Pyarrow is state of the art Parquet reader,

[jira] [Created] (ARROW-10620) [Rust][Parquet] move column chunk range logic to metadata.rs

2020-11-16 Thread Remi Dettai (Jira)
Remi Dettai created ARROW-10620: --- Summary: [Rust][Parquet] move column chunk range logic to metadata.rs Key: ARROW-10620 URL: https://issues.apache.org/jira/browse/ARROW-10620 Project: Apache Arrow

[jira] [Commented] (ARROW-10553) [Rust] [Parquet] Panic when reading Parquet file produced with parquet-cpp

2020-11-16 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-10553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17232911#comment-17232911 ] Remi Dettai commented on ARROW-10553: - Hi Michael! Thanks for reporting this. Have you tried writing

[jira] [Commented] (ARROW-10577) [Rust][DataFusion] Hash Aggregator stream finishes unexpectedly after going to Pending state

2020-11-13 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-10577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17231251#comment-17231251 ] Remi Dettai commented on ARROW-10577: - I can take a crack at it. [~jorgecarleitao] and [~andygrove]

[jira] [Created] (ARROW-10577) [Rust][DataFusion] Hash Aggregator stream finishes unexpectedly after going to Pending state

2020-11-13 Thread Remi Dettai (Jira)
Remi Dettai created ARROW-10577: --- Summary: [Rust][DataFusion] Hash Aggregator stream finishes unexpectedly after going to Pending state Key: ARROW-10577 URL: https://issues.apache.org/jira/browse/ARROW-10577

[jira] [Created] (ARROW-10389) [Rust][DataFusion] Make the custom source implementation API more explicit

2020-10-26 Thread Remi Dettai (Jira)
Remi Dettai created ARROW-10389: --- Summary: [Rust][DataFusion] Make the custom source implementation API more explicit Key: ARROW-10389 URL: https://issues.apache.org/jira/browse/ARROW-10389 Project:

[jira] [Updated] (ARROW-10387) [Rust] Avoid call for file size metadata when reading parquet footer

2020-10-26 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-10387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remi Dettai updated ARROW-10387: Component/s: Rust > [Rust] Avoid call for file size metadata when reading parquet footer >

[jira] [Created] (ARROW-10387) [Rust] Avoid call for file size metadata when reading parquet footer

2020-10-26 Thread Remi Dettai (Jira)
Remi Dettai created ARROW-10387: --- Summary: [Rust] Avoid call for file size metadata when reading parquet footer Key: ARROW-10387 URL: https://issues.apache.org/jira/browse/ARROW-10387 Project: Apache

[jira] [Closed] (ARROW-10368) [Rust][DataFusion] Refactor scan nodes to allow extensions

2020-10-26 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-10368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remi Dettai closed ARROW-10368. --- Resolution: Not A Problem > [Rust][DataFusion] Refactor scan nodes to allow extensions >

[jira] [Commented] (ARROW-10368) [Rust][DataFusion] Refactor scan nodes to allow extensions

2020-10-23 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-10368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219723#comment-17219723 ] Remi Dettai commented on ARROW-10368: - Just found out the addition of custom sources can be done

[jira] [Commented] (ARROW-10368) [Rust][DataFusion] Refactor scan nodes to allow extensions

2020-10-23 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-10368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219558#comment-17219558 ] Remi Dettai commented on ARROW-10368: - [~jorgecarleitao] thanks for looking into this. It's so nice

[jira] [Updated] (ARROW-10368) [Rust][DataFusion] Refactor scan nodes to allow extensions

2020-10-23 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-10368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remi Dettai updated ARROW-10368: Description: The first intention of this issue was to refactor InMemoryScan to use an iterator

[jira] [Updated] (ARROW-10368) [Rust][DataFusion] Refactor scan nodes to allow extensions

2020-10-23 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-10368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remi Dettai updated ARROW-10368: Description: The first intention was to refactor InMemoryScan to use an iterator.

[jira] [Updated] (ARROW-10368) [Rust][DataFusion] Refactor scan nodes to allow extensions

2020-10-23 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-10368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remi Dettai updated ARROW-10368: Summary: [Rust][DataFusion] Refactor scan nodes to allow extensions (was: [Rust][DataFusion]

[jira] [Commented] (ARROW-10368) [Rust][DataFusion] Make InMemoryScan work on iterators of RecordBatch

2020-10-22 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-10368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219210#comment-17219210 ] Remi Dettai commented on ARROW-10368: - If I summarize all of the above, these are the paths I could

[jira] [Commented] (ARROW-10368) [Rust][DataFusion] Make InMemoryScan work on iterators of RecordBatch

2020-10-22 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-10368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219181#comment-17219181 ] Remi Dettai commented on ARROW-10368: - _I know what this is: It's just myself talking to myself

[jira] [Comment Edited] (ARROW-10368) [Rust][DataFusion] Make InMemoryScan work on iterators of RecordBatch

2020-10-22 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-10368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219142#comment-17219142 ] Remi Dettai edited comment on ARROW-10368 at 10/22/20, 4:32 PM: bq. I

[jira] [Commented] (ARROW-10368) [Rust][DataFusion] Make InMemoryScan work on iterators of RecordBatch

2020-10-22 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-10368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219142#comment-17219142 ] Remi Dettai commented on ARROW-10368: - bq. I wonder if we could not go one step further and try to

[jira] [Comment Edited] (ARROW-10368) [Rust][DataFusion] Make InMemoryScan work on iterators of RecordBatch

2020-10-22 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-10368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17218952#comment-17218952 ] Remi Dettai edited comment on ARROW-10368 at 10/22/20, 11:59 AM: -

[jira] [Commented] (ARROW-10368) [Rust][DataFusion] Make InMemoryScan work on iterators of RecordBatch

2020-10-22 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-10368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17218952#comment-17218952 ] Remi Dettai commented on ARROW-10368: - [~andygrove] If this change seems reasonable to you, I can

[jira] [Updated] (ARROW-10368) [Rust][DataFusion] Make InMemoryScan work on iterators of RecordBatch

2020-10-22 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-10368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remi Dettai updated ARROW-10368: Summary: [Rust][DataFusion] Make InMemoryScan work on iterators of RecordBatch (was:

[jira] [Created] (ARROW-10368) [Rust][Datafusion] Make InMemoryScan work on iterators of RecordBatch

2020-10-22 Thread Remi Dettai (Jira)
Remi Dettai created ARROW-10368: --- Summary: [Rust][Datafusion] Make InMemoryScan work on iterators of RecordBatch Key: ARROW-10368 URL: https://issues.apache.org/jira/browse/ARROW-10368 Project: Apache

[jira] [Created] (ARROW-10307) [Rust] async parquet reader

2020-10-14 Thread Remi Dettai (Jira)
Remi Dettai created ARROW-10307: --- Summary: [Rust] async parquet reader Key: ARROW-10307 URL: https://issues.apache.org/jira/browse/ARROW-10307 Project: Apache Arrow Issue Type: Improvement

[jira] [Created] (ARROW-10300) [Rust] Parquet/CSV TPC-H data

2020-10-13 Thread Remi Dettai (Jira)
Remi Dettai created ARROW-10300: --- Summary: [Rust] Parquet/CSV TPC-H data Key: ARROW-10300 URL: https://issues.apache.org/jira/browse/ARROW-10300 Project: Apache Arrow Issue Type: Wish

[jira] [Commented] (ARROW-10135) [Rust] [Parquet] Refactor file module to help adding sources

2020-10-02 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-10135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17206012#comment-17206012 ] Remi Dettai commented on ARROW-10135: - [~alamb] thanks again for you response, very interesting

[jira] [Commented] (ARROW-10135) [Rust] [Parquet] Refactor file module to help adding sources

2020-10-01 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-10135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205356#comment-17205356 ] Remi Dettai commented on ARROW-10135: - Hi [~alamb] ! Thanks for your insight. The problem when you

[jira] [Updated] (ARROW-10135) [Rust] [Parquet] Refactor file module to help adding sources

2020-09-29 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-10135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remi Dettai updated ARROW-10135: Description: Currently, the Parquet reader is very strongly tied to file system reads. This

[jira] [Updated] (ARROW-10135) [Rust] [Parquet] Refactor file module to help adding sources

2020-09-29 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-10135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remi Dettai updated ARROW-10135: Description: Currently, the Parquet reader is very strongly tied to file system reads. This

[jira] [Updated] (ARROW-10135) [Rust] [Parquet] Refactor file module to help adding sources

2020-09-29 Thread Remi Dettai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-10135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remi Dettai updated ARROW-10135: Description: Currently, the Parquet reader is very strongly tied to file system reads. This

[jira] [Created] (ARROW-10135) [Rust] [Parquet] Refactor file module to help adding sources

2020-09-29 Thread Remi Dettai (Jira)
Remi Dettai created ARROW-10135: --- Summary: [Rust] [Parquet] Refactor file module to help adding sources Key: ARROW-10135 URL: https://issues.apache.org/jira/browse/ARROW-10135 Project: Apache Arrow