Re: Arrow Flight connector for SQL Server

2020-05-19 Thread Jacques Nadeau
Hey Brendan, Welcome to the community. At Dremio we've exposed flight as an input and output for sql result datasets. I'll have one of our guys share some details. I think a couple questions we've been struggling with include how to standardize additional metadata operations, what should the

[jira] [Created] (ARROW-8869) [Rust] [DataFusion] Type Coercion optimizer rule does not support new scan nodes

2020-05-19 Thread Andy Grove (Jira)
Andy Grove created ARROW-8869: - Summary: [Rust] [DataFusion] Type Coercion optimizer rule does not support new scan nodes Key: ARROW-8869 URL: https://issues.apache.org/jira/browse/ARROW-8869 Project:

[Rust] Vectorized traits for using arrays outside Arrow

2020-05-19 Thread Vertexclique
Hi; I wanted to discuss with Rust lib maintainers about how can we improve the current status of Rust's DictionaryArray and reading its' encoding array outside the Arrow. So a simple predicate filter needs to collect index over iterator and flat map over the optional values or map over the

[jira] [Created] (ARROW-8868) [Python] Feather format cannot store/retrieve lists correctly?

2020-05-19 Thread Farzad Abdolhosseini (Jira)
Farzad Abdolhosseini created ARROW-8868: --- Summary: [Python] Feather format cannot store/retrieve lists correctly? Key: ARROW-8868 URL: https://issues.apache.org/jira/browse/ARROW-8868 Project:

Re: [VOTE] Release Apache Arrow 0.17.1 - RC1

2020-05-19 Thread Neal Richardson
R submission to CRAN is done and accepted. I'm waiting to do Homebrew until after the website update, given their pushback last time. Neal On Tue, May 19, 2020 at 5:25 AM Uwe L. Korn wrote: > Current status: > > 1. [done] rebase (not required for a patch release) > 2. [done] upload source >

Arrow Flight connector for SQL Server

2020-05-19 Thread Brendan Niebruegge
Hi everyone, I wanted to informally introduce myself. My name is Brendan Niebruegge, I'm a Software Engineer in our SQL Server extensibility team here at Microsoft. I am leading an effort to explore how we could integrate Arrow Flight with SQL Server. We think this could be a very interesting

[jira] [Created] (ARROW-8866) [C++] Split Type::UNION into Type::SPARSE_UNION and Type::DENSE_UNION

2020-05-19 Thread Wes McKinney (Jira)
Wes McKinney created ARROW-8866: --- Summary: [C++] Split Type::UNION into Type::SPARSE_UNION and Type::DENSE_UNION Key: ARROW-8866 URL: https://issues.apache.org/jira/browse/ARROW-8866 Project: Apache

[jira] [Created] (ARROW-8865) windows distribution for 0.17.1 seems broken (conda only?

2020-05-19 Thread Maarten Breddels (Jira)
Maarten Breddels created ARROW-8865: --- Summary: windows distribution for 0.17.1 seems broken (conda only? Key: ARROW-8865 URL: https://issues.apache.org/jira/browse/ARROW-8865 Project: Apache Arrow

[jira] [Created] (ARROW-8864) [R] Add methods to Table/RecordBatch for consistency with data.frame

2020-05-19 Thread Neal Richardson (Jira)
Neal Richardson created ARROW-8864: -- Summary: [R] Add methods to Table/RecordBatch for consistency with data.frame Key: ARROW-8864 URL: https://issues.apache.org/jira/browse/ARROW-8864 Project:

[jira] [Created] (ARROW-8862) NumericBuilder does not use MemoryPool passed to CTOR

2020-05-19 Thread Simon Watts (Jira)
Simon Watts created ARROW-8862: -- Summary: NumericBuilder does not use MemoryPool passed to CTOR Key: ARROW-8862 URL: https://issues.apache.org/jira/browse/ARROW-8862 Project: Apache Arrow Issue

Re: Sparse Union format

2020-05-19 Thread Micah Kornfield
Hi Ryan, In addition to the limitations mentioned above another one is only 1 column of each type that can participate in the union. There are some old threads on these differences on the mailing list that should be searchable. Thanks, Micah On Tue, May 19, 2020 at 6:44 AM Antoine Pitrou

[jira] [Created] (ARROW-8861) Memory not released until Plasma process is killed

2020-05-19 Thread Chengxin Ma (Jira)
Chengxin Ma created ARROW-8861: -- Summary: Memory not released until Plasma process is killed Key: ARROW-8861 URL: https://issues.apache.org/jira/browse/ARROW-8861 Project: Apache Arrow Issue

[jira] [Created] (ARROW-8860) [C++] Compressed Feather file with struct array roundtrips incorrectly

2020-05-19 Thread Joris Van den Bossche (Jira)
Joris Van den Bossche created ARROW-8860: Summary: [C++] Compressed Feather file with struct array roundtrips incorrectly Key: ARROW-8860 URL: https://issues.apache.org/jira/browse/ARROW-8860

Re: Sparse Union format

2020-05-19 Thread Antoine Pitrou
Also, you may want to run the integration tests and inspect the generated JSON file for union data, it will probably be informative (look for type ids). Regards Antoine. Le 19/05/2020 à 15:38, Ryan Murray a écrit : > Thanks for the clarification! Next time I will read the whole document ;-)

Re: Sparse Union format

2020-05-19 Thread Ryan Murray
Thanks for the clarification! Next time I will read the whole document ;-) On Tue, May 19, 2020 at 2:38 PM Antoine Pitrou wrote: > > As explained in the comment below: > https://github.com/apache/arrow/blob/master/format/Schema.fbs#L91 > > Regards > > Antoine. > > > Le 19/05/2020 à 14:14, Ryan

[jira] [Created] (ARROW-8859) [Rust] [Integration Testing] Implement --quiet / verbose correctly

2020-05-19 Thread Andy Grove (Jira)
Andy Grove created ARROW-8859: - Summary: [Rust] [Integration Testing] Implement --quiet / verbose correctly Key: ARROW-8859 URL: https://issues.apache.org/jira/browse/ARROW-8859 Project: Apache Arrow

Re: Sparse Union format

2020-05-19 Thread Antoine Pitrou
As explained in the comment below: https://github.com/apache/arrow/blob/master/format/Schema.fbs#L91 Regards Antoine. Le 19/05/2020 à 14:14, Ryan Murray a écrit : > Thanks Antoine, > > Can you just clarify what you mean by 'type ids are logical'? In my mind > type ids are strongly coupled

Re: [VOTE] Release Apache Arrow 0.17.1 - RC1

2020-05-19 Thread Uwe L. Korn
Current status: 1. [done] rebase (not required for a patch release) 2. [done] upload source 3. [done] upload binaries 4. [done|in-pr] update website 5. [done] upload ruby gems 6. [ ] upload js packages 8. [done] upload C# packages 9. [ ] upload rust crates 10. [done] update conda recipes

Re: Sparse Union format

2020-05-19 Thread Ryan Murray
Thanks Antoine, Can you just clarify what you mean by 'type ids are logical'? In my mind type ids are strongly coupled to the types and their order in Schema.fbs [1]. Do you mean that the order there is only a convention and we can't assume that 0 === Null? Best, Ryan [1]

Re: Sparse Union format

2020-05-19 Thread Antoine Pitrou
Le 19/05/2020 à 13:43, Ryan Murray a écrit : > Hey All, > > While working on https://issues.apache.org/jira/browse/ARROW-1692 I noticed > that there is a difference between C++ and Java on the way Sparse Unions > are handled. I haven't seen in the format spec which the correct is so I > wanted

[jira] [Created] (ARROW-8858) [FlightRPC] Ensure headers are uniformly exposed

2020-05-19 Thread David Li (Jira)
David Li created ARROW-8858: --- Summary: [FlightRPC] Ensure headers are uniformly exposed Key: ARROW-8858 URL: https://issues.apache.org/jira/browse/ARROW-8858 Project: Apache Arrow Issue Type: Bug

Sparse Union format

2020-05-19 Thread Ryan Murray
Hey All, While working on https://issues.apache.org/jira/browse/ARROW-1692 I noticed that there is a difference between C++ and Java on the way Sparse Unions are handled. I haven't seen in the format spec which the correct is so I wanted to check with the wider community. c++ (and the

[NIGHTLY] Arrow Build Report for Job nightly-2020-05-19-0

2020-05-19 Thread Crossbow
Arrow Build Report for Job nightly-2020-05-19-0 All tasks: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-05-19-0 Failed Tasks: - conda-linux-gcc-py36: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-05-19-0-azure-conda-linux-gcc-py36 -