Re: [DISCUSS] Ongoing LZ4 problems with Parquet files

2020-07-02 Thread Christian Hudon
s available, and they span different parts of > the > > > > speed/compression spectrum, so it would be a pity to disable one of > them. > > > > > > It's true, however I think it's worse to write LZ4-compressed files > > > that cannot be read by other Parquet implementations (if that's what's > > > happening as I understand it?). If we are indeed shipping something > > > broken then we either should fix it or disable it until it can be > > > fixed. > > > > > > > Regards > > > > > > > > Antoine. > > > > > > > > > -- │ Christian Hudon │ Applied Research Scientist Element AI, 6650 Saint-Urbain #500 Montréal, QC, H2S 3G9, Canada Elementai.com

Re: Arrow for low-latency streaming of small batches?

2020-07-02 Thread Christian Hudon
gt; > rows (constructing and destructing builders, making Tables that can > only be > > > used once since they're immutable, etc). If the batches weren't so > small it > > > would probably make sense, but as is it's unworkable. I need to add > rows to > > &

[jira] [Created] (ARROW-8952) [C++] Support for textual, JSON schema representation

2020-05-26 Thread Christian Hudon (Jira)
Christian Hudon created ARROW-8952: -- Summary: [C++] Support for textual, JSON schema representation Key: ARROW-8952 URL: https://issues.apache.org/jira/browse/ARROW-8952 Project: Apache Arrow

Re: Re: Human-readable version of Arrow Schema

2020-05-08 Thread Christian Hudon
"m2": "meta 2", > "m3": "meta 3" > }, > "children": [] > } > ], > "

[jira] [Created] (ARROW-8714) [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType

2020-05-06 Thread Christian Hudon (Jira)
Christian Hudon created ARROW-8714: -- Summary: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType Key: ARROW-8714 URL: https://issues.apache.org/jira/browse/ARROW-8714

Re: Human-readable version of Arrow Schema?

2020-05-05 Thread Christian Hudon
ffer: offset=36416, length=384}, > > {Buffer: offset=36800, length=48000}, {Buffer: offset=84800, > > length=0}, {Buffer: offset=84800, length=384}, {Buffer: offset=85184, > > length=12032}, {Buffer: offset=97216, length=384}, {Buffer: > > offset=97600, length=12032}, {Buffer

[jira] [Created] (ARROW-7585) Plasma-store-server does not support --help, shows backtrace on getopt error

2020-01-15 Thread Christian Hudon (Jira)
Christian Hudon created ARROW-7585: -- Summary: Plasma-store-server does not support --help, shows backtrace on getopt error Key: ARROW-7585 URL: https://issues.apache.org/jira/browse/ARROW-7585

Re: Human-readable version of Arrow Schema?

2019-12-10 Thread Christian Hudon
> > readable schema. I could see adding a helper method someplace under > IPC for > > > this. Would that meet your needs? I think if there are other > > > requirements, then a proposal would be welcome. Ideally, a solution > would > > > not require additional build/run

Human-readable version of Arrow Schema?

2019-12-06 Thread Christian Hudon
d you folks be interested in me putting up an initial proposal for discussion? Any design constraints I should pay attention to, then? Thanks, Christian -- │ Christian Hudon │ Applied Research Scientist Element AI, 6650 Saint-Urbain #500 Montréal, QC, H2S 3G9, Canada Elementai.com

[jira] [Created] (ARROW-7041) PythonLibs setting found by CMake uses wrong version of Python on macOS

2019-10-31 Thread Christian Hudon (Jira)
Christian Hudon created ARROW-7041: -- Summary: PythonLibs setting found by CMake uses wrong version of Python on macOS Key: ARROW-7041 URL: https://issues.apache.org/jira/browse/ARROW-7041 Project

[jira] [Created] (ARROW-7033) Error in./configure step for jemalloc when building on OSX 10.14.6

2019-10-30 Thread Christian Hudon (Jira)
Christian Hudon created ARROW-7033: -- Summary: Error in./configure step for jemalloc when building on OSX 10.14.6 Key: ARROW-7033 URL: https://issues.apache.org/jira/browse/ARROW-7033 Project: Apache