Re: support dense union type in parquet

Thanks a lot for your rapid reply and direction. Let's see whether we can do for this. Best Regards, Zimo Zhang Micah Kornfield 于2021年4月24日周六 下午12:31写道： > Hi Zimo, > I don't think anybody is actively working on it. One of the challenges > would be how to represent a Union type in Parquet

Re: support dense union type in parquet

Hi Zimo, I don't think anybody is actively working on it. One of the challenges would be how to represent a Union type in Parquet because to the best of my knowledge parquet doesn't support this type. So if this is something you would like to contribute, proposing a mapping to and from the

support dense union type in parquet

Hi there, Dense union type is not supported yet in Parquet https://github.com/apache/arrow/blob/8e43f23dcc6a9e630516228f110c48b64d13cec6/cpp/src/parquet/arrow/schema.cc#L382 . Is it planned to support it nearly? And if I'd implement this myself, is there any other codes to take care of?

Re: [Python] Custom Metadata in PyArrow

On Fri, Apr 23, 2021 at 3:06 PM Weston Pace wrote: > > I have used the custom metadata feature in the past. I used it to > track (for example) which variables were independent variables and > which were dependent variables. This was used as input for later > tools to help present the data. > >

Re: [Python] Custom Metadata in PyArrow

I have used the custom metadata feature in the past. I used it to track (for example) which variables were independent variables and which were dependent variables. This was used as input for later tools to help present the data. > Is that how most people handle metadata they create the schema

Re: [Go, Rust] inter-language Arrow API compatibility

There is a matrix of features missing from each language [1][2]. [1] https://arrow.apache.org/docs/status.html [2] From master (soon to be 4.0.0 release) https://github.com/apache/arrow/blob/31c8c750513f8c75a5e76a267354cb897c56468e/docs/source/status.rst On Fri, Apr 23, 2021 at 11:51 AM Jorge

Re: [Go, Rust] inter-language Arrow API compatibility

Hi Agam, We have integration tests that produce data from Rust and consumes from Go, and vice-versa, for both IPC files and streams, so it is reasonable to expect that they work. Deviations are considered major out-of-spec bugs, which we often prioritize over e.g. new features. Best, Jorge On

[Go, Rust] inter-language Arrow API compatibility

Hi, Thanks for all the tips on the Go-API usage, I have the serialization going great there. I have a question about the compatibility of reader/writer API in different languages: if I have Rust services that also read/write or pass-through serialized arrow data, are there particular

Re: Arrow JS Meetup (02/13)

Thanks Dominik. Regards, Naveen On Tue, Apr 20, 2021 at 11:38 AM Dominik Moritz wrote: > Hi Naveen, > > I don’t know whether Brian took separate notes but here is the outline > with some links: > https://docs.google.com/document/d/1IPd9AS_0RxYOH8WiKWcqqaMoGW96xnC17cHxGwl6Ryc/edit?usp=sharing.

Re: apache arrow js documentation?

I may have figured out my issue: // 1. slowest based on average time of several runs with cache cleared before each const table = await Table.from(fetch('/table.ipc')); // 2. 5-10% faster than 1 const table = await Table.from(fetch('/table.ftr')); // 3. 10-15% faster than 2, presumably because

Re: apache arrow js documentation?

The documentation, from my understanding, is basically spread around 1. Within the actual files https://github.com/apache/arrow/blob/master/js/src/table.ts#L118 2. Within the examples/unit test https://github.com/apache/arrow/blob/master/js/test/unit/ipc/reader/stream-reader-tests.ts

RE: [Go] Flight client app metadata access

I have put up: https://github.com/apache/arrow/pull/10142 We can shift any further discussion to the Github PR / Jira card now. -Original Message- From: Matthew Topol Sent: Friday, April 23, 2021 10:07 AM To: 'dev@arrow.apache.org' Subject: RE: [Go] Flight client app metadata access

apache arrow js documentation?

Hey, Great project, super-useful. Is there some hidden, secret documentation somewhere that I'm just not finding? Here is what I know about so far: 1. https://arrow.apache.org/docs/js/ "Apache Arrow in JS" and the right sidebar of enum, class, interface definitions, etc... Clicking the Builder

RE: [Go] Flight client app metadata access

Awesomeness thanks much. I should hopefully be able to pick this up in the next day or so, so watch for the PR on the Jira card so you can comment on it (or just toss your Github username on here and I'll tag you if you like). --Matt -Original Message- From: Paul Whalen Sent:

[Python] Custom Metadata in PyArrow

Hello Team, The docs for Custom Metadata in PyArrow say TODO https://arrow.apache.org/docs/python/data.html#custom-schema-and-field-metadata So I am wondering if someone has any example of adding some custom metadata to PyArrow. I tried looking through the pyarrow github repo and looking at

Re: RE: [Go] expose ability to write arrow.Table to JSON

You can either use the provided server facility found in flight [1], or use stream directly via ipc [2]. You can look at the tests on how to use both facilities. François [1] https://github.com/apache/arrow/tree/master/go/arrow/flight [2] https://github.com/apache/arrow/tree/master/go/arrow/ipc

[NIGHTLY] Arrow Build Report for Job nightly-2021-04-23-0

Arrow Build Report for Job nightly-2021-04-23-0 All tasks: https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-23-0 Failed Tasks: - conda-linux-gcc-py36-arm64: URL: