Based on the response to using an empty IPC stream/file, it sounds to me like something substrait-like is ideal. Maybe an interface that can go between the equivalent of relational schemas and (generated) arrow code as you have shown. Then, there could be straightforward integration points with other libraries such as ibis or ones like Weston just mentioned (pydantic). But I think this leads me to the short answer of: I also don't know anyone working in the direction of what you're describing (or at least on something that satisfies what you want). Sent from Proton Mail for iOS On Mon, Jul 8, 2024 at 12:19, Ian Cook <ianmc...@apache.org> wrote: This has come up a few times in the past [1][2]. The main concern has been about cross-version compatibility guarantees.
[1] https://github.com/apache/arrow/issues/25078 [2] https://lists.apache.org/thread/02p37yxksxccsqfn9l6j4ryno404ttnl On Mon, Jul 8, 2024 at 3:10 PM Lee, David (PAG) <david....@blackrock.com.invalid> wrote: > Gah found a bug with my code.. Here's a corrected python version.. > > # iterate through possible nested columns > def _convert_to_arrow_type(field, obj): > """ > :param field: > :param obj: > :returns: pyarrow datatype > > """ > if isinstance(obj, list): > for child_obj in obj: > pa_type = _convert_to_arrow_type(field, child_obj) > return pa.list_(pa_type) > elif isinstance(obj, dict): > items = [] > for k, child_obj in obj.items(): > pa_type = _convert_to_arrow_type(k, child_obj) > items.append((k, pa_type)) > return pa.struct(items) > else: > if isinstance(obj, str): > if obj == "timestamp": > # default timestamp to microsecond precision > obj = "timestamp[us]" > elif obj == "date": > # default date to date32 which is an alias for date32[day] > obj = "date32" > elif obj == "int": > # default int to int32 > obj = "int32" > obj = pa.type_for_alias(obj) > return obj > > # iterate through columns to create a schema > def _convert_to_arrow_schema(fields_dict): > """ > > :param fields_dict: > :returns: pyarrow schema > > """ > columns = [] > for field, typ in fields_dict.items(): > pa_type = _convert_to_arrow_type(field, typ) > columns.append(pa.field(field, pa_type)) > schema = pa.schema(columns) > return schema > > -----Original Message----- > From: Lee, David (PAG) <david....@blackrock.com.INVALID> > Sent: Monday, July 8, 2024 11:58 AM > To: dev@arrow.apache.org > Subject: RE: [DISCUSS] Approach to generic schema representation > > External Email: Use caution with links and attachments > > > I came up with my own json representation that I could put into json / > yaml config files with some python code to convert this into a pyarrow > schema object.. > > ------------- yaml flat example------------- > fields: > cusip: string > start_date: date32 > end_date: date32 > purpose: string > source: string > flow: float32 > flow_usd: float32 > currency: string > > -------------yaml nested example------------- > fields: > cusip: string > start_date: date32 > regions: > [string] << list of strings > primary_benchmark: << struct > id: string > name: string > all_benchmarks: << list of structs > - > id: string > name: string > > Code: > > def _convert_to_arrow_type(field, obj): > """ > :param field: > :param obj: > :returns: pyarrow datatype > > """ > if isinstance(obj, list): > for child_obj in obj: > pa_type = _convert_to_arrow_type(field, child_obj) > return pa.list_(pa_type) > elif isinstance(obj, dict): > items = [] > for k, child_obj in obj.items(): > pa_type = _convert_to_arrow_type(k, child_obj) > items.append((k, pa_type)) > return pa.struct(items) > else: > if isinstance(obj, str): > obj = pa.type_for_alias(obj) > return obj > > > def _convert_to_arrow_schema(fields_dict): > """ > > :param fields_dict: > :returns: pyarrow schema > > """ > columns = [] > for field, typ in fields_dict.items(): > if typ == "timestamp": > # default timestamp to microsecond precision > typ = "timestamp[us]" > elif typ == "date": > # default date to date32 which is an alias for date32[day] > typ = "date32" > elif typ == "int": > # default int to int32 > typ = "int32" > pa_type = _convert_to_arrow_type(field, typ) > columns.append(pa.field(field, pa_type)) > schema = pa.schema(columns) > return schema > > -----Original Message----- > From: Weston Pace <weston.p...@gmail.com> > Sent: Monday, July 8, 2024 9:43 AM > To: dev@arrow.apache.org > Subject: Re: [DISCUSS] Approach to generic schema representation > > External Email: Use caution with links and attachments > > > +1 for empty stream/file as schema serialization. I have used this > approach myself on more than one occasion and it works well. It can even > be useful for transmitting schemas between different arrow-native libraries > in the same language (e.g. rust->rust) since it allows the different > libraries to use different arrow versions. > > There is one other approach if you only need intra-process serialization > (e.g. between threads / libraries in the same process). You can use the C > data interface ( > https://urldefense.com/v3/__https://arrow.apache.org/docs/format/CDataInterface.html__;!!KSjYCgUGsB4!ZpcpNRWAd5SeffO0-cFZpVsg1ze7lbt7Btmp3SdyCqvZcsa1naBsVkk2SXPTgQpHRR-fJd_bupsM0-v2oXAljCk$ > ). > It is maybe a slightly more complex API (because of the release callback) > and I think it is unlikely to be significantly faster (unless you have an > abnormally large schema). However, it has the same advantages and might be > useful if you are already using the C data interface elsewhere. > > > On Mon, Jul 8, 2024 at 8:27 AM Matt Topol <zotthewiz...@gmail.com> wrote: > > > Hey Jeremy, > > > > Currently the first message of an IPC stream is a Schema message which > > consists solely of a flatbuffer message and defined in the Schema.fbs > > file of the Arrow repo. All of the libraries that can read Arrow IPC > > should be able to also handle converting a single IPC schema message > > back into an Arrow schema without issue. Would that be sufficient for > you? > > > > On Mon, Jul 8, 2024 at 11:12 AM Jeremy Leibs <jer...@rerun.io> wrote: > > > > > I'm looking for any advice folks may have on a generic way to > > > document > > and > > > represent expected arrow schemas as part of an interface definition. > > > > > > For context, our library provides a cross-language (python, c++, > > > rust) > > SDK > > > for logging semantic multi-modal data (point clouds, images, > > > geometric transforms, bounding boxes, etc.). Each of these primitive > > > types has an associated arrow schema, but to date we have largely > > > abstracted that from our users through language-native object types, > > > and a bunch of generated code to "serialize" stuff into the arrow > > > buffers before transmitting via our IPC. > > > > > > We're trying to take steps in the direction of making it easier for > > > advanced users to write and read data from the store directly using > > arrow, > > > without needing to go in-and-out of an intermediate object-oriented > > > representation. However, doing this means documenting to users, for > > > example: "This is the arrow schema to use when sending a point cloud > > with a > > > color channel". > > > > > > I would love it if, eventually, the arrow project had a way of > > > defining a spec file similar to a .proto or a .fbs, with all > > > libraries supporting loading of a schema object by directly parsing > > > the spec. Has anyone taken steps in this direction? > > > > > > The best alternative I have at the moment is to redundantly define > > > the schema for each of the 3 languages implicitly by directly > > > providing the code to construct a datatype instance with the correct > > > schema. But this feels unfortunately messy and hard to maintain. > > > > > > Thanks, > > > Jeremy > > > > > > > > This message may contain information that is confidential or privileged. > If you are not the intended recipient, please advise the sender immediately > and delete this message. See > http://www.blackrock.com/corporate/compliance/email-disclaimers for > further information. Please refer to > http://www.blackrock.com/corporate/compliance/privacy-policy for more > information about BlackRock’s Privacy Policy. > > > For a list of BlackRock's office addresses worldwide, see > http://www.blackrock.com/corporate/about-us/contacts-locations. > > © 2024 BlackRock, Inc. All rights reserved. >
signature.asc
Description: OpenPGP digital signature
