Re: [VOTE] Move Variant to Parquet

2024-09-03 Thread Micah Kornfield
ared in the near future. > > Thanks, > Gene > > On Tue, Sep 3, 2024 at 10:28 AM Micah Kornfield > wrote: > >> I think maybe we should finalize the details before having a vote, to >> make sure everyone understands the implications? >> >> On Tue, Sep 3, 2024

Re: [VOTE] Move Variant to Parquet

2024-09-03 Thread Micah Kornfield
I think maybe we should finalize the details before having a vote, to make sure everyone understands the implications? On Tue, Sep 3, 2024 at 9:12 AM Gene Pang wrote: > Hi, > > In general, the Iceberg community is in favor of moving it to Parquet, and > the Parquet community is in support of rec

Re: Write to big query via bq connector is slow

2022-07-05 Thread Micah Kornfield
Hi Abhinav, This question is probably best asked for the Spark BigQuery connector project you linked. It is not part of the core Spark implementation. Cheers, Micah On Tue, Jun 28, 2022 at 12:12 AM Abhinav Ranjan wrote: > Hi, > > I am trying to write to a big query table via the bq-spark-conn

Re: Nabble archive is down

2021-08-17 Thread Micah Kornfield
https://lists.apache.org/list.html?u...@spark.apache.org should be searchable (although the UI is a little clunky). On Tue, Aug 17, 2021 at 12:52 PM Sean Owen wrote: > If the links are down and not evidently coming back, yeah let's change any > website links. Probably best to depend on ASF resou

Contract for PartitionReader/InputPartition for ColumnarBatch?

2020-06-27 Thread Micah Kornfield
Hello spark-dev, Looking at ColumnarBatch [1] it seems to indicate a single object is meant to be used for the entire loading process. Does this imply that Spark assumes the ColumnarBatch and any direct references to ColumnarBatch (e.g. UTF8Strings) returned by InputPartitionReader/PartitionReade

Re: Apache Arrow data in buffer to RDD/DataFrame/Dataset?

2016-08-05 Thread Micah Kornfield
Hi Everyone, I'm an Arrow contributor mostly on the C++ side of things, but I'll try to give a brief update of where I believe the project currently is (the views are my own, but hopefully are fairly accurate :). I think in the long run the diagram mentioned by Jim, is were we would like Arrow to