Re: [VOTE][RUST] Release Apache Arrow Rust 52.0.0 RC1

2024-06-04 Thread Andrew Lamb
+1 (binding) Verified on M3 mac. Thank you - this one is a good one. Andrew On Mon, Jun 3, 2024 at 12:18 PM Andy Grove wrote: > +1 (binding) > > Verified on Ubuntu 22.04.4 LTS. > > Thanks, Raphael. > > On Mon, Jun 3, 2024 at 10:12 AM L. C. Hsieh wrote: > > > +1 (binding) > > > > Verified on

Re: [DISCUSS] Move sqlparser-rs back into DataFusion project?

2024-05-30 Thread Andrew Lamb
t; project, > > and that it continues to have its own release process. > > > > The main benefit here is that it would bring it under ASF governance and > > allow those who have permission from their employers to contribute to > > Apache Arrow/DataFusion to be able to help with the mai

Re: [DISCUSS] Migration of parquet-cpp issues to GitHub

2024-05-28 Thread Andrew Lamb
I think it is a great idea -- github has served arrow (and datafusion) very well in my opinion. Specifically, having to sign up for a JIRA account (which can not be created self-service) adds a small, but real barrier to engagement and contribution. Removing the barrier I think encourages more

Re: [VOTE][RUST] Release Apache Arrow Rust Object Store 0.10.1 RC1

2024-05-11 Thread Andrew Lamb
+1 (binding) Thank you for the quick release and response Verified on M3 Mac On Fri, May 10, 2024 at 2:00 PM L. C. Hsieh wrote: > +1 (binding) > > Verified on M3 Mac. > > Thanks Raphael. > > On Fri, May 10, 2024 at 10:31 AM Raphael Taylor-Davies > wrote: > > > > Hi, > > > > I would like to

Re: Fwd: [C++] Parquet and Arrow overlap

2024-05-11 Thread Andrew Lamb
It is great to see some additional enthusiasm and momentum around the Apache Parquet implementation (congratulations on the release of parquet-mr 1.14[1]!). As activity picks up, if the desire is to build more community around Parquet, perhaps the Parquet PMC wants to encourage moving code back

[CROWDSOURCING] May 2024 ASF Board Report

2024-04-30 Thread Andrew Lamb
As part of being a new project, we need to submit reports to the board every month for the first three months[1]. In the tradition of Apache Arrow, I hope the community can help draft this report. Please take a look and add anything that might be relevant[2]. Thanks, Andrew [1]:

[RESULT] [VOTE][RUST][DataFusion] Release Apache Arrow DataFusion 37.1.0 RC2

2024-04-22 Thread Andrew Lamb
; > > Verified on Ubuntu 24.04.4 LTS. > > > > Thanks, Andrew. > > > > On Thu, Apr 18, 2024 at 8:04 PM L. C. Hsieh wrote: > > > > > +1 (binding) > > > > > > Verified on M3 Mac. > > > > > > Thanks Andrew. > > > &

Re: [VOTE][RUST][DataFusion] Release Apache Arrow DataFusion 37.1.0 RC1

2024-04-18 Thread Andrew Lamb
Unfortunately it appears I made an error and forgot to update the release version in this RC I have started a new thread[1] for a second RC Andrew [1] https://lists.apache.org/thread/0md6qyhw0hody8p0v9wddvt7vo8r8z2x On Thu, Apr 18, 2024 at 2:23 PM Andrew Lamb wrote: > Thanks Andy, J

[VOTE][RUST][DataFusion] Release Apache Arrow DataFusion 37.1.0 RC2

2024-04-18 Thread Andrew Lamb
I would like to propose a release of Apache Arrow DataFusion Implementation, version 37.1.0. Note this is the second RC (the first RC[4] did not include the change to the version numbers[5] :facepalm:). I apologize for the runaround. This release candidate is based on commit:

Re: [VOTE][RUST][DataFusion] Release Apache Arrow DataFusion 37.1.0 RC1

2024-04-18 Thread Andrew Lamb
or your work and efforts. > > > > > > > > On Thu, Apr 18, 2024 at 10:01 PM Andrew Lamb > wrote: > > > > > Hi, > > > > > > I would like to propose a release of Apache Arrow DataFusion > > > Implementation, > > > version 37.1

Re: [VOTE][RUST] Release Apache Arrow Rust Object Store 0.10.0 RC1

2024-04-18 Thread Andrew Lamb
+1 (binding) I reviewed the breaking API changes and the changelog and ran the verification scripts Thank you Raphael Andrew On Thu, Apr 18, 2024 at 6:55 AM Raphael Taylor-Davies wrote: > Hi, > > I would like to propose a release of Apache Arrow Rust Object > Store Implementation, version

[VOTE][RUST][DataFusion] Release Apache Arrow DataFusion 37.1.0 RC1

2024-04-18 Thread Andrew Lamb
Hi, I would like to propose a release of Apache Arrow DataFusion Implementation, version 37.1.0, a patch release with some bug fixes. Please see [4] for details. There is a failing CI test which only affects development tools [6]. While DataFusion is now officially its own top level Apache

Re: [RESULT] [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project

2024-04-18 Thread Andrew Lamb
ache/arrow-datafusion/issues/9691 On Sat, Mar 9, 2024 at 3:23 AM Andrew Lamb wrote: > With 30 +1 votes (18 binding by my count), and 0 -1 votes, the proposal is > approved > > Thank you everyone who voted and participated in helping form the proposal > (and the community over the

[DataFusion] 37.1.0 maintenance release

2024-04-16 Thread Andrew Lamb
We are making a 37.1.0 maintenance release[1]. If anyone wants to add anything or review the PRs please feel free. Andrew [1] https://github.com/apache/arrow-datafusion/issues/9904

Re: [ANNOUNCE] New Arrow committer: Sarah Gilmore

2024-04-11 Thread Andrew Lamb
Welcome! On Thu, Apr 11, 2024 at 2:43 PM Ian Cook wrote: > Congrats Sarah! > > On Thu, Apr 11, 2024 at 12:31 Bryce Mecum wrote: > > > Congratulations! > > > > On Thu, Apr 11, 2024 at 3:13 AM Sutou Kouhei wrote: > > > > > > Hi, > > > > > > On behalf of the Arrow PMC, I'm happy to announce that

Re: [DISCUSS] Versioning and releases for apache/arrow components

2024-04-07 Thread Andrew Lamb
I agree with all the other comments on this thread Having smaller releases is key to being able to release more frequently and finding the relevant expertise in my opinion. We have had separate releases / votes for Arrow Rust (and Arrow DataFusion) and it has served us quite well. The version

Re: [VOTE] Bulk ingestion support for Flight SQL (vote #2)

2024-04-06 Thread Andrew Lamb
+1 On Sat, Apr 6, 2024 at 3:48 AM wish maple wrote: > +1 (non binding) > > Best, > Xuwei Fu > ulk ingestion support for Flight SQL > > David Li 于2024年4月5日周五 16:38写道: > > > Hello, > > > > Joel Lubinitsky has proposed adding bulk ingestion support to Arrow > Flight > > SQL [1]. This provides a

Re: [VOTE] Add new info codes and options keys to ADBC specification

2024-04-06 Thread Andrew Lamb
+1 On Fri, Apr 5, 2024 at 9:55 PM Jacob Wujciak wrote: > + 1 (non-binding) > > Am Sa., 6. Apr. 2024 um 01:57 Uhr schrieb Joel Lubinitsky < > joell...@gmail.com>: > > > Yes, just updated both the issue and the PR. > > > > Thanks, > > Joel > > > > On Fri, Apr 5, 2024 at 7:51 PM Sutou Kouhei

Re: [RESULT][VOTE][RUST][DataFusion] Release Apache Arrow DataFusion 37.0.0 RC2

2024-04-04 Thread Andrew Lamb
The release is available here: https://dist.apache.org/repos/dist/release/arrow/arrow-datafusion-37.0.0 The release has also been published to crates.io: https://crates.io/crates/datafusion/37.0.0 Thank you all for your help! Andrew On Thu, Apr 4, 2024 at 4:31 PM Andrew Lamb wrote

[RESULT][VOTE][RUST][DataFusion] Release Apache Arrow DataFusion 37.0.0 RC2

2024-04-04 Thread Andrew Lamb
With 4 +1 votes (3 binding) the release is approved. I will now upload to crates.io On Mon, Apr 1, 2024 at 11:59 AM Andrew Lamb wrote: > +1 > > I verified on M3 mac > > Thanks Andy for the quick turnaround on making a new RC > > > Andrew > > > On Mon, Apr 1, 202

Re: [VOTE][RUST][DataFusion] Release Apache Arrow DataFusion 37.0.0 RC2

2024-04-01 Thread Andrew Lamb
+1 I verified on M3 mac Thanks Andy for the quick turnaround on making a new RC Andrew On Mon, Apr 1, 2024 at 1:51 AM Jean-Baptiste Onofré wrote: > +1 (non binding) > > - Hashed and signatures are OK > - ASF headers are present on expected file > - No binary file found in the source

Re: [ANNOUNCE] New Committer Joel Lubinitsky

2024-04-01 Thread Andrew Lamb
Congratulations Joel. On Mon, Apr 1, 2024 at 11:53 AM Raúl Cumplido wrote: > Congratulations and welcome Joel! > > > El lun, 1 abr 2024, 17:18, Kevin Gurney > escribió: > > > Congratulations, Joel! > > > > > > From: Jason Z > > Sent: Monday, April 1, 2024

Re: [VOTE][RUST][DataFusion] Release Apache Arrow DataFusion 37.0.0 RC1

2024-03-30 Thread Andrew Lamb
#issuecomment-2028018095 [3]: https://github.com/apache/arrow-datafusion/pull/9871 On Fri, Mar 29, 2024 at 12:09 PM Wayne Xia wrote: > +1 (non-binding) > > Same here on x86 Linux. > > Thanks. > > On Fri, Mar 29, 2024 at 7:22 PM Andrew Lamb wrote: > > > +1 (binding) > >

Re: [VOTE][RUST][DataFusion] Release Apache Arrow DataFusion 37.0.0 RC1

2024-03-29 Thread Andrew Lamb
+1 (binding) Thank you Andy. I likewise had to up the min stack size. FYI we (InfluxData) are still chasing down what we think may be a regression introduced by the TreeNode refactor [1] for certain queries. If this turns out to be the case I will coordinate making a patch release. Andrew [1]

Re: [VOTE] Stateless prepared statements in FlightSQL

2024-03-21 Thread Andrew Lamb
at 10:04 PM Adam C wrote: > Hello, > > I would like to propose a change to the FlightSQL specification as > originally described in this Github issue [1] by Andrew Lamb. The > specification change would allow servers to support prepared > statements with parameters, without needi

Re: [ANNOUNCE] New Arrow committer: Bryce Mecum

2024-03-18 Thread Andrew Lamb
Congratulations Bryce! On Mon, Mar 18, 2024 at 3:35 AM Alenka Frim wrote: > Congratulations Bryce and thank you for all your contributions!! > > On Mon, Mar 18, 2024 at 6:43 AM Raúl Cumplido > wrote: > > > Congratulations Bryce!!! > > > > El lun, 18 mar 2024, 5:21, Anja escribió: > > > > >

Re: [VOTE][RUST] Release Apache Arrow Rust 51.0.0 RC1

2024-03-15 Thread Andrew Lamb
+1 (binding) I verified on an M3 mac. Thank you Raphael Looks like a great release to me. I reviewed the changelog. Given there were so few breaking changes, it seems like we are very close to being able to do incremental releases (e.g. 51.1.0) which is a sign of maturity in my mind. Andrew On

Re: [DISCUSS] Conventions for transporting Arrow data over HTTP

2024-03-11 Thread Andrew Lamb
Update -- turns out there was already a Rust client/server -- linked to the ticket now On Mon, Mar 11, 2024 at 3:07 PM Andrew Lamb wrote: > I sadly don't have time to help with this directly, however, I did file a > ticket with the request to help with a Rust prototype [1]. Hopefully

Re: [DISCUSS] Conventions for transporting Arrow data over HTTP

2024-03-11 Thread Andrew Lamb
I sadly don't have time to help with this directly, however, I did file a ticket with the request to help with a Rust prototype [1]. Hopefully we'll get a taker [1] https://github.com/apache/arrow-rs/issues/5496 On Tue, Mar 5, 2024 at 11:03 PM Ian Cook wrote: > Update on recent progress in

Re: [DISCUSS] [FlightSQL] Supporting parameters and prepared statements with a stateless server

2024-03-11 Thread Andrew Lamb
We are working to incorporate all the feedback we have received so far -- thank you to everyone who commented. If you would like to participate in this discussion, please leave your comments in github. I expect we'll call a formal vote later in the week Andrew On Sun, Mar 3, 2024 at 3:26 AM

[RESULT] [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project

2024-03-09 Thread Andrew Lamb
; > > > > > > > > > On Fri, Mar 1, 2024 at 5:49 PM Micah Kornfield < > > > > > > > > emkornfi...@gmail.com> > > > > > > > > > > > wrote: > > > > > > > > > >

Re: [DISCUSS] Donation of a Spark native engine based on DataFusion & Arrow

2024-03-06 Thread Andrew Lamb
The blog post is now live at https://arrow.apache.org/blog/2024/03/06/comet-donation/ On Thu, Feb 29, 2024 at 9:32 AM Andrew Lamb wrote: > In case anyone is interested, we are working on a blog post related to > this donation here [1]. All feedback more than welcome. > > [1] https:

Re: [DISCUSS] [FlightSQL] Supporting parameters and prepared statements with a stateless server

2024-03-03 Thread Andrew Lamb
Thanks a lot for pulling this together Adam. I likewise took a look at the proposal and I think it looks great (disclaimer, I wrote the original issue, so I am not unbiased). I reviewed the spec proposal and the Rust implementation. If no one has time to review the go implementation, I can take a

Re: [VOTE][RUST] Release Apache Arrow Rust Object Store 0.9.1 RC1

2024-03-01 Thread Andrew Lamb
+1 (binding) Verified on M3 mac Thank you Raphael Andrew On Fri, Mar 1, 2024 at 2:45 AM L. C. Hsieh wrote: > > +1 (binding) > > Verified on M1 Mac. > > Thanks Raphael. > > > On Thu, Feb 29, 2024 at 11:10 PM Raphael Taylor-Davies > wrote: > > > > Hi, > > > > I would like to propose a release

[VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project

2024-03-01 Thread Andrew Lamb
the initial members of the Apache DataFusion Project: * Andy Grove (agr...@apache.org) * Andrew Lamb (al...@apache.org) * Daniël Heres (dhe...@apache.org) * Jie Wen (jake...@apache.org) * Kun Liu (liu...@apache.org) * Liang-Chi Hsieh (vii...@apache.org) * Qingping Hou: (ho...@apache.org) * Wes McKi

Re: [DISCUSS][RFC] Draft Proposal for new Top Level Project for DataFusion

2024-03-01 Thread Andrew Lamb
I will now start an official vote thread. Thank you all very much Andrew On Thu, Feb 29, 2024 at 5:36 AM Andrew Lamb wrote: > > Thank you Wes. I have added your name to the proposed PMC. > > On Wed, Feb 28, 2024 at 4:08 PM Wes McKinney wrote: > > > > I'd be happy to he

Re: [DISCUSS] Donation of a Spark native engine based on DataFusion & Arrow

2024-02-29 Thread Andrew Lamb
roject! > > On Sun, Feb 11, 2024 at 12:51 PM Wes McKinney wrote: > > > > Congrats all! It's great to see the Arrow+DataFusion ecosystem expand in > > this way and to bring the work under the ASF umbrella. > > > > On Sun, Feb 11, 2024 at 5:02 AM Andrew

Re: [DISCUSS][RFC] Draft Proposal for new Top Level Project for DataFusion

2024-02-29 Thread Andrew Lamb
pective from cases in the past). > > On Wed, Feb 28, 2024 at 2:13 PM Andrew Lamb wrote: >> >> Wes brought up a great point on the document[1] that I wanted to discuss >> here more broadly: >> >> > Others may point out that (I think) you don't have any ASF Membe

Re: [VOTE] Flight RPC: add 'fallback' URI scheme

2024-02-28 Thread Andrew Lamb
he comments - I've updated the implementation [1] and added > > Go + integration tests. If this all checks out I'd like to start a vote > > soon. > > > > [1]: https://github.com/apache/arrow/pull/40084 > > > > On Fri, Feb 16, 2024, at 13:43, Andrew Lamb wrote: >

Re: [DISCUSS][RFC] Draft Proposal for new Top Level Project for DataFusion

2024-02-28 Thread Andrew Lamb
tps://www.apache.org/foundation/members On Mon, Feb 26, 2024 at 5:10 PM Andrew Lamb wrote: > An update: > > I have updated the proposal [1] with additional information (new > committers Jeffrey Vo and Jay Zhan, and the new datafusion-comet repository) > > I plan to: > 1

Re: [DISCUSS] Move sqlparser-rs back into DataFusion project?

2024-02-28 Thread Andrew Lamb
odel? > > > > > > > Once DataFusion becomes a top-level project, we could move it to > > > datafusion-sqlparser-rs — it would be a quasi-independent project just > like > > > how DataFusion is today w.r.t. Arrow. But it would get most benefits of > > >

[DataFusion] Proposed changes to the TreeNode API

2024-02-27 Thread Andrew Lamb
I would like to draw some additional attention to any DataFusion user who uses the TreeNode API heavily. There is a PR with a proposed improvement to the API in [1]. Please share any comments you may have on the PR. Andrew [1] https://github.com/apache/arrow-datafusion/pull/8891

Re: [DISCUSS] Move sqlparser-rs back into DataFusion project?

2024-02-27 Thread Andrew Lamb
cessary in the Rust community, where there are many exciting > projects, but they are almost all happening outside ASF. (This is > exactly where Java was in ~2005. Maybe we need a rust-commons or > rust-db?) > > My conclusion is to leave sqlparser-rs where it is for now, but to > conti

Re: [DISCUSS][RFC] Draft Proposal for new Top Level Project for DataFusion

2024-02-26 Thread Andrew Lamb
/11WTNYS8KWScOt3ySTX39WVS6krPhUvHsuJRY9PZQx4g/edit [2] https://github.com/apache/arrow-datafusion/discussions/6475 On Fri, Jan 5, 2024 at 11:19 AM Andrew Lamb wrote: > Thank you very much > > On Fri, Jan 5, 2024 at 11:17 AM Jean-Baptiste Onofré > wrote: > >> Hi Andrew, >> >>

Re: [DISCUSS] Move sqlparser-rs back into DataFusion project?

2024-02-26 Thread Andrew Lamb
Sorry for the late reply, I think sqlparser-rs users are quite a bit more varied than DataFusion and there is not a large overlap between the contributors of the two projects. I currently seem to be the one reviewing / merging most sqlparser-rs reviews, and I would definitely love some more help.

Re: [VOTE][RUST][DataFusion] Release Apache Arrow DataFusion 36.0.0 RC1

2024-02-17 Thread Andrew Lamb
+1 (binding) Verified on M3 Mac Thank you for keeping the release training humming Andy Andrew On Fri, Feb 16, 2024 at 12:23 PM L. C. Hsieh wrote: > +1 (binding) > > Verified on M3 Mac. > > Thanks Andy. > > > On Fri, Feb 16, 2024 at 9:08 AM Andy Grove wrote: > > > > Hi, > > > > I would like

Re: [DISCUSS] Flight RPC: add 'fallback' URI scheme

2024-02-16 Thread Andrew Lamb
Thank you -- I think the usecase is great, but agree with the other reviewers that the name may be confusing. I left some notes on the ticket Andrew On Wed, Feb 14, 2024 at 3:52 PM David Li wrote: > I've put up a candidate implementation sans integration test [1]. > > Some caveats: > -

Re: [VOTE] Explicit session management for Flight RPC

2024-02-16 Thread Andrew Lamb
+1 On Fri, Feb 16, 2024 at 1:46 AM Jean-Baptiste Onofré wrote: > +1 > > Regards > JB > > On Wed, Feb 14, 2024 at 5:38 PM David Li wrote: > > > > Paul Nienaber would like to propose explicit session management for > Flight RPC. This proposal was previously discussed at [1]. A candidate >

Re: [Discussion][C++][FlightRPC] What stage to submit a PR for Flight SQL ODBC driver

2024-02-16 Thread Andrew Lamb
Thank you for driving this -- it will be a very valuable addition to FlightSQL On Tue, Feb 13, 2024 at 1:15 PM Jean-Baptiste Onofré wrote: > Hi guys, > > A quick update about the ODBC driver donation: > - I finished all required changes (ASF header everywhere, build update, > etc) > - I'm now

[ANNOUNCE] New Arrow committer: Jay Zhan

2024-02-16 Thread Andrew Lamb
On behalf of the Arrow PMC, I'm happy to announce that Jay Zhan has accepted an invitation to become a committer on Apache Arrow. Welcome, and thank you for your contributions! Andrew

Re: [FlightSQL] Supporting binding parameters to prepared statements with a stateless server

2024-02-12 Thread Andrew Lamb
th > each request. This sort of maps to the ticket notion already present in > many of the APIs, but could perhaps be formalized. > > Kind Regards, > > Raphael Taylor-Davies > > On 14/09/2023 11:52, Andrew Lamb wrote: > > Hello, > > > > As FlightSQL gets more w

Re: [DISCUSS] [DataFusion] Unifying BuiltIn and User Defined Functions

2024-02-12 Thread Andrew Lamb
/apache/arrow-datafusion/issues/9100 On Tue, Jan 23, 2024 at 8:05 AM Andrew Lamb wrote: > I would like to bring attention to the project of unifying built in and > user defined functions [1], and specifically a PR[2] that starts > implementing that approach. > > Please provide any fe

Re: [DISCUSS] Donation of a Spark native engine based on DataFusion & Arrow

2024-02-11 Thread Andrew Lamb
itive support! > > > > > > > > > > > > > > > > > > Andy, we plan to name the project Comet (BTW if you have > > better > > > > > > > > > suggestions please let us know). Could you help to create a > > repo > >

Re: [ANNOUNCE] New Arrow committer: Jeffrey Vo

2024-02-07 Thread Andrew Lamb
Congratulations Jeffrey! Well deserved! On Tue, Feb 6, 2024 at 1:30 PM Raphael Taylor-Davies wrote: > On behalf of the Arrow PMC, I am happy to announce that Jeffrey Vo has > accepted an invitation to become a committer on Apache Arrow. Welcome, > and thank you for your contributions! > >

Re: [VOTE][RUST][Ballista] Release Apache Arrow Ballista 0.12.0 RC4

2024-02-04 Thread Andrew Lamb
+1 (binding) Verified in M3 Mac Thanks Andy On Sat, Feb 3, 2024 at 5:35 PM L. C. Hsieh wrote: > +1 (binding) > > Verified on M1 Mac. > > Thanks Andy. > > On Sat, Feb 3, 2024 at 2:15 PM Andy Grove wrote: > > > > Hi, > > > > I would like to propose a release of Apache Arrow Ballista >

Re: [VOTE][RUST][DataFusion] Release DataFusion Python Bindings 35.0.0 RC1

2024-02-02 Thread Andrew Lamb
+1 (binding) Verified on M3 mac As before it seems as if python 3.11 isn't supported in the verification script, only python 3.10. When I used 3.10 everything looks good. Thanks a lot, Andrew On Thu, Feb 1, 2024 at 7:27 PM L. C. Hsieh wrote: > +1 (binding) > > Verified on M1 Mac. > > Thanks

[DISCUSS] [DATAFUSION] Minimum Supported Rust Version Policy

2024-01-31 Thread Andrew Lamb
Hello, We are thinking about a min rust version policy for DataFusion[1]. Please leave any thoughts you would like to share on the ticket. Thank you, Andrew [1]: https://github.com/apache/arrow-datafusion/issues/9082

Re: [VOTE] Accept donation of Comet Spark native engine

2024-01-27 Thread Andrew Lamb
+1 (binding) This is super exciting On Sat, Jan 27, 2024 at 11:00 AM Daniël Heres wrote: > +1 (binding). Awesome addition to the DataFusion ecosystem!!! > > Daniël > > > On Sat, Jan 27, 2024, 16:57 vin jake wrote: > > > +1 (binding) > > > > Andy Grove 于 2024年1月27日周六 下午11:43写道: > > > > >

[DataFusion] New Blog Post -- DataFusion 34.0

2024-01-23 Thread Andrew Lamb
If anyone is interested, here is a new blog post about the last 6 months in DataFusion[1] and where we are heading this year. Andrew [1]: https://arrow.apache.org/blog/2024/01/19/datafusion-34.0.0/

[DISCUSS] [DataFusion] Unifying BuiltIn and User Defined Functions

2024-01-23 Thread Andrew Lamb
I would like to bring attention to the project of unifying built in and user defined functions [1], and specifically a PR[2] that starts implementing that approach. Please provide any feedback you have on the ticket or PR. Thank you, Andrew [1]:

Re: [VOTE][RUST][DataFusion] Release Apache Arrow DataFusion 35.0.0 RC1

2024-01-21 Thread Andrew Lamb
+1 (binding) I verified it on Mac (M3). I got the same error in test_partial_ord and I agree it looks very much the the same as https://github.com/apache/arrow-datafusion/pull/8908 -- a test only issue that should not block the release Thanks Andy On Sat, Jan 20, 2024 at 10:43 AM Andy Grove

Re: [DISC] Improve Arrow Release verification process

2024-01-19 Thread Andrew Lamb
I would second this notion that manually running tests that are already covered as part of CI as part of the release process is of (very) limited value. While we do the same thing (compile and run some tests) as part of the Rust release this has never caught any serious defect I am aware of and

Re: invite to the #arrow-rust channel

2024-01-17 Thread Andrew Lamb
I sent an invite for Max. 455954986 I am not sure what you are asking -- can you please clarify? Thanks, Andrew On Wed, Jan 17, 2024 at 1:40 AM 455954986 <455954...@qq.com.invalid> wrote: > Thank you for the invitation. No questions asked. I look forward to > participating. What should I do

Re: [DISCUSS] Donation of a Spark native engine based on DataFusion & Arrow

2024-01-11 Thread Andrew Lamb
I am very supportive of this donation. I know of at least one other DataFusion-based project, blaze-rs[1], which has the same design goal and bringing this project into the ASF may help consolidate these efforts As Andy said, I believe it was very valuable to have a major consumer project (e.g.

Re: [VOTE][RUST] Release Apache Arrow Rust 50.0.0 RC1

2024-01-09 Thread Andrew Lamb
+1 (binding) I verified this on a Mac M3 I hit an error while running the verification scripts[1], but upon investigation it looks like a test only issue and related to an update in a dependent library Thanks Raphael, ANdrew [1] https://github.com/apache/arrow-rs/issues/5292 On Tue, Jan 9,

Re: [DISCUSS] Linear Formula Types

2024-01-08 Thread Andrew Lamb
ch "application" (pandas, DataFusion, etc.) have their own implementation? > > Best regards, > Elliot Morrison-Reed > > -Original Message- > From: Andrew Lamb > Sent: Saturday, January 6, 2024 8:22 AM > To: dev@arrow.apache.org > Subject: Re: [DISCUSS] Lin

Re: [DISCUSS] Linear Formula Types

2024-01-06 Thread Andrew Lamb
Hi Elliot, Given your description, I agree extension types sound like they may be a good idea, similar to geoarrow[1] for Geospatial data where there is extra metadata[2] needed to interpret underlying types (e.g. factor and offset) Andrew [1] https://github.com/geoarrow/geoarrow [2]

Re: [VOTE] Accept donation of flightsql-odbc

2024-01-06 Thread Andrew Lamb
+1 (binding) Thank you for helping make this happen On Sat, Jan 6, 2024 at 2:10 AM Sutou Kouhei wrote: > +1 > > In > "[VOTE] Accept donation of flightsql-odbc" on Fri, 05 Jan 2024 10:41:21 > -0500, > "David Li" wrote: > > > Hello, > > > > This vote is to determine if the Arrow PMC is in

Re: [DISCUSS][RFC] Draft Proposal for new Top Level Project for DataFusion

2024-01-05 Thread Andrew Lamb
ng Mark. He should get back to us soon. > > Regards > JB > > On Fri, Jan 5, 2024 at 3:38 PM Andrew Lamb wrote: > > > > Thanks JB, > > > > I did do a name search and posted the results here [1] > > > > However, I am not sure what the next steps

Re: [DISCUSS][RFC] Draft Proposal for new Top Level Project for DataFusion

2024-01-05 Thread Andrew Lamb
ve a new Apache name moving from Arrow > DataFusion to DataFusion). > > Please let me know if I can help on that. > > Thanks ! > Regards > JB > > On Fri, Jan 5, 2024 at 12:26 PM Andrew Lamb wrote: > > > > Upon reviewing the board report template, I am pla

Re: [VOTE][RUST] Release Apache Arrow Rust Object Store 0.9.0 RC1

2024-01-05 Thread Andrew Lamb
+1 (binding) I verified on M3 mac, both by running the tests manually via `cargo test --all-features` as well as running the release verification script Thank you Raphael for all your work on this release. Andrew On Fri, Jan 5, 2024 at 8:30 AM Raphael Taylor-Davies wrote: > Hi, > > I would

Re: [DISCUSS][RFC] Draft Proposal for new Top Level Project for DataFusion

2024-01-05 Thread Andrew Lamb
sal. It looks great to me too. > I added a few comments on it. > > On Wed, Dec 27, 2023 at 3:05 PM Andy Grove wrote: > > > > Thank you for creating the draft proposal, Andrew. I have reviewed this > and > > I think it looks great. > > > > Andy. > > >

Re: [VOTE][RUST][DataFusion] Release DataFusion Python Bindings 34.0.0 RC1

2024-01-03 Thread Andrew Lamb
t; Andy. > > > > On Fri, Dec 29, 2023 at 4:18 AM Andrew Lamb wrote: > > > I had some trouble running the verification script -- got an error that > a > > specific version of numpy was not available. I am running on an Apple M3 > > Max. > > > >

Re: [VOTE][RUST][DataFusion] Release DataFusion Python Bindings 34.0.0 RC1

2023-12-29 Thread Andrew Lamb
I had some trouble running the verification script -- got an error that a specific version of numpy was not available. I am running on an Apple M3 Max. ERROR: No matching distribution found for numpy==1.21.3 However that version does appear to be available:

[DISCUSS][RFC] Draft Proposal for new Top Level Project for DataFusion

2023-12-27 Thread Andrew Lamb
): Arrow PMC We’d like to propose Andrew Lamb as the initial Chair, (and thus ASF VP) for the DataFusion project. Affiliations Andy Grove (agrove): NVidia Andrew Lamb (alamb): InfluxData Daniël Heres (dheres): Coralogix Jie Wen (jakevin): SelectDB Kun Liu (liukun): Ebay Liang-Chi Hsieh (viirya): Apple

Re: Question: Arrow Flight handshake protocol version

2023-12-27 Thread Andrew Lamb
To the best of my knowledge, the Rust crate implements the Handshake portion of the protocol described in the Flight documentation [1] Depending on your usecase I think your application needs to decide how to use the Handshake message, if at all Andrew [1]:

Re: [RESULT][VOTE][RUST] Release Apache Arrow Rust 49.0.0 RC1

2023-12-21 Thread Andrew Lamb
ase is approved > > The release is available here: > > It has also been released to crates.io. I opted to omit arrow-avro as it > is still a work in progress. > > Thank you to everyone who helped verify this release > > On 09/11/2023 20:37, Andrew Lamb wrote: > > +1 (binding)

[DISUCSS] [DATAFUSION] Repositories

2023-12-21 Thread Andrew Lamb
This is another chain related to the previous discussion [1], about planning to propose [2] "graduating" the DataFusion to its own top level Apache project. In this chain, I would like to discuss which of the current arrow repositories[3] to propose moving to the new PMC I propose the following

[DISCUSS] [DATAFUSION] PMC for new DataFusion top level project

2023-12-20 Thread Andrew Lamb
d PMC members who so desired, would start as committers or PMC members on the new DataFusion project (assuming this is allowed by the process) >From what I can tell, this means the PMC would be the following from the current Arrow PMC [3]: Andy Grove (NVidia) Andrew Lamb (InfuxData) Daniël Heres

Re: [VOTE][RUST][DataFusion] Release Apache Arrow DataFusion 34.0.0 RC1

2023-12-13 Thread Andrew Lamb
d, Dec 13, 2023 at 12:26 PM Andy Grove wrote: > > > > Thanks, Andrew. > > > > That does seem quite serious. I agree it would be better to hold off on > the > > release until this is resolved. > > > > On Wed, Dec 13, 2023 at 1:09 PM Andrew Lamb > wrote:

Re: [VOTE][RUST][DataFusion] Release Apache Arrow DataFusion 34.0.0 RC1

2023-12-13 Thread Andrew Lamb
I have found a regression [1] that I think is fairly serious and should be fixed prior to a release. If others think we should release anyways I can plan to make a patch release shortly with the fix [1] https://github.com/apache/arrow-datafusion/issues/8532 On Wed, Dec 13, 2023 at 2:36 AM Kun

Re: [VOTE][RUST][DataFusion] Release Apache Arrow DataFusion 34.0.0 RC1

2023-12-12 Thread Andrew Lamb
+1 (binding) Verified on Mac M3 Thanks for keeping things rolling Andy On Tue, Dec 12, 2023 at 9:18 AM Andy Grove wrote: > Hi, > > I would like to propose a release of Apache Arrow DataFusion > Implementation, > version 34.0.0. > > This release candidate is based on commit: >

Re: [VOTE] Flight SQL as experimental

2023-12-08 Thread Andrew Lamb
ing) > > On Fri, Dec 8, 2023 at 3:11 PM Aldrin wrote: > > > This thread does have [VOTE] for me. does it not for you? > > > > Sent from Proton Mail <https://proton.me/mail/home> for iOS > > > > > > On Fri, Dec 8, 2023 at 12:09, Andrew Lamb > >

Re: [VOTE] Flight SQL as experimental

2023-12-08 Thread Andrew Lamb
Keep Flight SQL experimental because... > > On Fri, Dec 8, 2023, at 13:37, Weston Pace wrote: > > +1 > > > > On Fri, Dec 8, 2023 at 10:33 AM Micah Kornfield > > wrote: > > > >> +1 > >> > >> On Fri, Dec 8, 2023 at 10:29 AM A

Re: [DISCUSS] Flight SQL as experimental

2023-12-08 Thread Andrew Lamb
I agree it is time to "promote" ArrowFlightSQL to the same level as other standards in Arrow Now that it is used widely (we use and count on it too at InfluxData) I agree it makes sense to remove the experimental label from the overall spec. It would make sense to leave experimental / caveats on

Re: [ANNOUNCE] New Arrow committer: Felipe Oliveira Carvalho

2023-12-07 Thread Andrew Lamb
Congratulations! On Thu, Dec 7, 2023 at 3:09 PM Kevin Gurney wrote: > Congratulations, Felipe! > > From: Daniël Heres > Sent: Thursday, December 7, 2023 2:59 PM > To: dev@arrow.apache.org > Subject: Re: [ANNOUNCE] New Arrow committer: Felipe Oliveira Carvalho

Re: [ANNOUNCE] New Arrow PMC chair: Andy Grove

2023-11-28 Thread Andrew Lamb
t; > > On Tue, Nov 28, 2023 at 3:20 AM Li Jin wrote: > > > > > Congrats Andy! > > > > > > On Tue, Nov 28, 2023 at 3:25 PM Weston Pace > > wrote: > > > > > > > Congrats Andy! > > > > > > > > On Mon, Nov 27, 2023,

[ANNOUNCE] New Arrow PMC chair: Andy Grove

2023-11-27 Thread Andrew Lamb
I am pleased to announce that the Arrow Project has a new PMC chair and VP as per our tradition of rotating the chair once a year. I have resigned and Andy Grove was duly elected by the PMC and approved unanimously by the board. Please join me in congratulating Andy Grove! Thanks, Andrew

Re: [DISCUSS][Format] C data interface for Utf8View

2023-11-15 Thread Andrew Lamb
Given the constraints of not changing the existing struct definitions, adding a new buffer seems like the only way forward from what I understand. It is unfortunate that each array now needs need a new allocation (the buffer lengths) when passing via FFI, but I don't have any other suggestions

Re: [VOTE][RUST][DataFusion] Release Apache Arrow DataFusion 33.0.0 RC2

2023-11-13 Thread Andrew Lamb
+1 (binding) Verified on x86 mac Thanks for doing this Andy On Mon, Nov 13, 2023 at 1:11 PM L. C. Hsieh wrote: > +1 (binding) > > Verified on M1 Mac. > > Thanks Andy. > > On Mon, Nov 13, 2023 at 7:39 AM Andy Grove wrote: > > > > Hi, > > > > I would like to propose a release of Apache Arrow

[ANNOUNCE] New Arrow PMC member: Raúl Cumplido

2023-11-13 Thread Andrew Lamb
The Project Management Committee (PMC) for Apache Arrow has invited Raúl Cumplido to become a PMC member and we are pleased to announce that Raúl Cumplido has accepted. Please join me in congratulating them. Andrew

[DISCUSS] DataFusion Meetup

2023-11-13 Thread Andrew Lamb
I am attempting to organize a face to face meetup of DataFusion community if anyone is interested[1] and would like to solicit input from anyone who has it Andrew [1]: https://github.com/apache/arrow-datafusion/discussions/8152

Re: [VOTE][RUST] Release Apache Arrow Rust 48.0.1 RC1

2023-11-13 Thread Andrew Lamb
at 2:10 PM Raphael Taylor-Davies > > wrote: > > > > > +1 (binding) > > > > > > Verified on x86_64 GNU/Linux > > > > > > On 09/11/2023 20:31, Andrew Lamb wrote: > > > > As discussed on [5], I would like to propose a patch release

Re: [VOTE][RUST] Release Apache Arrow Rust 49.0.0 RC1

2023-11-09 Thread Andrew Lamb
+1 (binding) on Mac x86 Thank you Raphael Andrew On Thu, Nov 9, 2023 at 11:49 AM Chao Sun wrote: > +1 (non-binding) > > Verified on M1 Mac. Thanks Raphael. > > On Thu, Nov 9, 2023 at 12:47 AM Wayne Xia wrote: > > > > +1 (non-binding) > > > > Verified on Intel Linux > > > > Thanks Raphael > >

[VOTE][RUST] Release Apache Arrow Rust 48.0.1 RC1

2023-11-09 Thread Andrew Lamb
As discussed on [5], I would like to propose a patch release of Apache Arrow Rust Implementation, version 48.0.1 to include two bug fixes. This release candidate is based on commit: b60fc7bb09ada1385d3542b784fff2915fbc9cff [1] The proposed release tarball and signatures are hosted at [2]. The

Re: [DataFusion] Introduce qualified alias expressions

2023-11-09 Thread Andrew Lamb
Thank you for raising this Marko -- for anyone else following along there is some great discussion on the ticket and I would encourage anyone who is interested to share your thoughts there as well. On Wed, Nov 8, 2023 at 7:24 PM Jeremy Dyer wrote: > From an Arrow Datafusion Python consumer

[DISCUSS] arrow-rs 48.0.1 patch release

2023-11-09 Thread Andrew Lamb
For various reasons, I plan to make an arrow-rs patch release with bug fixes. Please comment on the ticket[1] if you have any thoughts or other suggestions of what to include. Andrew [1] https://github.com/apache/arrow-rs/issues/5050

Re: [VOTE][RUST][DataFusion] Release Apache Arrow DataFusion 33.0.0 RC1

2023-11-08 Thread Andrew Lamb
As a follow up here, I am working on an arrow 48.0.1 patch release [1] to fix some outstanding issues and unblock the DataFusion 33.0.0 release. Please comment on the ticket if you have any questions or other things you would like to include in the release. Otherwise I plan to make a RC tomorrow

[DISCUSS] [DataFusion] Unify Function Interface (remove BuiltInScalarFunction)

2023-11-08 Thread Andrew Lamb
I would like to bring wider visibility to a proposal[1] to unify the function interface in DataFusion. The TLDR is to remove BuiltInScalarFunction and ensure all functions can implemented as `ScalarUDF` There is a small API change to ScalarUDF proposed as well[2]. If you are interested, please

Re: [VOTE][RUST][DataFusion] Release Apache Arrow DataFusion 33.0.0 RC1

2023-11-07 Thread Andrew Lamb
I filed https://github.com/apache/arrow-rs/issues/5050 to discuss some different possibilities (for example, do an arrow patch release or hold the datafusion release) Please share your thoughts on the ticket On Tue, Nov 7, 2023 at 6:51 AM Andrew Lamb wrote: > I think we should create a tic

  1   2   3   4   5   6   7   8   9   >