[DISCUSS] Possibility of 12.0.2 release

2023-06-23 Thread Bryan Cutler
Hi All, I recently became aware of CVE issue https://github.com/advisories/GHSA-6mjq-h674-j845 with the Java netty libraries and using the fixed Netty library in version 4.1.94.Final required a patch for Arrow, already merged in https://github.com/apache/arrow/issues/36209. I know the freeze for

Re: [ANNOUNCE] New Arrow PMC member: Benjamin Kietzman

2021-05-06 Thread Bryan Cutler
Congrats Ben! On Thu, May 6, 2021 at 12:05 PM Antoine Pitrou wrote: > > Congratulations Ben :-) > > > Le 06/05/2021 à 21:02, Rok Mihevc a écrit : > > Congrats! > > > > On Thu, May 6, 2021 at 10:49 AM Krisztián Szűcs < > szucs.kriszt...@gmail.com> > > wrote: > > > >> Congrats Ben! > >> > >> On Th

Re: [ANNOUNCE] New Arrow committer: Kazuaki Ishizaki

2021-06-07 Thread Bryan Cutler
Congratulations!! On Sun, Jun 6, 2021, 7:28 PM Sutou Kouhei wrote: > Hi, > > On behalf of the Arrow PMC, I'm happy to announce that > Kazuaki Ishizaki has accepted an invitation to become a > committer on Apache Arrow. Welcome, and thank you for your > contributions! > > > Thanks, > -- > kou >

Re: [ANNOUNCE] New Arrow PMC member: David M Li

2021-06-23 Thread Bryan Cutler
Congrats David! On Tue, Jun 22, 2021, 7:24 PM Micah Kornfield wrote: > Congrats David! > > On Tue, Jun 22, 2021 at 7:13 PM Fan Liya wrote: > > > Congratulations David! > > > > Best, > > Liya Fan > > > > > > On Wed, Jun 23, 2021 at 9:44 AM Yibo Cai wrote: > > > > > Congrats David! > > > > > > O

Re: [STRAW POLL] (How) should Arrow define storage for "Instant"s

2021-06-28 Thread Bryan Cutler
C first choice, E second On Mon, Jun 28, 2021, 8:40 AM Julian Hyde wrote: > D > > (2nd choice E if we’re doing ranked-choice voting) > > Julian > > > On Jun 24, 2021, at 12:24 PM, Weston Pace wrote: > > > > The discussion in [1] led to the following question. Before we > > proceed on a vote i

Re: [VOTE] Arrow should state a convention for encoding instants as Timestamp with "UTC" as the time zone

2021-06-30 Thread Bryan Cutler
+1 non-binding On Wed, Jun 30, 2021, 2:53 AM Weston Pace wrote: > This vote is a result of previous discussion[1][2]. This vote is also > a prerequisite for the PR in [5]. > > --- > Some date & time libraries have three temporal concepts. For the sake > of this document we will call them Local

Re: [ANNOUNCE] New Arrow PMC chair: Kouhei Sutou

2022-01-27 Thread Bryan Cutler
Congratulations Kou, thanks for all your work! On Thu, Jan 27, 2022, 4:36 PM Sutou Kouhei wrote: > Thanks everyone!!! > > In > "[ANNOUNCE] New Arrow PMC chair: Kouhei Sutou" on Tue, 25 Jan 2022 > 11:32:56 -0500, > Wes McKinney wrote: > > > I am pleased to announce that we have a new PMC ch

Is 7.0.0 release missing the Java arrow-flight POM?

2022-02-17 Thread Bryan Cutler
Hi All, Congrats on the 7.0.0 release! I was trying it out and got an error not being able to find arrow-flight-7.0.0.pom. This looks like a new parent POM for Flight, so I checked maven central and don't see it deployed there. Not sure what could have happened, but maybe it's only me. Anyone else

Re: Is 7.0.0 release missing the Java arrow-flight POM?

2022-02-17 Thread Bryan Cutler
ven.org/maven2/org/apache/arrow/flight-grpc/ > > Can we use flight-core (and flight-grpc) instead of > arrow-flight? > > Thanks, > -- > kou > > In > "Is 7.0.0 release missing the Java arrow-flight POM?" on Thu, 17 Feb > 2022 09:48:57 -0800, > Bry

Re: Is 7.0.0 release missing the Java arrow-flight POM?

2022-02-17 Thread Bryan Cutler
w why our upload script > https://github.com/apache/arrow/blob/master/dev/release/06-java-upload.sh > doesn't upload it... > > Could you take a look at it? > > > Thanks, > -- > kou > > In > "Re: Is 7.0.0 release missing the Java arrow-flight POM?"

Re: Is 7.0.0 release missing the Java arrow-flight POM?

2022-02-18 Thread Bryan Cutler
, Feb 17, 2022 at 10:06 PM Bryan Cutler wrote: > Sure, I'll take a look at the script. > > On Thu, Feb 17, 2022 at 4:39 PM Sutou Kouhei wrote: > >> Hi, >> >> Ah, arrow-flight-*.pom exists on our CI artifacts: >> >> https://github.com/ursacomputing/cr

Re: Is 7.0.0 release missing the Java arrow-flight POM?

2022-02-21 Thread Bryan Cutler
les at > > https://repository.apache.org/#stagingRepositories with your > > ASF account. > > Note that you MUST not press the "Close" button! I'll remove > > them by pressing "Drop" button when we fix this. > > > > > > Th

Re: Flight/FlightSQL Optimization for Small Results?

2022-03-01 Thread Bryan Cutler
I think this would be a useful feature and be nice to have in Flight core. For cases like previewing data, you usually just want to get a small amount of data quickly. Would it make sense to make this part of DoGet since it still would be returning a record batch? Perhaps a Ticket could be made to

Re: [VOTE] Release Apache Arrow 7.0.0 - Java artifacts

2022-03-14 Thread Bryan Cutler
+1 (non-binding) On Mon, Mar 14, 2022 at 10:26 AM David Li wrote: > My vote: +1 (binding) > > Are any other PMC members able to take a quick look? > > Thanks, > David > > On Sat, Mar 12, 2022, at 07:31, Kun Liu wrote: > > +1 non-binding > > just uploaded the missing pom and don't need to releas

Re: [ANNOUNCE] New Arrow committers: Raphael Taylor-Davies, Wang Xudong, Yijie Shen, and Kun Liu

2022-03-14 Thread Bryan Cutler
Congrats to all! On Thu, Mar 10, 2022 at 12:11 AM Alenka Frim wrote: > Congratulations all! > > On Thu, Mar 10, 2022 at 1:55 AM Yang hao <1371656737...@gmail.com> wrote: > > > Congratulations to all! > > > > From: Benson Muite > > Date: Thursday, March 10, 2022 at 03:45 > > To: dev@arrow.apache

Re: [VOTE] Extend Arrow Flight SQL with GetXdbcTypeInfo, SQL type info in schemas

2022-03-28 Thread Bryan Cutler
+1 (non-binding) On Mon, Mar 28, 2022, 7:07 AM Andrew Lamb wrote: > Thank you David for pushing this through -- I think the overall FlightSQL > story is very compelling for the Arrow ecosystem > > I am also +1 on the idea, but I haven't had enough time to study the > implementation in detail yet

Re: [JAVA] JDK Support Policy?

2022-04-05 Thread Bryan Cutler
Thanks for bringing this up Micah. Given that we have finite resources for CI, I think the oldest active LTS version sounds pretty reasonable. Ultimately it should be community driven and balance between the available resources we have and peoples time to patch any issues that come up. On Tue, Mar

Re: [ANNOUNCE] New Arrow committer: Liang-Chi Hsieh

2022-04-27 Thread Bryan Cutler
Congratulations!! That's great news and really glad to have you on the project! On Wed, Apr 27, 2022, 11:44 AM Andrew Lamb wrote: > On behalf of the Arrow PMC, I'm happy to announce that Liang-Chi Hsieh > has accepted an invitation to become a committer on Apache > Arrow. Welcome, and thank you

Re: [VOTE] Release Apache Arrow 8.0.0 - RC3

2022-05-04 Thread Bryan Cutler
+1 (non-binding) I ran: TEST_DEFAULT=0 TEST_INTEGRATION_CPP=1 TEST_INTEGRATION_JAVA=1 ARROW_GANDIVA=OFF ARROW_PLASMA=OFF dev/release/verify-release-candidate.sh 8.0.0 3 On Wed, May 4, 2022 at 3:23 PM Sutou Kouhei wrote: > +1 > > I ran the followings on Debian GNU/Linux sid: > > * TEST_DEFAULT

Re: Trouble installing archery?

2020-04-13 Thread Bryan Cutler
I had the same problem and Antoine's suggestion was exactly what was wrong. On Mon, Apr 13, 2020 at 1:27 AM Antoine Pitrou wrote: > > Le 13/04/2020 à 02:42, Micah Kornfield a écrit : > > When I follow the instructions at > > https://arrow.apache.org/docs/developers/benchmarks.html > > > > "pip i

Re: [DISCUSS] Reducing scope of work for Arrow 1.0.0 release

2020-04-21 Thread Bryan Cutler
I really would like to see a 1.0.0 release with complete implementations for C++ and Java. From my experience, that interoperability has been a major selling point for the project. That being said, my time for contributions has been pretty limited lately and I know that Java has been lagging, so if

Python is there support for extension types in Parquet?

2020-04-24 Thread Bryan Cutler
I've been trying out IO with Arrow's extension types and I was able write a parquet file but reading it back causes an error: "pyarrow.lib.ArrowInvalid: Unsupported nested type: ...". Looking at the code for the parquet reader, it checks nested types and only allows a few specific ones. Is this a k

Re: Python is there support for extension types in Parquet?

2020-04-24 Thread Bryan Cutler
ed out but > > https://issues.apache.org/jira/browse/ARROW-1644 (and related subtasks) > > might be a good place to track this. > > > > Thanks, > > Micah > > > > On Fri, Apr 24, 2020 at 11:13 AM Bryan Cutler wrote: > > > > >

Re: [VOTE] Release Apache Arrow 0.17.1 - RC1

2020-05-15 Thread Bryan Cutler
+1 (non-binding) I ran: ARROW_TMPDIR=/tmp/arrow-test TEST_DEFAULT=0 TEST_SOURCE=1 TEST_CPP=1 TEST_PYTHON=1 TEST_JAVA=1 TEST_INTEGRATION_CPP=1 TEST_INTEGRATION_JAVA=1 dev/release/verify-release-candidate.sh source 0.17.1 1 On Fri, May 15, 2020 at 8:38 AM Francois Saint-Jacques < fsaintjacq...@gmai

Re: [ANNOUNCE] New Arrow committers: Ji Liu and Liya Fan

2020-06-12 Thread Bryan Cutler
Congratulations! On Thu, Jun 11, 2020, 9:29 PM Fan Liya wrote: > Dear all, > > I want to thank you all for all your kind help. > It is a great honor to work with you in this great community. > I Hope we can contribute more and make the community better. > > Best, > Liya Fan > > On Fri, Jun 12, 2

Re: [VOTE] Add Decimal::bitWidth field to Schema.fbs for forward compatibility

2020-06-25 Thread Bryan Cutler
+1 On Wed, Jun 24, 2020, 10:38 AM Francois Saint-Jacques < fsaintjacq...@gmail.com> wrote: > +1 (binding) >

Re: [VOTE] Release Apache Arrow 1.0.0 - RC1

2020-07-19 Thread Bryan Cutler
+0 (non-binding) I ran verification script for binaries and then source, as below, and both look good ARROW_TMPDIR=/tmp/arrow-test TEST_DEFAULT=0 TEST_SOURCE=1 TEST_CPP=1 TEST_PYTHON=1 TEST_JAVA=1 TEST_INTEGRATION_CPP=1 TEST_INTEGRATION_JAVA=1 dev/release/verify-release-candidate.sh source 1.0.0 1

Re: [VOTE] Release Apache Arrow 1.0.0 - RC1

2020-07-19 Thread Bryan Cutler
> > > time zone) and Arrow (which has naive timestamps -- a > > concept > > > > similar > > > > > > > > > > but different from the SQL concept TIMESTAMP WITHOUT TIME > > ZONE > > > > -- and > > > > > > &g

Re: [VOTE] Release Apache Arrow 1.0.0 - RC2

2020-07-22 Thread Bryan Cutler
+1 (non-binding) I ran release verification script with the following args ARROW_TMPDIR=/tmp/arrow-test TEST_DEFAULT=0 TEST_SOURCE=1 TEST_CPP=1 TEST_PYTHON=1 TEST_JAVA=1 TEST_INTEGRATION_CPP=1 TEST_INTEGRATION_JAVA=1 dev/release/verify-release-candidate.sh source 1.0.0 2 On Wed, Jul 22, 2020 at 1

change in pyarrow scalar equality?

2020-08-05 Thread Bryan Cutler
Hi all, I came across a behavior change from 0.17.1 when comparing array scalar values with python objects. This used to work for 0.17.1 and before, but in 1.0.0 equals always returns false. I saw there was a previous discussion on Python equality semantics, but not sure if the conclusion is the b

Re: change in pyarrow scalar equality?

2020-08-14 Thread Bryan Cutler
l(a[0], 1) > ... > TypeError: Got unexpected argument type for compute function > > For this last one, we should probably do an attempt to convert the python > scalar to a pyarrow scalar, and maybe for the "a[0] == 1" case as well > (however, coerce to which type if th

Re: [DISCUSS] Big Endian support in Arrow (was: Re: [Java] Supporting Big Endian)

2020-08-31 Thread Bryan Cutler
I also think this would be a worthwhile addition and help the project expand in more areas. Beyond the Apache Spark optimization use case, having Arrow interoperability with the Python data science stack on BE would be very useful. I have looked at the remaining PRs for Java and they seem pretty mi

Re: conversion between pyspark.DataFrame and pyarrow.Table

2020-09-01 Thread Bryan Cutler
There isn't a direct conversion to/from Spark, I made https://issues.apache.org/jira/browse/SPARK-29040 a while ago for conversion to Spark from an Arrow table. If possible, make a comment there for your use case which might help get support for it. Bryan On Mon, Aug 31, 2020, 9:12 PM Micah Kornf

Re: [VOTE] Release Apache Arrow 2.0.0 - RC2

2020-10-14 Thread Bryan Cutler
+1 (non-binding) I verified binaries and source with: ARROW_TMPDIR=/tmp/arrow-test ARROW_GANDIVA=0 ARROW_PLASMA=0 TEST_DEFAULT=0 TEST_SOURCE=1 TEST_CPP=1 TEST_PYTHON=1 dev/release/verify-release-candidate.sh source 2.0.0 2 On Wed, Oct 14, 2020 at 2:02 PM Sutou Kouhei wrote: > Hi, > > I forgot t

Re: [ANNOUNCE] New Arrow PMC chair: Wes McKinney

2020-10-26 Thread Bryan Cutler
Congrats Wes, well deserved! On Sun, Oct 25, 2020, 10:17 PM Jorge Cardoso Leitão < jorgecarlei...@gmail.com> wrote: > Thanks a lot Jacques for taking the flag until now, and congratulations, > Wes! > > On Sun, Oct 25, 2020 at 2:58 PM Wes McKinney wrote: > > > Thanks all! > > > > On Sun, Oct 25,

Github check error with ORC JNI adapter

2020-11-03 Thread Bryan Cutler
There seems to be a Github check error with the Java JNI tests for the ORC adapter that is affecting a lot of recent PRs, see https://github.com/apache/arrow/pull/8577/checks?check_run_id=1346780145. >From the log, it looks like some env setting, but I can't tell what's wrong. Anyone else know? Th

Re: [Java] PR review for ARROW-11173

2021-01-19 Thread Bryan Cutler
Hi Nick, I left a note in the PR that I will try to review soon, thanks! On Sun, Jan 17, 2021 at 8:22 PM Nick Bruno wrote: > Hi All, > > I'd like to get feedback on the pull request I created a little over a > week ago - https://github.com/apache/arrow/pull/9151 > > It adds support for Map type

Re: [VOTE] Release Apache Arrow 3.0.0 - RC2

2021-01-20 Thread Bryan Cutler
+1 (non-binding) I verified binaries and source with the following: ARROW_TMPDIR=/tmp/arrow-test ARROW_GANDIVA=0 ARROW_PLASMA=0 TEST_DEFAULT=0 TEST_SOURCE=1 TEST_CPP=1 TEST_PYTHON=1 TEST_JAVA=1 TEST_INTEGRATION_CPP=1 TEST_INTEGRATION_JAVA=1 dev/release/verify-release-candidate.sh source 3.0.0 2 I

Re: [C++][CI] Make "C++ on s390x" build mandatory?

2021-02-23 Thread Bryan Cutler
+1 sgtm On Tue, Feb 23, 2021, 9:47 AM Micah Kornfield wrote: > +1, but let's keep an eye on it to make sure it remains stable. > > On Tue, Feb 23, 2021 at 5:34 AM Kazuaki Ishizaki > wrote: > > > Thank you. +1 for this proposal, > > > > Kazuaki Ishizaki > > > > > > > > From: Benjamin Kietzman

Re: [VOTE] Adopt FORMAT and LIBRARY SemVer-based version schemes for Arrow 1.0.0 and beyond

2019-07-31 Thread Bryan Cutler
+1 (non-binding) On Wed, Jul 31, 2019 at 8:59 AM Uwe L. Korn wrote: > +1 from me. > > I really like the separate versions > > Uwe > > On Tue, Jul 30, 2019, at 2:21 PM, Antoine Pitrou wrote: > > > > +1 from me. > > > > Regards > > > > Antoine. > > > > > > > > On Fri, 26 Jul 2019 14:33:30 -0500 >

Re: [DISCUSS] Add GetFlightSchema to Flight RPC

2019-08-01 Thread Bryan Cutler
Sounds good to me, I would just echo what others have said. On Thu, Aug 1, 2019 at 8:17 AM Ryan Murray wrote: > Thanks Wes, > > The descriptor is only there to maintain a bit of symmetry with > GetFlightInfo. Happy to remove it, I don't think its necessary and already > a few people agree. Simil

Re: [VOTE] Proposed addition to Arrow Flight Protocol

2019-08-16 Thread Bryan Cutler
+1 (non-binding) On Fri, Aug 16, 2019, 8:36 AM Micah Kornfield wrote: > My vote +1 (binding) > > On Friday, August 16, 2019, David Li wrote: > > > +1 (non-binding) > > > > Thanks Ryan for working on this! > > > > Best, > > David > > > > On 8/16/19, Micah Kornfield wrote: > > > Hello, > > > Rya

Re: [VOTE] Alter Arrow binary protocol to address 8-byte Flatbuffer alignment requirements (2nd vote)

2019-08-20 Thread Bryan Cutler
+1 (non-binding) On Tue, Aug 20, 2019, 7:43 AM Antoine Pitrou wrote: > > Sorry, had forgotten to send my vote on this. > > +1 from me. > > Regards > > Antoine. > > > On Wed, 14 Aug 2019 17:42:33 -0500 > Wes McKinney wrote: > > hi all, > > > > As we've been discussing [1], there is a need to int

Re: [ANNOUNCE] New Arrow committer: David M Li

2019-08-30 Thread Bryan Cutler
Congrats David! On Fri, Aug 30, 2019 at 10:19 AM Antoine Pitrou wrote: > > Congratulations David and welcome to the team :-) > > Regards > > Antoine. > > > Le 30/08/2019 à 18:21, Wes McKinney a écrit : > > On behalf of the Arrow PMC I'm happy to announce that David has > > accepted an invitatio

Re: [ANNOUNCE] New committers: Ben Kietzman, Kenta Murata, and Neal Richardson

2019-09-06 Thread Bryan Cutler
Congrats Ben, Kenta and Neal! On Fri, Sep 6, 2019, 12:15 PM Krisztián Szűcs wrote: > Congratulations! > > On Fri, Sep 6, 2019 at 8:12 PM Ben Kietzman > wrote: > > > Thanks! > > > > On Fri, Sep 6, 2019 at 1:09 PM Micah Kornfield > > wrote: > > > > > Congrats everyone! (apologies if I double sen

Re: [RESULT] [VOTE] Alter Arrow binary protocol to address 8-byte Flatbuffer alignment requirements (2nd vote)

2019-09-09 Thread Bryan Cutler
(today maybe). I think we should create a branch > > > > > > where we can stack the patches that implement this for each > > language. > > > > > > > > > > > > On Fri, Aug 23, 2019 at 4:04 AM Paul Taylor > > > > > > > > > >

Re: [RESULT] [VOTE] Alter Arrow binary protocol to address 8-byte Flatbuffer alignment requirements (2nd vote)

2019-09-10 Thread Bryan Cutler
I have the patch for the EOS with Java writers up here https://github.com/apache/arrow/pull/5345. Just to clarify, the EOS of {0x, 0x} is used for both stream and file formats, in non-legacy writing mode. On Mon, Sep 9, 2019 at 8:01 PM Bryan Cutler wrote: > Sounds good to

Re: [NIGHTLY] Arrow Build Report for Job nightly-2019-09-24-0

2019-09-24 Thread Bryan Cutler
I'm able to pass Spark integration tests locally with the build patch from https://github.com/apache/arrow/pull/5465, so I'm reasonably confident all the issues have been resolved and it's just flaky timeouts now. We are trying some things to fix the timeouts, but nothing to hold up the release for

Re: [VOTE] Release Apache Arrow 0.15.0 - RC2

2019-10-02 Thread Bryan Cutler
prefix/src/orc_ep-stamp/orc_ep-build] Error 1 CMakeFiles/Makefile2:1248: recipe for target 'CMakeFiles/orc_ep.dir/all' failed make[1]: *** [CMakeFiles/orc_ep.dir/all] Error 2 On Wed, Oct 2, 2019 at 4:12 PM Bryan Cutler wrote: > +1 (non-binding) > > I ran the following on Ubunt

Re: [VOTE] Release Apache Arrow 0.15.0 - RC2

2019-10-02 Thread Bryan Cutler
+1 (non-binding) I ran the following on Ubuntu 16.04 4.15.0-64-generic: > dev/release/verify-release-candidate.sh binaries 0.15.0 2 > ARROW_CUDA=OFF \ TEST_DEFAULT=0 \ TEST_SOURCE=1 \ TEST_CPP=1 \ TEST_PYTHON=1 \ TEST_JAVA=1 \ TEST_INTEGRATION=1 \ dev/release/verify-release-candidate.sh source 0.1

Re: Docker organization for development images

2019-10-03 Thread Bryan Cutler
Sounds good, thanks Krisztian! On Thu, Oct 3, 2019 at 6:10 AM Krisztián Szűcs wrote: > Hi, > > We've created a docker hub organisation called "arrowdev" > to host the images defined in the docker-compose.yml, see > the following commit [1]. > So now it is possible to speed up the image builds by

Re: Collecting Arrow critique and our roadmap on that

2019-10-03 Thread Bryan Cutler
A lot of good info here, I added a point that has come up often for me. On Thu, Oct 3, 2019 at 10:03 AM Wes McKinney wrote: > I read through and left some comments. > > Would be great to turn into an FAQ section in the docs and add a link > to the navigation on the front page of the website. > >

Re: [ANNOUNCE] Apache Arrow 0.15.0 released

2019-10-07 Thread Bryan Cutler
Great work everyone! On Sun, Oct 6, 2019 at 1:46 PM Wes McKinney wrote: > Congrats everyone! > > On Sat, Oct 5, 2019 at 10:09 AM Krisztián Szűcs wrote: > > > > The Apache Arrow community is pleased to announce the 0.15.0 release. > > The release includes 711 resolved issues ([1]) since the 0.14

Re: [ANNOUNCE] New Arrow committer: Eric Erhardt

2019-10-18 Thread Bryan Cutler
Congrats! On Thu, Oct 17, 2019, 6:26 PM Fan Liya wrote: > Congrats Eric! > > Best, > Liya Fan > > On Fri, Oct 18, 2019 at 3:06 AM paddy horan > wrote: > > > Congrats Eric! > > > > > > From: Micah Kornfield > > Sent: Thursday, October 17, 2019 12:45:15 PM > > To

Re: [DISCUSS][Java] Builders for java classes

2019-10-29 Thread Bryan Cutler
Just to clarify, how will this be different than the current vector writers that they are wrapping? Is it just the ability to add multiple values at once, or more efficiently? Also, if we are going to be adding new APIs, maybe we can try to match more closely the existing builders in C++? I believ

Re: [VOTE] Release Apache Arrow 0.15.1 - RC0

2019-10-31 Thread Bryan Cutler
+1 (non-binding), although I could not complete the source verification script On Ubuntu 16.04 I ran * verification script for binaries, no issues * verification script for source, could not complete: TEST_DEFAULT=0 TEST_SOURCE=1 TEST_PYTHON=1 TEST_INTEGRATION_CPP=1 TEST_INTEGRATION_JAVA=1 ARROW

Re: [VOTE] Release Apache Arrow 0.15.1 - RC0

2019-10-31 Thread Bryan Cutler
2019 at 2:17 PM Wes McKinney wrote: > hi Bryan -- I think `pip3 install setuptools` will take care of it > > On Thu, Oct 31, 2019 at 2:06 PM Bryan Cutler wrote: > > > > +1 (non-binding), although I could not complete the source verification > > script > > > >

Re: [VOTE] Release Apache Arrow 0.15.1 - RC0

2019-10-31 Thread Bryan Cutler
mmands. --help Show this message and exit. Commands: benchmark Arrow benchmarking. build Initialize an Arrow C++ build On Thu, Oct 31, 2019 at 2:38 PM Bryan Cutler wrote: > I am using a conda env, so that will install the package there. When > archery runs the integration

Re: [Java] Append multiple record batches together?

2019-11-08 Thread Bryan Cutler
I think having a chunked array with multiple vector buffers would be ideal, similar to C++. It might take a fair amount of work to add this but would open up a lot more functionality. As for the API, VectorSchemaRoot.concat(Collection) seems good to me. On Thu, Nov 7, 2019 at 12:09 AM Fan Liya wr

Re: [Java] Append multiple record batches together?

2019-11-12 Thread Bryan Cutler
this but >> would open up a lot more functionality. > > > There are potentially two different use-cases. ChunkedArray is > logical/lazy concatenation where as concat, physically rebuilds the vectors > to be a single vector. > > On Fri, Nov 8, 2019 at 10:51 AM Bryan Cutler

[Discuss][Java] Provide default for io.netty.tryReflectionSetAccessible to prevent errors

2019-11-17 Thread Bryan Cutler
After ARROW-3191 [1], consumers of Arrow Java with a JDK 9 and above are required to set the JVM property "io.netty.tryReflectionSetAccessible=true" at startup, each time Arrow code is run, as documented at [2]. Not doing this will result in the error "java.lang.UnsupportedOperationException: sun.m

Re: [Discuss][Java] Provide default for io.netty.tryReflectionSetAccessible to prevent errors

2019-11-20 Thread Bryan Cutler
t; > I've had in the past. > > > > Do we need to handle jdk8 as a special case? Do you think it pays to try > > to find an alternate library that doesn't require special flags for > > whatever we are using this functionality for? > > > > Thanks, >

Re: Looking to 1.0

2020-01-06 Thread Bryan Cutler
I agree on a 0.16.0 release. In the meantime I'll try to help out with getting the Java side ready for 1.0. On Sat, Jan 4, 2020 at 7:21 PM Fan Liya wrote: > Hi Jacques, > > ARROW-4526 is interesting. I would like to try to resolve it. > Thanks a lot for the information. > > Best, > Liya Fan > >

Re: PySpark failure [RE: [NIGHTLY] Arrow Build Report for Job nightly-2020-01-15-0]

2020-01-23 Thread Bryan Cutler
Thanks for investigating this and the quick fix Joris and Wes! I just have a couple questions about the behavior observed here. The pyspark code assigns either the same series back to the pandas.DataFrame or makes some modifications if it is a timestamp. In the case there are no timestamps, is th

Re: PySpark failure [RE: [NIGHTLY] Arrow Build Report for Job nightly-2020-01-15-0]

2020-01-24 Thread Bryan Cutler
itted > blocks will get combined (resulting in a memory copy then), and it also > means you can't modify the dataframe (if this dataframe is used in python > UDFs, it might limit what can be done in those UDFs. Just guessing here, I > don't know the pyspark code well enough). &

Re: [Java] PR Reviewers

2020-01-27 Thread Bryan Cutler
Hi Micah, I don't have a ton of bandwidth at the moment, but I'll try to review some more PRs. Anyone, please feel free to ping me too if you have a stale PR that needs some help getting through. Outreach to other Java communities sounds like a good idea - more Java users would definitely be a good

PR Dashboard for Java?

2020-01-27 Thread Bryan Cutler
I saw on Confluence that other Arrow components have PR dashboards, but I don't see one for Java? I think it would be helpful, is it difficult to add one for Java? I'm happy to do it if someone could point me in the right direction. Thanks! Bryan

Re: [DISCUSS][JAVA] Correct the behavior of ListVector isEmpty

2020-01-27 Thread Bryan Cutler
Return a null might be more correct since `getObject(int index)` also return a null value if not set, but I don't think it's worth making a more complicated API for this. It should be fine to return `false` for a null value. +1 for treating nulls as empty. On Fri, Jan 24, 2020 at 9:12 AM Brian Hul

Re: PR Dashboard for Java?

2020-01-27 Thread Bryan Cutler
t; open. > > Or you could bookmark > > https://github.com/apache/arrow/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Aopen+%22%5BJava%5D%22 > or https://github.com/apache/arrow/labels/lang-java > > Neal > > On Mon, Jan 27, 2020 at 11:26 AM Bryan Cutler wrote: > > > I saw on

Re: [VOTE] Release Apache Arrow 0.16.0 - RC1

2020-01-28 Thread Bryan Cutler
-1 There is a bug in Pandas conversion for timestamps that looks to be a regression, https://issues.apache.org/jira/browse/ARROW-7709 On Tue, Jan 28, 2020 at 11:30 AM Wes McKinney wrote: > I opened https://issues.apache.org/jira/browse/ARROW-7708. > > On Tue, Jan 28, 2020 at 1:24 PM Wes McKinney

Re: [VOTE] Release Apache Arrow 0.16.0 - RC1

2020-01-28 Thread Bryan Cutler
: > Bryan -- was this tested somewhere that we missed (eg a nightly)? > > On Tue, Jan 28, 2020, 4:31 PM Bryan Cutler wrote: > > > -1 > > There is a bug in Pandas conversion for timestamps that looks to be a > > regression, https://issues.apache.org/jira/browse/ARROW-7709 &

Re: [VOTE] Release Apache Arrow 0.16.0 - RC1

2020-01-28 Thread Bryan Cutler
Actually, it looks like they weren't passing after the `split_blocks` change either. From a quick glance, the Spark tests were passing up to 1/14 and they started failing after that. On Tue, Jan 28, 2020 at 3:17 PM Bryan Cutler wrote: > The nightly Spark integration was failing becaus

Re: [VOTE] Release Apache Arrow 0.16.0 - RC1

2020-01-29 Thread Bryan Cutler
An update on Spark integration tests: the new error looks to be a regression so I made https://issues.apache.org/jira/browse/ARROW-7723 and marked as a blocker. It's possible to work around this bug, so I wouldn't call it a hard blocker if we need to proceed with the release. On Wed, Jan 29, 2020

Re: [Java] Issues with IntelliJ + errorprone + OpenJDK

2020-02-03 Thread Bryan Cutler
Hi Andy, What is your JDK version? I haven't seen that exact error, did you open Arrow as a Maven project in Intellij? On Mon, Feb 3, 2020 at 7:47 AM Andy Grove wrote: > I'm working on the Java codebase and cannot run code inside IntelliJ and it > looks like some kind of compatibility issue betw

Re: [Java] Issues with IntelliJ + errorprone + OpenJDK

2020-02-03 Thread Bryan Cutler
line with Maven works fine. > > > > Very odd. I'll guess I'll do a little more research on errorprone. > > > > Thanks, > > > > Andy. > > > > > > On Mon, Feb 3, 2020 at 12:50 PM Bryan Cutler wrote: > > > >> Hi Andy, > >&g

Re: [Java] Issues with IntelliJ + errorprone + OpenJDK

2020-02-04 Thread Bryan Cutler
ne Compiler" plugin to intellij > >> 2. setting "Settings/Build, Execution, Deployment/Compiler/Java > >> Compiler/Use compiler" to "Javac with error-prone" > >> > >> I am using Intellij 2019.3 (Community Edition) > >> > >> Best,

Re: [VOTE] Release Apache Arrow 0.16.0 - RC2

2020-02-04 Thread Bryan Cutler
+1 I had some trouble due to ARROW-7760 at first, but applied the same patch and passed. I ran the command: TMPDIR=/tmp/arrow TEST_DEFAULT=0 TEST_SOURCE=1 TEST_CPP=1 TEST_PYTHON=1 TEST_JAVA=1 TEST_INTEGRATION_CPP=1 TEST_INTEGRATION_JAVA=1 dev/release/verify-release-candidate.sh source 0.16.0 2 On

Re: PR Dashboard for Java?

2020-02-11 Thread Bryan Cutler
at 2:23 PM Wes McKinney wrote: > Bryan -- I just gave you (cutlerb) Confluence edit privileges. These > have to be explicitly managed on a per-user basis to avoid spam > problems > > On Mon, Jan 27, 2020 at 4:12 PM Bryan Cutler wrote: > > > > Thanks Neal, but it does

Re: PR Dashboard for Java?

2020-02-12 Thread Bryan Cutler
Works now, thanks! I added a page for Java open PRs https://cwiki.apache.org/confluence/display/ARROW/Java+Open+Patches On Tue, Feb 11, 2020 at 12:08 PM Wes McKinney wrote: > Weird. Try now > > On Tue, Feb 11, 2020 at 1:03 PM Bryan Cutler wrote: > > > > Wes, it doesn&

[DISCUSS] Flight testing inconsistency for empty batches

2020-02-24 Thread Bryan Cutler
While looking into Null type testing for ARROW-7899, a couple small issues came up regarding Flight integration testing with empty batches (row count == 0) that could be worked out with a quick discussion. It seems there is a small difference between the C++ and Java Flight servers when there are e

Re: [DISCUSS] Flight testing inconsistency for empty batches

2020-02-28 Thread Bryan Cutler
that are empty, > > because Flight lets you attach metadata to batches, and so an empty > > batch might still have metadata that the client or server wants. > > > > Best, > > David > > > > On 2/24/20, Bryan Cutler wrote: > > > While looking into Nu

Re: Proposal to use Black for automatic formatting of Python code

2020-03-27 Thread Bryan Cutler
+1 for using black On Fri, Mar 27, 2020 at 11:53 AM Joris Van den Bossche < jorisvandenboss...@gmail.com> wrote: > On Fri, 27 Mar 2020 at 18:49, Antoine Pitrou wrote: > > > > > I don't want to be the small minority opposing this so let's go for it. > > One question though: will we continue to ch

Re: [VOTE] Release Apache Arrow 0.11.0 (RC1)

2018-10-07 Thread Bryan Cutler
+1 (non-binding) I ran tests for C++, Python, Java, integration. I also tried to run spark-integration, but failed with the linking error below - not a blocker though /usr/bin/ld.gold: error: /opt/conda/envs/pyarrow-dev/lib/libgtest.a(gtest-all.cc.o): requires dynamic R_X86_64_32 reloc against '_

Re: [ANNOUNCE] New Arrow committers: Romain François, Sebastien Binet, Yosuke Shiro

2018-11-09 Thread Bryan Cutler
Congrats to all and welcome! On Thu, Nov 8, 2018, 2:00 PM Li Jin Welcome! > > On Thu, Nov 8, 2018 at 4:01 PM Uwe L. Korn wrote: > > > Welcome to all of you! > > > > On Thu, Nov 8, 2018, at 8:56 PM, Wes McKinney wrote: > > > On behalf of the Arrow PMC, I'm happy to announce that Romain > > > Fran

Re: [ANNOUNCE] New Arrow PMC member: Krisztián Szűcs

2018-11-09 Thread Bryan Cutler
Congrats Krisztián! On Thu, Nov 8, 2018, 2:00 PM Li Jin Congrats! > > On Thu, Nov 8, 2018 at 4:02 PM Uwe L. Korn wrote: > > > Congratulations Krisztián! > > > > On Thu, Nov 8, 2018, at 9:56 PM, Philipp Moritz wrote: > > > Congrats and welcome Krisztián! > > > > > > On Thu, Nov 8, 2018 at 11:48 A

Re: [DISCUSS] Joda Time -> Java8 Time

2018-11-14 Thread Bryan Cutler
Hi all, I'm picking this back up again and have WIP pr at https://github.com/apache/arrow/pull/2966. Please take a look at the new APIs and see if they impact you downstream. In addition to the API changes mentioned before by Li, there is also (4) IntervalDayVector now uses java.time.Duration, w

pyarrow data type casting problem when safe=True

2019-01-10 Thread Bryan Cutler
Hi All, I have a question about using pyarrow.Array.from_pandas with the safe flag set to True. When the Pandas data contains integers and NULL values, it will get changed to a floating point dtype and then if the type is casted back to an integer in Arrow, it will raise an error "ArrowInvalid: F

Re: [VOTE] Release Apache Arrow 0.12.0 RC4

2019-01-17 Thread Bryan Cutler
+1 I ran ARROW_HAVE_CUDA=NO dev/release/verify-release-candidate.sh source 0.12.0 4 - 4.15.0-43-generic #46~16.04.1-Ubuntu - openjdk version "1.8.0_191" - gcc version 5.4.0 I also ran Spark integration tests and was able to get all tests passing after some minor modifications. Bryan O

Re: [ANNOUNCE] New Arrow committer: Ravindra Pindikura

2019-02-04 Thread Bryan Cutler
Congratulations Ravindra! On Mon, Feb 4, 2019, 7:40 PM Praveen Kumar Congrats again Ravindra :) > > On Tue, Feb 5, 2019 at 12:32 AM Robert Nishihara < > robertnishih...@gmail.com> > wrote: > > > Congratulations! > > > > On Mon, Feb 4, 2019 at 10:06 AM Antoine Pitrou > wrote: > > > > > > > > Cong

Re: [ANNOUNCE] New Arrow PMC member: Andy Grove

2019-02-04 Thread Bryan Cutler
Congratulations Andy! On Mon, Feb 4, 2019, 3:29 PM Philipp Moritz Congratulations! > > On Mon, Feb 4, 2019 at 3:16 PM Krisztián Szűcs > wrote: > > > Congrats Andy! :) > > > > On Mon, Feb 4, 2019 at 4:39 PM Wes McKinney wrote: > > > > > The Project Management Committee (PMC) for Apache Arrow has

Re: [DISCUSS] Joda Time -> Java8 Time

2019-02-19 Thread Bryan Cutler
nk we (dremio) need to better understand impact. Praveen, maybe you > > could take a look at impact? > > > > On Wed, Nov 14, 2018, 4:27 PM Bryan Cutler > > > > Hi all, > > > > > > I'm picking this back up again and have WIP pr at > > > htt

Re: [Discuss][Java] Codebase Housekeeping?

2019-02-27 Thread Bryan Cutler
These all sound good to me Micah, thanks for taking this on! Regarding the javadoc codestyle in (2), I believe it was disabled because there were just too many issues of missing docs at the time. Any documentation additions are definitely welcome and hopefully we can eventually enable the check :

Re: [ANNOUNCE] New Arrow committer: Chao Sun

2019-02-28 Thread Bryan Cutler
Congratulations Chao! On Thu, Feb 28, 2019 at 9:27 AM Neville Dipale wrote: > Congratulations Chao and Paddy! I'm loving the increase in velocity on the > Rust side > > On Thu, 28 Feb 2019, 17:17 Wes McKinney, wrote: > > > thank you Chao, and welcome! > > > > On Thu, Feb 28, 2019 at 6:18 AM pad

Re: [ANNOUNCE] New Arrow committer: Paddy Horan

2019-02-28 Thread Bryan Cutler
Congratulations Paddy! On Thu, Feb 28, 2019 at 7:14 AM Wes McKinney wrote: > Welcome Paddy and thank you! > > > On Thu, Feb 28, 2019 at 4:29 AM Uwe L. Korn wrote: > > > > On behalf of the Arrow PMC, I'm happy to announce that Paddy has an > > accepted an invitation to become a committer on Apac

Re: Arrow sync call today 12:00 EDT

2019-03-20 Thread Bryan Cutler
Attendees: Praveen Kumar Siddharth Teotia Bryan Cutler Since there were not many participants this time, we concluded early. On Wed, Mar 20, 2019 at 8:37 AM Wes McKinney wrote: > I'm unable to join today due to a scheduling conflict > > https://meet.google.com/vtm-teks-phx >

tensorflow-io Arrow Datasets and thoughts on support for tensor columns

2019-03-22 Thread Bryan Cutler
Hi All, Recently I have been working with the TensorFlow SIG-IO community to introduce Apache Arrow based Datasets for bringing Arrow data into TensorFlow. SIG-IO is a community maintained repository focused on input/output support for TF, see https://github.com/tensorflow/io (a lot of formats fro

Re: tensorflow-io Arrow Datasets and thoughts on support for tensor columns

2019-03-27 Thread Bryan Cutler
aryArray with 64-bit > offsets, so that buffers can be arbitrarily large (well, within 64-bit > address space at least) > > - Wes > > On Fri, Mar 22, 2019 at 1:24 PM Bryan Cutler wrote: > > > > Hi All, > > > > Recently I have been working with the Tens

Re: [VOTE] Release Apache Arrow 0.13.0 - RC4

2019-03-28 Thread Bryan Cutler
+1 (non-binding) Ran: ARROW_HAVE_CUDA=NO dev/release/verify-release-candidate.sh source 0.13.0 4 With tests: : ${TEST_DEFAULT:=1} : ${TEST_JAVA:=${TEST_DEFAULT}} : ${TEST_CPP:=${TEST_DEFAULT}} : ${TEST_GLIB:=0} : ${TEST_RUBY:=0} : ${TEST_PYTHON:=${TEST_DEFAULT}} : ${TEST_JS:=0} :

Re: [VOTE] Release Apache Arrow 0.13.0 - RC4

2019-03-28 Thread Bryan Cutler
ates > > Thanks, > -- > kou > > In > "Re: [VOTE] Release Apache Arrow 0.13.0 - RC4" on Thu, 28 Mar 2019 > 12:10:13 -0700, > Bryan Cutler wrote: > > > +1 (non-binding) > > > > Ran: > > ARROW_HAVE_CUDA=NO dev/release/verify-release

Re: [VOTE] Proposed changes to Arrow Flight protocol

2019-04-03 Thread Bryan Cutler
+1 (non-binding) On Wed, Apr 3, 2019 at 7:52 AM Jacques Nadeau wrote: > I'm +1 to all four (binding) > > On Wed, Apr 3, 2019 at 1:56 AM Antoine Pitrou wrote: > > > > > > > Le 03/04/2019 à 02:05, Wes McKinney a écrit : > > > Hi, > > > > > > David Li has proposed to make the following additions o

  1   2   3   4   >