Re: Authentication Redeisgn

Hey James, Looks like the doc is not entirely readable. I am getting: ' File is in owner's trash You will soon permanently lose access to this file. For continued access, please make a copy. ' When I access it. Best, Ryan Murray | OSS Engineer +447540852009 | rym...@dremio.com On Wed

Re: [VOTE] Release Apache Arrow 1.0.0 - RC2

LVM/Clang requirements? Best Ryan On Fri, Jul 24, 2020 at 1:18 PM Krisztián Szűcs wrote: > On Tue, Jul 21, 2020 at 3:57 PM Ryan Murray wrote: > > > > +0 (non-binding) > > > > > > I verified source, release, binaries, integration tests for Python, C++, > &g

Re: [VOTE] Release Apache Arrow 1.0.0 - RC2

+0 (non-binding) I verified source, release, binaries, integration tests for Python, C++, Java. All went fine except for a failed test in c++ Gandiva: [ FAILED ] TestProjector.TestDateTime Not sure if this is known or expected? On Tue, Jul 21, 2020 at 1:32 PM Andy Grove wrote: > +1

Re: [VOTE] Release Apache Arrow 1.0.0 - RC1

I've tested Java and it looks good. However the verify script keeps on bailing with protobuf related errors: 'cpp/build/orc_ep-prefix/src/orc_ep-build/c++/src/orc_proto.pb.cc' and friends cant find protobuf definitions. A bit odd as cmake can see protobuf headers and builds directly off master

Re: Timeline for next major Arrow release (1.0.0)

I just submitted https://github.com/apache/arrow/pull/7697 to update Docs for Java type support for 1.0.0, docs probably are not a blocker but should be an easy review. On Thu, Jul 9, 2020 at 7:19 PM Wes McKinney wrote: > hi David -- I agree with you, I just set ARROW-9265 to a Blocker and >

Re: Union integration test

sigh...thanks Antoine, that did the trick. That's what I get for programming pre-coffee ;-) On Tue, Jul 7, 2020 at 10:31 AM Antoine Pitrou wrote: > > Hello Ryan > > Le 07/07/2020 à 11:21, Ryan Murray a écrit : > > > > While trying to finish up the Union integrati

Union integration test

Hey all, While trying to finish up the Union integration test between Java & C++ https://github.com/apache/arrow/pull/7290 I came across a minor problem that I hope I could get some clarification on. I have the integration test for C++ -> Java passing but Java -> C++ is giving me trouble. The

Re: [VOTE] Removing validity bitmap from Arrow union types

+1 (non binding) On Tue, Jun 30, 2020 at 5:29 AM Ben Kietzman wrote: > +1 (non binding) > > On Tue, Jun 30, 2020, 00:24 Wes McKinney wrote: > > > +1 (binding) > > > > On Mon, Jun 29, 2020 at 11:09 PM Micah Kornfield > > wrote: > > > > > > +1 (binding) (I had a couple of nits on language,

Re: Missing artifact io.netty:netty-transport-native-unix-common:jar

Hey Talal, Try adding the `kr.motd.maven` extension as per https://github.com/rymurr/flight-spark-source/blob/master/pom.xml#L34 That should populate the ${os.detected.arch} variable required to pull in the netty native dependencies. Best, Ryan Murray | Principal Consulting Engineer

[jira] [Created] (ARROW-9016) [Java] Remove direct references to Netty/Unsafe Allocators

Ryan Murray created ARROW-9016: -- Summary: [Java] Remove direct references to Netty/Unsafe Allocators Key: ARROW-9016 URL: https://issues.apache.org/jira/browse/ARROW-9016 Project: Apache Arrow

[jira] [Created] (ARROW-9015) [Java] Make BaseBuffer package private

Ryan Murray created ARROW-9015: -- Summary: [Java] Make BaseBuffer package private Key: ARROW-9015 URL: https://issues.apache.org/jira/browse/ARROW-9015 Project: Apache Arrow Issue Type

[jira] [Created] (ARROW-8948) [Java][Integration] enable duplicate field names integration tests

Ryan Murray created ARROW-8948: -- Summary: [Java][Integration] enable duplicate field names integration tests Key: ARROW-8948 URL: https://issues.apache.org/jira/browse/ARROW-8948 Project: Apache Arrow

[jira] [Created] (ARROW-8947) [Java] MapWithOrdinal javadoc doesn't describe actual behaviour

Ryan Murray created ARROW-8947: -- Summary: [Java] MapWithOrdinal javadoc doesn't describe actual behaviour Key: ARROW-8947 URL: https://issues.apache.org/jira/browse/ARROW-8947 Project: Apache Arrow

Re: Arrow Flight connector for SQL Server

Hey Brendan, As Jacques promised here are a few things to act as pointers for your work on Flight: Our early release Flight connector[1] this fully supports single flight streams and partially supports parallel streams I also have a Spark DataSourceV2 client which may be of interest to you[2]

Re: Sparse Union format

is only a convention and we can't > > assume that 0 === Null? > > > > Best, > > Ryan > > > > [1] https://github.com/apache/arrow/blob/master/format/Schema.fbs#L235 > > > > On Tue, May 19, 2020 at 2:04 PM Antoine Pitrou > wrote: > > > >>

Re: Sparse Union format

/apache/arrow/blob/master/format/Schema.fbs#L235 On Tue, May 19, 2020 at 2:04 PM Antoine Pitrou wrote: > > Le 19/05/2020 à 13:43, Ryan Murray a écrit : > > Hey All, > > > > While working on https://issues.apache.org/jira/browse/ARROW-1692 I > noticed > > tha

Sparse Union format

Hey All, While working on https://issues.apache.org/jira/browse/ARROW-1692 I noticed that there is a difference between C++ and Java on the way Sparse Unions are handled. I haven't seen in the format spec which the correct is so I wanted to check with the wider community. c++ (and the

[jira] [Created] (ARROW-8739) Standardise Logger naming

Ryan Murray created ARROW-8739: -- Summary: Standardise Logger naming Key: ARROW-8739 URL: https://issues.apache.org/jira/browse/ARROW-8739 Project: Apache Arrow Issue Type: Task

[jira] [Created] (ARROW-8738) Investigate adding a getUnsafe method to vectors

Ryan Murray created ARROW-8738: -- Summary: Investigate adding a getUnsafe method to vectors Key: ARROW-8738 URL: https://issues.apache.org/jira/browse/ARROW-8738 Project: Apache Arrow Issue Type

[jira] [Created] (ARROW-8696) [Java] Convert tests to integration tests

Ryan Murray created ARROW-8696: -- Summary: [Java] Convert tests to integration tests Key: ARROW-8696 URL: https://issues.apache.org/jira/browse/ARROW-8696 Project: Apache Arrow Issue Type: Task

[jira] [Created] (ARROW-8695) [Java] remove references to NettyUtils in memory module

Ryan Murray created ARROW-8695: -- Summary: [Java] remove references to NettyUtils in memory module Key: ARROW-8695 URL: https://issues.apache.org/jira/browse/ARROW-8695 Project: Apache Arrow

[jira] [Created] (ARROW-8687) [Java] Finish move of io.netty.buffer.ArrowBuf

Ryan Murray created ARROW-8687: -- Summary: [Java] Finish move of io.netty.buffer.ArrowBuf Key: ARROW-8687 URL: https://issues.apache.org/jira/browse/ARROW-8687 Project: Apache Arrow Issue Type

[jira] [Created] (ARROW-8664) Add isSet skip check to all Vector types

Ryan Murray created ARROW-8664: -- Summary: Add isSet skip check to all Vector types Key: ARROW-8664 URL: https://issues.apache.org/jira/browse/ARROW-8664 Project: Apache Arrow Issue Type: Task

Re: [VOTE] Accept "DoExchange" RPC to Arrow Flight protocol

+1 non-binding On Sat, Mar 28, 2020 at 1:44 AM Wes McKinney wrote: > Hello, > > David M Li has proposed adding a "bidirectional" DoExchange RPC [1] to > the Arrow Flight Protocol [2]. In this client call, datasets (possibly > having different schemas) are sent by both the > client and server

[jira] [Created] (ARROW-8183) [c++][FlightRPC] Expose transport error metadata

Ryan Murray created ARROW-8183: -- Summary: [c++][FlightRPC] Expose transport error metadata Key: ARROW-8183 URL: https://issues.apache.org/jira/browse/ARROW-8183 Project: Apache Arrow Issue Type

[jira] [Created] (ARROW-8181) [Java][FlightRPC] Expose transport error metadata

Ryan Murray created ARROW-8181: -- Summary: [Java][FlightRPC] Expose transport error metadata Key: ARROW-8181 URL: https://issues.apache.org/jira/browse/ARROW-8181 Project: Apache Arrow Issue

Re: [Java] PR Reviewers

Hey all, I would love to help out. Is there any specific ones that are relatively easy for me to get started on? On Mon, 27 Jan 2020, 18:31 Bryan Cutler, wrote: > Hi Micah, I don't have a ton of bandwidth at the moment, but I'll try to > review some more PRs. Anyone, please feel free to ping me

Re: Horizontal scaling design suggestion: Apache arrow flight

hubusercontent.com/6141965/67092386-b0012c00-f1cc-11e9-9ce2-d657001a85f7.png > > > > Just wanted to discuss if any PR is in progress for horizontal scaling in > Arrow flight, or any design doc is under discussion. > -- Ryan Murray | Principal Consulting Engineer +447540852

Re: [Discuss][FlightRPC] Extensions to Flight: "DoBidirectional"

server goes away in between the two > calls, but it then restarts or there is another instance available, > the client will happily reconnect to the new server without warning. > > Thanks, > David > > On 10/15/19, Ryan Murray wrote: > > Hey David, > > > > I think this propos

Re: [Discuss][FlightRPC] Extensions to Flight: "DoBidirectional"

tively straightforward > addition - it is essentially "DoPutThenGet". > > This is a format change and would require a vote. I've decided to > table the other format change I had proposed (on DoPut), as it doesn't > functionally change Flight, just the interpretation of the semant

Re: [ANNOUNCE] New Arrow committer: David M Li

Congratulations David! On Sat, 31 Aug 2019, 03:56 Micah Kornfield, wrote: > Congrats David, well desrved. > > On Fri, Aug 30, 2019 at 2:02 PM Bryan Cutler wrote: > > > Congrats David! > > > > On Fri, Aug 30, 2019 at 10:19 AM Antoine Pitrou > > wrote: > > > > > > > > Congratulations David and

Re: [VOTE] Proposed addition to Arrow Flight Protocol

> [1] https://github.com/apache/arrow/pull/4980 > > > > On Mon, Aug 19, 2019 at 2:52 AM Antoine Pitrou wrote: > > > > > +1 (binding) > > > > Regards > > > > Antoine. > > > > > > Le 16/08/2019 à 07:44, Micah Kornfield a é

Re: [DISCUSS] Add GetFlightSchema to Flight RPC

public and private votes in my role as a PMC > > member. I would appreciate if another PMC would assist with this vote. > > > > Thanks > > > > On Wed, Aug 14, 2019 at 5:37 PM Ryan Murray wrote: > > > > > > Hi All, > > > > > >

Re: [DISCUSS] Add GetFlightSchema to Flight RPC

Hi All, Does this require a vote? If yes what is the process for initiating one & if no I hope this is enough time for feedback and I would like to remove the draft designation from the PR Best, Ryan On Wed, Aug 7, 2019 at 9:31 AM Ryan Murray wrote: > As per everyone's feedback I have

Re: [DISCUSS] Add GetFlightSchema to Flight RPC

e said. > > On Thu, Aug 1, 2019 at 8:17 AM Ryan Murray wrote: > > > Thanks Wes, > > > > The descriptor is only there to maintain a bit of symmetry with > > GetFlightInfo. Happy to remove it, I don't think its necessary and > already > > a few people a

Re: [DISCUSS] Add GetFlightSchema to Flight RPC

; > Hi Ryan, > > > > Thanks for writing this up! I made a couple of minor comments in the > > doc/implementation, but overall I'm in favor of having this RPC > > method. > > > > Best, > > David > > > > On 8/1/19, Ryan Murray wrote: > > >

[DISCUSS] Add GetFlightSchema to Flight RPC

Hi All, Please see the attached document for a proposed addition to the Flight RPC[1]. This is the result of a previous mailing list discussion[2]. I have created the Pull Request[3] to make the proposal a little more concrete. Please let me know if you have any

[jira] [Created] (ARROW-6094) Add GetFlightSchema to Flight RPC

Ryan Murray created ARROW-6094: -- Summary: Add GetFlightSchema to Flight RPC Key: ARROW-6094 URL: https://issues.apache.org/jira/browse/ARROW-6094 Project: Apache Arrow Issue Type: Task

Re: Spark and Arrow Flight

ply specifying it as best-effort.) > > Best, > David > > On 7/10/19, Ryan Murray wrote: > > Hey Wes, > > > > Would be happy to! Jacques and I had originally thought to try and get it > > into Spark but perhaps Arrow might be a better home. I think the only &g

Re: Spark and Arrow Flight

PM Wes McKinney wrote: > Hi Ryan, have you thought about developing this inside Apache Arrow? > > On Tue, Jul 9, 2019, 5:42 PM Bryan Cutler wrote: > > > Great, thanks Ryan! I'll take a look > > > > On Tue, Jul 9, 2019 at 3:31 PM Ryan Murray wrote: > >

Re: Spark and Arrow Flight

turn the schema > > >> as null from GetFlightInfo. > > >> > > >> I think it's valid to want to make a single GetFlightInfo RPC request > > >> that returns _both_ the schema and the query plan. > > >> > > >> Thoughts from others? &

Re: Flight authentication interoperability

compatibility. > - We should start thinking about async APIs and/or timeouts in any > sort of API that makes a network call (there's already a JIRA: > https://issues.apache.org/jira/browse/ARROW-1009), since "never > returns" is a terrible failure mode > > Best, > David

Re: Flight authentication interoperability

emented is: > >>> - the Arrow codebase simply passes opaque tokens around > >>> - interpretation of tokens is handled by application code > >>> - marshalling of tokens is handled by Protocol Buffers > >>> > >>> So unless something s

Re: [VOTE] Release Apache Arrow 0.14.0 - RC0

7a82e6be-f4a5-240b-389a-4cf9cd4fb...@python.org> > >> >> > >> "Re: [VOTE] Release Apache Arrow 0.14.0 - RC0" on Tue, 2 Jul > 2019 > >> >> > >> 11:36:09 +0200, > >> >> > >> Antoine Pitrou wrote: > >

Spark and Arrow Flight

Hi All, I have been working on building an arrow flight source for spark. The goal here is for Spark to be able to use a group of arrow flight endpoints to get a dataset pulled over to spark in parallel. I am unsure of the best model for the spark <-> flight conversation and wanted to get your

[jira] [Created] (ARROW-5249) Java Flight client doesn't handle auth correctly in some cases

Ryan Murray created ARROW-5249: -- Summary: Java Flight client doesn't handle auth correctly in some cases Key: ARROW-5249 URL: https://issues.apache.org/jira/browse/ARROW-5249 Project: Apache Arrow

Job posting

Hey All, Apologies ahead of time for the potential spam. I have been working with Jacques and co for about 6 months to bring Dremio to our front office trading organisation at UBS. We are now expanding the team and I am looking to hire some killer devs in London to work on Arrow, Gandiva and