Re: [VOTE] Proposed addition to Arrow Flight: Arrow Flight RPC

+1 On Wed, Dec 8, 2021 at 11:42 PM Micah Kornfield wrote: > > > > Given that the C++ and Java components are in separate PRs, would it be > > acceptable to add after the initial merge? > > > OK by me. We could also create a branch to merge the PRs add the > integration tests, and then merge

Re: [Gandiva] How to optimize per CPU feature

On Mon, Dec 16, 2019 at 7:55 AM Yibo Cai wrote: > On 12/13/19 7:45 PM, Ravindra Pindikura wrote: > > On Fri, Dec 13, 2019 at 3:41 PM Yibo Cai wrote: > > > >> Hi, > >> > >> Thanks to pravindra's patch [1], Gandiva loop vectorization is okay now.

Re: [Gandiva] How to optimize per CPU feature

On Fri, Dec 13, 2019 at 3:41 PM Yibo Cai wrote: > Hi, > > Thanks to pravindra's patch [1], Gandiva loop vectorization is okay now. > > Will Gandiva detects CPU feature at runtime? My test CPU supports sse to > avx2, but I only > see "target-features"="+fxsr,+mmx,+sse,+sse2,+x87" in IR, and final

Re: [Gandiva] question about IR optimization

On Wed, Dec 11, 2019 at 4:09 PM Yibo Cai wrote: > Hi, > > I'm trying to figure out how Gandiva works by tracing unit test > TestSimpleArichmetic[1]. > I met with a problem about Gandiva IR generator and optimizer, would like > to seek for > help from community. > > I'm focusing on case "b+1",

Re: [Gandiva] question about IR optimization

I've opened jira to track this issue. Thanks for catching this ! https://issues.apache.org/jira/browse/ARROW-7378 On Wed, Dec 11, 2019 at 11:48 PM Ravindra Pindikura wrote: > > I found that there is something in this PR (the last change to > llvm_generator.cc) that broke

Re: [Gandiva] question about IR optimization

4, !alias.scope !6, !noalias !3* * %index.next = add i64 %index, 8* * %13 = icmp eq i64 %index.next, %n.vec* * br i1 %13, label %middle.block, label %vector.body, !llvm.loop !8* On Wed, Dec 11, 2019 at 11:25 PM Ravindra Pindikura wrote: > I'll debug this and get back to you - I suspect s

Re: [Gandiva] question about IR optimization

I'll debug this and get back to you - I suspect some recent change broke this functionality. On Wed, Dec 11, 2019 at 10:06 PM Francois Saint-Jacques < fsaintjacq...@gmail.com> wrote: > It seems that LLVM can't auto vectorize. I don't have a debug build, > so I can't get the `-debug-only`

Re: questions about Gandiva

On Thu, Oct 31, 2019 at 10:56 PM Wes McKinney wrote: > hi > > On Thu, Oct 31, 2019 at 12:11 AM Yibo Cai wrote: > > > > Hi, > > > > Arrow cpp integrates Gandiva to provide low level operations on arrow > buffers. [1][2] > > I have some questions, any help is appreciated: > > - Arrow cpp already

Re: questions about Gandiva

On Fri, Nov 1, 2019 at 10:41 AM Yibo Cai wrote: > Thanks Wes. Arrow is a very exciting project. > I'm from Arm. We are interested in arrow and would like to study and help > improving arrow. > If you are familiar with LLVM/JIT, you could help us with improving the optimisation passes in gandiva

Re: [DISCUSS][Java] Builders for java classes

On Thu, Oct 24, 2019 at 10:01 AM Micah Kornfield wrote: > As part a PR Ji Liu has made to help populate data for test cases [1], the > question came up on whether we should provide a more builder classes in > java for ValueVectors. The proposed implementation would wrap the existing > Writer

Re: [DISCUSS][Java] Should null values in VariableWidthVector/ListVector always takes 0 space?

On Wed, Aug 28, 2019 at 12:32 PM Fan Liya wrote: > Dear all, > > In the discussion of this PR (https://github.com/apache/arrow/pull/5073), > we are faced with a problem: > > Normally, in a VariableWidthVector (e.g. VarCharVector), a null value is > supposed to take no space in the data buffer.

Re: Gandiva Java benchmarks

On Sat, Aug 17, 2019 at 5:09 AM Rui Wang wrote: > I got help for a pointer to Gandiva cpp's micro benchmark > < > https://github.com/apache/arrow/blob/master/cpp/src/gandiva/tests/micro_benchmarks.cc > >. > I will start from there. > There is also a primitive java variant here :

Re: Re: Sharing Java Arrow Buffer with C++ in same process

*gandiva* does the sharing in one direction i.e buffers allocated in java, and passed along to c++ in jni calls. The java side code of extracting of the address and passing it down is here

Re: [Discuss][Java] Make the semantics of lastSet consistent

On Sat, Jul 6, 2019 at 5:48 AM Jacques Nadeau wrote: > Ravindra, Praveen and Prudhvi, can you confirm the ramifications of this > change and what impact this inconsistency has had downstream? > Looks like the ListVector treats lastSet as the "last set index" in the offsets buffer. It treats it

Re: [VOTE] Release Apache Arrow 0.14.0 - RC0

rg/jira/browse/ARROW-5786 > > > Thanks, > -- > kou > > In > "Re: [VOTE] Release Apache Arrow 0.14.0 - RC0" on Wed, 3 Jul 2019 > 06:55:52 +0530, > Ravindra Pindikura wrote: > > > I tried "./dev/release/verify-release-candidate.sh source 0.14.0

Re: [VOTE] Release Apache Arrow 0.14.0 - RC0

On Wed, Jul 3, 2019 at 7:06 AM Wes McKinney wrote: > For the record, because Flight is so new and isn't being tested by very > many contributors in their environments, I would expect a lot of problems > and don't think they pose an issue for releasing. Let's open follow up > JIRAs > done,

Re: [VOTE] Release Apache Arrow 0.14.0 - RC0

On Wed, Jul 3, 2019 at 7:04 AM Wes McKinney wrote: > @Ravindra, could you clarify what point #2 means? > As part of the release verify script, the mvn tests are run for all modules (memory, vector, flight, jdbc, ..). I saw that the gandiva tests aren't running as part of that. @kou confirmed

Re: [VOTE] Release Apache Arrow 0.14.0 - RC0

I tried "./dev/release/verify-release-candidate.sh source 0.14.0 0" on mac mojave. 1. I consistently get this error with flight tests [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.04 s <<< FAILURE! - in org.apache.arrow.flight.TestServerOptions [ERROR]

Re: [ANNOUNCE] New Arrow committer: Praveen Kumar

Congrats Praveen! Sent from my iPhone > On 28-Jun-2019, at 1:55 AM, Wes McKinney wrote: > > On behalf of the Arrow PMC I'm happy to announce that Praveen has > accepted an invitation to become an Arrow committer! > > Welcome, and thank you for your contributions!

Re: [Disscuss][Java] Add more check style rule for Java code

+1 On Fri, Jun 14, 2019 at 9:51 AM Micah Kornfield wrote: > I'm in favor of the changes also, but since they affect the entire Java > codebase I wanted people to be aware of them in case there are any > objections. I plan to merge the PR tomorrow evening Past or over the > weekend unless

Re: [VOTE] Formalizing "Extension Type" metadata in Arrow binary protocol

+1 On Tue, Jun 11, 2019 at 10:24 PM Micah Kornfield wrote: > +1 (non-binding) > > On Tue, Jun 11, 2019 at 6:08 AM Antoine Pitrou wrote: > > > > > Le 10/06/2019 à 22:28, Wes McKinney a écrit : > > > > > > Please vote to accept these changes (see [3] for the actual changes). > > > The vote will

Re: [DISCUSS] 32- and 64-bit decimal types

On Tue, Jun 11, 2019 at 2:48 AM Wes McKinney wrote: > On the 1.0.0 protocol discussion, one item that we've skirted for some > time is other decimal sizes: > > https://issues.apache.org/jira/browse/ARROW-2009 > > I understand this is a loaded subject since a deliberate decision was > made to

Re: [Discuss][Format][Java] Finalizing Union Types

On Sat, May 25, 2019 at 12:29 PM Micah Kornfield wrote: > Thanks for the responses, I've clipped the questions and provided responses > inline. > > is the proposal that both cpp & java will support only option 2 ? > > I guess 1 is a subset of 2 anyway. > > CPP already supports option 2. I would

Re: [DISCUSS][Java] How to solve the problem of OutOfMemoryException when there is sufficient memory?

On Thu, May 30, 2019 at 12:23 PM Micah Kornfield wrote: > (Adding Java to thread title) > > For more context, I pushed back on the changes in > https://github.com/apache/arrow/pull/4358 because they don't seem typical > in memory management systems (i.e. they expose internal implementation >

Re: [Discuss][Format][Java] Finalizing Union Types

Micah, Couple of questions inline : On Tue, May 21, 2019 at 10:21 AM Micah Kornfield wrote: > In the past [1] there hasn't been agreement on the final requirements for > union types. > > Briefly the two approaches that are currently advocated: > 1. Limit unions to only contain one field of

Re: [Discuss][Format] Zero size record batches

On Tue, May 21, 2019 at 10:35 AM Micah Kornfield wrote: > Today, the format docs are ambiguous on whether zero sized batches are > supported. Wes opened a PR [1] for empty record batches that shows C++ > handles them but Java and javascript fail to handle them. > > > I'd like to propose: > 1.

Arrow sync call

Hi, We had only a few attendees today : Neal Richardson (Ursa labs) Wes (Ursa labs) Ravindra (dremio) Hatem Halel (MathWorks) Areg Melik (Intel) There were no specific issues, and wes seemed to have network issues. So, we had a very short meeting. - Neal described a mac build issue due to

Re: CMake refactor Heads-up

On Sat, Mar 16, 2019 at 2:38 AM Suvayu Ali wrote: > Hi Uwe, > > On Fri, Mar 15, 2019 at 10:38:32AM -0400, Uwe L. Korn wrote: > > > > we have merged the CMake refactor yesterday > https://github.com/apache/arrow/pull/3688 and this means that the build > system behaves a bit different. The main

Re: [Discuss][Java, Non-C++ generally] Support for 64-bit int array lengths?

@Jacques Nadeau would have more background on this. Here's my understanding : On Thu, Mar 14, 2019 at 12:08 PM Micah Kornfield wrote: > I was working on a proof of concept java implementation for LargeList [1] > implementation (64-bit array offsets). Our Java implementation doesn't > appear

Re: Boost and manylinux CI builds

g through the > email logs Sure Wes. Will do via ARROW-4756 > > Thanks! > Wes > > On Fri, Mar 1, 2019 at 9:58 PM Ravindra Pindikura wrote: >> >> Thanks Uwe. >> >> For the record (in case someone needs to do it again), these are the steps : >>

Re: Boost and manylinux CI builds

image > to build. > > Uwe > > On Wed, Feb 27, 2019, at 8:41 AM, Krisztián Szűcs wrote: >> Hi Ravindra! >> >> You'll need to rebuild the docker image and change this line accordingly: >> https://github.com/apache/arrow/blob/master/ci/travis_script_manylinux.

Re: Flaky Travis CI builds on master

> On Feb 28, 2019, at 2:10 PM, Antoine Pitrou wrote: > > > Le 28/02/2019 à 07:53, Ravindra Pindikura a écrit : >> >> >>> On Feb 27, 2019, at 1:48 AM, Antoine Pitrou wrote: >>> >>> On Tue, 26 Feb 2019 13:39:08 -0600 >>> Wes McKin

Re: Flaky Travis CI builds on master

> On Feb 27, 2019, at 1:48 AM, Antoine Pitrou wrote: > > On Tue, 26 Feb 2019 13:39:08 -0600 > Wes McKinney wrote: >> hi folks, >> >> We haven't had a green build on master for about 5 days now (the last >> one was February 21). Has anyone else been paying attention to this? >> It seems we

Boost and manylinux CI builds

Hi, I added an include for boost header file in gandiva. This compiles on ubuntu/Mac/windows, but fails with the manylinux CI entry. I’m getting a compilation failure : https://travis-ci.org/apache/arrow/jobs/498718755

Re: [Discuss][C++] Hashing floating point numbers

> On Feb 26, 2019, at 10:32 AM, Micah Kornfield wrote: > > Implementing compute kernels that depend on hashing has raised a couple of > edge cases that are worth discussing. In particular > the following points need to be resolved (I opened a JIRA [1] to track the > fixes). In particular: >

Re: [DISCUSSION] Representing Map datatype using ValueVectors

> On Feb 25, 2019, at 8:02 PM, Ihor Huzenko wrote: > > Hello Arrow Team, > > My name is Igor Guzenko. I'm currently working on task related to > complex types in Apache Drill [1], and bumped into an issue that Drill > hasn't > appropriate vector for representing canonical (java-like) Map

Re: Compute kernels and Gandiva operators

avindra, >> >> >> On Wed, Feb 13, 2019 at 1:34 AM Ravindra Pindikura >> wrote: >>> >>> Hi, >>> >>> I was looking at the recent checkin for arrow kernels, and started to >> think of how they would work alongside Gandiva.

Compute kernels and Gandiva operators

Hi, I was looking at the recent checkin for arrow kernels, and started to think of how they would work alongside Gandiva. Here are my thoughts : 1. Gandiva already has two high-level operators namely project and filter, with runtime code generation * It already supports 100s of functions

Re: [ANNOUNCE] New Arrow committer: Ravindra Pindikura

Thanks everyone. It’s been a pleasure to work with the arrow community, and I feel honored. > On Feb 5, 2019, at 9:28 AM, Bryan Cutler wrote: > > Congratulations Ravindra! > > On Mon, Feb 4, 2019, 7:40 PM Praveen Kumar >> Congrats again Ravindra :) >> >> On Tue, Feb 5, 2019 at 12:32 AM

Re: Git workflow question

Ok. Thanks, wes. > On Jan 30, 2019, at 8:43 PM, Wes McKinney wrote: > > hi Ravindra, > > On Wed, Jan 30, 2019 at 12:00 AM Ravindra Pindikura <mailto:ravin...@dremio.com>> wrote: >> >> >> >> >>> On Jan 30, 2019, at 11:05 AM, Andy Gr

Re: Git workflow question

> On Jan 30, 2019, at 11:05 AM, Andy Grove wrote: > > Got it. Thanks for the clarification. > > On Tue, Jan 29, 2019 at 10:30 PM Wes McKinney wrote: > >> hi Andy, >> >> yes, in this project I recommend never using "git merge". Merge >> commits just make branches harder to maintain when

Re: [Format] Passing selection masks with Arrow record batches

> On Jan 28, 2019, at 11:47 AM, Wes McKinney wrote: > > On Mon, Jan 28, 2019 at 12:05 AM Ravindra Pindikura <mailto:ravin...@dremio.com>> wrote: >> >> >> >>> On Jan 28, 2019, at 11:22 AM, Wes McKinney wrote: >>> >>&g

Re: [Format] Passing selection masks with Arrow record batches

> On Jan 28, 2019, at 11:22 AM, Wes McKinney wrote: > > I was having a discussion recently about Arrow and the topic of > server-side filtering vs. client-side filtering came up. > > The basic problem is this: > > If you have a RecordBatch that you wish to filter out some of the > "rows",

Re: Cleaning up the Arrow Java build output

I’ve assigned to myself. > On Jan 21, 2019, at 11:39 PM, Wes McKinney wrote: > > Hello, could someone take a look at this? > > Thank you > > On Thu, Jan 17, 2019 at 2:34 PM Wes McKinney wrote: >> >> hi folks, >> >> The Java build has recently acquired a bunch of new verbose debugging >>

Re: Debugging Gandiva failures in python CI tests

as previously in Java. > Here is a workaround that works here: > https://gist.github.com/pitrou/7eed7b980eec3187d619bc146b75cc4b > > Regards > > Antoine. > > > Le 07/01/2019 à 07:16, Ravindra Pindikura a écrit : >> I missed mentioning this. The repro requires my changes for decimal (in the >> PR). >> >> https://github.com/apache/arrow/pull/2942 >> <https://github.com/apache/arrow/pull/2942>

Re: Debugging Gandiva failures in python CI tests

I missed mentioning this. The repro requires my changes for decimal (in the PR). https://github.com/apache/arrow/pull/2942 <https://github.com/apache/arrow/pull/2942> > On Jan 6, 2019, at 4:23 PM, Ravindra Pindikura wrote: > > > >> On Jan 6, 2019, at 3:38 PM, Antoi

Re: Debugging Gandiva failures in python CI tests

The test silently dies without printing any error msg. So, I modified test_gandiva.py to use pdb and stepped through till I got the error (pdb.set_trace() in test_tree_expr_builder()). Thanks & regards, Ravindra. > > Regards > > Antoine. > > > Le 06/

Re: Debugging Gandiva failures in python CI tests

> On Dec 29, 2018, at 8:03 PM, Ravindra Pindikura wrote: > > > >> On Dec 28, 2018, at 1:38 AM, Wes McKinney wrote: >> >> You might have to wait for Antoine's help after the holidays. In the >> meantime: do you get the error when building both with gcc 4

Re: Debugging Gandiva failures in python CI tests

m > is a common troublemaker for LLVM. I wonder if the code can be > refactored to avoid using std::cout Thanks Wes. I’ll wait for Antoine. In the meanwhile, I’ll work on the refactoring, just in case. > > On Thu, Dec 27, 2018 at 3:56 AM Ravindra Pindikura > wrote: >> >>

Re: Debugging Gandiva failures in python CI tests

> > On Tue, Dec 25, 2018 at 12:53 PM Ravindra Pindikura > wrote: >> >> Hi, >> >> My change (decimal support in gandiva) causes a CI failure in python >> test_gandiva.py >> >> https://travis-ci.org/apache/arrow/jobs/471874254#L7167 >> >&

Debugging Gandiva failures in python CI tests

Hi, My change (decimal support in gandiva) causes a CI failure in python test_gandiva.py https://travis-ci.org/apache/arrow/jobs/471874254#L7167 However, the travis log has no further information on the failure. To debug this, I tried setting up a python dev environment on ubuntu 14 using

Re: Weird symbol resolution error with llvm JIT in gandiva jni

. https://github.com/apache/arrow/blob/master/cpp/src/gandiva/engine.cc#L61 <https://github.com/apache/arrow/blob/master/cpp/src/gandiva/engine.cc#L61> > > > Le 17/12/2018 à 13:50, Ravindra Pindikura a écrit : >> Antoine, >> >> I set ARROW_GANDIVA_STATIC_LI

Re: Weird symbol resolution error with llvm JIT in gandiva jni

ng. > > Regards > > Antoine. > > > Le 16/12/2018 à 14:03, Antoine Pitrou a écrit : >> >> Does the static gandiva lib bundle libstdc++? Otherwise you'll probably >> need to load it explicitly in the Java process. >> >> Regards >> >> Anto

Re: Weird symbol resolution error with llvm JIT in gandiva jni

I think this happens if any of the cpp files that are used for IR generation (clang —emit-llvm) includes . For decimal support, I’m generating IR code from arrow/util/decimal.cc <http://decimal.cc/> which includes (many times in the header file chain). > On Dec 16, 2018, at 9:14 AM,

Weird symbol resolution error with llvm JIT in gandiva jni

Hi, I’ve been working on adding decimal support in gandiva - I’m hitting a symbol resolution error with std::ios_base::Init with the java tests on ubuntu. Any ideas on what I may be doing wrong ? LLVM ERROR: Program used external function '_ZNSt8ios_base4InitC1Ev' which could not be resolved!

Re: Access Gandiva filter result by array index

> On Dec 14, 2018, at 3:22 AM, Suvayu Ali wrote: > > Hi everyone, > > Maybe I'm missing something obvious, but for the life of me, I can't > figure out how I can access the elements of an array after a Gandiva > filter operation. > > I have linked a minimal example at the end which I compile

Re: Arrow sync call starting shortly at 12p Eastern / 17:00 UTC

Attendees : Wes, Sidd, Bryan, Francois, Hatem, Nick, Shyam, Ravindra, Matt Wes: - do not rush the 0.12 release before the holidays, instead target the release for early next year - request everyone to look at PRs in the queue, and help by doing reviews Wes/Nick - queried about Interest in

Windows support for gandiva

Hi all, Would anyone be interested in working on adding windows support for gandiva ? My knowledge of windows is severely limited - but the potential issues could be : - interaction with the llvm APIs (should work, but there may be surprises if some of them aren’t available on windows) - the