Decimal Format

Hi all, I’d like to propose the following changes to the in-memory Decimal128 format and solicit feedback. 1. When converting to and from an array of bytes, the input bytes are *assumed* to be in big-endian order and the output bytes are *guaranteed* to be in big-endian order.

Re: [ANNOUNCE] New Arrow committers: Li Jin, Siddharth Teotia, and Paul Taylor

Sweet! Welcome! On Wed, Oct 4, 2017 at 10:22 AM Wes McKinney wrote: > On behalf of the Arrow PMC, I'm pleased to announce that Li Jin, > Siddharth Teotia, and Paul Taylor have been invited to be Arrow > committers. > > We are grateful for your contributions to the project

Re: [VOTE] Release Apache Arrow 0.7.1 - RC1

+1 (non-binding) * Verified signatures with release verification script * Ran C++, Python + parquet support unit tests On Thu, Sep 28, 2017 at 1:58 PM Gang Wang wrote: > +1 Looks good to me. > > Gary > > On 2017-09-27 07:01, Wes McKinney wrote: > > Hello

Re: [ANNOUNCE] New Arrow committers: Phillip Cloud and Bryan Cutler

ehalf of the Arrow PMC, I'm pleased to announce that Phillip Cloud > > and Bryan Cutler have been invited to be Arrow committers. > > > > We are grateful for your contributions to the project and look forward > > to growing the community together. > > > > Welcome, Phillip and Bryan, and congrats! > > > > - Wes > > >

Re: [VOTE] Release Apache Arrow 0.8.0 RC2

+1 Verified signatures and checksums Ran unit tests for arrow python and c++ using clang on Arch Linux as well as parquet-cpp unit tests. On Sat, Dec 16, 2017, 00:25 Jacques Nadeau wrote: > Sounds good. > > +1 (binding) > > On Dec 15, 2017 7:05 PM, "Wes McKinney"

Re: [DISCUSS] Removing the "page" field from the Buffer record batch Arrow metadata

+1 agree it should be solved separately. On Fri, Oct 20, 2017, 21:24 Wes McKinney wrote: > Patch here: https://github.com/apache/arrow/pull/1225 > > On Fri, Oct 20, 2017 at 12:13 AM, Li Jin wrote: > > +1 for the change too. >

Re: Faster PySpark UDFs using Apache Arrow in Spark 2.3.0

Congrats Li! This is awesome. On Mon, Oct 30, 2017 at 2:05 PM Wes McKinney wrote: > hi all, > > One of our newest committers, Li Jin, has been driving efforts to > speed up Python UDFs in Spark using Arrow. This was just written about > today: > > >

Re: Arrow sync at 12pm EDT today

Meeting notes from the call: Attendees/Topics to discuss - Wes - Packaging - Uwe - Packaging - Simba - Li Two Sigma - Ethan Two Sigma - Josh Two Sigma - Exceptions vs status codes - Class design question

Re: Arrow sync at 12pm Eastern today

I won't be able to make it today. On Wed, Jun 27, 2018 at 10:48 AM Wes McKinney wrote: > https://meet.google.com/vtm-teks-phx >

Re: Gandiva Initiative

This is super exciting. In particular, I think for ibis ( http://docs.ibis-project.org/) building up expressions and executing them using gandiva would fit nicely as another in-memory backend alongside the pandas backend. I think it would also drive some use cases forward for more complex datatype

Re: Arrow PR backlog: please help

What is the main barrier to getting CircleCI to work with Apache projects? On Thu, Feb 1, 2018 at 2:03 PM Uwe L. Korn wrote: > I just went over a lot of open PRs and sadly I wasn't able to reduce the > number of open ones significantly. Some of them make slow progress and it >

Re: Arrow PR backlog: please help

eb 1, 2018 at 2:15 PM, Uwe L. Korn <uw...@xhochy.com> wrote: > > CircleCI requires more permissions than Travis and Apache Infra don't > want to give it to them. This might be different now that we have the > gitbox setup instead of the previous Apache git mirroring. > > &

Re: Arrow PR backlog: please help

JIRA-ized: https://issues.apache.org/jira/browse/INFRA-15964 On Thu, Feb 1, 2018 at 3:59 PM Phillip Cloud <cpcl...@gmail.com> wrote: > Ok, will do. > > On Thu, Feb 1, 2018 at 3:56 PM Wes McKinney <wesmck...@gmail.com> wrote: > >> You'll have to open an INFRA tick

Re: Arrow PR backlog: please help

Ok, will do. On Thu, Feb 1, 2018 at 3:56 PM Wes McKinney <wesmck...@gmail.com> wrote: > You'll have to open an INFRA ticket on JIRA > > On Thu, Feb 1, 2018 at 3:53 PM, Phillip Cloud <cpcl...@gmail.com> wrote: > > I'll follow up with them and shoot an email over to s

Re: Arrow for MATLAB?

The MathWorks is in the process of starting to contribute. I spoke with them a couple weeks ago about this and they were excited about it. I can ping them to see if they are still interested. On Tue, Feb 13, 2018, 09:24 Uwe L. Korn wrote: > Hello Joris, > > this is only due to

Decimal NaNs

Recently someone opened ARROW-2145 asking for support for non-finite values, such as NaN and infinity. It may seem like a “no-brainer” to implement this, but there’s no real consistency on how to implement it or *even to implement

Re: [ANNOUNCE] New Arrow committers

Congratulations to everyone, welcome! On Wed, Feb 14, 2018 at 12:23 PM Li Jin wrote: > Congrats! > > On Wed, Feb 14, 2018 at 12:14 PM, Philipp Moritz > wrote: > > > Congrats to the new committers! > > > > On Wed, Feb 14, 2018 at 9:07 AM, Robert

Upgrade to Clang-5.0

After https://github.com/apache/arrow/pull/1597 is merged arrow-cpp will now pin clang to version 5.0. Please upgrade your clang installations. Thanks, Phillip

Re: [DISCUSS] Concerns about the Arrow Slack channel

Dhruv, I'm curious why the dev mailing list is considered intrusive. Can you expand a bit on that? I've always thought of mailing lists to be *the* place where people go to ask questions about a project in a way that is open to all. They are also archived and organized in some way that makes it

Re: [VOTE] Release Apache Arrow 0.10.0 (RC0)

Marco, what would be even better is if you would test your application against Antoine's PR before it gets merged. On Thu, Aug 2, 2018 at 2:37 PM Phillip Cloud wrote: > I'll cut another one. Can someone review Antoine's PR, like ASAP, so that > I can cut another RC? > > On Thu, Aug

Re: [VOTE] Release Apache Arrow 0.10.0 (RC0)

I'll cut another one. Can someone review Antoine's PR, like ASAP, so that I can cut another RC? On Thu, Aug 2, 2018 at 2:32 PM Wes McKinney wrote: > -1 > > It's up to Phillip whether he wants to cancel the RC, but note that > releases cannot be vetoed. > > @Marco, thanks for reporting -- is

Re: [VOTE] Release Apache Arrow 0.10.0 (RC0)

t was just > >implicitly assumed to exist (fork stability). So the regression kinda > >"normal". > >> > >> Marco > >> > >> On August 2, 2018 8:40:05 PM GMT+02:00, Phillip Cloud > > wrote: > >>>Marco, what would be even bet

[VOTE] Release Apache Arrow 0.10.0 (RC1)

Hello all, I'd like to propose the 1st release candidate (rc1) of Apache Arrow version 0.10.0. rc0 was invalidated because of ARROW-2963, the fix for which is incorporated into rc1. This release candidate is distinct from previous releases in that it includes binary artifacts such as python

Re: [VOTE] Release Apache Arrow 0.10.0 (RC1)

+1 (binding) ran ./dev/release/verify-release-candidate.sh 0.10.0 1 * Arch Linux * java8 * ruby 2.5.1p57 * python3.6 * clang 6.0.0 * node v10.5.0 On Fri, Aug 3, 2018 at 10:55 AM Phillip Cloud wrote: > Hello all, > > I'd like to propose the 1st release candidate (rc1) of Apache Arro

Re: [VOTE] Release Apache Arrow 0.10.0 (RC1)

That should say "2nd" release candidate by the way. Apologies. On Fri, Aug 3, 2018 at 11:29 AM Phillip Cloud wrote: > +1 (binding) > > ran ./dev/release/verify-release-candidate.sh 0.10.0 1 > > * Arch Linux > * java8 > * ruby 2.5.1p57 > * python3.6 > * cl

Re: Working towards 0.10.0 release candidate

ttps://github.com/apache/arrow/pull/2340 that would be helpful. I > think this is the last code-related patch outstanding > > Thank you, > Wes > > On Mon, Jul 30, 2018 at 9:38 AM, Phillip Cloud wrote: > > Wanted to update everyone here regarding the ability to cut a release &g

Re: Working towards 0.10.0 release candidate

Wanted to update everyone here regarding the ability to cut a release candidate for 0.10.0. The last remaining set of tasks is to be able to use the new packaging tool (crossbow.py) to build binary artifacts from a source archive. What this means is that we'll have to move the release scripts

Re: [VOTE] Release Apache Arrow 0.10.0 (RC1)

; On Fri, Aug 3, 2018, at 5:45 PM, Li Jin wrote: > >> > +1 > >> > > >> > ran ./dev/release/verify-release-candidate.sh on MacOS > >> > * Java 8 > >> > * clang-902.0.39.2 > >> > > >> > On Fri, Aug 3, 2018 at 3:31 P

Re: [VOTE] Release Apache Arrow 0.10.0 (RC1)

Thanks, the edit button is now available. On Sat, Aug 4, 2018 at 4:35 PM Wes McKinney wrote: > I just made you an admin (all PMCs should be admin on this space). I > turned on "Add" permission for pages for all confluence-users. > > On Sat, Aug 4, 2018 at 4:17 PM, Phill

Re: [VOTE] Release Apache Arrow 0.10.0 (RC1)

I've updated the "Main source release and vote" section. I will update the section about uploading pip wheels etc tomorrow. On Sat, Aug 4, 2018 at 4:37 PM Phillip Cloud wrote: > Thanks, the edit button is now available. > > On Sat, Aug 4, 2018 at 4:35 PM Wes McKinney wrot

Re: [VOTE] Release Apache Arrow 0.10.0 (RC1)

Just updated the pip wheels section On Sat, Aug 4, 2018 at 4:52 PM Phillip Cloud wrote: > I've updated the "Main source release and vote" section. I will update the > section about uploading pip wheels etc tomorrow. > > On Sat, Aug 4, 2018 at 4:37 PM Phillip Cloud wrote:

[VOTE] Release Apache Arrow 0.10.0 (RC0)

Hello all, I'd like to propose the 1st release candidate (rc0) of Apache Arrow version 0.10.0. This is a major release consisting of 454 resolved JIRAs [1]. This release candidate is distinct from previous releases in that it includes binary artifacts such as python wheels, conda packages, and

Re: [VOTE] Release Apache Arrow 0.10.0 (RC0)

+1 (binding) ran dev/release/verify-release-candidate.sh on Arch Linux * clang 6 * jdk8 * ruby 2.5.1p57 * nodejs v10.5.0 -Phillip On Wed, Aug 1, 2018 at 1:35 PM Phillip Cloud wrote: > Hello all, > > I'd like to propose the 1st release candidate (rc0) of Apache Arrow version

Re: [RESULT][VOTE] Release Apache Arrow 0.10.0 (RC1)

issue is that I noticed Java doc is comment out from > dev/gen_apidocs/create_documents.sh > (probably due to Java8 lint-checks). I have uncommented it and it works. > (Opened ARROW-3003) > > On Mon, Aug 6, 2018 at 9:33 PM, Phillip Cloud wrote: > > > Excellent. Thanks for

Re: [RESULT][VOTE] Release Apache Arrow 0.10.0 (RC1)

https://pypi.org/project/pyarrow/#files > > On Mon, Aug 6, 2018 at 4:38 PM, Phillip Cloud wrote: > > > The release artifacts are uploaded to SVN. > > > > > > On Mon, Aug 6, 2018 at 4:01 PM Wes McKinney > wrote: > > > > We can comment out the JS do

Re: [RESULT][VOTE] Release Apache Arrow 0.10.0 (RC1)

additional logging output above. > >> > >> > >> npm ERR! Please include the following file with any support request: > >> > >> npm ERR! /apache-arrow/arrow/js/npm-debug.log > >> > >> > >> On Mon, Aug 6, 2018 at 7:41 PM, Wes McKinney &g

Re: [ANNOUNCE] Apache Arrow 0.10.0 released

n we remove > https://dist.apache.org/repos/dist/release/arrow/arrow-0.9.0/ ? > > If we can do, can I try doing it? > > > Thanks, > -- > kou > > In > "[ANNOUNCE] Apache Arrow 0.10.0 released" on Tue, 7 Aug 2018 11:59:44 > -0400, > Phillip Cloud wrote

Re: [VOTE] Accept donation of Gandiva to Apache Arrow

+1 On Thu, Aug 16, 2018 at 9:26 PM Andy Grove wrote: > +1 > > On Thu, Aug 16, 2018 at 9:56 AM Wes McKinney wrote: > > > Dear all, > > > > The developers of Gandiva, an LLVM-based vectorized expression > > evaluation engine for Arrow columnar memory, are proposing to donate > > the project to

[ANNOUNCE] Apache Arrow 0.10.0 released

The Apache Arrow community is pleased to announce the 0.10.0 release. This is a major release including 470 resolved issues ([1]) since the 0.9.0 release. The release is available now from our website and [2]: http://arrow.apache.org/install/ Changelog

Re: [ANNOUNCE] Apache Arrow 0.10.0 released

@Julian Sorry I misread your first comment. It looks like that's actually missing a link. The sha256/sha512 link is also wrong. On Tue, Aug 7, 2018 at 12:31 PM Phillip Cloud wrote: > @Julian > > First issue: the link should be to the sha256 checksum (we're not > computing the sh

Re: [ANNOUNCE] Apache Arrow 0.10.0 released

Congrats on the release everyone, there was a ton of work that went into it! On Tue, Aug 7, 2018 at 12:10 PM Wes McKinney wrote: > Congrats all! > > On Tue, Aug 7, 2018 at 11:59 AM, Phillip Cloud wrote: > > The Apache Arrow community is pleased to announce the 0.1

Re: [ANNOUNCE] Apache Arrow 0.10.0 released

ile the http://arrow.apache.org/release/0.10.0.html < > http://arrow.apache.org/release/0.10.0.html> has the source release but > no link to the checksums. > > Julian > > > > On Aug 7, 2018, at 9:16 AM, Phillip Cloud wrote: > > > > Congrats on the release everyon

[RESULT][VOTE] Release Apache Arrow 0.10.0 (RC1)

With 4 binding +1 votes (myself, Uwe, Wes, Kou), 2 non-binding +1 (Krisztián, Li), and no other votes, the vote passes. Thanks everyone! I will upload the Java packages as per the release management wiki. Would some folks please volunteer to get the Python packaging, documentation and website

Re: [VOTE] Combining Arrow C++ development process with Apache Parquet C++

+1 On Tue, Aug 21, 2018 at 6:26 PM Jacques Nadeau wrote: > +1 > > > > On Tue, Aug 21, 2018 at 3:21 PM Philipp Moritz wrote: > > > +1 for the monorepo plan and push access to Parquet C++ committers > > > > -- Philipp. > > > > On Tue, Aug 21, 2018 at 3:12 PM, Antoine Pitrou > > wrote: > > > > >

Re: [ANNOUNCE] New Arrow PMC member: Antoine Pitrou

Welcome! Great to have you on the team! On Tue, Aug 21, 2018 at 4:27 PM Wes McKinney wrote: > The Project Management Committee (PMC) for Apache Arrow has invited > Antoine Pitrou to become a PMC member and we are pleased to announce > that he has accepted. > > Congratulations and welcome,

Re: Arrow Sync

I won't be able to make the call today, I have a conflict. On Wed, Aug 22, 2018 at 11:04 AM Wes McKinney wrote: > No worries. It's 12pm Eastern today at > https://meet.google.com/vtm-teks-phx > > On Wed, Aug 22, 2018 at 10:56 AM, Siddharth Teotia > wrote: > > I have a clash this morning so

Re: Need a release manager for Arrow 0.10.0 [was Re: Packaging status update]

I can certainly help with the release vote. I'm out of town from the 20th to the 23rd, so it would be after that. Is there a particular target date we have in mind? On Wed, Jul 18, 2018 at 12:24 PM Wes McKinney wrote: > hi Li, > > Thanks for volunteering -- we need a PMC to sign the source and

Re: [DISCUSS] Contribution of Gandiva to Apache Arrow

While I'm not a level 83 LLVM wizard like Antoine :) I have a small amount of experience with it and would also be happy to review/merge patches. Having Gandiva in arrow will simplify packaging and building the library, which IME has always been annoying with large cross-platform and

Re: Working towards 0.10.0 release candidate

That sounds great to me. I'll make sure I'm able to build and sign artifacts this week and surface any issues I find along the way. Looking forward to a smooth release! On Tue, Jul 24, 2018 at 11:24 AM Wes McKinney wrote: > hi folks, > > Tuesday's update: it looks like we're going to be able to

RC Cutting

I'm think I'm still a bit confused about the order in which things need to happen to cut a release candidate. My understanding is that the ordering is: 1. create the source release 2. build packages from the source release (wheels, conda packages, etc) 3. commit source release + binary packages

Re: [VOTE] Close down Arrow Slack channel

+1 On Mon, Jul 9, 2018 at 2:47 PM Wes McKinney wrote: > hi Paul -- I would suggest operating an unofficial channel for > real-time discussions that don't necessarily concern the project > roadmap and governance of the Arrow project. > > The basic problems with Slack are, to summarize our

Re: [ANNOUNCE] New Arrow PMC member: Phillip Cloud

gmail.com> > wrote: > > > On behalf of the Apache Arrow PMC, I'm happy to announce that Phillip > > Cloud has been invited to join the PMC. > > > > Congrats Phillip, and thank you for your hard work growing the Arrow > > project and developer community! > > > > - Wes > > >

Re: Next Arrow sync call

Please add me as well. On Thu, Mar 29, 2018 at 3:36 PM Li Jin wrote: > Please add me in the gcal invite too. Thx > On Thu, Mar 29, 2018 at 3:12 PM Paul Taylor wrote: > > > I'd like to join the gcal invite as well. Thanks! > > > > > On Mar 29, 2018, at

Re: Arrow sync tomorrow: 12:00 US/Eastern, please review packaging thread

I didn't realize one needs chrome to use google meet, so it'll be a minute or so before I'm there. On Wed, Apr 4, 2018 at 11:54 AM Wes McKinney wrote: > hi Sidd -- your e-mail is on the Google calendar invite, not sure how > to send a link to it > > Here is the link for the

Re: [Py] writing 2- or 4-byte decimal columns to Parquet

ytes that fits, though, is that right? > > > https://github.com/apache/parquet-cpp/blob/c405bf36506ec584e8009a6d53349277e600467d/src/parquet/arrow/schema.cc#L635 > > On Thu, Apr 19, 2018 at 8:09 AM, Phillip Cloud <cpcl...@gmail.com> wrote: > > Hi Colin, > > > &

Re: Confronting Arrow packaging problems

I think we need to use a tool that can perform every single step of the deployment process, end-to-end. Right now, cmake isn't cutting it IMO because it lends itself quite heavily to copy pasting and oodles of bash scripts that are indecipherable by anyone except the original author. With that in

Re: Confronting Arrow packaging problems

Responses inline. This kind of information is extremely helpful and informative. On Mon, Mar 26, 2018 at 11:26 AM Antoine Pitrou wrote: > > Hi, > > As someone who started contributing recently, I'd like to raise a few > points. I hope this post doesn't come accross as

Re: Confronting Arrow packaging problems

o make things simpler, eventually the project will grow > until we are dealing with a different kind of development workflow > crisis. > I don't want to punt on automation, we need to do that regardless. What do you think about having specific owners of packaging areas, documented

Re: Adding support for Bazel?

I'm in agreement with Wes here. Packaging is much higher priority right now. When we have an MVP for that we can revisit the build system issue, which is definitely worth exploring. On Wed, Mar 28, 2018, 20:50 Wes McKinney wrote: > > As far as Bazel goes, I can put up a

Re: [VOTE] Apache Arrow JavaScript 0.3.1 - RC1

+1 (binding). Ran js-verify-release-candidate.sh with Node 9.8.0. On Thu, Mar 15, 2018 at 9:25 AM Brian Hulette wrote: > +1 (non-binding). Ran js-verify-release-candidate.sh with Node 8.9.1 on > Ubuntu 16.04. Thanks Wes! > > > On 03/15/2018 05:17 AM, Uwe L. Korn wrote: >

Re: [ANNOUNCE] New Arrow PMC member: Philipp Moritz

Sweet! Welcome to the PMC Philipp On Fri, Mar 16, 2018 at 4:58 PM Wes McKinney wrote: > On behalf of the Apache Arrow PMC, I'm happy to announce that Philipp > Moritz has been invited to join the PMC. > > Congrats Philipp, and thank you for your hard work growing the Arrow

Re: [RESULT] [VOTE] Release Apache Arrow 0.9.0 (RC2)

/blob/master/dev/release/RELEASE_MANAGEMENT.md > , > > let's definitely add them. I'd like the RM process to be reasonably > > fool-proof > > > > On Wed, Mar 21, 2018 at 10:07 AM, Phillip Cloud <cpcl...@gmail.com> > wrote: > >> Charles.Cloud > >>

Re: [RESULT] [VOTE] Release Apache Arrow 0.9.0 (RC2)

here is already a pyarrow 0.9.0 > package, but it does not seem to be there yet: > > https://arrow.apache.org/install/#python-wheels-on-pypi-unofficial > https://pypi.python.org/pypi/pyarrow > > BTW, why are Python packages unofficial? > > > Mitar > > On Thu, Mar 22, 2

Re: [VOTE] Release Apache Arrow 0.9.0 (RC2)

+1 (binding), tested on Arch Linux. I will verify the RC tomorrow morning (Eastern time) on Windows. On Sun, Mar 18, 2018 at 9:40 AM Uwe L. Korn wrote: > +1 (binding), tested on Ubuntu 16.04 > > > Am 16.03.2018 um 18:41 schrieb Wes McKinney : > > > > +1

Re: [RESULT] [VOTE] Release Apache Arrow 0.9.0 (RC2)

Wes > > On Mon, Mar 19, 2018 at 2:47 PM, Phillip Cloud <cpcl...@gmail.com> wrote: > > Just verified on windows, all systems are go for launch. > > > > On Mon, Mar 19, 2018 at 12:51 PM Li Jin <ice.xell...@gmail.com> wrote: > > > >> +1 > &

Re: [RESULT] [VOTE] Release Apache Arrow 0.9.0 (RC2)

raft a 0.9.0 blog post for the website > >> > later today or tomorrow morning. I suggest we announce the release on > >> > Wednesday morning after we have a chance to move along the binary > >> > packaging process. > >> > > >> > Thanks > &

Re: [RESULT] [VOTE] Release Apache Arrow 0.9.0 (RC2)

Charles.Cloud On Wed, Mar 21, 2018 at 8:53 AM Uwe L. Korn <uw...@xhochy.com> wrote: > At least I have not. Philip, what is your login on pypi.python.org so I > can add you as a maintainer there? > > On Wed, Mar 21, 2018, at 1:49 PM, Phillip Cloud wrote: > > Has anyone s

Re: [RESULT] [VOTE] Release Apache Arrow 0.9.0 (RC2)

time to do a release, properly. > > > Mitar > > On Thu, Mar 22, 2018 at 11:35 AM, Phillip Cloud <cpcl...@gmail.com> wrote: > > We are working on getting those wheels up as fast as we can. They should > be > > available very soon. In the meantime, you can install pyarrow 0

Re: Apache Arrow .NET implementation

+1. I agree that .NET + Arrow is a good match. Generally speaking, I'm not sure there are many systems programming languages whose communities wouldn't benefit from an Arrow implementation. I do think it's worth discussing what to do about the growing numbers of implementations, but that

Re: [VOTE] Accept donation of Arrow C# .NET implementation

+1 On Mon, Oct 15, 2018 at 9:33 PM Kouhei Sutou wrote: > +1 > > In > "[VOTE] Accept donation of Arrow C# .NET implementation" on Mon, 15 Oct > 2018 11:27:59 -0400, > Wes McKinney wrote: > > > hi folks, > > > > Individuals from Feyen Zylstra LLC have developed a C# implementation > > of

Re: Assign/update : NA bitmap vs sentinel

ed to deal with the NAs > (either way: including the NA group or excluding the NA group), if I > understand correctly. > > On Thu, Nov 8, 2018 at 3:18 PM Phillip Cloud wrote: > > > There is one database that I'm aware of that uses sentinels _and_ > supports > > com

Re: Assign/update : NA bitmap vs sentinel

There is one database that I'm aware of that uses sentinels _and_ supports complex types with missing values: Kx's KDB+. This has led to some seriously strange choices like the ASCII space character being used as the sentinel value for strings. See https://code.kx.com/wiki/Reference/Datatypes for

Re: [ANNOUNCE] New Arrow PMC member: Krisztián Szűcs

Huzzah! Welcome! On Fri, Nov 9, 2018, 23:39 Srinivas Reddy wrote: > Congratulations Krisztián.. > > > > -- > Srinivas Reddy > > http://mrsrinivas.com/ > > > (Sent via gmail web) > > > On Fri, 9 Nov 2018 at 03:48, Wes McKinney wrote: > > > The Project Management Committee (PMC) for Apache Arrow

Re: [VOTE] Accept donation of C GLib bindings to Parquet C++ libraries

+1, nice work. On Tue, Sep 25, 2018 at 10:53 AM Krisztián Szűcs wrote: > +1 > > On Sep 25 2018, at 1:36 pm, Wes McKinney wrote: > > > > hello, > > Kouhei Sutou is proposing to donate C GLib bindings to the Parquet C++ > > libraries (which can read Arrow tables back), designed to work > >

Re: [VOTE] Accept donation of Rust Parquet implementation

+1 from me. On Sat, Dec 1, 2018, 06:24 Uwe L. Korn +1, nice to see this joining the Apache community > > Uwe > > > Am 01.12.2018 um 10:16 schrieb Antoine Pitrou : > > > > > >> Le 01/12/2018 à 00:50, Wes McKinney a écrit : > >> > >> This vote is to determine if the Arrow PMC is in favor of

Re: [DISCUSS] Splitting out the Arrow format directory

synchronously. Is that what you're after here? > E.g. thrift does not offer async -> parquet-format-rs does not offer async > -> parquet does not offer async -> datafusion wraps all parquet "IO-bounded > and CPU-bounded operations" in spawn_blocking or something

Re: [DISCUSS] Splitting out the Arrow format directory

On Fri, Aug 13, 2021 at 11:43 AM Antoine Pitrou wrote: > > Le 13/08/2021 à 17:35, Phillip Cloud a écrit : > > > >> I.e. make the ability to read and write by humans be more important than > >> speed of validation. > > > > I think I differ on wheth

Re: [DISCUSS] Developing an "Arrow Compute IR [Intermediate Representation]" to decouple language front ends from Arrow-native compute engines

but I now think that using an enum for "built-ins" would be > superior (because of the code-generated enum interface) and not a > premature optimization. > > [1]: https://github.com/apache/arrow/pull/10934 > > On Fri, Aug 13, 2021 at 11:26 PM Phillip Cloud wrote: > &

Re: [DISCUSS] Developing an "Arrow Compute IR [Intermediate Representation]" to decouple language front ends from Arrow-native compute engines

able. > > > [1] > > https://github.com/apache/arrow/pull/10979/files#diff-e40fbc40cf7a131efd2cb098444931774cfad046b8665b38452258ffaa2e3423R34 > [2] > > https://github.com/apache/arrow/commit/611a4b951e24f4f967c3d382a2027dc035fc37f0 > [3] https://github.com/gre

Re: [DISCUSS] Developing an "Arrow Compute IR [Intermediate Representation]" to decouple language front ends from Arrow-native compute engines

peration named X stored in Y that maintains properties > 1,2 > > > and > > > disrupts property 3. Putting just a black box of bytes will > > > substantially > > >reduce the compatibility and extensibility of the ecosystem of tools > > >wor

Re: [DISCUSS] Developing an "Arrow Compute IR [Intermediate Representation]" to decouple language front ends from Arrow-native compute engines

of batches. > > Elsethread, someone mentioned the LoLePop concept and the Kohn/Leis/Neuman > paper [1]. The LoLePop concept sounds good for our purposes. > > Julian > > [1] https://db.in.tum.de/~kohn/papers/lolepops-sigmod21.pdf > > > > On Aug 12, 2021, at 5:1

Re: [DISCUSS] Splitting out the Arrow format directory

21 at 6:46 AM Jacob Quinn > > wrote: > > > > > > > > > > > I just thought of one other requirement: the format needs to support > > > > arbitrary byte sequences. > > > > > > > Can you clarify why this is needed? Is it that c

Re: [DISCUSS] Developing an "Arrow Compute IR [Intermediate Representation]" to decouple language front ends from Arrow-native compute engines

value here as well. > > So what is the scope of this initiative? If it is just #1 for example > then I don't see any need to put types in the IR (and I've commented > as such in the PR). From a relational perspective isn't a UDF just a > black box Table -> UDF -> Table? >

Re: [ANNOUNCE] New Arrow committer: Nic Crane

Congrats and welcome! On Thu, Sep 9, 2021, 16:16 Matthew Topol wrote: > Congrats! > > -Original Message- > From: Krisztián Szűcs > Sent: Thursday, September 9, 2021 3:47 PM > To: dev@arrow.apache.org > Subject: Re: [ANNOUNCE] New Arrow committer: Nic Crane > > Congrats Nic! > > On

Re: [DISCUSS] Splitting out the Arrow format directory

t; format definition files. > > > > My rationale is that the Rust implementation has a copy of the `format` > > directory [1] and potential drift worries me (a bit). Having a single > > source of truth for the format that is not part of the large mono repo > > would be a

Re: [DISCUSS] Splitting out the Arrow format directory

t; >> > > >> I support the idea of an independent repo that has the arrow > flatbuffers > > >> format definition files. > > >> > > >> My rationale is that the Rust implementation has a copy of the > `format` > > >> directory [1] and p

Re: [DISCUSS] Developing an "Arrow Compute IR [Intermediate Representation]" to decouple language front ends from Arrow-native compute engines

t; next week if anyone would like to chat in person about it, and we can > relay the content of any discussions back to the document/PR/e-mail > thread. > > I know that Phillip Cloud expressed interest in working on the PR and > helping work through many of the details, so

Re: [DISCUSS] Splitting out the Arrow format directory

think you're right that specs, regardless of wire format should remain together. > On Wed, Aug 11, 2021 at 11:34 AM Antoine Pitrou > wrote: > > > > > > Le 11/08/2021 à 23:06, Phillip Cloud a écrit : > > > On Wed, Aug 11, 2021 at 4:22 PM Antoine Pitrou

[DISCUSS] Splitting out the Arrow format directory

Hi all, I'd like to bring up an idea from a recent thread ([1]) about moving the `format/` directory out of the primary apache/arrow repository. I understand from that thread there are some concerns about using submodules, and I definitely sympathize with those concerns. In talking with David

Re: [DISCUSS] Splitting out the Arrow format directory

On Wed, Aug 11, 2021 at 3:51 PM Antoine Pitrou wrote: > > > Le 11/08/2021 à 21:39, Phillip Cloud a écrit : > > The benefit is that IR components don't interact much with `flatbuffers` > or > > `flatc` directly. > > > [...] > > > > One counter-pro

Re: [DISCUSS] Splitting out the Arrow format directory

as an option. On Wed, Aug 11, 2021 at 3:52 PM Phillip Cloud wrote: > On Wed, Aug 11, 2021 at 3:51 PM Antoine Pitrou wrote: > >> >> >> Le 11/08/2021 à 21:39, Phillip Cloud a écrit : >> > The benefit is that IR components don't interact much with >>

Re: [DISCUSS] Splitting out the Arrow format directory

or the specific language) their library code. End users of, say, ibis never need to think about having flatc around. > > On Wed, Aug 11, 2021, at 16:16, Phillip Cloud wrote: > > On Wed, Aug 11, 2021 at 4:05 PM Antoine Pitrou > wrote: > > > > > > > > Le 11/

Re: [DISCUSS] Developing an "Arrow Compute IR [Intermediate Representation]" to decouple language front ends from Arrow-native compute engines

, and it's up to the consumer to figure out how to turn that count into something that makes sense for itself. In your example that's a series of partial counts followed by a sum. > > > > > > On Wed, Aug 11, 2021 at 6:10 PM Phillip Cloud wrote: > > > Thanks Wes, > &

Re: [DISCUSS] Splitting out the Arrow format directory

On Wed, Aug 11, 2021 at 4:22 PM Antoine Pitrou wrote: > > Le 11/08/2021 à 22:16, Phillip Cloud a écrit : > > > > Yeah, that is a drawback here, though I don't see needing to run flatc > as a > > major downside given the upside > > of not having to write additi

Re: [DISCUSS] Splitting out the Arrow format directory

On Wed, Aug 11, 2021 at 3:58 PM Antoine Pitrou wrote: > > Le 11/08/2021 à 21:56, Phillip Cloud a écrit : > > I can see how that might be a bit circular. Let me start from the > > perspective of requirements. We want to be able to reuse the arrow's > types > > and sche

Re: [DISCUSS] Splitting out the Arrow format directory

On Wed, Aug 11, 2021 at 4:05 PM Antoine Pitrou wrote: > > Le 11/08/2021 à 22:02, Phillip Cloud a écrit : > > On Wed, Aug 11, 2021 at 3:58 PM Antoine Pitrou > wrote: > > > >> > >> Le 11/08/2021 à 21:56, Phillip Cloud a écrit : > >>> I can see

Temporal Arithmetic

Hi all, I wanted to draw some attention to ARROW-11090 [1] in an effort to start getting the implementation going for temporal arithmetic kernels. The supported operation matrix is decently large. I would like to propose that we adopt the behavior of PostgreSQL where things are at first glance

Re: [JIRA] New "Compute IR" component

Thanks Antoine. For future reference, can anyone add these labels? On Mon, Sep 27, 2021 at 7:39 AM Antoine Pitrou wrote: > > Hello, > > I've added a JIRA component called "Compute IR". Please use > accordingly for all compute IR-related issues. > > Regards > > Antoine. > > >

Re: [DISCUSS] Deprecate user@ in favor for github issues/discussions

I am +1 on steering users towards GitHub issues for support questions. I think there's a lot of value in someone being able to use a search engine to potentially find an answer to their problem. On Wed, Sep 29, 2021 at 2:16 PM Micah Kornfield wrote: > We discussed briefly on the sync this

Re: [DISCUSS] Deprecate user@ in favor for github issues/discussions

On Wed, Sep 29, 2021 at 3:08 PM Antoine Pitrou wrote: > > Le 29/09/2021 à 20:51, Micah Kornfield a écrit : > >> > >> Cons: > >> - Github is a not a mailing-list and does not integrate well in a normal > >> e-mail workflow. > > > > > > Would a mailing list mirror of the issues work for you (I

  1   2   3   >