Re: Arrow sync call tomorrow at 12:00 US/Eastern, 16:00 UTC

2019-05-15 Thread Neal Richardson
Attendees: * Bryan Cutler * Ben Kietsman * Praveen Kumar * Wes McKinney * John Muehlhausen * Neal Richardson Bryan to start on Map type in Java, interested in the Spark-to-Arrow connection. Ben volunteered to look into the C++ implementation. Wes raised the issue of the large number of open Java

Re: [DISCUSS] PR Backlog reduction

2019-05-31 Thread Neal Richardson
There are also some "dashboards" (mostly just saved filters, in list view) here: https://cwiki.apache.org/confluence/display/ARROW/Dashboards They're not great either, but at least they can give us some common views to pay attention to. Neal On Fri, May 31, 2019 at 5:56 AM Wes McKinney wrote:

Re: Arrow sync call tomorrow (May 29) at 12:00 US/Eastern, 16:00 UTC

2019-05-29 Thread Neal Richardson
Attendees: * Bryan Cutler * François Saint-Jacques * John Muehlhausen * Neal Richardson * Praveen Kumar * Wes McKinney John: Raised question of custom metadata in file footer (see https://lists.apache.org/thread.html/c3b3d1456b7062a435f6795c0308ccb7c8fe55c818cfed2cf55f76c5@%3Cdev.arrow.apache.org

Re: [DISCUSS] Timing of release and making a 1.0.0 release marking Arrow protocol stability

2019-06-14 Thread Neal Richardson
Thanks Kou for volunteering to be the release manager. I can't imagine there are any objections :) As this is my first release with the project, I'm eager to learn how things work and to help however I can. One thing that occurs to me is the management of the backlog for 0.14 and the

Arrow sync call tomorrow (June 12) at 12:00 US/Eastern, 16:00 UTC

2019-06-11 Thread Neal Richardson
Hi everyone, Reminder that the biweekly Arrow call is tomorrow at https://meet.google.com/vtm-teks-phx. All are welcome to join. Given the current discussions on the list, including around timing and content of upcoming releases, there's lots to chat about. Notes will be sent out to the mailing

Re: [DISCUSS] Making the Arrow 0.14 release, outstanding items

2019-06-24 Thread Neal Richardson
I've got one of the blockers, https://issues.apache.org/jira/browse/ARROW-5497, about the non-sphinx language doc building. I'll spend some time on it this week. It should get done before we're wrapped up with 0.14, but if I get held up, it doesn't need to block making RC0. In other news, in

Re: Arrow as a common open standard for machine learning data

2019-06-12 Thread Neal Richardson
Hi Joaquin, I recognize that this doesn't answer all of your questions, but we are in the process of adding a FAQ to the arrow.apache.org website that speaks to some of them: https://github.com/apache/arrow/blob/master/site/faq.md Neal On Wed, Jun 12, 2019 at 3:39 AM Joaquin Vanschoren <

Re: Arrow sync call tomorrow (June 12) at 12:00 US/Eastern, 16:00 UTC

2019-06-12 Thread Neal Richardson
Attendees: Neal Richardson François Saint-Jacques Micah Kornfield Ravindra Pindikura Ben Kietzman John Muehlhausen Micah raised several issues: * C++ status: 2 PRs: 1. make status pluggable, clean up error codes ( https://github.com/apache/arrow/pull/4484). Antoine has been reviewing

Re: Python development setup and LLVM 7 / Gandiva

2019-05-22 Thread Neal Richardson
Since you seem to be using conda on macOS, I'll point you to this recent addition to the pyarrow dev docs (which has not been deployed to arrow.apache.org/docs): https://github.com/nealrichardson/arrow/blob/master/docs/source/developers/python.rst#using-conda Not sure if that's what's causing

Re: [DISCUSS] PR Backlog reduction

2019-05-21 Thread Neal Richardson
Automatically close stale PRs? https://github.com/probot/stale On Tue, May 21, 2019 at 11:00 AM Wes McKinney wrote: > Any other thoughts about process to manage the backlog? > > On Thu, May 16, 2019 at 2:58 PM Wes McKinney wrote: > > > > hi Micah, > > > > This sounds like a reasonable

Arrow sync call tomorrow at 12:00 US/Eastern, 16:00 UTC

2019-05-14 Thread Neal Richardson
Hi all, Just a reminder that the biweekly Arrow call is tomorrow at https://meet.google.com/vtm-teks-phx. All are welcome to join. Notes will be sent out to the mailing list afterwards. Neal

Release verification process

2019-07-04 Thread Neal Richardson
Hi all, [Splitting off from the [VOTE] discussion thread] I wonder if there are things we can do to make for a smoother release next time. This was the first release I participated in with the Arrow project, and there were some aspects that were surprising to me. It seemed odd that "release

Re: [RESULT][VOTE] Release Apache Arrow 0.14.0 - RC0

2019-07-04 Thread Neal Richardson
I'll get the ball rolling on the R packages. Neal On Thu, Jul 4, 2019 at 12:46 AM Sutou Kouhei wrote: > Hi, > > I need your help! > Could Rust developers see "Failed:" section? > Could someone take over tasks in "Need helped:" section? > > Failed: > > * Updating Rust packages > >

Re: Arrow sync call tomorrow (June 26) at 12:00 US/Eastern, 16:00 UTC

2019-06-26 Thread Neal Richardson
Attendees: Ben Kietzman Wes McKinney John Muehlhausen Neal Richardson François Saint-Jacques Shyam Singh Discussion: 0.14 release: * Trouble in past releases with Java Gandiva ( https://issues.apache.org/jira/browse/ARROW-4301). Shyam says that they'll follow up on email * Travis-CI backups

Re: [DISCUSS] Making the Arrow 0.14 release, outstanding items

2019-06-27 Thread Neal Richardson
-ish to determine if a fix is in > >>> scope for the release > >>> * ARROW-5497: Publishing R / JS / Java docs -- not essential before > >>> release candidate can be cut > >>> > >>> Is there anything else? > >>> > >>

Re: Arrow biweekly sync call today at 12pm US/Eastern / 16:00 UTC

2019-07-10 Thread Neal Richardson
Attendees: Hatem Helal Uwe Korn Micah Kornfield Wes McKinney Prudhvi Porandla Neal Richardson Krisztián Szűcs Topics discussed: Issues with 0.14: * Python manylinux2010 wheels broken, runtime dependency on lz4: fixed in master, bad wheels removed * Python macOS wheels have runtime dependency

Re: [DISCUSS] Need for 0.14.1 release due to Python package problems, Parquet forward compatibility problems

2019-07-11 Thread Neal Richardson
I just moved https://issues.apache.org/jira/browse/ARROW-5850 from 1.0.0 to 0.14.1. On Thu, Jul 11, 2019 at 8:12 AM Wes McKinney wrote: > To limit uncertainty, I'm going to start preparing a 0.14.1 patch > release branch. I will update the list with the patches that are being > cherry-picked.

Arrow sync call tomorrow (June 26) at 12:00 US/Eastern, 16:00 UTC

2019-06-25 Thread Neal Richardson
Hi everyone, Reminder that the biweekly Arrow call is tomorrow at https://meet.google.com/vtm-teks-phx. All are welcome to join. I suspect a significant topic of discussion will be the upcoming release and any remaining blockers (of which

Re: Proposal to move website source to arrow-site, add automatic builds

2019-08-12 Thread Neal Richardson
ome point soon. > > On Thu, Aug 8, 2019 at 2:29 PM Neal Richardson > wrote: > > > > I need a committer to make a master branch on arrow-site so that I can > > PR to it. I thought it could be just an empty orphan branch but that > > proved not to work, so

Proposal to move website source to arrow-site, add automatic builds

2019-08-02 Thread Neal Richardson
Hi all, https://issues.apache.org/jira/browse/ARROW-5746 requested to move the source for https://arrow.apache.org out of `apache/arrow` due to the growing number of binary files (mostly images) there. https://issues.apache.org/jira/browse/ARROW-4473 requested improvements to the ability to make

Re: Proposal to move website source to arrow-site, add automatic builds

2019-08-02 Thread Neal Richardson
ng a token with "repo" scope out there. We > might need to think about this some more -- the general idea of making > it easier to deploy the website I'm totally on board with > > - Wes > > > On Fri, Aug 2, 2019 at 1:35 PM Neal Richardson > wrote: > >

Re: Proposal to move website source to arrow-site, add automatic builds

2019-08-03 Thread Neal Richardson
an environment variable there after > > > it's set, it could still technically be compromised. Personally I > > > wouldn't be comfortable having a token with "repo" scope out there. We > > > might need to think about this some more -- the general idea of making > &

Re: [DISCUSS] Apache Arrow manylinux1 support

2019-08-16 Thread Neal Richardson
For R's official support for various C++ versions, see https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Using-C_002b_002b11-code and below. Empirically, C++ > 11 is not really used: there are only 6 packages on CRAN that declare it as a requirement, and none of those are widely used.

Re: Arrow sync call tomorrow (August 21) at 12:00 US/Eastern, 16:00 UTC

2019-08-21 Thread Neal Richardson
Attendees: 刘吉 Micah Kornfield Wes McKinney Rok Mihevc Antoine Pitrou Prudhvi Porandla Neal Richardson Discussion: * alignment vote: Wes and Micah discussed implementation and testing forwards and backwards compatibility * 0.15: Alignment issues will be "blockers"; doesn't seem the

Arrow sync call tomorrow (August 21) at 12:00 US/Eastern, 16:00 UTC

2019-08-20 Thread Neal Richardson
Hi all, Reminder that the biweekly Arrow call is tomorrow at https://meet.google.com/vtm-teks-phx. All are welcome to join. Notes will be sent out to the mailing list afterwards. Neal

Re: [DISCUSS][Format][C++] Improvement of sparse tensor format and implementation

2019-08-27 Thread Neal Richardson
Forgive me if this is off topic; I haven't been following this closely and I haven't used scipy.sparse. But there are some very reasonable cases where you might want to fill sparse data with a value other than 0: * The sparseness is missing data, and 0 is not the same as NA * Better compression:

Broken nightly builds needing attention before 0.15

2019-09-03 Thread Neal Richardson
Hi all, As there is currently not great visibility into the nightly builds and tests that crossbow runs, it's not surprising that many of them are failing. See https://github.com/ursa-labs/crossbow/branches/all?page=2=nightly-624=%E2%9C%93 for the builds and

Arrow sync call tomorrow (September 4) at 12:00 US/Eastern, 16:00 UTC

2019-09-03 Thread Neal Richardson
Hi all, Reminder that the biweekly Arrow call is tomorrow (or today, depending on your time zone ;) at https://meet.google.com/vtm-teks-phx. All are welcome to join. Notes will be sent out to the mailing list afterwards. Neal

Re: Arrow sync call tomorrow (September 4) at 12:00 US/Eastern, 16:00 UTC

2019-09-04 Thread Neal Richardson
We're meeting at meet.google.com/nwj-xado-dtu instead today. On Tue, Sep 3, 2019 at 4:24 PM Neal Richardson wrote: > > Hi all, > Reminder that the biweekly Arrow call is tomorrow (or today, depending > on your time zone ;) at https://meet.google.com/vtm-teks-phx. All are > welcome

Re: [RESULT] [VOTE] Alter Arrow binary protocol to address 8-byte Flatbuffer alignment requirements (2nd vote)

2019-09-13 Thread Neal Richardson
Nice work! On Fri, Sep 13, 2019 at 10:43 AM Wes McKinney wrote: > > Integration tests are passing: > > https://travis-ci.org/apache/arrow/jobs/584361357 > > I'm merging the feature branch into master. Thanks all > > On Thu, Sep 12, 2019 at 6:15 PM Wes McKinney wrote: > > > > Thanks all! > > > >

Re: [DISCUSS][C++] Rethinking our current C++ shared library (.so / .dll) approach

2019-09-12 Thread Neal Richardson
Just to be clear: this is not part of the 0.15 release discussion, right? On Thu, Sep 12, 2019 at 1:29 PM Wes McKinney wrote: > > On Thu, Sep 12, 2019 at 3:14 PM Antoine Pitrou wrote: > > > > > > Le 12/09/2019 à 20:14, Wes McKinney a écrit : > > > hi folks, > > > > > > I wanted to share some

Re: Nightly build report for crossbow job nightly-2019-09-10-1

2019-09-10 Thread Neal Richardson
I think they're ticketed. According to

Re: Can the R interface to write_parquet accept strings?

2019-09-09 Thread Neal Richardson
Hi Daniel, This works on my machine: > library(arrow) > write_parquet(data.frame(y = c("a", "b", "c"), stringsAsFactors=FALSE), file= > "string.parquet") > read_parquet("string.parquet") y 1 a 2 b 3 c > (The function masking warnings are all from library(tidyverse) and aren't relevant here.)

Re: Timeline for 0.15.0 release

2019-09-17 Thread Neal Richardson
Sounds good to me. Do we have a release manager yet? Any volunteers? Neal On Tue, Sep 17, 2019 at 4:06 PM Wes McKinney wrote: > hi all, > > It looks like we're drawing close to be able to make the 0.15.0 > release. I would suggest "pencils down" at the end of this week and > see if a release

Arrow sync call September 19 at 12:00 US/Eastern, 16:00 UTC

2019-09-17 Thread Neal Richardson
Hi all, Belated reminder that the biweekly Arrow call is coming up in less than 12 hours at https://meet.google.com/vtm-teks-phx. All are welcome to join. Notes will be sent out to the mailing list afterwards. Neal

Re: [ANNOUNCE] New committers: Ben Kietzman, Kenta Murata, and Neal Richardson

2019-09-06 Thread Neal Richardson
; > > >> > On Thu, Sep 5, 2019 at 7:06 PM Ji Liu > >> > wrote: > >> > > >> > > Congratulations! > >> > > > >> > > Thanks, > >> > > Ji Liu > >> > > > >> > > > >> > >

Draft blog post for 0.15 release

2019-09-18 Thread Neal Richardson
Hi all, In preparation for next week, I've started a release announcement blog post here: https://github.com/apache/arrow-site/pull/27 Please fill in the parts you know best. Committers can just push edits to my branch; also feel free to reply to this thread with content, or email me directly,

Re: Arrow sync call tomorrow (August 7) at 12:00 US/Eastern, 16:00 UTC

2019-08-07 Thread Neal Richardson
Attendees: Hatem Helal Micah Kornfield Wes McKinney Rok Mihevc Ravindra Pindikura Antoine Pitrou Neal Richardson François Saint-Jacques Discussion: * Alignment vote: Antoine raised proposal to pad end of stream too. Related need to make official an EOS marker. Decision to suspend vote for now

Arrow sync call tomorrow (August 7) at 12:00 US/Eastern, 16:00 UTC

2019-08-06 Thread Neal Richardson
Hi all, Reminder that the biweekly Arrow call is tomorrow at https://meet.google.com/vtm-teks-phx. All are welcome to join. Notes will be sent out to the mailing list afterwards. Neal

Re: Proposal to move website source to arrow-site, add automatic builds

2019-08-08 Thread Neal Richardson
d for > > $FOO.apache.org tend to be ones where the documentation _is_ the > > website. I think we would need to commission a significant web design > > overhaul to be able to make our documentation page adequate as the > > landing point for visitors to https://arrow.a

Re: Proposal to move website source to arrow-site, add automatic builds

2019-08-08 Thread Neal Richardson
u spell out what > are the next steps? > > On Thu, Aug 8, 2019 at 2:01 PM Neal Richardson > wrote: > > > > Have we reached "lazy consensus" here? No further comments in the last > > three days. > > > > Thanks, > > Neal > > > > On

Re: [ANNOUNCE] New Arrow PMC member: Micah Kornfield

2019-08-09 Thread Neal Richardson
Congratulations, Micah! Neal On Fri, Aug 9, 2019 at 8:52 AM Ji Liu wrote: > > Congrats Micah! Well deserved. > > Thanks, > Ji Liu > > > -- > From:paddy horan > Send Time:2019年8月9日(星期五) 23:31 > To:dev@arrow.apache.org >

Re: Caution about CI builds on personal forks

2019-07-17 Thread Neal Richardson
Won't moving CI away from Travis to our own infrastructure mean that we won't get any CI on our personal forks? On Wed, Jul 17, 2019 at 8:23 AM Wes McKinney wrote: > On Wed, Jul 17, 2019 at 10:22 AM Wes McKinney wrote: > > > > hi folks -- I noticed this last night on > >

Re: [VOTE] Release Apache Arrow 0.14.1 - RC0

2019-07-23 Thread Neal Richardson
I'll handle R. On Mon, Jul 22, 2019 at 5:42 PM Krisztián Szűcs wrote: > On Tue, Jul 23, 2019 at 12:31 AM Krisztián Szűcs < > szucs.kriszt...@gmail.com> > wrote: > > > The remaining tasks are: > > - Updating website (after https://github.com/apache/arrow/pull/4922 is > > merged) > > > I'm

Arrow sync call July 24 at 12:00 US/Eastern, 16:00 UTC

2019-07-23 Thread Neal Richardson
Hi everyone, Reminder that the biweekly Arrow call is tomorrow (well, already today for some of you) at https://meet.google.com/vtm-teks-phx. All are welcome to join. Notes will be sent out to the mailing list afterwards. Neal

Re: Arrow sync call July 24 at 12:00 US/Eastern, 16:00 UTC

2019-07-25 Thread Neal Richardson
Notes from the meeting: Attendees: Micah Kornfield Antoine Pitrou Neal Richardson We followed up on some mailing list threads. * Proposal on versioning post 1.0.0: need PMC to call a vote * Flatbuffers alignment: Micah to summarize the discussion so far to help us make a decision * Encodings

Re: [NIGHTLY] Arrow Build Report for Job nightly-2019-09-21-0

2019-09-21 Thread Neal Richardson
I made https://issues.apache.org/jira/browse/ARROW-6651 for the conda R failure. On Sat, Sep 21, 2019 at 7:02 AM Crossbow wrote: > > > Arrow Build Report for Job nightly-2019-09-21-0 > > All tasks: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-09-21-0 > > Failed Tasks:

Re: Arrow sync call September 19 at 12:00 US/Eastern, 16:00 UTC

2019-09-19 Thread Neal Richardson
A little late but here are the notes from the call. Attendees: Ben Kietzman Micah Kornfield Rok Mihevc Prudhvi Porandla Antoine Pitrou Neal Richardson François Saint-Jacques Discussion: * Arrow compatibility branding and minimal packaging discussion threads: talked about problems and possible

Re: Build issues on macOS [newbie]

2019-09-19 Thread Neal Richardson
Others have different experience, but I ultimately had better luck using Homebrew for system dependencies rather than Conda. You might also want to start with more cmake flags OFF, just to keep things simpler while you work out the basic environment. Though your current error is happening before

Re: Collecting Arrow critique and our roadmap on that

2019-09-19 Thread Neal Richardson
Uwe, I think this is an excellent idea. I've started https://docs.google.com/document/d/1cgN7mYzH30URDTaioHsCP2d80wKKHDNs9f5s7vdb2mA/edit?usp=sharing to collect some ideas and notes. Once we have gathered our thoughts there, we can put them in the appropriate places. I think that some of the

Arrow sync call October 2 at 12:00 US/Eastern, 16:00 UTC

2019-10-02 Thread Neal Richardson
Hi all, our biweekly call is about to begin at https://meet.google.com/vtm-teks-phx. All are welcome to join. Notes will be sent out to the mailing list afterwards. Neal

[DISCUSS] Understanding Arrow's CI problems and needs

2019-10-01 Thread Neal Richardson
Hi all, Over the last few months, I've seen a lot of frustration and discussion around the shortcomings of our current CI. I'm also seeing debate over a few possible solutions; unfortunately, the debates tend not to resolve in a clear, decisive way, and we end up having the same debates

Re: [CI] Disable coverage on Travis-CI?

2019-11-04 Thread Neal Richardson
A nightly coverage build would also have the benefit of actually being useful. Because we rarely run all of the coverage jobs on a single pull request, the report of the change in coverage from a diff is worthless. Neal On Mon, Nov 4, 2019 at 9:16 AM Wes McKinney wrote: > > I'm okay with

Re: [VOTE] Release Apache Arrow 0.15.1 - RC0

2019-11-04 Thread Neal Richardson
R package 0.15.1 has been accepted on CRAN. I'll keep monitoring CRAN for the macOS and Windows binary builds, but it shouldn't require anything additional from us at this point. Neal On Sun, Nov 3, 2019 at 6:51 AM Neal Richardson wrote: > > Yes, I’m on it. > > Neal > > >

Re: [VOTE] Release Apache Arrow 0.15.1 - RC0

2019-11-03 Thread Neal Richardson
Yes, I’m on it. Neal > On Nov 3, 2019, at 4:38 AM, Krisztián Szűcs wrote: > > Thanks Wes! > > The cpp, python and R conda packages are updated, so the only remaining > task is to update the R packages. @Neal could you please help with that? > > - [x] marking the released version as

Re: [NIGHTLY] Arrow Build Report for Job nightly-2019-11-10-0

2019-11-10 Thread Neal Richardson
I made https://issues.apache.org/jira/browse/ARROW-7105 for the homebrew-cpp failure. Neal On Sun, Nov 10, 2019 at 5:01 AM Crossbow wrote: > > > Arrow Build Report for Job nightly-2019-11-10-0 > > All tasks: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-11-10-0 > >

Re: [NIGHTLY] Arrow Build Report for Job nightly-2019-11-08-0

2019-11-08 Thread Neal Richardson
The homebrew-cpp failure is a little hard (for me) to determine which is the real error, but https://travis-ci.org/ursa-labs/crossbow/builds/609138711#L6672 seems to implicate aws-sdk-cpp, which was upgraded on homebrew-core a couple of days ago:

Re: Arrow sync call October 30 at 12:00 US/Eastern, 16:00 UTC

2019-10-30 Thread Neal Richardson
Attendees: * Uwe Korn * Micah Kornfield * Praveen Kumar * Wes McKinney * Rok Mihevc * Neal Richardson Discussion: * docker-compose/github-actions (https://github.com/apache/arrow/pull/5589). Needs review, needs to be merged and have followup issues made. Currently too many jobs being run

Re: [NIGHTLY] Arrow Build Report for Job nightly-2019-10-30-0

2019-10-30 Thread Neal Richardson
https://issues.apache.org/jira/browse/ARROW-7034 (pending +1/merge) will rid us of these meddlesome failures. Neal On Wed, Oct 30, 2019 at 11:25 AM Wes McKinney wrote: > > The failed tasks here are a nuisance. If they can't be fixed, should > they be removed from the nightlies? > > On Wed, Oct

Re: Building Arrow 0.15.1 using dependencies in local source folder

2019-11-14 Thread Neal Richardson
I am not an expert on this, but it seems you can specify `*_ROOT` arguments to cmake, like https://github.com/apache/arrow/blob/master/ci/PKGBUILD#L90-L91 Maybe that does what you need? Neal On Thu, Nov 14, 2019 at 12:45 PM Tahsin Hassan wrote: > Hi all, > > I am trying to build out arrow

Re: [NIGHTLY] Arrow Build Report for Job nightly-2019-11-17-0

2019-11-17 Thread Neal Richardson
I just merged ARROW-7181 (wheel failure, thanks Kou!) so that should be passing tomorrow. All other failures were reported previously. ubuntu-xenial is https://issues.apache.org/jira/browse/ARROW-6727 again. Neal On Sun, Nov 17, 2019 at 5:01 AM Crossbow wrote: > > Arrow Build Report for Job

Re: [NIGHTLY] Arrow Build Report for Job nightly-2019-11-15-0

2019-11-15 Thread Neal Richardson
I ticketed a few of the nightly failures: * https://issues.apache.org/jira/browse/ARROW-7181 (wheels) * https://issues.apache.org/jira/browse/ARROW-7182 (fuzzit, looks like an easy fix) * https://issues.apache.org/jira/browse/ARROW-7183 (re-skip r-sanitizer, I'll handle) I didn't have time this

Re: [NIGHTLY] Arrow Build Report for Job nightly-2019-11-15-0

2019-11-15 Thread Neal Richardson
://issues.apache.org/jira/browse/ARROW-7184 Neal On Fri, Nov 15, 2019 at 10:12 AM Krisztián Szűcs wrote: > Thanks Neal! I'll try to allocate time for the wheels (including > python 3.8) next week. > > On Fri, Nov 15, 2019 at 7:05 PM Neal Richardson > wrote: > > > > I ticketed a

Re: [NIGHTLY] Arrow Build Report for Job nightly-2019-11-16-0

2019-11-16 Thread Neal Richardson
Yesterday I dealt with ARROW-7105 (homebrew-cpp failure), ARROW-7183 (re-skip r-sanitizer tests), and ARROW-7187 (doxygen failure on master, which hadn't yet triggered a nightly failure). New failures: * https://issues.apache.org/jira/browse/ARROW-7188 (cpp docs missing parameter, easy fix) *

Re: [VOTE] Adopt Arrow in-process C Data Interface specification

2019-12-03 Thread Neal Richardson
+1 (non-binding) On Tue, Dec 3, 2019 at 10:56 AM Wes McKinney wrote: > +1 (binding) > > On Tue, Dec 3, 2019 at 12:54 PM Wes McKinney wrote: > > > > hello, > > > > We have been discussing the creation of a minimalist C-based data > > interface for applications to exchange Arrow columnar data

Re: [ANNOUNCE] New Arrow committer: Joris van den Bossche

2019-12-09 Thread Neal Richardson
Congratulations! > On Dec 9, 2019, at 4:38 AM, Fan Liya wrote: > > Congratulations, Joris! > > Best, > Liya Fan > >> On Mon, Dec 9, 2019 at 7:55 PM Wes McKinney wrote: >> >> On behalf of the Arrow PMC, I'm happy to announce that Joris has >> accepted an invitation to become a committer on

Arrow sync call December 11 at 12:00 US/Eastern, 17:00 UTC

2019-12-10 Thread Neal Richardson
Hi all, reminder for our biweekly call at https://meet.google.com/vtm-teks-phx. All are welcome to join. Notes will be sent out to the mailing list afterwards. Neal

Re: [NIGHTLY] Arrow Build Report for Job nightly-2019-12-17-0

2019-12-17 Thread Neal Richardson
I'm working on the macos-r-autobrew failure, which is related to last week's R 3.6.2 release: https://issues.apache.org/jira/browse/ARROW-7416 On Tue, Dec 17, 2019 at 6:03 AM Crossbow wrote: > > Arrow Build Report for Job nightly-2019-12-17-0 > > All tasks: >

Re: Planned Support for ORC Dataset?

2019-12-11 Thread Neal Richardson
Hi William, ORC is part of the C++ Datasets grand vision: see https://docs.google.com/document/d/1bVhzifD38qDypnSjtf8exvpP3sSB5x_Kw9m-n66FB2c/edit#heading=h.22aikbvt54fv. That said, I don't think anyone in the Arrow community is currently prioritizing work on ORC, and we'd welcome contributions in

Re: [NIGHTLY] Arrow Build Report for Job nightly-2019-12-03-0

2019-12-03 Thread Neal Richardson
Is there a jira for this yet? On Tue, Dec 3, 2019 at 12:31 PM Wes McKinney wrote: > The manylinux builds are failing because of a missing Docker image > > $ docker-compose pull $BUILD_IMAGE > 228Pulling centos-python-manylinux2010 ... > 229ERROR: for centos-python-manylinux2010 manifest for >

Re: [NIGHTLY] Arrow Build Report for Job nightly-2019-10-18-0

2019-10-18 Thread Neal Richardson
docker-r-sanitizer failure is a failure to install a dependency: https://circleci.com/gh/ursa-labs/crossbow/3945 I'll watch it and make a ticket if the failure persists. Neal On Fri, Oct 18, 2019 at 5:01 AM Crossbow wrote: > > > Arrow Build Report for Job nightly-2019-10-18-0 > > All tasks: >

Re: Possible Arrow 0.15.1 release

2019-10-21 Thread Neal Richardson
dded 0.15.1 back to a couple of fixed issues where the merge > > >>>> script > > >>>>>>> removed it :-/ > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>&

Arrow sync call October 16 at 12:00 US/Eastern, 16:00 UTC

2019-10-16 Thread Neal Richardson
Hi all, our biweekly call is coming up in a couple of hours at https://meet.google.com/vtm-teks-phx. All are welcome to join. Notes will be sent out to the mailing list afterwards. Neal

Re: [NIGHTLY] Arrow Build Report for Job nightly-2019-10-29-0

2019-10-29 Thread Neal Richardson
I'll fix the docker-r-conda here: https://issues.apache.org/jira/browse/ARROW-7024 On Tue, Oct 29, 2019 at 5:17 AM Crossbow wrote: > > > Arrow Build Report for Job nightly-2019-10-29-0 > > All tasks: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-29-0 > > Failed

Re: [NIGHTLY] Arrow Build Report for Job nightly-2019-10-22-0

2019-10-22 Thread Neal Richardson
I ticketed the R sanitizer failure here: https://issues.apache.org/jira/browse/ARROW-6957 Will try to take a look sometime this week, but it's safe to ignore in the meantime--it's a setup failure, not an actual sanitizer error. Neal On Tue, Oct 22, 2019 at 7:12 AM Krisztián Szűcs wrote: > >

Re: Arrow sync call October 16 at 12:00 US/Eastern, 16:00 UTC

2019-10-16 Thread Neal Richardson
adapters: a few open discussions * Use of std::regex in dataset partition detection: not well supported on older GCC. Ben to replace. * Issue with failing macOS/Homebrew builds: will make a Jira On Wed, Oct 16, 2019 at 6:24 AM Neal Richardson wrote: > > Hi all, our biweekly call is com

Arrow sync call October 30 at 12:00 US/Eastern, 16:00 UTC

2019-10-29 Thread Neal Richardson
Hi all, reminder that our biweekly call is 12 hours from now at https://meet.google.com/vtm-teks-phx. All are welcome to join. Notes will be sent out to the mailing list afterwards. Neal

Re: [DISCUSS] Changing C++ build system default options to produce more barebones builds

2019-09-20 Thread Neal Richardson
I wonder if having a core "format" C++ library, which the io, compute, etc. library/libraries would depend on, is a natural step. Particularly since we're coming up on 1.0 and the format is being declared stable. Neal On Fri, Sep 20, 2019 at 8:28 AM Wes McKinney wrote: > > We would have to be

Re: Timeline for 0.15.0 release

2019-09-25 Thread Neal Richardson
;> > > > > > > >> > > Sure, I'll be available next week. We can also grant access > > to > > > > > > >> > > https://github.com/ursa-labs/crossbow because configuring > > all > > > > > > >> > > the CI backends can be time consuming. > &g

Re: Timeline for 0.15.0 release

2019-09-25 Thread Neal Richardson
p 25, 2019 at 5:28 PM Micah Kornfield >> wrote: >>> >>> OK, I'll start the process today. I'll send up e-mail updates as I make >>> progress. >>> >>> On Wed, Sep 25, 2019 at 8:22 AM Wes McKinney wrote: >>>> >>>> Yes, a

Re: [NIGHTLY] Arrow Build Report for Job nightly-2019-10-07-0

2019-10-07 Thread Neal Richardson
homebrew-cpp failure is a timeout in `brew update`, which is part of the macos builder init stage: https://travis-ci.org/ursa-labs/crossbow/builds/594462704#L115 I don't think we can add a travis_wait to that, so we may just have to accept the occasional flaky failure. Neal On Mon, Oct 7, 2019

Re: [VOTE] Release Apache Arrow 0.15.0 - RC2

2019-10-10 Thread Neal Richardson
m the > > dist system > > > > Can we change our process to only remove old dist artifacts when we > > are about to upload a new RC? Otherwise it's harder to make > > improvements to the release verification scripts without any old RC to > > test a

Re: [NIGHTLY] Arrow Build Report for Job nightly-2019-10-09-0

2019-10-09 Thread Neal Richardson
FWIW there appears to have been a recent update to grpc on Homebrew involving protobuf: https://github.com/Homebrew/homebrew-core/commits/master/Formula/grpc.rb Last time we had a Homebrew grpc issue, I made this at Kou's suggestion: https://github.com/Homebrew/homebrew-core/pull/44198 I think

Looking ahead to 1.0

2019-10-09 Thread Neal Richardson
Congratulations everyone on 0.15! I know a lot of hard work went into it, not only in the software itself but also in the build and release process. Once you've caught your breath from the release, we should start thinking about what's in scope for our next release, the big 1.0. To get us started

Re: [VOTE] Release Apache Arrow 0.15.0 - RC2

2019-10-05 Thread Neal Richardson
R release steps per https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide#ReleaseManagementGuide-UpdatingRpackages are underway. Neal On Sat, Oct 5, 2019 at 8:40 AM Krisztián Szűcs wrote: > > - website updated with the release notes > - conda-forge updates are merged > >

Re: Should Arrow adopt C++14 / 17?

2019-10-04 Thread Neal Richardson
We do have to care about more than just conda and Python. For R, for example, C++14 support is limited: https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Using-C_002b_002b14-code That's not to say that we can't try (if C++ devs wanted to, and I can't speak for them), but it might be

Re: Possible Arrow 0.15.1 release

2019-10-11 Thread Neal Richardson
If 6844 isn't Python-only (I can't tell from the description) and we're updating C++ libraries, I'd like to include https://issues.apache.org/jira/browse/ARROW-3808 for R (not a bug fix but would be helpful to get out). But if there's no C++ change involved, then I probably wouldn't bother to

Re: [NIGHTLY] Arrow Build Report for Job nightly-2019-10-12-0

2019-10-12 Thread Neal Richardson
I just merged https://github.com/apache/arrow/pull/5617 so hopefully that will address the docker failures on circleCI. Neal On Sat, Oct 12, 2019 at 5:01 AM Crossbow wrote: > > > Arrow Build Report for Job nightly-2019-10-12-0 > > All tasks: >

Re: [NIGHTLY] Arrow Build Report for Job nightly-2019-12-18-0

2019-12-18 Thread Neal Richardson
conda-osx wheel builds have been failing for a long time. I found https://issues.apache.org/jira/browse/ARROW-7189 from over a month ago, but these builds here failed earlier than the error described on that ticket. Could someone take a look? Neal On Wed, Dec 18, 2019 at 5:47 AM Crossbow

Re: [NIGHTLY] Arrow Build Report for Job nightly-2020-02-22-0

2020-02-23 Thread Neal Richardson
I made https://issues.apache.org/jira/browse/ARROW-7922 for the macOS wheels and have a PR up. https://issues.apache.org/jira/browse/ARROW-7923 for the macos-r-autobrew will be more involved to fix. On Sat, Feb 22, 2020 at 5:31 AM Crossbow wrote: > > Arrow Build Report for Job

Re: [NIGHTLY] Arrow Build Report for Job nightly-2020-02-29-0

2020-02-29 Thread Neal Richardson
I'm looking into the R failures (https://github.com/apache/arrow/pull/6509). Since all of those docker-compose jobs are failing on Crossbow on Azure, but the one that we run on push/pull_request on GHA is passing ( https://github.com/apache/arrow/actions/runs/46824058), my guess is something

Arrow sync call March 4 at 12:00 US/Eastern, 17:00 UTC

2020-03-03 Thread Neal Richardson
Hi all, Reminder that our biweekly call is coming up tomorrow/later today at https://meet.google.com/vtm-teks-phx. All are welcome to join. Notes will be sent out to the mailing list afterward. Neal

Re: Integration testing

2020-03-05 Thread Neal Richardson
e out Dense/Sparse union testing into their own > issues. > > Thanks, > Micah > > On Fri, Feb 21, 2020 at 2:12 PM Neal Richardson < > neal.p.richard...@gmail.com> > wrote: > > > Thanks, I've marked that as not required for 1.0 (and added a legend to >

Re: [ANNOUNCE] New Arrow PMC member: Neal Richardson

2020-03-05 Thread Neal Richardson
bject:Re: [ANNOUNCE] New Arrow PMC member: Neal Richardson > > Congratulations! > > On Thu, Mar 5, 2020 at 5:57 AM Fan Liya wrote: > > > Congratulations, Neal Richardson! > > > > Best, > > Liya Fan > > > > On Thu, Mar 5, 2020 at 12:51 AM Wes McK

Re: [VOTE] Release Apache Arrow 0.16.0 - RC1

2020-01-29 Thread Neal Richardson
The place where the segfault is triggered in the R nightlies is a couple of tests after the one I added in that patch. If that patch is causing the segfaults, we can skip the new test ( https://github.com/apache/arrow/blob/master/r/tests/testthat/test-parquet.R#L125) and investigate later. The

Re: PR Dashboard for Java?

2020-01-27 Thread Neal Richardson
If you have confluence privileges, duplicate a page like https://cwiki.apache.org/confluence/display/ARROW/Ruby+JIRA+Dashboard and then edit the Jira query (something like status in open/in progress/reopened, labels = pull-request-available, component = java, project = ARROW) if you want to make

Re: [DISCUSS] Improving GitHub Actions configurations + tooling

2020-02-04 Thread Neal Richardson
What if we wrote our own action(s) to wrap up some of the boilerplate? It doesn't seem that there are any off-the-shelf actions we could use to drive docker-compose: https://github.com/marketplace?utf8=%E2%9C%93=actions=docker-compose but I don't think it would be that difficult to wrap

Re: [VOTE] Release Apache Arrow 0.16.0 - RC2

2020-02-05 Thread Neal Richardson
I have a fix on my GHA test branch, I think: https://github.com/apache/arrow/pull/6349/files#diff-3ebb5f8536634a43ccb884fcd453b1cd On Wed, Feb 5, 2020 at 9:14 AM Wes McKinney wrote: > The Windows wheel RC script is broken > > wget --no-check-certificate -O

Re: Arrow sync call February 5 at 12:00 US/Eastern, 17:00 UTC

2020-02-05 Thread Neal Richardson
Attendees: Todd Hendricks Ben Kietzman Micah Kornfield Wes McKinney Prudhvi Porandla Neal Richardson François Saint-Jacques Krisztián Szucs Discussion: * Release verification: ways to put some/more/all of the logic in CI so that we catch problems sooner and it's less expensive to cut a release

Arrow sync call February 5 at 12:00 US/Eastern, 17:00 UTC

2020-02-04 Thread Neal Richardson
Hi all, Reminder that our biweekly call is tomorrow at https://meet.google.com/vtm-teks-phx. All are welcome to join. Notes will be sent out to the mailing list afterwards (like, right after, I promise). Neal

  1   2   3   4   5   6   7   8   >