[jira] [Created] (ARROW-6814) [C++] Resolve compiler warnings occurred on release build
Kenta Murata created ARROW-6814: --- Summary: [C++] Resolve compiler warnings occurred on release build Key: ARROW-6814 URL: https://issues.apache.org/jira/browse/ARROW-6814 Project: Apache Arrow Issue Type: Task Components: C++, C++ - Gandiva Reporter: Kenta Murata Assignee: Kenta Murata I encountered some compiler warnings on release build when I used gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1). [https://gist.github.com/mrkn/f7739edb301988a24e9d6066410b0625] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ARROW-6813) [RUBY] Arrow::Table.load with headers=true leads to exception in Arrow 0.15
Rick created ARROW-6813: --- Summary: [RUBY] Arrow::Table.load with headers=true leads to exception in Arrow 0.15 Key: ARROW-6813 URL: https://issues.apache.org/jira/browse/ARROW-6813 Project: Apache Arrow Issue Type: Bug Components: Ruby Affects Versions: 0.15.0 Environment: Ubuntu 18.04, Debian Stretch Reporter: Rick ``` Error: undefined method `n_header_rows=' for # ``` It appears that 0.15 has changed the name for this option to `n_skip_rows` ``` (byebug) options #(byebug) (options.methods - Object.new.methods).sort [:add_column_name, :add_column_type, :add_column_type_raw, :add_false_value, :add_null_value, :add_schema, :add_true_value, :allow_newlines_in_values=, :allow_newlines_in_values?, :allow_null_strings=, :allow_null_strings?, :bind_property, :block_size, :block_size=, :check_utf8=, :check_utf8?, :column_names, :column_names=, :column_types, :delimiter, :delimiter=, :destroyed?, :double_quoted=, :double_quoted?, :escape_character, :escape_character=, :escaped=, :escaped?, :false_values, :false_values=, :floating?, :freeze_notify, :generate_column_names=, :generate_column_names?, :get_property, :gtype, :ignore_empty_lines=, :ignore_empty_lines?, :n_skip_rows, :n_skip_rows=, :notify, :null_values, :null_values=, :parent_instance, :quote_character, :quote_character=, :quoted=, :quoted?, :ref_count, :set_allow_newlines_in_values, :set_allow_null_strings, :set_block_size, :set_check_utf8, :set_column_names, :set_delimiter, :set_double_quoted, :set_escape_character, :set_escaped, :set_false_values, :set_generate_column_names, :set_ignore_empty_lines, :set_n_skip_rows, :set_null_values, :set_property, :set_quote_character, :set_quoted, :set_true_values, :set_use_threads, :signal_connect, :signal_connect_after, :signal_emit, :signal_emit_stop, :signal_handler_block, :signal_handler_disconnect, :signal_handler_is_connected?, :signal_handler_unblock, :signal_has_handler_pending?, :thaw_notify, :true_values, :true_values=, :type_name, :unref, :use_threads=, :use_threads?] ``` -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ARROW-6812) [Java] Remove Dremio Corp. from License Header
Rohit Gupta created ARROW-6812: -- Summary: [Java] Remove Dremio Corp. from License Header Key: ARROW-6812 URL: https://issues.apache.org/jira/browse/ARROW-6812 Project: Apache Arrow Issue Type: Bug Components: Java Reporter: Rohit Gupta Remove Dremio Corporation from License Headers -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [VOTE] Release Apache Arrow 0.15.0 - RC2
I think we might be a little aggressive at removing artifacts from the dist system Can we change our process to only remove old dist artifacts when we are about to upload a new RC? Otherwise it's harder to make improvements to the release verification scripts without any old RC to test against On Mon, Oct 7, 2019 at 5:17 PM Neal Richardson wrote: > > The R package has been accepted by CRAN. Binaries for macOS and > Windows should become available in the next few days. > > Neal > > On Mon, Oct 7, 2019 at 1:41 AM Krisztián Szűcs > wrote: > > > > Thanks Andy! > > > > I've just removed the RC source artefacts from SVN. > > > > We have two remaining post release tasks: > > - homebrew > > - apidocs > > > > On Mon, Oct 7, 2019 at 1:47 AM Andy Grove wrote: > > > > > I released the Rust crates from the RC2 source tarball. I had to comment > > > out the benchmark references in the Cargo.toml first since the tarball > > > does > > > not include the benchmark source code. I filed > > > https://issues.apache.org/jira/browse/ARROW-6801 for this bug and will > > > fix the packaging before the 1.0.0 release. > > > > > > On Sun, Oct 6, 2019 at 2:01 AM Krisztián Szűcs > > > wrote: > > > > > >> The rust publishing script fails because it cannot find the benchmarks. > > >> Seems to be related to cargo changes. > > >> I cannot investigate it right now, @Andy could you take a look? > > >> > > >> On Sun, Oct 6, 2019, 9:11 AM Krisztián Szűcs > > >> wrote: > > >> > > >>> - published js packages to npm, please check that they are working > > >>> properly > > >>> > > >>> On Sat, Oct 5, 2019 at 10:14 PM Neal Richardson < > > >>> neal.p.richard...@gmail.com> wrote: > > >>> > > R release steps per > > > > https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide#ReleaseManagementGuide-UpdatingRpackages > > are underway. > > > > Neal > > > > On Sat, Oct 5, 2019 at 8:40 AM Krisztián Szűcs > > wrote: > > > > > > - website updated with the release notes > > > - conda-forge updates are merged > > > > > > Remaining: > > > - Javascript > > > - Rust > > > - R > > > - Homebrew > > > - Apidocs > > > > > > On Sat, Oct 5, 2019 at 2:58 PM Sutou Kouhei > > wrote: > > > > > > > - uploaded C# packages > > > > > > > > In > t1k_vz68rcb3m...@mail.gmail.com> > > > > "Re: [VOTE] Release Apache Arrow 0.15.0 - RC2" on Sat, 5 Oct 2019 > > > > 14:50:51 +0200, > > > > Krisztián Szűcs wrote: > > > > > > > > > - uploaded python wheels to pypi > > > > > - uploaded java artifacts to maven central > > > > > > > > > > I'm going to update the conda recipes. > > > > > > > > > > Remaining: > > > > > - Javascript > > > > > - Rust > > > > > - C# > > > > > - R > > > > > - Homebrew > > > > > - Site > > > > > > > > > > > > > > > > > > > > On Sat, Oct 5, 2019 at 2:29 PM Krisztián Szűcs < > > > > szucs.kriszt...@gmail.com> > > > > > wrote: > > > > > > > > > >> - rebased master > > > > >> - rebased the pull requests > > > > >> - released the jira version > > > > >> - started the new jira version > > > > >> - uploaded source artifacts to svn > > > > >> - uploaded binary artifacts to bintray > > > > >> - currently uploading python wheels to pypi > > > > >> > > > > >> > > > > >> On Sat, Oct 5, 2019 at 2:04 PM Sutou Kouhei > > > > >> > > wrote: > > > > >> > > > > >>> I'll release RubyGems. > > > > >>> > > > > >>> In < > > cahm19a5pxw5mq5zgb7pxoerg9rkxrhmadcrpmkw12jkjokw...@mail.gmail.com > > > > > > > > > >>> "Re: [VOTE] Release Apache Arrow 0.15.0 - RC2" on Sat, 5 Oct > > 2019 > > > > >>> 11:46:16 +0200, > > > > >>> Krisztián Szűcs wrote: > > > > >>> > > > > >>> > On Sat, Oct 5, 2019 at 11:40 AM Krisztián Szűcs < > > > > >>> szucs.kriszt...@gmail.com> > > > > >>> > wrote: > > > > >>> > > > > > >>> >> The VOTE carries with 5 binding +1 votes and 2 non-binding > > +1 votes. > > > > >>> >> > > > > >>> >> On Fri, Oct 4, 2019 at 10:04 PM Wes McKinney < > > wesmck...@gmail.com> > > > > >>> wrote: > > > > >>> >> > > > > >>> >>> The commits from your local RC branch aren't available so I > > cannot > > > > >>> >>> rebase master yet, I'll just wait for you to be available > > again. > > > > >>> >> > > > > >>> >> Actually those commits are available under the tag. > > > > >>> >> > > > > >>> > The new version updates are indeed only available locally. > > I'm doing > > > > the > > > > >>> > rebase. > > > > >>> > > > > > >>> >> If anyone has some spare time we should try to complete as > > many > > > > >>> >>> post-release tasks this weekend so we can announce the > >
[jira] [Created] (ARROW-6811) [R] Assorted post-0.15 release cleanups
Neal Richardson created ARROW-6811: -- Summary: [R] Assorted post-0.15 release cleanups Key: ARROW-6811 URL: https://issues.apache.org/jira/browse/ARROW-6811 Project: Apache Arrow Issue Type: Improvement Components: R Reporter: Neal Richardson Assignee: Neal Richardson -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ARROW-6810) [Website] Add docs for R package 0.15 release
Neal Richardson created ARROW-6810: -- Summary: [Website] Add docs for R package 0.15 release Key: ARROW-6810 URL: https://issues.apache.org/jira/browse/ARROW-6810 Project: Apache Arrow Issue Type: Improvement Components: Documentation, Website Reporter: Neal Richardson Assignee: Neal Richardson Fix For: 1.0.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [VOTE] Release Apache Arrow 0.15.0 - RC2
The R package has been accepted by CRAN. Binaries for macOS and Windows should become available in the next few days. Neal On Mon, Oct 7, 2019 at 1:41 AM Krisztián Szűcs wrote: > > Thanks Andy! > > I've just removed the RC source artefacts from SVN. > > We have two remaining post release tasks: > - homebrew > - apidocs > > On Mon, Oct 7, 2019 at 1:47 AM Andy Grove wrote: > > > I released the Rust crates from the RC2 source tarball. I had to comment > > out the benchmark references in the Cargo.toml first since the tarball does > > not include the benchmark source code. I filed > > https://issues.apache.org/jira/browse/ARROW-6801 for this bug and will > > fix the packaging before the 1.0.0 release. > > > > On Sun, Oct 6, 2019 at 2:01 AM Krisztián Szűcs > > wrote: > > > >> The rust publishing script fails because it cannot find the benchmarks. > >> Seems to be related to cargo changes. > >> I cannot investigate it right now, @Andy could you take a look? > >> > >> On Sun, Oct 6, 2019, 9:11 AM Krisztián Szűcs > >> wrote: > >> > >>> - published js packages to npm, please check that they are working > >>> properly > >>> > >>> On Sat, Oct 5, 2019 at 10:14 PM Neal Richardson < > >>> neal.p.richard...@gmail.com> wrote: > >>> > R release steps per > > https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide#ReleaseManagementGuide-UpdatingRpackages > are underway. > > Neal > > On Sat, Oct 5, 2019 at 8:40 AM Krisztián Szűcs > wrote: > > > > - website updated with the release notes > > - conda-forge updates are merged > > > > Remaining: > > - Javascript > > - Rust > > - R > > - Homebrew > > - Apidocs > > > > On Sat, Oct 5, 2019 at 2:58 PM Sutou Kouhei > wrote: > > > > > - uploaded C# packages > > > > > > In t1k_vz68rcb3m...@mail.gmail.com> > > > "Re: [VOTE] Release Apache Arrow 0.15.0 - RC2" on Sat, 5 Oct 2019 > > > 14:50:51 +0200, > > > Krisztián Szűcs wrote: > > > > > > > - uploaded python wheels to pypi > > > > - uploaded java artifacts to maven central > > > > > > > > I'm going to update the conda recipes. > > > > > > > > Remaining: > > > > - Javascript > > > > - Rust > > > > - C# > > > > - R > > > > - Homebrew > > > > - Site > > > > > > > > > > > > > > > > On Sat, Oct 5, 2019 at 2:29 PM Krisztián Szűcs < > > > szucs.kriszt...@gmail.com> > > > > wrote: > > > > > > > >> - rebased master > > > >> - rebased the pull requests > > > >> - released the jira version > > > >> - started the new jira version > > > >> - uploaded source artifacts to svn > > > >> - uploaded binary artifacts to bintray > > > >> - currently uploading python wheels to pypi > > > >> > > > >> > > > >> On Sat, Oct 5, 2019 at 2:04 PM Sutou Kouhei > wrote: > > > >> > > > >>> I'll release RubyGems. > > > >>> > > > >>> In < > cahm19a5pxw5mq5zgb7pxoerg9rkxrhmadcrpmkw12jkjokw...@mail.gmail.com > > > > > > > >>> "Re: [VOTE] Release Apache Arrow 0.15.0 - RC2" on Sat, 5 Oct > 2019 > > > >>> 11:46:16 +0200, > > > >>> Krisztián Szűcs wrote: > > > >>> > > > >>> > On Sat, Oct 5, 2019 at 11:40 AM Krisztián Szűcs < > > > >>> szucs.kriszt...@gmail.com> > > > >>> > wrote: > > > >>> > > > > >>> >> The VOTE carries with 5 binding +1 votes and 2 non-binding > +1 votes. > > > >>> >> > > > >>> >> On Fri, Oct 4, 2019 at 10:04 PM Wes McKinney < > wesmck...@gmail.com> > > > >>> wrote: > > > >>> >> > > > >>> >>> The commits from your local RC branch aren't available so I > cannot > > > >>> >>> rebase master yet, I'll just wait for you to be available > again. > > > >>> >> > > > >>> >> Actually those commits are available under the tag. > > > >>> >> > > > >>> > The new version updates are indeed only available locally. > I'm doing > > > the > > > >>> > rebase. > > > >>> > > > > >>> >> If anyone has some spare time we should try to complete as > many > > > >>> >>> post-release tasks this weekend so we can announce the > release on > > > >>> >>> Monday or Tuesday next week. > > > >>> >>> > > > >>> >> > > > >>> >> We have many post-release tasks, any help is appreciated. > > > >>> >> > > > >>> >>> > > > >>> >>> Thanks all for your help getting this release ready! > > > >>> >>> > > > >>> >>> On Fri, Oct 4, 2019 at 6:40 AM Krisztián Szűcs > > > >>> >>> wrote: > > > >>> >>> > > > > >>> >>> > We have 5 binding +1 votes and 2 non-binding +1 votes so > far. > > > >>> >>> > The 72 hours has passed, so we can close the release vote. > > > >>> >>> > > > > >>> >>> > Sadly I won't be avail
[DRAFT] Apache Arrow Board Report - October 2019
Here is a start for our Q3 board report ## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange ## Issues: There are no issues requiring board attention at this time ## Membership Data: * Apache Arrow was founded 2016-01-19 (4 years ago) * There are currently 48 committers and 28 PMC members in this project. * The Committer-to-PMC ratio is roughly 3:2. Community changes, past quarter: - Micah Kornfield was added to the PMC on 2019-08-21 - Sebastien Binet was added to the PMC on 2019-08-21 - Ben Kietzman was added as committer on 2019-09-07 - David Li was added as committer on 2019-08-30 - Kenta Murata was added as committer on 2019-09-05 - Neal Richardson was added as committer on 2019-09-05 - Praveen Kumar was added as committer on 2019-07-14 ## Project Activity: * The project has just made a 0.15.0 release. * We are discussing ways to make the Arrow libraries as accessible as possible to downstream projects for minimal use cases while allowing the development of more comprehensive "standard libraries" with larger dependency stacks in the project * We plan to make a 1.0.0 release as our next major release, at which time we will declare that the Arrow binary protocol is stable with forward and backward compatibility guarantees * We are struggling with Continuous Integration scalability as the project has definitely outgrown what Travis CI and Appveyor can do for us. We are exploring alternative solutions such as Buildbot, Buildkite (see INFRA-19217), and GitHub Actions to provide a path to migrate away from Travis CI / Appveyor ## Community Health: * The community is overall healthy, with the aforementioned concerns around CI scalability. New contributors frequently take notice of the long build queue times when submitting pull requests.
Re: [Proposal]: Expose Flight gRPC for Dremio use case (Java)
Corresponding JIRA issue: https://issues.apache.org/jira/browse/ARROW-6807 Wes, It's possible for a single gRPC server to expose multiple services.The unit test in my patch highlights that functionality. PR: https://github.com/apache/arrow/pull/5597 On 2019/10/04 19:36:52, Wes McKinney wrote: > Is it possible for a single gRPC server to expose multiple services > through the same port (it sounds like it is)? It would be a good idea > to do similar refactoring in C++ so that Flight RPC endpoints can be > provided alongside some other non-Flight endpoints in the same gRPC > server > > On Fri, Oct 4, 2019 at 1:49 PM David Li wrote: > > > > Hi Rohit, > > > > This sounds interesting, and I think we've voiced support for > > something similar before :) > > > > Given that Flight does want to abstract over the exact backends, > > though, how should we approach this? Is the proposal to also refactor > > Flight/Java such that the core classes are just interfaces (or > > delegate to interfaces) that anyone can implement, and have the gRPC > > implementation as the reference one? Or is this just proposing to > > expose the gRPC implementation under a separate namespace, and leave > > that question for later? > > > > Best, > > David > > > > On 10/4/19, Rohit Gupta wrote: > > > Hi, > > > > > > At dremio we are using gRPC for JobsService. One of the api's relies on > > > Arrow Flight. We want access to the Flight service so we can bind it to > > > the > > > same managed channel as the rest of JobsService (& not have a completely > > > separate server). > > > > > > The approach would be to create a new module within the same package > > > (org.apache.arrow.flight) and have 2 classes FlightGrpcServer & > > > FlightGrpcClient that expose the client & server, and also make > > > FlightClient ctor package-private. > > > > > > Please let us know if you have questions or concerns. > > > > > > Best, > > > Rohit > > > >
PSA: TensorFlow Extended (TFX) is proposing to use Arrow for in-memory representations
The proposal is in PR form at: [1]. I thought I'd mention it here in case people are interested but haven't seen it yet. [1] https://github.com/tensorflow/community/pull/162
Re: [DISCUSS] Dictionary Encoding Clarifications/Future Proofing
> > > So, why would we allow dictionary replacement instead of have the > > emitter use a new dictionary id? Is it to optimize memory consumption > > on the receiver? > The dictionary id's are set in the schema, so it's not possible to > change the dictionary id after the schema has been sent. It is conceivably possible to add another field to DictionaryBatch to indicate the dictionary is a replacement for another ID, but this seems like it would make the IPC protocol messier (i.e. we could eliminate isDelta). But I think this is more confusing from a protocol perspective. It might make implementations simpler though. On Sun, Oct 6, 2019 at 1:25 PM Wes McKinney wrote: > On Sun, Oct 6, 2019 at 4:30 AM Antoine Pitrou wrote: > > > > On Sat, 5 Oct 2019 17:01:27 -0600 > > Micah Kornfield wrote: > > > I've opened a pull request [1] to clarify some recent conversations > about > > > semantics/edge cases for dictionary encoding [2][3] around interleaved > > > batches and when isDelta=False. > > > > > > Specifically, it proposes isDelta=False indicates dictionary > replacement. > > > > So, why would we allow dictionary replacement instead of have the > > emitter use a new dictionary id? Is it to optimize memory consumption > > on the receiver? > > The dictionary id's are set in the schema, so it's not possible to > change the dictionary id after the schema has been sent. > > > Regards > > > > Antoine. > > > > >
[jira] [Created] (ARROW-6809) [RUBY] Gem does not install on macOS due to glib2 3.3.7 compilation failure
Keith Wedinger created ARROW-6809: - Summary: [RUBY] Gem does not install on macOS due to glib2 3.3.7 compilation failure Key: ARROW-6809 URL: https://issues.apache.org/jira/browse/ARROW-6809 Project: Apache Arrow Issue Type: Bug Components: Ruby Affects Versions: 0.15.0 Environment: macOS Mojave 10.14.6 Ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-darwin18] Xcode 10.3 Reporter: Keith Wedinger *System information:* * macOS Mojave 10.14.6 * Ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-darwin18] managed via rbenv *Reproduction steps:* Run `gem install red-arrow` *Observe:* The following compilation errors occur during compilation of dependent gem glib2 3.3.7: ``` Building native extensions. This could take a while... ERROR: Error installing red-arrow: ERROR: Failed to build gem native extension. current directory: /Users/kwedinger/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/glib2-3.3.7/ext/glib2 /Users/kwedinger/.rbenv/versions/2.6.3/bin/ruby -I /Users/kwedinger/.rbenv/versions/2.6.3/lib/ruby/2.6.0 -r ./siteconf20191007-84053-1y4ly2q.rb extconf.rb checking for --enable-debug-build option... no checking for -Wall option to compiler... yes checking for -Waggregate-return option to compiler... yes checking for -Wcast-align option to compiler... yes checking for -Wextra option to compiler... no checking for -Wformat=2 option to compiler... yes checking for -Winit-self option to compiler... yes checking for -Wlarger-than-65500 option to compiler... yes checking for -Wmissing-declarations option to compiler... yes checking for -Wmissing-format-attribute option to compiler... yes checking for -Wmissing-include-dirs option to compiler... yes checking for -Wmissing-noreturn option to compiler... yes checking for -Wmissing-prototypes option to compiler... yes checking for -Wnested-externs option to compiler... yes checking for -Wold-style-definition option to compiler... yes checking for -Wpacked option to compiler... yes checking for -Wp,-D_FORTIFY_SOURCE=2 option to compiler... yes checking for -Wpointer-arith option to compiler... yes checking for -Wswitch-default option to compiler... yes checking for -Wswitch-enum option to compiler... yes checking for -Wundef option to compiler... yes checking for -Wout-of-line-declaration option to compiler... yes checking for -Wunsafe-loop-optimizations option to compiler... no checking for -Wwrite-strings option to compiler... yes checking for Homebrew... yes checking for gobject-2.0 version (>= 2.12.0)... yes checking for gthread-2.0... yes checking for unistd.h... yes checking for io.h... no checking for g_spawn_close_pid() in glib.h... yes checking for g_thread_init() in glib.h... yes checking for g_main_depth() in glib.h... yes checking for g_listenv() in glib.h... yes checking for rb_check_array_type() in ruby.h... yes checking for rb_check_hash_type() in ruby.h... yes checking for rb_exec_recursive() in ruby.h... yes checking for rb_errinfo() in ruby.h... yes checking for rb_thread_call_without_gvl() in ruby.h... yes checking for ruby_native_thread_p() in ruby.h... yes checking for rb_thread_call_with_gvl() in ruby.h... yes checking for rb_gc_register_mark_object() in ruby.h... yes checking for rb_exc_new_str() in ruby.h... yes checking for rb_enc_str_new_static() in ruby.h... yes checking for curr_thread in ruby.h,node.h... no checking for rb_curr_thread in ruby.h,node.h... no creating ruby-glib2.pc creating glib-enum-types.c creating glib-enum-types.h creating Makefile current directory: /Users/kwedinger/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/glib2-3.3.7/ext/glib2 make "DESTDIR=" clean current directory: /Users/kwedinger/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/glib2-3.3.7/ext/glib2 make "DESTDIR=" compiling rbglib-gc.c compiling rbgobj_signal.c compiling rbglib_int64.c compiling rbglib_convert.c compiling rbglib_bookmarkfile.c compiling rbglib-variant.c compiling glib-enum-types.c glib-enum-types.c:632:9: warning: 'G_SPAWN_ERROR_2BIG' is deprecated: Use 'G_SPAWN_ERROR_TOO_BIG' instead [-Wdeprecated-declarations] { G_SPAWN_ERROR_2BIG, "G_SPAWN_ERROR_2BIG", "2big" }, ^ /usr/local/Cellar/glib/2.62.1/include/glib-2.0/glib/gspawn.h:76:22: note: 'G_SPAWN_ERROR_2BIG' has been explicitly marked deprecated here G_SPAWN_ERROR_2BIG GLIB_DEPRECATED_ENUMERATOR_IN_2_32_FOR(G_SPAWN_ERROR_TOO_BIG) = G_SPAWN_ERROR_TOO_BIG, ^ /usr/local/Cellar/glib/2.62.1/include/glib-2.0/glib/gversionmacros.h:421:54: note: expanded from macro 'GLIB_DEPRECATED_ENUMERATOR_IN_2_32_FOR' # define GLIB_DEPRECATED_ENUMERATOR_IN_2_32_FOR(f) GLIB_DEPRECATED_ENUMERATOR_FOR(f) ^ /usr/local/Cellar/glib/2.62.1/include/glib-2.0/glib/gmacros.h:1002:43: note: expanded from macro 'GLIB_DEPRECATED_ENUMERATOR_FOR' #define GLIB_
Re: [ANNOUNCE] Apache Arrow 0.15.0 released
Great work everyone! On Sun, Oct 6, 2019 at 1:46 PM Wes McKinney wrote: > Congrats everyone! > > On Sat, Oct 5, 2019 at 10:09 AM Krisztián Szűcs wrote: > > > > The Apache Arrow community is pleased to announce the 0.15.0 release. > > The release includes 711 resolved issues ([1]) since the 0.14.0 release. > > > > The release is available now from our website, [2] and [3]: > > http://arrow.apache.org/install/ > > > > Release notes are available at: > > https://arrow.apache.org/release/0.15.0.html > > > > What is Apache Arrow? > > - > > > > Apache Arrow is a cross-language development platform for in-memory > data. It > > specifies a standardized language-independent columnar memory format for > > flat > > and hierarchical data, organized for efficient analytic operations on > modern > > hardware. It also provides computational libraries and zero-copy > streaming > > messaging and interprocess communication. Languages currently supported > > include > > C, C++, C#, Go, Java, JavaScript, MATLAB, Python, R, Ruby, and Rust. > > > > Please report any feedback to the mailing lists ([4]) > > > > Regards, > > The Apache Arrow community > > > > [1]: https://issues.apache.org/jira/projects/ARROW/versions/12345978 > > [2]: https://www.apache.org/dyn/closer.cgi/arrow/arrow-0.15.0/ > > [3]: https://bintray.com/apache/arrow > > [4]: https://lists.apache.org/list.html?dev@arrow.apache.org >
[jira] [Created] (ARROW-6808) [ruby] Doesn't build on windows msys2
Dominic Sisneros created ARROW-6808: --- Summary: [ruby] Doesn't build on windows msys2 Key: ARROW-6808 URL: https://issues.apache.org/jira/browse/ARROW-6808 Project: Apache Arrow Issue Type: Bug Environment: windows 10 msys2 Reporter: Dominic Sisneros C:\Users\Dominic E Sisneros>gem update red-arrow Updating installed gems Updating red-arrow Temporarily enhancing PATH for MSYS/MINGW... Installing required msys2 packages: mingw-w64-x86_64-arrow warning: mingw-w64-x86_64-arrow-0.14.0-2 is up to date -- skipping Building native extensions. This could take a while... ERROR: Error installing red-arrow: ERROR: Failed to build gem native extension. current directory: c:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/red-arrow-0.15.0/ext/arrow c:/Ruby26-x64/bin/ruby.exe -I c:/Ruby26-x64/lib/ruby/site_ruby/2.6.0 -r ./siteconf20191007-20416-1m6q3x3.rb extconf.rb checking --enable-debug-build option... no checking C++ compiler... x86_64-w64-mingw32-g++ checking g++ version... 9.2 (gnu++14) checking for --enable-debug-build option... no checking for -Wall option to compiler... yes checking for -Waggregate-return option to compiler... yes checking for -Wcast-align option to compiler... yes checking for -Wextra option to compiler... no checking for -Wformat=2 option to compiler... yes checking for -Winit-self option to compiler... yes checking for -Wlarger-than-65500 option to compiler... yes checking for -Wmissing-declarations option to compiler... yes checking for -Wmissing-format-attribute option to compiler... yes checking for -Wmissing-include-dirs option to compiler... yes checking for -Wmissing-noreturn option to compiler... yes checking for -Wmissing-prototypes option to compiler... yes checking for -Wnested-externs option to compiler... no checking for -Wold-style-definition option to compiler... yes checking for -Wpacked option to compiler... yes checking for -Wp,-D_FORTIFY_SOURCE=2 option to compiler... yes checking for -Wpointer-arith option to compiler... yes checking for -Wswitch-default option to compiler... yes checking for -Wswitch-enum option to compiler... yes checking for -Wundef option to compiler... yes checking for -Wout-of-line-declaration option to compiler... no checking for -Wunsafe-loop-optimizations option to compiler... yes checking for -Wwrite-strings option to compiler... yes checking for Homebrew... no checking for arrow... yes checking for arrow-glib... yes creating Makefile current directory: c:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/red-arrow-0.15.0/ext/arrow make "DESTDIR=" clean current directory: c:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/red-arrow-0.15.0/ext/arrow make "DESTDIR=" generating arrow-x64-mingw32.def compiling arrow.cpp compiling converters.cpp compiling raw-records.cpp raw-records.cpp: In lambda function: raw-records.cpp:61:52: error: 'class arrow::Column' has no member named 'chunks' 61 | for (const auto array : chunked_array->chunks()) { |^~ make: *** [Makefile:236: raw-records.o] Error 1 make failed, exit code 2 Gem files will remain installed in c:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/red-arrow-0.15.0 for inspection. Results logged to c:/Ruby26-x64/lib/ruby/gems/2.6.0/extensions/x64-mingw32/2.6.0/red-arrow-0.15.0/gem_make.out Gems updated: red-arrow -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [NIGHTLY] Arrow Build Report for Job nightly-2019-10-07-0
homebrew-cpp failure is a timeout in `brew update`, which is part of the macos builder init stage: https://travis-ci.org/ursa-labs/crossbow/builds/594462704#L115 I don't think we can add a travis_wait to that, so we may just have to accept the occasional flaky failure. Neal On Mon, Oct 7, 2019 at 5:09 AM Crossbow wrote: > > > Arrow Build Report for Job nightly-2019-10-07-0 > > All tasks: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0 > > Failed Tasks: > - debian-stretch-arm64: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-azure-debian-stretch-arm64 > - ubuntu-bionic-arm64: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-azure-ubuntu-bionic-arm64 > - ubuntu-disco-arm64: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-azure-ubuntu-disco-arm64 > - centos-7-aarch64: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-azure-centos-7-aarch64 > - gandiva-jar-osx: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-travis-gandiva-jar-osx > - wheel-osx-cp36m: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-travis-wheel-osx-cp36m > - wheel-osx-cp35m: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-travis-wheel-osx-cp35m > - homebrew-cpp: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-travis-homebrew-cpp > - wheel-osx-cp27m: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-travis-wheel-osx-cp27m > - debian-buster-arm64: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-azure-debian-buster-arm64 > - gandiva-jar-trusty: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-travis-gandiva-jar-trusty > - ubuntu-xenial-arm64: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-azure-ubuntu-xenial-arm64 > - wheel-osx-cp37m: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-travis-wheel-osx-cp37m > - wheel-manylinux2010-cp35m: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-travis-wheel-manylinux2010-cp35m > > Succeeded Tasks: > - wheel-manylinux2010-cp37m: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-travis-wheel-manylinux2010-cp37m > - debian-buster: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-azure-debian-buster > - docker-turbodbc-integration: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-circle-docker-turbodbc-integration > - docker-dask-integration: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-circle-docker-dask-integration > - conda-linux-gcc-py27: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-azure-conda-linux-gcc-py27 > - docker-python-2.7: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-circle-docker-python-2.7 > - ubuntu-bionic: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-azure-ubuntu-bionic > - docker-go: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-circle-docker-go > - wheel-manylinux2010-cp36m: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-travis-wheel-manylinux2010-cp36m > - docker-cpp-release: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-circle-docker-cpp-release > - docker-python-2.7-nopandas: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-circle-docker-python-2.7-nopandas > - conda-linux-gcc-py36: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-azure-conda-linux-gcc-py36 > - docker-python-3.6-nopandas: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-circle-docker-python-3.6-nopandas > - docker-cpp-cmake32: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-circle-docker-cpp-cmake32 > - wheel-manylinux2010-cp27m: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-travis-wheel-manylinux2010-cp27m > - wheel-win-cp36m: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-appveyor-wheel-win-cp36m > - docker-docs: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-circle-docker-docs > - wheel-manylinux1-cp27m: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-travis
[jira] [Created] (ARROW-6807) [Java] Expose gRPC service
Rohit Gupta created ARROW-6807: -- Summary: [Java] Expose gRPC service Key: ARROW-6807 URL: https://issues.apache.org/jira/browse/ARROW-6807 Project: Apache Arrow Issue Type: New Feature Components: Java Reporter: Rohit Gupta Have a utility class that exposes the flight service & client so that multiple services can be plugged into the same endpoint. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ARROW-6806) Segfault deserializing ListArray containing null
Max Bolingbroke created ARROW-6806: -- Summary: Segfault deserializing ListArray containing null Key: ARROW-6806 URL: https://issues.apache.org/jira/browse/ARROW-6806 Project: Apache Arrow Issue Type: Bug Components: C++ Affects Versions: 0.15.0 Reporter: Max Bolingbroke The following code segfaults for me (Windows and Linux, pyarrow 0.13 and 0.15): {code:java} import pyarrow as pa from io import BytesIO x = b'\xdc\x00\x00\x00\x10\x00\x00\x00\x0c\x00\x0e\x00\x06\x00\r\x00\x08\x00\x00\x00\x0c\x00\x00\x00\x00\x00\x03\x00\x10\x00\x00\x00\x00\x01\n\x00\x0c\x00\x00\x00\x08\x00\x04\x00\n\x00\x00\x00\x08\x00\x00\x00\x08\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x18\x00\x00\x00\x00\x00\x12\x00\x18\x00\x14\x00\x13\x00\x12\x00\x0c\x00\x00\x00\x08\x00\x04\x00\x12\x00\x00\x00\x14\x00\x00\x00\x14\x00\x00\x00`\x00\x00\x00\x00\x00\x0c\x01\\\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x18\x00\x00\x00\x00\x00\x12\x00\x18\x00\x14\x00\x00\x00\x13\x00\x0c\x00\x00\x00\x08\x00\x04\x00\x12\x00\x00\x00\x14\x00\x00\x00\x14\x00\x00\x00\x14\x00\x00\x00\x00\x00\x00\x05\x10\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xf0\xff\xff\xff\x06\x00\x00\x00$data$\x00\x00\x04\x00\x04\x00\x04\x00\x00\x00\x10\x00\x00\x00exchangeCodeList\x00\x00\x00\x00\xcc\x00\x00\x00\x14\x00\x00\x00\x00\x00\x00\x00\x0c\x00\x16\x00\x0e\x00\x15\x00\x10\x00\x04\x00\x0c\x00\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x00\x03\x00\x10\x00\x00\x00\x00\x03\n\x00\x18\x00\x0c\x00\x08\x00\x04\x00\n\x00\x00\x00\x14\x00\x00\x00h\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x08\x00\x00\x00\x00\x00\x00\x00\x08\x00\x00\x00\x00\x00\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' r = pa.RecordBatchStreamReader(BytesIO(x)) r.read_all() {code} I *think* what should happen instead is that I should get a Table with a single column named "exchangeCodeList", where the column is a ChunkedArray with a single chunk, where that chunk is a ListArray containing just a single element (a null). Failing that (i.e. if the bytestring is actually malformed), pyarrow should maybe throw an error instead of segfaulting? I'm not 100% sure how the bytestring was generated: I think it comes from a Java-based server. I can deserialize the server response fine if all the records have at least one element in the "exchangeCodeList" column, but not if at least one of them is null. I've tried to reproduce the failure by generating the bytestring with pyarrow but can't trigger the segfault. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ARROW-6805) Should the new field_ argument on Table.set_column() be optional? (Unnecessary breaking change)
Igor Tavares created ARROW-6805: --- Summary: Should the new field_ argument on Table.set_column() be optional? (Unnecessary breaking change) Key: ARROW-6805 URL: https://issues.apache.org/jira/browse/ARROW-6805 Project: Apache Arrow Issue Type: Bug Components: Python Affects Versions: 0.15.0 Environment: All Reporter: Igor Tavares The new *field_* argument (Table.set_column() on version 0.15.0) helps to add extra information on the column metadata. But it should not be mandatory, in some cases it is simply redundant like that: {code:python} import pandas as pd import pyarrow as pa df = pd.DataFrame({"foo": [1, 2, 3]}) tbl = pa.Table.from_pandas(df=df, preserve_index=False) if pa.__version__ == "0.15.0": field = pa.field(name="foo", type="double") tbl = tbl.set_column(0, field, tbl.column("foo").cast("double")) else: tbl = tbl.set_column(0, tbl.column("foo").cast("double")) {code} I think that this argument should be optional to avoid redundant code and to keep compatibility with version <0.15.0. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ARROW-6804) [CI] [Rust] Migrate Travis Rust job to Github Actions
Antoine Pitrou created ARROW-6804: - Summary: [CI] [Rust] Migrate Travis Rust job to Github Actions Key: ARROW-6804 URL: https://issues.apache.org/jira/browse/ARROW-6804 Project: Apache Arrow Issue Type: Improvement Components: Continuous Integration, Rust Reporter: Antoine Pitrou Should be easy, as it's already using docker-compose. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[NIGHTLY] Arrow Build Report for Job nightly-2019-10-07-0
Arrow Build Report for Job nightly-2019-10-07-0 All tasks: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0 Failed Tasks: - debian-stretch-arm64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-azure-debian-stretch-arm64 - ubuntu-bionic-arm64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-azure-ubuntu-bionic-arm64 - ubuntu-disco-arm64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-azure-ubuntu-disco-arm64 - centos-7-aarch64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-azure-centos-7-aarch64 - gandiva-jar-osx: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-travis-gandiva-jar-osx - wheel-osx-cp36m: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-travis-wheel-osx-cp36m - wheel-osx-cp35m: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-travis-wheel-osx-cp35m - homebrew-cpp: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-travis-homebrew-cpp - wheel-osx-cp27m: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-travis-wheel-osx-cp27m - debian-buster-arm64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-azure-debian-buster-arm64 - gandiva-jar-trusty: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-travis-gandiva-jar-trusty - ubuntu-xenial-arm64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-azure-ubuntu-xenial-arm64 - wheel-osx-cp37m: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-travis-wheel-osx-cp37m - wheel-manylinux2010-cp35m: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-travis-wheel-manylinux2010-cp35m Succeeded Tasks: - wheel-manylinux2010-cp37m: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-travis-wheel-manylinux2010-cp37m - debian-buster: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-azure-debian-buster - docker-turbodbc-integration: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-circle-docker-turbodbc-integration - docker-dask-integration: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-circle-docker-dask-integration - conda-linux-gcc-py27: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-azure-conda-linux-gcc-py27 - docker-python-2.7: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-circle-docker-python-2.7 - ubuntu-bionic: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-azure-ubuntu-bionic - docker-go: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-circle-docker-go - wheel-manylinux2010-cp36m: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-travis-wheel-manylinux2010-cp36m - docker-cpp-release: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-circle-docker-cpp-release - docker-python-2.7-nopandas: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-circle-docker-python-2.7-nopandas - conda-linux-gcc-py36: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-azure-conda-linux-gcc-py36 - docker-python-3.6-nopandas: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-circle-docker-python-3.6-nopandas - docker-cpp-cmake32: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-circle-docker-cpp-cmake32 - wheel-manylinux2010-cp27m: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-travis-wheel-manylinux2010-cp27m - wheel-win-cp36m: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-appveyor-wheel-win-cp36m - docker-docs: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-circle-docker-docs - wheel-manylinux1-cp27m: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-travis-wheel-manylinux1-cp27m - ubuntu-xenial: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-azure-ubuntu-xenial - centos-6: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-azure-centos-6 - docker-r-conda: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-circle-docker-r-conda - docker-r: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-10-07-0-circle-docker-r - homebrew-cpp-autobrew: URL:
[jira] [Created] (ARROW-6803) [Rust] [DataFusion] Aggregate queries are slower with new physical query plan
Andy Grove created ARROW-6803: - Summary: [Rust] [DataFusion] Aggregate queries are slower with new physical query plan Key: ARROW-6803 URL: https://issues.apache.org/jira/browse/ARROW-6803 Project: Apache Arrow Issue Type: Sub-task Components: Rust, Rust - DataFusion Affects Versions: 1.0.0 Reporter: Andy Grove Assignee: Andy Grove Fix For: 1.0.0 executing direct from logical plan: {code:java} aggregate_query_no_group_by time: [13.096 us 13.187 us 13.294 us] change: [-88.712% -88.554% -88.398%] (p = 0.00 < 0.05) Performance has improved. Found 5 outliers among 100 measurements (5.00%) 5 (5.00%) high mildaggregate_query_group_by time: [44.153 us 44.816 us 45.541 us] change: [-77.984% -77.485% -77.009%] (p = 0.00 < 0.05) Performance has improved. Found 4 outliers among 100 measurements (4.00%) 4 (4.00%) high mildaggregate_query_group_by_with_filter time: [75.383 us 76.076 us 76.817 us] change: [-72.345% -71.811% -71.097%] (p = 0.00 < 0.05) Performance has improved. Found 16 outliers among 100 measurements (16.00%) {code} executing from physical plan: {code:java} aggregate_query_no_group_by time: [112.13 us 113.63 us 115.26 us] change: [-3.8005% -2.0342% -0.3584%] (p = 0.02 < 0.05) Change within noise threshold. Found 1 outliers among 100 measurements (1.00%) 1 (1.00%) high mildaggregate_query_group_by time: [195.12 us 198.63 us 202.39 us] change: [-1.3814% +1.0612% +3.5732%] (p = 0.40 > 0.05) No change in performance detected. Found 6 outliers among 100 measurements (6.00%) 4 (4.00%) high mild 2 (2.00%) high severeaggregate_query_group_by_with_filter time: [270.69 us 272.18 us 273.63 us] change: [-2.1583% -0.4877% +1.0161%] (p = 0.56 > 0.05) No change in performance detected. {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [VOTE] Release Apache Arrow 0.15.0 - RC2
Thanks Andy! I've just removed the RC source artefacts from SVN. We have two remaining post release tasks: - homebrew - apidocs On Mon, Oct 7, 2019 at 1:47 AM Andy Grove wrote: > I released the Rust crates from the RC2 source tarball. I had to comment > out the benchmark references in the Cargo.toml first since the tarball does > not include the benchmark source code. I filed > https://issues.apache.org/jira/browse/ARROW-6801 for this bug and will > fix the packaging before the 1.0.0 release. > > On Sun, Oct 6, 2019 at 2:01 AM Krisztián Szűcs > wrote: > >> The rust publishing script fails because it cannot find the benchmarks. >> Seems to be related to cargo changes. >> I cannot investigate it right now, @Andy could you take a look? >> >> On Sun, Oct 6, 2019, 9:11 AM Krisztián Szűcs >> wrote: >> >>> - published js packages to npm, please check that they are working >>> properly >>> >>> On Sat, Oct 5, 2019 at 10:14 PM Neal Richardson < >>> neal.p.richard...@gmail.com> wrote: >>> R release steps per https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide#ReleaseManagementGuide-UpdatingRpackages are underway. Neal On Sat, Oct 5, 2019 at 8:40 AM Krisztián Szűcs wrote: > > - website updated with the release notes > - conda-forge updates are merged > > Remaining: > - Javascript > - Rust > - R > - Homebrew > - Apidocs > > On Sat, Oct 5, 2019 at 2:58 PM Sutou Kouhei wrote: > > > - uploaded C# packages > > > > In >>> t1k_vz68rcb3m...@mail.gmail.com> > > "Re: [VOTE] Release Apache Arrow 0.15.0 - RC2" on Sat, 5 Oct 2019 > > 14:50:51 +0200, > > Krisztián Szűcs wrote: > > > > > - uploaded python wheels to pypi > > > - uploaded java artifacts to maven central > > > > > > I'm going to update the conda recipes. > > > > > > Remaining: > > > - Javascript > > > - Rust > > > - C# > > > - R > > > - Homebrew > > > - Site > > > > > > > > > > > > On Sat, Oct 5, 2019 at 2:29 PM Krisztián Szűcs < > > szucs.kriszt...@gmail.com> > > > wrote: > > > > > >> - rebased master > > >> - rebased the pull requests > > >> - released the jira version > > >> - started the new jira version > > >> - uploaded source artifacts to svn > > >> - uploaded binary artifacts to bintray > > >> - currently uploading python wheels to pypi > > >> > > >> > > >> On Sat, Oct 5, 2019 at 2:04 PM Sutou Kouhei wrote: > > >> > > >>> I'll release RubyGems. > > >>> > > >>> In < cahm19a5pxw5mq5zgb7pxoerg9rkxrhmadcrpmkw12jkjokw...@mail.gmail.com > > > > > >>> "Re: [VOTE] Release Apache Arrow 0.15.0 - RC2" on Sat, 5 Oct 2019 > > >>> 11:46:16 +0200, > > >>> Krisztián Szűcs wrote: > > >>> > > >>> > On Sat, Oct 5, 2019 at 11:40 AM Krisztián Szűcs < > > >>> szucs.kriszt...@gmail.com> > > >>> > wrote: > > >>> > > > >>> >> The VOTE carries with 5 binding +1 votes and 2 non-binding +1 votes. > > >>> >> > > >>> >> On Fri, Oct 4, 2019 at 10:04 PM Wes McKinney < wesmck...@gmail.com> > > >>> wrote: > > >>> >> > > >>> >>> The commits from your local RC branch aren't available so I cannot > > >>> >>> rebase master yet, I'll just wait for you to be available again. > > >>> >> > > >>> >> Actually those commits are available under the tag. > > >>> >> > > >>> > The new version updates are indeed only available locally. I'm doing > > the > > >>> > rebase. > > >>> > > > >>> >> If anyone has some spare time we should try to complete as many > > >>> >>> post-release tasks this weekend so we can announce the release on > > >>> >>> Monday or Tuesday next week. > > >>> >>> > > >>> >> > > >>> >> We have many post-release tasks, any help is appreciated. > > >>> >> > > >>> >>> > > >>> >>> Thanks all for your help getting this release ready! > > >>> >>> > > >>> >>> On Fri, Oct 4, 2019 at 6:40 AM Krisztián Szűcs > > >>> >>> wrote: > > >>> >>> > > > >>> >>> > We have 5 binding +1 votes and 2 non-binding +1 votes so far. > > >>> >>> > The 72 hours has passed, so we can close the release vote. > > >>> >>> > > > >>> >>> > Sadly I won't be available for the rest of the day, so I will be > > >>> able > > >>> >>> > to close the vote and start to work on the the post release tasks > > >>> >>> > from tomorrow. > > >>> >>> > @Wes if you have bandwidth feel free to close the vote sooner. > > >>> >>> > > > >>> >>> > > > >>> >>> > On Thu, Oct 3, 2019 at 1:14 AM Bryan Cutler < cutl...@gmail.com> > > >>> wrote: > > >>> >>> > > > >>> >>> > > Accident