[jira] [Updated] (ARROW-7324) [Rust] Add Timezone to Timestamp
[ https://issues.apache.org/jira/browse/ARROW-7324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated ARROW-7324: -- Labels: pull-request-available (was: ) > [Rust] Add Timezone to Timestamp > > > Key: ARROW-7324 > URL: https://issues.apache.org/jira/browse/ARROW-7324 > Project: Apache Arrow > Issue Type: Sub-task > Components: Rust >Reporter: Neville Dipale >Priority: Major > Labels: pull-request-available > > Proposal to add timestamp to timezone type -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ARROW-7325) [Rust] [Parquet] Update to parquet-format 2.6 and thrift 0.12
Kornelijus Survila created ARROW-7325:
-
Summary: [Rust] [Parquet] Update to parquet-format 2.6 and thrift
0.12
Key: ARROW-7325
URL: https://issues.apache.org/jira/browse/ARROW-7325
Project: Apache Arrow
Issue Type: Improvement
Components: Rust
Reporter: Kornelijus Survila
{{parquet-format 2.5}} and {{thrift 0.0.4}} bring in outdated versions of
third-party crates such as {{byteorder}}, {{ordered-float}}, and
{{num-traits}}. Let's update as few of them have reached 1.0.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ARROW-7324) [Rust] Add Timezone to Timestamp
Neville Dipale created ARROW-7324: - Summary: [Rust] Add Timezone to Timestamp Key: ARROW-7324 URL: https://issues.apache.org/jira/browse/ARROW-7324 Project: Apache Arrow Issue Type: Sub-task Components: Rust Reporter: Neville Dipale Proposal to add timestamp to timezone type -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ARROW-6637) [C++] Zero-dependency default core build
[ https://issues.apache.org/jira/browse/ARROW-6637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kouhei Sutou resolved ARROW-6637. - Resolution: Fixed Issue resolved by pull request 5890 [https://github.com/apache/arrow/pull/5890] > [C++] Zero-dependency default core build > > > Key: ARROW-6637 > URL: https://issues.apache.org/jira/browse/ARROW-6637 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Reporter: Wes McKinney >Assignee: Wes McKinney >Priority: Major > Labels: pull-request-available > Fix For: 1.0.0 > > Time Spent: 5h 20m > Remaining Estimate: 0h > > This is a tracking JIRA for items relating to having few or no dependencies > for minimal out-of-the-box builds -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ARROW-7277) [Document] Add discussion about vector lifecycle
[ https://issues.apache.org/jira/browse/ARROW-7277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated ARROW-7277: -- Labels: pull-request-available (was: ) > [Document] Add discussion about vector lifecycle > > > Key: ARROW-7277 > URL: https://issues.apache.org/jira/browse/ARROW-7277 > Project: Apache Arrow > Issue Type: Improvement > Components: Java >Reporter: Liya Fan >Assignee: Liya Fan >Priority: Minor > Labels: pull-request-available > > As discussed in > https://issues.apache.org/jira/browse/ARROW-7254?focusedCommentId=16983284=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16983284, > we need a discussion about the lifecycle of a vector. > Each vector has a lifecycle, and different operations should be performed in > particular phases of the lifecycle. If we violate this, some unexpected > results may be produced. This may cause some confusion for Arrow users. So we > want to add a new section to the prose document, to make it clear and > explicit. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ARROW-7322) [CI][Python] Fall back to arrowdev dockerhub organization for manylinux images
[ https://issues.apache.org/jira/browse/ARROW-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kouhei Sutou resolved ARROW-7322. - Fix Version/s: 1.0.0 Resolution: Fixed Issue resolved by pull request 5966 [https://github.com/apache/arrow/pull/5966] > [CI][Python] Fall back to arrowdev dockerhub organization for manylinux images > -- > > Key: ARROW-7322 > URL: https://issues.apache.org/jira/browse/ARROW-7322 > Project: Apache Arrow > Issue Type: Task > Components: Continuous Integration, Python >Reporter: Krisztian Szucs >Assignee: Krisztian Szucs >Priority: Major > Labels: pull-request-available > Fix For: 1.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Until https://issues.apache.org/jira/browse/INFRA-19499 is resolved. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ARROW-7323) [CI][Rust] Nightly CI is failed by different toolchain
[
https://issues.apache.org/jira/browse/ARROW-7323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kouhei Sutou resolved ARROW-7323.
-
Fix Version/s: 1.0.0
Resolution: Fixed
Issue resolved by pull request 5968
[https://github.com/apache/arrow/pull/5968]
> [CI][Rust] Nightly CI is failed by different toolchain
> --
>
> Key: ARROW-7323
> URL: https://issues.apache.org/jira/browse/ARROW-7323
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Continuous Integration, Rust
>Reporter: Kouhei Sutou
>Assignee: Kouhei Sutou
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.0.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> https://circleci.com/gh/ursa-labs/crossbow/5685
> {noformat}
> Step 8/8 : RUN rustup component add rustfmt --toolchain
> nightly-2019-11-14-x86_64-unknown-linux-gnu
> ---> Running in b28e33fbf36d
> error: toolchain 'nightly-2019-11-14-x86_64-unknown-linux-gnu' is not
> installed
> ERROR: Service 'debian-rust' failed to build: The command '/bin/sh -c rustup
> component add rustfmt --toolchain
> nightly-2019-11-14-x86_64-unknown-linux-gnu' returned a non-zero code: 1
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (ARROW-7321) [CI][GLib] Failed to build with GLib warning
[
https://issues.apache.org/jira/browse/ARROW-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kouhei Sutou resolved ARROW-7321.
-
Fix Version/s: 1.0.0
Resolution: Fixed
Issue resolved by pull request 5964
[https://github.com/apache/arrow/pull/5964]
> [CI][GLib] Failed to build with GLib warning
>
>
> Key: ARROW-7321
> URL: https://issues.apache.org/jira/browse/ARROW-7321
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Continuous Integration, GLib
>Reporter: Kouhei Sutou
>Assignee: Kouhei Sutou
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.0.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> https://circleci.com/gh/ursa-labs/crossbow/5681
> {noformat}
> FAILED: arrow-glib/20f505c@@arrow-glib@sha/file.cpp.o
> ccache c++ -Iarrow-glib/20f505c@@arrow-glib@sha -Iarrow-glib
> -I../../arrow/c_glib/arrow-glib -I. -I../../arrow/c_glib/
> -I/usr/local/include -I/usr/include/glib-2.0
> -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -fdiagnostics-color=always -pipe
> -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -g
> -Werror -DARROW_NO_DEPRECATED_API -fPIC -pthread -MD -MQ
> 'arrow-glib/20f505c@@arrow-glib@sha/file.cpp.o' -MF
> 'arrow-glib/20f505c@@arrow-glib@sha/file.cpp.o.d' -o
> 'arrow-glib/20f505c@@arrow-glib@sha/file.cpp.o' -c
> ../../arrow/c_glib/arrow-glib/file.cpp
> In file included from /usr/include/glib-2.0/gobject/gobject.h:24:0,
> from /usr/include/glib-2.0/gobject/gbinding.h:29,
> from /usr/include/glib-2.0/glib-object.h:23,
> from ../../arrow/c_glib/arrow-glib/error.h:22,
> from ../../arrow/c_glib/arrow-glib/error.hpp:24,
> from ../../arrow/c_glib/arrow-glib/file.cpp:26:
> ../../arrow/c_glib/arrow-glib/file.cpp: In function 'GType
> garrow_file_get_type()':
> /usr/include/glib-2.0/gobject/gtype.h:219:50: error: '<<' in boolean context,
> did you mean '<' ? [-Werror=int-in-bool-context]
> #define G_TYPE_MAKE_FUNDAMENTAL(x) ((GType) ((x) <<
> G_TYPE_FUNDAMENTAL_SHIFT))
> ~^~~~
> /usr/include/glib-2.0/gobject/gtype.h:2015:11: note: in definition of macro
> '_G_DEFINE_INTERFACE_EXTENDED_BEGIN'
>if (TYPE_PREREQ) \
>^~~
> /usr/include/glib-2.0/gobject/gtype.h:1756:47: note: in expansion of macro
> 'G_DEFINE_INTERFACE_WITH_CODE'
> #define G_DEFINE_INTERFACE(TN, t_n, T_P)
> G_DEFINE_INTERFACE_WITH_CODE(TN, t_n, T_P, ;)
>^~~~
> ../../arrow/c_glib/arrow-glib/file.cpp:40:1: note: in expansion of macro
> 'G_DEFINE_INTERFACE'
> G_DEFINE_INTERFACE(GArrowFile,
> ^~
> /usr/include/glib-2.0/gobject/gtype.h:178:25: note: in expansion of macro
> 'G_TYPE_MAKE_FUNDAMENTAL'
> #define G_TYPE_OBJECT G_TYPE_MAKE_FUNDAMENTAL (20)
> ^~~
> ../../arrow/c_glib/arrow-glib/file.cpp:42:20: note: in expansion of macro
> 'G_TYPE_OBJECT'
> G_TYPE_OBJECT)
> ^
> cc1plus: all warnings being treated as errors
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ARROW-7323) [CI][Rust] Nightly CI is failed by different toolchain
Kouhei Sutou created ARROW-7323:
---
Summary: [CI][Rust] Nightly CI is failed by different toolchain
Key: ARROW-7323
URL: https://issues.apache.org/jira/browse/ARROW-7323
Project: Apache Arrow
Issue Type: Improvement
Components: Continuous Integration, Rust
Reporter: Kouhei Sutou
Assignee: Kouhei Sutou
https://circleci.com/gh/ursa-labs/crossbow/5685
{noformat}
Step 8/8 : RUN rustup component add rustfmt --toolchain
nightly-2019-11-14-x86_64-unknown-linux-gnu
---> Running in b28e33fbf36d
error: toolchain 'nightly-2019-11-14-x86_64-unknown-linux-gnu' is not installed
ERROR: Service 'debian-rust' failed to build: The command '/bin/sh -c rustup
component add rustfmt --toolchain nightly-2019-11-14-x86_64-unknown-linux-gnu'
returned a non-zero code: 1
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ARROW-7323) [CI][Rust] Nightly CI is failed by different toolchain
[
https://issues.apache.org/jira/browse/ARROW-7323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated ARROW-7323:
--
Labels: pull-request-available (was: )
> [CI][Rust] Nightly CI is failed by different toolchain
> --
>
> Key: ARROW-7323
> URL: https://issues.apache.org/jira/browse/ARROW-7323
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Continuous Integration, Rust
>Reporter: Kouhei Sutou
>Assignee: Kouhei Sutou
>Priority: Major
> Labels: pull-request-available
>
> https://circleci.com/gh/ursa-labs/crossbow/5685
> {noformat}
> Step 8/8 : RUN rustup component add rustfmt --toolchain
> nightly-2019-11-14-x86_64-unknown-linux-gnu
> ---> Running in b28e33fbf36d
> error: toolchain 'nightly-2019-11-14-x86_64-unknown-linux-gnu' is not
> installed
> ERROR: Service 'debian-rust' failed to build: The command '/bin/sh -c rustup
> component add rustfmt --toolchain
> nightly-2019-11-14-x86_64-unknown-linux-gnu' returned a non-zero code: 1
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ARROW-7146) [R][CI] Pre-install the R dependencies in the dockerfile
[ https://issues.apache.org/jira/browse/ARROW-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated ARROW-7146: -- Labels: pull-request-available (was: ) > [R][CI] Pre-install the R dependencies in the dockerfile > > > Key: ARROW-7146 > URL: https://issues.apache.org/jira/browse/ARROW-7146 > Project: Apache Arrow > Issue Type: Improvement > Components: Continuous Integration, R >Reporter: Krisztian Szucs >Assignee: Neal Richardson >Priority: Major > Labels: pull-request-available > > Here: > https://github.com/apache/arrow/blob/master/ci/docker/linux-apt-r.dockerfile#L61 > Using the https://github.com/apache/arrow/blob/master/ci/scripts/r_deps.sh > script -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (ARROW-7146) [R][CI] Pre-install the R dependencies in the dockerfile
[ https://issues.apache.org/jira/browse/ARROW-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neal Richardson reassigned ARROW-7146: -- Assignee: Neal Richardson > [R][CI] Pre-install the R dependencies in the dockerfile > > > Key: ARROW-7146 > URL: https://issues.apache.org/jira/browse/ARROW-7146 > Project: Apache Arrow > Issue Type: Improvement > Components: Continuous Integration, R >Reporter: Krisztian Szucs >Assignee: Neal Richardson >Priority: Major > > Here: > https://github.com/apache/arrow/blob/master/ci/docker/linux-apt-r.dockerfile#L61 > Using the https://github.com/apache/arrow/blob/master/ci/scripts/r_deps.sh > script -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ARROW-7322) [CI][Python] Fall back to arrowdev dockerhub organization for manylinux images
[ https://issues.apache.org/jira/browse/ARROW-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated ARROW-7322: -- Labels: pull-request-available (was: ) > [CI][Python] Fall back to arrowdev dockerhub organization for manylinux images > -- > > Key: ARROW-7322 > URL: https://issues.apache.org/jira/browse/ARROW-7322 > Project: Apache Arrow > Issue Type: Task > Components: Continuous Integration, Python >Reporter: Krisztian Szucs >Assignee: Krisztian Szucs >Priority: Major > Labels: pull-request-available > > Until https://issues.apache.org/jira/browse/INFRA-19499 is resolved. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ARROW-7322) [CI][Python] Fall back to arrowdev dockerhub organization for manylinux images
Krisztian Szucs created ARROW-7322: -- Summary: [CI][Python] Fall back to arrowdev dockerhub organization for manylinux images Key: ARROW-7322 URL: https://issues.apache.org/jira/browse/ARROW-7322 Project: Apache Arrow Issue Type: Task Components: Continuous Integration, Python Reporter: Krisztian Szucs Assignee: Krisztian Szucs Until https://issues.apache.org/jira/browse/INFRA-19499 is resolved. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ARROW-7233) [C++] Add Result APIs to IPC module
[ https://issues.apache.org/jira/browse/ARROW-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988276#comment-16988276 ] Wes McKinney commented on ARROW-7233: - I agree a cleanup would be useful. Some of the APIs may still be needed but can be moved to an internal namespace. It's hard to satisfy the matrix of all use cases -- for example both InputStream and RandomAccessFile APIs are required > [C++] Add Result APIs to IPC module > -- > > Key: ARROW-7233 > URL: https://issues.apache.org/jira/browse/ARROW-7233 > Project: Apache Arrow > Issue Type: Sub-task > Components: C++ >Reporter: Micah Kornfield >Priority: Major > > src/arrow/ipc -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ARROW-5801) [CI] Dockerize (add to docker-compose) all Travis CI Linux tasks
[ https://issues.apache.org/jira/browse/ARROW-5801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney resolved ARROW-5801. - Resolution: Fixed The proper state for this issue is probably "Resolved' not "Closed" > [CI] Dockerize (add to docker-compose) all Travis CI Linux tasks > > > Key: ARROW-5801 > URL: https://issues.apache.org/jira/browse/ARROW-5801 > Project: Apache Arrow > Issue Type: Improvement > Components: Continuous Integration >Reporter: Wes McKinney >Assignee: Krisztian Szucs >Priority: Major > Fix For: 1.0.0 > > > Parent issue for migrating Travis CI tasks to Docker and decoupling from > Travis-specific environment variables and image state -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (ARROW-5801) [CI] Dockerize (add to docker-compose) all Travis CI Linux tasks
[ https://issues.apache.org/jira/browse/ARROW-5801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney reopened ARROW-5801: - Assignee: Krisztian Szucs > [CI] Dockerize (add to docker-compose) all Travis CI Linux tasks > > > Key: ARROW-5801 > URL: https://issues.apache.org/jira/browse/ARROW-5801 > Project: Apache Arrow > Issue Type: Improvement > Components: Continuous Integration >Reporter: Wes McKinney >Assignee: Krisztian Szucs >Priority: Major > Fix For: 1.0.0 > > > Parent issue for migrating Travis CI tasks to Docker and decoupling from > Travis-specific environment variables and image state -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ARROW-7318) [C#] TimestampArray serialization failure
[ https://issues.apache.org/jira/browse/ARROW-7318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated ARROW-7318: -- Labels: pull-request-available (was: ) > [C#] TimestampArray serialization failure > - > > Key: ARROW-7318 > URL: https://issues.apache.org/jira/browse/ARROW-7318 > Project: Apache Arrow > Issue Type: Bug > Components: C# >Affects Versions: 0.15.1 >Reporter: Chris Hutchinson >Assignee: Chris Hutchinson >Priority: Major > Labels: pull-request-available > Original Estimate: 2h > Remaining Estimate: 2h > > *TimestampArray* throws during deserialization from a record batch previously > serialized with the library. This appears to be because the timezone is not > serialized for *TimestampType* in the Flatbuffer serialization implementation. > This does not appear in the round-trip serialization tests because the > *TimestampArray* field is omitted from the test data, along with other > parametric types. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ARROW-7321) [CI][GLib] Failed to build with GLib warning
[
https://issues.apache.org/jira/browse/ARROW-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated ARROW-7321:
--
Labels: pull-request-available (was: )
> [CI][GLib] Failed to build with GLib warning
>
>
> Key: ARROW-7321
> URL: https://issues.apache.org/jira/browse/ARROW-7321
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Continuous Integration, GLib
>Reporter: Kouhei Sutou
>Assignee: Kouhei Sutou
>Priority: Major
> Labels: pull-request-available
>
> https://circleci.com/gh/ursa-labs/crossbow/5681
> {noformat}
> FAILED: arrow-glib/20f505c@@arrow-glib@sha/file.cpp.o
> ccache c++ -Iarrow-glib/20f505c@@arrow-glib@sha -Iarrow-glib
> -I../../arrow/c_glib/arrow-glib -I. -I../../arrow/c_glib/
> -I/usr/local/include -I/usr/include/glib-2.0
> -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -fdiagnostics-color=always -pipe
> -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -g
> -Werror -DARROW_NO_DEPRECATED_API -fPIC -pthread -MD -MQ
> 'arrow-glib/20f505c@@arrow-glib@sha/file.cpp.o' -MF
> 'arrow-glib/20f505c@@arrow-glib@sha/file.cpp.o.d' -o
> 'arrow-glib/20f505c@@arrow-glib@sha/file.cpp.o' -c
> ../../arrow/c_glib/arrow-glib/file.cpp
> In file included from /usr/include/glib-2.0/gobject/gobject.h:24:0,
> from /usr/include/glib-2.0/gobject/gbinding.h:29,
> from /usr/include/glib-2.0/glib-object.h:23,
> from ../../arrow/c_glib/arrow-glib/error.h:22,
> from ../../arrow/c_glib/arrow-glib/error.hpp:24,
> from ../../arrow/c_glib/arrow-glib/file.cpp:26:
> ../../arrow/c_glib/arrow-glib/file.cpp: In function 'GType
> garrow_file_get_type()':
> /usr/include/glib-2.0/gobject/gtype.h:219:50: error: '<<' in boolean context,
> did you mean '<' ? [-Werror=int-in-bool-context]
> #define G_TYPE_MAKE_FUNDAMENTAL(x) ((GType) ((x) <<
> G_TYPE_FUNDAMENTAL_SHIFT))
> ~^~~~
> /usr/include/glib-2.0/gobject/gtype.h:2015:11: note: in definition of macro
> '_G_DEFINE_INTERFACE_EXTENDED_BEGIN'
>if (TYPE_PREREQ) \
>^~~
> /usr/include/glib-2.0/gobject/gtype.h:1756:47: note: in expansion of macro
> 'G_DEFINE_INTERFACE_WITH_CODE'
> #define G_DEFINE_INTERFACE(TN, t_n, T_P)
> G_DEFINE_INTERFACE_WITH_CODE(TN, t_n, T_P, ;)
>^~~~
> ../../arrow/c_glib/arrow-glib/file.cpp:40:1: note: in expansion of macro
> 'G_DEFINE_INTERFACE'
> G_DEFINE_INTERFACE(GArrowFile,
> ^~
> /usr/include/glib-2.0/gobject/gtype.h:178:25: note: in expansion of macro
> 'G_TYPE_MAKE_FUNDAMENTAL'
> #define G_TYPE_OBJECT G_TYPE_MAKE_FUNDAMENTAL (20)
> ^~~
> ../../arrow/c_glib/arrow-glib/file.cpp:42:20: note: in expansion of macro
> 'G_TYPE_OBJECT'
> G_TYPE_OBJECT)
> ^
> cc1plus: all warnings being treated as errors
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ARROW-7321) [CI][GLib] Failed to build with GLib warning
Kouhei Sutou created ARROW-7321:
---
Summary: [CI][GLib] Failed to build with GLib warning
Key: ARROW-7321
URL: https://issues.apache.org/jira/browse/ARROW-7321
Project: Apache Arrow
Issue Type: Improvement
Components: Continuous Integration, GLib
Reporter: Kouhei Sutou
Assignee: Kouhei Sutou
https://circleci.com/gh/ursa-labs/crossbow/5681
{noformat}
FAILED: arrow-glib/20f505c@@arrow-glib@sha/file.cpp.o
ccache c++ -Iarrow-glib/20f505c@@arrow-glib@sha -Iarrow-glib
-I../../arrow/c_glib/arrow-glib -I. -I../../arrow/c_glib/ -I/usr/local/include
-I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include
-fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
-Wnon-virtual-dtor -std=c++11 -g -Werror -DARROW_NO_DEPRECATED_API -fPIC
-pthread -MD -MQ 'arrow-glib/20f505c@@arrow-glib@sha/file.cpp.o' -MF
'arrow-glib/20f505c@@arrow-glib@sha/file.cpp.o.d' -o
'arrow-glib/20f505c@@arrow-glib@sha/file.cpp.o' -c
../../arrow/c_glib/arrow-glib/file.cpp
In file included from /usr/include/glib-2.0/gobject/gobject.h:24:0,
from /usr/include/glib-2.0/gobject/gbinding.h:29,
from /usr/include/glib-2.0/glib-object.h:23,
from ../../arrow/c_glib/arrow-glib/error.h:22,
from ../../arrow/c_glib/arrow-glib/error.hpp:24,
from ../../arrow/c_glib/arrow-glib/file.cpp:26:
../../arrow/c_glib/arrow-glib/file.cpp: In function 'GType
garrow_file_get_type()':
/usr/include/glib-2.0/gobject/gtype.h:219:50: error: '<<' in boolean context,
did you mean '<' ? [-Werror=int-in-bool-context]
#define G_TYPE_MAKE_FUNDAMENTAL(x) ((GType) ((x) << G_TYPE_FUNDAMENTAL_SHIFT))
~^~~~
/usr/include/glib-2.0/gobject/gtype.h:2015:11: note: in definition of macro
'_G_DEFINE_INTERFACE_EXTENDED_BEGIN'
if (TYPE_PREREQ) \
^~~
/usr/include/glib-2.0/gobject/gtype.h:1756:47: note: in expansion of macro
'G_DEFINE_INTERFACE_WITH_CODE'
#define G_DEFINE_INTERFACE(TN, t_n, T_P) G_DEFINE_INTERFACE_WITH_CODE(TN,
t_n, T_P, ;)
^~~~
../../arrow/c_glib/arrow-glib/file.cpp:40:1: note: in expansion of macro
'G_DEFINE_INTERFACE'
G_DEFINE_INTERFACE(GArrowFile,
^~
/usr/include/glib-2.0/gobject/gtype.h:178:25: note: in expansion of macro
'G_TYPE_MAKE_FUNDAMENTAL'
#define G_TYPE_OBJECT G_TYPE_MAKE_FUNDAMENTAL (20)
^~~
../../arrow/c_glib/arrow-glib/file.cpp:42:20: note: in expansion of macro
'G_TYPE_OBJECT'
G_TYPE_OBJECT)
^
cc1plus: all warnings being treated as errors
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (ARROW-7307) [CI][GLib] Documentation isn't generated
[
https://issues.apache.org/jira/browse/ARROW-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kouhei Sutou resolved ARROW-7307.
-
Fix Version/s: 1.0.0
Resolution: Fixed
Issue resolved by pull request 5953
[https://github.com/apache/arrow/pull/5953]
> [CI][GLib] Documentation isn't generated
>
>
> Key: ARROW-7307
> URL: https://issues.apache.org/jira/browse/ARROW-7307
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Continuous Integration, GLib
>Reporter: Kouhei Sutou
>Assignee: Kouhei Sutou
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.0.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> https://github.com/apache/arrow/runs/332154195#step:5:12092
> {noformat}
> + rsync -a --delete '/usr/local/share/gtk-doc/html/*' /build/docs/c_glib
> rsync: change_dir "/usr/local/share/gtk-doc/html" failed: No such file or
> directory (2)
> rsync error: some files/attrs were not transferred (see previous errors)
> (code 23) at main.c(1196) [sender=3.1.2]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ARROW-7307) [CI][GLib] Documentation isn't generated
[
https://issues.apache.org/jira/browse/ARROW-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kouhei Sutou updated ARROW-7307:
Summary: [CI][GLib] Documentation isn't generated (was: [CI][GLib]
Document isn't generated)
> [CI][GLib] Documentation isn't generated
>
>
> Key: ARROW-7307
> URL: https://issues.apache.org/jira/browse/ARROW-7307
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Continuous Integration, GLib
>Reporter: Kouhei Sutou
>Assignee: Kouhei Sutou
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> https://github.com/apache/arrow/runs/332154195#step:5:12092
> {noformat}
> + rsync -a --delete '/usr/local/share/gtk-doc/html/*' /build/docs/c_glib
> rsync: change_dir "/usr/local/share/gtk-doc/html" failed: No such file or
> directory (2)
> rsync error: some files/attrs were not transferred (see previous errors)
> (code 23) at main.c(1196) [sender=3.1.2]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (ARROW-5523) [Python] [Packaging] Use HTTPS consistently for downloading dependencies
[
https://issues.apache.org/jira/browse/ARROW-5523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neal Richardson reassigned ARROW-5523:
--
Assignee: Neal Richardson
> [Python] [Packaging] Use HTTPS consistently for downloading dependencies
>
>
> Key: ARROW-5523
> URL: https://issues.apache.org/jira/browse/ARROW-5523
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Packaging, Python
>Affects Versions: 0.13.0
>Reporter: Antoine Pitrou
>Assignee: Neal Richardson
>Priority: Major
> Fix For: 1.0.0
>
>
> Some download URLs in the manylinux scripts use plain-text protocols:
> {code:bash}
> python/manylinux1/scripts/build_thrift.sh:wget
> http://archive.apache.org/dist/thrift/${THRIFT_VERSION}/thrift-${THRIFT_VERSION}.tar.gz
> python/manylinux1/scripts/build_python.sh:AUTOCONF_DOWNLOAD_URL=http://ftp.gnu.org/gnu/autoconf
> python/manylinux1/scripts/build_python.sh:AUTOMAKE_DOWNLOAD_URL=http://ftp.gnu.org/gnu/automake
> python/manylinux1/scripts/build_python.sh:LIBTOOL_DOWNLOAD_URL=http://ftp.gnu.org/gnu/libtool
> python/manylinux1/scripts/build_bison.sh:wget
> http://ftp.gnu.org/gnu/bison/bison-3.0.4.tar.gz
> python/manylinux1/scripts/build_re2.sh:curl -sL
> "http://github.com/google/re2/archive/${RE2_VERSION}.tar.gz; -o
> re2-${RE2_VERSION}.tar.gz
> python/manylinux1/scripts/build_llvm.sh:curl -sL
> http://releases.llvm.org/${LLVM_VERSION}/llvm-${LLVM_VERSION}.src.tar.xz -o
> llvm-${LLVM_VERSION}.src.tar.xz
> python/manylinux1/scripts/build_clang.sh:curl -sL
> http://releases.llvm.org/${LLVM_VERSION}/cfe-${LLVM_VERSION}.src.tar.xz -o
> cfe-${LLVM_VERSION}.src.tar.xz
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-5523) [Python] [Packaging] Use HTTPS consistently for downloading dependencies
[
https://issues.apache.org/jira/browse/ARROW-5523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988227#comment-16988227
]
Neal Richardson commented on ARROW-5523:
(y) will do
> [Python] [Packaging] Use HTTPS consistently for downloading dependencies
>
>
> Key: ARROW-5523
> URL: https://issues.apache.org/jira/browse/ARROW-5523
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Packaging, Python
>Affects Versions: 0.13.0
>Reporter: Antoine Pitrou
>Priority: Major
> Fix For: 1.0.0
>
>
> Some download URLs in the manylinux scripts use plain-text protocols:
> {code:bash}
> python/manylinux1/scripts/build_thrift.sh:wget
> http://archive.apache.org/dist/thrift/${THRIFT_VERSION}/thrift-${THRIFT_VERSION}.tar.gz
> python/manylinux1/scripts/build_python.sh:AUTOCONF_DOWNLOAD_URL=http://ftp.gnu.org/gnu/autoconf
> python/manylinux1/scripts/build_python.sh:AUTOMAKE_DOWNLOAD_URL=http://ftp.gnu.org/gnu/automake
> python/manylinux1/scripts/build_python.sh:LIBTOOL_DOWNLOAD_URL=http://ftp.gnu.org/gnu/libtool
> python/manylinux1/scripts/build_bison.sh:wget
> http://ftp.gnu.org/gnu/bison/bison-3.0.4.tar.gz
> python/manylinux1/scripts/build_re2.sh:curl -sL
> "http://github.com/google/re2/archive/${RE2_VERSION}.tar.gz; -o
> re2-${RE2_VERSION}.tar.gz
> python/manylinux1/scripts/build_llvm.sh:curl -sL
> http://releases.llvm.org/${LLVM_VERSION}/llvm-${LLVM_VERSION}.src.tar.xz -o
> llvm-${LLVM_VERSION}.src.tar.xz
> python/manylinux1/scripts/build_clang.sh:curl -sL
> http://releases.llvm.org/${LLVM_VERSION}/cfe-${LLVM_VERSION}.src.tar.xz -o
> cfe-${LLVM_VERSION}.src.tar.xz
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ARROW-7320) [C++] Target arrow-type-benchmark failed to be built on bullx Linux
[
https://issues.apache.org/jira/browse/ARROW-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antoine Pitrou updated ARROW-7320:
--
Affects Version/s: (was: 1.0.0)
> [C++] Target arrow-type-benchmark failed to be built on bullx Linux
> ---
>
> Key: ARROW-7320
> URL: https://issues.apache.org/jira/browse/ARROW-7320
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
> Environment: bullx Linux
>Reporter: Chengxin Ma
>Assignee: Chengxin Ma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> I was building Arrow on bullx Linux (a Linux distribution compatible with Red
> Hat Enterprise Linux).
> CMake options:
> {code}
> -DCMAKE_BUILD_TYPE=Debug
> -DARROW_FLIGHT=ON
> -DARROW_BUILD_BENCHMARKS=ON
> {code}
> {{make}} failed with the following error message:
> {code}
> Scanning dependencies of target arrow-type-benchmark
> [ 72%] Building CXX object
> src/arrow/CMakeFiles/arrow-type-benchmark.dir/type_benchmark.cc.o
> make[2]: *** No rule to make target
> `gbenchmark_ep/src/gbenchmark_ep-install/lib/libbenchmark_main.a', needed by
> `debug/arrow-type-benchmark'. Stop.
> make[1]: *** [src/arrow/CMakeFiles/arrow-type-benchmark.dir/all] Error 2
> make: *** [all] Error 2
> {code}
> This is due to the same reason as mentioned in [this
> commit|https://github.com/apache/arrow/pull/4246/commits/f6b0bc7f8dc56f02e2778752235e728b7623a9ee]:
> If {{-DCMAKE_INSTALL_LIBDIR=lib}} is not explicitly set,
> {{libbenchmark_main.a}} will be put in {{lib64}} instead of {{lib}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-5523) [Python] [Packaging] Use HTTPS consistently for downloading dependencies
[
https://issues.apache.org/jira/browse/ARROW-5523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988225#comment-16988225
]
Antoine Pitrou commented on ARROW-5523:
---
[~npr] If you're looking at the manylinux scripts you might want to tackle this
(easy?) issue.
> [Python] [Packaging] Use HTTPS consistently for downloading dependencies
>
>
> Key: ARROW-5523
> URL: https://issues.apache.org/jira/browse/ARROW-5523
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Packaging, Python
>Affects Versions: 0.13.0
>Reporter: Antoine Pitrou
>Priority: Major
> Fix For: 1.0.0
>
>
> Some download URLs in the manylinux scripts use plain-text protocols:
> {code:bash}
> python/manylinux1/scripts/build_thrift.sh:wget
> http://archive.apache.org/dist/thrift/${THRIFT_VERSION}/thrift-${THRIFT_VERSION}.tar.gz
> python/manylinux1/scripts/build_python.sh:AUTOCONF_DOWNLOAD_URL=http://ftp.gnu.org/gnu/autoconf
> python/manylinux1/scripts/build_python.sh:AUTOMAKE_DOWNLOAD_URL=http://ftp.gnu.org/gnu/automake
> python/manylinux1/scripts/build_python.sh:LIBTOOL_DOWNLOAD_URL=http://ftp.gnu.org/gnu/libtool
> python/manylinux1/scripts/build_bison.sh:wget
> http://ftp.gnu.org/gnu/bison/bison-3.0.4.tar.gz
> python/manylinux1/scripts/build_re2.sh:curl -sL
> "http://github.com/google/re2/archive/${RE2_VERSION}.tar.gz; -o
> re2-${RE2_VERSION}.tar.gz
> python/manylinux1/scripts/build_llvm.sh:curl -sL
> http://releases.llvm.org/${LLVM_VERSION}/llvm-${LLVM_VERSION}.src.tar.xz -o
> llvm-${LLVM_VERSION}.src.tar.xz
> python/manylinux1/scripts/build_clang.sh:curl -sL
> http://releases.llvm.org/${LLVM_VERSION}/cfe-${LLVM_VERSION}.src.tar.xz -o
> cfe-${LLVM_VERSION}.src.tar.xz
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-5766) [Python] Unpin jpype1 version
[ https://issues.apache.org/jira/browse/ARROW-5766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988223#comment-16988223 ] Antoine Pitrou commented on ARROW-5766: --- Hmm... I'm not sure we're testing jpype on CI anymore. [~uwe] you might want to take a look :-) > [Python] Unpin jpype1 version > - > > Key: ARROW-5766 > URL: https://issues.apache.org/jira/browse/ARROW-5766 > Project: Apache Arrow > Issue Type: Improvement > Components: Python >Reporter: Wes McKinney >Priority: Major > Fix For: 1.0.0 > > > According to the discussion in > https://github.com/conda-forge/jpype1-feedstock/issues/8 htere are some > changes that we must make to our code to stay on the released version of > jpype1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ARROW-2873) [Python] Micro-optimize scalar value instantiation
[ https://issues.apache.org/jira/browse/ARROW-2873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988222#comment-16988222 ] Antoine Pitrou commented on ARROW-2873: --- [~kszucs] Perhaps we can close this since you're re-doing the Scalar / Value hierarchy? > [Python] Micro-optimize scalar value instantiation > -- > > Key: ARROW-2873 > URL: https://issues.apache.org/jira/browse/ARROW-2873 > Project: Apache Arrow > Issue Type: Task > Components: Python >Reporter: Krisztian Szucs >Priority: Minor > Fix For: 1.0.0 > > > This lead to a 20% time increase in __getitem__: > https://pandas.pydata.org/speed/arrow/#array_ops.ScalarAccess.time_getitem > See conversation: > https://github.com/apache/arrow/commit/dc80a768c0a15e62998ccd32d8353d2035302cb6#r29746119 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ARROW-5679) [Python] Drop Python 3.5 from support matrix
[ https://issues.apache.org/jira/browse/ARROW-5679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988220#comment-16988220 ] Antoine Pitrou commented on ARROW-5679: --- I don't see Python 3.5 in any CI entry, so I think we've dropped it unknowingly? :-) > [Python] Drop Python 3.5 from support matrix > > > Key: ARROW-5679 > URL: https://issues.apache.org/jira/browse/ARROW-5679 > Project: Apache Arrow > Issue Type: Improvement > Components: Python >Reporter: Wes McKinney >Priority: Major > Fix For: 1.0.0 > > > We probably need to maintain Python 3.5 on Linux and macOS for the time > being, but we may want to drop it for Windows since conda-forge isn't > supporting Python 3.5 anymore, so maintaining wheels for Python 3.5 will come > with extra cost -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ARROW-6654) [Python] Consider adding some user-friendly conveniences to Filesystem API
[
https://issues.apache.org/jira/browse/ARROW-6654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988219#comment-16988219
]
Antoine Pitrou commented on ARROW-6654:
---
In this case we should simply allow passing a single path.
> [Python] Consider adding some user-friendly conveniences to Filesystem API
> --
>
> Key: ARROW-6654
> URL: https://issues.apache.org/jira/browse/ARROW-6654
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
>Reporter: Wes McKinney
>Priority: Major
> Fix For: 1.0.0
>
>
> For example:
> {code}
> In [12]: lfs.get_target_stats('/home/wesm')
>
>
> ---
> TypeError Traceback (most recent call last)
> in
> > 1 lfs.get_target_stats('/home/wesm')
> ~/code/arrow/python/pyarrow/_fs.pyx in
> pyarrow._fs.FileSystem.get_target_stats()
> 239 check_status(self.fs.GetTargetStats(paths, ))
> 240 else:
> --> 241 raise TypeError('Must pass either paths or a Selector')
> 242
> 243 return [FileStats.wrap(stat) for stat in stats]
> TypeError: Must pass either paths or a Selector
> {code}
> Some conveniences like {{listdir}} might be kind to the user
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (ARROW-2600) [Python] Add additional LocalFileSystem filesystem methods
[
https://issues.apache.org/jira/browse/ARROW-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antoine Pitrou closed ARROW-2600.
-
Fix Version/s: (was: 1.0.0)
Resolution: Won't Fix
The old {{pyarrow.filesystem}} API is deprecated, we won't be changing it
anymore.
> [Python] Add additional LocalFileSystem filesystem methods
> --
>
> Key: ARROW-2600
> URL: https://issues.apache.org/jira/browse/ARROW-2600
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
>Reporter: Alex Hagerman
>Priority: Minor
> Labels: filesystem, pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Related to https://issues.apache.org/jira/browse/ARROW-1319 I noticed the
> methods Martin listed are also not part of the LocalFileSystem class.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (ARROW-5806) [CI] Dockerize (add to docker-compose) Integration tests Travis CI entry
[ https://issues.apache.org/jira/browse/ARROW-5806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antoine Pitrou closed ARROW-5806. - Assignee: Krisztian Szucs (was: Wes McKinney) Resolution: Duplicate > [CI] Dockerize (add to docker-compose) Integration tests Travis CI entry > > > Key: ARROW-5806 > URL: https://issues.apache.org/jira/browse/ARROW-5806 > Project: Apache Arrow > Issue Type: Improvement > Components: Continuous Integration >Reporter: Wes McKinney >Assignee: Krisztian Szucs >Priority: Major > Fix For: 1.0.0 > > > https://github.com/apache/arrow/blob/master/.travis.yml#L228 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (ARROW-5807) [JS] Dockerize NodeJS Travis CI entry
[ https://issues.apache.org/jira/browse/ARROW-5807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antoine Pitrou closed ARROW-5807. - Resolution: Duplicate > [JS] Dockerize NodeJS Travis CI entry > - > > Key: ARROW-5807 > URL: https://issues.apache.org/jira/browse/ARROW-5807 > Project: Apache Arrow > Issue Type: Improvement > Components: JavaScript >Reporter: Wes McKinney >Priority: Major > Fix For: 1.0.0 > > > Add to docker-compose, use in Travis CI instead of current scripts -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (ARROW-5810) [Go] Dockerize Travis CI Go build
[ https://issues.apache.org/jira/browse/ARROW-5810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antoine Pitrou closed ARROW-5810. - Resolution: Duplicate > [Go] Dockerize Travis CI Go build > - > > Key: ARROW-5810 > URL: https://issues.apache.org/jira/browse/ARROW-5810 > Project: Apache Arrow > Issue Type: Improvement > Components: Continuous Integration, Go >Reporter: Wes McKinney >Assignee: Krisztian Szucs >Priority: Major > Fix For: 1.0.0 > > > https://github.com/apache/arrow/blob/master/.travis.yml#L330 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (ARROW-5810) [Go] Dockerize Travis CI Go build
[ https://issues.apache.org/jira/browse/ARROW-5810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antoine Pitrou reassigned ARROW-5810: - Assignee: Krisztian Szucs > [Go] Dockerize Travis CI Go build > - > > Key: ARROW-5810 > URL: https://issues.apache.org/jira/browse/ARROW-5810 > Project: Apache Arrow > Issue Type: Improvement > Components: Continuous Integration, Go >Reporter: Wes McKinney >Assignee: Krisztian Szucs >Priority: Major > Fix For: 1.0.0 > > > https://github.com/apache/arrow/blob/master/.travis.yml#L330 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (ARROW-5808) [GLib][Ruby] Dockerize (add to docker-compose) current GLib + Ruby Travis CI entry
[ https://issues.apache.org/jira/browse/ARROW-5808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antoine Pitrou closed ARROW-5808. - Assignee: Krisztian Szucs (was: Yosuke Shiro) Resolution: Duplicate > [GLib][Ruby] Dockerize (add to docker-compose) current GLib + Ruby Travis CI > entry > -- > > Key: ARROW-5808 > URL: https://issues.apache.org/jira/browse/ARROW-5808 > Project: Apache Arrow > Issue Type: Improvement > Components: GLib, Ruby >Reporter: Wes McKinney >Assignee: Krisztian Szucs >Priority: Major > Fix For: 1.0.0 > > > Add to docker-compose and use in Travis CI > https://github.com/apache/arrow/blob/master/.travis.yml#L265 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (ARROW-5801) [CI] Dockerize (add to docker-compose) all Travis CI Linux tasks
[ https://issues.apache.org/jira/browse/ARROW-5801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antoine Pitrou closed ARROW-5801. - Resolution: Done > [CI] Dockerize (add to docker-compose) all Travis CI Linux tasks > > > Key: ARROW-5801 > URL: https://issues.apache.org/jira/browse/ARROW-5801 > Project: Apache Arrow > Issue Type: Improvement > Components: Continuous Integration >Reporter: Wes McKinney >Priority: Major > Fix For: 1.0.0 > > > Parent issue for migrating Travis CI tasks to Docker and decoupling from > Travis-specific environment variables and image state -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (ARROW-7320) [C++] Target arrow-type-benchmark failed to be built on bullx Linux
[
https://issues.apache.org/jira/browse/ARROW-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chengxin Ma reassigned ARROW-7320:
--
Assignee: Chengxin Ma
> [C++] Target arrow-type-benchmark failed to be built on bullx Linux
> ---
>
> Key: ARROW-7320
> URL: https://issues.apache.org/jira/browse/ARROW-7320
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
>Affects Versions: 1.0.0
> Environment: bullx Linux
>Reporter: Chengxin Ma
>Assignee: Chengxin Ma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I was building Arrow on bullx Linux (a Linux distribution compatible with Red
> Hat Enterprise Linux).
> CMake options:
> {code}
> -DCMAKE_BUILD_TYPE=Debug
> -DARROW_FLIGHT=ON
> -DARROW_BUILD_BENCHMARKS=ON
> {code}
> {{make}} failed with the following error message:
> {code}
> Scanning dependencies of target arrow-type-benchmark
> [ 72%] Building CXX object
> src/arrow/CMakeFiles/arrow-type-benchmark.dir/type_benchmark.cc.o
> make[2]: *** No rule to make target
> `gbenchmark_ep/src/gbenchmark_ep-install/lib/libbenchmark_main.a', needed by
> `debug/arrow-type-benchmark'. Stop.
> make[1]: *** [src/arrow/CMakeFiles/arrow-type-benchmark.dir/all] Error 2
> make: *** [all] Error 2
> {code}
> This is due to the same reason as mentioned in [this
> commit|https://github.com/apache/arrow/pull/4246/commits/f6b0bc7f8dc56f02e2778752235e728b7623a9ee]:
> If {{-DCMAKE_INSTALL_LIBDIR=lib}} is not explicitly set,
> {{libbenchmark_main.a}} will be put in {{lib64}} instead of {{lib}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-6463) [C++][Python] Rename arrow::fs::Selector to FileSelector
[ https://issues.apache.org/jira/browse/ARROW-6463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988211#comment-16988211 ] Antoine Pitrou commented on ARROW-6463: --- [~kszucs] it would be better to do this before 1.0.0. > [C++][Python] Rename arrow::fs::Selector to FileSelector > > > Key: ARROW-6463 > URL: https://issues.apache.org/jira/browse/ARROW-6463 > Project: Apache Arrow > Issue Type: Improvement > Components: C++, Python >Reporter: Krisztian Szucs >Assignee: Krisztian Szucs >Priority: Major > Labels: filesystem > Fix For: 1.0.0 > > > In both the C++ implementation and the python binding. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ARROW-6547) [C++] valgrind errors in diff-test
[ https://issues.apache.org/jira/browse/ARROW-6547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antoine Pitrou updated ARROW-6547: -- Summary: [C++] valgrind errors in diff-test (was: [C++] valgrind errors in arrow-ipc-read-write-test) > [C++] valgrind errors in diff-test > -- > > Key: ARROW-6547 > URL: https://issues.apache.org/jira/browse/ARROW-6547 > Project: Apache Arrow > Issue Type: Bug > Components: C++ >Reporter: Wes McKinney >Priority: Minor > Fix For: 1.0.0 > > > Not sure when these crept in but I encountered when looking into a segfault > in a build today > https://gist.github.com/wesm/b388dda4f0e2e38a8aa77dfc9bd91914 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ARROW-6547) [C++] valgrind errors in arrow-ipc-read-write-test
[ https://issues.apache.org/jira/browse/ARROW-6547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antoine Pitrou updated ARROW-6547: -- Priority: Minor (was: Major) > [C++] valgrind errors in arrow-ipc-read-write-test > -- > > Key: ARROW-6547 > URL: https://issues.apache.org/jira/browse/ARROW-6547 > Project: Apache Arrow > Issue Type: Bug > Components: C++ >Reporter: Wes McKinney >Priority: Minor > Fix For: 1.0.0 > > > Not sure when these crept in but I encountered when looking into a segfault > in a build today > https://gist.github.com/wesm/b388dda4f0e2e38a8aa77dfc9bd91914 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (ARROW-7120) [C++][CI] Add .ccache to the docker-compose volume mounts
[ https://issues.apache.org/jira/browse/ARROW-7120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antoine Pitrou closed ARROW-7120. - Resolution: Duplicate > [C++][CI] Add .ccache to the docker-compose volume mounts > - > > Key: ARROW-7120 > URL: https://issues.apache.org/jira/browse/ARROW-7120 > Project: Apache Arrow > Issue Type: Improvement > Components: C++, Continuous Integration >Reporter: Krisztian Szucs >Priority: Major > Fix For: 1.0.0 > > > This should speed up the compilation steps. Although I'm not sure whether we > should define it as a "docker named volume" or a standard volume mounted from > the host. The former one provides better isolation whereas the latter one > makes the integration with GitHub actions and it's cache plugin easier. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ARROW-7049) [C++] warnings building on mingw-w64
[
https://issues.apache.org/jira/browse/ARROW-7049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988209#comment-16988209
]
Antoine Pitrou commented on ARROW-7049:
---
[~jeroenooms] Do you want to provide a PR for this?
> [C++] warnings building on mingw-w64
>
>
> Key: ARROW-7049
> URL: https://issues.apache.org/jira/browse/ARROW-7049
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
>Affects Versions: 0.15.1
>Reporter: Jeroen
>Priority: Minor
> Fix For: 1.0.0
>
>
> Two warnings when building libarrow 0.15.1 on mingw-w64:
> {code}
> [ 2%] Running thrift compiler on parquet.thrift
> [WARNING:C:/msys64/home/mingw-packages/mingw-w64-arrow/src/apache-arrow-0.15.1/cpp/src/parquet/parquet.thrift:297]
> The "byte" type is a compatibility alias for "i8". Use "i8" to emphasize the
> signedness of this type.
> {code}
> And later:
> {code}
> 81%] Building CXX object
> src/parquet/CMakeFiles/parquet_static.dir/column_reader.cc.obj
> C:/msys64/home/mingw-packages/mingw-w64-arrow/src/apache-arrow-0.15.1/cpp/src/parquet/arrow/writer.cc:
> In member function 'virtual arrow::Status
> parquet::arrow::FileWriterImpl::WriteColumnChunk(const
> std::shared_ptr&, int64_t, int64_t)':
> C:/msys64/home/mingw-packages/mingw-w64-arrow/src/apache-arrow-0.15.1/cpp/src/parquet/arrow/writer.cc:79:41:
> warning: 'schema_field' may be used uninitialized in this function
> [-Wmaybe-uninitialized]
> schema_manifest_(schema_manifest) {}
> ^
> C:/msys64/home/mingw-packages/mingw-w64-arrow/src/apache-arrow-0.15.1/cpp/src/parquet/arrow/writer.cc:466:24:
> note: 'schema_field' was declared here
> const SchemaField* schema_field;
> {code}
> Maybe CI with `CXXFLAGS += -Werror` ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (ARROW-7319) [C++] Refactor Iterator to yield Result
[ https://issues.apache.org/jira/browse/ARROW-7319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Kietzman closed ARROW-7319. --- Resolution: Fixed > [C++] Refactor Iterator to yield Result > - > > Key: ARROW-7319 > URL: https://issues.apache.org/jira/browse/ARROW-7319 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Affects Versions: 0.15.1 >Reporter: Ben Kietzman >Assignee: Ben Kietzman >Priority: Minor > Fix For: 1.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ARROW-6786) [C++] arrow-dataset-file-parquet-test is slow
[
https://issues.apache.org/jira/browse/ARROW-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antoine Pitrou resolved ARROW-6786.
---
Resolution: Done
Seems to have been fixed in ARROW-6951
> [C++] arrow-dataset-file-parquet-test is slow
> -
>
> Key: ARROW-6786
> URL: https://issues.apache.org/jira/browse/ARROW-6786
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
>Reporter: Antoine Pitrou
>Priority: Major
> Fix For: 1.0.0
>
>
> It takes 15 seconds in debug mode (probably more with ASAN / UBSAN /etc.) to
> run 2 tests that simply iterated through a generated in-memory dataset:
> {code}
> $ ./build-test/debug/arrow-dataset-file-parquet-test
> Running main() from
> /home/conda/feedstock_root/build_artifacts/gtest_1551008230529/work/googletest/src/gtest_main.cc
> [==] Running 2 tests from 1 test case.
> [--] Global test environment set-up.
> [--] 2 tests from TestParquetFileFormat
> [ RUN ] TestParquetFileFormat.ScanRecordBatchReader
> [ OK ] TestParquetFileFormat.ScanRecordBatchReader (7338 ms)
> [ RUN ] TestParquetFileFormat.Inspect
> [ OK ] TestParquetFileFormat.Inspect (6222 ms)
> [--] 2 tests from TestParquetFileFormat (13560 ms total)
> [--] Global test environment tear-down
> [==] 2 tests from 1 test case ran. (13560 ms total)
> [ PASSED ] 2 tests.
> {code}
> Unless it is stressing something in particular, the number of repetitions or
> the batch size can probably be reduced dramatically.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-6929) [C++] ValidateArray is out of sync with the ListArray IPC specification
[ https://issues.apache.org/jira/browse/ARROW-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988204#comment-16988204 ] Antoine Pitrou commented on ARROW-6929: --- [~emkornfi...@gmail.com] Can you check the current implementation? It's been significantly overhauled. > [C++] ValidateArray is out of sync with the ListArray IPC specification > --- > > Key: ARROW-6929 > URL: https://issues.apache.org/jira/browse/ARROW-6929 > Project: Apache Arrow > Issue Type: Bug > Components: C++ >Reporter: Micah Kornfield >Priority: Major > Fix For: 1.0.0 > > > * It appears to check that null values take zero space > * It still checks for a begin offset of 0 if the array isn't sliced > (technically this doesn't seem necessary and it could be non-zero even if the > array wasn't sliced.) > * I think it also fails if an array is sliced to truncate it since it should > compare length to data_extent instead of last_offset. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ARROW-7077) [C++] Unsupported Dict->T cast crashes instead of returning error
[
https://issues.apache.org/jira/browse/ARROW-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated ARROW-7077:
--
Labels: pull-request-available (was: )
> [C++] Unsupported Dict->T cast crashes instead of returning error
> -
>
> Key: ARROW-7077
> URL: https://issues.apache.org/jira/browse/ARROW-7077
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++, C++ - Compute
>Affects Versions: 0.15.1
>Reporter: Antoine Pitrou
>Assignee: Antoine Pitrou
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.0.0
>
>
> {code:python}
> >>> arr = pa.array(["foo", "bar"])
> >>>
> >>>
> >>> arr.dictionary_encode().cast('binary')
> >>>
> >>>
> F1106 16:53:22.411165 28503 cast.cc:919] Check failed:
> values_type.Equals(*output->type) Dictionary type: string target type: binary
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (ARROW-7077) [C++] Unsupported Dict->T cast crashes instead of returning error
[
https://issues.apache.org/jira/browse/ARROW-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antoine Pitrou reassigned ARROW-7077:
-
Assignee: Antoine Pitrou
> [C++] Unsupported Dict->T cast crashes instead of returning error
> -
>
> Key: ARROW-7077
> URL: https://issues.apache.org/jira/browse/ARROW-7077
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++, C++ - Compute
>Affects Versions: 0.15.1
>Reporter: Antoine Pitrou
>Assignee: Antoine Pitrou
>Priority: Major
> Fix For: 1.0.0
>
>
> {code:python}
> >>> arr = pa.array(["foo", "bar"])
> >>>
> >>>
> >>> arr.dictionary_encode().cast('binary')
> >>>
> >>>
> F1106 16:53:22.411165 28503 cast.cc:919] Check failed:
> values_type.Equals(*output->type) Dictionary type: string target type: binary
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ARROW-7320) [C++] Target arrow-type-benchmark failed to be built on bullx Linux
[
https://issues.apache.org/jira/browse/ARROW-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated ARROW-7320:
--
Labels: pull-request-available (was: )
> [C++] Target arrow-type-benchmark failed to be built on bullx Linux
> ---
>
> Key: ARROW-7320
> URL: https://issues.apache.org/jira/browse/ARROW-7320
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
>Affects Versions: 1.0.0
> Environment: bullx Linux
>Reporter: Chengxin Ma
>Priority: Major
> Labels: pull-request-available
>
> I was building Arrow on bullx Linux (a Linux distribution compatible with Red
> Hat Enterprise Linux).
> CMake options:
> {code}
> -DCMAKE_BUILD_TYPE=Debug
> -DARROW_FLIGHT=ON
> -DARROW_BUILD_BENCHMARKS=ON
> {code}
> {{make}} failed with the following error message:
> {code}
> Scanning dependencies of target arrow-type-benchmark
> [ 72%] Building CXX object
> src/arrow/CMakeFiles/arrow-type-benchmark.dir/type_benchmark.cc.o
> make[2]: *** No rule to make target
> `gbenchmark_ep/src/gbenchmark_ep-install/lib/libbenchmark_main.a', needed by
> `debug/arrow-type-benchmark'. Stop.
> make[1]: *** [src/arrow/CMakeFiles/arrow-type-benchmark.dir/all] Error 2
> make: *** [all] Error 2
> {code}
> This is due to the same reason as mentioned in [this
> commit|https://github.com/apache/arrow/pull/4246/commits/f6b0bc7f8dc56f02e2778752235e728b7623a9ee]:
> If {{-DCMAKE_INSTALL_LIBDIR=lib}} is not explicitly set,
> {{libbenchmark_main.a}} will be put in {{lib64}} instead of {{lib}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ARROW-7320) Target arrow-type-benchmark failed to be built on bullx Linux
Chengxin Ma created ARROW-7320:
--
Summary: Target arrow-type-benchmark failed to be built on bullx
Linux
Key: ARROW-7320
URL: https://issues.apache.org/jira/browse/ARROW-7320
Project: Apache Arrow
Issue Type: Bug
Components: C++
Affects Versions: 1.0.0
Environment: bullx Linux
Reporter: Chengxin Ma
I was building Arrow on bullx Linux (a Linux distribution compatible with Red
Hat Enterprise Linux).
CMake options:
{code}
-DCMAKE_BUILD_TYPE=Debug
-DARROW_FLIGHT=ON
-DARROW_BUILD_BENCHMARKS=ON
{code}
{{make}} failed with the following error message:
{code}
Scanning dependencies of target arrow-type-benchmark
[ 72%] Building CXX object
src/arrow/CMakeFiles/arrow-type-benchmark.dir/type_benchmark.cc.o
make[2]: *** No rule to make target
`gbenchmark_ep/src/gbenchmark_ep-install/lib/libbenchmark_main.a', needed by
`debug/arrow-type-benchmark'. Stop.
make[1]: *** [src/arrow/CMakeFiles/arrow-type-benchmark.dir/all] Error 2
make: *** [all] Error 2
{code}
This is due to the same reason as mentioned in [this
commit|https://github.com/apache/arrow/pull/4246/commits/f6b0bc7f8dc56f02e2778752235e728b7623a9ee]:
If {{-DCMAKE_INSTALL_LIBDIR=lib}} is not explicitly set,
{{libbenchmark_main.a}} will be put in {{lib64}} instead of {{lib}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ARROW-7320) [C++] Target arrow-type-benchmark failed to be built on bullx Linux
[
https://issues.apache.org/jira/browse/ARROW-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chengxin Ma updated ARROW-7320:
---
Summary: [C++] Target arrow-type-benchmark failed to be built on bullx
Linux (was: Target arrow-type-benchmark failed to be built on bullx Linux)
> [C++] Target arrow-type-benchmark failed to be built on bullx Linux
> ---
>
> Key: ARROW-7320
> URL: https://issues.apache.org/jira/browse/ARROW-7320
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
>Affects Versions: 1.0.0
> Environment: bullx Linux
>Reporter: Chengxin Ma
>Priority: Major
>
> I was building Arrow on bullx Linux (a Linux distribution compatible with Red
> Hat Enterprise Linux).
> CMake options:
> {code}
> -DCMAKE_BUILD_TYPE=Debug
> -DARROW_FLIGHT=ON
> -DARROW_BUILD_BENCHMARKS=ON
> {code}
> {{make}} failed with the following error message:
> {code}
> Scanning dependencies of target arrow-type-benchmark
> [ 72%] Building CXX object
> src/arrow/CMakeFiles/arrow-type-benchmark.dir/type_benchmark.cc.o
> make[2]: *** No rule to make target
> `gbenchmark_ep/src/gbenchmark_ep-install/lib/libbenchmark_main.a', needed by
> `debug/arrow-type-benchmark'. Stop.
> make[1]: *** [src/arrow/CMakeFiles/arrow-type-benchmark.dir/all] Error 2
> make: *** [all] Error 2
> {code}
> This is due to the same reason as mentioned in [this
> commit|https://github.com/apache/arrow/pull/4246/commits/f6b0bc7f8dc56f02e2778752235e728b7623a9ee]:
> If {{-DCMAKE_INSTALL_LIBDIR=lib}} is not explicitly set,
> {{libbenchmark_main.a}} will be put in {{lib64}} instead of {{lib}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ARROW-7319) [C++] Refactor Iterator to yield Result
Ben Kietzman created ARROW-7319: --- Summary: [C++] Refactor Iterator to yield Result Key: ARROW-7319 URL: https://issues.apache.org/jira/browse/ARROW-7319 Project: Apache Arrow Issue Type: Improvement Components: C++ Affects Versions: 0.15.1 Reporter: Ben Kietzman Assignee: Ben Kietzman Fix For: 1.0.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ARROW-7318) [C#] TimestampArray serialization failure
Chris Hutchinson created ARROW-7318: --- Summary: [C#] TimestampArray serialization failure Key: ARROW-7318 URL: https://issues.apache.org/jira/browse/ARROW-7318 Project: Apache Arrow Issue Type: Bug Components: C# Affects Versions: 0.15.1 Reporter: Chris Hutchinson Assignee: Chris Hutchinson *TimestampArray* throws during deserialization from a record batch previously serialized with the library. This appears to be because the timezone is not serialized for *TimestampType* in the Flatbuffer serialization implementation. This does not appear in the round-trip serialization tests because the *TimestampArray* field is omitted from the test data, along with other parametric types. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ARROW-7256) [C++] Remove ARROW_DEFAULT_MEMORY_POOL option
[ https://issues.apache.org/jira/browse/ARROW-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988157#comment-16988157 ] Antoine Pitrou commented on ARROW-7256: --- [~emkornfield] What do you think? > [C++] Remove ARROW_DEFAULT_MEMORY_POOL option > - > > Key: ARROW-7256 > URL: https://issues.apache.org/jira/browse/ARROW-7256 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Reporter: Wes McKinney >Priority: Major > Fix For: 1.0.0 > > > As mentioned elsewhere in a JIRA I recall, we aren't testing adequately the > CMake option for "no default memory pool", so it would either be better to > require explicit memory pools or pass the default, rather than having a > build-time option to set whether a default will be passed -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ARROW-7233) [C++] Add Result APIs to IPC module
[ https://issues.apache.org/jira/browse/ARROW-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988148#comment-16988148 ] Antoine Pitrou commented on ARROW-7233: --- cc [~wesm]. Perhaps this is an opportunity to clean up the existing APIs? They are a bit plethoric. > [C++] Add Result APIs to IPC module > -- > > Key: ARROW-7233 > URL: https://issues.apache.org/jira/browse/ARROW-7233 > Project: Apache Arrow > Issue Type: Sub-task > Components: C++ >Reporter: Micah Kornfield >Priority: Major > > src/arrow/ipc -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (ARROW-7237) [C++] Add Result to APIs to arrow/json
[ https://issues.apache.org/jira/browse/ARROW-7237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antoine Pitrou reassigned ARROW-7237: - Assignee: Ben Kietzman > [C++] Add Result to APIs to arrow/json > - > > Key: ARROW-7237 > URL: https://issues.apache.org/jira/browse/ARROW-7237 > Project: Apache Arrow > Issue Type: Sub-task > Components: C++ >Reporter: Micah Kornfield >Assignee: Ben Kietzman >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ARROW-7274) [C++] Add Result APIs to Decimal class
[ https://issues.apache.org/jira/browse/ARROW-7274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated ARROW-7274: -- Labels: pull-request-available (was: ) > [C++] Add Result APIs to Decimal class > - > > Key: ARROW-7274 > URL: https://issues.apache.org/jira/browse/ARROW-7274 > Project: Apache Arrow > Issue Type: Sub-task > Components: C++ >Reporter: Micah Kornfield >Assignee: Kenta Murata >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ARROW-7317) [C++] Migrate Iterator API to Result
Antoine Pitrou created ARROW-7317: - Summary: [C++] Migrate Iterator API to Result Key: ARROW-7317 URL: https://issues.apache.org/jira/browse/ARROW-7317 Project: Apache Arrow Issue Type: Sub-task Components: C++ Reporter: Micah Kornfield Assignee: Antoine Pitrou -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (ARROW-7317) [C++] Migrate Iterator API to Result
[ https://issues.apache.org/jira/browse/ARROW-7317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antoine Pitrou reassigned ARROW-7317: - Assignee: Ben Kietzman (was: Antoine Pitrou) > [C++] Migrate Iterator API to Result > --- > > Key: ARROW-7317 > URL: https://issues.apache.org/jira/browse/ARROW-7317 > Project: Apache Arrow > Issue Type: Sub-task > Components: C++ >Reporter: Micah Kornfield >Assignee: Ben Kietzman >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ARROW-6611) [C++] Make ARROW_JSON=OFF the default
[ https://issues.apache.org/jira/browse/ARROW-6611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antoine Pitrou resolved ARROW-6611. --- Resolution: Duplicate > [C++] Make ARROW_JSON=OFF the default > - > > Key: ARROW-6611 > URL: https://issues.apache.org/jira/browse/ARROW-6611 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Reporter: Wes McKinney >Priority: Major > Fix For: 1.0.0 > > > The JSON-based functionality is only needed for > * Integration tests > * Unit tests > * JSON scanning > If the user opts in to unit tests or integration tests, then we can flip it > on, but I think that the user should opt in when building libarrow -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (ARROW-5804) [C++] Dockerize C++ CI job with conda-forge toolchain, code coverage from Travis CI
[ https://issues.apache.org/jira/browse/ARROW-5804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antoine Pitrou closed ARROW-5804. - Resolution: Done We have left Travis and are now using docker-compose consistently for Linux CI builds. > [C++] Dockerize C++ CI job with conda-forge toolchain, code coverage from > Travis CI > --- > > Key: ARROW-5804 > URL: https://issues.apache.org/jira/browse/ARROW-5804 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Reporter: Wes McKinney >Priority: Major > Fix For: 1.0.0 > > > https://github.com/apache/arrow/blob/master/.travis.yml#L87 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (ARROW-4091) [C++] Curate default list of CSV null spellings
[ https://issues.apache.org/jira/browse/ARROW-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antoine Pitrou closed ARROW-4091. - Resolution: Not A Problem Nobody complained about the current defaults, so closing. > [C++] Curate default list of CSV null spellings > --- > > Key: ARROW-4091 > URL: https://issues.apache.org/jira/browse/ARROW-4091 > Project: Apache Arrow > Issue Type: Wish > Components: C++ >Affects Versions: 0.11.1 >Reporter: Antoine Pitrou >Priority: Major > Labels: csv > Fix For: 1.0.0 > > > "NaN" is not null in Arrow-land (at least not for float columns?). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ARROW-7316) compile error due to incomplete type for unique_ptr
Danny Kim created ARROW-7316:
Summary: compile error due to incomplete type for unique_ptr
Key: ARROW-7316
URL: https://issues.apache.org/jira/browse/ARROW-7316
Project: Apache Arrow
Issue Type: Bug
Components: C++
Affects Versions: 0.15.1
Environment: WSL, conda, arrow version 0.15
Reporter: Danny Kim
Hi,
I am getting following compile error from Arrow c++
{code:java}
Warning: Can't read registry to find the necessary compiler setting
Make sure that Python modules winreg, win32api or win32con are installed.C
compiler: /home/danny/miniconda3/envs/DEV/bin/x86_64-conda_cos6-linux-gnu-cc
-Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall
-Wstrict-prototypes -march=nocona -mtune=haswell -ftree-vectorize -fPIC
-fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell
-ftree-vectorize -fPIC -fstack-protector-strong -fno-plt
-O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC
-fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -DNDEBUG
-D_FORTIFY_SOURCE=2 -O2 -fPIC
compile options: '-DBUILTIN_PARQUET_READER -I.
-I/home/danny/miniconda3/envs/DEV/include
-I/home/danny/miniconda3/envs/DEV/include/python3.7m -c'
extra options: '-std=c++11 -g0 -O3'
x86_64-conda_cos6-linux-gnu-cc: bodo/io/_parquet.cpp
x86_64-conda_cos6-linux-gnu-cc: bodo/io/_parquet_reader.cpp
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC
but not for C++
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC
but not for C++
In file included from
/home/danny/miniconda3/envs/DEV/x86_64-conda_cos6-linux-gnu/include/c++/7.3.0/memory:80:0,
from /home/danny/miniconda3/envs/DEV/include/parquet/arrow/reader.h:22,
from
bodo/io/_parquet.cpp:13:/home/danny/miniconda3/envs/DEV/x86_64-conda_cos6-linux-gnu/include/c++/7.3.0/bits/unique_ptr.h:
In instantiation of 'void std::default_delete<_Tp>::operator()(_Tp*) const
[with _Tp = arrow::RecordBatchReader]':
/home/danny/miniconda3/envs/DEV/x86_64-conda_cos6-linux-gnu/include/c++/7.3.0/bits/unique_ptr.h:268:17:
required from 'std::unique_ptr<_Tp, _Dp>::~unique_ptr() [with _Tp =
arrow::RecordBatchReader; _Dp =
std::default_delete]'/home/danny/miniconda3/envs/DEV/include/parquet/arrow/reader.h:161:49:
required from here
/home/danny/miniconda3/envs/DEV/x86_64-conda_cos6-linux-gnu/include/c++/7.3.0/bits/unique_ptr.h:76:22:
error: invalid application of 'sizeof' to incomplete type
'arrow::RecordBatchReader'
static_assert(sizeof(_Tp)>0, ^In file included from
/home/danny/miniconda3/envs/DEV/x86_64-conda_cos6-linux-gnu/include/c++/7.3.0/bits/shared_ptr.h:52:0,
from
/home/danny/miniconda3/envs/DEV/x86_64-conda_cos6-linux-gnu/include/c++/7.3.0/memory:81,
from /home/danny/miniconda3/envs/DEV/include/parquet/arrow/reader.h:22,
from bodo/io/_parquet.cpp:13:
/home/danny/miniconda3/envs/DEV/x86_64-conda_cos6-linux-gnu/include/c++/7.3.0/bits/shared_ptr_base.h:
In instantiation of 'std::__shared_ptr<_Tp, _Lp>::__shared_ptr(_Yp*)
[with _Yp = arrow::RecordBatchReader; = void; _Tp =
arrow::RecordBatchReader; __gnu_cxx::_Lock_policy _Lp =
(__gnu_cxx::_Lock_policy)2]':
/home/danny/miniconda3/envs/DEV/x86_64-conda_cos6-linux-gnu/include/c++/7.3.0/bits/shared_ptr_base.h:1243:4:
required from 'std::__shared_ptr<_Tp, _Lp>::_SafeConv<_Yp>
std::__shared_ptr<_Tp, _Lp>::reset(_Yp*) [with _Yp = arrow::RecordBatchReader;
_Tp = arrow::RecordBatchReader; __gnu_cxx::_Lock_policy _Lp =
(__gnu_cxx::_Lock_policy)2; std::__shared_ptr<_Tp, _Lp>::_SafeConv<_Yp> = void]'
/home/danny/miniconda3/envs/DEV/include/parquet/arrow/reader.h:164:29: required
from here
/home/danny/miniconda3/envs/DEV/x86_64-conda_cos6-linux-gnu/include/c++/7.3.0/bits/shared_ptr_base.h:1082:25:
error: invalid application of 'sizeof' to incomplete type
'arrow::RecordBatchReader'
static_assert( sizeof(_Yp) > 0, "incomplete type" );
^
error: Command
"/home/danny/miniconda3/envs/DEV/bin/x86_64-conda_cos6-linux-gnu-cc
-Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall
-Wstrict-prototypes -march=nocona -mtune=haswell -ftree-vectorize -fPIC
-fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell
-ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe
-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong
-fno-plt -O2 -ffunction-sections -pipe -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -fPIC
-DBUILTIN_PARQUET_READER -I. -I/home/danny/miniconda3/envs/DEV/include
-I/home/danny/miniconda3/envs/DEV/include/python3.7m -c bodo/io/_parquet.cpp -o
build/temp.linux-x86_64-3.7/bodo/io/_parquet.o -std=c++11 -g0 -O3" failed with
exit status 1{code}
The actual error in above entire error message is
{code:java}
/home/danny/miniconda3/envs/DEV/include/parquet/arrow/reader.h:161:49: required
from here
[jira] [Resolved] (ARROW-5508) [C++] Create reusable Iterator interface
[
https://issues.apache.org/jira/browse/ARROW-5508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antoine Pitrou resolved ARROW-5508.
---
Resolution: Done
> [C++] Create reusable Iterator interface
>
>
> Key: ARROW-5508
> URL: https://issues.apache.org/jira/browse/ARROW-5508
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
>Reporter: Wes McKinney
>Priority: Major
> Fix For: 1.0.0
>
>
> We have various iterator-like classes. I envision a reusable interface like
> {code}
> template
> class Iterator {
> public:
> virtual ~Iterator() = default;
> virtual Status Next(T* out) = 0;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-5508) [C++] Create reusable Iterator interface
[
https://issues.apache.org/jira/browse/ARROW-5508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988129#comment-16988129
]
Antoine Pitrou commented on ARROW-5508:
---
Is there something left to do here? It seems the {{Iterator}} interface and
its {{Visit}} method are sufficient.
> [C++] Create reusable Iterator interface
>
>
> Key: ARROW-5508
> URL: https://issues.apache.org/jira/browse/ARROW-5508
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
>Reporter: Wes McKinney
>Priority: Major
> Fix For: 1.0.0
>
>
> We have various iterator-like classes. I envision a reusable interface like
> {code}
> template
> class Iterator {
> public:
> virtual ~Iterator() = default;
> virtual Status Next(T* out) = 0;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ARROW-7315) [Python] Remove unused cmake flag from wheel scripts
[
https://issues.apache.org/jira/browse/ARROW-7315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated ARROW-7315:
--
Labels: pull-request-available (was: )
> [Python] Remove unused cmake flag from wheel scripts
>
>
> Key: ARROW-7315
> URL: https://issues.apache.org/jira/browse/ARROW-7315
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Reporter: Neal Richardson
>Assignee: Neal Richardson
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.0.0
>
>
> Saw this while building a wheel locally:
> {code}
> CMake Warning:
> Manually-specified variables were not used by the project:
> OPENSSL_USE_STATIC_LIBS
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-823) [Python] Devise a means to serialize arrays of arbitrary Python objects in Arrow IPC messages
[ https://issues.apache.org/jira/browse/ARROW-823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988126#comment-16988126 ] Antoine Pitrou commented on ARROW-823: -- cc [~jorisvandenbossche] > [Python] Devise a means to serialize arrays of arbitrary Python objects in > Arrow IPC messages > - > > Key: ARROW-823 > URL: https://issues.apache.org/jira/browse/ARROW-823 > Project: Apache Arrow > Issue Type: New Feature > Components: Python >Reporter: Wes McKinney >Priority: Major > Fix For: 1.0.0 > > > Practically speaking, this would involve a "custom" logical type that is > "pyobject", represented physically as an array of 64-bit pointers. On > serialization, this would need to be converted to a BinaryArray containing > pickled objects as binary values > At the moment, we don't yet have the machinery to deal with "custom" types > where the in-memory representation is different from the on-wire > representation. This would be a useful use case to work through the design > issues > Interestingly, if done properly, this would enable other Arrow > implementations to manipulate (filter, etc.) serialized Python objects as > binary blobs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ARROW-6904) [Python] Implement MapArray and MapType
[ https://issues.apache.org/jira/browse/ARROW-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antoine Pitrou resolved ARROW-6904. --- Resolution: Fixed Issue resolved by pull request 5774 [https://github.com/apache/arrow/pull/5774] > [Python] Implement MapArray and MapType > --- > > Key: ARROW-6904 > URL: https://issues.apache.org/jira/browse/ARROW-6904 > Project: Apache Arrow > Issue Type: Improvement > Components: Python >Reporter: Bryan Cutler >Assignee: Bryan Cutler >Priority: Major > Labels: pull-request-available > Fix For: 1.0.0 > > Time Spent: 11h 10m > Remaining Estimate: 0h > > Map arrays are already added to C++, need to expose them in the Python API > also -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ARROW-7282) [Python] IO functions should raise FileNotFoundError when appropriate
[
https://issues.apache.org/jira/browse/ARROW-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated ARROW-7282:
--
Labels: exception-reporting pull-request-available pyarrow (was:
exception-reporting pyarrow)
> [Python] IO functions should raise FileNotFoundError when appropriate
> -
>
> Key: ARROW-7282
> URL: https://issues.apache.org/jira/browse/ARROW-7282
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++, Python
>Affects Versions: 0.15.1
> Environment: Arch Linux, Python 3.7
>Reporter: Scott Gigante
>Assignee: Antoine Pitrou
>Priority: Minor
> Labels: exception-reporting, pull-request-available, pyarrow
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> I get the following error when trying to open a file that does not exist.
> ```
> pyarrow.lib.ArrowIOError: Failed to open local file 'filename', error: No
> such file or directory
> ```
> In my opinion, this particular error should also subclass from Python
> FileNotFoundError. It currently only inherits from IOError (which is a
> superclass of FileNotFoundError).
> ```
> >>> import pyarrow
> >>> try:
> ... raise pyarrow.lib.ArrowIOError
> ... except IOError:
> ... pass
> ...
> >>> try:
> ... raise pyarrow.lib.ArrowIOError
> ... except FileNotFoundError:
> ... print('caught')
> ...
> Traceback (most recent call last):
> File "", line 2, in
> pyarrow.lib.ArrowIOError
> >>> try:
> ... raise pyarrow.lib.ArrowIOError
> ... except IOError:
> ... print('caught')
> ...
> caught
> ```
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ARROW-7282) [Python] IO functions should raise FileNotFoundError when appropriate
[
https://issues.apache.org/jira/browse/ARROW-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antoine Pitrou updated ARROW-7282:
--
Summary: [Python] IO functions should raise FileNotFoundError when
appropriate (was: [Python] IO functions should raise FileNotFoundError)
> [Python] IO functions should raise FileNotFoundError when appropriate
> -
>
> Key: ARROW-7282
> URL: https://issues.apache.org/jira/browse/ARROW-7282
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++, Python
>Affects Versions: 0.15.1
> Environment: Arch Linux, Python 3.7
>Reporter: Scott Gigante
>Assignee: Antoine Pitrou
>Priority: Minor
> Labels: exception-reporting, pyarrow
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> I get the following error when trying to open a file that does not exist.
> ```
> pyarrow.lib.ArrowIOError: Failed to open local file 'filename', error: No
> such file or directory
> ```
> In my opinion, this particular error should also subclass from Python
> FileNotFoundError. It currently only inherits from IOError (which is a
> superclass of FileNotFoundError).
> ```
> >>> import pyarrow
> >>> try:
> ... raise pyarrow.lib.ArrowIOError
> ... except IOError:
> ... pass
> ...
> >>> try:
> ... raise pyarrow.lib.ArrowIOError
> ... except FileNotFoundError:
> ... print('caught')
> ...
> Traceback (most recent call last):
> File "", line 2, in
> pyarrow.lib.ArrowIOError
> >>> try:
> ... raise pyarrow.lib.ArrowIOError
> ... except IOError:
> ... print('caught')
> ...
> caught
> ```
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ARROW-7282) [Python] let ArrowIOError subclass from FileNotFoundError when appropriate
[
https://issues.apache.org/jira/browse/ARROW-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antoine Pitrou updated ARROW-7282:
--
Component/s: C++
> [Python] let ArrowIOError subclass from FileNotFoundError when appropriate
> --
>
> Key: ARROW-7282
> URL: https://issues.apache.org/jira/browse/ARROW-7282
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++, Python
>Affects Versions: 0.15.1
> Environment: Arch Linux, Python 3.7
>Reporter: Scott Gigante
>Assignee: Antoine Pitrou
>Priority: Minor
> Labels: exception-reporting, pyarrow
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> I get the following error when trying to open a file that does not exist.
> ```
> pyarrow.lib.ArrowIOError: Failed to open local file 'filename', error: No
> such file or directory
> ```
> In my opinion, this particular error should also subclass from Python
> FileNotFoundError. It currently only inherits from IOError (which is a
> superclass of FileNotFoundError).
> ```
> >>> import pyarrow
> >>> try:
> ... raise pyarrow.lib.ArrowIOError
> ... except IOError:
> ... pass
> ...
> >>> try:
> ... raise pyarrow.lib.ArrowIOError
> ... except FileNotFoundError:
> ... print('caught')
> ...
> Traceback (most recent call last):
> File "", line 2, in
> pyarrow.lib.ArrowIOError
> >>> try:
> ... raise pyarrow.lib.ArrowIOError
> ... except IOError:
> ... print('caught')
> ...
> caught
> ```
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ARROW-7282) [Python] IO functions should raise FileNotFoundError
[
https://issues.apache.org/jira/browse/ARROW-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antoine Pitrou updated ARROW-7282:
--
Summary: [Python] IO functions should raise FileNotFoundError (was:
[Python] let ArrowIOError subclass from FileNotFoundError when appropriate)
> [Python] IO functions should raise FileNotFoundError
>
>
> Key: ARROW-7282
> URL: https://issues.apache.org/jira/browse/ARROW-7282
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++, Python
>Affects Versions: 0.15.1
> Environment: Arch Linux, Python 3.7
>Reporter: Scott Gigante
>Assignee: Antoine Pitrou
>Priority: Minor
> Labels: exception-reporting, pyarrow
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> I get the following error when trying to open a file that does not exist.
> ```
> pyarrow.lib.ArrowIOError: Failed to open local file 'filename', error: No
> such file or directory
> ```
> In my opinion, this particular error should also subclass from Python
> FileNotFoundError. It currently only inherits from IOError (which is a
> superclass of FileNotFoundError).
> ```
> >>> import pyarrow
> >>> try:
> ... raise pyarrow.lib.ArrowIOError
> ... except IOError:
> ... pass
> ...
> >>> try:
> ... raise pyarrow.lib.ArrowIOError
> ... except FileNotFoundError:
> ... print('caught')
> ...
> Traceback (most recent call last):
> File "", line 2, in
> pyarrow.lib.ArrowIOError
> >>> try:
> ... raise pyarrow.lib.ArrowIOError
> ... except IOError:
> ... print('caught')
> ...
> caught
> ```
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-7314) [Python] Compiler warning in pyarrow
[
https://issues.apache.org/jira/browse/ARROW-7314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988097#comment-16988097
]
Wes McKinney commented on ARROW-7314:
-
It can be difficult to get warning-free Cython builds, if is this is not a
coding error then we may suppress this warning
> [Python] Compiler warning in pyarrow
>
>
> Key: ARROW-7314
> URL: https://issues.apache.org/jira/browse/ARROW-7314
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Reporter: Neal Richardson
>Priority: Minor
> Fix For: 1.0.0
>
>
> Saw this while building a wheel locally
> {code}
> -- Running cmake --build for pyarrow
> cmake --build . --config release -- -j3
> [23/24] Building CXX object CMakeFiles/lib.dir/lib.cpp.o
> lib.cpp: In function ‘PyObject* __pyx_pf_7pyarrow_3lib_90union(PyObject*,
> PyObject*, PyObject*, PyObject*)’:
> lib.cpp:37184:29: warning: comparison of integer expressions of different
> signedness: ‘Py_ssize_t’ {aka ‘long int’} and
> ‘std::vector >::size_type’ {aka ‘long unsigned
> int’} [-Wsign-compare]
> __pyx_t_2 = ((__pyx_t_6 != __pyx_v_c_fields.size()) != 0);
>~~^~
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ARROW-7315) [Python] Remove unused cmake flag from wheel scripts
Neal Richardson created ARROW-7315:
--
Summary: [Python] Remove unused cmake flag from wheel scripts
Key: ARROW-7315
URL: https://issues.apache.org/jira/browse/ARROW-7315
Project: Apache Arrow
Issue Type: Bug
Components: Python
Reporter: Neal Richardson
Assignee: Neal Richardson
Fix For: 1.0.0
Saw this while building a wheel locally:
{code}
CMake Warning:
Manually-specified variables were not used by the project:
OPENSSL_USE_STATIC_LIBS
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ARROW-7314) [Python] Compiler warning in pyarrow
Neal Richardson created ARROW-7314:
--
Summary: [Python] Compiler warning in pyarrow
Key: ARROW-7314
URL: https://issues.apache.org/jira/browse/ARROW-7314
Project: Apache Arrow
Issue Type: Bug
Components: Python
Reporter: Neal Richardson
Fix For: 1.0.0
Saw this while building a wheel locally
{code}
-- Running cmake --build for pyarrow
cmake --build . --config release -- -j3
[23/24] Building CXX object CMakeFiles/lib.dir/lib.cpp.o
lib.cpp: In function ‘PyObject* __pyx_pf_7pyarrow_3lib_90union(PyObject*,
PyObject*, PyObject*, PyObject*)’:
lib.cpp:37184:29: warning: comparison of integer expressions of different
signedness: ‘Py_ssize_t’ {aka ‘long int’} and
‘std::vector >::size_type’ {aka ‘long unsigned
int’} [-Wsign-compare]
__pyx_t_2 = ((__pyx_t_6 != __pyx_v_c_fields.size()) != 0);
~~^~
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (ARROW-6508) [C++] Add Tensor and SparseTensor factory function with validations
[ https://issues.apache.org/jira/browse/ARROW-6508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antoine Pitrou resolved ARROW-6508. --- Resolution: Fixed Issue resolved by pull request 5862 [https://github.com/apache/arrow/pull/5862] > [C++] Add Tensor and SparseTensor factory function with validations > --- > > Key: ARROW-6508 > URL: https://issues.apache.org/jira/browse/ARROW-6508 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Reporter: Kenta Murata >Assignee: Kenta Murata >Priority: Major > Labels: pull-request-available > Fix For: 1.0.0 > > Time Spent: 5h > Remaining Estimate: 0h > > Now Tensor and SparseTensor only have their constructors, but not factory > functions that validate the parameters. > We need such factory functions for creating Tensor and SparseTensor from > parameters given from the external source. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ARROW-7279) [C++] Rename UnionArray::type_ids to UnionArray::type_codes
[
https://issues.apache.org/jira/browse/ARROW-7279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney resolved ARROW-7279.
-
Fix Version/s: 1.0.0
Resolution: Fixed
Issue resolved by pull request 5934
[https://github.com/apache/arrow/pull/5934]
> [C++] Rename UnionArray::type_ids to UnionArray::type_codes
> ---
>
> Key: ARROW-7279
> URL: https://issues.apache.org/jira/browse/ARROW-7279
> Project: Apache Arrow
> Issue Type: Wish
> Components: C++
>Affects Versions: 1.0.0
>Reporter: Antoine Pitrou
>Assignee: Antoine Pitrou
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.0.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> This would be consistent with {{UnionType::type_codes}}. Furthermore,
> "type_id" already means something else in the C++ API, so it would be less
> confusing as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ARROW-7309) [Python] Support HDFS federation viewfs://
[ https://issues.apache.org/jira/browse/ARROW-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Höring updated ARROW-7309: - Summary: [Python] Support HDFS federation viewfs:// (was: [Python] Support HDFS federation viewfs:// in resolve_filesystem_and_path) > [Python] Support HDFS federation viewfs:// > -- > > Key: ARROW-7309 > URL: https://issues.apache.org/jira/browse/ARROW-7309 > Project: Apache Arrow > Issue Type: Improvement > Components: Python >Affects Versions: 0.15.1 >Reporter: Fabian Höring >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > - Add viewfs support to pyarrow.filesystem.resolve_filesystem_and_path > - libhdfs already supports injecting the scheme and will automatically > resolve federation in > fs = FileSystem#get(URI, conf, ugi) > - works with Hadoop 2/3 > see: > https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L770 > https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L637 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (ARROW-7282) [Python] let ArrowIOError subclass from FileNotFoundError when appropriate
[
https://issues.apache.org/jira/browse/ARROW-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antoine Pitrou reassigned ARROW-7282:
-
Assignee: Antoine Pitrou
> [Python] let ArrowIOError subclass from FileNotFoundError when appropriate
> --
>
> Key: ARROW-7282
> URL: https://issues.apache.org/jira/browse/ARROW-7282
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.15.1
> Environment: Arch Linux, Python 3.7
>Reporter: Scott Gigante
>Assignee: Antoine Pitrou
>Priority: Minor
> Labels: exception-reporting, pyarrow
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> I get the following error when trying to open a file that does not exist.
> ```
> pyarrow.lib.ArrowIOError: Failed to open local file 'filename', error: No
> such file or directory
> ```
> In my opinion, this particular error should also subclass from Python
> FileNotFoundError. It currently only inherits from IOError (which is a
> superclass of FileNotFoundError).
> ```
> >>> import pyarrow
> >>> try:
> ... raise pyarrow.lib.ArrowIOError
> ... except IOError:
> ... pass
> ...
> >>> try:
> ... raise pyarrow.lib.ArrowIOError
> ... except FileNotFoundError:
> ... print('caught')
> ...
> Traceback (most recent call last):
> File "", line 2, in
> pyarrow.lib.ArrowIOError
> >>> try:
> ... raise pyarrow.lib.ArrowIOError
> ... except IOError:
> ... print('caught')
> ...
> caught
> ```
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-7309) [Python] Support HDFS federation viewfs:// in resolve_filesystem_and_path
[ https://issues.apache.org/jira/browse/ARROW-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987968#comment-16987968 ] Fabian Höring commented on ARROW-7309: -- FileSystemFromUri doesn't support viewfs either. https://github.com/apache/arrow/blob/master/cpp/src/arrow/filesystem/filesystem.cc#L358 > [Python] Support HDFS federation viewfs:// in resolve_filesystem_and_path > - > > Key: ARROW-7309 > URL: https://issues.apache.org/jira/browse/ARROW-7309 > Project: Apache Arrow > Issue Type: Improvement > Components: Python >Affects Versions: 0.15.1 >Reporter: Fabian Höring >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > - Add viewfs support to pyarrow.filesystem.resolve_filesystem_and_path > - libhdfs already supports injecting the scheme and will automatically > resolve federation in > fs = FileSystem#get(URI, conf, ugi) > - works with Hadoop 2/3 > see: > https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L770 > https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L637 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ARROW-7313) [C++] Add function for retrieving a scalar from an array slot
Ben Kietzman created ARROW-7313:
---
Summary: [C++] Add function for retrieving a scalar from an array
slot
Key: ARROW-7313
URL: https://issues.apache.org/jira/browse/ARROW-7313
Project: Apache Arrow
Issue Type: Improvement
Components: C++
Affects Versions: 0.15.1
Reporter: Ben Kietzman
Assignee: Ben Kietzman
Fix For: 1.0.0
It'd be useful to construct scalar values given an array and an index.
{code}
/* static */ std::shared_ptr Scalar::FromArray(const Array&, int64_t);
{code}
Since this is much less efficient than unboxing the entire array and accessing
its buffers directly, it should not be used in hot loops.
[~kszucs] [~fsaintjacques]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-7304) clang-tidy diagnostics not emitted for most headers
[
https://issues.apache.org/jira/browse/ARROW-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987957#comment-16987957
]
Ben Kietzman commented on ARROW-7304:
-
It might be useful to handle this by making {{.clang-tidy-ignore}} (which is
currently unreferenced) the default {{lint_exclusions}} file for
{{run_clang_tidy.py}}. It could duplicate the existing lint exclusions and add
any other files which should be ignored. Alternatively if lint_exclusions is
sufficiently exclusive then the HeaderFilterRegex field and
{{.clang-tidy-ignore}} can both be deleted.
> clang-tidy diagnostics not emitted for most headers
> ---
>
> Key: ARROW-7304
> URL: https://issues.apache.org/jira/browse/ARROW-7304
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++, CI
>Affects Versions: 0.15.1
>Reporter: Elvis Stansvik
>Priority: Minor
>
> The {{HeaderFilterRegex}} in {{.clang-tidy}} is written
> {code}
> HeaderFilterRegex:
> '^(.*codegen.*|.*_generated.*|.*windows_compatibility.h|.*pyarrow_api.h|.*pyarrow_lib.h|.*python/config.h|.*python/platform.h|.*thirdparty/ae/.*|.*vendored/.*|.*RcppExports.cpp.*|)$'
> {code}
> as if it was an exclusion filter, but {{HeaderFilterRegex}} is in fact an
> inclusion mechanism. So clang-tidy diagnostics are not emitted for I guess
> most of the headers in Arrow.
> See
> [https://github.com/apache/arrow/commit/72b553147e4bd47e100fbfd58ed49041561b7bc4#r36225046]
> which is where I came across this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-7310) [Python] Expose HDFS implementation for pyarrow.fs
[
https://issues.apache.org/jira/browse/ARROW-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987923#comment-16987923
]
Krisztian Szucs commented on ARROW-7310:
Because of the optional dependency I suppose we should expose it under
pyarrow.hdfs, although it'd be nice to be available from pyarrow.fs.
> [Python] Expose HDFS implementation for pyarrow.fs
> --
>
> Key: ARROW-7310
> URL: https://issues.apache.org/jira/browse/ARROW-7310
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
>Reporter: Antoine Pitrou
>Assignee: Krisztian Szucs
>Priority: Major
>
> There's a C++ implementation for HDFS (see {{arrow/filesystem/hdfs.h}}) but
> it's not exposed from Python.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (ARROW-7310) [Python] Expose HDFS implementation for pyarrow.fs
[
https://issues.apache.org/jira/browse/ARROW-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Szucs reassigned ARROW-7310:
--
Assignee: Krisztian Szucs
> [Python] Expose HDFS implementation for pyarrow.fs
> --
>
> Key: ARROW-7310
> URL: https://issues.apache.org/jira/browse/ARROW-7310
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
>Reporter: Antoine Pitrou
>Assignee: Krisztian Szucs
>Priority: Major
>
> There's a C++ implementation for HDFS (see {{arrow/filesystem/hdfs.h}}) but
> it's not exposed from Python.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ARROW-7312) [Rust] ArrowError should implement std::error:Error
[ https://issues.apache.org/jira/browse/ARROW-7312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated ARROW-7312: -- Labels: pull-request-available (was: ) > [Rust] ArrowError should implement std::error:Error > --- > > Key: ARROW-7312 > URL: https://issues.apache.org/jira/browse/ARROW-7312 > Project: Apache Arrow > Issue Type: Improvement >Reporter: Renjie Liu >Assignee: Renjie Liu >Priority: Major > Labels: pull-request-available > > ArrowError should implement this trait so that other crates can handle error > from this crate more friendly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ARROW-6890) [Rust] [Parquet] ArrowReader fails with seg fault
[ https://issues.apache.org/jira/browse/ARROW-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987902#comment-16987902 ] Andy Grove commented on ARROW-6890: --- Yes, thanks [~liurenjie1024] I hope to get to it this weekend > [Rust] [Parquet] ArrowReader fails with seg fault > - > > Key: ARROW-6890 > URL: https://issues.apache.org/jira/browse/ARROW-6890 > Project: Apache Arrow > Issue Type: Bug > Components: Rust >Affects Versions: 1.0.0 >Reporter: Andy Grove >Assignee: Renjie Liu >Priority: Major > Fix For: 1.0.0 > > > ArrowReader fails with seg fault when trying to read an unsupported type, > like Utf8. We should have it return an Err instead of causing a segmentation > fault. > > See [https://github.com/apache/arrow/pull/5641] for a reproducible test. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ARROW-7312) [Rust] ArrowError should implement std::error:Error
Renjie Liu created ARROW-7312: - Summary: [Rust] ArrowError should implement std::error:Error Key: ARROW-7312 URL: https://issues.apache.org/jira/browse/ARROW-7312 Project: Apache Arrow Issue Type: Improvement Reporter: Renjie Liu Assignee: Renjie Liu ArrowError should implement this trait so that other crates can handle error from this crate more friendly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ARROW-6890) [Rust] [Parquet] ArrowReader fails with seg fault
[ https://issues.apache.org/jira/browse/ARROW-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987900#comment-16987900 ] Renjie Liu commented on ARROW-6890: --- [~andygrove] Are you going to retry with new version of arrow reader? > [Rust] [Parquet] ArrowReader fails with seg fault > - > > Key: ARROW-6890 > URL: https://issues.apache.org/jira/browse/ARROW-6890 > Project: Apache Arrow > Issue Type: Bug > Components: Rust >Affects Versions: 1.0.0 >Reporter: Andy Grove >Assignee: Renjie Liu >Priority: Major > Fix For: 1.0.0 > > > ArrowReader fails with seg fault when trying to read an unsupported type, > like Utf8. We should have it return an Err instead of causing a segmentation > fault. > > See [https://github.com/apache/arrow/pull/5641] for a reproducible test. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ARROW-7309) [Python] Support HDFS federation viewfs:// in resolve_filesystem_and_path
[ https://issues.apache.org/jira/browse/ARROW-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987872#comment-16987872 ] Antoine Pitrou commented on ARROW-7309: --- HDFS should be relatively easy to expose (similar to S3). See ARROW-7310. Ditto for URI resolving, since it just needs to wrap the corresponding C++ API. Note that the C++ API may need to be expanded to support more URI schemes and parameters. See ARROW-7311. > [Python] Support HDFS federation viewfs:// in resolve_filesystem_and_path > - > > Key: ARROW-7309 > URL: https://issues.apache.org/jira/browse/ARROW-7309 > Project: Apache Arrow > Issue Type: Improvement > Components: Python >Affects Versions: 0.15.1 >Reporter: Fabian Höring >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > - Add viewfs support to pyarrow.filesystem.resolve_filesystem_and_path > - libhdfs already supports injecting the scheme and will automatically > resolve federation in > fs = FileSystem#get(URI, conf, ugi) > - works with Hadoop 2/3 > see: > https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L770 > https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L637 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ARROW-7309) [Python] Support HDFS federation viewfs:// in resolve_filesystem_and_path
[ https://issues.apache.org/jira/browse/ARROW-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987868#comment-16987868 ] Fabian Höring commented on ARROW-7309: -- [~apitrou] I had a look at the new implementations. I get the idea. So what I need is the new hdfs wrapper and then a Python wrapper resolve_filesystem_and_path that exposes all those filesystems based on the path. The registry idea from [fsspec|https://github.com/intake/filesystem_spec/blob/master/fsspec/registry.py] is nice or just if blocks based on the scheme (as it is handled now) Also some internal caching would be nice (for hdfs filesystems, we have many different namenodes) When will hdfs be exposed in Python with the new format ? I could do a proposal for the fs resolver. But basically it would just be - move resolve_filesystem_and_path to a new module - expose new python objects based on the scheme - create the fs - add some internal caching > [Python] Support HDFS federation viewfs:// in resolve_filesystem_and_path > - > > Key: ARROW-7309 > URL: https://issues.apache.org/jira/browse/ARROW-7309 > Project: Apache Arrow > Issue Type: Improvement > Components: Python >Affects Versions: 0.15.1 >Reporter: Fabian Höring >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > - Add viewfs support to pyarrow.filesystem.resolve_filesystem_and_path > - libhdfs already supports injecting the scheme and will automatically > resolve federation in > fs = FileSystem#get(URI, conf, ugi) > - works with Hadoop 2/3 > see: > https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L770 > https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L637 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (ARROW-7309) [Python] Support HDFS federation viewfs:// in resolve_filesystem_and_path
[
https://issues.apache.org/jira/browse/ARROW-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987853#comment-16987853
]
Fabian Höring edited comment on ARROW-7309 at 12/4/19 1:29 PM:
---
That, imo, is the only way to properly handle viewfs,
I also had a look a fsspec and exactly this is also missing.
Something like `fs = get_cached_fs_from_path("viewfs://root/user/f.horing/..")`
I don't mind where to implement this here or in fsspec. I just would like to
avoid doing this in my code.
was (Author: fhoering):
That, imo, is the only way to properly handle viewfs,
I also had a look a fsspec and exactly this is also missing.
Sth like `fs = get_cached_fs_from_path("viewfs://root/useR/f.horing/..")`
I don't mind where to implement this here or in fsspec. I just would like to
avoid doing this in my code.
> [Python] Support HDFS federation viewfs:// in resolve_filesystem_and_path
> -
>
> Key: ARROW-7309
> URL: https://issues.apache.org/jira/browse/ARROW-7309
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
>Affects Versions: 0.15.1
>Reporter: Fabian Höring
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> - Add viewfs support to pyarrow.filesystem.resolve_filesystem_and_path
> - libhdfs already supports injecting the scheme and will automatically
> resolve federation in
> fs = FileSystem#get(URI, conf, ugi)
> - works with Hadoop 2/3
> see:
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L770
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L637
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (ARROW-7309) [Python] Support HDFS federation viewfs:// in resolve_filesystem_and_path
[
https://issues.apache.org/jira/browse/ARROW-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987853#comment-16987853
]
Fabian Höring edited comment on ARROW-7309 at 12/4/19 1:20 PM:
---
That, imo, is the only way to properly handle viewfs,
I also had a look a fsspec and exactly this is also missing.
Sth like `fs = get_cached_fs_from_path("viewfs://root/useR/f.horing/..")`
I don't mind where to implement this here or in fsspec. I just would like to
avoid doing this in my code.
was (Author: fhoering):
That, imo, is the only way to properly handle viewfs,
I also had a look a fsspec and exactly this is also missing.
Sth like `fs = fs.get_cache_fs_from_path("viewfs://root/useR/f.horing/..")`
I don't mind where to implement this here or in fsspec. I just would like to
avoid doing this in my code.
> [Python] Support HDFS federation viewfs:// in resolve_filesystem_and_path
> -
>
> Key: ARROW-7309
> URL: https://issues.apache.org/jira/browse/ARROW-7309
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
>Affects Versions: 0.15.1
>Reporter: Fabian Höring
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> - Add viewfs support to pyarrow.filesystem.resolve_filesystem_and_path
> - libhdfs already supports injecting the scheme and will automatically
> resolve federation in
> fs = FileSystem#get(URI, conf, ugi)
> - works with Hadoop 2/3
> see:
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L770
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L637
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (ARROW-7309) [Python] Support HDFS federation viewfs:// in resolve_filesystem_and_path
[
https://issues.apache.org/jira/browse/ARROW-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987853#comment-16987853
]
Fabian Höring edited comment on ARROW-7309 at 12/4/19 1:17 PM:
---
That, imo, is the only way to properly handle viewfs,
I also had a look a fsspec and exactly this is also missing.
Sth like `fs = fs.get_cache_fs_from_path("viewfs://root/useR/f.horing/..")`
I don't mind where to implement this here or in fsspec. I just would like to
avoid doing this in my code.
was (Author: fhoering):
That, imo, is the only way to properly handle viewfs,
I also had a look a fsspec and exactly this is also missing.
Sth like `fs = fs.get_cache_fs_from_path("viewfs://root/useR/f.horing/..")`
I don't mind where to implement this here or in fsspec. I just would like to
avoid doign this in my code.
> [Python] Support HDFS federation viewfs:// in resolve_filesystem_and_path
> -
>
> Key: ARROW-7309
> URL: https://issues.apache.org/jira/browse/ARROW-7309
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
>Affects Versions: 0.15.1
>Reporter: Fabian Höring
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> - Add viewfs support to pyarrow.filesystem.resolve_filesystem_and_path
> - libhdfs already supports injecting the scheme and will automatically
> resolve federation in
> fs = FileSystem#get(URI, conf, ugi)
> - works with Hadoop 2/3
> see:
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L770
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L637
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-7309) [Python] Support HDFS federation viewfs:// in resolve_filesystem_and_path
[
https://issues.apache.org/jira/browse/ARROW-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987853#comment-16987853
]
Fabian Höring commented on ARROW-7309:
--
That, imo, is the only way to properly handle viewfs,
I also had a look a fsspec and exactly this is also missing.
Sth like `fs = fs.get_cache_fs_from_path("viewfs://root/useR/f.horing/..")`
I don't mind where to implement this here or in fsspec. I just would like to
avoid doign this in my code.
> [Python] Support HDFS federation viewfs:// in resolve_filesystem_and_path
> -
>
> Key: ARROW-7309
> URL: https://issues.apache.org/jira/browse/ARROW-7309
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
>Affects Versions: 0.15.1
>Reporter: Fabian Höring
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> - Add viewfs support to pyarrow.filesystem.resolve_filesystem_and_path
> - libhdfs already supports injecting the scheme and will automatically
> resolve federation in
> fs = FileSystem#get(URI, conf, ugi)
> - works with Hadoop 2/3
> see:
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L770
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L637
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-7311) [Python] Return filesystem and path from URI
[
https://issues.apache.org/jira/browse/ARROW-7311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987850#comment-16987850
]
Antoine Pitrou commented on ARROW-7311:
---
[~kszucs]
> [Python] Return filesystem and path from URI
>
>
> Key: ARROW-7311
> URL: https://issues.apache.org/jira/browse/ARROW-7311
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
>Reporter: Antoine Pitrou
>Priority: Major
>
> The C++ API {{FileSystemFromUri}} should be exposed in Python.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ARROW-7311) [Python] Return filesystem and path from URI
Antoine Pitrou created ARROW-7311:
-
Summary: [Python] Return filesystem and path from URI
Key: ARROW-7311
URL: https://issues.apache.org/jira/browse/ARROW-7311
Project: Apache Arrow
Issue Type: Improvement
Components: Python
Reporter: Antoine Pitrou
The C++ API {{FileSystemFromUri}} should be exposed in Python.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-7310) [Python] Expose HDFS implementation for pyarrow.fs
[
https://issues.apache.org/jira/browse/ARROW-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987852#comment-16987852
]
Antoine Pitrou commented on ARROW-7310:
---
[~kszucs]
> [Python] Expose HDFS implementation for pyarrow.fs
> --
>
> Key: ARROW-7310
> URL: https://issues.apache.org/jira/browse/ARROW-7310
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
>Reporter: Antoine Pitrou
>Priority: Major
>
> There's a C++ implementation for HDFS (see {{arrow/filesystem/hdfs.h}}) but
> it's not exposed from Python.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-7309) [Python] Support HDFS federation viewfs:// in resolve_filesystem_and_path
[ https://issues.apache.org/jira/browse/ARROW-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987849#comment-16987849 ] Antoine Pitrou commented on ARROW-7309: --- We need to expose that in Python as well :-) > [Python] Support HDFS federation viewfs:// in resolve_filesystem_and_path > - > > Key: ARROW-7309 > URL: https://issues.apache.org/jira/browse/ARROW-7309 > Project: Apache Arrow > Issue Type: Improvement > Components: Python >Affects Versions: 0.15.1 >Reporter: Fabian Höring >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > - Add viewfs support to pyarrow.filesystem.resolve_filesystem_and_path > - libhdfs already supports injecting the scheme and will automatically > resolve federation in > fs = FileSystem#get(URI, conf, ugi) > - works with Hadoop 2/3 > see: > https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L770 > https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L637 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ARROW-7309) [Python] Support HDFS federation viewfs:// in resolve_filesystem_and_path
[ https://issues.apache.org/jira/browse/ARROW-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987848#comment-16987848 ] Fabian Höring commented on ARROW-7309: -- Yes, but now you have those classes exposed separately. I need some kind of generic resolver based on the path. Exactly what `resolve_filesystem_and_path` is doing. > [Python] Support HDFS federation viewfs:// in resolve_filesystem_and_path > - > > Key: ARROW-7309 > URL: https://issues.apache.org/jira/browse/ARROW-7309 > Project: Apache Arrow > Issue Type: Improvement > Components: Python >Affects Versions: 0.15.1 >Reporter: Fabian Höring >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > - Add viewfs support to pyarrow.filesystem.resolve_filesystem_and_path > - libhdfs already supports injecting the scheme and will automatically > resolve federation in > fs = FileSystem#get(URI, conf, ugi) > - works with Hadoop 2/3 > see: > https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L770 > https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L637 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ARROW-7309) [Python] Support HDFS federation viewfs:// in resolve_filesystem_and_path
[
https://issues.apache.org/jira/browse/ARROW-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987847#comment-16987847
]
Antoine Pitrou commented on ARROW-7309:
---
There's also ARROW-7102 open to make {{pyarrow.fs}} interoperate with the
Python {{fsspec}}.
> [Python] Support HDFS federation viewfs:// in resolve_filesystem_and_path
> -
>
> Key: ARROW-7309
> URL: https://issues.apache.org/jira/browse/ARROW-7309
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
>Affects Versions: 0.15.1
>Reporter: Fabian Höring
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> - Add viewfs support to pyarrow.filesystem.resolve_filesystem_and_path
> - libhdfs already supports injecting the scheme and will automatically
> resolve federation in
> fs = FileSystem#get(URI, conf, ugi)
> - works with Hadoop 2/3
> see:
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L770
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L637
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
