[jira] [Updated] (ARROW-3013) [Website] Fix download links on website for tarballs, checksums
[ https://issues.apache.org/jira/browse/ARROW-3013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-3013: Fix Version/s: (was: 0.10.0) 0.11.0 > [Website] Fix download links on website for tarballs, checksums > --- > > Key: ARROW-3013 > URL: https://issues.apache.org/jira/browse/ARROW-3013 > Project: Apache Arrow > Issue Type: Bug > Components: Website >Reporter: Wes McKinney >Priority: Major > Fix For: 0.11.0 > > > See discussion on the ANNOUNCE thread for 0.10.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-1894) [Python] Treat CPython memoryview or buffer objects equivalently to pyarrow.Buffer in pyarrow.serialize
[ https://issues.apache.org/jira/browse/ARROW-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-1894: Fix Version/s: (was: 0.11.0) 0.13.0 > [Python] Treat CPython memoryview or buffer objects equivalently to > pyarrow.Buffer in pyarrow.serialize > --- > > Key: ARROW-1894 > URL: https://issues.apache.org/jira/browse/ARROW-1894 > Project: Apache Arrow > Issue Type: Improvement > Components: Python >Reporter: Wes McKinney >Priority: Major > Labels: beginner > Fix For: 0.13.0 > > > These should be treated as Buffer-like on serialize. We should consider how > to "box" the buffers as the appropriate kind of object (Buffer, memoryview, > etc.) when being deserialized -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-1860) [C++] Add data structure to "stage" a sequence of IPC messages from in-memory data
[
https://issues.apache.org/jira/browse/ARROW-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598010#comment-16598010
]
Wes McKinney commented on ARROW-1860:
-
I will need to revisit this in the context of Flight RPC, where this data
structure may be used to coordinate writes onto the GRPC output stream
> [C++] Add data structure to "stage" a sequence of IPC messages from in-memory
> data
> --
>
> Key: ARROW-1860
> URL: https://issues.apache.org/jira/browse/ARROW-1860
> Project: Apache Arrow
> Issue Type: New Feature
> Components: C++
>Reporter: Wes McKinney
>Assignee: Wes McKinney
>Priority: Major
> Labels: flight, pull-request-available
> Fix For: 0.12.0
>
> Attachments: text.html
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently, when you need to pre-allocate space for a record batch or a stream
> (schema + dictionaries + record batches), you must make multiple passes over
> the data structures of interest (and use e.g. {{MockOutputStream}} to compute
> the size of the output buffer). It would be useful to make a single pass to
> "prepare" the IPC payload for both sizing and writing to prevent having to
> make multiple passes
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-1860) [C++] Add data structure to "stage" a sequence of IPC messages from in-memory data
[
https://issues.apache.org/jira/browse/ARROW-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney updated ARROW-1860:
Labels: flight pull-request-available (was: pull-request-available)
> [C++] Add data structure to "stage" a sequence of IPC messages from in-memory
> data
> --
>
> Key: ARROW-1860
> URL: https://issues.apache.org/jira/browse/ARROW-1860
> Project: Apache Arrow
> Issue Type: New Feature
> Components: C++
>Reporter: Wes McKinney
>Assignee: Wes McKinney
>Priority: Major
> Labels: flight, pull-request-available
> Fix For: 0.12.0
>
> Attachments: text.html
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently, when you need to pre-allocate space for a record batch or a stream
> (schema + dictionaries + record batches), you must make multiple passes over
> the data structures of interest (and use e.g. {{MockOutputStream}} to compute
> the size of the output buffer). It would be useful to make a single pass to
> "prepare" the IPC payload for both sizing and writing to prevent having to
> make multiple passes
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-1860) [C++] Add data structure to "stage" a sequence of IPC messages from in-memory data
[
https://issues.apache.org/jira/browse/ARROW-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney updated ARROW-1860:
Fix Version/s: (was: 0.11.0)
0.12.0
> [C++] Add data structure to "stage" a sequence of IPC messages from in-memory
> data
> --
>
> Key: ARROW-1860
> URL: https://issues.apache.org/jira/browse/ARROW-1860
> Project: Apache Arrow
> Issue Type: New Feature
> Components: C++
>Reporter: Wes McKinney
>Assignee: Wes McKinney
>Priority: Major
> Labels: flight, pull-request-available
> Fix For: 0.12.0
>
> Attachments: text.html
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently, when you need to pre-allocate space for a record batch or a stream
> (schema + dictionaries + record batches), you must make multiple passes over
> the data structures of interest (and use e.g. {{MockOutputStream}} to compute
> the size of the output buffer). It would be useful to make a single pass to
> "prepare" the IPC payload for both sizing and writing to prevent having to
> make multiple passes
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-1807) [JAVA] Reduce Heap Usage (Phase 3): consolidate buffers
[ https://issues.apache.org/jira/browse/ARROW-1807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-1807: Fix Version/s: (was: 0.11.0) 0.12.0 > [JAVA] Reduce Heap Usage (Phase 3): consolidate buffers > --- > > Key: ARROW-1807 > URL: https://issues.apache.org/jira/browse/ARROW-1807 > Project: Apache Arrow > Issue Type: Improvement >Reporter: Siddharth Teotia >Assignee: Siddharth Teotia >Priority: Major > Fix For: 0.12.0 > > > Consolidate buffers for reducing the volume of objects and heap usage > => single buffer for fixed width > < validity + offsets> = single buffer for var width, list vector -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-1823) [C++] Add hash kernel benchmarks, investigate faster alternative non-SIMD hash functions
[ https://issues.apache.org/jira/browse/ARROW-1823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-1823: Fix Version/s: (was: 0.11.0) 0.13.0 > [C++] Add hash kernel benchmarks, investigate faster alternative non-SIMD > hash functions > > > Key: ARROW-1823 > URL: https://issues.apache.org/jira/browse/ARROW-1823 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Reporter: Wes McKinney >Priority: Major > Fix For: 0.13.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-1833) [Java] Add accessor methods for data buffers that skip null checking
[ https://issues.apache.org/jira/browse/ARROW-1833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-1833: Fix Version/s: (was: 0.11.0) 0.13.0 > [Java] Add accessor methods for data buffers that skip null checking > > > Key: ARROW-1833 > URL: https://issues.apache.org/jira/browse/ARROW-1833 > Project: Apache Arrow > Issue Type: Improvement > Components: Java >Reporter: Wes McKinney >Assignee: Jingyuan Wang >Priority: Major > Fix For: 0.13.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-1786) [Format] List expected on-wire buffer layouts for each kind of Arrow physical type in specification
[ https://issues.apache.org/jira/browse/ARROW-1786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-1786: Labels: columnar-format-1.0 (was: ) > [Format] List expected on-wire buffer layouts for each kind of Arrow physical > type in specification > --- > > Key: ARROW-1786 > URL: https://issues.apache.org/jira/browse/ARROW-1786 > Project: Apache Arrow > Issue Type: Improvement > Components: Format >Reporter: Wes McKinney >Assignee: Wes McKinney >Priority: Major > Labels: columnar-format-1.0 > Fix For: 0.13.0 > > > see ARROW-1693, ARROW-1785 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-1696) [C++] Add codec benchmarks
[ https://issues.apache.org/jira/browse/ARROW-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-1696: Fix Version/s: (was: 0.11.0) 0.12.0 > [C++] Add codec benchmarks > -- > > Key: ARROW-1696 > URL: https://issues.apache.org/jira/browse/ARROW-1696 > Project: Apache Arrow > Issue Type: New Feature > Components: C++ >Reporter: Wes McKinney >Priority: Major > Fix For: 0.12.0 > > > This will also help users validate in release builds that the compression > libraries have been built with the appropriate optimization levels, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-1786) [Format] List expected on-wire buffer layouts for each kind of Arrow physical type in specification
[ https://issues.apache.org/jira/browse/ARROW-1786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-1786: Fix Version/s: (was: 0.11.0) 0.13.0 > [Format] List expected on-wire buffer layouts for each kind of Arrow physical > type in specification > --- > > Key: ARROW-1786 > URL: https://issues.apache.org/jira/browse/ARROW-1786 > Project: Apache Arrow > Issue Type: Improvement > Components: Format >Reporter: Wes McKinney >Assignee: Wes McKinney >Priority: Major > Labels: columnar-format-1.0 > Fix For: 0.13.0 > > > see ARROW-1693, ARROW-1785 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-1789) [Format] Consolidate specification documents and improve clarity for new implementation authors
[ https://issues.apache.org/jira/browse/ARROW-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-1789: Fix Version/s: (was: 0.11.0) 0.12.0 > [Format] Consolidate specification documents and improve clarity for new > implementation authors > --- > > Key: ARROW-1789 > URL: https://issues.apache.org/jira/browse/ARROW-1789 > Project: Apache Arrow > Issue Type: Improvement > Components: Format >Reporter: Wes McKinney >Priority: Major > Fix For: 0.12.0 > > > See discussion in https://github.com/apache/arrow/issues/1296 > I believe the specification documents Layout.md, Metadata.md, and IPC.md > would benefit from being consolidated into a single Markdown document that > would be sufficient (along with the Flatbuffers schemas) to create a complete > Arrow implementation capable of reading and writing the binary format -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-1382) [Python] Deduplicate non-scalar Python objects when using pyarrow.serialize
[
https://issues.apache.org/jira/browse/ARROW-1382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney updated ARROW-1382:
Fix Version/s: (was: 0.11.0)
0.13.0
> [Python] Deduplicate non-scalar Python objects when using pyarrow.serialize
> ---
>
> Key: ARROW-1382
> URL: https://issues.apache.org/jira/browse/ARROW-1382
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Reporter: Robert Nishihara
>Priority: Major
> Fix For: 0.13.0
>
>
> If a Python object appears multiple times within a list/tuple/dictionary,
> then when pyarrow serializes the object, it will duplicate the object many
> times. This leads to a potentially huge expansion in the size of the object
> (e.g., the serialized version of {{100 * [np.zeros(10 ** 6)]}} will be 100
> times bigger than it needs to be).
> {code}
> import pyarrow as pa
> l = [0]
> original_object = [l, l]
> # Serialize and deserialize the object.
> buf = pa.serialize(original_object).to_buffer()
> new_object = pa.deserialize(buf)
> # This works.
> assert original_object[0] is original_object[1]
> # This fails.
> assert new_object[0] is new_object[1]
> {code}
> One potential way to address this is to use the Arrow dictionary encoding.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-976) [Python] Provide API for defining and reading Parquet datasets with more ad hoc partition schemes
[ https://issues.apache.org/jira/browse/ARROW-976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-976: --- Fix Version/s: (was: 0.11.0) 0.12.0 > [Python] Provide API for defining and reading Parquet datasets with more ad > hoc partition schemes > - > > Key: ARROW-976 > URL: https://issues.apache.org/jira/browse/ARROW-976 > Project: Apache Arrow > Issue Type: New Feature > Components: Python >Reporter: Wes McKinney >Priority: Major > Fix For: 0.12.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-1837) [Java] Unable to read unsigned integers outside signed range for bit width in integration tests
[ https://issues.apache.org/jira/browse/ARROW-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-1837: Fix Version/s: (was: 0.11.0) 0.12.0 > [Java] Unable to read unsigned integers outside signed range for bit width in > integration tests > --- > > Key: ARROW-1837 > URL: https://issues.apache.org/jira/browse/ARROW-1837 > Project: Apache Arrow > Issue Type: Bug > Components: Java >Reporter: Wes McKinney >Priority: Blocker > Labels: columnar-format-1.0 > Fix For: 0.12.0 > > Attachments: generated_primitive.json > > > I believe this was introduced recently (perhaps in the refactors), but there > was a problem where the integration tests weren't being properly run that hid > the error from us > see https://github.com/apache/arrow/pull/1294#issuecomment-345553066 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-1837) [Java] Unable to read unsigned integers outside signed range for bit width in integration tests
[ https://issues.apache.org/jira/browse/ARROW-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-1837: Labels: columnar-format-1.0 (was: ) > [Java] Unable to read unsigned integers outside signed range for bit width in > integration tests > --- > > Key: ARROW-1837 > URL: https://issues.apache.org/jira/browse/ARROW-1837 > Project: Apache Arrow > Issue Type: Bug > Components: Java >Reporter: Wes McKinney >Priority: Blocker > Labels: columnar-format-1.0 > Fix For: 0.12.0 > > Attachments: generated_primitive.json > > > I believe this was introduced recently (perhaps in the refactors), but there > was a problem where the integration tests weren't being properly run that hid > the error from us > see https://github.com/apache/arrow/pull/1294#issuecomment-345553066 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-3151) [C++] Create Protocol Buffers interface for iterating over the semantic "rows" of a record batch, and accessing the rows using the protobuf API
[
https://issues.apache.org/jira/browse/ARROW-3151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597948#comment-16597948
]
Wes McKinney commented on ARROW-3151:
-
Thanks Paul -- indeed this is a C++ initiative but I agree it would be useful
to align on some target use cases, requirements, etc. What I intended here is
pretty orthogonal to memory management, or any details of where the Arrow
columnar memory comes from; a user might have some code which utilizes a
Protobuf-based row interface and we want to feed that Arrow data as quickly as
possible
> [C++] Create Protocol Buffers interface for iterating over the semantic
> "rows" of a record batch, and accessing the rows using the protobuf API
> ---
>
> Key: ARROW-3151
> URL: https://issues.apache.org/jira/browse/ARROW-3151
> Project: Apache Arrow
> Issue Type: New Feature
> Components: C++
>Reporter: Wes McKinney
>Priority: Major
>
> The desired workflow:
> * User writes a .proto file describing the structure of a "row" as a Message
> * Given the generated pb.h bindings, an Arrow users can iterate over an
> {{arrow::RecordBatch}}, each iteration populating an instance of the Row
> message
> * The values of the row can then be accessed via the standard Protobuf APIs
> A corresponding interface could be developed to write a RecordBatch using
> protobufs as input, but that could be its own project
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-3151) [C++] Create Protocol Buffers interface for iterating over the semantic "rows" of a record batch, and accessing the rows using the protobuf API
[
https://issues.apache.org/jira/browse/ARROW-3151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597920#comment-16597920
]
Paul Rogers commented on ARROW-3151:
Let's see if we can coordinate on this. I'm starting work on a proposal for a
"RowSet" interface to be ported over from Drill that provides a simple
row-based API to read from, and write to, vectors. On the write site, the
mechanism also enforces memory limits, which is the key reason Drill created
the "RowSet" abstraction.
Given that this project will need a way to assemble a row from a bundle of
vectors, the "columnar-to-row" mechanism of RowSet might be a way to populate
the row buffer.
On the other hand, the RowSet code from Drill is in Java, this is C++. Still,
might make sense to port the mechanism to C++ so it can be used in multiple
contexts.
Any background docs I could read to get a better understanding of the project
context to determine if what was just said above makes sense in this context?
Thanks.
> [C++] Create Protocol Buffers interface for iterating over the semantic
> "rows" of a record batch, and accessing the rows using the protobuf API
> ---
>
> Key: ARROW-3151
> URL: https://issues.apache.org/jira/browse/ARROW-3151
> Project: Apache Arrow
> Issue Type: New Feature
> Components: C++
>Reporter: Wes McKinney
>Priority: Major
>
> The desired workflow:
> * User writes a .proto file describing the structure of a "row" as a Message
> * Given the generated pb.h bindings, an Arrow users can iterate over an
> {{arrow::RecordBatch}}, each iteration populating an instance of the Row
> message
> * The values of the row can then be accessed via the standard Protobuf APIs
> A corresponding interface could be developed to write a RecordBatch using
> protobufs as input, but that could be its own project
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-3072) [C++] Use ARROW_RETURN_NOT_OK instead of RETURN_NOT_OK in header files
[
https://issues.apache.org/jira/browse/ARROW-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597903#comment-16597903
]
Wes McKinney commented on ARROW-3072:
-
As a matter of code hygiene we should add a lint check for this in
https://github.com/apache/arrow/blob/master/cpp/build-support/lint_cpp_cli.py
Additionally, to avoid collisions in third party software, we should make
{{ARROW_RETURN_NOT_OK}} the default macro definition, and aliasing this to
{{RETURN_NOT_OK}} _only_ if the latter is not already defined (so we do not
clobber someone else's macro). This way we don't have to change any .cc files
and our builds will continue to work.
> [C++] Use ARROW_RETURN_NOT_OK instead of RETURN_NOT_OK in header files
> --
>
> Key: ARROW-3072
> URL: https://issues.apache.org/jira/browse/ARROW-3072
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
>Reporter: Wes McKinney
>Priority: Major
> Fix For: 0.11.0
>
>
> The {{RETURN_NOT_OK}} macro could conceivably collide with macros in other
> libraries. It would be better to use the scoped macro in public headers
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (ARROW-3151) [C++] Create Protocol Buffers interface for iterating over the semantic "rows" of a record batch, and accessing the rows using the protobuf API
Wes McKinney created ARROW-3151:
---
Summary: [C++] Create Protocol Buffers interface for iterating
over the semantic "rows" of a record batch, and accessing the rows using the
protobuf API
Key: ARROW-3151
URL: https://issues.apache.org/jira/browse/ARROW-3151
Project: Apache Arrow
Issue Type: New Feature
Components: C++
Reporter: Wes McKinney
The desired workflow:
* User writes a .proto file describing the structure of a "row" as a Message
* Given the generated pb.h bindings, an Arrow users can iterate over an
{{arrow::RecordBatch}}, each iteration populating an instance of the Row message
* The values of the row can then be accessed via the standard Protobuf APIs
A corresponding interface could be developed to write a RecordBatch using
protobufs as input, but that could be its own project
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-3149) [C++] Use gRPC (when it exists) from conda-forge for CI builds
[ https://issues.apache.org/jira/browse/ARROW-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597882#comment-16597882 ] Wes McKinney commented on ARROW-3149: - Here is a partially complete recipe: https://github.com/AnacondaRecipes/grpcio-feedstock/tree/master/recipe. The vendoring situation with gRPC makes things pretty complicated -- the CMake build for example installs all of the transitive dependencies. It seems that the patch here is intended to undo this logic; there may be some other way to avoid installing the vendored dependencies without having to patch the CMakeLists.txt (which could grow unmaintainable) > [C++] Use gRPC (when it exists) from conda-forge for CI builds > -- > > Key: ARROW-3149 > URL: https://issues.apache.org/jira/browse/ARROW-3149 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Reporter: Wes McKinney >Priority: Major > Fix For: 0.12.0 > > > gRPC is not available in conda-forge yet. It has a rather complex dependency > chain if you want secure RPCs (e.g. Google is maintaining a fork of OpenSSL, > BoringSSL). Some of these dependencies will have to be added to conda-forge > in turn. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (ARROW-3142) [C++] Fetch all libs from toolchain environment
[ https://issues.apache.org/jira/browse/ARROW-3142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney resolved ARROW-3142. - Resolution: Fixed Fix Version/s: 0.11.0 Issue resolved by pull request 2495 [https://github.com/apache/arrow/pull/2495] > [C++] Fetch all libs from toolchain environment > --- > > Key: ARROW-3142 > URL: https://issues.apache.org/jira/browse/ARROW-3142 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Affects Versions: 0.10.0 >Reporter: Antoine Pitrou >Priority: Major > Labels: pull-request-available > Fix For: 0.11.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > When setting ARROW_BUILD_TOOLCHAIN, gtest and orc are currently not taken > from the toolchain environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-2461) [Python] Build wheels for manylinux2010 tag
[
https://issues.apache.org/jira/browse/ARROW-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597864#comment-16597864
]
Wes McKinney commented on ARROW-2461:
-
My understanding is that the libstdc++ details are all handled by RedHat's
devtoolset toolchain. So I think what symbols are statically linked and binary
compatibility should be the same (save for the glibc issue) whether the build
is run on centos5 or centos6
> [Python] Build wheels for manylinux2010 tag
> ---
>
> Key: ARROW-2461
> URL: https://issues.apache.org/jira/browse/ARROW-2461
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Reporter: Uwe L. Korn
>Priority: Major
> Fix For: 0.11.0
>
>
> There is now work in progress on an updated manylinux tag based on CentOS6.
> We should provide wheels for this tag and the old {{manylinux1}} tag for one
> release and then switch to the new tag in the release afterwards. This should
> enable us also to raise the minimum compiler requirement to gcc 4.9 (or
> higher once conda-forge has migrated to a newer compiler).
> The relevant PEP is https://www.python.org/dev/peps/pep-0571/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-2461) [Python] Build wheels for manylinux2010 tag
[
https://issues.apache.org/jira/browse/ARROW-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597862#comment-16597862
]
Antoine Pitrou commented on ARROW-2461:
---
{quote}AFAIK the only thing "bad" about using the manylinux1 tag on a wheel
built on centos6 would be the minimum glibc version.\{quote}
Do we link libstdc++ statically? If so, hopefully that would be the only thing,
yes.
> [Python] Build wheels for manylinux2010 tag
> ---
>
> Key: ARROW-2461
> URL: https://issues.apache.org/jira/browse/ARROW-2461
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Reporter: Uwe L. Korn
>Priority: Major
> Fix For: 0.11.0
>
>
> There is now work in progress on an updated manylinux tag based on CentOS6.
> We should provide wheels for this tag and the old {{manylinux1}} tag for one
> release and then switch to the new tag in the release afterwards. This should
> enable us also to raise the minimum compiler requirement to gcc 4.9 (or
> higher once conda-forge has migrated to a newer compiler).
> The relevant PEP is https://www.python.org/dev/peps/pep-0571/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (ARROW-2461) [Python] Build wheels for manylinux2010 tag
[
https://issues.apache.org/jira/browse/ARROW-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597862#comment-16597862
]
Antoine Pitrou edited comment on ARROW-2461 at 8/30/18 7:59 PM:
{quote}AFAIK the only thing "bad" about using the manylinux1 tag on a wheel
built on centos6 would be the minimum glibc version.{quote}
Do we link libstdc++ statically? If so, hopefully that would be the only thing,
yes.
was (Author: pitrou):
{quote}AFAIK the only thing "bad" about using the manylinux1 tag on a wheel
built on centos6 would be the minimum glibc version.\{quote}
Do we link libstdc++ statically? If so, hopefully that would be the only thing,
yes.
> [Python] Build wheels for manylinux2010 tag
> ---
>
> Key: ARROW-2461
> URL: https://issues.apache.org/jira/browse/ARROW-2461
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Reporter: Uwe L. Korn
>Priority: Major
> Fix For: 0.11.0
>
>
> There is now work in progress on an updated manylinux tag based on CentOS6.
> We should provide wheels for this tag and the old {{manylinux1}} tag for one
> release and then switch to the new tag in the release afterwards. This should
> enable us also to raise the minimum compiler requirement to gcc 4.9 (or
> higher once conda-forge has migrated to a newer compiler).
> The relevant PEP is https://www.python.org/dev/peps/pep-0571/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-2461) [Python] Build wheels for manylinux2010 tag
[
https://issues.apache.org/jira/browse/ARROW-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597861#comment-16597861
]
Antoine Pitrou commented on ARROW-2461:
---
Ha, apparently pip may still not have support for manylinux2010:
https://github.com/pypa/pip/pull/5410
As for compatibility, the PEP says:
{quote}manylinux1 wheels are considered manylinux2010 wheels. A pip that
recognizes the manylinux2010 platform tag will thus install manylinux1 wheels
for manylinux2010 platforms -- even when explicitly set -- when no
manylinux2010 wheels are available.{quote}
> [Python] Build wheels for manylinux2010 tag
> ---
>
> Key: ARROW-2461
> URL: https://issues.apache.org/jira/browse/ARROW-2461
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Reporter: Uwe L. Korn
>Priority: Major
> Fix For: 0.11.0
>
>
> There is now work in progress on an updated manylinux tag based on CentOS6.
> We should provide wheels for this tag and the old {{manylinux1}} tag for one
> release and then switch to the new tag in the release afterwards. This should
> enable us also to raise the minimum compiler requirement to gcc 4.9 (or
> higher once conda-forge has migrated to a newer compiler).
> The relevant PEP is https://www.python.org/dev/peps/pep-0571/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-3150) [Python] Ship Flight-enabled Python wheels
[
https://issues.apache.org/jira/browse/ARROW-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597854#comment-16597854
]
Wes McKinney commented on ARROW-3150:
-
gRPC build does not create shared libraries so there are many layers to this
onion
> [Python] Ship Flight-enabled Python wheels
> --
>
> Key: ARROW-3150
> URL: https://issues.apache.org/jira/browse/ARROW-3150
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
>Reporter: Wes McKinney
>Assignee: Uwe L. Korn
>Priority: Major
> Labels: flight
> Fix For: 0.12.0
>
>
> This may involve statically-linking (or bundling where shared libs makes
> sense) the various required dependencies with {{libarrow_flight.so}} in the
> manylinux1 wheel build
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-2461) [Python] Build wheels for manylinux2010 tag
[
https://issues.apache.org/jira/browse/ARROW-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597851#comment-16597851
]
Wes McKinney commented on ARROW-2461:
-
So I guess my main question is: If we stop shipping manylinux1 wheels in favor
of manylinux2010, will that disrupt users, or will it "just work"? Are there
other examples of projects using manylinux2010? AFAIK the only thing "bad"
about using the manylinux1 tag on a wheel built on centos6 would be the minimum
glibc version. So while still _bad_ I think it's significantly less bad than
what TensorFlow is doing, which is using a non-conforming compile and link
procedure, in addition AFAIK to building on a newer Linux image. The only
people impacted by this change would be users on very old Linux
> [Python] Build wheels for manylinux2010 tag
> ---
>
> Key: ARROW-2461
> URL: https://issues.apache.org/jira/browse/ARROW-2461
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Reporter: Uwe L. Korn
>Priority: Major
> Fix For: 0.11.0
>
>
> There is now work in progress on an updated manylinux tag based on CentOS6.
> We should provide wheels for this tag and the old {{manylinux1}} tag for one
> release and then switch to the new tag in the release afterwards. This should
> enable us also to raise the minimum compiler requirement to gcc 4.9 (or
> higher once conda-forge has migrated to a newer compiler).
> The relevant PEP is https://www.python.org/dev/peps/pep-0571/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-2461) [Python] Build wheels for manylinux2010 tag
[
https://issues.apache.org/jira/browse/ARROW-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597826#comment-16597826
]
Antoine Pitrou commented on ARROW-2461:
---
I'm not sure what you're proposing here. The manylinux1 PEP requires that we
don't rely on anything younger than glibc 2.5. So using Abseil in a manylinux1
wheel is out of question (unless we want to be a bad citizen like Tensorflow
is).
Note that manylinux2010 is based on CentOS 6, i.e. its compatibility
requirements for base libraries are based on what CentOS 6 provides.
> [Python] Build wheels for manylinux2010 tag
> ---
>
> Key: ARROW-2461
> URL: https://issues.apache.org/jira/browse/ARROW-2461
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Reporter: Uwe L. Korn
>Priority: Major
> Fix For: 0.11.0
>
>
> There is now work in progress on an updated manylinux tag based on CentOS6.
> We should provide wheels for this tag and the old {{manylinux1}} tag for one
> release and then switch to the new tag in the release afterwards. This should
> enable us also to raise the minimum compiler requirement to gcc 4.9 (or
> higher once conda-forge has migrated to a newer compiler).
> The relevant PEP is https://www.python.org/dev/peps/pep-0571/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-3150) [Python] Ship Flight-enabled Python wheels
[
https://issues.apache.org/jira/browse/ARROW-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597796#comment-16597796
]
Uwe L. Korn commented on ARROW-3150:
We should link dynamically. With the correct RPATH settings this is the better
solution. See boost as an example
> [Python] Ship Flight-enabled Python wheels
> --
>
> Key: ARROW-3150
> URL: https://issues.apache.org/jira/browse/ARROW-3150
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
>Reporter: Wes McKinney
>Priority: Major
> Labels: flight
> Fix For: 0.12.0
>
>
> This may involve statically-linking (or bundling where shared libs makes
> sense) the various required dependencies with {{libarrow_flight.so}} in the
> manylinux1 wheel build
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (ARROW-3150) [Python] Ship Flight-enabled Python wheels
[
https://issues.apache.org/jira/browse/ARROW-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uwe L. Korn reassigned ARROW-3150:
--
Assignee: Uwe L. Korn
> [Python] Ship Flight-enabled Python wheels
> --
>
> Key: ARROW-3150
> URL: https://issues.apache.org/jira/browse/ARROW-3150
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
>Reporter: Wes McKinney
>Assignee: Uwe L. Korn
>Priority: Major
> Labels: flight
> Fix For: 0.12.0
>
>
> This may involve statically-linking (or bundling where shared libs makes
> sense) the various required dependencies with {{libarrow_flight.so}} in the
> manylinux1 wheel build
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (ARROW-3150) [Python] Ship Flight-enabled Python wheels
Wes McKinney created ARROW-3150:
---
Summary: [Python] Ship Flight-enabled Python wheels
Key: ARROW-3150
URL: https://issues.apache.org/jira/browse/ARROW-3150
Project: Apache Arrow
Issue Type: Improvement
Components: C++
Reporter: Wes McKinney
Fix For: 0.12.0
This may involve statically-linking (or bundling where shared libs makes sense)
the various required dependencies with {{libarrow_flight.so}} in the manylinux1
wheel build
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (ARROW-3149) [C++] Use gRPC (when it exists) from conda-forge for CI builds
Wes McKinney created ARROW-3149: --- Summary: [C++] Use gRPC (when it exists) from conda-forge for CI builds Key: ARROW-3149 URL: https://issues.apache.org/jira/browse/ARROW-3149 Project: Apache Arrow Issue Type: Improvement Components: C++ Reporter: Wes McKinney Fix For: 0.12.0 gRPC is not available in conda-forge yet. It has a rather complex dependency chain if you want secure RPCs (e.g. Google is maintaining a fork of OpenSSL, BoringSSL). Some of these dependencies will have to be added to conda-forge in turn. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-2461) [Python] Build wheels for manylinux2010 tag
[
https://issues.apache.org/jira/browse/ARROW-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597784#comment-16597784
]
Wes McKinney commented on ARROW-2461:
-
According to https://access.redhat.com/support/policy/updates/errata RedHat has
retired RHEL5 and is continuing extended support to those paying for it through
end of 2020. I expect these cases to be quite exceptional and not reflecting of
the average Arrow adopter or even Python user
> [Python] Build wheels for manylinux2010 tag
> ---
>
> Key: ARROW-2461
> URL: https://issues.apache.org/jira/browse/ARROW-2461
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Reporter: Uwe L. Korn
>Priority: Major
> Fix For: 0.11.0
>
>
> There is now work in progress on an updated manylinux tag based on CentOS6.
> We should provide wheels for this tag and the old {{manylinux1}} tag for one
> release and then switch to the new tag in the release afterwards. This should
> enable us also to raise the minimum compiler requirement to gcc 4.9 (or
> higher once conda-forge has migrated to a newer compiler).
> The relevant PEP is https://www.python.org/dev/peps/pep-0571/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-2461) [Python] Build wheels for manylinux2010 tag
[
https://issues.apache.org/jira/browse/ARROW-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597779#comment-16597779
]
Wes McKinney commented on ARROW-2461:
-
Oy. So RHEL 6 has glibc 2.12
https://distrowatch.com/table.php?distribution=redhat
The base image for manylinux1 is centos5 which tracks RHEL 5.
https://distrowatch.com/table.php?distribution=centos
I don't know if this is a radical proposal but we should promote our manylinux1
image to be based on centos6, which I guess is the same thing as manylinux2010.
PyTorch and TensorFlow have already created toolchain problems -- not to say
that we should make things worse -- but I am not sure that this would do very
much harm compared with, say, using a different compiler (like Clang with
statically linked libstdc++ instead of using the RedHat devtoolset)
I don't know if we are doing ourselves any favors by being stuck on centos5
glibc, and the impacted users may be slow-to-upgrade enterprises which, I would
hazard a guess, aren't contributing to Apache Arrow.
> [Python] Build wheels for manylinux2010 tag
> ---
>
> Key: ARROW-2461
> URL: https://issues.apache.org/jira/browse/ARROW-2461
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Reporter: Uwe L. Korn
>Priority: Major
> Fix For: 0.11.0
>
>
> There is now work in progress on an updated manylinux tag based on CentOS6.
> We should provide wheels for this tag and the old {{manylinux1}} tag for one
> release and then switch to the new tag in the release afterwards. This should
> enable us also to raise the minimum compiler requirement to gcc 4.9 (or
> higher once conda-forge has migrated to a newer compiler).
> The relevant PEP is https://www.python.org/dev/peps/pep-0571/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-2461) [Python] Build wheels for manylinux2010 tag
[
https://issues.apache.org/jira/browse/ARROW-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597769#comment-16597769
]
Antoine Pitrou commented on ARROW-2461:
---
Turns out glibc 2.12, which is the baseline in manylinux2010 but not
manylinux1, is required by the Abseil C++ library.
> [Python] Build wheels for manylinux2010 tag
> ---
>
> Key: ARROW-2461
> URL: https://issues.apache.org/jira/browse/ARROW-2461
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Reporter: Uwe L. Korn
>Priority: Major
> Fix For: 0.11.0
>
>
> There is now work in progress on an updated manylinux tag based on CentOS6.
> We should provide wheels for this tag and the old {{manylinux1}} tag for one
> release and then switch to the new tag in the release afterwards. This should
> enable us also to raise the minimum compiler requirement to gcc 4.9 (or
> higher once conda-forge has migrated to a newer compiler).
> The relevant PEP is https://www.python.org/dev/peps/pep-0571/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (ARROW-3147) [C++] MSVC version isn't detected in code page 932
[ https://issues.apache.org/jira/browse/ARROW-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney resolved ARROW-3147. - Resolution: Fixed Fix Version/s: 0.11.0 Issue resolved by pull request 2499 [https://github.com/apache/arrow/pull/2499] > [C++] MSVC version isn't detected in code page 932 > -- > > Key: ARROW-3147 > URL: https://issues.apache.org/jira/browse/ARROW-3147 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Affects Versions: 0.10.0 >Reporter: Kouhei Sutou >Assignee: Kouhei Sutou >Priority: Minor > Labels: pull-request-available > Fix For: 0.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > > There is no space between "Microsoft" and "(R)" in cl.exe output on code page > 932. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (ARROW-3148) [C++] MSVC shows C4819 warning on code page 932
[ https://issues.apache.org/jira/browse/ARROW-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney resolved ARROW-3148. - Resolution: Fixed Fix Version/s: 0.11.0 > [C++] MSVC shows C4819 warning on code page 932 > > > Key: ARROW-3148 > URL: https://issues.apache.org/jira/browse/ARROW-3148 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Affects Versions: 0.10.0 >Reporter: Kouhei Sutou >Assignee: Kouhei Sutou >Priority: Minor > Labels: pull-request-available > Fix For: 0.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-1669) [C++] Consider adding Abseil (Google C++11 standard library extensions) to toolchain
[ https://issues.apache.org/jira/browse/ARROW-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated ARROW-1669: -- Labels: pull-request-available (was: ) > [C++] Consider adding Abseil (Google C++11 standard library extensions) to > toolchain > > > Key: ARROW-1669 > URL: https://issues.apache.org/jira/browse/ARROW-1669 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Reporter: Wes McKinney >Priority: Major > Labels: pull-request-available > > Google has released a library of C++11-compliant extensions to the STL that > may help make a lot of Arrow code simpler: > https://github.com/abseil/abseil-cpp/ > This code is not header-only and so would require some effort to add to the > toolchain at the moment since it only supports the Bazel build system -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARROW-3148) [C++] MSVC shows C4819 warning on code page 932
Kouhei Sutou created ARROW-3148: --- Summary: [C++] MSVC shows C4819 warning on code page 932 Key: ARROW-3148 URL: https://issues.apache.org/jira/browse/ARROW-3148 Project: Apache Arrow Issue Type: Improvement Components: C++ Affects Versions: 0.10.0 Reporter: Kouhei Sutou Assignee: Kouhei Sutou -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-3148) [C++] MSVC shows C4819 warning on code page 932
[ https://issues.apache.org/jira/browse/ARROW-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated ARROW-3148: -- Labels: pull-request-available (was: ) > [C++] MSVC shows C4819 warning on code page 932 > > > Key: ARROW-3148 > URL: https://issues.apache.org/jira/browse/ARROW-3148 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Affects Versions: 0.10.0 >Reporter: Kouhei Sutou >Assignee: Kouhei Sutou >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-3147) [C++] MSVC version isn't detected in code page 932
[ https://issues.apache.org/jira/browse/ARROW-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated ARROW-3147: -- Labels: pull-request-available (was: ) > [C++] MSVC version isn't detected in code page 932 > -- > > Key: ARROW-3147 > URL: https://issues.apache.org/jira/browse/ARROW-3147 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Affects Versions: 0.10.0 >Reporter: Kouhei Sutou >Assignee: Kouhei Sutou >Priority: Minor > Labels: pull-request-available > > There is no space between "Microsoft" and "(R)" in cl.exe output on code page > 932. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARROW-3147) [C++] MSVC version isn't detected in code page 932
Kouhei Sutou created ARROW-3147: --- Summary: [C++] MSVC version isn't detected in code page 932 Key: ARROW-3147 URL: https://issues.apache.org/jira/browse/ARROW-3147 Project: Apache Arrow Issue Type: Improvement Components: C++ Affects Versions: 0.10.0 Reporter: Kouhei Sutou Assignee: Kouhei Sutou There is no space between "Microsoft" and "(R)" in cl.exe output on code page 932. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-3086) [Glib] GISCAN fails due to conda-shipped openblas
[
https://issues.apache.org/jira/browse/ARROW-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uwe L. Korn updated ARROW-3086:
---
Fix Version/s: 0.11.0
> [Glib] GISCAN fails due to conda-shipped openblas
> -
>
> Key: ARROW-3086
> URL: https://issues.apache.org/jira/browse/ARROW-3086
> Project: Apache Arrow
> Issue Type: Bug
> Components: GLib
>Affects Versions: 0.10.0
>Reporter: Uwe L. Korn
>Assignee: Kouhei Sutou
>Priority: Major
> Fix For: 0.11.0
>
>
> With the changes in [https://github.com/apache/arrow/pull/2374], the
> libraries provided by conda are now in the library path when running the
> GISCAN step. This sadly leads to the poisoning of the search path with the
> conda provided openblas which is incompatible with the system provided
> libLAPACK.dylib
> {code:java}
> dyld: Library not loaded:
> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib
> Referenced from:
> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/vecLib
> Reason: Incompatible library version: vecLib requires version 1.0.0 or later,
> but libLAPACK.dylib provides version 0.0.0{code}
> While mentioned that it explicitly loads
> {{/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib}},
> it seems that {{liblapack.so}} from the conda installation gets picked up
> first. This only provides the library symbols with version 0.0.0 and thus is
> incompatible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-3086) [Glib] GISCAN fails due to conda-shipped openblas
[
https://issues.apache.org/jira/browse/ARROW-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597392#comment-16597392
]
Uwe L. Korn commented on ARROW-3086:
I will have a look at this in 2-3 weeks again. conda-forge is currently
updating their blas dependency. This may solve it already magically.
> [Glib] GISCAN fails due to conda-shipped openblas
> -
>
> Key: ARROW-3086
> URL: https://issues.apache.org/jira/browse/ARROW-3086
> Project: Apache Arrow
> Issue Type: Bug
> Components: GLib
>Affects Versions: 0.10.0
>Reporter: Uwe L. Korn
>Assignee: Kouhei Sutou
>Priority: Major
> Fix For: 0.11.0
>
>
> With the changes in [https://github.com/apache/arrow/pull/2374], the
> libraries provided by conda are now in the library path when running the
> GISCAN step. This sadly leads to the poisoning of the search path with the
> conda provided openblas which is incompatible with the system provided
> libLAPACK.dylib
> {code:java}
> dyld: Library not loaded:
> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib
> Referenced from:
> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/vecLib
> Reason: Incompatible library version: vecLib requires version 1.0.0 or later,
> but libLAPACK.dylib provides version 0.0.0{code}
> While mentioned that it explicitly loads
> {{/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib}},
> it seems that {{liblapack.so}} from the conda installation gets picked up
> first. This only provides the library symbols with version 0.0.0 and thus is
> incompatible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (ARROW-3140) [Plasma] Plasma fails building with GPU enabled
[
https://issues.apache.org/jira/browse/ARROW-3140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uwe L. Korn reassigned ARROW-3140:
--
Assignee: Antoine Pitrou
> [Plasma] Plasma fails building with GPU enabled
> ---
>
> Key: ARROW-3140
> URL: https://issues.apache.org/jira/browse/ARROW-3140
> Project: Apache Arrow
> Issue Type: Bug
> Components: GPU, Plasma (C++)
>Reporter: Antoine Pitrou
>Assignee: Antoine Pitrou
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.11.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> {code}
> In file included from ../src/plasma/client.h:30:0,
> from ../src/plasma/client.cc:20:
> ../src/plasma/common.h:120:19: error: ‘CudaIpcMemHandle’ was not declared in
> this scope
>std::shared_ptr ipc_handle;
>^~~~
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (ARROW-3140) [Plasma] Plasma fails building with GPU enabled
[
https://issues.apache.org/jira/browse/ARROW-3140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uwe L. Korn resolved ARROW-3140.
Resolution: Fixed
Fix Version/s: 0.11.0
Issue resolved by pull request 2498
[https://github.com/apache/arrow/pull/2498]
> [Plasma] Plasma fails building with GPU enabled
> ---
>
> Key: ARROW-3140
> URL: https://issues.apache.org/jira/browse/ARROW-3140
> Project: Apache Arrow
> Issue Type: Bug
> Components: GPU, Plasma (C++)
>Reporter: Antoine Pitrou
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.11.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {code}
> In file included from ../src/plasma/client.h:30:0,
> from ../src/plasma/client.cc:20:
> ../src/plasma/common.h:120:19: error: ‘CudaIpcMemHandle’ was not declared in
> this scope
>std::shared_ptr ipc_handle;
>^~~~
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-3140) [Plasma] Plasma fails building with GPU enabled
[
https://issues.apache.org/jira/browse/ARROW-3140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated ARROW-3140:
--
Labels: pull-request-available (was: )
> [Plasma] Plasma fails building with GPU enabled
> ---
>
> Key: ARROW-3140
> URL: https://issues.apache.org/jira/browse/ARROW-3140
> Project: Apache Arrow
> Issue Type: Bug
> Components: GPU, Plasma (C++)
>Reporter: Antoine Pitrou
>Priority: Major
> Labels: pull-request-available
>
> {code}
> In file included from ../src/plasma/client.h:30:0,
> from ../src/plasma/client.cc:20:
> ../src/plasma/common.h:120:19: error: ‘CudaIpcMemHandle’ was not declared in
> this scope
>std::shared_ptr ipc_handle;
>^~~~
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-3086) [Glib] GISCAN fails due to conda-shipped openblas
[
https://issues.apache.org/jira/browse/ARROW-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597201#comment-16597201
]
Kouhei Sutou commented on ARROW-3086:
-
[~xhochy] Sorry. I couldn't reproduce this case...
Can you show the output of {{make V=1}}?
> [Glib] GISCAN fails due to conda-shipped openblas
> -
>
> Key: ARROW-3086
> URL: https://issues.apache.org/jira/browse/ARROW-3086
> Project: Apache Arrow
> Issue Type: Bug
> Components: GLib
>Affects Versions: 0.10.0
>Reporter: Uwe L. Korn
>Assignee: Kouhei Sutou
>Priority: Major
>
> With the changes in [https://github.com/apache/arrow/pull/2374], the
> libraries provided by conda are now in the library path when running the
> GISCAN step. This sadly leads to the poisoning of the search path with the
> conda provided openblas which is incompatible with the system provided
> libLAPACK.dylib
> {code:java}
> dyld: Library not loaded:
> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib
> Referenced from:
> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/vecLib
> Reason: Incompatible library version: vecLib requires version 1.0.0 or later,
> but libLAPACK.dylib provides version 0.0.0{code}
> While mentioned that it explicitly loads
> {{/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib}},
> it seems that {{liblapack.so}} from the conda installation gets picked up
> first. This only provides the library symbols with version 0.0.0 and thus is
> incompatible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
