[jira] [Commented] (ARROW-1983) [Python] Add ability to write parquet `_metadata` file

[ https://issues.apache.org/jira/browse/ARROW-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833141#comment-16833141 ] Pearu Peterson commented on ARROW-1983: --- For this issue, questions raised in

[jira] [Commented] (ARROW-1983) [Python] Add ability to write parquet `_metadata` file

[ https://issues.apache.org/jira/browse/ARROW-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833140#comment-16833140 ] Pearu Peterson commented on ARROW-1983: --- ARROW-5258 provides a way to collect file metadata objects

[jira] [Updated] (ARROW-5258) Expose file metadata of dataset pieces to caller

[ https://issues.apache.org/jira/browse/ARROW-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pearu Peterson updated ARROW-5258: -- Description: This issue resolves partly the issue raised in ARROW-1983 by providing a way to

[jira] [Updated] (ARROW-5258) [C++/Python] Expose file metadata of dataset pieces to caller

[ https://issues.apache.org/jira/browse/ARROW-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pearu Peterson updated ARROW-5258: -- Summary: [C++/Python] Expose file metadata of dataset pieces to caller (was: Expose file

[jira] [Updated] (ARROW-5258) Expose file metadata of dataset pieces to caller

[ https://issues.apache.org/jira/browse/ARROW-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pearu Peterson updated ARROW-5258: -- Description: This issue resolves partly the issue raised in ARROW-1983 > Expose file metadata

[jira] [Created] (ARROW-5258) Expose file metadata of dataset pieces to caller

Pearu Peterson created ARROW-5258: - Summary: Expose file metadata of dataset pieces to caller Key: ARROW-5258 URL: https://issues.apache.org/jira/browse/ARROW-5258 Project: Apache Arrow

[jira] [Commented] (ARROW-1983) [Python] Add ability to write parquet `_metadata` file

[ https://issues.apache.org/jira/browse/ARROW-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831195#comment-16831195 ] Pearu Peterson commented on ARROW-1983: --- Arrow [PR 4236|https://github.com/apache/arrow/pull/4236] 

[jira] [Commented] (ARROW-1983) [Python] Add ability to write parquet `_metadata` file

[ https://issues.apache.org/jira/browse/ARROW-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820504#comment-16820504 ] Pearu Peterson commented on ARROW-1983: --- Arrow [PR 4166|https://github.com/apache/arrow/pull/4166] 

[jira] [Commented] (ARROW-1983) [Python] Add ability to write parquet `_metadata` file

[ https://issues.apache.org/jira/browse/ARROW-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820084#comment-16820084 ] Pearu Peterson commented on ARROW-1983: --- There seems to be two options to write a separate metadata

[jira] [Comment Edited] (ARROW-1983) [Python] Add ability to write parquet `_metadata` file

[ https://issues.apache.org/jira/browse/ARROW-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818343#comment-16818343 ] Pearu Peterson edited comment on ARROW-1983 at 4/15/19 8:06 PM: Note that

[jira] [Commented] (ARROW-1983) [Python] Add ability to write parquet `_metadata` file

[ https://issues.apache.org/jira/browse/ARROW-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818343#comment-16818343 ] Pearu Peterson commented on ARROW-1983: --- Note that the Parquet format has three different metadata

[jira] [Commented] (ARROW-1983) [Python] Add ability to write parquet `_metadata` file

[ https://issues.apache.org/jira/browse/ARROW-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817308#comment-16817308 ] Pearu Peterson commented on ARROW-1983: --- Currently, ParquetDataset metadata has the following

[jira] [Commented] (ARROW-5090) Linking failure on MacOS due to @rpath in dylib

[ https://issues.apache.org/jira/browse/ARROW-5090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808126#comment-16808126 ] Pearu Peterson commented on ARROW-5090: --- To avoid this failure, I have used {{export

[jira] [Commented] (ARROW-4861) [C++] Introduce MemoryPool::Memset method.

[ https://issues.apache.org/jira/browse/ARROW-4861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16792802#comment-16792802 ] Pearu Peterson commented on ARROW-4861: --- No. As discussed in ARROW-2447 , a Device would not be a

[jira] [Commented] (ARROW-4861) [C++] Introduce MemoryPool::Memset method.

[ https://issues.apache.org/jira/browse/ARROW-4861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16792460#comment-16792460 ] Pearu Peterson commented on ARROW-4861: --- Yes, also places with fill and memcpy would need to be

[jira] [Created] (ARROW-4861) [C++] Introduce MemoryPool::Memset method.

Pearu Peterson created ARROW-4861: - Summary: [C++] Introduce MemoryPool::Memset method. Key: ARROW-4861 URL: https://issues.apache.org/jira/browse/ARROW-4861 Project: Apache Arrow Issue

[jira] [Assigned] (ARROW-4825) [Python][C++] MemoryPool is destructed before deallocating its buffers leads to segfault

[ https://issues.apache.org/jira/browse/ARROW-4825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pearu Peterson reassigned ARROW-4825: - Assignee: Pearu Peterson > [Python][C++] MemoryPool is destructed before deallocating

[jira] [Updated] (ARROW-4825) [Python][C++] MemoryPool is destructed before deallocating its buffers leads to segfault

[ https://issues.apache.org/jira/browse/ARROW-4825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pearu Peterson updated ARROW-4825: -- Component/s: C++ > [Python][C++] MemoryPool is destructed before deallocating its buffers

[jira] [Commented] (ARROW-4825) [Python][C++] MemoryPool is destructed before deallocating its buffers leads to segfault

[ https://issues.apache.org/jira/browse/ARROW-4825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16790005#comment-16790005 ] Pearu Peterson commented on ARROW-4825: --- So, as pointer out by [~wesmckinn], the pool is already

[jira] [Updated] (ARROW-4825) [Python][C++] MemoryPool is destructed before deallocating its buffers leads to segfault

[ https://issues.apache.org/jira/browse/ARROW-4825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pearu Peterson updated ARROW-4825: -- Summary: [Python][C++] MemoryPool is destructed before deallocating its buffers leads to

[jira] [Commented] (ARROW-2447) [C++] Create a device abstraction

[ https://issues.apache.org/jira/browse/ARROW-2447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789753#comment-16789753 ] Pearu Peterson commented on ARROW-2447: --- Thanks for the pointer! I'll give shared_ptr-approach a

[jira] [Commented] (ARROW-2447) [C++] Create a device abstraction

[ https://issues.apache.org/jira/browse/ARROW-2447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789498#comment-16789498 ] Pearu Peterson commented on ARROW-2447: --- It turns out that MemoryPool should be always attached to

[jira] [Created] (ARROW-4825) MemoryPool is destructed before deallocating its buffers leads to segfault

Pearu Peterson created ARROW-4825: - Summary: MemoryPool is destructed before deallocating its buffers leads to segfault Key: ARROW-4825 URL: https://issues.apache.org/jira/browse/ARROW-4825 Project:

[jira] [Commented] (ARROW-2447) [C++] Create a device abstraction

[ https://issues.apache.org/jira/browse/ARROW-2447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1673#comment-1673 ] Pearu Peterson commented on ARROW-2447: --- Also the MemoryPool plays an important role in this issue:

[jira] [Commented] (ARROW-2447) [C++] Create a device abstraction

[ https://issues.apache.org/jira/browse/ARROW-2447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16788780#comment-16788780 ] Pearu Peterson commented on ARROW-2447: --- Re [~pitrou] comment:  need a way to query device-specific

[jira] [Comment Edited] (ARROW-2447) [C++] Create a device abstraction

[ https://issues.apache.org/jira/browse/ARROW-2447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16788777#comment-16788777 ] Pearu Peterson edited comment on ARROW-2447 at 3/9/19 7:12 PM: --- If a CUDA

[jira] [Commented] (ARROW-2447) [C++] Create a device abstraction

[ https://issues.apache.org/jira/browse/ARROW-2447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16788777#comment-16788777 ] Pearu Peterson commented on ARROW-2447: --- If a CUDA device supports compute capability 7.0 or higher

[jira] [Commented] (ARROW-2447) [C++] Create a device abstraction

[ https://issues.apache.org/jira/browse/ARROW-2447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16788770#comment-16788770 ] Pearu Peterson commented on ARROW-2447: --- FYI, CUDA introduces

[jira] [Commented] (ARROW-2447) [C++] Create a device abstraction

[ https://issues.apache.org/jira/browse/ARROW-2447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16788766#comment-16788766 ] Pearu Peterson commented on ARROW-2447: --- Re [~pitrou] comment: In case (iii), the memory still

[jira] [Assigned] (ARROW-2447) [C++] Create a device abstraction

[ https://issues.apache.org/jira/browse/ARROW-2447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pearu Peterson reassigned ARROW-2447: - Assignee: Pearu Peterson > [C++] Create a device abstraction >

[jira] [Updated] (ARROW-4486) [Python][CUDA] pyarrow.cuda.Context.foreign_buffer should have a `base=None` argument

[ https://issues.apache.org/jira/browse/ARROW-4486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pearu Peterson updated ARROW-4486: -- External issue URL: https://github.com/apache/arrow/pull/3850 > [Python][CUDA]

[jira] [Assigned] (ARROW-4486) [Python][CUDA] pyarrow.cuda.Context.foreign_buffer should have a `base=None` argument

[ https://issues.apache.org/jira/browse/ARROW-4486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pearu Peterson reassigned ARROW-4486: - Assignee: Pearu Peterson > [Python][CUDA] pyarrow.cuda.Context.foreign_buffer should

[jira] [Commented] (ARROW-4486) [Python][CUDA] pyarrow.cuda.Context.foreign_buffer should have a `base=None` argument

[ https://issues.apache.org/jira/browse/ARROW-4486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771901#comment-16771901 ] Pearu Peterson commented on ARROW-4486: --- Sure. > [Python][CUDA]

[jira] [Comment Edited] (ARROW-4547) [Python][Documentation] Update python/development.rst with instructions for CUDA-enabled builds

[ https://issues.apache.org/jira/browse/ARROW-4547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16769077#comment-16769077 ] Pearu Peterson edited comment on ARROW-4547 at 2/15/19 8:38 AM:

[jira] [Commented] (ARROW-4547) [Python][Documentation] Update python/development.rst with instructions for CUDA-enabled builds

[ https://issues.apache.org/jira/browse/ARROW-4547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16769077#comment-16769077 ] Pearu Peterson commented on ARROW-4547: --- [~Andrew_Palumbo], there are many possibly superfluous

[jira] [Assigned] (ARROW-3653) [Python/C++] Support data copying between different GPU devices

[ https://issues.apache.org/jira/browse/ARROW-3653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pearu Peterson reassigned ARROW-3653: - Assignee: Pearu Peterson > [Python/C++] Support data copying between different GPU

[jira] [Commented] (ARROW-3653) [Python/C++] Support data copying between different GPU devices

[ https://issues.apache.org/jira/browse/ARROW-3653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762882#comment-16762882 ] Pearu Peterson commented on ARROW-3653: --- Sounds good. As long as the nvidia driver is working and

[jira] [Commented] (ARROW-3653) [Python/C++] Support data copying between different GPU devices

[ https://issues.apache.org/jira/browse/ARROW-3653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762846#comment-16762846 ] Pearu Peterson commented on ARROW-3653: --- Yes, I can submit a patch. However, I don't have access to

[jira] [Updated] (ARROW-4486) [Python][CUDA] pyarrow.cuda.Context.foreign_buffer should have a `base=None` argument

[ https://issues.apache.org/jira/browse/ARROW-4486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pearu Peterson updated ARROW-4486: -- Summary: [Python][CUDA] pyarrow.cuda.Context.foreign_buffer should have a `base=None` argument

[jira] [Created] (ARROW-4486) pyarrow.cuda.Context.foreign_buffer should have a `base=None` argument

Pearu Peterson created ARROW-4486: - Summary: pyarrow.cuda.Context.foreign_buffer should have a `base=None` argument Key: ARROW-4486 URL: https://issues.apache.org/jira/browse/ARROW-4486 Project:

[jira] [Assigned] (ARROW-4212) [Python] [CUDA] Creating a CUDA buffer from Numba device array should be easier

[ https://issues.apache.org/jira/browse/ARROW-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pearu Peterson reassigned ARROW-4212: - Assignee: Pearu Peterson > [Python] [CUDA] Creating a CUDA buffer from Numba device

[jira] [Created] (ARROW-3653) [Python/C++] Support data copying between different GPU devices

Pearu Peterson created ARROW-3653: - Summary: [Python/C++] Support data copying between different GPU devices Key: ARROW-3653 URL: https://issues.apache.org/jira/browse/ARROW-3653 Project: Apache

[jira] [Assigned] (ARROW-3624) [Python/C++] Support for zero-sized device buffers

[ https://issues.apache.org/jira/browse/ARROW-3624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pearu Peterson reassigned ARROW-3624: - Assignee: Pearu Peterson > [Python/C++] Support for zero-sized device buffers >

[jira] [Created] (ARROW-3624) [Python/C++] Support for zero-sized device buffers

Pearu Peterson created ARROW-3624: - Summary: [Python/C++] Support for zero-sized device buffers Key: ARROW-3624 URL: https://issues.apache.org/jira/browse/ARROW-3624 Project: Apache Arrow

[jira] [Assigned] (ARROW-3451) [Python] Allocate CUDA memory from a CUcontext created by numba.cuda

[ https://issues.apache.org/jira/browse/ARROW-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pearu Peterson reassigned ARROW-3451: - Assignee: Pearu Peterson > [Python] Allocate CUDA memory from a CUcontext created by

[jira] [Reopened] (ARROW-3354) [Python] read_record_batch interfaces differ in pyarrow and pyarrow.cuda

[ https://issues.apache.org/jira/browse/ARROW-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pearu Peterson reopened ARROW-3354: --- Reopening because the PR is not merged. Sorry for the noise.. > [Python] read_record_batch

[jira] [Resolved] (ARROW-3354) [Python] read_record_batch interfaces differ in pyarrow and pyarrow.cuda

[ https://issues.apache.org/jira/browse/ARROW-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pearu Peterson resolved ARROW-3354. --- Resolution: Fixed > [Python] read_record_batch interfaces differ in pyarrow and pyarrow.cuda

[jira] [Updated] (ARROW-3354) [Python] read_record_batch interfaces differ in pyarrow and pyarrow.cuda

[ https://issues.apache.org/jira/browse/ARROW-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pearu Peterson updated ARROW-3354: -- External issue URL: https://github.com/apache/arrow/pull/2657 > [Python] read_record_batch

[jira] [Assigned] (ARROW-3354) [Python] read_record_batch interfaces differ in pyarrow and pyarrow.cuda

[ https://issues.apache.org/jira/browse/ARROW-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pearu Peterson reassigned ARROW-3354: - Assignee: Pearu Peterson > [Python] read_record_batch interfaces differ in pyarrow and

[jira] [Created] (ARROW-3354) read_record_patch interfaces differ in pyarrow and pyarrow.cuda

Pearu Peterson created ARROW-3354: - Summary: read_record_patch interfaces differ in pyarrow and pyarrow.cuda Key: ARROW-3354 URL: https://issues.apache.org/jira/browse/ARROW-3354 Project: Apache

[jira] [Created] (ARROW-3228) [Python] Immutability of bytes is ignored

Pearu Peterson created ARROW-3228: - Summary: [Python] Immutability of bytes is ignored Key: ARROW-3228 URL: https://issues.apache.org/jira/browse/ARROW-3228 Project: Apache Arrow Issue Type:

[jira] [Created] (ARROW-3221) [Python] Add a virtual Slice method to buffers

Pearu Peterson created ARROW-3221: - Summary: [Python] Add a virtual Slice method to buffers Key: ARROW-3221 URL: https://issues.apache.org/jira/browse/ARROW-3221 Project: Apache Arrow Issue

[jira] [Created] (ARROW-3220) [Python] Add writeat method to writeable NativeFile

Pearu Peterson created ARROW-3220: - Summary: [Python] Add writeat method to writeable NativeFile Key: ARROW-3220 URL: https://issues.apache.org/jira/browse/ARROW-3220 Project: Apache Arrow

[jira] [Created] (ARROW-2944) Arrow format documentation mentions VectorLayout that does not exist anymore

Pearu Peterson created ARROW-2944: - Summary: Arrow format documentation mentions VectorLayout that does not exist anymore Key: ARROW-2944 URL: https://issues.apache.org/jira/browse/ARROW-2944

[jira] [Commented] (ARROW-2903) [C++] Setting -DARROW_HDFS=OFF breaks arrow build when linking against boost libraries

[ https://issues.apache.org/jira/browse/ARROW-2903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554830#comment-16554830 ] Pearu Peterson commented on ARROW-2903: --- Yes, I was using 0.9.0. When starting to prepare a PR, I

[jira] [Created] (ARROW-2903) Setting -DARROW_HDFS=OFF breaks arrow build when linking against boost libraries

Pearu Peterson created ARROW-2903: - Summary: Setting -DARROW_HDFS=OFF breaks arrow build when linking against boost libraries Key: ARROW-2903 URL: https://issues.apache.org/jira/browse/ARROW-2903