[jira] [Commented] (ARROW-6352) [Java] Add implementation of DenseUnionVector.
[ https://issues.apache.org/jira/browse/ARROW-6352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915467#comment-16915467 ] Liya Fan commented on ARROW-6352: - [~emkornfi...@gmail.com] This sounds interesting and useful. Can you please point out the "other JIRAs covering"? > [Java] Add implementation of DenseUnionVector. > -- > > Key: ARROW-6352 > URL: https://issues.apache.org/jira/browse/ARROW-6352 > Project: Apache Arrow > Issue Type: Improvement >Reporter: Micah Kornfield >Priority: Major > > Today only Sparse unions are supported. We should have a dense union > implementation vector that conforms to the IPC protocol (the current spare > union vector doesn't do this and there are other JIRAs covering making these > compatible). -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (ARROW-6352) [Java] Add implementation of DenseUnionVector.
Micah Kornfield created ARROW-6352: -- Summary: [Java] Add implementation of DenseUnionVector. Key: ARROW-6352 URL: https://issues.apache.org/jira/browse/ARROW-6352 Project: Apache Arrow Issue Type: Improvement Reporter: Micah Kornfield Today only Sparse unions are supported. We should have a dense union implementation vector that conforms to the IPC protocol (the current spare union vector doesn't do this and there are other JIRAs covering making these compatible). -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (ARROW-6335) [Java] Improve the performance of DictionaryHashTable
[ https://issues.apache.org/jira/browse/ARROW-6335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Micah Kornfield resolved ARROW-6335. Fix Version/s: 0.15.0 Resolution: Fixed Issue resolved by pull request 5178 [https://github.com/apache/arrow/pull/5178] > [Java] Improve the performance of DictionaryHashTable > - > > Key: ARROW-6335 > URL: https://issues.apache.org/jira/browse/ARROW-6335 > Project: Apache Arrow > Issue Type: Improvement > Components: Java >Reporter: Liya Fan >Assignee: Liya Fan >Priority: Trivial > Labels: pull-request-available > Fix For: 0.15.0 > > Time Spent: 40m > Remaining Estimate: 0h > > when comparing two entries in the dictionary hash table, it is more efficient > to compare the index directly, rather than using Objects.equals, because they > are both ints. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (ARROW-6099) [JAVA] Has the ability to not using slf4j logging framework
[ https://issues.apache.org/jira/browse/ARROW-6099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated ARROW-6099: -- Labels: pull-request-available (was: ) > [JAVA] Has the ability to not using slf4j logging framework > --- > > Key: ARROW-6099 > URL: https://issues.apache.org/jira/browse/ARROW-6099 > Project: Apache Arrow > Issue Type: Improvement > Components: Java >Affects Versions: 0.14.1 >Reporter: Haowei Yu >Priority: Major > Labels: pull-request-available > > Currently, the java library directly calls slf4j api, and there is no > abstract layer. This leads to user need to install slf4j as a requirement > even if we don't use slf4j at all. > > It is best if you can change the slf4j dependency scope to provided and log > content only if slf4j jar file is provided at runtime. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [arrow-testing] emkornfield commented on issue #9: ARROW-6318: add generated files from integration test to testing
emkornfield commented on issue #9: ARROW-6318: add generated files from integration test to testing URL: https://github.com/apache/arrow-testing/pull/9#issuecomment-524704788 @wesm gzipped the JSON. The decimal test was quite comprehensive on precision/scale, I limited it to single value This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (ARROW-6216) [C++] Allow user to select the compression level
[ https://issues.apache.org/jira/browse/ARROW-6216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney resolved ARROW-6216. - Fix Version/s: 0.15.0 Resolution: Fixed Issue resolved by pull request 5192 [https://github.com/apache/arrow/pull/5192] > [C++] Allow user to select the compression level > > > Key: ARROW-6216 > URL: https://issues.apache.org/jira/browse/ARROW-6216 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Reporter: Martin Radev >Assignee: Martin Radev >Priority: Minor > Labels: pull-request-available > Fix For: 0.15.0 > > Original Estimate: 2h > Time Spent: 13h 40m > Remaining Estimate: 0h > > The compression level selected in Arrow for ZSTD is 1 which is the minimal > compression level for the compressor. This leads to very high compression > speed at the sacrifice of compression ratio. > The user should be allowed to select the compression level as both speed and > ratio are data specific. > The proposed solution is to expose the knob via an environment variable such > as ARROW_ZSTD_COMPRESSION_LEVEL. > Example: > export ARROW_ZSTD_COMPRESSION_LEVEL=10 > ./my_parquet_app > Here is a test run with compression levels of 1, 2 and 5: > Level Time (s) Size (mb) > 1 13.02 181 > 2 13.10 177 > 5 19.44 148 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (ARROW-6216) [C++] Allow user to select the compression level
[ https://issues.apache.org/jira/browse/ARROW-6216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney reassigned ARROW-6216: --- Assignee: Martin Radev > [C++] Allow user to select the compression level > > > Key: ARROW-6216 > URL: https://issues.apache.org/jira/browse/ARROW-6216 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Reporter: Martin Radev >Assignee: Martin Radev >Priority: Minor > Labels: pull-request-available > Original Estimate: 2h > Time Spent: 13h 40m > Remaining Estimate: 0h > > The compression level selected in Arrow for ZSTD is 1 which is the minimal > compression level for the compressor. This leads to very high compression > speed at the sacrifice of compression ratio. > The user should be allowed to select the compression level as both speed and > ratio are data specific. > The proposed solution is to expose the knob via an environment variable such > as ARROW_ZSTD_COMPRESSION_LEVEL. > Example: > export ARROW_ZSTD_COMPRESSION_LEVEL=10 > ./my_parquet_app > Here is a test run with compression levels of 1, 2 and 5: > Level Time (s) Size (mb) > 1 13.02 181 > 2 13.10 177 > 5 19.44 148 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (ARROW-6351) [Ruby] Improve Arrow#values performance
[ https://issues.apache.org/jira/browse/ARROW-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated ARROW-6351: -- Labels: pull-request-available (was: ) > [Ruby] Improve Arrow#values performance > --- > > Key: ARROW-6351 > URL: https://issues.apache.org/jira/browse/ARROW-6351 > Project: Apache Arrow > Issue Type: Improvement > Components: Ruby >Reporter: Sutou Kouhei >Assignee: Sutou Kouhei >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (ARROW-6351) [Ruby] Improve Arrow#values performance
Sutou Kouhei created ARROW-6351: --- Summary: [Ruby] Improve Arrow#values performance Key: ARROW-6351 URL: https://issues.apache.org/jira/browse/ARROW-6351 Project: Apache Arrow Issue Type: Improvement Components: Ruby Reporter: Sutou Kouhei Assignee: Sutou Kouhei -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (ARROW-6322) [C#] Implement a plasma client
[ https://issues.apache.org/jira/browse/ARROW-6322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915386#comment-16915386 ] Sutou Kouhei commented on ARROW-6322: - FYI: Plasma doesn't support Windows yet. > [C#] Implement a plasma client > -- > > Key: ARROW-6322 > URL: https://issues.apache.org/jira/browse/ARROW-6322 > Project: Apache Arrow > Issue Type: New Feature > Components: C# >Reporter: Eric Erhardt >Assignee: Eric Erhardt >Priority: Major > Original Estimate: 96h > Remaining Estimate: 96h > > We should create a C# plasma client, so .NET code can get and put objects > into the plasma store. > An easy-ish way of implementing this would be to build on the c_glib C APIs > already exposed for the plasma client. Unfortunately, I haven't found a > decent C# GObject generator, so I think the C bindings will need to be > written by hand, but there isn't too many of them. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (ARROW-6349) [Python/numpy] ArrowInvalid: Negative ndarray strides not supported
[
https://issues.apache.org/jira/browse/ARROW-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915354#comment-16915354
]
Wes McKinney commented on ARROW-6349:
-
Arrow 0.11.1 is almost a year old, I recommend using a newer version. This is
still the case in master, though
{code}
import pyarrow
import numpy as np
a=np.zeros([500,600,3])
b=a[:,:,::-1]
pyarrow.serialize(b)
## -- End pasted text --
---
ArrowInvalid Traceback (most recent call last)
in
3 a=np.zeros([500,600,3])
4 b=a[:,:,::-1]
> 5 pyarrow.serialize(b)
~/code/arrow/python/pyarrow/serialization.pxi in pyarrow.lib.serialize()
356
357 with nogil:
--> 358 check_status(SerializeObject(context, wrapped_value,
))
359 return serialized
360
~/code/arrow/python/pyarrow/error.pxi in pyarrow.lib.check_status()
76 message = frombytes(status.message())
77 if status.IsInvalid():
---> 78 raise ArrowInvalid(message)
79 elif status.IsIOError():
80 raise ArrowIOError(message)
ArrowInvalid: Negative ndarray strides not supported
In ../src/arrow/python/serialize.cc, line 510, code:
NdarrayToTensor(default_memory_pool(), reinterpret_cast(array), {},
)
In ../src/arrow/python/serialize.cc, line 464, code: AppendArray(context,
reinterpret_cast(elem), builder, recursion_depth, blobs_out)
In ../src/arrow/python/iterators.h, line 70, code: func(value,
static_cast(i), _going)
In ../src/arrow/python/serialize.cc, line 538, code: internal::VisitIterable(
sequence, [&](PyObject* obj, bool* keep_going ) { return Append(context, obj,
, 0, out); })
{code}
> [Python/numpy] ArrowInvalid: Negative ndarray strides not supported
> ---
>
> Key: ARROW-6349
> URL: https://issues.apache.org/jira/browse/ARROW-6349
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Environment: Ubuntu 16.04, python 3.6.6, numpy 1.14.6, pyarrow 0.11.1
>Reporter: YUTING ZHANG
>Priority: Major
>
> When indexing a np.ndarray with negative stride, arrow cannot serialize it.
> However, this could be a common use case for reversing along an axis.
>
> Code:
> {code:java}
> import pyarrow
> import numpy as np
> a=np.zeros([500,600,3])
> b=a[:,:,::-1]
> pyarrow.serialize(b)
> {code}
>
> Error:
> {code:java}
> ArrowInvalid Traceback (most recent call last)
> in ()
> > 1 pyarrow.serialize(b)
> ~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/pyarrow/serialization.pxi
> in pyarrow.lib.serialize()
> ~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/pyarrow/error.pxi in
> pyarrow.lib.check_status()
> ArrowInvalid: Negative ndarray strides not supported
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Resolved] (ARROW-6350) [Ruby] Remove Arrow::Struct and use Hash instead
[ https://issues.apache.org/jira/browse/ARROW-6350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yosuke Shiro resolved ARROW-6350. - Fix Version/s: 0.15.0 Resolution: Fixed Issue resolved by pull request 5190 [https://github.com/apache/arrow/pull/5190] > [Ruby] Remove Arrow::Struct and use Hash instead > > > Key: ARROW-6350 > URL: https://issues.apache.org/jira/browse/ARROW-6350 > Project: Apache Arrow > Issue Type: Improvement > Components: Ruby >Reporter: Sutou Kouhei >Assignee: Sutou Kouhei >Priority: Major > Labels: pull-request-available > Fix For: 0.15.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (ARROW-6338) [R] Type function names don't match type names
[
https://issues.apache.org/jira/browse/ARROW-6338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915298#comment-16915298
]
Benjamin Kietzman commented on ARROW-6338:
--
When writing the C API "double" can't be used because it's reserved, so we use
"float64" for the DataType factory and "DoubleArray", "DoubleType",
"Type::DOUBLE" everywhere else because those variations are not reserved.
Similarly, when deciding what string to return from DoubleType::name(),
"double" was chosen just because that's what one calls a 64 bit floating point
number in C. I don't believe it would break anything to amend
DoubleType::name() to return "float64" instead.
> [R] Type function names don't match type names
> --
>
> Key: ARROW-6338
> URL: https://issues.apache.org/jira/browse/ARROW-6338
> Project: Apache Arrow
> Issue Type: Improvement
> Components: R
>Reporter: Neal Richardson
>Assignee: Neal Richardson
>Priority: Major
> Fix For: 0.15.0
>
>
> I noticed this while working on documentation for ARROW-5505, trying to show
> how you could pass an explicit schema definition to make a table. For a few
> types, the name of the type that gets printed (and comes from the C++
> library) doesn't match the name of the function you use to specify the type
> in a schema:
> {code:r}
> > tab <- to_arrow(data.frame(
> + a = 1:10,
> + b = as.numeric(1:10),
> + c = sample(c(TRUE, FALSE, NA), 10, replace = TRUE),
> + d = letters[1:10],
> + stringsAsFactors = FALSE
> + ))
> > tab$schema
> arrow::Schema
> a: int32
> b: double
> c: bool
> d: string
> # Alright, let's make that schema
> > schema(a = int32(), b = double(), c = bool(), d = string())
> Error in bool() : could not find function "bool"
> # Hmm, ok, so bool --> boolean()
> > schema(a = int32(), b = double(), c = boolean(), d = string())
> Error in string() : could not find function "string"
> # string --> utf8()
> > schema(a = int32(), b = double(), c = boolean(), d = utf8())
> Error: type does not inherit from class arrow::DataType
> # Wha?
> > double()
> numeric(0)
> # Oh. double is a base R function.
> > schema(a = int32(), b = float64(), c = boolean(), d = utf8())
> arrow::Schema
> a: int32
> b: double
> c: bool
> d: string
> {code}
> If you believe this switch statement is correct, these three, along with
> float and half_float, are the only mismatches:
> [https://github.com/apache/arrow/blob/master/r/R/R6.R#L81-L109]
> {code:r}
> > schema(b = float64(), c = boolean(), d = utf8(), e = float32(), f =
> > float16())
> arrow::Schema
> b: double
> c: bool
> d: string
> e: float
> f: halffloat
> {code}
> I can add aliases (i.e. another function that does the same thing) for bool,
> string, float, and halffloat, and I can add some magic so that double() (and
> even integer()) work inside the schema() function. But in looking into the
> C++ side to confirm where these alternate type names were coming from, I saw
> some inconsistencies. For example,
> https://github.com/apache/arrow/blob/master/cpp/src/arrow/type.h#L773-L788
> suggests that the StringType should report its name as "utf8". But the
> ToString method here
> https://github.com/apache/arrow/blob/master/cpp/src/arrow/type.cc#L191 has it
> report as "string". It's unclear why those should report differently.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (ARROW-6348) [R] arrow::read_csv_arrow namespace error when package not loaded
[
https://issues.apache.org/jira/browse/ARROW-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915254#comment-16915254
]
Neal Richardson commented on ARROW-6348:
Yeah, I hit that last week too. Will be fixed by ARROW-6323. You can work
around in the meantime by using {{normalizePath()}} if you have a tilde in your
file path.
> [R] arrow::read_csv_arrow namespace error when package not loaded
> -
>
> Key: ARROW-6348
> URL: https://issues.apache.org/jira/browse/ARROW-6348
> Project: Apache Arrow
> Issue Type: Bug
> Components: R
>Reporter: hugh marera
>Priority: Minor
> Fix For: 0.15.0
>
>
> {quote}I am not sure if the arrow::read_csv_arrow() error below is a bug or a
> feature?
>
> data("iris")
> write.csv(iris, "iris.csv")
> test <- arrow::read_csv_arrow("iris.csv")
> Error in read_delim_arrow(file = "iris.csv", delim = ",") :
> could not find function "read_delim_arrow"
> test <- arrow::read_delim_arrow("iris.csv")
> sessionInfo()
> R version 3.6.1 (2019-07-05)
> Platform: x86_64-apple-darwin18.6.0 (64-bit)
> Running under: macOS Mojave 10.14.6
> {quote}
> Matrix products: default
> BLAS:
> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
> LAPACK: /usr/local/Cellar/openblas/0.3.7/lib/libopenblasp-r0.3.7.dylib
> locale:
> [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
> attached base packages:
> [1] stats graphics grDevices utils datasets
> [6] methods base
> loaded via a namespace (and not attached):
> [1] tidyselect_0.2.5 bit_1.1-14 compiler_3.6.1
> [4] magrittr_1.5 assertthat_0.2.1 R6_2.4.0
> [7] tools_3.6.1 fs_1.3.1 glue_1.3.1
> [10] Rcpp_1.0.2 bit64_0.9-7 arrow_0.14.1.1
> [13] rlang_0.4.0 purrr_0.3.2
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (ARROW-6090) [Rust] [DataFusion] Implement parallel execution for hash aggregate
[ https://issues.apache.org/jira/browse/ARROW-6090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated ARROW-6090: -- Labels: pull-request-available (was: ) > [Rust] [DataFusion] Implement parallel execution for hash aggregate > --- > > Key: ARROW-6090 > URL: https://issues.apache.org/jira/browse/ARROW-6090 > Project: Apache Arrow > Issue Type: Sub-task > Components: Rust - DataFusion >Reporter: Andy Grove >Assignee: Andy Grove >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (ARROW-6348) [R] arrow::read_csv_arrow namespace error when package not loaded
[
https://issues.apache.org/jira/browse/ARROW-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915172#comment-16915172
]
hugh marera commented on ARROW-6348:
I am now having a slightly different error.. appears to be caused by "~/"?
> library("arrow")
> data("iris")
> write.csv(iris, "~/iris.csv", row.names = FALSE)
> test1 <- arrow::read_csv_arrow("~/iris.csv")
Error in io___MemoryMappedFile__Open(fs::path_abs(path), mode) :
IOError: Failed to open local file '~/iris.csv', error: No such file or
directory
> test2 <- read.csv("~/iris.csv")
> [R] arrow::read_csv_arrow namespace error when package not loaded
> -
>
> Key: ARROW-6348
> URL: https://issues.apache.org/jira/browse/ARROW-6348
> Project: Apache Arrow
> Issue Type: Bug
> Components: R
>Reporter: hugh marera
>Priority: Minor
> Fix For: 0.15.0
>
>
> {quote}I am not sure if the arrow::read_csv_arrow() error below is a bug or a
> feature?
>
> data("iris")
> write.csv(iris, "iris.csv")
> test <- arrow::read_csv_arrow("iris.csv")
> Error in read_delim_arrow(file = "iris.csv", delim = ",") :
> could not find function "read_delim_arrow"
> test <- arrow::read_delim_arrow("iris.csv")
> sessionInfo()
> R version 3.6.1 (2019-07-05)
> Platform: x86_64-apple-darwin18.6.0 (64-bit)
> Running under: macOS Mojave 10.14.6
> {quote}
> Matrix products: default
> BLAS:
> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
> LAPACK: /usr/local/Cellar/openblas/0.3.7/lib/libopenblasp-r0.3.7.dylib
> locale:
> [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
> attached base packages:
> [1] stats graphics grDevices utils datasets
> [6] methods base
> loaded via a namespace (and not attached):
> [1] tidyselect_0.2.5 bit_1.1-14 compiler_3.6.1
> [4] magrittr_1.5 assertthat_0.2.1 R6_2.4.0
> [7] tools_3.6.1 fs_1.3.1 glue_1.3.1
> [10] Rcpp_1.0.2 bit64_0.9-7 arrow_0.14.1.1
> [13] rlang_0.4.0 purrr_0.3.2
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (ARROW-6350) [Ruby] Remove Arrow::Struct and use Hash instead
[ https://issues.apache.org/jira/browse/ARROW-6350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated ARROW-6350: -- Labels: pull-request-available (was: ) > [Ruby] Remove Arrow::Struct and use Hash instead > > > Key: ARROW-6350 > URL: https://issues.apache.org/jira/browse/ARROW-6350 > Project: Apache Arrow > Issue Type: Improvement > Components: Ruby >Reporter: Sutou Kouhei >Assignee: Sutou Kouhei >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (ARROW-6350) [Ruby] Remove Arrow::Struct and use Hash instead
Sutou Kouhei created ARROW-6350: --- Summary: [Ruby] Remove Arrow::Struct and use Hash instead Key: ARROW-6350 URL: https://issues.apache.org/jira/browse/ARROW-6350 Project: Apache Arrow Issue Type: Improvement Components: Ruby Reporter: Sutou Kouhei Assignee: Sutou Kouhei -- This message was sent by Atlassian Jira (v8.3.2#803003)
