[
https://issues.apache.org/jira/browse/ARROW-1992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16338640#comment-16338640
]
ASF GitHub Bot commented on ARROW-1992:
---------------------------------------
wesm commented on a change in pull request #1508: ARROW-1992: [C++/Python] Fix
segfault when string to categorical empty string array
URL: https://github.com/apache/arrow/pull/1508#discussion_r163738105
##########
File path: cpp/src/arrow/compute/kernels/hash.cc
##########
@@ -406,20 +406,29 @@ class HashTableKernel<Type, Action,
enable_if_binary<Type>> : public HashTable {
}
Status Append(const ArrayData& arr) override {
+ constexpr uint8_t empty_value = 0;
if (!initialized_) {
RETURN_NOT_OK(Init());
}
const int32_t* offsets = GetValues<int32_t>(arr, 1);
- const uint8_t* data = GetValues<uint8_t>(arr, 2);
+ const uint8_t* data;
+ if (arr.buffers[2].get() == nullptr) {
+ data = &empty_value;
+ } else {
+ data = GetValues<uint8_t>(arr, 2);
+ }
auto action = static_cast<Action*>(this);
RETURN_NOT_OK(action->Reserve(arr.length));
#define HASH_INNER_LOOP()
\
const int32_t position = offsets[i];
\
const int32_t length = offsets[i + 1] - position;
\
- const uint8_t* value = data + position;
\
+ const uint8_t* value = data;
\
+ if (value != &empty_value) {
\
+ value += position;
\
Review comment:
When this case comes up, `position` is always going to be 0, so we could
maybe handle this by substituting a dummy static buffer for `data` above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> [Python] to_pandas crashes when using strings_to_categoricals on empty string
> cols on 0.8.0
> -------------------------------------------------------------------------------------------
>
> Key: ARROW-1992
> URL: https://issues.apache.org/jira/browse/ARROW-1992
> Project: Apache Arrow
> Issue Type: Bug
> Affects Versions: 0.8.0
> Environment: OS: Windows
> Python: PY36 x64
> Pandas: 0.22.0
> pyarrow: 0.8.0
> Reporter: Victor Uriarte
> Assignee: Licht Takeuchi
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.9.0
>
>
> When trying to read back a table, Python crashes when pyarrow is used to
> read/convert a table that has a column of 0 length `strings and
> strings_to_categorical=True`. Example code below.
> This same test ran ok with pyarrow 0.7.1
> {code:none}
> import pandas as pd
> import pyarrow as pa
> df = pd.DataFrame({
> 'Foo': ['A', 'A', 'B', 'B', 'C'],
> 'Bar': ['A1', 'A2', 'B2', 'D3', ''],
> 'Baz': ['', '', '', '', ''],
> })
> table = pa.Table.from_pandas(df)
> df = table.to_pandas(strings_to_categorical=False) # Works
> print('Categoricals=False', len(df))
> df = table.to_pandas(strings_to_categorical=True) # Crashes
> print('Categoricals=True', len(df))
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
