[
https://issues.apache.org/jira/browse/ARROW-6425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yan Zhou updated ARROW-6425:
----------------------------
Description:
The ValidateArray function fails for slice of list array (and MapArray I
believe though haven't tested it yet). The validation will only pass for an
unsliced array or the last slice.
For example,
{code:java}
auto value_builder = std::make_shared<::arrow::DoubleBuilder>();
::arrow::ListBuilder builder(
::arrow::default_memory_pool(), value_builder);
auto status = builder.Append(true);
status = value_builder->Append(1);
status = builder.Append(true);
status = value_builder->Append(2);
std::shared_ptr<::arrow::Array> list;
status = builder.Finish(&list);
std::cout << ::arrow::ValidateArray(*list).ok() << std::endl;
std::cout << ::arrow::ValidateArray(*list->Slice(0, 1)).ok() << std::endl;
std::cout << ::arrow::ValidateArray(*list->Slice(1, 1)).ok() << std::endl;
{code}
This will print 1, 0, 1.
The problem lies in array.cc around line 1167
{code:java}
const int32_t last_offset = array.value_offset(array.length());
if (array.values()->length() != last_offset) {
return Status::Invalid("Final offset invariant not equal to values
length: ",
last_offset, "!=", array.values()->length());
}
{code}
Here values() does not account for the array offset and thus its length is the
whole array while value_offset and array.length() will account for that offset.
was:
The ValidateArray function fails for slice of list array (and MapArray I
believe though haven't tested it yet). The validation will only pass for an
unsliced array or the last slice.
For example,
{code:java}
auto value_builder = std::make_shared<::arrow::DoubleBuilder>();
::arrow::ListBuilder builder( ::arrow::default_memory_pool(), value_builder);
auto status = builder.Append(true); status = value_builder->Append(1); status =
builder.Append(true); status = value_builder->Append(2);
std::shared_ptr<::arrow::Array> list; status = builder.Finish(&list); std::cout
<< ::arrow::ValidateArray(*list).ok() << std::endl; std::cout <<
::arrow::ValidateArray(*list->Slice(0, 1)).ok() << std::endl; std::cout <<
::arrow::ValidateArray(*list->Slice(1, 1)).ok() << std::endl;
{code}
This will print 1, 0, 1.
The problem lies in array.cc around line 1167
{code:java}
const int32_t last_offset = array.value_offset(array.length());
if (array.values()->length() != last_offset) {
return Status::Invalid("Final offset invariant not equal to values
length: ",
last_offset, "!=", array.values()->length());
}
{code}
Here values() does not account for the array offset and thus its length is the
whole array while value_offset and array.length() will account for that offset.
> ValidateArray fail for slice of list array
> -------------------------------------------
>
> Key: ARROW-6425
> URL: https://issues.apache.org/jira/browse/ARROW-6425
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
> Affects Versions: 0.14.1
> Reporter: Yan Zhou
> Priority: Major
>
> The ValidateArray function fails for slice of list array (and MapArray I
> believe though haven't tested it yet). The validation will only pass for an
> unsliced array or the last slice.
>
> For example,
>
>
> {code:java}
> auto value_builder = std::make_shared<::arrow::DoubleBuilder>();
> ::arrow::ListBuilder builder(
> ::arrow::default_memory_pool(), value_builder);
> auto status = builder.Append(true);
> status = value_builder->Append(1);
> status = builder.Append(true);
> status = value_builder->Append(2);
> std::shared_ptr<::arrow::Array> list;
> status = builder.Finish(&list);
> std::cout << ::arrow::ValidateArray(*list).ok() << std::endl;
> std::cout << ::arrow::ValidateArray(*list->Slice(0, 1)).ok() << std::endl;
> std::cout << ::arrow::ValidateArray(*list->Slice(1, 1)).ok() << std::endl;
> {code}
>
> This will print 1, 0, 1.
>
> The problem lies in array.cc around line 1167
>
> {code:java}
> const int32_t last_offset = array.value_offset(array.length());
> if (array.values()->length() != last_offset) {
> return Status::Invalid("Final offset invariant not equal to values
> length: ",
> last_offset, "!=", array.values()->length());
> }
> {code}
> Here values() does not account for the array offset and thus its length is
> the whole array while value_offset and array.length() will account for that
> offset.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
