[
https://issues.apache.org/jira/browse/IMPALA-12606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798596#comment-17798596
]
ASF subversion and git services commented on IMPALA-12606:
----------------------------------------------------------
Commit b6ce98b0b60c12869b2cd88db481cbd16bbc21fe in impala's branch
refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=b6ce98b0b ]
IMPALA-12606: Sporadic failures around
query_test.test_queries.TestQueries.test_intersect
test_intersect failed when ASYNC_CODEGEN was enabled. This happened
because we were using codegened 'ProcessProbeBatch' in the HASH JOIN
operator with non-codegened InsertBatch/ProcessBuildBatch at the Builder
side, or vice versa.
Only the NULL StringValues were hit by the bug, turned out NULLs are
handled differently in the hash table. We are using the
HashUtil::FNV_SEED number to represent NULL values. This number was
chosen arbitrarily, we just wanted to use some random value.
In the codegened path we invoke StringValue::Assign(ptr, len) with both
params being HashUtil::FNV_SEED. HashUtil::FNV_SEED is a negative value
in int32_t, so StringValue::Assign(ptr, len) stored 0 as len actually,
and not HashUtil::FNV_SEED. This is needed to be resilient against
invalid values in corrupt files.
In non-codegened path we are creating a StringValue object by
reinterpret casting a [HashUtil::FNV_SEED, HashUtil::FNV_SEED, ...]
array to StringValue. Then in RawValue::WriteNonNullPrimitive() we
invoke StringValue::Assign(StringValue&) that just memcopies the
parameter to this. It cannot check for negative values, because the
parameter StringValue might be a valid small string.
To sum up, this is how a NULL string was represented in the HashTable:
* Codegen path: ptr = HashUtil::FNV_SEED, len = 0
* Non-codegen path: ptr = HashUtil::FNV_SEED, len = HashUtil::FNV_SEED
This is why the hash join operator was working incorrectly on NULL
String values when some parts of it used Codegen'ed path while other
parts were using the non-codegened path.
To fix the issue, I introduced UnsafeAssign(ptr, len) which doesn't
do any checks for 'ptr' or 'len', so we have the same StringValue for
objects for NULLs at both the codegened and non-codegened paths.
Testing:
* Executed TestQueries.test_intersect multiple times
Change-Id: I6b855c59808db80fd7ac596ce338fc4c3c9c7667
Reviewed-on: http://gerrit.cloudera.org:8080/20793
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Sporadic failures around query_test.test_queries.TestQueries.test_intersect
> ---------------------------------------------------------------------------
>
> Key: IMPALA-12606
> URL: https://issues.apache.org/jira/browse/IMPALA-12606
> Project: IMPALA
> Issue Type: Bug
> Affects Versions: Impala 4.4.0
> Reporter: Peter Rozsa
> Assignee: Zoltán Borók-Nagy
> Priority: Critical
>
> test_intersect fails with the following setup randomly through multiple
> executions:
> {code:java}
> set
> DEBUG_ACTION="BEFORE_CODEGEN_IN_ASYNC_CODEGEN_THREAD:JITTER@1000|AFTER_STARTING_ASYNC_CODEGEN_IN_FRAGMENT_THREAD:JITTER@1000"
> {code}
> Failing query:
> {code:java}
> select group_str, null_str from nullrows where bool_nulls = true intersect
> select group_str, null_str from nullrows where bool_nulls = false {code}
> The bisecting points to 0e5374e080 which is IMPALA-12373.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
