Rohini Palaniswamy created PIG-5478:
---------------------------------------
Summary: HashValuePartitioner for Union must not hash
floating-point bits
Key: PIG-5478
URL: https://issues.apache.org/jira/browse/PIG-5478
Project: Pig
Issue Type: Bug
Reporter: Joshua Martell
Assignee: Joshua Martell
Fix For: 0.19.0
This issue applies to UNION with a PARALLEL clause and can cause silent row
loss or duplication on task retry when float or double columns are present in
the record.
The Tez UNION edge uses HashValuePartitioner, which hashes every tuple field.
Java's Double/Float hashCode is derived from the exact bit pattern
(doubleToLongBits), and a floating-point aggregate is order-of-operations
dependent. When a task attempt is re-executed (speculation, node loss, retry),
it can recompute the same logical record's float in a different order and
produce a bit-different double. That routes the re-computed copy to a different
partition, so downstream tasks that consume a mix of attempt versions silently
drop and duplicate records.
The class already skips DataBag fields for exactly this reproducibility reason.
Floats (and floats nested in tuples / map entries) were the missed case.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
