[
https://issues.apache.org/jira/browse/HUDI-8902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Y Ethan Guo updated HUDI-8902:
------------------------------
Description:
After adding stricter data validation in the test `TestSpark3DDL`.`Test alter
table properties and add rename drop column with table services`, we see that
reading the field value field after promotion from float to double could be
different between COW, MOR, before and after compaction and clustering. The
value has precision difference, e.g., 102.0199966430664 vs 102.02. Right now
the test has to differentiate this by using "getDouble". We should look into
this and see if this is expected, and if the behavior can be made consistent.
Note that running the same test without enabling the fg reader in compaction or
clustering for schema on read also pass. So this is related to existing
discrepancy on how we evolve the field from float to double type.
was:After adding stricter data validation in the test `TestSpark3DDL`.`Test
alter table properties and add rename drop column with table services`, we see
that reading the field value field after promotion from float to double could
be different between COW, MOR, before and after compaction and clustering. The
value has precision difference, e.g., 102.0199966430664 vs 102.02. Right now
the test has to differentiate this by using "getDouble". We should look into
this and see if this is expected, and if the behavior can be made consistent.
> Investigate different read behavior on a field after promotion from float to
> double
> -----------------------------------------------------------------------------------
>
> Key: HUDI-8902
> URL: https://issues.apache.org/jira/browse/HUDI-8902
> Project: Apache Hudi
> Issue Type: Sub-task
> Reporter: Y Ethan Guo
> Assignee: Davis Zhang
> Priority: Critical
> Fix For: 1.0.1
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> After adding stricter data validation in the test `TestSpark3DDL`.`Test alter
> table properties and add rename drop column with table services`, we see that
> reading the field value field after promotion from float to double could be
> different between COW, MOR, before and after compaction and clustering. The
> value has precision difference, e.g., 102.0199966430664 vs 102.02. Right now
> the test has to differentiate this by using "getDouble". We should look into
> this and see if this is expected, and if the behavior can be made consistent.
> Note that running the same test without enabling the fg reader in compaction
> or clustering for schema on read also pass. So this is related to existing
> discrepancy on how we evolve the field from float to double type.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
