[
https://issues.apache.org/jira/browse/HUDI-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Y Ethan Guo updated HUDI-8425:
------------------------------
Description:
[https://github.com/apache/hudi/pull/11943#discussion_r1805676748]
Currently we set default ordering value if the precombine/ordering field is not
set, so the natural ordering is used (based on processing or instant time).
This is fine given it's only used on the reader path. However, there's a risk
of writing the default ordering field to the storage if the file group reader
is used on the writer side for compaction, clustering, etc. We should revisit
precombine/ordering value passing in the file group reader and record buffer to
see if we can get rid of such default ordering value and let the record merger
handle the cases that the ordering value is null / not present directly.
was:
[https://github.com/apache/hudi/pull/11943#discussion_r1805676748]
Currently we set default ordering value if the precombine/ordering field is not
set, so the natural ordering is used (based on processing or instant time).
This is fine given it's only used on the reader path. However, there's a risk
of writing the default ordering field to the storage if the file group reader
is used on the writer side for compaction, clustering, etc. We should revisit
> Revisit precombine/ordering value passing in the file group reader and record
> buffer
> ------------------------------------------------------------------------------------
>
> Key: HUDI-8425
> URL: https://issues.apache.org/jira/browse/HUDI-8425
> Project: Apache Hudi
> Issue Type: Improvement
> Reporter: Y Ethan Guo
> Priority: Blocker
> Fix For: 1.1.0
>
>
> [https://github.com/apache/hudi/pull/11943#discussion_r1805676748]
> Currently we set default ordering value if the precombine/ordering field is
> not set, so the natural ordering is used (based on processing or instant
> time). This is fine given it's only used on the reader path. However,
> there's a risk of writing the default ordering field to the storage if the
> file group reader is used on the writer side for compaction, clustering, etc.
> We should revisit precombine/ordering value passing in the file group reader
> and record buffer to see if we can get rid of such default ordering value and
> let the record merger handle the cases that the ordering value is null / not
> present directly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
