[
https://issues.apache.org/jira/browse/HUDI-8150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17890770#comment-17890770
]
Y Ethan Guo edited comment on HUDI-8150 at 10/18/24 2:02 AM:
-------------------------------------------------------------
In Hudi 1.0, we'll maintain the
On master: we don't write the payload class name to hoodie.properties if not
set explicitly.
- payload class set (user sets it) in hoodie.properties -> Spark SQL writer
uses it and skips payload deduction logic (got overriden)
- payload class not set (using default) in hoodie.properties -> Spark SQL
deducts the payload class (can be different from default) to use
SQL writer merge behavior:
- (fine with master and the PR) MIT should always use the expression payload,
because MIT statement dictates the update/merge behavior.
- (problem to solve) INSERT INTO has some complex logic on determining
internal payload to use
{code:java}
val payloadClassName = if (useLegacyInsertDropDupFlow) {
deducePayloadClassNameLegacy(operation, tableType, insertMode)
} else {
if (insertDupPolicy == FAIL_INSERT_DUP_POLICY) {
classOf[ValidateDuplicateKeyPayload].getCanonicalName
} else {
classOf[DefaultHoodieRecordPayload].getCanonicalName
}
}
private def deducePayloadClassNameLegacy(operation: String, tableType: String,
insertMode: InsertMode): String = {
if (operation == UPSERT_OPERATION_OPT_VAL &&
tableType == COW_TABLE_TYPE_OPT_VAL && insertMode == InsertMode.STRICT) {
// Validate duplicate key for COW, for MOR it will do the merge with the
DefaultHoodieRecordPayload
// on reading.
// TODO use HoodieSparkValidateDuplicateKeyRecordMerger when
SparkRecordMerger is default
classOf[ValidateDuplicateKeyPayload].getCanonicalName
} else if (operation == INSERT_OPERATION_OPT_VAL && tableType ==
COW_TABLE_TYPE_OPT_VAL &&
insertMode == InsertMode.STRICT) {
// Validate duplicate key for inserts to COW table when using strict insert
mode.
classOf[ValidateDuplicateKeyPayload].getCanonicalName
} else {
classOf[DefaultHoodieRecordPayload].getCanonicalName
}
}{code}
- master branch follows this
- In the PR, there is currently a conflict between payload config in table
config vs the deducted payload, causing behavior change
- (fine with master and the PR) All other DMLs use custom payload avaiable.
In the new behavior / merge mode PR:
- Always merge mode config into hoodie.properties
Proposal 1
- Maintain the behavior, where we don't write the payload class name to
hoodie.properties if not set explicitly.
- always set merge mode table config others (CUSTOM: custom payload class is
set always, and OVERWRITE_WITH_LATEST and EVENT_TIME_ORDERING we can infer from
merge mode, i.e., we know the relationship between merge mode and payload class)
- In table config,
- For Spark SQL, do not add the payload class name when loading the
HoodieTableConfig
- For other places, still attach the payload from the Table Configs
Proposal 2 (go with this)
- (Prefer this) Always write the payload class config in hoodie.properties
(what in the current PR)
- For Spark SQL INSERT INTO,
- for CUSTOM payload, always use it which is the current behavior
- (change needed) if it's default hoodie payload, use deduction logic without
honoring the table config
was (Author: JIRAUSER280684):
In Hudi 1.0
On master: we don't write the payload class name to hoodie.properties if not
set explicitly.
- payload class set (user sets it) in hoodie.properties -> Spark SQL writer
uses it and skips payload deduction logic (got overriden)
- payload class not set (using default) in hoodie.properties -> Spark SQL
deducts the payload class (can be different from default) to use
SQL writer merge behavior:
- (fine with master and the PR) MIT should always use the expression payload,
because MIT statement dictates the update/merge behavior.
- (problem to solve) INSERT INTO has some complex logic on determining internal
payload to use
{code:java}
val payloadClassName = if (useLegacyInsertDropDupFlow) {
deducePayloadClassNameLegacy(operation, tableType, insertMode)
} else {
if (insertDupPolicy == FAIL_INSERT_DUP_POLICY) {
classOf[ValidateDuplicateKeyPayload].getCanonicalName
} else {
classOf[DefaultHoodieRecordPayload].getCanonicalName
}
}
private def deducePayloadClassNameLegacy(operation: String, tableType: String,
insertMode: InsertMode): String = {
if (operation == UPSERT_OPERATION_OPT_VAL &&
tableType == COW_TABLE_TYPE_OPT_VAL && insertMode == InsertMode.STRICT) {
// Validate duplicate key for COW, for MOR it will do the merge with the
DefaultHoodieRecordPayload
// on reading.
// TODO use HoodieSparkValidateDuplicateKeyRecordMerger when
SparkRecordMerger is default
classOf[ValidateDuplicateKeyPayload].getCanonicalName
} else if (operation == INSERT_OPERATION_OPT_VAL && tableType ==
COW_TABLE_TYPE_OPT_VAL &&
insertMode == InsertMode.STRICT) {
// Validate duplicate key for inserts to COW table when using strict insert
mode.
classOf[ValidateDuplicateKeyPayload].getCanonicalName
} else {
classOf[DefaultHoodieRecordPayload].getCanonicalName
}
}{code}
- master branch follows this
- In the PR, there is currently a conflict between payload config in table
config vs the deducted payload, causing behavior change
- (fine with master and the PR) All other DMLs use custom payload avaiable.
In the new behavior / merge mode PR:
- Always merge mode config into hoodie.properties
Proposal 1
- Maintain the behavior, where we don't write the payload class name to
hoodie.properties if not set explicitly.
- always set merge mode table config others (CUSTOM: custom payload class is
set always, and OVERWRITE_WITH_LATEST and EVENT_TIME_ORDERING we can infer from
merge mode, i.e., we know the relationship between merge mode and payload class)
- In table config,
- For Spark SQL, do not add the payload class name when loading the
HoodieTableConfig
- For other places, still attach the payload from the Table Configs
Proposal 2 (go with this)
- (Prefer this) Always write the payload class config in hoodie.properties
(what in the current PR)
- For Spark SQL INSERT INTO,
- for CUSTOM payload, always use it which is the current behavior
- (change needed) if it's default hoodie payload, use deduction logic without
honoring the table config
> Review semantics of SQL and RecordMerger interplay
> --------------------------------------------------
>
> Key: HUDI-8150
> URL: https://issues.apache.org/jira/browse/HUDI-8150
> Project: Apache Hudi
> Issue Type: New Feature
> Reporter: Ethan Guo (this is the old account; please use "yihua")
> Assignee: Y Ethan Guo
> Priority: Blocker
> Fix For: 1.0.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
