[
https://issues.apache.org/jira/browse/HUDI-8150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17890770#comment-17890770
]
Y Ethan Guo edited comment on HUDI-8150 at 10/21/24 9:37 PM:
-------------------------------------------------------------
In Hudi 1.0, we'll maintain the way how the Spark SQL writer and reader paths
work regarding how the custom payload classes are used to fulfill specific DML
logic. Here are more details. HUDI-8203 also needs the following information
to use the merge mode and payload classes properly in the Spark SQL.
On master and existing behavior: we don't write the payload class name to
hoodie.properties if not set explicitly.
- payload class set (user sets it) in hoodie.properties -> Spark SQL writer
uses it and skips payload deduction logic (got overriden)
- payload class not set (using default) in hoodie.properties -> Spark SQL
deducts the payload class (can be different from default) to use
HUDI-8203 SQL writer merge behavior:
- (fine with master and the PR) MIT should always use the expression payload,
because MIT statement dictates the update/merge behavior.
- (problem to solve) INSERT INTO has some complex logic on determining
internal payload to use
{code:java}
val payloadClassName = if (useLegacyInsertDropDupFlow) {
deducePayloadClassNameLegacy(operation, tableType, insertMode)
} else {
if (insertDupPolicy == FAIL_INSERT_DUP_POLICY) {
classOf[ValidateDuplicateKeyPayload].getCanonicalName
} else {
classOf[DefaultHoodieRecordPayload].getCanonicalName
}
}
private def deducePayloadClassNameLegacy(operation: String, tableType: String,
insertMode: InsertMode): String = {
if (operation == UPSERT_OPERATION_OPT_VAL &&
tableType == COW_TABLE_TYPE_OPT_VAL && insertMode == InsertMode.STRICT) {
// Validate duplicate key for COW, for MOR it will do the merge with the
DefaultHoodieRecordPayload
// on reading.
// TODO use HoodieSparkValidateDuplicateKeyRecordMerger when
SparkRecordMerger is default
classOf[ValidateDuplicateKeyPayload].getCanonicalName
} else if (operation == INSERT_OPERATION_OPT_VAL && tableType ==
COW_TABLE_TYPE_OPT_VAL &&
insertMode == InsertMode.STRICT) {
// Validate duplicate key for inserts to COW table when using strict insert
mode.
classOf[ValidateDuplicateKeyPayload].getCanonicalName
} else {
classOf[DefaultHoodieRecordPayload].getCanonicalName
}
}{code}
- master branch follows this
- In the PR, there is currently a conflict between payload config in table
config vs the deducted payload, causing behavior change
- (fine with master and the PR) All other DMLs use custom payload avaiable.
In the new behavior / merge mode PR:
- Always merge mode config into hoodie.properties
Proposal 1
- Maintain the behavior, where we don't write the payload class name to
hoodie.properties if not set explicitly.
- always set merge mode table config others (CUSTOM: custom payload class is
set always, and OVERWRITE_WITH_LATEST and EVENT_TIME_ORDERING we can infer from
merge mode, i.e., we know the relationship between merge mode and payload class)
- In table config,
- For Spark SQL, do not add the payload class name when loading the
HoodieTableConfig
- For other places, still attach the payload from the Table Configs
Proposal 2 (go with this)
- (Prefer this) Always write the payload class config in hoodie.properties
(what in the current PR)
- For Spark SQL INSERT INTO,
- for CUSTOM payload, always use it which is the current behavior
- (change needed) if it's default hoodie payload, use deduction logic without
honoring the table config
was (Author: JIRAUSER280684):
In Hudi 1.0, we'll maintain the
On master: we don't write the payload class name to hoodie.properties if not
set explicitly.
- payload class set (user sets it) in hoodie.properties -> Spark SQL writer
uses it and skips payload deduction logic (got overriden)
- payload class not set (using default) in hoodie.properties -> Spark SQL
deducts the payload class (can be different from default) to use
SQL writer merge behavior:
- (fine with master and the PR) MIT should always use the expression payload,
because MIT statement dictates the update/merge behavior.
- (problem to solve) INSERT INTO has some complex logic on determining
internal payload to use
{code:java}
val payloadClassName = if (useLegacyInsertDropDupFlow) {
deducePayloadClassNameLegacy(operation, tableType, insertMode)
} else {
if (insertDupPolicy == FAIL_INSERT_DUP_POLICY) {
classOf[ValidateDuplicateKeyPayload].getCanonicalName
} else {
classOf[DefaultHoodieRecordPayload].getCanonicalName
}
}
private def deducePayloadClassNameLegacy(operation: String, tableType: String,
insertMode: InsertMode): String = {
if (operation == UPSERT_OPERATION_OPT_VAL &&
tableType == COW_TABLE_TYPE_OPT_VAL && insertMode == InsertMode.STRICT) {
// Validate duplicate key for COW, for MOR it will do the merge with the
DefaultHoodieRecordPayload
// on reading.
// TODO use HoodieSparkValidateDuplicateKeyRecordMerger when
SparkRecordMerger is default
classOf[ValidateDuplicateKeyPayload].getCanonicalName
} else if (operation == INSERT_OPERATION_OPT_VAL && tableType ==
COW_TABLE_TYPE_OPT_VAL &&
insertMode == InsertMode.STRICT) {
// Validate duplicate key for inserts to COW table when using strict insert
mode.
classOf[ValidateDuplicateKeyPayload].getCanonicalName
} else {
classOf[DefaultHoodieRecordPayload].getCanonicalName
}
}{code}
- master branch follows this
- In the PR, there is currently a conflict between payload config in table
config vs the deducted payload, causing behavior change
- (fine with master and the PR) All other DMLs use custom payload avaiable.
In the new behavior / merge mode PR:
- Always merge mode config into hoodie.properties
Proposal 1
- Maintain the behavior, where we don't write the payload class name to
hoodie.properties if not set explicitly.
- always set merge mode table config others (CUSTOM: custom payload class is
set always, and OVERWRITE_WITH_LATEST and EVENT_TIME_ORDERING we can infer from
merge mode, i.e., we know the relationship between merge mode and payload class)
- In table config,
- For Spark SQL, do not add the payload class name when loading the
HoodieTableConfig
- For other places, still attach the payload from the Table Configs
Proposal 2 (go with this)
- (Prefer this) Always write the payload class config in hoodie.properties
(what in the current PR)
- For Spark SQL INSERT INTO,
- for CUSTOM payload, always use it which is the current behavior
- (change needed) if it's default hoodie payload, use deduction logic without
honoring the table config
> Review semantics of SQL and RecordMerger interplay
> --------------------------------------------------
>
> Key: HUDI-8150
> URL: https://issues.apache.org/jira/browse/HUDI-8150
> Project: Apache Hudi
> Issue Type: New Feature
> Reporter: Ethan Guo (this is the old account; please use "yihua")
> Assignee: Y Ethan Guo
> Priority: Blocker
> Fix For: 1.0.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
