deniskuzZ commented on code in PR #6701: URL: https://github.com/apache/hive/pull/6701#discussion_r3892696130
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java:
##########
@@ -640,6 +641,7 @@ protected void initializeOp(Configuration hconf) throws
HiveException {
jc = new JobConf(hconf);
setWriteOperation(jc, getConf().getTableInfo().getTableName(),
getConf().getWriteOperation());
+ setIsMergeStatement(jc, getConf().getTableInfo().getTableName(),
getConf().isMergeStatement());
Review Comment:
WriterBuilder re-derives COW/MOR from table properties at runtime, and it
must land on the same answer the rewriter factory already computed at plan time:
````
┌────────────────────┬────────────────────────────┬──────────────────────────────┐
│ statement │ factory decision │ FS writeOperation per
branch │
├────────────────────┼────────────────────────────┼──────────────────────────────┤
│ DELETE │ shouldOverwrite(t, DELETE) │ DELETE ✓
│
├────────────────────┼────────────────────────────┼──────────────────────────────┤
│ UPDATE (COW) │ shouldOverwrite(t, UPDATE) │ UPDATE/INSERT ✓
│
├────────────────────┼────────────────────────────┼──────────────────────────────┤
│ UPDATE (split MOR) │ shouldOverwrite(t, UPDATE) │ INSERT + DELETE ✗
│
├────────────────────┼────────────────────────────┼──────────────────────────────┤
│ MERGE (COW) │ shouldOverwrite(t, MERGE) │ MERGE ✓
│
├────────────────────┼────────────────────────────┼──────────────────────────────┤
│ MERGE (split MOR) │ shouldOverwrite(t, MERGE) │ INSERT + DELETE ✗
│
└────────────────────┴────────────────────────────┴──────────────────────────────┘
````
Suggested shape
// FileSinkDesc — replace isMergeStatement
private Context.Operation statementOperation = Context.Operation.OTHER;
// + get/setStatementOperation, and add BOTH to clone() — writeOperation
is missing there today
// SemanticAnalyzer.createFileSinkDesc— next to the existing
setWriteOperation
fileSinkDesc.setStatementOperation(ctx.getOperation());
// HiveCustomStorageHandlerUtils
public static final String STATEMENT_OPERATION_CONFIG_PREFIX =
"file.sink.statement.operation.";
// setStatementOperation / getStatementOperation, mirroring
setWriteOperation/getWriteOperation
// FileSinkOperator.initializeOp — next to the existing setWriteOperation
setStatementOperation(jc, getConf().getTableInfo().getTableName(),
getConf().getStatementOperation());
// WriterBuilder — ctor
this.statementOperation =
HiveCustomStorageHandlerUtils.getStatementOperation(ops, tableName);
// WriterBuilder.build()
boolean isCOW = IcebergTableUtil.isCopyOnWriteMode(
ObjectUtils.defaultIfNull(statementOperation, operation),
table.properties()::getOrDefault);
cc @kasakrisz
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
