cshuo commented on code in PR #12967:
URL: https://github.com/apache/hudi/pull/12967#discussion_r2003219986
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/OptionsResolver.java:
##########
@@ -70,6 +76,32 @@ public static boolean isAppendMode(Configuration conf) {
return isInsertOperation(conf) && ((isCowTable(conf) &&
!conf.getBoolean(FlinkOptions.INSERT_CLUSTER)) || isMorTable(conf));
}
+ /**
+ * Returns whether the RowData append is enabled with given configuration
{@code conf}.
+ * <p>
+ * todo:
+ * <p> support RowData append for COW, see HUDI-9149
+ * <p> support RowData append for operation besides UPSERT, see HUDI-9149
+ * <p> support RowData append for nested complex DataType field, see
HUDI-9146, currently RowData
+ * reader for MOR table is not supported yet, Flink uses Avro parquet reader
to read parquet data
+ * blocks in log file written by RowData writer, there exists some
discrepancies between Avro parquet
+ * support and RowData parquet support regarding nested complex type,
leading to failure during reading.
+ * After the RowData reader is supported, there would be no problems.
+ */
+ public static boolean supportRowDataAppend(Configuration conf, RowType
rowType) {
+ return conf.get(FlinkOptions.INSERT_ROWDATA_MODE_ENABLED)
+ && HoodieTableType.valueOf(conf.get(FlinkOptions.TABLE_TYPE)) ==
HoodieTableType.MERGE_ON_READ
+ &&
(WriteOperationType.valueOf(conf.get(FlinkOptions.OPERATION).toUpperCase()) ==
WriteOperationType.UPSERT
+ ||
WriteOperationType.valueOf(conf.get(FlinkOptions.OPERATION).toUpperCase()) ==
WriteOperationType.DELETE)
+ && !DataTypeUtils.containsNestedComplexType(rowType);
Review Comment:
Nested complex type is not worked now because there exists discrepancies
between Avro parquet
read support and RowData parquet write support regarding nested complex
type.
If we want to fix the issue, I think we may need to update Flink RowData
Parquet writer, so that Avro parquet reader can read the nested complex type
field correctly, but this would incur compatible problem, i.e., we should make
Flink RowData parquet reader compatible to correctly old and new parquet file
containing fields of nested type. I actually prefer to put this fix in a
separate PR. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
