prashantwason commented on code in PR #17777:
URL: https://github.com/apache/hudi/pull/17777#discussion_r2738249607
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/HoodieStreamer.java:
##########
@@ -122,6 +122,12 @@ public class HoodieStreamer implements Serializable {
public static final String CHECKPOINT_KEY =
HoodieWriteConfig.STREAMER_CHECKPOINT_KEY;
public static final String CHECKPOINT_RESET_KEY =
STREAMER_CHECKPOINT_RESET_KEY_V1;
+ // Ensure that all columns in the schema are nullable. This is required to
keep the schema backwards compatible when
+ // new columns are added via SQL queries (e.g. When using SQLSource or
SQLFileBasedSource as source of records
+ // written into a table.
+ public static final String ENSURE_ALL_COLUMNS_NULLABLE_KEY =
"hoodie.streamer.transformed.row.nullable";
Review Comment:
Done. Moved to `HoodieStreamerConfig` as a `ConfigProperty<Boolean>` with:
- Primary key: `hoodie.streamer.transformed.row.nullable`
- Alternative key: `hoodie.deltastreamer.transformed.row.nullable`
Also added unit tests in `TestUtilHelpers.java`.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java:
##########
@@ -728,9 +729,10 @@ Pair<InputBatch, Boolean>
fetchNextBatchFromSource(Option<Checkpoint> resumeChec
} else {
// Deduce proper target (writer's) schema for the input dataset,
reconciling its
// schema w/ the table's one
- HoodieSchema incomingSchema = transformed.map(df ->
-
HoodieSchemaConversionUtils.convertStructTypeToHoodieSchema(df.schema(),
getRecordQualifiedName(cfg.targetTableName)))
-
.orElseGet(dataAndCheckpoint.getSchemaProvider()::getTargetHoodieSchema);
+ HoodieSchema incomingSchema = transformed.map(df -> {
+ StructType structType = UtilHelpers.extractSchemaFromDataset(df,
props);
Review Comment:
Addressed. Using a single `ConfigProperty` in `HoodieStreamerConfig` with
`hoodie.deltastreamer.transformed.row.nullable` as an alternative key for
backwards compatibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
