nsivabalan commented on code in PR #17946:
URL: https://github.com/apache/hudi/pull/17946#discussion_r2720319106
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/SourceFormatAdapter.java:
##########
@@ -206,9 +206,9 @@ public InputBatch<JavaRDD<GenericRecord>>
fetchNewDataInAvroFormat(Option<Checkp
// pass in the schema for the Row-to-Avro conversion
// to avoid nullability mismatch between Avro schema and
Row schema
? HoodieSparkUtils.createRdd(rdd,
HOODIE_RECORD_STRUCT_NAME, HOODIE_RECORD_NAMESPACE, true,
-
Option.ofNullable(r.getSchemaProvider().getSourceHoodieSchema())
- ).toJavaRDD() : HoodieSparkUtils.createRdd(rdd,
- HOODIE_RECORD_STRUCT_NAME, HOODIE_RECORD_NAMESPACE, false,
Option.empty()).toJavaRDD();
+
Option.ofNullable(r.getSchemaProvider().getSourceHoodieSchema())).toJavaRDD()
+ : HoodieSparkUtils.createRdd(rdd,
HOODIE_RECORD_STRUCT_NAME, HOODIE_RECORD_NAMESPACE, false,
+
Option.ofNullable(r.getSchemaProvider().getTargetHoodieSchema())).toJavaRDD();
Review Comment:
Yes, adding a test would help for sure.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/SourceFormatAdapter.java:
##########
@@ -206,9 +206,9 @@ public InputBatch<JavaRDD<GenericRecord>>
fetchNewDataInAvroFormat(Option<Checkp
// pass in the schema for the Row-to-Avro conversion
// to avoid nullability mismatch between Avro schema and
Row schema
? HoodieSparkUtils.createRdd(rdd,
HOODIE_RECORD_STRUCT_NAME, HOODIE_RECORD_NAMESPACE, true,
-
Option.ofNullable(r.getSchemaProvider().getSourceHoodieSchema())
- ).toJavaRDD() : HoodieSparkUtils.createRdd(rdd,
- HOODIE_RECORD_STRUCT_NAME, HOODIE_RECORD_NAMESPACE, false,
Option.empty()).toJavaRDD();
+
Option.ofNullable(r.getSchemaProvider().getSourceHoodieSchema())).toJavaRDD()
+ : HoodieSparkUtils.createRdd(rdd,
HOODIE_RECORD_STRUCT_NAME, HOODIE_RECORD_NAMESPACE, false,
+
Option.ofNullable(r.getSchemaProvider().getTargetHoodieSchema())).toJavaRDD();
Review Comment:
lets add java docs to call out the need for target schema in this code
block.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
