the-other-tim-brown commented on code in PR #8574:
URL: https://github.com/apache/hudi/pull/8574#discussion_r1191525650
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/transform/Transformer.java:
##########
@@ -45,4 +47,9 @@ public interface Transformer {
*/
@PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
Dataset<Row> apply(JavaSparkContext jsc, SparkSession sparkSession,
Dataset<Row> rowDataset, TypedProperties properties);
+
+ @PublicAPIMethod(maturity = ApiMaturityLevel.EVOLVING)
+ default Option<Schema> transformedSchema(JavaSparkContext jsc, SparkSession
sparkSession, Schema incomingSchema, TypedProperties properties) {
+ return Option.empty();
Review Comment:
Another note on Avro vs StructType. There are subtle differences between the
two and possible struct types that cannot be covered by Avro schemas like
non-string map keys. Since this is some intermediate state, I think we should
be using the schema format that represents this state so we don't getting into
any edge cases where the user cannot define the proper schema of their row
before and after the transform
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
