lokeshj1703 commented on code in PR #8574:
URL: https://github.com/apache/hudi/pull/8574#discussion_r1177815595
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java:
##########
@@ -276,9 +276,19 @@ public static class Config implements Serializable {
+ ". Allows transforming raw source Dataset to a target Dataset
(conforming to target schema) before "
+ "writing. Default : Not set. E:g -
org.apache.hudi.utilities.transform.SqlQueryBasedTransformer (which "
+ "allows a SQL query templated to be passed as a transformation
function). "
- + "Pass a comma-separated list of subclass names to chain the
transformations.")
+ + "Pass a comma-separated list of subclass names to chain the
transformations. Transformer can also include "
+ + "a suffix. This suffix can be appended with the property keys to
identify properties related to the transformer. "
+ + "E:g -
tr1:org.apache.hudi.utilities.transform.SqlQueryBasedTransformer can be used
along with property key "
+ + "hoodie.deltastreamer.transformer.sql.tr1. Here tr1 is a suffix
used to identify the keys specific to this transformer. "
Review Comment:
This scenario is already allowed currently. The behaviour is `configA.idt1`
would override any config `configA` defined in properties. If no key
corresponding to `configA.idt1` is defined then default is used.
Also currently we do not allow configurations where id is passed for some
transformers and not for others:
```
tr1:io.bytearray.TransformerA
io.bytearray.TransformerB
tr3:io.bytearray.TransformerA
```
Please let me know if this looks good.
cc @rmahindra123 @vinothchandar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
