pratyakshsharma commented on a change in pull request #1566:
URL: https://github.com/apache/incubator-hudi/pull/1566#discussion_r416694602
##########
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##########
@@ -162,18 +162,23 @@ public DeltaSync(HoodieDeltaStreamer.Config cfg,
SparkSession sparkSession, Sche
this.fs = fs;
this.onInitializingHoodieWriteClient = onInitializingHoodieWriteClient;
this.props = props;
- this.schemaProvider = schemaProvider;
refreshTimeline();
-
this.transformer =
UtilHelpers.createTransformer(cfg.transformerClassNames);
this.keyGenerator = DataSourceUtils.createKeyGenerator(props);
-
- this.formatAdapter = new SourceFormatAdapter(
- UtilHelpers.createSource(cfg.sourceClassName, props, jssc,
sparkSession, schemaProvider));
-
this.conf = conf;
+ refreshSchemaProvider(schemaProvider);
+ }
+ /**
+ * Very useful when DeltaStreamer is running in continuous mode.
+ * @param schemaProvider
+ * @throws IOException
+ */
+ public void refreshSchemaProvider(SchemaProvider schemaProvider) throws
IOException {
Review comment:
> It looks like refreshSchemaProvider not only refreshes schema-provider
but also recreates Source and setup WriteClient
Do you see any side effects of doing this? @bvaradar
> have delta-streamer call this ?
This call will happen exactly at the same point where I am calling
refreshSchemaProvider in delta-streamer, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
