nsivabalan commented on code in PR #11660:
URL: https://github.com/apache/hudi/pull/11660#discussion_r1973933358
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/SchemaRegistryProvider.java:
##########
@@ -82,34 +93,53 @@ public static class Config {
public static final String SSL_KEY_PASSWORD_PROP =
"schema.registry.ssl.key.password";
}
- protected Schema cachedSourceSchema;
- protected Schema cachedTargetSchema;
+ private final Option<SchemaConverter> schemaConverter;
+ private final SerializableFunctionUnchecked<String, RestService>
restServiceProvider;
+ private final SerializableFunctionUnchecked<RestService,
SchemaRegistryClient> registryClientProvider;
- private final String srcSchemaRegistryUrl;
- private final String targetSchemaRegistryUrl;
+ public SchemaRegistryProvider(TypedProperties props, JavaSparkContext jssc) {
+ super(props, jssc);
+ checkRequiredConfigProperties(props,
Collections.singletonList(HoodieSchemaProviderConfig.SRC_SCHEMA_REGISTRY_URL));
+ if (config.containsKey(Config.SSL_KEYSTORE_LOCATION_PROP)
+ || config.containsKey(Config.SSL_TRUSTSTORE_LOCATION_PROP)) {
+ setUpSSLStores();
+ }
+ String schemaConverter = getStringWithAltKeys(config,
HoodieSchemaProviderConfig.SCHEMA_CONVERTER, true);
+ this.schemaConverter = !StringUtils.isNullOrEmpty(schemaConverter)
+ ? Option.of((SchemaConverter) ReflectionUtils.loadClass(
+ schemaConverter, new Class<?>[] {TypedProperties.class}, config))
+ : Option.empty();
+ this.restServiceProvider = RestService::new;
+ this.registryClientProvider = restService -> new
CachedSchemaRegistryClient(restService, 100,
+ Arrays.asList(new ProtobufSchemaProvider(), new JsonSchemaProvider(),
new AvroSchemaProvider()), null, null);
Review Comment:
can you help throw some light @the-other-tim-brown
why are we instantiating all 3 providers? In reality only one will be
applicable per stream right?
And based on schema converter object at L 112, we should be able to gauge
which provider we wanted to go w/ .
Can we avoid instantiating all 3, but just 1.
this could be hard blocker for simple OOB user as reported in
https://issues.apache.org/jira/browse/HUDI-9057
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
