skoppu22 commented on code in PR #169:
URL:
https://github.com/apache/cassandra-analytics/pull/169#discussion_r3475667308
##########
cassandra-analytics-core/src/main/java/org/apache/cassandra/spark/bulkwriter/AbstractBulkWriterContext.java:
##########
@@ -98,10 +101,32 @@ protected AbstractBulkWriterContext(@NotNull BulkSparkConf
conf,
this.conf = conf;
this.sparkDefaultParallelism = sparkDefaultParallelism;
- // Build everything fresh on driver
- this.clusterInfo = buildClusterInfo();
+ // Retrieve lowest Cassandra version without building full ClusterInfo
+ String lowestCassandraVersion = getLowestCassandraVersion(conf);
+ Set<String> sstableVersionsOnCluster = null;
+
+ // Get SSTable versions from cluster only if SSTable version-based
selection is enabled
+ // If disabled, skip retrieval to allow job to proceed even when
SSTable version detection fails
+ if (!conf.isSSTableVersionBasedBridgeDisabled())
+ {
+ sstableVersionsOnCluster = getSSTableVersionsOnCluster(conf);
+ }
+
+ // Determine bridge version
+ this.bridgeVersion =
SSTableVersionAnalyzer.determineBridgeVersionForWrite(
+ sstableVersionsOnCluster,
+ CassandraVersion.configuredSSTableFormat(),
+ lowestCassandraVersion,
+ conf.isSSTableVersionBasedBridgeDisabled()
+ );
+
+ // Validate that Kryo registrator exists for this bridge version
+ KryoRegister.validateKryoRegistratorExists(this.bridgeVersion,
lowestCassandraVersion);
Review Comment:
We added this call KryoRegister.validateKryoRegistratorExists when we were
adding only one cassandra.version registrator class to KRYO_SERIALIZERS, which
may not be the same as determined bridgerversion. Now I have modified
KryoRegister to add all implemented registrator classes to KRYO_SERIALIZERS, so
this check is not needed. We can leave it as it is just to verify upfront or we
can remove it.
SSTableVersionAnalyzer already ensures only recognized bridgeversion is
returned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
