codope commented on code in PR #8993:
URL: https://github.com/apache/hudi/pull/8993#discussion_r1233171801
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/validator/SqlQueryEqualityPreCommitValidator.java:
##########
@@ -54,13 +54,11 @@ protected String getQueryConfigName() {
@Override
protected void validateUsingQuery(String query, String prevTableSnapshot,
String newTableSnapshot, SQLContext sqlContext) {
- String queryWithPrevSnapshot =
query.replaceAll(HoodiePreCommitValidatorConfig.VALIDATOR_TABLE_VARIABLE,
prevTableSnapshot);
- String queryWithNewSnapshot =
query.replaceAll(HoodiePreCommitValidatorConfig.VALIDATOR_TABLE_VARIABLE,
newTableSnapshot);
- LOG.info("Running query on previous state: " + queryWithPrevSnapshot);
- Dataset<Row> prevRows = sqlContext.sql(queryWithPrevSnapshot).cache();
+ LOG.info("Running query on previous state: " + prevTableSnapshot);
+ Dataset<Row> prevRows = executeSqlQuery(sqlContext, query,
prevTableSnapshot).cache();
Review Comment:
It looks like a simple refactoring. Trying to understand the root cause. Was
it because of successive `replaceAll` calls before executing prev snapshot and
new snapshot queries?
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/validator/SparkPreCommitValidator.java:
##########
@@ -118,4 +120,11 @@ public HoodieEngineContext getEngineContext() {
public HoodieWriteConfig getWriteConfig() {
return this.writeConfig;
}
+
+ protected Dataset<Row> executeSqlQuery(SQLContext sqlContext, String
sqlQuery, String tableName) {
+ String queryWithTempTableName = sqlQuery.replaceAll(
+ HoodiePreCommitValidatorConfig.VALIDATOR_TABLE_VARIABLE, tableName);
+ LOG.debug("Running query: " + queryWithTempTableName);
+ return sqlContext.sql(queryWithTempTableName);
Review Comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
