yihua commented on code in PR #8993:
URL: https://github.com/apache/hudi/pull/8993#discussion_r1233384462
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/validator/SqlQueryEqualityPreCommitValidator.java:
##########
@@ -54,13 +54,11 @@ protected String getQueryConfigName() {
@Override
protected void validateUsingQuery(String query, String prevTableSnapshot,
String newTableSnapshot, SQLContext sqlContext) {
- String queryWithPrevSnapshot =
query.replaceAll(HoodiePreCommitValidatorConfig.VALIDATOR_TABLE_VARIABLE,
prevTableSnapshot);
- String queryWithNewSnapshot =
query.replaceAll(HoodiePreCommitValidatorConfig.VALIDATOR_TABLE_VARIABLE,
newTableSnapshot);
- LOG.info("Running query on previous state: " + queryWithPrevSnapshot);
- Dataset<Row> prevRows = sqlContext.sql(queryWithPrevSnapshot).cache();
+ LOG.info("Running query on previous state: " + prevTableSnapshot);
+ Dataset<Row> prevRows = executeSqlQuery(sqlContext, query,
prevTableSnapshot).cache();
LOG.info("Total rows in prevRows " + prevRows.count());
- LOG.info("Running query on new state: " + queryWithNewSnapshot);
- Dataset<Row> newRows = sqlContext.sql(queryWithNewSnapshot).cache();
+ LOG.info("Running query on new state: " + newTableSnapshot);
Review Comment:
Sorry, the actual fix is in `SparkValidatorUtils` which was somehow revert
in my force push. Fixed now.
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/validator/SqlQueryEqualityPreCommitValidator.java:
##########
@@ -54,13 +54,11 @@ protected String getQueryConfigName() {
@Override
protected void validateUsingQuery(String query, String prevTableSnapshot,
String newTableSnapshot, SQLContext sqlContext) {
- String queryWithPrevSnapshot =
query.replaceAll(HoodiePreCommitValidatorConfig.VALIDATOR_TABLE_VARIABLE,
prevTableSnapshot);
- String queryWithNewSnapshot =
query.replaceAll(HoodiePreCommitValidatorConfig.VALIDATOR_TABLE_VARIABLE,
newTableSnapshot);
- LOG.info("Running query on previous state: " + queryWithPrevSnapshot);
- Dataset<Row> prevRows = sqlContext.sql(queryWithPrevSnapshot).cache();
+ LOG.info("Running query on previous state: " + prevTableSnapshot);
+ Dataset<Row> prevRows = executeSqlQuery(sqlContext, query,
prevTableSnapshot).cache();
Review Comment:
Sorry, the actual fix is in `SparkValidatorUtils` which was somehow revert
in my force push. Fixed now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
