xushiyan commented on code in PR #7092:
URL: https://github.com/apache/hudi/pull/7092#discussion_r1014056424
##########
hudi-integ-test/src/main/scala/org/apache/hudi/integ/testsuite/dag/nodes/SparkInsertNode.scala:
##########
@@ -71,6 +71,7 @@ class SparkInsertNode(dagNodeConfig: Config) extends
DagNode[RDD[WriteStatus]] {
.option(DataSourceWriteOptions.TABLE_NAME.key,
context.getHoodieTestSuiteWriter.getCfg.targetTableName)
.option(DataSourceWriteOptions.TABLE_TYPE.key,
context.getHoodieTestSuiteWriter.getCfg.tableType)
.option(DataSourceWriteOptions.OPERATION.key, getOperation())
+ .option("hoodie.index.type",
context.getHoodieTestSuiteWriter.getCfg.indexType)
Review Comment:
use constant for the key
##########
hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/dag/nodes/ValidateDatasetNode.java:
##########
@@ -51,7 +51,7 @@ public Dataset<Row> getDatasetToValidate(SparkSession
session, ExecutionContext
StructType inputSchema) {
String partitionPathField =
context.getWriterContext().getProps().getString(DataSourceWriteOptions.PARTITIONPATH_FIELD().key());
String hudiPath =
context.getHoodieTestSuiteWriter().getCfg().targetBasePath +
(partitionPathField.isEmpty() ? "/" : "/*/*/*");
- Dataset<Row> hudiDf =
session.read().option(HoodieMetadataConfig.ENABLE.key(),
String.valueOf(config.isEnableMetadataValidate()))
+ Dataset<Row> hudiDf =
session.read().option(HoodieMetadataConfig.ENABLE.key(),
String.valueOf(context.getHoodieTestSuiteWriter().getCfg().enableMetadataOnRead))
Review Comment:
so we prefer `context.getHoodieTestSuiteWriter().getCfg()` over
`this.config` ? how would people know this is the preferred way: we either make
`this.config` usable or remove it so people only use context to retrieve config?
##########
hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/HoodieTestSuiteJob.java:
##########
@@ -340,5 +343,11 @@ public static class HoodieTestSuiteConfig extends
HoodieDeltaStreamer.Config {
@Parameter(names = {"--trino-jdbc-password"}, description = "Password
corresponding to the username to use for authentication")
public String trinoPassword;
+
+ @Parameter(names = {"--index-type"}, description = "Index type to use for
writes")
+ public String indexType = "SIMPLE";
+
+ @Parameter(names = {"--enable-metadata-on-read"}, description = "Enable's
metadata for queries")
+ public Boolean enableMetadataOnRead = false;
Review Comment:
ditto; and applies to other manually assigned default values
##########
hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/dag/nodes/PrestoQueryNode.java:
##########
@@ -35,25 +35,32 @@ public PrestoQueryNode(DeltaConfig.Config config) {
@Override
public void execute(ExecutionContext context, int curItrCount) throws
Exception {
- log.info("Executing presto query node {}", this.getName());
- String url = context.getHoodieTestSuiteWriter().getCfg().prestoJdbcUrl;
- if (StringUtils.isNullOrEmpty(url)) {
- throw new IllegalArgumentException("Presto JDBC connection url not
provided. Please set --presto-jdbc-url.");
- }
- String user = context.getHoodieTestSuiteWriter().getCfg().prestoUsername;
- String pass = context.getHoodieTestSuiteWriter().getCfg().prestoPassword;
- try {
- Class.forName("com.facebook.presto.jdbc.PrestoDriver");
- } catch (ClassNotFoundException e) {
- throw new HoodieValidationException("Presto query validation failed due
to " + e.getMessage(), e);
- }
- try (Connection connection = DriverManager.getConnection(url, user, pass))
{
- Statement stmt = connection.createStatement();
- setSessionProperties(this.config.getPrestoProperties(), stmt);
- executeAndValidateQueries(this.config.getPrestoQueries(), stmt);
- stmt.close();
- } catch (Exception e) {
- throw new HoodieValidationException("Presto query validation failed due
to " + e.getMessage(), e);
+ if (context.getHoodieTestSuiteWriter().getCfg().enablePrestoValidation) {
Review Comment:
pls follow the early-return style
##########
hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/HoodieTestSuiteJob.java:
##########
@@ -340,5 +343,11 @@ public static class HoodieTestSuiteConfig extends
HoodieDeltaStreamer.Config {
@Parameter(names = {"--trino-jdbc-password"}, description = "Password
corresponding to the username to use for authentication")
public String trinoPassword;
+
+ @Parameter(names = {"--index-type"}, description = "Index type to use for
writes")
+ public String indexType = "SIMPLE";
Review Comment:
use the default value from ConfigProperty
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
