yihua commented on code in PR #11792:
URL: https://github.com/apache/hudi/pull/11792#discussion_r1722602002
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/testutils/HoodieMergeOnReadTestUtils.java:
##########
@@ -57,48 +55,9 @@ public class HoodieMergeOnReadTestUtils {
private static final Logger LOG =
LoggerFactory.getLogger(HoodieMergeOnReadTestUtils.class);
- public static List<RecordReader>
getRecordReadersUsingInputFormat(Configuration conf, List<String> inputPaths,
Review Comment:
Is this not used any more?
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/storage/hadoop/HadoopStorageConfiguration.java:
##########
@@ -38,6 +38,10 @@ public class HadoopStorageConfiguration extends
StorageConfiguration<Configurati
private transient Configuration configuration;
+ public HadoopStorageConfiguration() {
+ this(true);
Review Comment:
Using `loadDefaults=true` adds non-trivial overhead of initializing the
configuration. If not necessary (only used in tests), let's remove this
constructor and use existing constructor explicitly.
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/testutils/GenericRecordValidationTestUtils.java:
##########
@@ -93,29 +92,31 @@ public static void assertGenericRecords(GenericRecord
record1, GenericRecord rec
}
public static void assertDataInMORTable(HoodieWriteConfig config, String
instant1, String instant2,
- StorageConfiguration<Configuration>
storageConf, List<String> partitionPaths) {
+ StorageConfiguration<?> storageConf,
List<String> partitionPaths) {
List<String> excludeFields =
CollectionUtils.createImmutableList(COMMIT_TIME_METADATA_FIELD,
COMMIT_SEQNO_METADATA_FIELD,
FILENAME_METADATA_FIELD, OPERATION_METADATA_FIELD);
assertDataInMORTable(config, instant1, instant2, storageConf,
partitionPaths, excludeFields);
}
public static void assertDataInMORTable(HoodieWriteConfig config, String
instant1, String instant2,
- StorageConfiguration<Configuration>
storageConf, List<String> partitionPaths, List<String> excludeFields) {
- JobConf jobConf = new JobConf(storageConf.unwrap());
+ StorageConfiguration<?> storageConf,
List<String> partitionPaths, List<String> excludeFields) {
+ StorageConfiguration<?> storageConfCopy = storageConf.newInstance();
Review Comment:
We might still keep the same logic as before as
`HadoopFSUtils.convertToJobConf` is used to convert the config back to
`JobConf`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
