danny0405 commented on a change in pull request #4699:
URL: https://github.com/apache/hudi/pull/4699#discussion_r803550671
##########
File path: hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java
##########
@@ -101,7 +102,7 @@ public static TypedProperties getProps(FlinkStreamerConfig
cfg) {
return new TypedProperties();
}
return readConfig(
- getHadoopConf(),
+ getHadoopConf(cfg),
new Path(cfg.propsFilePath), cfg.configs).getProps();
Review comment:
Fix the indentation.
##########
File path: hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java
##########
@@ -140,9 +141,21 @@ public static DFSPropertiesConfiguration
readConfig(org.apache.hadoop.conf.Confi
return conf;
}
- // Keep the redundant to avoid too many modifications.
public static org.apache.hadoop.conf.Configuration getHadoopConf() {
- return FlinkClientUtil.getHadoopConf();
+ return getHadoopConf(null);
+ }
+
+ // Keep the redundant to avoid too many modifications.
+ public static org.apache.hadoop.conf.Configuration
getHadoopConf(Configuration configuration) {
+ if (configuration == null) {
+ return FlinkClientUtil.getHadoopConf();
Review comment:
`configuration` => `conf`,
Can we make the param `configuration` never null ?
##########
File path:
hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java
##########
@@ -660,25 +661,18 @@ private FlinkOptions() {
// Prefix for Hoodie specific properties.
private static final String PROPERTIES_PREFIX = "properties.";
- /**
- * Collects the config options that start with 'properties.' into a
'key'='value' list.
- */
- public static Map<String, String> getHoodieProperties(Map<String, String>
options) {
- return getHoodiePropertiesWithPrefix(options, PROPERTIES_PREFIX);
- }
-
/**
* Collects the config options that start with specified prefix {@code
prefix} into a 'key'='value' list.
*/
- public static Map<String, String> getHoodiePropertiesWithPrefix(Map<String,
String> options, String prefix) {
+ public static Map<String, String> getPropertiesWithPrefix(Map<String,
String> options, String subprefix) {
final Map<String, String> hoodieProperties = new HashMap<>();
Review comment:
Can we use the `prefix` directly ? There is no need to prefix the option
with `properties.`, prefix the option with `hadoop.` directly is okey.
And can we also add a too method named `getHadoopOptions(Configuration
conf)` here ?
##########
File path:
hudi-flink/src/test/java/org/apache/hudi/utils/TestViewStorageProperties.java
##########
@@ -45,11 +45,11 @@ void testReadWriteProperties() throws IOException {
.withStorageType(FileSystemViewStorageType.SPILLABLE_DISK)
.withRemoteServerHost("host1")
.withRemoteServerPort(1234).build();
- ViewStorageProperties.createProperties(basePath, config);
- ViewStorageProperties.createProperties(basePath, config);
- ViewStorageProperties.createProperties(basePath, config);
+ ViewStorageProperties.createProperties(basePath, config, null);
+ ViewStorageProperties.createProperties(basePath, config, null);
+ ViewStorageProperties.createProperties(basePath, config, null);
- FileSystemViewStorageConfig readConfig =
ViewStorageProperties.loadFromProperties(basePath);
+ FileSystemViewStorageConfig readConfig =
ViewStorageProperties.loadFromProperties(basePath, null);
Review comment:
ditto.
##########
File path: hudi-flink/src/test/java/org/apache/hudi/utils/TestStreamerUtil.java
##########
@@ -106,7 +106,7 @@ void testInstantTimeDiff() {
void testDumpRemoteViewStorageConfig() throws IOException {
Configuration conf =
TestConfigurations.getDefaultConf(tempFile.getAbsolutePath());
StreamerUtil.createWriteClient(conf);
- FileSystemViewStorageConfig storageConfig =
ViewStorageProperties.loadFromProperties(conf.getString(FlinkOptions.PATH));
+ FileSystemViewStorageConfig storageConfig =
ViewStorageProperties.loadFromProperties(conf.getString(FlinkOptions.PATH),
null);
assertThat(storageConfig.getStorageType(),
is(FileSystemViewStorageType.REMOTE_FIRST));
Review comment:
Use `new Configuration` to avoid null.
##########
File path:
hudi-flink/src/main/java/org/apache/hudi/util/ViewStorageProperties.java
##########
@@ -61,10 +63,10 @@ public static void createProperties(
/**
* Read the {@link FileSystemViewStorageConfig} with given table base path.
*/
- public static FileSystemViewStorageConfig loadFromProperties(String
basePath) {
+ public static FileSystemViewStorageConfig loadFromProperties(String
basePath, Configuration configuration) {
Path propertyPath = getPropertiesFilePath(basePath);
LOG.info("Loading filesystem view storage properties from " +
propertyPath);
Review comment:
`configuration` => `conf`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
