nsivabalan commented on a change in pull request #2449:
URL: https://github.com/apache/hudi/pull/2449#discussion_r567256239
##########
File path:
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java
##########
@@ -63,6 +63,10 @@
public HiveSyncTool(HiveSyncConfig cfg, HiveConf configuration, FileSystem
fs) {
super(configuration.getAllProperties(), fs);
+ //Set Hive meta store URI which enables synchronize to remote hive
+ if (cfg.hiveMetaStoreUri != null) {
Review comment:
I see that we set a default value for this new config in
DataSourceUtils( thrift://localhost:9083). So, for someone who does not set
this, I assume it should be no-op even though we set to some default value. I
don't have much knowledge on hive metastore and sync. So, might be a naive
question.
##########
File path:
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java
##########
@@ -63,6 +63,10 @@
public HiveSyncTool(HiveSyncConfig cfg, HiveConf configuration, FileSystem
fs) {
super(configuration.getAllProperties(), fs);
+ //Set Hive meta store URI which enables synchronize to remote hive
+ if (cfg.hiveMetaStoreUri != null) {
+ configuration.set("hive.metastore.uris",cfg.hiveMetaStoreUri);
Review comment:
might be a naive question. Can you help me understand where else can I
find the configs where we translate hoodie params to hive params. I tried to
search in our repo, but couldn't find any. or is this the first one we do like
this? for eg, wrt JDBC url, hoodie param is named as
"hoodie.datasource.hive_sync.jdbcurl", but we directly pass this value while
creating a connection to hive. But are there other configs which gets converted
from hoodie param to hive param somewhere? Basically trying to understand why
this new config is handled specially here and not along w/ any other such
translations.
##########
File path:
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/DataSourceUtils.java
##########
@@ -284,6 +284,8 @@ public static HiveSyncConfig
buildHiveSyncConfig(TypedProperties props, String b
props.getString(DataSourceWriteOptions.HIVE_PASS_OPT_KEY(),
DataSourceWriteOptions.DEFAULT_HIVE_PASS_OPT_VAL());
hiveSyncConfig.jdbcUrl =
props.getString(DataSourceWriteOptions.HIVE_URL_OPT_KEY(),
DataSourceWriteOptions.DEFAULT_HIVE_URL_OPT_VAL());
+ hiveSyncConfig.hiveMetaStoreUri =
Review comment:
is this not "HIVE_METASTORE_URI_OPT_VAL()" ?
##########
File path:
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/DataSourceUtils.java
##########
@@ -284,6 +284,8 @@ public static HiveSyncConfig
buildHiveSyncConfig(TypedProperties props, String b
props.getString(DataSourceWriteOptions.HIVE_PASS_OPT_KEY(),
DataSourceWriteOptions.DEFAULT_HIVE_PASS_OPT_VAL());
hiveSyncConfig.jdbcUrl =
props.getString(DataSourceWriteOptions.HIVE_URL_OPT_KEY(),
DataSourceWriteOptions.DEFAULT_HIVE_URL_OPT_VAL());
+ hiveSyncConfig.hiveMetaStoreUri =
Review comment:
wondering if we have any tests around this. if not, we would have caught
this. would be nice if we can have one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
