aasha commented on a change in pull request #1883:
URL: https://github.com/apache/hive/pull/1883#discussion_r564994663
##########
File path:
ql/src/test/org/apache/hadoop/hive/ql/exec/repl/TestAtlasDumpTask.java
##########
@@ -242,6 +251,25 @@ public void testAtlasClientTimeouts() throws Exception {
AtlasRestClientBuilder.ATLAS_PROPERTY_READ_TIMEOUT_IN_MS));
}
+ @Test
+ public void testCreateExportRequest() throws Exception {
Review comment:
what happens if policy is modified to remove the table expression or
modify the table expression? Does it fallback to db level replication?
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/AtlasLoadTask.java
##########
@@ -121,7 +121,7 @@ AtlasReplInfo createAtlasReplInfo() throws
SemanticException, MalformedURLExcept
String srcCluster =
ReplUtils.getNonEmpty(HiveConf.ConfVars.REPL_SOURCE_CLUSTER_NAME.varname, conf,
errorFormat);
String tgtCluster =
ReplUtils.getNonEmpty(HiveConf.ConfVars.REPL_TARGET_CLUSTER_NAME.varname, conf,
errorFormat);
AtlasReplInfo atlasReplInfo = new AtlasReplInfo(endpoint, work.getSrcDB(),
work.getTgtDB(),
- srcCluster, tgtCluster, work.getStagingDir(), conf);
+ srcCluster, tgtCluster, work.getStagingDir(), null, conf);
Review comment:
Can have 2 constructors, with and without table list
##########
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/atlas/AtlasRequestBuilder.java
##########
@@ -105,6 +126,50 @@ private String getQualifiedName(String clusterName, String
srcDb) {
return qualifiedName;
}
+ private String getQualifiedName(String clusterName, String srcDB, String
tableName) {
+ String qualifiedTableName =
String.format(QUALIFIED_NAME_HIVE_TABLE_FORMAT, srcDB, tableName);
+ return getQualifiedName(clusterName, qualifiedTableName);
+ }
+
+ private List<String> getQualifiedNames(String clusterName, String srcDb,
Path listOfTablesFile, HiveConf conf)
+ throws SemanticException {
+ List<String> qualifiedNames = new ArrayList<>();
+ List<String> tableNames = getFileAsList(listOfTablesFile, conf);
+ if (CollectionUtils.isEmpty(tableNames)) {
+ LOG.info("Empty file encountered: {}", listOfTablesFile);
+ return qualifiedNames;
+ }
+ for (String tableName : tableNames) {
+ qualifiedNames.add(getQualifiedName(clusterName, srcDb, tableName));
+ }
+ return qualifiedNames;
+ }
+
+ private static List<String> getFileAsList(Path listOfTablesFile, HiveConf
conf) throws SemanticException {
+ List<String> list = new ArrayList<>();
+ InputStream is = null;
+ try {
+ FileSystem fs = FileSystem.get(listOfTablesFile.toUri(), conf);
Review comment:
Anything needed based on staging is on source /target or HA cases?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
