[jira] [Work logged] (HIVE-23611) Mandate fully qualified absolute path for external table base dir during REPL operation

2020-09-08 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-23611?focusedWorklogId=480534&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-480534
 ]

ASF GitHub Bot logged work on HIVE-23611:
-

Author: ASF GitHub Bot
Created on: 09/Sep/20 00:47
Start Date: 09/Sep/20 00:47
Worklog Time Spent: 10m 
  Work Description: github-actions[bot] closed pull request #1120:
URL: https://github.com/apache/hive/pull/1120


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 480534)
Time Spent: 1h 10m  (was: 1h)

> Mandate fully qualified absolute path for external table base dir during REPL 
> operation
> ---
>
> Key: HIVE-23611
> URL: https://issues.apache.org/jira/browse/HIVE-23611
> Project: Hive
>  Issue Type: Improvement
>Reporter: Pravin Sinha
>Assignee: Pravin Sinha
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-23611.01.patch, HIVE-23611.02.patch, 
> HIVE-23611.03.patch
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (HIVE-23611) Mandate fully qualified absolute path for external table base dir during REPL operation

2020-08-31 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-23611?focusedWorklogId=476938&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-476938
 ]

ASF GitHub Bot logged work on HIVE-23611:
-

Author: ASF GitHub Bot
Created on: 01/Sep/20 00:47
Start Date: 01/Sep/20 00:47
Worklog Time Spent: 10m 
  Work Description: github-actions[bot] commented on pull request #1120:
URL: https://github.com/apache/hive/pull/1120#issuecomment-684124169


   This pull request has been automatically marked as stale because it has not 
had recent activity. It will be closed if no further activity occurs.
   Feel free to reach out on the d...@hive.apache.org list if the patch is in 
need of reviews.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 476938)
Time Spent: 1h  (was: 50m)

> Mandate fully qualified absolute path for external table base dir during REPL 
> operation
> ---
>
> Key: HIVE-23611
> URL: https://issues.apache.org/jira/browse/HIVE-23611
> Project: Hive
>  Issue Type: Improvement
>Reporter: Pravin Sinha
>Assignee: Pravin Sinha
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-23611.01.patch, HIVE-23611.02.patch, 
> HIVE-23611.03.patch
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (HIVE-23611) Mandate fully qualified absolute path for external table base dir during REPL operation

2020-06-29 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-23611?focusedWorklogId=452575&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-452575
 ]

ASF GitHub Bot logged work on HIVE-23611:
-

Author: ASF GitHub Bot
Created on: 29/Jun/20 18:45
Start Date: 29/Jun/20 18:45
Worklog Time Spent: 10m 
  Work Description: pkumarsinha commented on a change in pull request #1120:
URL: https://github.com/apache/hive/pull/1120#discussion_r447178750



##
File path: 
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosExternalTables.java
##
@@ -517,7 +508,7 @@ public void externalTableIncrementalReplication() throws 
Throwable {
 + "'")
 .run("alter table t1 add partition(country='india')")
 .run("alter table t1 add partition(country='us')")
-.dump(primaryDbName, withClause);

Review comment:
   Not needed anymore





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 452575)
Time Spent: 50m  (was: 40m)

> Mandate fully qualified absolute path for external table base dir during REPL 
> operation
> ---
>
> Key: HIVE-23611
> URL: https://issues.apache.org/jira/browse/HIVE-23611
> Project: Hive
>  Issue Type: Improvement
>Reporter: Pravin Sinha
>Assignee: Pravin Sinha
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-23611.01.patch, HIVE-23611.02.patch
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (HIVE-23611) Mandate fully qualified absolute path for external table base dir during REPL operation

2020-06-29 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-23611?focusedWorklogId=452574&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-452574
 ]

ASF GitHub Bot logged work on HIVE-23611:
-

Author: ASF GitHub Bot
Created on: 29/Jun/20 18:45
Start Date: 29/Jun/20 18:45
Worklog Time Spent: 10m 
  Work Description: pkumarsinha commented on a change in pull request #1120:
URL: https://github.com/apache/hive/pull/1120#discussion_r447178408



##
File path: 
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosExternalTables.java
##
@@ -159,14 +154,14 @@ public void externalTableReplicationWithDefaultPaths() 
throws Throwable {
 .run("insert into table t2 partition(country='india') values 
('bangalore')")
 .run("insert into table t2 partition(country='us') values ('austin')")
 .run("insert into table t2 partition(country='france') values 
('paris')")
-.dump(primaryDbName, withClauseOptions);
+.dump(primaryDbName);

Review comment:
   Not needed anymore

##
File path: 
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosExternalTables.java
##
@@ -503,8 +495,7 @@ public void externalTableIncrementalCheckpointing() throws 
Throwable {
 
   @Test
   public void externalTableIncrementalReplication() throws Throwable {
-List withClause = externalTableBasePathWithClause();
-WarehouseInstance.Tuple tuple = primary.dump(primaryDbName, withClause);
+WarehouseInstance.Tuple tuple = primary.dump(primaryDbName);

Review comment:
   Not needed anymore

##
File path: 
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosExternalTables.java
##
@@ -623,7 +612,7 @@ public void bootstrapExternalTablesDuringIncrementalPhase() 
throws Throwable {
 assertFalse(primary.miniDFSCluster.getFileSystem()
 .exists(new Path(metadataPath + relativeExtInfoPath(null;
 
-replica.load(replicatedDbName, primaryDbName, loadWithClause)
+replica.load(replicatedDbName, primaryDbName)

Review comment:
   Not needed anymore





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 452574)
Time Spent: 40m  (was: 0.5h)

> Mandate fully qualified absolute path for external table base dir during REPL 
> operation
> ---
>
> Key: HIVE-23611
> URL: https://issues.apache.org/jira/browse/HIVE-23611
> Project: Hive
>  Issue Type: Improvement
>Reporter: Pravin Sinha
>Assignee: Pravin Sinha
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-23611.01.patch, HIVE-23611.02.patch
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (HIVE-23611) Mandate fully qualified absolute path for external table base dir during REPL operation

2020-06-29 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-23611?focusedWorklogId=452569&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-452569
 ]

ASF GitHub Bot logged work on HIVE-23611:
-

Author: ASF GitHub Bot
Created on: 29/Jun/20 18:38
Start Date: 29/Jun/20 18:38
Worklog Time Spent: 10m 
  Work Description: pkumarsinha commented on a change in pull request #1120:
URL: https://github.com/apache/hive/pull/1120#discussion_r447174957



##
File path: 
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/BaseReplicationAcrossInstances.java
##
@@ -103,6 +114,12 @@ public static void classLevelTearDown() throws IOException 
{
 replica.close();
   }
 
+  private static void setReplicaExternalBase(FileSystem fs, Map confMap) throws IOException {
+fs.mkdirs(REPLICA_EXTERNAL_BASE);
+fullyQualifiedReplicaExternalBase =  
fs.getFileStatus(REPLICA_EXTERNAL_BASE).getPath().toString();
+confMap.put(HiveConf.ConfVars.REPL_EXTERNAL_TABLE_BASE_DIR.varname, 
fullyQualifiedReplicaExternalBase);

Review comment:
   Yes, needed both the places. Both are different case.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 452569)
Time Spent: 0.5h  (was: 20m)

> Mandate fully qualified absolute path for external table base dir during REPL 
> operation
> ---
>
> Key: HIVE-23611
> URL: https://issues.apache.org/jira/browse/HIVE-23611
> Project: Hive
>  Issue Type: Improvement
>Reporter: Pravin Sinha
>Assignee: Pravin Sinha
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-23611.01.patch, HIVE-23611.02.patch
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (HIVE-23611) Mandate fully qualified absolute path for external table base dir during REPL operation

2020-06-29 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-23611?focusedWorklogId=452316&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-452316
 ]

ASF GitHub Bot logged work on HIVE-23611:
-

Author: ASF GitHub Bot
Created on: 29/Jun/20 13:30
Start Date: 29/Jun/20 13:30
Worklog Time Spent: 10m 
  Work Description: aasha commented on a change in pull request #1120:
URL: https://github.com/apache/hive/pull/1120#discussion_r444723985



##
File path: 
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosExternalTables.java
##
@@ -517,7 +508,7 @@ public void externalTableIncrementalReplication() throws 
Throwable {
 + "'")
 .run("alter table t1 add partition(country='india')")
 .run("alter table t1 add partition(country='us')")
-.dump(primaryDbName, withClause);

Review comment:
   why is the withclause removed from here?

##
File path: 
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/BaseReplicationAcrossInstances.java
##
@@ -103,6 +114,12 @@ public static void classLevelTearDown() throws IOException 
{
 replica.close();
   }
 
+  private static void setReplicaExternalBase(FileSystem fs, Map confMap) throws IOException {
+fs.mkdirs(REPLICA_EXTERNAL_BASE);
+fullyQualifiedReplicaExternalBase =  
fs.getFileStatus(REPLICA_EXTERNAL_BASE).getPath().toString();
+confMap.put(HiveConf.ConfVars.REPL_EXTERNAL_TABLE_BASE_DIR.varname, 
fullyQualifiedReplicaExternalBase);

Review comment:
   this is set at 104 line also

##
File path: 
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/ReplicationTestUtils.java
##
@@ -502,19 +502,11 @@ public static void insertForMerge(WarehouseInstance 
primary, String primaryDbNam
 "creation", "creation", "merge_update", "merge_insert", 
"merge_insert"});
   }
 
-  public static List externalTableBasePathWithClause(String 
replExternalBase, WarehouseInstance replica)
-  throws IOException, SemanticException {
-Path externalTableLocation = new Path(replExternalBase);
-DistributedFileSystem fileSystem = replica.miniDFSCluster.getFileSystem();
-externalTableLocation = 
PathBuilder.fullyQualifiedHDFSUri(externalTableLocation, fileSystem);
-fileSystem.mkdirs(externalTableLocation);
-
-// this is required since the same filesystem is used in both source and 
target
-return Arrays.asList(
-"'" + HiveConf.ConfVars.REPL_EXTERNAL_TABLE_BASE_DIR.varname + 
"'='"
-+ externalTableLocation.toString() + "'",
-"'distcp.options.pugpb'=''"
-);
+  public static List externalTableClause(boolean enable) {

Review comment:
   should this be include external table clause

##
File path: 
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplExternalTables.java
##
@@ -66,6 +67,9 @@ private ReplExternalTables(){}
 
   public static String externalTableLocation(HiveConf hiveConf, String 
location) throws SemanticException {
 String baseDir = 
hiveConf.get(HiveConf.ConfVars.REPL_EXTERNAL_TABLE_BASE_DIR.varname);
+if (StringUtils.isEmpty(baseDir)) {

Review comment:
   At the time of load the REPL_EXTERNAL_TABLE_BASE_DIR fully qualified 
path is not needed?

##
File path: 
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosExternalTables.java
##
@@ -503,8 +495,7 @@ public void externalTableIncrementalCheckpointing() throws 
Throwable {
 
   @Test
   public void externalTableIncrementalReplication() throws Throwable {
-List withClause = externalTableBasePathWithClause();
-WarehouseInstance.Tuple tuple = primary.dump(primaryDbName, withClause);
+WarehouseInstance.Tuple tuple = primary.dump(primaryDbName);

Review comment:
   with clause removed?

##
File path: 
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosExternalTables.java
##
@@ -159,14 +154,14 @@ public void externalTableReplicationWithDefaultPaths() 
throws Throwable {
 .run("insert into table t2 partition(country='india') values 
('bangalore')")
 .run("insert into table t2 partition(country='us') values ('austin')")
 .run("insert into table t2 partition(country='france') values 
('paris')")
-.dump(primaryDbName, withClauseOptions);
+.dump(primaryDbName);

Review comment:
   why is the with clause removed?

##
File path: 
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosExternalTables.java
##
@@ -623,7 +612,7 @@ public void bootstrapExternalTablesDuringIncrementalPhase() 
throws Throwable {
 assertFalse(primary.miniDFSCluster.getFileSystem()
 .exists(new Path(metadataPath + relativeExtInfoPath(null;
 
-replica.load(repli

[jira] [Work logged] (HIVE-23611) Mandate fully qualified absolute path for external table base dir during REPL operation

2020-06-16 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-23611?focusedWorklogId=446362&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446362
 ]

ASF GitHub Bot logged work on HIVE-23611:
-

Author: ASF GitHub Bot
Created on: 16/Jun/20 08:40
Start Date: 16/Jun/20 08:40
Worklog Time Spent: 10m 
  Work Description: pkumarsinha opened a new pull request #1120:
URL: https://github.com/apache/hive/pull/1120


   … base dir during REPL operation
   
   ## NOTICE
   
   Please create an issue in ASF JIRA before opening a pull request,
   and you need to set the title of the pull request which starts with
   the corresponding JIRA issue number. (e.g. HIVE-X: Fix a typo in YYY)
   For more details, please see 
https://cwiki.apache.org/confluence/display/Hive/HowToContribute
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 446362)
Remaining Estimate: 0h
Time Spent: 10m

> Mandate fully qualified absolute path for external table base dir during REPL 
> operation
> ---
>
> Key: HIVE-23611
> URL: https://issues.apache.org/jira/browse/HIVE-23611
> Project: Hive
>  Issue Type: Improvement
>Reporter: Pravin Sinha
>Assignee: Pravin Sinha
>Priority: Major
> Attachments: HIVE-23611.01.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)