[jira] [Commented] (HIVE-16727) REPL DUMP for insert event should't fail if the table is already dropped.

2017-06-07 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16040548#comment-16040548
 ] 

ASF GitHub Bot commented on HIVE-16727:
---

Github user sankarh closed the pull request at:

https://github.com/apache/hive/pull/186


> REPL DUMP for insert event should't fail if the table is already dropped.
> -
>
> Key: HIVE-16727
> URL: https://issues.apache.org/jira/browse/HIVE-16727
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive, repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-16727.01.patch, HIVE-16727.02.patch
>
>
> Currently, insert event doesn't log the table object as part of event 
> notification message. During dump, the table object is obtained from 
> metastore which can be null if the table is already dropped and hence REPL 
> DUMP fails.
> Steps:
> 1. Bootstrap dump/load with a table.
> 2. Insert into the table.
> 3. Drop the table.
> 4. REPL DUMP (incremental).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16727) REPL DUMP for insert event should't fail if the table is already dropped.

2017-05-26 Thread anishek (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026180#comment-16026180
 ] 

anishek commented on HIVE-16727:


+1 Looks good . over to you [~thejas]/[~sushanth]

> REPL DUMP for insert event should't fail if the table is already dropped.
> -
>
> Key: HIVE-16727
> URL: https://issues.apache.org/jira/browse/HIVE-16727
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive, repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR, replication
> Attachments: HIVE-16727.01.patch, HIVE-16727.02.patch
>
>
> Currently, insert event doesn't log the table object as part of event 
> notification message. During dump, the table object is obtained from 
> metastore which can be null if the table is already dropped and hence REPL 
> DUMP fails.
> Steps:
> 1. Bootstrap dump/load with a table.
> 2. Insert into the table.
> 3. Drop the table.
> 4. REPL DUMP (incremental).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16727) REPL DUMP for insert event should't fail if the table is already dropped.

2017-05-26 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026153#comment-16026153
 ] 

Hive QA commented on HIVE-16727:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12870033/HIVE-16727.02.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10790 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[create_merge_compressed]
 (batchId=237)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
 (batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=145)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5440/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5440/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5440/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12870033 - PreCommit-HIVE-Build

> REPL DUMP for insert event should't fail if the table is already dropped.
> -
>
> Key: HIVE-16727
> URL: https://issues.apache.org/jira/browse/HIVE-16727
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive, repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR, replication
> Attachments: HIVE-16727.01.patch, HIVE-16727.02.patch
>
>
> Currently, insert event doesn't log the table object as part of event 
> notification message. During dump, the table object is obtained from 
> metastore which can be null if the table is already dropped and hence REPL 
> DUMP fails.
> Steps:
> 1. Bootstrap dump/load with a table.
> 2. Insert into the table.
> 3. Drop the table.
> 4. REPL DUMP (incremental).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16727) REPL DUMP for insert event should't fail if the table is already dropped.

2017-05-26 Thread anishek (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026044#comment-16026044
 ] 

anishek commented on HIVE-16727:


[~sankarh] You are correct in that there is common code for partitioned and 
unpartitioned code but that can may be abstracted out as a common code, its 
just that the test becomes difficult to read, And the idea of having two test 
cases as incrementalInsertDropPartitionTable and 
incrementalInsertDropUnpartitionTable vs having one method doing both, on 
failures the test name will tell you what failed rather than looking at the 
line in the test where it failed and trying to figure out which one of them is 
failing. I just wanted to bring out the point that readability might be better.

I was just going by the comments on the method get_table_core, as you said 
there is no difference in execution path for anyone of them just additional 
request object setup and as such in get_table_req method. 


> REPL DUMP for insert event should't fail if the table is already dropped.
> -
>
> Key: HIVE-16727
> URL: https://issues.apache.org/jira/browse/HIVE-16727
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive, repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR, replication
> Attachments: HIVE-16727.01.patch
>
>
> Currently, insert event doesn't log the table object as part of event 
> notification message. During dump, the table object is obtained from 
> metastore which can be null if the table is already dropped and hence REPL 
> DUMP fails.
> Steps:
> 1. Bootstrap dump/load with a table.
> 2. Insert into the table.
> 3. Drop the table.
> 4. REPL DUMP (incremental).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16727) REPL DUMP for insert event should't fail if the table is already dropped.

2017-05-26 Thread Sankar Hariappan (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026037#comment-16026037
 ] 

Sankar Hariappan commented on HIVE-16727:
-

[~anishek], 
Please find my responses here and let me know if any concerns.

{quote}
TestReplicationScenarios : The test case seems to test both partitioned and 
unpartitioned cases in the same test case, It would be easier to spot errors if 
they were split if into two tests cases one for partition and another for 
unpartitioned tables, will also additionally make it more readable. Might be 
possible to abstract most of the commands using the WarehouseInstance 
{quote}
- As the InsertEvent handling is common for both partitioned and unpartitioned 
table, I intended to keep one test case for this scenario. Also, REPL dump will 
process these insert events one by one which helps to spot the errors easily if 
one fails.Using WarehouseInstance, I shall able to abstract the dump commands 
but it is not too many in this test case and this change can be more, so, think 
of doing it for future patches. :)

{quote}
InsertEvent: get_table_core could be used since this is for internal hive 
classes.
{quote}
- In the base code for Repl dump was using get_table_req after setting this 
req.setCapabilities(HiveMetaStoreClient.TEST_VERSION); Not sure, why is it 
there. However, even get_table_req internally invokes get_table_core. So, 
shouldn't be a problem I guess.

{quote}
There are unused Imports in most of the classes above.
{quote}
- Yes, agreed. Will share a patch removing all these unused imports from files 
modified.

> REPL DUMP for insert event should't fail if the table is already dropped.
> -
>
> Key: HIVE-16727
> URL: https://issues.apache.org/jira/browse/HIVE-16727
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive, repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR, replication
> Attachments: HIVE-16727.01.patch
>
>
> Currently, insert event doesn't log the table object as part of event 
> notification message. During dump, the table object is obtained from 
> metastore which can be null if the table is already dropped and hence REPL 
> DUMP fails.
> Steps:
> 1. Bootstrap dump/load with a table.
> 2. Insert into the table.
> 3. Drop the table.
> 4. REPL DUMP (incremental).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16727) REPL DUMP for insert event should't fail if the table is already dropped.

2017-05-26 Thread anishek (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025967#comment-16025967
 ] 

anishek commented on HIVE-16727:


[~sankarh]
* TestReplicationScenarios : The test case seems to test both partitioned and 
unpartitioned cases in the same test case, It would be easier to spot errors if 
they were split if into two tests cases one for partition and another for 
unpartitioned tables, will also additionally make it more readable. Might be 
possible to abstract most of the commands using the WarehouseInstance 
representation.
* InsertEvent: get_table_core could be used since this is for internal hive 
classes. 
* There are unused Imports in most of the classes above.


> REPL DUMP for insert event should't fail if the table is already dropped.
> -
>
> Key: HIVE-16727
> URL: https://issues.apache.org/jira/browse/HIVE-16727
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive, repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR, replication
> Attachments: HIVE-16727.01.patch
>
>
> Currently, insert event doesn't log the table object as part of event 
> notification message. During dump, the table object is obtained from 
> metastore which can be null if the table is already dropped and hence REPL 
> DUMP fails.
> Steps:
> 1. Bootstrap dump/load with a table.
> 2. Insert into the table.
> 3. Drop the table.
> 4. REPL DUMP (incremental).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16727) REPL DUMP for insert event should't fail if the table is already dropped.

2017-05-23 Thread Sankar Hariappan (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022326#comment-16022326
 ] 

Sankar Hariappan commented on HIVE-16727:
-

The test failures are irrelevant to the patch!

> REPL DUMP for insert event should't fail if the table is already dropped.
> -
>
> Key: HIVE-16727
> URL: https://issues.apache.org/jira/browse/HIVE-16727
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive, repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR, replication
> Attachments: HIVE-16727.01.patch
>
>
> Currently, insert event doesn't log the table object as part of event 
> notification message. During dump, the table object is obtained from 
> metastore which can be null if the table is already dropped and hence REPL 
> DUMP fails.
> Steps:
> 1. Bootstrap dump/load with a table.
> 2. Insert into the table.
> 3. Drop the table.
> 4. REPL DUMP (incremental).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16727) REPL DUMP for insert event should't fail if the table is already dropped.

2017-05-23 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021488#comment-16021488
 ] 

Hive QA commented on HIVE-16727:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869480/HIVE-16727.01.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10750 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite]
 (batchId=236)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join30]
 (batchId=149)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5400/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5400/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5400/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12869480 - PreCommit-HIVE-Build

> REPL DUMP for insert event should't fail if the table is already dropped.
> -
>
> Key: HIVE-16727
> URL: https://issues.apache.org/jira/browse/HIVE-16727
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive, repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR, replication
> Attachments: HIVE-16727.01.patch
>
>
> Currently, insert event doesn't log the table object as part of event 
> notification message. During dump, the table object is obtained from 
> metastore which can be null if the table is already dropped and hence REPL 
> DUMP fails.
> Steps:
> 1. Bootstrap dump/load with a table.
> 2. Insert into the table.
> 3. Drop the table.
> 4. REPL DUMP (incremental).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16727) REPL DUMP for insert event should't fail if the table is already dropped.

2017-05-23 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021370#comment-16021370
 ] 

ASF GitHub Bot commented on HIVE-16727:
---

GitHub user sankarh opened a pull request:

https://github.com/apache/hive/pull/186

HIVE-16727: REPL DUMP for insert event should't fail if the table is 
already dropped.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sankarh/hive HIVE-16727

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/186.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #186


commit c06a9df4c2c1a735dc8c0139ebdf6451cc6c9017
Author: Sankar Hariappan 
Date:   2017-05-22T08:15:54Z

HIVE-16727: REPL DUMP for insert event should't fail if the table is 
already dropped.




> REPL DUMP for insert event should't fail if the table is already dropped.
> -
>
> Key: HIVE-16727
> URL: https://issues.apache.org/jira/browse/HIVE-16727
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive, repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>
> Currently, insert event doesn't log the table object as part of event 
> notification message. During dump, the table object is obtained from 
> metastore which can be null if the table is already dropped and hence REPL 
> DUMP fails.
> Steps:
> 1. Bootstrap dump/load with a table.
> 2. Insert into the table.
> 3. Drop the table.
> 4. REPL DUMP (incremental).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)