[jira] [Commented] (HIVE-22125) Move to Kafka 2.3 Clients
[
https://issues.apache.org/jira/browse/HIVE-22125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16911807#comment-16911807
]
Hive QA commented on HIVE-22125:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12978110/HIVE-22125.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16743 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18370/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18370/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18370/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12978110 - PreCommit-HIVE-Build
> Move to Kafka 2.3 Clients
> -
>
> Key: HIVE-22125
> URL: https://issues.apache.org/jira/browse/HIVE-22125
> Project: Hive
> Issue Type: Improvement
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-22125.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-22125) Move to Kafka 2.3 Clients
[
https://issues.apache.org/jira/browse/HIVE-22125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16911770#comment-16911770
]
Hive QA commented on HIVE-22125:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} kafka-handler: The patch generated 3 new + 0 unchanged
- 0 fixed = 3 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18370/dev-support/hive-personality.sh
|
| git revision | master / c0341dc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18370/yetus/diff-checkstyle-kafka-handler.txt
|
| modules | C: kafka-handler U: kafka-handler |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18370/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Move to Kafka 2.3 Clients
> -
>
> Key: HIVE-22125
> URL: https://issues.apache.org/jira/browse/HIVE-22125
> Project: Hive
> Issue Type: Improvement
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-22125.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-22068) Return the last event id dumped as repl status to avoid notification event missing error.
[
https://issues.apache.org/jira/browse/HIVE-22068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16911738#comment-16911738
]
Hive QA commented on HIVE-22068:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12978097/HIVE-22068.06.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 16741 tests
executed
*Failed tests:*
{noformat}
TestTriggersNoTezSessionPool - did not produce a TEST-*.xml file (likely timed
out) (batchId=280)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesBootstrapWithJsonMessage.testAcidTablesBootstrapDuringIncrementalWithOpenTxnsTimeout
(batchId=250)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTablesBootstrap.testAcidTablesBootstrapDuringIncrementalWithOpenTxnsTimeout
(batchId=248)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18369/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18369/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18369/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12978097 - PreCommit-HIVE-Build
> Return the last event id dumped as repl status to avoid notification event
> missing error.
> -
>
> Key: HIVE-22068
> URL: https://issues.apache.org/jira/browse/HIVE-22068
> Project: Hive
> Issue Type: Improvement
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22068.01.patch, HIVE-22068.02.patch,
> HIVE-22068.03.patch, HIVE-22068.04.patch, HIVE-22068.05.patch,
> HIVE-22068.06.patch
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> In repl load, update the status of target database to the last event dumped
> so that repl status returns that and next incremental can specify it as the
> event from which to start the dump. WIthout that repl status might return and
> old event which might cause, older events to be dumped again and/or a
> notification event missing error if the older events are cleaned by the
> cleaner.
> While at it
> * Add more logging to DB notification listener cleaner thread
> ** The time when it considered cleaning, the interval and time before which
> events were cleared, the min and max id at that time
> ** how many events were cleared
> ** min and max id after the cleaning.
> * In REPL::START document the starting event, end event if specified and the
> maximum number of events, if specified.
> *
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-22068) Return the last event id dumped as repl status to avoid notification event missing error.
[
https://issues.apache.org/jira/browse/HIVE-22068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16911726#comment-16911726
]
Hive QA commented on HIVE-22068:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
12s{color} | {color:blue} standalone-metastore/metastore-server in master has
181 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
4s{color} | {color:blue} ql in master has 2251 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 2 new + 25 unchanged - 0 fixed
= 27 total (was 25) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
15s{color} | {color:red} ql generated 3 new + 2251 unchanged - 0 fixed = 2254
total (was 2251) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Unread field:IncrementalDumpBegin.java:[line 52] |
| | Unread field:IncrementalDumpBegin.java:[line 54] |
| | Unread field:IncrementalDumpBegin.java:[line 53] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18369/dev-support/hive-personality.sh
|
| git revision | master / c0341dc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18369/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18369/yetus/new-findbugs-ql.html
|
| modules | C: standalone-metastore/metastore-server ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18369/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Return the last event id dumped as repl status to avoid notification event
> missing error.
> -
>
> Key: HIVE-22068
> URL: https://issues.apache.org/jira/browse/HIVE-22068
> Project: Hive
> Issue Type: Improvement
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels:
[jira] [Updated] (HIVE-22125) Move to Kafka 2.3 Clients
[ https://issues.apache.org/jira/browse/HIVE-22125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-22125: -- Status: Patch Available (was: Open) > Move to Kafka 2.3 Clients > - > > Key: HIVE-22125 > URL: https://issues.apache.org/jira/browse/HIVE-22125 > Project: Hive > Issue Type: Improvement >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Attachments: HIVE-22125.patch > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22125) Move to Kafka 2.3 Clients
[ https://issues.apache.org/jira/browse/HIVE-22125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-22125: -- Attachment: HIVE-22125.patch > Move to Kafka 2.3 Clients > - > > Key: HIVE-22125 > URL: https://issues.apache.org/jira/browse/HIVE-22125 > Project: Hive > Issue Type: Improvement >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Attachments: HIVE-22125.patch > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22127) Query Routing logging appender is leaking resources of RandomAccessFileManager.
[
https://issues.apache.org/jira/browse/HIVE-22127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-22127:
--
Component/s: Logging
> Query Routing logging appender is leaking resources of
> RandomAccessFileManager.
> ---
>
> Key: HIVE-22127
> URL: https://issues.apache.org/jira/browse/HIVE-22127
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
>
> Query routing appender registered by
> {code:java}
> org.apache.hadoop.hive.ql.log.LogDivertAppender#registerRoutingAppender
> {code}
> Is leaking reference to the
> {code} org.apache.hadoop.hive.ql.log.HushableRandomAccessFileAppender {code}
> On closing operation hooks
> {code}
> org.apache.hive.service.cli.operation.Operation#cleanupOperationLog
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22115) Prevent the creation of query-router logger in HS2 as per property
[
https://issues.apache.org/jira/browse/HIVE-22115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-22115:
--
Component/s: Logging
> Prevent the creation of query-router logger in HS2 as per property
> --
>
> Key: HIVE-22115
> URL: https://issues.apache.org/jira/browse/HIVE-22115
> Project: Hive
> Issue Type: Improvement
> Components: Logging
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-22115.patch, HIVE-22115.patch, HIVE-22115.patch
>
>
> Avoid the creation and registration of query-router logger if the Hive server
> Property is set to false by the user
> {code}
> HiveConf.ConfVars.HIVE_SERVER2_LOGGING_OPERATION_ENABLED
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Assigned] (HIVE-22127) Query Routing logging appender is leaking resources of RandomAccessFileManager.
[
https://issues.apache.org/jira/browse/HIVE-22127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra reassigned HIVE-22127:
-
> Query Routing logging appender is leaking resources of
> RandomAccessFileManager.
> ---
>
> Key: HIVE-22127
> URL: https://issues.apache.org/jira/browse/HIVE-22127
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
>
> Query routing appender registered by
> {code:java}
> org.apache.hadoop.hive.ql.log.LogDivertAppender#registerRoutingAppender
> {code}
> Is leaking reference to the
> {code} org.apache.hadoop.hive.ql.log.HushableRandomAccessFileAppender {code}
> On closing operation hooks
> {code}
> org.apache.hive.service.cli.operation.Operation#cleanupOperationLog
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22115) Prevent the creation of query-router logger in HS2 as per property
[
https://issues.apache.org/jira/browse/HIVE-22115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-22115:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
fixed by commit c0341dcf0602b06fd4e8441d833d708b709164a2
> Prevent the creation of query-router logger in HS2 as per property
> --
>
> Key: HIVE-22115
> URL: https://issues.apache.org/jira/browse/HIVE-22115
> Project: Hive
> Issue Type: Improvement
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-22115.patch, HIVE-22115.patch, HIVE-22115.patch
>
>
> Avoid the creation and registration of query-router logger if the Hive server
> Property is set to false by the user
> {code}
> HiveConf.ConfVars.HIVE_SERVER2_LOGGING_OPERATION_ENABLED
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22068) Return the last event id dumped as repl status to avoid notification event missing error.
[ https://issues.apache.org/jira/browse/HIVE-22068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-22068: -- Attachment: HIVE-22068.06.patch Status: Patch Available (was: In Progress) Patch addressing [~sankarh]'s comments. Added a new testcase to cover bootstrap during incremental case. > Return the last event id dumped as repl status to avoid notification event > missing error. > - > > Key: HIVE-22068 > URL: https://issues.apache.org/jira/browse/HIVE-22068 > Project: Hive > Issue Type: Improvement >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22068.01.patch, HIVE-22068.02.patch, > HIVE-22068.03.patch, HIVE-22068.04.patch, HIVE-22068.05.patch, > HIVE-22068.06.patch > > Time Spent: 2h 20m > Remaining Estimate: 0h > > In repl load, update the status of target database to the last event dumped > so that repl status returns that and next incremental can specify it as the > event from which to start the dump. WIthout that repl status might return and > old event which might cause, older events to be dumped again and/or a > notification event missing error if the older events are cleaned by the > cleaner. > While at it > * Add more logging to DB notification listener cleaner thread > ** The time when it considered cleaning, the interval and time before which > events were cleared, the min and max id at that time > ** how many events were cleared > ** min and max id after the cleaning. > * In REPL::START document the starting event, end event if specified and the > maximum number of events, if specified. > * -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22068) Return the last event id dumped as repl status to avoid notification event missing error.
[ https://issues.apache.org/jira/browse/HIVE-22068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-22068: -- Status: In Progress (was: Patch Available) > Return the last event id dumped as repl status to avoid notification event > missing error. > - > > Key: HIVE-22068 > URL: https://issues.apache.org/jira/browse/HIVE-22068 > Project: Hive > Issue Type: Improvement >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22068.01.patch, HIVE-22068.02.patch, > HIVE-22068.03.patch, HIVE-22068.04.patch, HIVE-22068.05.patch > > Time Spent: 2h 20m > Remaining Estimate: 0h > > In repl load, update the status of target database to the last event dumped > so that repl status returns that and next incremental can specify it as the > event from which to start the dump. WIthout that repl status might return and > old event which might cause, older events to be dumped again and/or a > notification event missing error if the older events are cleaned by the > cleaner. > While at it > * Add more logging to DB notification listener cleaner thread > ** The time when it considered cleaning, the interval and time before which > events were cleared, the min and max id at that time > ** how many events were cleared > ** min and max id after the cleaning. > * In REPL::START document the starting event, end event if specified and the > maximum number of events, if specified. > * -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (HIVE-22068) Return the last event id dumped as repl status to avoid notification event missing error.
[
https://issues.apache.org/jira/browse/HIVE-22068?focusedWorklogId=297970=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297970
]
ASF GitHub Bot logged work on HIVE-22068:
-
Author: ASF GitHub Bot
Created on: 20/Aug/19 15:38
Start Date: 20/Aug/19 15:38
Worklog Time Spent: 10m
Work Description: ashutosh-bapat commented on pull request #742:
HIVE-22068 : Add more logging to notification cleaner and replication to track
events
URL: https://github.com/apache/hive/pull/742#discussion_r315762673
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##
@@ -522,6 +525,41 @@ private int executeIncrementalLoad(DriverContext
driverContext) {
// bootstrap of tables if exist.
if (builder.hasMoreWork() || work.getPathsToCopyIterator().hasNext() ||
work.hasBootstrapLoadTasks()) {
DAGTraversal.traverse(childTasks, new
AddDependencyToLeaves(TaskFactory.get(work, conf)));
+ } else {
+// Nothing to be done for repl load now. Add a task to update the
last.repl.id of the
+// target database to the event id of the last event considered by the
dump. Next
+// incremental cycle if starts from this id, the events considered for
this dump, won't
+// be considered again.
+
+// The name of the database to be loaded into is either specified
directly or is
+// available from the dump metadata.
+String dbName = work.dbNameToLoadIn;
+if (dbName == null || StringUtils.isNotBlank(dbName)) {
+ if (work.currentReplScope != null) {
Review comment:
Done. Please check and suggest improvement if necessary.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 297970)
Time Spent: 2h 20m (was: 2h 10m)
> Return the last event id dumped as repl status to avoid notification event
> missing error.
> -
>
> Key: HIVE-22068
> URL: https://issues.apache.org/jira/browse/HIVE-22068
> Project: Hive
> Issue Type: Improvement
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22068.01.patch, HIVE-22068.02.patch,
> HIVE-22068.03.patch, HIVE-22068.04.patch, HIVE-22068.05.patch
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> In repl load, update the status of target database to the last event dumped
> so that repl status returns that and next incremental can specify it as the
> event from which to start the dump. WIthout that repl status might return and
> old event which might cause, older events to be dumped again and/or a
> notification event missing error if the older events are cleaned by the
> cleaner.
> While at it
> * Add more logging to DB notification listener cleaner thread
> ** The time when it considered cleaning, the interval and time before which
> events were cleared, the min and max id at that time
> ** how many events were cleared
> ** min and max id after the cleaning.
> * In REPL::START document the starting event, end event if specified and the
> maximum number of events, if specified.
> *
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (HIVE-22068) Return the last event id dumped as repl status to avoid notification event missing error.
[
https://issues.apache.org/jira/browse/HIVE-22068?focusedWorklogId=297846=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297846
]
ASF GitHub Bot logged work on HIVE-22068:
-
Author: ASF GitHub Bot
Created on: 20/Aug/19 12:54
Start Date: 20/Aug/19 12:54
Worklog Time Spent: 10m
Work Description: ashutosh-bapat commented on pull request #742:
HIVE-22068 : Add more logging to notification cleaner and replication to track
events
URL: https://github.com/apache/hive/pull/742#discussion_r315670865
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##
@@ -522,6 +525,41 @@ private int executeIncrementalLoad(DriverContext
driverContext) {
// bootstrap of tables if exist.
if (builder.hasMoreWork() || work.getPathsToCopyIterator().hasNext() ||
work.hasBootstrapLoadTasks()) {
DAGTraversal.traverse(childTasks, new
AddDependencyToLeaves(TaskFactory.get(work, conf)));
+ } else {
+// Nothing to be done for repl load now. Add a task to update the
last.repl.id of the
+// target database to the event id of the last event considered by the
dump. Next
+// incremental cycle if starts from this id, the events considered for
this dump, won't
+// be considered again.
+
+// The name of the database to be loaded into is either specified
directly or is
+// available from the dump metadata.
+String dbName = work.dbNameToLoadIn;
+if (dbName == null || StringUtils.isNotBlank(dbName)) {
Review comment:
Thanks for catching this. Fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 297846)
Time Spent: 2h 10m (was: 2h)
> Return the last event id dumped as repl status to avoid notification event
> missing error.
> -
>
> Key: HIVE-22068
> URL: https://issues.apache.org/jira/browse/HIVE-22068
> Project: Hive
> Issue Type: Improvement
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22068.01.patch, HIVE-22068.02.patch,
> HIVE-22068.03.patch, HIVE-22068.04.patch, HIVE-22068.05.patch
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> In repl load, update the status of target database to the last event dumped
> so that repl status returns that and next incremental can specify it as the
> event from which to start the dump. WIthout that repl status might return and
> old event which might cause, older events to be dumped again and/or a
> notification event missing error if the older events are cleaned by the
> cleaner.
> While at it
> * Add more logging to DB notification listener cleaner thread
> ** The time when it considered cleaning, the interval and time before which
> events were cleared, the min and max id at that time
> ** how many events were cleared
> ** min and max id after the cleaning.
> * In REPL::START document the starting event, end event if specified and the
> maximum number of events, if specified.
> *
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (HIVE-22068) Return the last event id dumped as repl status to avoid notification event missing error.
[
https://issues.apache.org/jira/browse/HIVE-22068?focusedWorklogId=297843=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297843
]
ASF GitHub Bot logged work on HIVE-22068:
-
Author: ASF GitHub Bot
Created on: 20/Aug/19 12:51
Start Date: 20/Aug/19 12:51
Worklog Time Spent: 10m
Work Description: ashutosh-bapat commented on pull request #742:
HIVE-22068 : Add more logging to notification cleaner and replication to track
events
URL: https://github.com/apache/hive/pull/742#discussion_r315669736
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosExternalTables.java
##
@@ -750,6 +766,38 @@ public Table apply(@Nullable Table table) {
.verifyResults(Arrays.asList("1", "2"));
}
+ @Test
+ public void testIncrementalDumpEmptyDumpDirectory() throws Throwable {
Review comment:
Added a testcase with external table bootstrap. There I could reproduce the
problem you mentioned and also fixed it. During an incremental the last repl id
is updated for the database after applying all the events but before
bootstrapping any tables.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 297843)
Time Spent: 2h (was: 1h 50m)
> Return the last event id dumped as repl status to avoid notification event
> missing error.
> -
>
> Key: HIVE-22068
> URL: https://issues.apache.org/jira/browse/HIVE-22068
> Project: Hive
> Issue Type: Improvement
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22068.01.patch, HIVE-22068.02.patch,
> HIVE-22068.03.patch, HIVE-22068.04.patch, HIVE-22068.05.patch
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> In repl load, update the status of target database to the last event dumped
> so that repl status returns that and next incremental can specify it as the
> event from which to start the dump. WIthout that repl status might return and
> old event which might cause, older events to be dumped again and/or a
> notification event missing error if the older events are cleaned by the
> cleaner.
> While at it
> * Add more logging to DB notification listener cleaner thread
> ** The time when it considered cleaning, the interval and time before which
> events were cleared, the min and max id at that time
> ** how many events were cleared
> ** min and max id after the cleaning.
> * In REPL::START document the starting event, end event if specified and the
> maximum number of events, if specified.
> *
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
