[jira] [Commented] (HIVE-14737) Problem accessing /logs in a Kerberized Hive Server 2 Web UI
[
https://issues.apache.org/jira/browse/HIVE-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16870083#comment-16870083
]
Hive QA commented on HIVE-14737:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} common: The patch generated 1 new + 20 unchanged - 0
fixed = 21 total (was 20) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17699/dev-support/hive-personality.sh
|
| git revision | master / 9890919 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17699/yetus/diff-checkstyle-common.txt
|
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17699/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Problem accessing /logs in a Kerberized Hive Server 2 Web UI
>
>
> Key: HIVE-14737
> URL: https://issues.apache.org/jira/browse/HIVE-14737
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Matyas Orhidi
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-14737.01.patch, HIVE-14737.02.patch,
> HIVE-14737.patch
>
>
> The /logs menu fails with error [1] when the cluster is Kerberized. Other
> menu items are working properly.
> [1] HTTP ERROR: 401
> Problem accessing /logs/. Reason:
> Unauthenticated users are not authorized to access this page.
> Powered by Jetty://
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21914) Move Function and Macro related DDL operations into the DDL framework
[
https://issues.apache.org/jira/browse/HIVE-21914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16870076#comment-16870076
]
Hive QA commented on HIVE-21914:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972504/HIVE-21914.02.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16335 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT[2] (batchId=350)
org.apache.hadoop.hive.metastore.TestPartitionManagement.testPartitionDiscoveryTransactionalTable
(batchId=222)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17698/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17698/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17698/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972504 - PreCommit-HIVE-Build
> Move Function and Macro related DDL operations into the DDL framework
> -
>
> Key: HIVE-21914
> URL: https://issues.apache.org/jira/browse/HIVE-21914
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: refactor-ddl
> Fix For: 4.0.0
>
> Attachments: HIVE-21914.01.patch, HIVE-21914.02.patch
>
>
> Some Function and Macro related operations are handled by FunctionTask, and
> FunctionWork while they belong to the DDL framework.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-14737) Problem accessing /logs in a Kerberized Hive Server 2 Web UI
[ https://issues.apache.org/jira/browse/HIVE-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-14737: -- Attachment: HIVE-14737.02.patch Status: Patch Available (was: Open) > Problem accessing /logs in a Kerberized Hive Server 2 Web UI > > > Key: HIVE-14737 > URL: https://issues.apache.org/jira/browse/HIVE-14737 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Matyas Orhidi >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-14737.01.patch, HIVE-14737.02.patch, > HIVE-14737.patch > > > The /logs menu fails with error [1] when the cluster is Kerberized. Other > menu items are working properly. > [1] HTTP ERROR: 401 > Problem accessing /logs/. Reason: > Unauthenticated users are not authorized to access this page. > Powered by Jetty:// -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-14737) Problem accessing /logs in a Kerberized Hive Server 2 Web UI
[ https://issues.apache.org/jira/browse/HIVE-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-14737: -- Status: Open (was: Patch Available) > Problem accessing /logs in a Kerberized Hive Server 2 Web UI > > > Key: HIVE-14737 > URL: https://issues.apache.org/jira/browse/HIVE-14737 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Matyas Orhidi >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-14737.01.patch, HIVE-14737.02.patch, > HIVE-14737.patch > > > The /logs menu fails with error [1] when the cluster is Kerberized. Other > menu items are working properly. > [1] HTTP ERROR: 401 > Problem accessing /logs/. Reason: > Unauthenticated users are not authorized to access this page. > Powered by Jetty:// -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-14737) Problem accessing /logs in a Kerberized Hive Server 2 Web UI
[ https://issues.apache.org/jira/browse/HIVE-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-14737: -- Attachment: HIVE-14737.02.patch > Problem accessing /logs in a Kerberized Hive Server 2 Web UI > > > Key: HIVE-14737 > URL: https://issues.apache.org/jira/browse/HIVE-14737 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Matyas Orhidi >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-14737.01.patch, HIVE-14737.02.patch, > HIVE-14737.patch > > > The /logs menu fails with error [1] when the cluster is Kerberized. Other > menu items are working properly. > [1] HTTP ERROR: 401 > Problem accessing /logs/. Reason: > Unauthenticated users are not authorized to access this page. > Powered by Jetty:// -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21914) Move Function and Macro related DDL operations into the DDL framework
[
https://issues.apache.org/jira/browse/HIVE-21914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16870054#comment-16870054
]
Hive QA commented on HIVE-21914:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
6s{color} | {color:blue} ql in master has 2254 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
44s{color} | {color:blue} llap-server in master has 82 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
44s{color} | {color:red} ql: The patch generated 1 new + 278 unchanged - 17
fixed = 279 total (was 295) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17698/dev-support/hive-personality.sh
|
| git revision | master / 9890919 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17698/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql llap-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17698/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Move Function and Macro related DDL operations into the DDL framework
> -
>
> Key: HIVE-21914
> URL: https://issues.apache.org/jira/browse/HIVE-21914
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: refactor-ddl
> Fix For: 4.0.0
>
> Attachments: HIVE-21914.01.patch, HIVE-21914.02.patch
>
>
> Some Function and Macro related operations are handled by FunctionTask, and
> FunctionWork while they belong to the DDL framework.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[ https://issues.apache.org/jira/browse/HIVE-21764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16870052#comment-16870052 ] Sankar Hariappan commented on HIVE-21764: - +1, [^HIVE-21764.03.patch] LGTM > REPL DUMP should detect and bootstrap any rename table events where old table > was excluded but renamed table is included. > - > > Key: HIVE-21764 > URL: https://issues.apache.org/jira/browse/HIVE-21764 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Sankar Hariappan >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch, > HIVE-21764.03.patch > > Time Spent: 7h 10m > Remaining Estimate: 0h > > REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular > expression + inclusion/exclusion list. So, in case of rename table event, the > event will be ignored if old table doesn't match the pattern but the new > table should be bootstrapped. REPL DUMP should have a mechanism to detect > such tables and automatically bootstrap with incremental replication.Also, if > renamed table is excluded from replication policy, then need to drop the old > table at target as well. > There are 4 scenarios that needs to be handled. > # Both new name and old name satisfies the table name pattern filter. > ## No need to do anything. The incremental event for rename should take care > of the replication. > # Both the names does not satisfy the table name pattern filter. > ## Both the names are not in the scope of the policy and thus nothing needs > to be done. > # New name satisfies the pattern but the old name does not. > ## The table will not be present at the target. > ## Rename event handler for dump should detect this case and add the new > table name to the list of table for bootstrap. > ## All the events related to the table (new name) should be ignored. > ## If there is a drop event for the table (with new name), then remove the > table from the list of tables to be bootstrapped. > ## In case of rename (double rename) > ### If the new name satisfies the table pattern, then add the new name to > the list of tables to be bootstrapped and remove the old name from the list > of tables to be bootstrapped. > ### If the new name does not satisfies then just removed the table name from > the list of tables to be bootstrapped. > # New name does not satisfies the pattern but the old name satisfies. > ## Change the rename event to a drop event. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21857) Sort conditions in a filter predicate to accelerate query processing
[
https://issues.apache.org/jira/browse/HIVE-21857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16870047#comment-16870047
]
Hive QA commented on HIVE-21857:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972498/HIVE-21857.06.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17697/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17697/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17697/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12972498/HIVE-21857.06.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972498 - PreCommit-HIVE-Build
> Sort conditions in a filter predicate to accelerate query processing
>
>
> Key: HIVE-21857
> URL: https://issues.apache.org/jira/browse/HIVE-21857
> Project: Hive
> Issue Type: New Feature
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21857.01.patch, HIVE-21857.02.patch,
> HIVE-21857.03.patch, HIVE-21857.04.patch, HIVE-21857.05.patch,
> HIVE-21857.06.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Following approach similar to
> http://db.cs.berkeley.edu/jmh/miscpapers/sigmod93.pdf .
> To reorder predicates in AND conditions, we could rank each of elements in
> the clauses in increasing order based on following formula:
> {code}
> rank = (selectivity - 1) / cost per tuple
> {code}
> Similarly, for OR conditions:
> {code}
> rank = (-selectivity) / cost per tuple
> {code}
> Selectivity can be computed with FilterSelectivityEstimator. For cost per
> tuple, we will need to come up with some heuristic based on how expensive is
> the evaluation of the functions contained in that predicate. Custom UDFs
> could be annotated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-14737) Problem accessing /logs in a Kerberized Hive Server 2 Web UI
[
https://issues.apache.org/jira/browse/HIVE-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16870045#comment-16870045
]
Hive QA commented on HIVE-14737:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972490/HIVE-14737.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16303 tests
executed
*Failed tests:*
{noformat}
TestDataSourceProviderFactory - did not produce a TEST-*.xml file (likely timed
out) (batchId=232)
TestObjectStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=232)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17696/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17696/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17696/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972490 - PreCommit-HIVE-Build
> Problem accessing /logs in a Kerberized Hive Server 2 Web UI
>
>
> Key: HIVE-14737
> URL: https://issues.apache.org/jira/browse/HIVE-14737
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Matyas Orhidi
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-14737.01.patch, HIVE-14737.patch
>
>
> The /logs menu fails with error [1] when the cluster is Kerberized. Other
> menu items are working properly.
> [1] HTTP ERROR: 401
> Problem accessing /logs/. Reason:
> Unauthenticated users are not authorized to access this page.
> Powered by Jetty://
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-14737) Problem accessing /logs in a Kerberized Hive Server 2 Web UI
[
https://issues.apache.org/jira/browse/HIVE-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16870028#comment-16870028
]
Hive QA commented on HIVE-14737:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} common: The patch generated 1 new + 20 unchanged - 0
fixed = 21 total (was 20) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17696/dev-support/hive-personality.sh
|
| git revision | master / 9890919 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17696/yetus/diff-checkstyle-common.txt
|
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17696/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Problem accessing /logs in a Kerberized Hive Server 2 Web UI
>
>
> Key: HIVE-14737
> URL: https://issues.apache.org/jira/browse/HIVE-14737
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Matyas Orhidi
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-14737.01.patch, HIVE-14737.patch
>
>
> The /logs menu fails with error [1] when the cluster is Kerberized. Other
> menu items are working properly.
> [1] HTTP ERROR: 401
> Problem accessing /logs/. Reason:
> Unauthenticated users are not authorized to access this page.
> Powered by Jetty://
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21825) Improve client error msg when Active/Passive HA is enabled
[
https://issues.apache.org/jira/browse/HIVE-21825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16870023#comment-16870023
]
Hive QA commented on HIVE-21825:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972485/Hive-21825.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 16336 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs
(batchId=224)
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
(batchId=224)
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
(batchId=224)
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
(batchId=224)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17695/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17695/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17695/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972485 - PreCommit-HIVE-Build
> Improve client error msg when Active/Passive HA is enabled
> --
>
> Key: HIVE-21825
> URL: https://issues.apache.org/jira/browse/HIVE-21825
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Richard Zhang
>Priority: Major
> Labels: pull-request-available
> Attachments: Hive-21825.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When Active/Passive HA is enabled and when client tries to connect to Passive
> HA or when HS2 is still starting up, clients will receive the following the
> error msg
> {code:java}
> 'Cannot open sessions on an inactive HS2 instance; use service discovery to
> connect'{code}
> This error msg can be improved to say that HS2 is still starting up (or more
> user-friendly error msg).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21825) Improve client error msg when Active/Passive HA is enabled
[
https://issues.apache.org/jira/browse/HIVE-21825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16870001#comment-16870001
]
Hive QA commented on HIVE-21825:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17695/dev-support/hive-personality.sh
|
| git revision | master / 9890919 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: service U: service |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17695/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improve client error msg when Active/Passive HA is enabled
> --
>
> Key: HIVE-21825
> URL: https://issues.apache.org/jira/browse/HIVE-21825
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Richard Zhang
>Priority: Major
> Labels: pull-request-available
> Attachments: Hive-21825.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When Active/Passive HA is enabled and when client tries to connect to Passive
> HA or when HS2 is still starting up, clients will receive the following the
> error msg
> {code:java}
> 'Cannot open sessions on an inactive HS2 instance; use service discovery to
> connect'{code}
> This error msg can be improved to say that HS2 is still starting up (or more
> user-friendly error msg).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21857) Sort conditions in a filter predicate to accelerate query processing
[
https://issues.apache.org/jira/browse/HIVE-21857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869988#comment-16869988
]
Hive QA commented on HIVE-21857:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972498/HIVE-21857.06.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16335 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_ts]
(batchId=197)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[external_jdbc_table_perf]
(batchId=184)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17694/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17694/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17694/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972498 - PreCommit-HIVE-Build
> Sort conditions in a filter predicate to accelerate query processing
>
>
> Key: HIVE-21857
> URL: https://issues.apache.org/jira/browse/HIVE-21857
> Project: Hive
> Issue Type: New Feature
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21857.01.patch, HIVE-21857.02.patch,
> HIVE-21857.03.patch, HIVE-21857.04.patch, HIVE-21857.05.patch,
> HIVE-21857.06.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Following approach similar to
> http://db.cs.berkeley.edu/jmh/miscpapers/sigmod93.pdf .
> To reorder predicates in AND conditions, we could rank each of elements in
> the clauses in increasing order based on following formula:
> {code}
> rank = (selectivity - 1) / cost per tuple
> {code}
> Similarly, for OR conditions:
> {code}
> rank = (-selectivity) / cost per tuple
> {code}
> Selectivity can be computed with FilterSelectivityEstimator. For cost per
> tuple, we will need to come up with some heuristic based on how expensive is
> the evaluation of the functions contained in that predicate. Custom UDFs
> could be annotated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21914) Move Function and Macro related DDL operations into the DDL framework
[ https://issues.apache.org/jira/browse/HIVE-21914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21914: -- Attachment: HIVE-21914.02.patch > Move Function and Macro related DDL operations into the DDL framework > - > > Key: HIVE-21914 > URL: https://issues.apache.org/jira/browse/HIVE-21914 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21914.01.patch, HIVE-21914.02.patch > > > Some Function and Macro related operations are handled by FunctionTask, and > FunctionWork while they belong to the DDL framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21857) Sort conditions in a filter predicate to accelerate query processing
[
https://issues.apache.org/jira/browse/HIVE-21857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869976#comment-16869976
]
Hive QA commented on HIVE-21857:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
28s{color} | {color:blue} ql in master has 2254 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} accumulo-handler in master has 21 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
37s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
23s{color} | {color:red} common: The patch generated 1 new + 437 unchanged - 0
fixed = 438 total (was 437) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
50s{color} | {color:red} ql: The patch generated 48 new + 257 unchanged - 2
fixed = 305 total (was 259) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17694/dev-support/hive-personality.sh
|
| git revision | master / 9890919 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17694/yetus/diff-checkstyle-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17694/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql accumulo-handler hbase-handler U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17694/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Sort conditions in a filter predicate to accelerate query processing
>
>
> Key: HIVE-21857
> URL: https://issues.apache.org/jira/browse/HIVE-21857
> Project: Hive
> Issue Type: New Feature
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>
[jira] [Commented] (HIVE-21914) Move Function and Macro related DDL operations into the DDL framework
[
https://issues.apache.org/jira/browse/HIVE-21914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869958#comment-16869958
]
Hive QA commented on HIVE-21914:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972484/HIVE-21914.01.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 16335 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_func1]
(batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_genericudaf]
(batchId=90)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_genericudf]
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_udaf] (batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[drop_udf] (batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_compare_java_string]
(batchId=73)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_logic_java_boolean]
(batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_testlength2]
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_testlength]
(batchId=12)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[create_function_nonexistent_class]
(batchId=100)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[create_function_nonudf_class]
(batchId=100)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[create_unknown_genericudf]
(batchId=102)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[create_unknown_udf_udaf]
(batchId=101)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[ivyDownload]
(batchId=100)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_function_does_not_implement_udf]
(batchId=102)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_nonexistent_resource]
(batchId=101)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[udf_local_resource]
(batchId=103)
org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testDropMacro
(batchId=315)
org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testDropMacroDoesNotExist
(batchId=315)
org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testDropMacroExistsDoNotIgnoreErrors
(batchId=315)
org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testDropMacroNonExistentWithIfExists
(batchId=315)
org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testDropMacroNonExistentWithIfExistsDoNotIgnoreNonExistent
(batchId=315)
org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testOneInputParamters
(batchId=315)
org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testThreeInputParamters
(batchId=315)
org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testTwoInputParamters
(batchId=315)
org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testZeroInputParamters
(batchId=315)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17693/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17693/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17693/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 26 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972484 - PreCommit-HIVE-Build
> Move Function and Macro related DDL operations into the DDL framework
> -
>
> Key: HIVE-21914
> URL: https://issues.apache.org/jira/browse/HIVE-21914
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: refactor-ddl
> Fix For: 4.0.0
>
> Attachments: HIVE-21914.01.patch
>
>
> Some Function and Macro related operations are handled by FunctionTask, and
> FunctionWork while they belong to the DDL framework.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21857) Sort conditions in a filter predicate to accelerate query processing
[
https://issues.apache.org/jira/browse/HIVE-21857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-21857:
---
Attachment: HIVE-21857.06.patch
> Sort conditions in a filter predicate to accelerate query processing
>
>
> Key: HIVE-21857
> URL: https://issues.apache.org/jira/browse/HIVE-21857
> Project: Hive
> Issue Type: New Feature
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21857.01.patch, HIVE-21857.02.patch,
> HIVE-21857.03.patch, HIVE-21857.04.patch, HIVE-21857.05.patch,
> HIVE-21857.06.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Following approach similar to
> http://db.cs.berkeley.edu/jmh/miscpapers/sigmod93.pdf .
> To reorder predicates in AND conditions, we could rank each of elements in
> the clauses in increasing order based on following formula:
> {code}
> rank = (selectivity - 1) / cost per tuple
> {code}
> Similarly, for OR conditions:
> {code}
> rank = (-selectivity) / cost per tuple
> {code}
> Selectivity can be computed with FilterSelectivityEstimator. For cost per
> tuple, we will need to come up with some heuristic based on how expensive is
> the evaluation of the functions contained in that predicate. Custom UDFs
> could be annotated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-14737) Problem accessing /logs in a Kerberized Hive Server 2 Web UI
[ https://issues.apache.org/jira/browse/HIVE-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-14737: -- Attachment: HIVE-14737.01.patch Status: Patch Available (was: Open) > Problem accessing /logs in a Kerberized Hive Server 2 Web UI > > > Key: HIVE-14737 > URL: https://issues.apache.org/jira/browse/HIVE-14737 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Matyas Orhidi >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-14737.01.patch, HIVE-14737.patch > > > The /logs menu fails with error [1] when the cluster is Kerberized. Other > menu items are working properly. > [1] HTTP ERROR: 401 > Problem accessing /logs/. Reason: > Unauthenticated users are not authorized to access this page. > Powered by Jetty:// -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21913) GenericUDTFGetSplits should handle usernames in the same way as LLAP
[ https://issues.apache.org/jira/browse/HIVE-21913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869935#comment-16869935 ] Jason Dere commented on HIVE-21913: --- +1 > GenericUDTFGetSplits should handle usernames in the same way as LLAP > > > Key: HIVE-21913 > URL: https://issues.apache.org/jira/browse/HIVE-21913 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21913.1.patch > > > LLAP ZK registry namespacing includes current user name which is typically > hive. But in some deployments, usernames are created with '_' like hive_dev. > RegistryUtils.currentUser() (that LLAP uses) replaces '_' with '-' for DNS > reasons. But GenericUDTFSplits uses UGI login user which does not do the > underscore replacement. As a result, LlapBaseInputFormat is finding any LLAP > daemons even though they are running. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21914) Move Function and Macro related DDL operations into the DDL framework
[
https://issues.apache.org/jira/browse/HIVE-21914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869932#comment-16869932
]
Hive QA commented on HIVE-21914:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
3s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
17s{color} | {color:blue} ql in master has 2254 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
47s{color} | {color:blue} llap-server in master has 82 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
47s{color} | {color:red} ql: The patch generated 2 new + 276 unchanged - 17
fixed = 278 total (was 293) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17693/dev-support/hive-personality.sh
|
| git revision | master / 9890919 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17693/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17693/yetus/whitespace-eol.txt
|
| modules | C: ql llap-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17693/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Move Function and Macro related DDL operations into the DDL framework
> -
>
> Key: HIVE-21914
> URL: https://issues.apache.org/jira/browse/HIVE-21914
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: refactor-ddl
> Fix For: 4.0.0
>
> Attachments: HIVE-21914.01.patch
>
>
> Some Function and Macro related operations are handled by FunctionTask, and
> FunctionWork while they belong to the DDL framework.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21857) Sort conditions in a filter predicate to accelerate query processing
[
https://issues.apache.org/jira/browse/HIVE-21857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-21857:
---
Attachment: HIVE-21857.05.patch
> Sort conditions in a filter predicate to accelerate query processing
>
>
> Key: HIVE-21857
> URL: https://issues.apache.org/jira/browse/HIVE-21857
> Project: Hive
> Issue Type: New Feature
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21857.01.patch, HIVE-21857.02.patch,
> HIVE-21857.03.patch, HIVE-21857.04.patch, HIVE-21857.05.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Following approach similar to
> http://db.cs.berkeley.edu/jmh/miscpapers/sigmod93.pdf .
> To reorder predicates in AND conditions, we could rank each of elements in
> the clauses in increasing order based on following formula:
> {code}
> rank = (selectivity - 1) / cost per tuple
> {code}
> Similarly, for OR conditions:
> {code}
> rank = (-selectivity) / cost per tuple
> {code}
> Selectivity can be computed with FilterSelectivityEstimator. For cost per
> tuple, we will need to come up with some heuristic based on how expensive is
> the evaluation of the functions contained in that predicate. Custom UDFs
> could be annotated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21857) Sort conditions in a filter predicate to accelerate query processing
[
https://issues.apache.org/jira/browse/HIVE-21857?focusedWorklogId=265034=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-265034
]
ASF GitHub Bot logged work on HIVE-21857:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 22:17
Start Date: 21/Jun/19 22:17
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #671: HIVE-21857
URL: https://github.com/apache/hive/pull/671#discussion_r296410050
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdDistinctRowCount.java
##
@@ -61,17 +62,11 @@ private HiveRelMdDistinctRowCount() {
@Override
public Double getDistinctRowCount(RelNode rel, RelMetadataQuery mq,
ImmutableBitSet groupKey,
RexNode predicate) {
-if (rel instanceof HiveTableScan) {
- return getDistinctRowCount((HiveTableScan) rel, mq, groupKey, predicate);
-}
-/*
- * For now use Calcite' default formulas for propagating NDVs up the Query
- * Tree.
- */
-return super.getDistinctRowCount(rel, mq, groupKey, predicate);
+return NumberUtil.multiply(mq.getRowCount(rel),
Review comment:
The logic in the super method was:
```
// REVIEW zfong 4/19/06 - Broadbase code does not take into
// consideration selectivity of predicates passed in. Also, they
// assume the rows are unique even if the table is not
boolean uniq = RelMdUtil.areColumnsDefinitelyUnique(mq, rel, groupKey);
if (uniq) {
return NumberUtil.multiply(mq.getRowCount(rel),
mq.getSelectivity(rel, predicate));
}
return null;
```
I cannot figure out the logic behind it... if columns are definitely unique,
return distinct row count, otherwise return null?
In any case, I thought about this again and I changed the logic. I left
implementation as it was (I removed the method because it was just calling to
the super method) and I changed slightly the logic in FilterEstimator so it can
handle null values. Basically, null is unknown and you just push those
predicates in the AND / OR to the very end since you do not know how selective
they are. This may happen when we find an operator in the tree that is not
recognized yet in RelMdDistinctRowCount (e.g. DruidQuery).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 265034)
Time Spent: 1h 10m (was: 1h)
> Sort conditions in a filter predicate to accelerate query processing
>
>
> Key: HIVE-21857
> URL: https://issues.apache.org/jira/browse/HIVE-21857
> Project: Hive
> Issue Type: New Feature
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21857.01.patch, HIVE-21857.02.patch,
> HIVE-21857.03.patch, HIVE-21857.04.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Following approach similar to
> http://db.cs.berkeley.edu/jmh/miscpapers/sigmod93.pdf .
> To reorder predicates in AND conditions, we could rank each of elements in
> the clauses in increasing order based on following formula:
> {code}
> rank = (selectivity - 1) / cost per tuple
> {code}
> Similarly, for OR conditions:
> {code}
> rank = (-selectivity) / cost per tuple
> {code}
> Selectivity can be computed with FilterSelectivityEstimator. For cost per
> tuple, we will need to come up with some heuristic based on how expensive is
> the evaluation of the functions contained in that predicate. Custom UDFs
> could be annotated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21825) Improve client error msg when Active/Passive HA is enabled
[
https://issues.apache.org/jira/browse/HIVE-21825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Richard Zhang updated HIVE-21825:
-
Attachment: Hive-21825.1.patch
Status: Patch Available (was: Open)
> Improve client error msg when Active/Passive HA is enabled
> --
>
> Key: HIVE-21825
> URL: https://issues.apache.org/jira/browse/HIVE-21825
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Richard Zhang
>Priority: Major
> Labels: pull-request-available
> Attachments: Hive-21825.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When Active/Passive HA is enabled and when client tries to connect to Passive
> HA or when HS2 is still starting up, clients will receive the following the
> error msg
> {code:java}
> 'Cannot open sessions on an inactive HS2 instance; use service discovery to
> connect'{code}
> This error msg can be improved to say that HS2 is still starting up (or more
> user-friendly error msg).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21825) Improve client error msg when Active/Passive HA is enabled
[
https://issues.apache.org/jira/browse/HIVE-21825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-21825:
--
Labels: pull-request-available (was: )
> Improve client error msg when Active/Passive HA is enabled
> --
>
> Key: HIVE-21825
> URL: https://issues.apache.org/jira/browse/HIVE-21825
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Richard Zhang
>Priority: Major
> Labels: pull-request-available
>
> When Active/Passive HA is enabled and when client tries to connect to Passive
> HA or when HS2 is still starting up, clients will receive the following the
> error msg
> {code:java}
> 'Cannot open sessions on an inactive HS2 instance; use service discovery to
> connect'{code}
> This error msg can be improved to say that HS2 is still starting up (or more
> user-friendly error msg).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21825) Improve client error msg when Active/Passive HA is enabled
[
https://issues.apache.org/jira/browse/HIVE-21825?focusedWorklogId=265026=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-265026
]
ASF GitHub Bot logged work on HIVE-21825:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 22:04
Start Date: 21/Jun/19 22:04
Worklog Time Spent: 10m
Work Description: rizhangcloud commented on pull request #682:
HIVE-21825: Improve client error msg when Active/Passive HA is enable…
URL: https://github.com/apache/hive/pull/682
When Active/Passive HA is enabled and when client tries to connect to
Passive HA or when HS2 is still starting up, clients will receive the following
the error msg
'Cannot open sessions on an inactive HS2 instance; use service discovery to
connect'
This error msg can be improved to say that HS2 is still starting up (or more
user-friendly error msg).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 265026)
Time Spent: 10m
Remaining Estimate: 0h
> Improve client error msg when Active/Passive HA is enabled
> --
>
> Key: HIVE-21825
> URL: https://issues.apache.org/jira/browse/HIVE-21825
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Richard Zhang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When Active/Passive HA is enabled and when client tries to connect to Passive
> HA or when HS2 is still starting up, clients will receive the following the
> error msg
> {code:java}
> 'Cannot open sessions on an inactive HS2 instance; use service discovery to
> connect'{code}
> This error msg can be improved to say that HS2 is still starting up (or more
> user-friendly error msg).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21914) Move Function and Macro related DDL operations into the DDL framework
[ https://issues.apache.org/jira/browse/HIVE-21914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21914: -- Component/s: Hive > Move Function and Macro related DDL operations into the DDL framework > - > > Key: HIVE-21914 > URL: https://issues.apache.org/jira/browse/HIVE-21914 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21914.01.patch > > > Some Function and Macro related operations are handled by FunctionTask, and > FunctionWork while they belong to the DDL framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21914) Move Function and Macro related DDL operations into the DDL framework
[ https://issues.apache.org/jira/browse/HIVE-21914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21914: -- Fix Version/s: 4.0.0 > Move Function and Macro related DDL operations into the DDL framework > - > > Key: HIVE-21914 > URL: https://issues.apache.org/jira/browse/HIVE-21914 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21914.01.patch > > > Some Function and Macro related operations are handled by FunctionTask, and > FunctionWork while they belong to the DDL framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21914) Move Function and Macro related DDL operations into the DDL framework
[ https://issues.apache.org/jira/browse/HIVE-21914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21914: -- Affects Version/s: 3.1.1 > Move Function and Macro related DDL operations into the DDL framework > - > > Key: HIVE-21914 > URL: https://issues.apache.org/jira/browse/HIVE-21914 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Attachments: HIVE-21914.01.patch > > > Some Function and Macro related operations are handled by FunctionTask, and > FunctionWork while they belong to the DDL framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21914) Move Function and Macro related DDL operations into the DDL framework
[ https://issues.apache.org/jira/browse/HIVE-21914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21914: -- Status: Patch Available (was: Open) > Move Function and Macro related DDL operations into the DDL framework > - > > Key: HIVE-21914 > URL: https://issues.apache.org/jira/browse/HIVE-21914 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Attachments: HIVE-21914.01.patch > > > Some Function and Macro related operations are handled by FunctionTask, and > FunctionWork while they belong to the DDL framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21914) Move Function and Macro related DDL operations into the DDL framework
[ https://issues.apache.org/jira/browse/HIVE-21914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21914: -- Attachment: HIVE-21914.01.patch > Move Function and Macro related DDL operations into the DDL framework > - > > Key: HIVE-21914 > URL: https://issues.apache.org/jira/browse/HIVE-21914 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Attachments: HIVE-21914.01.patch > > > Some Function and Macro related operations are handled by FunctionTask, and > FunctionWork while they belong to the DDL framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21914) Move Function and Macro related DDL operations into the DDL framework
[ https://issues.apache.org/jira/browse/HIVE-21914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21914: -- Labels: refactor-ddl (was: ) > Move Function and Macro related DDL operations into the DDL framework > - > > Key: HIVE-21914 > URL: https://issues.apache.org/jira/browse/HIVE-21914 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > > Some Function and Macro related operations are handled by FunctionTask, and > FunctionWork while they belong to the DDL framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21914) Move Function and Macro related DDL operations into the DDL framework
[ https://issues.apache.org/jira/browse/HIVE-21914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely reassigned HIVE-21914: - > Move Function and Macro related DDL operations into the DDL framework > - > > Key: HIVE-21914 > URL: https://issues.apache.org/jira/browse/HIVE-21914 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > > Some Function and Macro related operations are handled by FunctionTask, and > FunctionWork while they belong to the DDL framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21913) GenericUDTFGetSplits should handle usernames in the same way as LLAP
[
https://issues.apache.org/jira/browse/HIVE-21913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869906#comment-16869906
]
Hive QA commented on HIVE-21913:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972467/HIVE-21913.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16335 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17692/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17692/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17692/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972467 - PreCommit-HIVE-Build
> GenericUDTFGetSplits should handle usernames in the same way as LLAP
>
>
> Key: HIVE-21913
> URL: https://issues.apache.org/jira/browse/HIVE-21913
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21913.1.patch
>
>
> LLAP ZK registry namespacing includes current user name which is typically
> hive. But in some deployments, usernames are created with '_' like hive_dev.
> RegistryUtils.currentUser() (that LLAP uses) replaces '_' with '-' for DNS
> reasons. But GenericUDTFSplits uses UGI login user which does not do the
> underscore replacement. As a result, LlapBaseInputFormat is finding any LLAP
> daemons even though they are running.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21857) Sort conditions in a filter predicate to accelerate query processing
[
https://issues.apache.org/jira/browse/HIVE-21857?focusedWorklogId=265009=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-265009
]
ASF GitHub Bot logged work on HIVE-21857:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 21:43
Start Date: 21/Jun/19 21:43
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #671: HIVE-21857
URL: https://github.com/apache/hive/pull/671#discussion_r296403208
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterSortPredicates.java
##
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.optimizer.calcite.rules;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexDynamicParam;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexLiteral;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexVisitorImpl;
+import org.apache.calcite.util.Pair;
+import
org.apache.hadoop.hive.ql.optimizer.calcite.stats.FilterSelectivityEstimator;
+import org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdSize;
+
+
+/**
+ * Rule that sorts conditions in a filter predicate to accelerate query
processing
+ * based on selectivity and compute cost. Currently it is not applied
recursively,
+ * i.e., it is only applied to top predicates in the condition.
+ */
+public class HiveFilterSortPredicates extends RelOptRule {
+
+ public static final HiveFilterSortPredicates INSTANCE = new
HiveFilterSortPredicates();
+
+
+ private HiveFilterSortPredicates() {
+super(
+operand(Filter.class,
+operand(RelNode.class, any(;
+ }
+
+ @Override
+ public boolean matches(RelOptRuleCall call) {
+final Filter filter = call.rel(0);
+
+HiveRulesRegistry registry =
call.getPlanner().getContext().unwrap(HiveRulesRegistry.class);
+
+// If this operator has been visited already by the rule,
+// we do not need to apply the optimization
+if (registry != null && registry.getVisited(this).contains(filter)) {
+ return false;
+}
+
+return true;
+ }
+
+ @Override
+ public void onMatch(RelOptRuleCall call) {
+final Filter filter = call.rel(0);
+final RelNode input = call.rel(1);
+
+// Register that we have visited this operator in this rule
+HiveRulesRegistry registry =
call.getPlanner().getContext().unwrap(HiveRulesRegistry.class);
+if (registry != null) {
+ registry.registerVisited(this, filter);
+}
+
+final RexNode originalCond = filter.getCondition();
+RexSortPredicatesShuttle sortPredicatesShuttle = new
RexSortPredicatesShuttle(
+input, filter.getCluster().getMetadataQuery());
+final RexNode newCond = originalCond.accept(sortPredicatesShuttle);
+if (!sortPredicatesShuttle.modified) {
+ // We are done, bail out
+ return;
+}
+
+// We register the new filter so we do not fire the rule on it again
+final Filter newFilter = filter.copy(filter.getTraitSet(), input, newCond);
+if (registry != null) {
+ registry.registerVisited(this, newFilter);
+}
+
+call.transformTo(newFilter);
+ }
+
+ /**
+ * If the expression is an AND/OR, it will sort predicates accordingly
+ * to maximize performance.
+ * In particular, for AND clause:
+ * rank = (selectivity - 1) / cost per tuple
+ * Similarly, for OR clause:
+ * rank = (-selectivity) / cost per tuple
+ */
+ private static class RexSortPredicatesShuttle extends RexShuttle {
+
+private FilterSelectivityEstimator selectivityEstimator;
+private boolean modified;
+
+private RexSortPredicatesShuttle(RelNode inputRel,
[jira] [Work logged] (HIVE-21857) Sort conditions in a filter predicate to accelerate query processing
[
https://issues.apache.org/jira/browse/HIVE-21857?focusedWorklogId=265006=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-265006
]
ASF GitHub Bot logged work on HIVE-21857:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 21:37
Start Date: 21/Jun/19 21:37
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #671: HIVE-21857
URL: https://github.com/apache/hive/pull/671#discussion_r296402075
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterSortPredicates.java
##
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.optimizer.calcite.rules;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexDynamicParam;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexLiteral;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexVisitorImpl;
+import org.apache.calcite.util.Pair;
+import
org.apache.hadoop.hive.ql.optimizer.calcite.stats.FilterSelectivityEstimator;
+import org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdSize;
+
+
+/**
+ * Rule that sorts conditions in a filter predicate to accelerate query
processing
+ * based on selectivity and compute cost. Currently it is not applied
recursively,
+ * i.e., it is only applied to top predicates in the condition.
+ */
+public class HiveFilterSortPredicates extends RelOptRule {
+
+ public static final HiveFilterSortPredicates INSTANCE = new
HiveFilterSortPredicates();
+
+
+ private HiveFilterSortPredicates() {
+super(
+operand(Filter.class,
+operand(RelNode.class, any(;
+ }
+
+ @Override
+ public boolean matches(RelOptRuleCall call) {
+final Filter filter = call.rel(0);
+
+HiveRulesRegistry registry =
call.getPlanner().getContext().unwrap(HiveRulesRegistry.class);
+
+// If this operator has been visited already by the rule,
+// we do not need to apply the optimization
+if (registry != null && registry.getVisited(this).contains(filter)) {
+ return false;
+}
+
+return true;
+ }
+
+ @Override
+ public void onMatch(RelOptRuleCall call) {
+final Filter filter = call.rel(0);
+final RelNode input = call.rel(1);
+
+// Register that we have visited this operator in this rule
+HiveRulesRegistry registry =
call.getPlanner().getContext().unwrap(HiveRulesRegistry.class);
+if (registry != null) {
+ registry.registerVisited(this, filter);
+}
+
+final RexNode originalCond = filter.getCondition();
+RexSortPredicatesShuttle sortPredicatesShuttle = new
RexSortPredicatesShuttle(
+input, filter.getCluster().getMetadataQuery());
+final RexNode newCond = originalCond.accept(sortPredicatesShuttle);
+if (!sortPredicatesShuttle.modified) {
+ // We are done, bail out
+ return;
+}
+
+// We register the new filter so we do not fire the rule on it again
+final Filter newFilter = filter.copy(filter.getTraitSet(), input, newCond);
+if (registry != null) {
+ registry.registerVisited(this, newFilter);
+}
+
+call.transformTo(newFilter);
+ }
+
+ /**
+ * If the expression is an AND/OR, it will sort predicates accordingly
+ * to maximize performance.
+ * In particular, for AND clause:
+ * rank = (selectivity - 1) / cost per tuple
+ * Similarly, for OR clause:
Review comment:
Added comments to `rankingAnd` and `rankingOr` methods.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to
[jira] [Assigned] (HIVE-21825) Improve client error msg when Active/Passive HA is enabled
[
https://issues.apache.org/jira/browse/HIVE-21825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Richard Zhang reassigned HIVE-21825:
Assignee: Richard Zhang
> Improve client error msg when Active/Passive HA is enabled
> --
>
> Key: HIVE-21825
> URL: https://issues.apache.org/jira/browse/HIVE-21825
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Richard Zhang
>Priority: Major
>
> When Active/Passive HA is enabled and when client tries to connect to Passive
> HA or when HS2 is still starting up, clients will receive the following the
> error msg
> {code:java}
> 'Cannot open sessions on an inactive HS2 instance; use service discovery to
> connect'{code}
> This error msg can be improved to say that HS2 is still starting up (or more
> user-friendly error msg).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HIVE-18842) CLUSTER BY/DISTRIBUTE BY/SORT BY support for materialized views
[
https://issues.apache.org/jira/browse/HIVE-18842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-18842 started by Jesus Camacho Rodriguez.
--
> CLUSTER BY/DISTRIBUTE BY/SORT BY support for materialized views
> ---
>
> Key: HIVE-18842
> URL: https://issues.apache.org/jira/browse/HIVE-18842
> Project: Hive
> Issue Type: New Feature
> Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> We should support defining a {{CLUSTER BY/DISTRIBUTE BY/SORT BY}}
> specification for materialized views.
> The syntax should be extended as follows:
> {code:sql}
> CREATE MATERIALIZED VIEW [IF NOT EXISTS] [db_name.]materialized_view_name
> [COMMENT materialized_view_comment]
> [CLUSTER BY (col_name, ...) | ( [DISTRIBUTE BY (col_name, ...)] [SORT BY
> (col_name, ...)] ) ] -- NEW!
> [
>[ROW FORMAT row_format]
>[STORED AS file_format]
> | STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
> ]
> [LOCATION hdfs_path]
> [TBLPROPERTIES (property_name=property_value, ...)]
> AS select_statement;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21391) LLAP: Pool of column vector buffers can cause memory pressure
[ https://issues.apache.org/jira/browse/HIVE-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869875#comment-16869875 ] Prasanth Jayachandran commented on HIVE-21391: -- +1'ed in github PR. > LLAP: Pool of column vector buffers can cause memory pressure > - > > Key: HIVE-21391 > URL: https://issues.apache.org/jira/browse/HIVE-21391 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: slim bouguerra >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21391.1.patch > > Time Spent: 20m > Remaining Estimate: 0h > > Where there are too many columns (in the order of 100s), with decimal, string > types the column vector pool of buffers created here > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/EncodedDataConsumer.java#L59] > can cause memory pressure. > Example: > 128 (poolSize) * 300 (numCols) * 1024 (batchSize) * 80 (decimalSize) ~= 3GB > The pool size keeps increasing when there is slow consumer but fast llap io > (SSDs) leading to GC pressure when all LLAP io threads read splits from same > table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21391) LLAP: Pool of column vector buffers can cause memory pressure
[ https://issues.apache.org/jira/browse/HIVE-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-21391: Assignee: slim bouguerra (was: Prasanth Jayachandran) > LLAP: Pool of column vector buffers can cause memory pressure > - > > Key: HIVE-21391 > URL: https://issues.apache.org/jira/browse/HIVE-21391 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: slim bouguerra >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21391.1.patch > > Time Spent: 20m > Remaining Estimate: 0h > > Where there are too many columns (in the order of 100s), with decimal, string > types the column vector pool of buffers created here > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/EncodedDataConsumer.java#L59] > can cause memory pressure. > Example: > 128 (poolSize) * 300 (numCols) * 1024 (batchSize) * 80 (decimalSize) ~= 3GB > The pool size keeps increasing when there is slow consumer but fast llap io > (SSDs) leading to GC pressure when all LLAP io threads read splits from same > table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21391) LLAP: Pool of column vector buffers can cause memory pressure
[ https://issues.apache.org/jira/browse/HIVE-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869874#comment-16869874 ] Prasanth Jayachandran commented on HIVE-21391: -- [~bslim] could you upload patch file for precommit tests to run? > LLAP: Pool of column vector buffers can cause memory pressure > - > > Key: HIVE-21391 > URL: https://issues.apache.org/jira/browse/HIVE-21391 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: slim bouguerra >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21391.1.patch > > Time Spent: 20m > Remaining Estimate: 0h > > Where there are too many columns (in the order of 100s), with decimal, string > types the column vector pool of buffers created here > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/EncodedDataConsumer.java#L59] > can cause memory pressure. > Example: > 128 (poolSize) * 300 (numCols) * 1024 (batchSize) * 80 (decimalSize) ~= 3GB > The pool size keeps increasing when there is slow consumer but fast llap io > (SSDs) leading to GC pressure when all LLAP io threads read splits from same > table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21913) GenericUDTFGetSplits should handle usernames in the same way as LLAP
[
https://issues.apache.org/jira/browse/HIVE-21913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869868#comment-16869868
]
Hive QA commented on HIVE-21913:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 2254 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17692/dev-support/hive-personality.sh
|
| git revision | master / 9890919 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17692/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> GenericUDTFGetSplits should handle usernames in the same way as LLAP
>
>
> Key: HIVE-21913
> URL: https://issues.apache.org/jira/browse/HIVE-21913
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21913.1.patch
>
>
> LLAP ZK registry namespacing includes current user name which is typically
> hive. But in some deployments, usernames are created with '_' like hive_dev.
> RegistryUtils.currentUser() (that LLAP uses) replaces '_' with '-' for DNS
> reasons. But GenericUDTFSplits uses UGI login user which does not do the
> underscore replacement. As a result, LlapBaseInputFormat is finding any LLAP
> daemons even though they are running.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869853#comment-16869853
]
Hive QA commented on HIVE-21764:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972455/HIVE-21764.03.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16339 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17691/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17691/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17691/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972455 - PreCommit-HIVE-Build
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch,
> HIVE-21764.03.patch
>
> Time Spent: 7h 10m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869834#comment-16869834
]
Hive QA commented on HIVE-21764:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
58s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
13s{color} | {color:blue} standalone-metastore/metastore-server in master has
179 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
8s{color} | {color:blue} ql in master has 2254 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
47s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17691/dev-support/hive-personality.sh
|
| git revision | master / 9890919 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17691/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17691/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch,
> HIVE-21764.03.patch
>
> Time Spent: 7h 10m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the
[jira] [Commented] (HIVE-21896) SHOW FUNCTIONS / SHOW FUNCTIONS LIKE - clarify
[
https://issues.apache.org/jira/browse/HIVE-21896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869804#comment-16869804
]
Hive QA commented on HIVE-21896:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972452/HIVE-21896.02.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16336 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17690/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17690/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17690/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972452 - PreCommit-HIVE-Build
> SHOW FUNCTIONS / SHOW FUNCTIONS LIKE - clarify
> --
>
> Key: HIVE-21896
> URL: https://issues.apache.org/jira/browse/HIVE-21896
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: incompatibleChange, todoc4.0
> Fix For: 4.0.0
>
> Attachments: HIVE-21896.01.patch, HIVE-21896.02.patch
>
>
> According to
> [https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowFunctions]
> the currently available functions can be listed like this:
> {code:java}
> SHOW FUNCTIONS ;{code}
> If the user executes this command, they will get the correct list of
> functions, but they will also see this on the standard output:
> {code:java}
> SHOW FUNCTIONS is deprecated, please use SHOW FUNCTIONS LIKE instead.{code}
> If the user uses the
> {code:java}
> SHOW FUNCTIONS LIKE ;{code}
> command then they will receive the exact same result (though through
> different codes). The only difference is that one can get all the function
> names with "SHOW FUNCTIONS;", while "SHOW FUNCTIONS LIKE;" returns an
> exception, so in this case the pattern is mandatory.
> So there should be a decision if we still accept "SHOW FUNCTIONS" without the
> "LIKE". My suggestion is to accept it only if there is no pattern. so "SHOW
> FUNCTIONS;" is ok, without deprecation message, but "SHOW FUNCTIONS
> " should throw an exception.
> Whatever we decide, we should document it appropriately.
> cc [~krishahn]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21913) GenericUDTFGetSplits should handle usernames in the same way as LLAP
[ https://issues.apache.org/jira/browse/HIVE-21913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869755#comment-16869755 ] Prasanth Jayachandran commented on HIVE-21913: -- [~jdere] can you please review this small patch? > GenericUDTFGetSplits should handle usernames in the same way as LLAP > > > Key: HIVE-21913 > URL: https://issues.apache.org/jira/browse/HIVE-21913 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21913.1.patch > > > LLAP ZK registry namespacing includes current user name which is typically > hive. But in some deployments, usernames are created with '_' like hive_dev. > RegistryUtils.currentUser() (that LLAP uses) replaces '_' with '-' for DNS > reasons. But GenericUDTFSplits uses UGI login user which does not do the > underscore replacement. As a result, LlapBaseInputFormat is finding any LLAP > daemons even though they are running. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21913) GenericUDTFGetSplits should handle usernames in the same way as LLAP
[ https://issues.apache.org/jira/browse/HIVE-21913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21913: - Status: Patch Available (was: Open) > GenericUDTFGetSplits should handle usernames in the same way as LLAP > > > Key: HIVE-21913 > URL: https://issues.apache.org/jira/browse/HIVE-21913 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21913.1.patch > > > LLAP ZK registry namespacing includes current user name which is typically > hive. But in some deployments, usernames are created with '_' like hive_dev. > RegistryUtils.currentUser() (that LLAP uses) replaces '_' with '-' for DNS > reasons. But GenericUDTFSplits uses UGI login user which does not do the > underscore replacement. As a result, LlapBaseInputFormat is finding any LLAP > daemons even though they are running. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21913) GenericUDTFGetSplits should handle usernames in the same way as LLAP
[ https://issues.apache.org/jira/browse/HIVE-21913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21913: - Attachment: HIVE-21913.1.patch > GenericUDTFGetSplits should handle usernames in the same way as LLAP > > > Key: HIVE-21913 > URL: https://issues.apache.org/jira/browse/HIVE-21913 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21913.1.patch > > > LLAP ZK registry namespacing includes current user name which is typically > hive. But in some deployments, usernames are created with '_' like hive_dev. > RegistryUtils.currentUser() (that LLAP uses) replaces '_' with '-' for DNS > reasons. But GenericUDTFSplits uses UGI login user which does not do the > underscore replacement. As a result, LlapBaseInputFormat is finding any LLAP > daemons even though they are running. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21913) GenericUDTFGetSplits should handle usernames in the same way as LLAP
[ https://issues.apache.org/jira/browse/HIVE-21913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-21913: > GenericUDTFGetSplits should handle usernames in the same way as LLAP > > > Key: HIVE-21913 > URL: https://issues.apache.org/jira/browse/HIVE-21913 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > LLAP ZK registry namespacing includes current user name which is typically > hive. But in some deployments, usernames are created with '_' like hive_dev. > RegistryUtils.currentUser() (that LLAP uses) replaces '_' with '-' for DNS > reasons. But GenericUDTFSplits uses UGI login user which does not do the > underscore replacement. As a result, LlapBaseInputFormat is finding any LLAP > daemons even though they are running. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21896) SHOW FUNCTIONS / SHOW FUNCTIONS LIKE - clarify
[
https://issues.apache.org/jira/browse/HIVE-21896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869753#comment-16869753
]
Hive QA commented on HIVE-21896:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
56s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
12s{color} | {color:blue} ql in master has 2254 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} ql: The patch generated 0 new + 292 unchanged - 1
fixed = 292 total (was 293) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17690/dev-support/hive-personality.sh
|
| git revision | master / 9890919 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17690/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> SHOW FUNCTIONS / SHOW FUNCTIONS LIKE - clarify
> --
>
> Key: HIVE-21896
> URL: https://issues.apache.org/jira/browse/HIVE-21896
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: incompatibleChange, todoc4.0
> Fix For: 4.0.0
>
> Attachments: HIVE-21896.01.patch, HIVE-21896.02.patch
>
>
> According to
> [https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowFunctions]
> the currently available functions can be listed like this:
> {code:java}
> SHOW FUNCTIONS ;{code}
> If the user executes this command, they will get the correct list of
> functions, but they will also see this on the standard output:
> {code:java}
> SHOW FUNCTIONS is deprecated, please use SHOW FUNCTIONS LIKE instead.{code}
> If
[jira] [Commented] (HIVE-21905) Generics improvement around the FetchOperator class
[
https://issues.apache.org/jira/browse/HIVE-21905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869710#comment-16869710
]
Hive QA commented on HIVE-21905:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972446/HIVE-21905.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16335 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17689/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17689/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17689/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972446 - PreCommit-HIVE-Build
> Generics improvement around the FetchOperator class
> ---
>
> Key: HIVE-21905
> URL: https://issues.apache.org/jira/browse/HIVE-21905
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Ivan Suller
>Assignee: Ivan Suller
>Priority: Minor
> Attachments: HIVE-21905.1.patch
>
>
> In and around the org.apache.hadoop.hive.ql.exec.FetchOperator class the
> generics are handled poorly. Lot's of declarations are missing generics,
> which makes lots of noise in the IDE and makes it hard to be sure of the
> correctness of the code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21902) HiveServer2 UI: jetty response header needs X-Frame-Options
[
https://issues.apache.org/jira/browse/HIVE-21902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated HIVE-21902:
-
Summary: HiveServer2 UI: jetty response header needs X-Frame-Options (was:
HiveServer2 UI: Security Vulnerability in jetty response header)
> HiveServer2 UI: jetty response header needs X-Frame-Options
> ---
>
> Key: HIVE-21902
> URL: https://issues.apache.org/jira/browse/HIVE-21902
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Labels: security
> Attachments: HIVE-21902.01.patch, HIVE-21902.patch
>
>
> there are some vulnerability are reported for hiveserver2 ui
> X-Frame-Options or Content-Security-Policy: frame-ancestors HTTP Headers
> missing on port 10002.
> {code}
> GET / HTTP/1.1
> Host: HOSTNAME:10002
> Connection: Keep-Alive
> X-XSS-Protection HTTP Header missing on port 10002.
> X-Content-Type-Options HTTP Header missing on port 10002.
> {code}
> after the proposed changes
> {code}
> HTTP/1.1 200 OK
> Date: Thu, 20 Jun 2019 05:29:59 GMT
> Content-Type: text/html;charset=utf-8
> X-Content-Type-Options: nosniff
> X-FRAME-OPTIONS: SAMEORIGIN
> X-XSS-Protection: 1; mode=block
> Set-Cookie: JSESSIONID=15kscuow9cmy7qms6dzaxllqt;Path=/
> Expires: Thu, 01 Jan 1970 00:00:00 GMT
> Content-Length: 3824
> Server: Jetty(9.3.25.v20180904)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21905) Generics improvement around the FetchOperator class
[
https://issues.apache.org/jira/browse/HIVE-21905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869651#comment-16869651
]
Hive QA commented on HIVE-21905:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
15s{color} | {color:blue} ql in master has 2254 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 1 new + 152 unchanged - 1
fixed = 153 total (was 153) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
19s{color} | {color:green} ql generated 0 new + 2253 unchanged - 1 fixed = 2253
total (was 2254) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17689/dev-support/hive-personality.sh
|
| git revision | master / 9890919 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17689/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17689/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Generics improvement around the FetchOperator class
> ---
>
> Key: HIVE-21905
> URL: https://issues.apache.org/jira/browse/HIVE-21905
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Ivan Suller
>Assignee: Ivan Suller
>Priority: Minor
> Attachments: HIVE-21905.1.patch
>
>
> In and around the org.apache.hadoop.hive.ql.exec.FetchOperator class the
> generics are handled poorly. Lot's of declarations are missing generics,
> which makes lots of noise in the IDE and makes it hard to be sure of the
> correctness of the code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21906) Blacklist limping nodes
[ https://issues.apache.org/jira/browse/HIVE-21906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary reassigned HIVE-21906: - > Blacklist limping nodes > --- > > Key: HIVE-21906 > URL: https://issues.apache.org/jira/browse/HIVE-21906 > Project: Hive > Issue Type: New Feature > Components: llap >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > > If a specific LLAP node is limping it can degrade the performance of the > whole cluster. > We should find a way to identify the limping nodes and act on them either by > disabling the node altogether, or decreasing the load on this node -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[ https://issues.apache.org/jira/browse/HIVE-21764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21764: --- Attachment: HIVE-21764.03.patch > REPL DUMP should detect and bootstrap any rename table events where old table > was excluded but renamed table is included. > - > > Key: HIVE-21764 > URL: https://issues.apache.org/jira/browse/HIVE-21764 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Sankar Hariappan >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch, > HIVE-21764.03.patch > > Time Spent: 7h 10m > Remaining Estimate: 0h > > REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular > expression + inclusion/exclusion list. So, in case of rename table event, the > event will be ignored if old table doesn't match the pattern but the new > table should be bootstrapped. REPL DUMP should have a mechanism to detect > such tables and automatically bootstrap with incremental replication.Also, if > renamed table is excluded from replication policy, then need to drop the old > table at target as well. > There are 4 scenarios that needs to be handled. > # Both new name and old name satisfies the table name pattern filter. > ## No need to do anything. The incremental event for rename should take care > of the replication. > # Both the names does not satisfy the table name pattern filter. > ## Both the names are not in the scope of the policy and thus nothing needs > to be done. > # New name satisfies the pattern but the old name does not. > ## The table will not be present at the target. > ## Rename event handler for dump should detect this case and add the new > table name to the list of table for bootstrap. > ## All the events related to the table (new name) should be ignored. > ## If there is a drop event for the table (with new name), then remove the > table from the list of tables to be bootstrapped. > ## In case of rename (double rename) > ### If the new name satisfies the table pattern, then add the new name to > the list of tables to be bootstrapped and remove the old name from the list > of tables to be bootstrapped. > ### If the new name does not satisfies then just removed the table name from > the list of tables to be bootstrapped. > # New name does not satisfies the pattern but the old name satisfies. > ## Change the rename event to a drop event. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[ https://issues.apache.org/jira/browse/HIVE-21764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21764: --- Status: Patch Available (was: Open) > REPL DUMP should detect and bootstrap any rename table events where old table > was excluded but renamed table is included. > - > > Key: HIVE-21764 > URL: https://issues.apache.org/jira/browse/HIVE-21764 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Sankar Hariappan >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch, > HIVE-21764.03.patch > > Time Spent: 7h 10m > Remaining Estimate: 0h > > REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular > expression + inclusion/exclusion list. So, in case of rename table event, the > event will be ignored if old table doesn't match the pattern but the new > table should be bootstrapped. REPL DUMP should have a mechanism to detect > such tables and automatically bootstrap with incremental replication.Also, if > renamed table is excluded from replication policy, then need to drop the old > table at target as well. > There are 4 scenarios that needs to be handled. > # Both new name and old name satisfies the table name pattern filter. > ## No need to do anything. The incremental event for rename should take care > of the replication. > # Both the names does not satisfy the table name pattern filter. > ## Both the names are not in the scope of the policy and thus nothing needs > to be done. > # New name satisfies the pattern but the old name does not. > ## The table will not be present at the target. > ## Rename event handler for dump should detect this case and add the new > table name to the list of table for bootstrap. > ## All the events related to the table (new name) should be ignored. > ## If there is a drop event for the table (with new name), then remove the > table from the list of tables to be bootstrapped. > ## In case of rename (double rename) > ### If the new name satisfies the table pattern, then add the new name to > the list of tables to be bootstrapped and remove the old name from the list > of tables to be bootstrapped. > ### If the new name does not satisfies then just removed the table name from > the list of tables to be bootstrapped. > # New name does not satisfies the pattern but the old name satisfies. > ## Change the rename event to a drop event. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21874) Implement add partitions related methods on temporary table
[
https://issues.apache.org/jira/browse/HIVE-21874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869621#comment-16869621
]
Hive QA commented on HIVE-21874:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972437/HIVE-21874.01.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16593 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestPartitionManagement.testPartitionDiscoveryEnabledBothTableTypes
(batchId=222)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17688/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17688/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17688/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972437 - PreCommit-HIVE-Build
> Implement add partitions related methods on temporary table
> ---
>
> Key: HIVE-21874
> URL: https://issues.apache.org/jira/browse/HIVE-21874
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-21874.01.patch
>
>
> IMetaStoreClient exposes the following add partition related methods:
> {code:java}
> Partition add_partition(Partition partition);
> int add_partitions(List partitions);
> int add_partitions_pspec(PartitionSpecProxy partitionSpec);
> List add_partitions(List partitions, boolean
> ifNotExists, boolean needResults);
> {code}
> These methods should be implemented in order to handle addition of partitions
> to temporary tables.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21896) SHOW FUNCTIONS / SHOW FUNCTIONS LIKE - clarify
[
https://issues.apache.org/jira/browse/HIVE-21896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-21896:
--
Attachment: HIVE-21896.02.patch
> SHOW FUNCTIONS / SHOW FUNCTIONS LIKE - clarify
> --
>
> Key: HIVE-21896
> URL: https://issues.apache.org/jira/browse/HIVE-21896
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: incompatibleChange, todoc4.0
> Fix For: 4.0.0
>
> Attachments: HIVE-21896.01.patch, HIVE-21896.02.patch
>
>
> According to
> [https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowFunctions]
> the currently available functions can be listed like this:
> {code:java}
> SHOW FUNCTIONS ;{code}
> If the user executes this command, they will get the correct list of
> functions, but they will also see this on the standard output:
> {code:java}
> SHOW FUNCTIONS is deprecated, please use SHOW FUNCTIONS LIKE instead.{code}
> If the user uses the
> {code:java}
> SHOW FUNCTIONS LIKE ;{code}
> command then they will receive the exact same result (though through
> different codes). The only difference is that one can get all the function
> names with "SHOW FUNCTIONS;", while "SHOW FUNCTIONS LIKE;" returns an
> exception, so in this case the pattern is mandatory.
> So there should be a decision if we still accept "SHOW FUNCTIONS" without the
> "LIKE". My suggestion is to accept it only if there is no pattern. so "SHOW
> FUNCTIONS;" is ok, without deprecation message, but "SHOW FUNCTIONS
> " should throw an exception.
> Whatever we decide, we should document it appropriately.
> cc [~krishahn]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21874) Implement add partitions related methods on temporary table
[
https://issues.apache.org/jira/browse/HIVE-21874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869570#comment-16869570
]
Hive QA commented on HIVE-21874:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
1s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
12s{color} | {color:blue} standalone-metastore/metastore-server in master has
179 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
9s{color} | {color:blue} ql in master has 2254 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 7 new + 15 unchanged - 0 fixed
= 22 total (was 15) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17688/dev-support/hive-personality.sh
|
| git revision | master / 9890919 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17688/yetus/diff-checkstyle-ql.txt
|
| modules | C: standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17688/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Implement add partitions related methods on temporary table
> ---
>
> Key: HIVE-21874
> URL: https://issues.apache.org/jira/browse/HIVE-21874
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-21874.01.patch
>
>
> IMetaStoreClient exposes the following add partition related methods:
> {code:java}
> Partition add_partition(Partition partition);
> int add_partitions(List partitions);
> int add_partitions_pspec(PartitionSpecProxy partitionSpec);
> List add_partitions(List partitions, boolean
> ifNotExists, boolean needResults);
> {code}
> These methods should be implemented in order to handle addition of partitions
> to temporary tables.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264690=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264690
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 14:47
Start Date: 21/Jun/19 14:47
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296268041
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/event/filters/ReplEventFilter.java
##
@@ -30,12 +31,22 @@ public ReplEventFilter(final ReplScope replScope) {
this.replScope = replScope;
}
+ // Return false if all the tables are included, as bootstrap during rename
is done only in case filter set for tables.
+ boolean isAlterDuringTableLevelReplication(final NotificationEvent event) {
Review comment:
This is not alter table handler. It is ReplEventFilter.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264690)
Time Spent: 7h 10m (was: 7h)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 7h 10m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264688=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264688
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 14:43
Start Date: 21/Jun/19 14:43
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296266307
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/event/filters/ReplEventFilter.java
##
@@ -33,7 +33,7 @@ public ReplEventFilter(final ReplScope replScope) {
// Return false if all the tables are included, as bootstrap during rename
is done only in case filter set for tables.
boolean isAlterDuringTableLevelReplication(final NotificationEvent event) {
-if (replScope.includeAllTables() ||
!replScope.getDbNamePattern().matcher(event.getDbName()).matches()) {
+if (!replScope.dbIncludedInReplScope(event.getDbName())) {
return false;
}
return event.getEventType().equals(MessageBuilder.ALTER_TABLE_EVENT);
Review comment:
Can combine to one line as replScope.dbIncludedInReplScope() &&
event.getEventType().equals().
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264688)
Time Spent: 7h (was: 6h 50m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 7h
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21896) SHOW FUNCTIONS / SHOW FUNCTIONS LIKE - clarify
[
https://issues.apache.org/jira/browse/HIVE-21896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869531#comment-16869531
]
Hive QA commented on HIVE-21896:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972436/HIVE-21896.01.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 16336 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.parse.TestReplAcrossInstancesWithJsonMessageFormat.testBootstrapFunctionReplication
(batchId=255)
org.apache.hadoop.hive.ql.parse.TestReplAcrossInstancesWithJsonMessageFormat.testBootstrapReplLoadRetryAfterFailureForFunctions
(batchId=255)
org.apache.hadoop.hive.ql.parse.TestReplAcrossInstancesWithJsonMessageFormat.testBootstrapReplLoadRetryAfterFailureForPartitions
(batchId=255)
org.apache.hadoop.hive.ql.parse.TestReplAcrossInstancesWithJsonMessageFormat.testCreateFunctionIncrementalReplication
(batchId=255)
org.apache.hadoop.hive.ql.parse.TestReplAcrossInstancesWithJsonMessageFormat.testCreateFunctionWithFunctionBinaryJarsOnHDFS
(batchId=255)
org.apache.hadoop.hive.ql.parse.TestReplAcrossInstancesWithJsonMessageFormat.testIncrementalCreateFunctionWithFunctionBinaryJarsOnHDFS
(batchId=255)
org.apache.hadoop.hive.ql.parse.TestReplAcrossInstancesWithJsonMessageFormat.testReplLoadFromSourceUsingWithClause
(batchId=255)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testBootstrapFunctionReplication
(batchId=257)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testBootstrapReplLoadRetryAfterFailureForFunctions
(batchId=257)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testBootstrapReplLoadRetryAfterFailureForPartitions
(batchId=257)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testCreateFunctionIncrementalReplication
(batchId=257)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testCreateFunctionWithFunctionBinaryJarsOnHDFS
(batchId=257)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testIncrementalCreateFunctionWithFunctionBinaryJarsOnHDFS
(batchId=257)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testReplLoadFromSourceUsingWithClause
(batchId=257)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17687/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17687/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17687/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972436 - PreCommit-HIVE-Build
> SHOW FUNCTIONS / SHOW FUNCTIONS LIKE - clarify
> --
>
> Key: HIVE-21896
> URL: https://issues.apache.org/jira/browse/HIVE-21896
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: incompatibleChange, todoc4.0
> Fix For: 4.0.0
>
> Attachments: HIVE-21896.01.patch
>
>
> According to
> [https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowFunctions]
> the currently available functions can be listed like this:
> {code:java}
> SHOW FUNCTIONS ;{code}
> If the user executes this command, they will get the correct list of
> functions, but they will also see this on the standard output:
> {code:java}
> SHOW FUNCTIONS is deprecated, please use SHOW FUNCTIONS LIKE instead.{code}
> If the user uses the
> {code:java}
> SHOW FUNCTIONS LIKE ;{code}
> command then they will receive the exact same result (though through
> different codes). The only difference is that one can get all the function
> names with "SHOW FUNCTIONS;", while "SHOW FUNCTIONS LIKE;" returns an
> exception, so in this case the pattern is mandatory.
> So there should be a decision if we still accept "SHOW FUNCTIONS" without the
> "LIKE". My suggestion is to accept it only if there is no pattern. so "SHOW
> FUNCTIONS;" is ok, without deprecation message, but "SHOW FUNCTIONS
> " should throw an exception.
> Whatever we decide, we should document it appropriately.
> cc [~krishahn]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264681=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264681
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 14:28
Start Date: 21/Jun/19 14:28
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296259733
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -82,14 +92,71 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
+ // return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext) {
+String oldName = before.getTableName();
+String newName = after.getTableName();
+
+if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
+ // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
+ // list of tables to be bootstrapped.
+ boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
Review comment:
i think new Jira is required for this ..willl create one ..thus scenario
needs some more thinking
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264681)
Time Spent: 6h 50m (was: 6h 40m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 6h 50m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264680=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264680
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 14:28
Start Date: 21/Jun/19 14:28
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296259435
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/event/filters/ReplEventFilter.java
##
@@ -30,12 +31,22 @@ public ReplEventFilter(final ReplScope replScope) {
this.replScope = replScope;
}
+ // Return false if all the tables are included, as bootstrap during rename
is done only in case filter set for tables.
+ boolean isAlterDuringTableLevelReplication(final NotificationEvent event) {
+if (replScope.includeAllTables() ||
!replScope.getDbNamePattern().matcher(event.getDbName()).matches()) {
Review comment:
shall use dbIncludedInReplScope
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264680)
Time Spent: 6h 40m (was: 6.5h)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 6h 40m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264677=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264677
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 14:22
Start Date: 21/Jun/19 14:22
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296256649
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -82,14 +92,71 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
+ // return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext) {
+String oldName = before.getTableName();
+String newName = after.getTableName();
+
+if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
+ // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
+ // list of tables to be bootstrapped.
+ boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
Review comment:
I think, we look at beforeTableObject in case of shouldReplicate method
which allows this event as the before table was not external or ACID. However,
if at all this is a bug, it is not related to this patch. It is a general issue.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264677)
Time Spent: 6.5h (was: 6h 20m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264676=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264676
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 14:21
Start Date: 21/Jun/19 14:21
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296256649
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -82,14 +92,71 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
+ // return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext) {
+String oldName = before.getTableName();
+String newName = after.getTableName();
+
+if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
+ // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
+ // list of tables to be bootstrapped.
+ boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
Review comment:
I think, we look at beforeTableObject in case of shouldReplicate method
which allows this event as the before table was not external or ACID.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264676)
Time Spent: 6h 20m (was: 6h 10m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 6h 20m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264675=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264675
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 14:20
Start Date: 21/Jun/19 14:20
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296256293
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/event/filters/ReplEventFilter.java
##
@@ -30,12 +31,22 @@ public ReplEventFilter(final ReplScope replScope) {
this.replScope = replScope;
}
+ // Return false if all the tables are included, as bootstrap during rename
is done only in case filter set for tables.
+ boolean isAlterDuringTableLevelReplication(final NotificationEvent event) {
+if (replScope.includeAllTables() ||
!replScope.getDbNamePattern().matcher(event.getDbName()).matches()) {
Review comment:
Well. my point is not about exposing dbNamePattern. It's more about
functionality related to ReplScope. So, the db matching is a functionality
which can be kept isolated within ReplScope. Why need to expose this
functionality outside? Also, having it inside ReplScope would make others to
use it in future instead of duplicating it in the caller.
Anyways, it's your call.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264675)
Time Spent: 6h 10m (was: 6h)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264674=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264674
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 14:18
Start Date: 21/Jun/19 14:18
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296255409
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/event/filters/ReplEventFilter.java
##
@@ -30,12 +31,22 @@ public ReplEventFilter(final ReplScope replScope) {
this.replScope = replScope;
}
+ // Return false if all the tables are included, as bootstrap during rename
is done only in case filter set for tables.
+ boolean isAlterDuringTableLevelReplication(final NotificationEvent event) {
+if (replScope.includeAllTables() ||
!replScope.getDbNamePattern().matcher(event.getDbName()).matches()) {
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264674)
Time Spent: 6h (was: 5h 50m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 6h
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264673=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264673
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 14:18
Start Date: 21/Jun/19 14:18
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296255346
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -82,14 +92,71 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
+ // return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext) {
+String oldName = before.getTableName();
+String newName = after.getTableName();
+
+if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
+ // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
+ // list of tables to be bootstrapped.
+ boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+
+ // If old table satisfies the filter, but the new table does not, then
the old table should be dropped.
+ // This should be done, only if the old table is not in the list of
tables to be bootstrapped which is a multi
+ // rename case. In case of multi rename, only the first rename should do
the drop.
+ if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope,
newName)) {
+if (oldTableIsPresent) {
+ // If the old table was present in the list of tables to be
bootstrapped, then just ignore the event.
+ return false;
+} else {
+ scenario = Scenario.DROP;
+ LOG.info("Table " + oldName + " will be dropped as the table is
renamed to " + newName);
+ return true;
+}
+ }
+
+ // If the old table was in the list of tables to be bootstrapped which
is a multi rename case, the old table
+ // is removed from the list of tables to be bootstrapped and new one is
added.
+ if (oldTableIsPresent) {
+withinContext.addToListOfTablesForBootstrap(newName);
+return false;
+ }
+
+ // If both old and new table satisfies the filter and old table is
present at target, then dump the rename event.
+ LOG.info("both old and new table satisfies the filter");
+ return true;
+} else {
+ // if the old table does not satisfies the filter, but the new one
satisfies, then the new table should be
+ // added to the list of tables to be bootstrapped and don't dump the
event.
+ if (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
newName)) {
+LOG.info("Table " + newName + " is added for bootstrap " + " during
rename from " + oldName);
+withinContext.addToListOfTablesForBootstrap(newName);
+return false;
+ }
+
+ // if both old and new table does not satisfies the filter, then don't
dump the event.
+ LOG.info("both old and new table not satisfies the filter");
+ return false;
+}
+ }
+
@Override
public void handle(Context withinContext) throws Exception {
LOG.info("Processing#{} ALTER_TABLE message : {}", fromEventId(),
eventMessageAsJSON);
Table qlMdTableBefore = new Table(before);
+Set bootstrapTableList;
+if (Scenario.RENAME == scenario) {
+ // Handling for table level replication is done in
handleForTableLevelReplication method.
+ bootstrapTableList = null;
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264673)
Time Spent: 5h 50m (was: 5h 40m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication,
[jira] [Updated] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[ https://issues.apache.org/jira/browse/HIVE-21764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21764: --- Status: Open (was: Patch Available) > REPL DUMP should detect and bootstrap any rename table events where old table > was excluded but renamed table is included. > - > > Key: HIVE-21764 > URL: https://issues.apache.org/jira/browse/HIVE-21764 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Sankar Hariappan >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch > > Time Spent: 5h 40m > Remaining Estimate: 0h > > REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular > expression + inclusion/exclusion list. So, in case of rename table event, the > event will be ignored if old table doesn't match the pattern but the new > table should be bootstrapped. REPL DUMP should have a mechanism to detect > such tables and automatically bootstrap with incremental replication.Also, if > renamed table is excluded from replication policy, then need to drop the old > table at target as well. > There are 4 scenarios that needs to be handled. > # Both new name and old name satisfies the table name pattern filter. > ## No need to do anything. The incremental event for rename should take care > of the replication. > # Both the names does not satisfy the table name pattern filter. > ## Both the names are not in the scope of the policy and thus nothing needs > to be done. > # New name satisfies the pattern but the old name does not. > ## The table will not be present at the target. > ## Rename event handler for dump should detect this case and add the new > table name to the list of table for bootstrap. > ## All the events related to the table (new name) should be ignored. > ## If there is a drop event for the table (with new name), then remove the > table from the list of tables to be bootstrapped. > ## In case of rename (double rename) > ### If the new name satisfies the table pattern, then add the new name to > the list of tables to be bootstrapped and remove the old name from the list > of tables to be bootstrapped. > ### If the new name does not satisfies then just removed the table name from > the list of tables to be bootstrapped. > # New name does not satisfies the pattern but the old name satisfies. > ## Change the rename event to a drop event. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264670=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264670
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 14:11
Start Date: 21/Jun/19 14:11
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296252578
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -82,14 +90,49 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
+ // return true, if event needs to be dumped, else return false.
+ private boolean handleRenameDuringTableLevelReplication(Context
withinContext) {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264670)
Time Spent: 5h 40m (was: 5.5h)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264665=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264665
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 14:07
Start Date: 21/Jun/19 14:07
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296250742
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/event/filters/ReplEventFilter.java
##
@@ -30,12 +31,22 @@ public ReplEventFilter(final ReplScope replScope) {
this.replScope = replScope;
}
+ // Return false if all the tables are included, as bootstrap during rename
is done only in case filter set for tables.
+ boolean isAlterDuringTableLevelReplication(final NotificationEvent event) {
+if (replScope.includeAllTables() ||
!replScope.getDbNamePattern().matcher(event.getDbName()).matches()) {
Review comment:
if db name can be exposed ..this i don't think has any issue ..
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264665)
Time Spent: 5.5h (was: 5h 20m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264664=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264664
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 14:06
Start Date: 21/Jun/19 14:06
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296250437
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/event/filters/ReplEventFilter.java
##
@@ -30,12 +31,22 @@ public ReplEventFilter(final ReplScope replScope) {
this.replScope = replScope;
}
+ // Return false if all the tables are included, as bootstrap during rename
is done only in case filter set for tables.
+ boolean isAlterDuringTableLevelReplication(final NotificationEvent event) {
+if (replScope.includeAllTables() ||
!replScope.getDbNamePattern().matcher(event.getDbName()).matches()) {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264664)
Time Spent: 5h 20m (was: 5h 10m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264662=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264662
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 14:05
Start Date: 21/Jun/19 14:05
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296250117
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/event/filters/ReplEventFilter.java
##
@@ -30,12 +31,22 @@ public ReplEventFilter(final ReplScope replScope) {
this.replScope = replScope;
}
+ // Return false if all the tables are included, as bootstrap during rename
is done only in case filter set for tables.
+ boolean isAlterDuringTableLevelReplication(final NotificationEvent event) {
Review comment:
its inside alter table handler class so not a problem :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264662)
Time Spent: 5h 10m (was: 5h)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264658=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264658
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 14:02
Start Date: 21/Jun/19 14:02
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296248646
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -82,14 +92,71 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
+ // return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext) {
+String oldName = before.getTableName();
+String newName = after.getTableName();
+
+if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
+ // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
+ // list of tables to be bootstrapped.
+ boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
Review comment:
bootstrap external or bootstrap acid is handled in shuldreplicate method
..the control should not reach till here
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264658)
Time Spent: 5h (was: 4h 50m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 5h
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264651=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264651
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 14:00
Start Date: 21/Jun/19 14:00
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296247839
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/event/filters/ReplEventFilter.java
##
@@ -30,12 +31,22 @@ public ReplEventFilter(final ReplScope replScope) {
this.replScope = replScope;
}
+ // Return false if all the tables are included, as bootstrap during rename
is done only in case filter set for tables.
+ boolean isAlterDuringTableLevelReplication(final NotificationEvent event) {
+if (replScope.includeAllTables() ||
!replScope.getDbNamePattern().matcher(event.getDbName()).matches()) {
Review comment:
i think db rename is not supported
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264651)
Time Spent: 4h 50m (was: 4h 40m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264649=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264649
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 13:59
Start Date: 21/Jun/19 13:59
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296247443
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -82,14 +92,71 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
+ // return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext) {
+String oldName = before.getTableName();
+String newName = after.getTableName();
+
+if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
+ // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
+ // list of tables to be bootstrapped.
+ boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+
+ // If old table satisfies the filter, but the new table does not, then
the old table should be dropped.
+ // This should be done, only if the old table is not in the list of
tables to be bootstrapped which is a multi
+ // rename case. In case of multi rename, only the first rename should do
the drop.
+ if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope,
newName)) {
+if (oldTableIsPresent) {
+ // If the old table was present in the list of tables to be
bootstrapped, then just ignore the event.
+ return false;
+} else {
+ scenario = Scenario.DROP;
+ LOG.info("Table " + oldName + " will be dropped as the table is
renamed to " + newName);
+ return true;
+}
+ }
+
+ // If the old table was in the list of tables to be bootstrapped which
is a multi rename case, the old table
+ // is removed from the list of tables to be bootstrapped and new one is
added.
+ if (oldTableIsPresent) {
+withinContext.addToListOfTablesForBootstrap(newName);
+return false;
+ }
+
+ // If both old and new table satisfies the filter and old table is
present at target, then dump the rename event.
+ LOG.info("both old and new table satisfies the filter");
+ return true;
+} else {
+ // if the old table does not satisfies the filter, but the new one
satisfies, then the new table should be
+ // added to the list of tables to be bootstrapped and don't dump the
event.
+ if (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
newName)) {
+LOG.info("Table " + newName + " is added for bootstrap " + " during
rename from " + oldName);
+withinContext.addToListOfTablesForBootstrap(newName);
+return false;
+ }
+
+ // if both old and new table does not satisfies the filter, then don't
dump the event.
+ LOG.info("both old and new table not satisfies the filter");
+ return false;
+}
+ }
+
@Override
public void handle(Context withinContext) throws Exception {
LOG.info("Processing#{} ALTER_TABLE message : {}", fromEventId(),
eventMessageAsJSON);
Table qlMdTableBefore = new Table(before);
+Set bootstrapTableList;
+if (Scenario.RENAME == scenario) {
+ // Handling for table level replication is done in
handleForTableLevelReplication method.
+ bootstrapTableList = null;
Review comment:
replScope is for replace policy (old policy) ..its n not handled in this
patch
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264649)
Time Spent: 4h 40m (was: 4.5h)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh
[jira] [Commented] (HIVE-21896) SHOW FUNCTIONS / SHOW FUNCTIONS LIKE - clarify
[
https://issues.apache.org/jira/browse/HIVE-21896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869480#comment-16869480
]
Hive QA commented on HIVE-21896:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
11s{color} | {color:blue} ql in master has 2254 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} ql: The patch generated 0 new + 292 unchanged - 1
fixed = 292 total (was 293) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17687/dev-support/hive-personality.sh
|
| git revision | master / 9890919 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17687/yetus/whitespace-eol.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17687/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> SHOW FUNCTIONS / SHOW FUNCTIONS LIKE - clarify
> --
>
> Key: HIVE-21896
> URL: https://issues.apache.org/jira/browse/HIVE-21896
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: incompatibleChange, todoc4.0
> Fix For: 4.0.0
>
> Attachments: HIVE-21896.01.patch
>
>
> According to
> [https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowFunctions]
> the currently available functions can be listed like this:
> {code:java}
> SHOW FUNCTIONS ;{code}
> If the user executes this command, they will get the correct list of
> functions, but they will also see this on the standard output:
> {code:java}
> SHOW FUNCTIONS is deprecated, please use SHOW FUNCTIONS LIKE instead.{code}
> If the user uses the
> {code:java}
> SHOW FUNCTIONS LIKE ;{code}
> command then they will receive the exact same result (though through
> different codes). The only difference is that one can get all the function
> names with "SHOW FUNCTIONS;", while "SHOW FUNCTIONS LIKE;" returns an
> exception, so in this case the pattern is mandatory.
> So there should be a decision if we still accept "SHOW FUNCTIONS" without the
> "LIKE". My suggestion is to accept it only if there
[jira] [Commented] (HIVE-21894) Hadoop credential password storage for the Kafka Storage handler when security is SSL
[
https://issues.apache.org/jira/browse/HIVE-21894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869476#comment-16869476
]
Kristopher Kane commented on HIVE-21894:
[~bslim] Yes, this the Kafka method, but two things are missing within the
context of Hive execution:
1) Protection of the credentials for the keystore/truststore - will need Hadoop
credentials similar to the JDBC handler and then add them to the consumer
config once they are retrieved
2) That path is an absolute path to a linux file system only. We cannot assume
that the Hive user level would have the ability or infrastructure to distribute
the keystore/truststore to every compute node or HS2 instance (I'm not actually
sure when this would get loaded with regard to Hive bootstraping of the handler)
I wrote up my idea and some questions on the dev mailing list yesterday. I'll
post it here too:
{noformat}
I’m attempting to add SSL support to the Kafka storage handler and will need to
make the keystore/truststore available to KafkaConsumer via an absolute path on
a local file system.
The ideal steps are these:
1. TBLPROPERTIES describe an absolute HDFS path for the keystore/truststore
2. The Kafka storage handler copies both files from HDFS to the container’s
local FS and configures and builds the KafkaConsumer around an absolute
reference to this local YARN container directory.
3. Passwords for these files are stored using TBLPROPERTIES references to
Hadoop credentials similar to the examples in the JDBC storage handler
Looks like there is precedent for interacting with HDFS from a storage handler
level for HBase here:
https://github.com/apache/hive/blob/master/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeHelper.java#L366,
then in the Druid Storage handler there is a reference to using the local
filesystem for dependency jar additions:
https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandlerUtils.java#L692
Need some help with a few questions:
1. Is that the FS under the temporary YARN container example in the second
link where all software/configuration is distributed?
2. Is this Storage handler execution scope at the ‘mapper’ level on a
container running YARN?
3. Do these file movement steps seem ok and within a storage handler’s
scope?{noformat}
> Hadoop credential password storage for the Kafka Storage handler when
> security is SSL
> -
>
> Key: HIVE-21894
> URL: https://issues.apache.org/jira/browse/HIVE-21894
> Project: Hive
> Issue Type: Improvement
> Components: kafka integration
>Affects Versions: 4.0.0
>Reporter: Kristopher Kane
>Assignee: Kristopher Kane
>Priority: Minor
> Fix For: 4.0.0
>
>
> The Kafka storage handler assumes that if the Hive service is configured with
> Kerberos then the destination Kafka cluster is also secured with the same
> Kerberos realm or trust of realms. The security configuration of the Kafka
> client can be overwritten due to the additive operations of the Kafka client
> configs, but, the only way to specify SSL and the keystore/truststore
> user/pass is via plain text table properties.
> This ticket proposes adding Hadoop credential security to the Kafka storage
> handler in support of SSL secured Kafka clusters.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21905) Generics improvement around the FetchOperator class
[ https://issues.apache.org/jira/browse/HIVE-21905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Suller updated HIVE-21905: --- Attachment: HIVE-21905.1.patch > Generics improvement around the FetchOperator class > --- > > Key: HIVE-21905 > URL: https://issues.apache.org/jira/browse/HIVE-21905 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Minor > Attachments: HIVE-21905.1.patch > > > In and around the org.apache.hadoop.hive.ql.exec.FetchOperator class the > generics are handled poorly. Lot's of declarations are missing generics, > which makes lots of noise in the IDE and makes it hard to be sure of the > correctness of the code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21905) Generics improvement around the FetchOperator class
[ https://issues.apache.org/jira/browse/HIVE-21905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Suller updated HIVE-21905: --- Status: Patch Available (was: In Progress) > Generics improvement around the FetchOperator class > --- > > Key: HIVE-21905 > URL: https://issues.apache.org/jira/browse/HIVE-21905 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Minor > Attachments: HIVE-21905.1.patch > > > In and around the org.apache.hadoop.hive.ql.exec.FetchOperator class the > generics are handled poorly. Lot's of declarations are missing generics, > which makes lots of noise in the IDE and makes it hard to be sure of the > correctness of the code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869453#comment-16869453
]
Hive QA commented on HIVE-21764:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972430/HIVE-21764.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 16339 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.parse.TestTableLevelReplicationScenarios.testBasicReplaceReplPolicy
(batchId=255)
org.apache.hadoop.hive.ql.parse.TestTableLevelReplicationScenarios.testCaseInSensitiveNatureOfReplPolicy
(batchId=255)
org.apache.hadoop.hive.ql.parse.TestTableLevelReplicationScenarios.testIncorrectTablePolicyInReplDump
(batchId=255)
org.apache.hadoop.hive.ql.parse.TestTableLevelReplicationScenarios.testReplacePolicyOnBootstrapAcidTablesIncrementalPhase
(batchId=255)
org.apache.hadoop.hive.ql.parse.TestTableLevelReplicationScenarios.testReplacePolicyOnBootstrapExternalTablesIncrementalPhase
(batchId=255)
org.apache.hadoop.hive.ql.parse.TestTableLevelReplicationScenarios.testReplacePolicyWhenAcidTablesDisabledForRepl
(batchId=255)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17686/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17686/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17686/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972430 - PreCommit-HIVE-Build
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21905) Generics improvement around the FetchOperator class
[ https://issues.apache.org/jira/browse/HIVE-21905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Suller reassigned HIVE-21905: -- > Generics improvement around the FetchOperator class > --- > > Key: HIVE-21905 > URL: https://issues.apache.org/jira/browse/HIVE-21905 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Minor > > In and around the org.apache.hadoop.hive.ql.exec.FetchOperator class the > generics are handled poorly. Lot's of declarations are missing generics, > which makes lots of noise in the IDE and makes it hard to be sure of the > correctness of the code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-21905) Generics improvement around the FetchOperator class
[ https://issues.apache.org/jira/browse/HIVE-21905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-21905 started by Ivan Suller. -- > Generics improvement around the FetchOperator class > --- > > Key: HIVE-21905 > URL: https://issues.apache.org/jira/browse/HIVE-21905 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Minor > > In and around the org.apache.hadoop.hive.ql.exec.FetchOperator class the > generics are handled poorly. Lot's of declarations are missing generics, > which makes lots of noise in the IDE and makes it hard to be sure of the > correctness of the code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21896) SHOW FUNCTIONS / SHOW FUNCTIONS LIKE - clarify
[
https://issues.apache.org/jira/browse/HIVE-21896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869449#comment-16869449
]
Zoltan Haindrich commented on HIVE-21896:
-
added todoc label; and marked it "incompatibleChange" since the pattern will
change from ".*" to (sql like) "%" for "SHOW FUNTIONS LIKE"
> SHOW FUNCTIONS / SHOW FUNCTIONS LIKE - clarify
> --
>
> Key: HIVE-21896
> URL: https://issues.apache.org/jira/browse/HIVE-21896
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: incompatibleChange, todoc4.0
> Fix For: 4.0.0
>
> Attachments: HIVE-21896.01.patch
>
>
> According to
> [https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowFunctions]
> the currently available functions can be listed like this:
> {code:java}
> SHOW FUNCTIONS ;{code}
> If the user executes this command, they will get the correct list of
> functions, but they will also see this on the standard output:
> {code:java}
> SHOW FUNCTIONS is deprecated, please use SHOW FUNCTIONS LIKE instead.{code}
> If the user uses the
> {code:java}
> SHOW FUNCTIONS LIKE ;{code}
> command then they will receive the exact same result (though through
> different codes). The only difference is that one can get all the function
> names with "SHOW FUNCTIONS;", while "SHOW FUNCTIONS LIKE;" returns an
> exception, so in this case the pattern is mandatory.
> So there should be a decision if we still accept "SHOW FUNCTIONS" without the
> "LIKE". My suggestion is to accept it only if there is no pattern. so "SHOW
> FUNCTIONS;" is ok, without deprecation message, but "SHOW FUNCTIONS
> " should throw an exception.
> Whatever we decide, we should document it appropriately.
> cc [~krishahn]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21896) SHOW FUNCTIONS / SHOW FUNCTIONS LIKE - clarify
[
https://issues.apache.org/jira/browse/HIVE-21896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-21896:
Labels: incompatibleChange todoc4.0 (was: )
> SHOW FUNCTIONS / SHOW FUNCTIONS LIKE - clarify
> --
>
> Key: HIVE-21896
> URL: https://issues.apache.org/jira/browse/HIVE-21896
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: incompatibleChange, todoc4.0
> Fix For: 4.0.0
>
> Attachments: HIVE-21896.01.patch
>
>
> According to
> [https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowFunctions]
> the currently available functions can be listed like this:
> {code:java}
> SHOW FUNCTIONS ;{code}
> If the user executes this command, they will get the correct list of
> functions, but they will also see this on the standard output:
> {code:java}
> SHOW FUNCTIONS is deprecated, please use SHOW FUNCTIONS LIKE instead.{code}
> If the user uses the
> {code:java}
> SHOW FUNCTIONS LIKE ;{code}
> command then they will receive the exact same result (though through
> different codes). The only difference is that one can get all the function
> names with "SHOW FUNCTIONS;", while "SHOW FUNCTIONS LIKE;" returns an
> exception, so in this case the pattern is mandatory.
> So there should be a decision if we still accept "SHOW FUNCTIONS" without the
> "LIKE". My suggestion is to accept it only if there is no pattern. so "SHOW
> FUNCTIONS;" is ok, without deprecation message, but "SHOW FUNCTIONS
> " should throw an exception.
> Whatever we decide, we should document it appropriately.
> cc [~krishahn]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869444#comment-16869444
]
Hive QA commented on HIVE-21764:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
40s{color} | {color:blue} standalone-metastore/metastore-common in master has
31 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
10s{color} | {color:blue} standalone-metastore/metastore-server in master has
179 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 2254 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 1 new + 39 unchanged - 0 fixed
= 40 total (was 39) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17686/dev-support/hive-personality.sh
|
| git revision | master / 9890919 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17686/yetus/diff-checkstyle-ql.txt
|
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17686/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
>
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264577=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264577
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 12:23
Start Date: 21/Jun/19 12:23
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296202122
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/event/filters/ReplEventFilter.java
##
@@ -30,12 +31,22 @@ public ReplEventFilter(final ReplScope replScope) {
this.replScope = replScope;
}
+ // Return false if all the tables are included, as bootstrap during rename
is done only in case filter set for tables.
+ boolean isAlterDuringTableLevelReplication(final NotificationEvent event) {
+if (replScope.includeAllTables() ||
!replScope.getDbNamePattern().matcher(event.getDbName()).matches()) {
Review comment:
replScope.includeAllTables() will always be false. If true, then it wouldn't
reach here as replScope.includedInReplScope will be true in the caller.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264577)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264575=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264575
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 12:23
Start Date: 21/Jun/19 12:23
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296212406
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -82,14 +92,71 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
+ // return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext) {
+String oldName = before.getTableName();
+String newName = after.getTableName();
+
+if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
+ // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
+ // list of tables to be bootstrapped.
+ boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
Review comment:
Let's say, a non-acid table is converted to external. Now, since, both names
match ReplScope, this event is dumped. Now, all the other events on this alter
table would be skipped as BOOTSTRAP_EXTERNAL config is true. Now, we replicate
the event and also, bootstrap the table which may fail. This is not related to
this patch but still want to check if it can happen. I noticed, we block change
table type only if STRICT_MANAGED=true.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264575)
Time Spent: 4h 20m (was: 4h 10m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264578=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264578
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 12:23
Start Date: 21/Jun/19 12:23
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296202255
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/event/filters/ReplEventFilter.java
##
@@ -30,12 +31,22 @@ public ReplEventFilter(final ReplScope replScope) {
this.replScope = replScope;
}
+ // Return false if all the tables are included, as bootstrap during rename
is done only in case filter set for tables.
+ boolean isAlterDuringTableLevelReplication(final NotificationEvent event) {
+if (replScope.includeAllTables() ||
!replScope.getDbNamePattern().matcher(event.getDbName()).matches()) {
Review comment:
Is it possible to rename to different DB altogether?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264578)
Time Spent: 4.5h (was: 4h 20m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264574=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264574
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 12:23
Start Date: 21/Jun/19 12:23
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296203142
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/event/filters/ReplEventFilter.java
##
@@ -30,12 +31,22 @@ public ReplEventFilter(final ReplScope replScope) {
this.replScope = replScope;
}
+ // Return false if all the tables are included, as bootstrap during rename
is done only in case filter set for tables.
+ boolean isAlterDuringTableLevelReplication(final NotificationEvent event) {
+if (replScope.includeAllTables() ||
!replScope.getDbNamePattern().matcher(event.getDbName()).matches()) {
Review comment:
Instead of exposing dbNamePattern outside ReplScope, can we add a method
that checks if Db matches the replScope. That seems cleaner.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264574)
Time Spent: 4h 20m (was: 4h 10m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264576=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264576
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 12:23
Start Date: 21/Jun/19 12:23
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296201139
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -82,14 +92,71 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
+ // return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext) {
+String oldName = before.getTableName();
+String newName = after.getTableName();
+
+if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
+ // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
+ // list of tables to be bootstrapped.
+ boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+
+ // If old table satisfies the filter, but the new table does not, then
the old table should be dropped.
+ // This should be done, only if the old table is not in the list of
tables to be bootstrapped which is a multi
+ // rename case. In case of multi rename, only the first rename should do
the drop.
+ if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope,
newName)) {
+if (oldTableIsPresent) {
+ // If the old table was present in the list of tables to be
bootstrapped, then just ignore the event.
+ return false;
+} else {
+ scenario = Scenario.DROP;
+ LOG.info("Table " + oldName + " will be dropped as the table is
renamed to " + newName);
+ return true;
+}
+ }
+
+ // If the old table was in the list of tables to be bootstrapped which
is a multi rename case, the old table
+ // is removed from the list of tables to be bootstrapped and new one is
added.
+ if (oldTableIsPresent) {
+withinContext.addToListOfTablesForBootstrap(newName);
+return false;
+ }
+
+ // If both old and new table satisfies the filter and old table is
present at target, then dump the rename event.
+ LOG.info("both old and new table satisfies the filter");
+ return true;
+} else {
+ // if the old table does not satisfies the filter, but the new one
satisfies, then the new table should be
+ // added to the list of tables to be bootstrapped and don't dump the
event.
+ if (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
newName)) {
+LOG.info("Table " + newName + " is added for bootstrap " + " during
rename from " + oldName);
+withinContext.addToListOfTablesForBootstrap(newName);
+return false;
+ }
+
+ // if both old and new table does not satisfies the filter, then don't
dump the event.
+ LOG.info("both old and new table not satisfies the filter");
+ return false;
+}
+ }
+
@Override
public void handle(Context withinContext) throws Exception {
LOG.info("Processing#{} ALTER_TABLE message : {}", fromEventId(),
eventMessageAsJSON);
Table qlMdTableBefore = new Table(before);
+Set bootstrapTableList;
+if (Scenario.RENAME == scenario) {
+ // Handling for table level replication is done in
handleForTableLevelReplication method.
+ bootstrapTableList = null;
Review comment:
If bootstrapTableList is passed as null, then if old table doesn't match the
replScope, then this event will be skipped before reaching
handleForTableLevelReplication.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264576)
Time Spent: 4h 20m (was: 4h 10m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
>
[jira] [Updated] (HIVE-21896) SHOW FUNCTIONS / SHOW FUNCTIONS LIKE - clarify
[
https://issues.apache.org/jira/browse/HIVE-21896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-21896:
--
Status: Patch Available (was: Open)
> SHOW FUNCTIONS / SHOW FUNCTIONS LIKE - clarify
> --
>
> Key: HIVE-21896
> URL: https://issues.apache.org/jira/browse/HIVE-21896
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21896.01.patch
>
>
> According to
> [https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowFunctions]
> the currently available functions can be listed like this:
> {code:java}
> SHOW FUNCTIONS ;{code}
> If the user executes this command, they will get the correct list of
> functions, but they will also see this on the standard output:
> {code:java}
> SHOW FUNCTIONS is deprecated, please use SHOW FUNCTIONS LIKE instead.{code}
> If the user uses the
> {code:java}
> SHOW FUNCTIONS LIKE ;{code}
> command then they will receive the exact same result (though through
> different codes). The only difference is that one can get all the function
> names with "SHOW FUNCTIONS;", while "SHOW FUNCTIONS LIKE;" returns an
> exception, so in this case the pattern is mandatory.
> So there should be a decision if we still accept "SHOW FUNCTIONS" without the
> "LIKE". My suggestion is to accept it only if there is no pattern. so "SHOW
> FUNCTIONS;" is ok, without deprecation message, but "SHOW FUNCTIONS
> " should throw an exception.
> Whatever we decide, we should document it appropriately.
> cc [~krishahn]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21874) Implement add partitions related methods on temporary table
[
https://issues.apache.org/jira/browse/HIVE-21874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-21874:
-
Attachment: HIVE-21874.01.patch
> Implement add partitions related methods on temporary table
> ---
>
> Key: HIVE-21874
> URL: https://issues.apache.org/jira/browse/HIVE-21874
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-21874.01.patch
>
>
> IMetaStoreClient exposes the following add partition related methods:
> {code:java}
> Partition add_partition(Partition partition);
> int add_partitions(List partitions);
> int add_partitions_pspec(PartitionSpecProxy partitionSpec);
> List add_partitions(List partitions, boolean
> ifNotExists, boolean needResults);
> {code}
> These methods should be implemented in order to handle addition of partitions
> to temporary tables.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21874) Implement add partitions related methods on temporary table
[
https://issues.apache.org/jira/browse/HIVE-21874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-21874:
-
Status: Patch Available (was: Open)
> Implement add partitions related methods on temporary table
> ---
>
> Key: HIVE-21874
> URL: https://issues.apache.org/jira/browse/HIVE-21874
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-21874.01.patch
>
>
> IMetaStoreClient exposes the following add partition related methods:
> {code:java}
> Partition add_partition(Partition partition);
> int add_partitions(List partitions);
> int add_partitions_pspec(PartitionSpecProxy partitionSpec);
> List add_partitions(List partitions, boolean
> ifNotExists, boolean needResults);
> {code}
> These methods should be implemented in order to handle addition of partitions
> to temporary tables.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21896) SHOW FUNCTIONS / SHOW FUNCTIONS LIKE - clarify
[
https://issues.apache.org/jira/browse/HIVE-21896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-21896:
--
Attachment: HIVE-21896.01.patch
> SHOW FUNCTIONS / SHOW FUNCTIONS LIKE - clarify
> --
>
> Key: HIVE-21896
> URL: https://issues.apache.org/jira/browse/HIVE-21896
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21896.01.patch
>
>
> According to
> [https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowFunctions]
> the currently available functions can be listed like this:
> {code:java}
> SHOW FUNCTIONS ;{code}
> If the user executes this command, they will get the correct list of
> functions, but they will also see this on the standard output:
> {code:java}
> SHOW FUNCTIONS is deprecated, please use SHOW FUNCTIONS LIKE instead.{code}
> If the user uses the
> {code:java}
> SHOW FUNCTIONS LIKE ;{code}
> command then they will receive the exact same result (though through
> different codes). The only difference is that one can get all the function
> names with "SHOW FUNCTIONS;", while "SHOW FUNCTIONS LIKE;" returns an
> exception, so in this case the pattern is mandatory.
> So there should be a decision if we still accept "SHOW FUNCTIONS" without the
> "LIKE". My suggestion is to accept it only if there is no pattern. so "SHOW
> FUNCTIONS;" is ok, without deprecation message, but "SHOW FUNCTIONS
> " should throw an exception.
> Whatever we decide, we should document it appropriately.
> cc [~krishahn]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21868) Vectorize CAST...FORMAT
[
https://issues.apache.org/jira/browse/HIVE-21868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869410#comment-16869410
]
Hive QA commented on HIVE-21868:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972413/HIVE-21868.02.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16341 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17685/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17685/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17685/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972413 - PreCommit-HIVE-Build
> Vectorize CAST...FORMAT
> ---
>
> Key: HIVE-21868
> URL: https://issues.apache.org/jira/browse/HIVE-21868
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21868.01.patch, HIVE-21868.01.patch,
> HIVE-21868.02.patch
>
>
> Vectorize UDFs for CAST ( AS STRING/CHAR/VARCHAR FORMAT
> ) and CAST ( AS TIMESTAMP/DATE FORMAT ).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264545=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264545
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 11:22
Start Date: 21/Jun/19 11:22
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296197082
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/Utils.java
##
@@ -214,29 +215,35 @@ public static boolean shouldReplicate(ReplicationSpec
replicationSpec, Table tab
return false;
}
-// Skip dumping events related to ACID tables if bootstrap is enabled
on it.
-// Also, skip if current table is included only in new policy but not
in old policy.
-if (isEventDump) {
- return
!hiveConf.getBoolVar(HiveConf.ConfVars.REPL_BOOTSTRAP_ACID_TABLES)
- && ReplUtils.tableIncludedInReplScope(oldReplScope,
tableHandle.getTableName());
+// Skip dumping events related to ACID tables if bootstrap is enabled
for ACID tables.
+if (isEventDump &&
hiveConf.getBoolVar(HiveConf.ConfVars.REPL_BOOTSTRAP_ACID_TABLES)) {
+ return false;
}
}
- // If replication policy is replaced with new included/excluded tables
list, then events
- // corresponding to tables which are not included in old policy but
included in new policy
- // should be skipped. Those tables would be bootstrapped along with the
current incremental
- // replication dump.
- // Note: If any event dump reaches here, it means, table is included in
new replication policy.
- if (isEventDump && !ReplUtils.tableIncludedInReplScope(oldReplScope,
tableHandle.getTableName())) {
-return false;
+ // Tables which are selected for bootstrap should be skipped. Those
tables would be bootstrapped
+ // along with the current incremental replication dump and thus no need
to dump events for them.
+ // Note: If any event (other than alter table with table level
replication) dump reaches here, it means, table is
+ // included in new replication policy.
+ if (isEventDump) {
+// If replication policy is replaced with new included/excluded tables
list, then events
+// corresponding to tables which are not included in old policy but
included in new policy
+// should be skipped.
+if (!ReplUtils.tableIncludedInReplScope(oldReplScope,
tableHandle.getTableName())) {
+ return false;
+}
+
+// Tables in the list of tables to be bootstrapped should be skipped.
Review comment:
I meant, "Events are skipped not tables."
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264545)
Time Spent: 4h 10m (was: 4h)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264542=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264542
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 11:20
Start Date: 21/Jun/19 11:20
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296196665
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/event/filters/ReplEventFilter.java
##
@@ -30,12 +31,22 @@ public ReplEventFilter(final ReplScope replScope) {
this.replScope = replScope;
}
+ // Return false if all the tables are included, as bootstrap during rename
is done only in case filter set for tables.
+ boolean isAlterDuringTableLevelReplication(final NotificationEvent event) {
Review comment:
ok... shall change to isAlterTableEvent. We have alter event for partitions
as well. :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264542)
Time Spent: 3h 50m (was: 3h 40m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264543=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264543
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 11:21
Start Date: 21/Jun/19 11:21
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296196726
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -101,6 +144,14 @@ public void handle(Context withinContext) throws
Exception {
}
}
+// If the tables are filtered based on name, then needs to handle the
rename scenarios.
+if (withinContext.replScope != null &&
!withinContext.replScope.includeAllTables()) {
+ if (!handleRenameDuringTableLevelReplication(withinContext)) {
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264543)
Time Spent: 4h (was: 3h 50m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
[
https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264540=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264540
]
ASF GitHub Bot logged work on HIVE-21764:
-
Author: ASF GitHub Bot
Created on: 21/Jun/19 11:18
Start Date: 21/Jun/19 11:18
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #679: HIVE-21764 :
REPL DUMP should detect and bootstrap any rename table events where old table
was excluded but renamed table is included.
URL: https://github.com/apache/hive/pull/679#discussion_r296196167
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -82,14 +90,49 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
+ // return true, if event needs to be dumped, else return false.
+ private boolean handleRenameDuringTableLevelReplication(Context
withinContext) {
Review comment:
It says, handleRename but it actually handles all alter events. So, name can
be relevant to that.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 264540)
Time Spent: 3.5h (was: 3h 20m)
> REPL DUMP should detect and bootstrap any rename table events where old table
> was excluded but renamed table is included.
> -
>
> Key: HIVE-21764
> URL: https://issues.apache.org/jira/browse/HIVE-21764
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular
> expression + inclusion/exclusion list. So, in case of rename table event, the
> event will be ignored if old table doesn't match the pattern but the new
> table should be bootstrapped. REPL DUMP should have a mechanism to detect
> such tables and automatically bootstrap with incremental replication.Also, if
> renamed table is excluded from replication policy, then need to drop the old
> table at target as well.
> There are 4 scenarios that needs to be handled.
> # Both new name and old name satisfies the table name pattern filter.
> ## No need to do anything. The incremental event for rename should take care
> of the replication.
> # Both the names does not satisfy the table name pattern filter.
> ## Both the names are not in the scope of the policy and thus nothing needs
> to be done.
> # New name satisfies the pattern but the old name does not.
> ## The table will not be present at the target.
> ## Rename event handler for dump should detect this case and add the new
> table name to the list of table for bootstrap.
> ## All the events related to the table (new name) should be ignored.
> ## If there is a drop event for the table (with new name), then remove the
> table from the list of tables to be bootstrapped.
> ## In case of rename (double rename)
> ### If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ### If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # New name does not satisfies the pattern but the old name satisfies.
> ## Change the rename event to a drop event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
