[jira] [Commented] (HIVE-23359) "show tables like" support for SQL wildcard characters (% and _)
[
https://issues.apache.org/jira/browse/HIVE-23359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101375#comment-17101375
]
Hive QA commented on HIVE-23359:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/1300/HIVE-23359.05.patch
{color:green}SUCCESS:{color} +1 due to 19 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17232 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.exec.spark.TestLocalHiveSparkClient.testMultiSessionSparkContextReUse
(batchId=289)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22172/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22172/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22172/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 1300 - PreCommit-HIVE-Build
> "show tables like" support for SQL wildcard characters (% and _)
>
>
> Key: HIVE-23359
> URL: https://issues.apache.org/jira/browse/HIVE-23359
> Project: Hive
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 2.3.7
>Reporter: Yuming Wang
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-23359.01.patch, HIVE-23359.02.patch,
> HIVE-23359.03.patch, HIVE-23359.04.patch, HIVE-23359.05.patch
>
>
> https://docs.snowflake.com/en/sql-reference/sql/show-tables.html
> https://clickhouse.tech/docs/en/sql-reference/statements/show/
> https://www.mysqltutorial.org/mysql-show-tables/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23390) Duplicate entry for a table in TAB_COL_STATS
[
https://issues.apache.org/jira/browse/HIVE-23390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Antony updated HIVE-23390:
-
Description:
When *_analyze _* command was executed from presto to update the stats
of a table for the first time from multiple cluster sharing the same Hive
metastore. Duplicate entry for the same table is inserted to the
*_TAB_COL_STATS_* table.
This lead to failure executing further *_analyze _* commands.
{code:java}
Query failed: Multiple entries with same key:
dummy=HiveColumnStatistics{integerStatistics=Optional[IntegerStatistics{min=OptionalLong[1],
max=OptionalLong[1]}], doubleStatistics=Optional.empty,
decimalStatistics=Optional.empty, dateStatistics=Optional.empty,
booleanStatistics=Optional.empty, maxValueSizeInBytes=OptionalLong.empty,
totalSizeInBytes=OptionalLong.empty, nullsCount=OptionalLong[0],
distinctValuesCount=OptionalLong[1]} and
dummy=HiveColumnStatistics{integerStatistics=Optional[IntegerStatistics{min=OptionalLong[1],
max=OptionalLong[1]}], doubleStatistics=Optional.empty,
decimalStatistics=Optional.empty, dateStatistics=Optional.empty,
booleanStatistics=Optional.empty, maxValueSizeInBytes=OptionalLong.empty,
totalSizeInBytes=OptionalLong.empty, nullsCount=OptionalLong[0],
distinctValuesCount=OptionalLong[1]}.
{code}
Duplicate records in the *_TAB_COL_STATS_*
{code:java}
'7','default','dual','dummy','smallint','245671','1','1',NULL,NULL,NULL,NULL,'0','1',NULL,NULL,NULL,NULL,'1588345509'
'11','default','dual','dummy','smallint','245671','1','1',NULL,NULL,NULL,NULL,'0','1',NULL,NULL,NULL,NULL,'1588345509'{code}
was:
When `analyze ` command was executed from presto to update the stats of
a table for the first time from multiple cluster sharing the same Hive
metastore. Duplicate entry for the same table is inserted to the
`TAB_COL_STATS` table.
This lead to failure executing further `analyze ` commands.
{code:java}
Query failed: Multiple entries with same key:
dummy=HiveColumnStatistics{integerStatistics=Optional[IntegerStatistics{min=OptionalLong[1],
max=OptionalLong[1]}], doubleStatistics=Optional.empty,
decimalStatistics=Optional.empty, dateStatistics=Optional.empty,
booleanStatistics=Optional.empty, maxValueSizeInBytes=OptionalLong.empty,
totalSizeInBytes=OptionalLong.empty, nullsCount=OptionalLong[0],
distinctValuesCount=OptionalLong[1]} and
dummy=HiveColumnStatistics{integerStatistics=Optional[IntegerStatistics{min=OptionalLong[1],
max=OptionalLong[1]}], doubleStatistics=Optional.empty,
decimalStatistics=Optional.empty, dateStatistics=Optional.empty,
booleanStatistics=Optional.empty, maxValueSizeInBytes=OptionalLong.empty,
totalSizeInBytes=OptionalLong.empty, nullsCount=OptionalLong[0],
distinctValuesCount=OptionalLong[1]}.
{code}
Duplicate records in the `TAB_COL_STATS`
{code:java}
'7','default','dual','dummy','smallint','245671','1','1',NULL,NULL,NULL,NULL,'0','1',NULL,NULL,NULL,NULL,'1588345509'
'11','default','dual','dummy','smallint','245671','1','1',NULL,NULL,NULL,NULL,'0','1',NULL,NULL,NULL,NULL,'1588345509'{code}
> Duplicate entry for a table in TAB_COL_STATS
> -
>
> Key: HIVE-23390
> URL: https://issues.apache.org/jira/browse/HIVE-23390
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.3.4
>Reporter: Mithun Antony
>Priority: Major
>
> When *_analyze _* command was executed from presto to update the stats
> of a table for the first time from multiple cluster sharing the same Hive
> metastore. Duplicate entry for the same table is inserted to the
> *_TAB_COL_STATS_* table.
> This lead to failure executing further *_analyze _* commands.
> {code:java}
> Query failed: Multiple entries with same key:
> dummy=HiveColumnStatistics{integerStatistics=Optional[IntegerStatistics{min=OptionalLong[1],
> max=OptionalLong[1]}], doubleStatistics=Optional.empty,
> decimalStatistics=Optional.empty, dateStatistics=Optional.empty,
> booleanStatistics=Optional.empty, maxValueSizeInBytes=OptionalLong.empty,
> totalSizeInBytes=OptionalLong.empty, nullsCount=OptionalLong[0],
> distinctValuesCount=OptionalLong[1]} and
> dummy=HiveColumnStatistics{integerStatistics=Optional[IntegerStatistics{min=OptionalLong[1],
> max=OptionalLong[1]}], doubleStatistics=Optional.empty,
> decimalStatistics=Optional.empty, dateStatistics=Optional.empty,
> booleanStatistics=Optional.empty, maxValueSizeInBytes=OptionalLong.empty,
> totalSizeInBytes=OptionalLong.empty, nullsCount=OptionalLong[0],
> distinctValuesCount=OptionalLong[1]}.
> {code}
> Duplicate records in the *_TAB_COL_STATS_*
> {code:java}
> '7','default','dual','dummy','smallint','245671','1','1',NULL,NULL,NULL,NULL,'0','1',NULL,NULL,NULL,NULL,'1588345509'
>
>
[jira] [Updated] (HIVE-23390) Duplicate entry for a table in TAB_COL_STATS
[
https://issues.apache.org/jira/browse/HIVE-23390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Antony updated HIVE-23390:
-
Description:
When `analyze ` command was executed from presto to update the stats of
a table for the first time from multiple cluster sharing the same Hive
metastore. Duplicate entry for the same table is inserted to the
`TAB_COL_STATS` table.
This lead to failure executing further `analyze ` commands.
{code:java}
Query failed: Multiple entries with same key:
dummy=HiveColumnStatistics{integerStatistics=Optional[IntegerStatistics{min=OptionalLong[1],
max=OptionalLong[1]}], doubleStatistics=Optional.empty,
decimalStatistics=Optional.empty, dateStatistics=Optional.empty,
booleanStatistics=Optional.empty, maxValueSizeInBytes=OptionalLong.empty,
totalSizeInBytes=OptionalLong.empty, nullsCount=OptionalLong[0],
distinctValuesCount=OptionalLong[1]} and
dummy=HiveColumnStatistics{integerStatistics=Optional[IntegerStatistics{min=OptionalLong[1],
max=OptionalLong[1]}], doubleStatistics=Optional.empty,
decimalStatistics=Optional.empty, dateStatistics=Optional.empty,
booleanStatistics=Optional.empty, maxValueSizeInBytes=OptionalLong.empty,
totalSizeInBytes=OptionalLong.empty, nullsCount=OptionalLong[0],
distinctValuesCount=OptionalLong[1]}.
{code}
Duplicate records in the `TAB_COL_STATS`
{code:java}
'7','default','dual','dummy','smallint','245671','1','1',NULL,NULL,NULL,NULL,'0','1',NULL,NULL,NULL,NULL,'1588345509'
'11','default','dual','dummy','smallint','245671','1','1',NULL,NULL,NULL,NULL,'0','1',NULL,NULL,NULL,NULL,'1588345509'{code}
was:
When `analyze ` command was executed from presto to update the stats of
a table for the first time from multiple cluster sharing the same Hive
metastore. Duplicate entry for the same table is inserted to the
`TAB_COL_STATS` table.
This lead to failure executing further `analyze ` commands.
Query failed: Multiple entries with same key:
dummy=HiveColumnStatistics\{integerStatistics=Optional[IntegerStatistics{min=OptionalLong[1],
max=OptionalLong[1]}], doubleStatistics=Optional.empty,
decimalStatistics=Optional.empty, dateStatistics=Optional.empty,
booleanStatistics=Optional.empty, maxValueSizeInBytes=OptionalLong.empty,
totalSizeInBytes=OptionalLong.empty, nullsCount=OptionalLong[0],
distinctValuesCount=OptionalLong[1]} and
dummy=HiveColumnStatistics\{integerStatistics=Optional[IntegerStatistics{min=OptionalLong[1],
max=OptionalLong[1]}], doubleStatistics=Optional.empty,
decimalStatistics=Optional.empty, dateStatistics=Optional.empty,
booleanStatistics=Optional.empty, maxValueSizeInBytes=OptionalLong.empty,
totalSizeInBytes=OptionalLong.empty, nullsCount=OptionalLong[0],
distinctValuesCount=OptionalLong[1]}.
Duplicate records in the `TAB_COL_STATS`
'7','default','dual','dummy','smallint','245671','1','1',NULL,NULL,NULL,NULL,'0','1',NULL,NULL,NULL,NULL,'1588345509',
'11','default','dual','dummy','smallint','245671','1','1',NULL,NULL,NULL,NULL,'0','1',NULL,NULL,NULL,NULL,'1588345509'.
> Duplicate entry for a table in TAB_COL_STATS
> -
>
> Key: HIVE-23390
> URL: https://issues.apache.org/jira/browse/HIVE-23390
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.3.4
>Reporter: Mithun Antony
>Priority: Major
>
> When `analyze ` command was executed from presto to update the stats
> of a table for the first time from multiple cluster sharing the same Hive
> metastore. Duplicate entry for the same table is inserted to the
> `TAB_COL_STATS` table.
> This lead to failure executing further `analyze ` commands.
> {code:java}
> Query failed: Multiple entries with same key:
> dummy=HiveColumnStatistics{integerStatistics=Optional[IntegerStatistics{min=OptionalLong[1],
> max=OptionalLong[1]}], doubleStatistics=Optional.empty,
> decimalStatistics=Optional.empty, dateStatistics=Optional.empty,
> booleanStatistics=Optional.empty, maxValueSizeInBytes=OptionalLong.empty,
> totalSizeInBytes=OptionalLong.empty, nullsCount=OptionalLong[0],
> distinctValuesCount=OptionalLong[1]} and
> dummy=HiveColumnStatistics{integerStatistics=Optional[IntegerStatistics{min=OptionalLong[1],
> max=OptionalLong[1]}], doubleStatistics=Optional.empty,
> decimalStatistics=Optional.empty, dateStatistics=Optional.empty,
> booleanStatistics=Optional.empty, maxValueSizeInBytes=OptionalLong.empty,
> totalSizeInBytes=OptionalLong.empty, nullsCount=OptionalLong[0],
> distinctValuesCount=OptionalLong[1]}.
> {code}
> Duplicate records in the `TAB_COL_STATS`
> {code:java}
> '7','default','dual','dummy','smallint','245671','1','1',NULL,NULL,NULL,NULL,'0','1',NULL,NULL,NULL,NULL,'1588345509'
>
> '11','default','dual','dummy','smallint','245671','1','1',NULL,NULL,NULL,NULL,'0','1',NULL,NULL,NULL,NULL,'1588345509'{code}
[jira] [Commented] (HIVE-23359) "show tables like" support for SQL wildcard characters (% and _)
[
https://issues.apache.org/jira/browse/HIVE-23359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101367#comment-17101367
]
Hive QA commented on HIVE-23359:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
34s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22172/dev-support/hive-personality.sh
|
| git revision | master / 54b8799 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22172/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> "show tables like" support for SQL wildcard characters (% and _)
>
>
> Key: HIVE-23359
> URL: https://issues.apache.org/jira/browse/HIVE-23359
> Project: Hive
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 2.3.7
>Reporter: Yuming Wang
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-23359.01.patch, HIVE-23359.02.patch,
> HIVE-23359.03.patch, HIVE-23359.04.patch, HIVE-23359.05.patch
>
>
> https://docs.snowflake.com/en/sql-reference/sql/show-tables.html
> https://clickhouse.tech/docs/en/sql-reference/statements/show/
> https://www.mysqltutorial.org/mysql-show-tables/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23301) Optimising privilege synchroniser: UDF for updating privileges
[
https://issues.apache.org/jira/browse/HIVE-23301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101356#comment-17101356
]
Hive QA commented on HIVE-23301:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002219/hive-23301.6.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17234 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=169)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22171/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22171/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22171/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002219 - PreCommit-HIVE-Build
> Optimising privilege synchroniser: UDF for updating privileges
> --
>
> Key: HIVE-23301
> URL: https://issues.apache.org/jira/browse/HIVE-23301
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, UDF
>Affects Versions: 3.1.1
>Reporter: Simhadri G
>Assignee: Simhadri G
>Priority: Minor
> Attachments: UDFSplitMapPrivs.patch, UDFSplitMapPrivs.patch,
> hive-23301-2.patch, hive-23301.3.patch, hive-23301.4.patch,
> hive-23301.5.patch, hive-23301.6.patch
>
>
> Privilege synchronizer pulls the list of databases, tables and columns from
> the Hive Metastore. For each of these objects it fetches the privilege
> information and invokes HMS API to refresh the privilege information in HMS.
> The current UDF Maps a bit string to a privilege based on if the privilege
> is granted or not.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23301) Optimising privilege synchroniser: UDF for updating privileges
[
https://issues.apache.org/jira/browse/HIVE-23301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101344#comment-17101344
]
Hive QA commented on HIVE-23301:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
2s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
48s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 2 new + 82 unchanged - 0 fixed
= 84 total (was 82) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22171/dev-support/hive-personality.sh
|
| git revision | master / 54b8799 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22171/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22171/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Optimising privilege synchroniser: UDF for updating privileges
> --
>
> Key: HIVE-23301
> URL: https://issues.apache.org/jira/browse/HIVE-23301
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, UDF
>Affects Versions: 3.1.1
>Reporter: Simhadri G
>Assignee: Simhadri G
>Priority: Minor
> Attachments: UDFSplitMapPrivs.patch, UDFSplitMapPrivs.patch,
> hive-23301-2.patch, hive-23301.3.patch, hive-23301.4.patch,
> hive-23301.5.patch, hive-23301.6.patch
>
>
> Privilege synchronizer pulls the list of databases, tables and columns from
> the Hive Metastore. For each of these objects it fetches the privilege
> information and invokes HMS API to refresh the privilege information in HMS.
> The current UDF Maps a bit string to a privilege based on if the privilege
> is granted or not.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23266) Remove QueryWrapper from ObjectStore
[
https://issues.apache.org/jira/browse/HIVE-23266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101324#comment-17101324
]
Hive QA commented on HIVE-23266:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002212/HIVE-23266.11.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17232 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22170/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22170/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22170/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002212 - PreCommit-HIVE-Build
> Remove QueryWrapper from ObjectStore
>
>
> Key: HIVE-23266
> URL: https://issues.apache.org/jira/browse/HIVE-23266
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23266.1.patch, HIVE-23266.10.patch,
> HIVE-23266.11.patch, HIVE-23266.2.patch, HIVE-23266.2.patch,
> HIVE-23266.3.patch, HIVE-23266.4.patch, HIVE-23266.5.patch,
> HIVE-23266.6.patch, HIVE-23266.7.patch, HIVE-23266.8.patch, HIVE-23266.9.patch
>
>
> There is currently a utility called {{QueryWrapper}} that makes a normal
> {{Query}} auto-closable. However, {{Query}} is now in fact already
> auto-closing, so there is no need for this class. In trying to remove it, I
> realized that this wrapper was being passed around in pretty convoluted ways
> and also it was sometimes being created in a {{try-with-resources}} block but
> then never actually used in any way.
> Remove the {{QueryWrapper}} from the class and simplify some of the DB
> interactions.
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L178
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[ https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101322#comment-17101322 ] Naresh P R commented on HIVE-23354: --- We are facing this issue as well. Should we check task attempt status before comparing the file size ? Does it matter to compare failed attempt vs succeeded attempt file size ? Can we use task counters to validate the output records in case of speculative execution ? > Remove file size sanity checking from compareTempOrDuplicateFiles > - > > Key: HIVE-23354 > URL: https://issues.apache.org/jira/browse/HIVE-23354 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-23354.1.patch > > > [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010] > compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to > determine which file(s) to keep. > I've seen instances where this function throws an exception due to the fact > that the newer attemptId file size is less than the older attemptId (thus > failing the query). > I think this assumption is faulty, due to various factors such as file > compression and the order in which values are written. It may be prudent to > trust that the newest attemptId is in fact the best choice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23266) Remove QueryWrapper from ObjectStore
[
https://issues.apache.org/jira/browse/HIVE-23266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101309#comment-17101309
]
Hive QA commented on HIVE-23266:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
15s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
20s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 29 new + 285 unchanged - 28 fixed = 314 total (was 313) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
22s{color} | {color:red} standalone-metastore/metastore-server generated 2 new
+ 186 unchanged - 1 fixed = 188 total (was 187) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 58s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | Impossible cast from String to
org.apache.hadoop.hive.metastore.model.MPartition in
org.apache.hadoop.hive.metastore.ObjectStore.listPartitionsPsWithAuth(String,
String, String, List, short, String, List) At
ObjectStore.java:org.apache.hadoop.hive.metastore.model.MPartition in
org.apache.hadoop.hive.metastore.ObjectStore.listPartitionsPsWithAuth(String,
String, String, List, short, String, List) At ObjectStore.java:[line 3306] |
| |
org.apache.hadoop.hive.metastore.ObjectStore.getMPartitionColumnStatistics(Table,
List, List, String) concatenates strings using + in a loop At
ObjectStore.java:strings using + in a loop At ObjectStore.java:[line 9312] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22170/dev-support/hive-personality.sh
|
| git revision | master / 54b8799 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22170/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22170/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22170/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove QueryWrapper from ObjectStore
>
>
> Key: HIVE-23266
> URL: https://issues.apache.org/jira/browse/HIVE-23266
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23266.1.patch,
[jira] [Commented] (HIVE-23325) Consolidate acid-related cleanup tasks
[
https://issues.apache.org/jira/browse/HIVE-23325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101301#comment-17101301
]
Hive QA commented on HIVE-23325:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002208/HIVE-23325.8.patch
{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17236 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22169/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22169/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22169/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002208 - PreCommit-HIVE-Build
> Consolidate acid-related cleanup tasks
> --
>
> Key: HIVE-23325
> URL: https://issues.apache.org/jira/browse/HIVE-23325
> Project: Hive
> Issue Type: Improvement
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Attachments: HIVE-23325.1.patch, HIVE-23325.2.patch,
> HIVE-23325.2.patch, HIVE-23325.2.patch, HIVE-23325.3.patch,
> HIVE-23325.4.patch, HIVE-23325.5.patch, HIVE-23325.5.patch,
> HIVE-23325.6.patch, HIVE-23325.6.patch, HIVE-23325.7.patch,
> HIVE-23325.8.patch, HIVE-23325.8.patch
>
>
> Currently there are multiple ways to clean up ACID related stuff:
> * AcidHouseKeeperService
> * AcidWriteSetService
> * AcidCompactionHistoryService
> * Initiator
> * Cleaner
> * etc.
> We should consolidate them where possible and improve logging.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22633) GroupByOperator may throw NullPointerException when setting data skew optimization parameters
[
https://issues.apache.org/jira/browse/HIVE-22633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101300#comment-17101300
]
Renukaprasad C commented on HIVE-22633:
---
[~zhangbutao] are you working on this issue?
> GroupByOperator may throw NullPointerException when setting data skew
> optimization parameters
> -
>
> Key: HIVE-22633
> URL: https://issues.apache.org/jira/browse/HIVE-22633
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0, 4.0.0, 3.1.1
>Reporter: zhangbutao
>Priority: Major
>
> if hive.map.aggr and hive.groupby.skewindata set true，exception will be
> thrown.
> step to repro:
> 1. create table:
> set hive.map.aggr=true;
> set hive.groupby.skewindata=true;
> create table test1 (id1 bigint);
> create table test2 (id2 bigint) partitioned by(dt2 string);
> insert into test2 partition(dt2='2020') select a.id1 from test1 a group by
> a.id1;
> 2.NullPointerException:
> {code:java}
> ], TaskAttempt 2 failed, info=[Error: Error while running task ( failure ) :
> attempt_1585641455670_0001_2_03_00_2:java.lang.RuntimeException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFComputeStats$GenericUDAFNumericStatsEvaluator.init(GenericUDAFComputeStats.java:373)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.initializeOp(GroupByOperator.java:373)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:360)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:191)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23325) Consolidate acid-related cleanup tasks
[
https://issues.apache.org/jira/browse/HIVE-23325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101297#comment-17101297
]
Hive QA commented on HIVE-23325:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
39s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
18s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
40s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch metastore-common passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} standalone-metastore/metastore-server: The patch
generated 0 new + 895 unchanged - 3 fixed = 895 total (was 898) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} ql: The patch generated 0 new + 716 unchanged - 4
fixed = 716 total (was 720) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22169/dev-support/hive-personality.sh
|
| git revision | master / 54b8799 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-common common
standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22169/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Consolidate acid-related cleanup tasks
> --
>
> Key: HIVE-23325
> URL: https://issues.apache.org/jira/browse/HIVE-23325
[jira] [Commented] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[ https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101295#comment-17101295 ] John Sherman commented on HIVE-23354: - [~kuczoram] So it would be my understanding that the current code also doesn't work with speculative execution. Is that a correct reading of the situation? Since it relies on attemptIds but will throw an error if the file size of the newest attempt not >= than the older attempts. I just want to make sure I'm not missing a nuance of the existing code path. > Remove file size sanity checking from compareTempOrDuplicateFiles > - > > Key: HIVE-23354 > URL: https://issues.apache.org/jira/browse/HIVE-23354 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-23354.1.patch > > > [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010] > compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to > determine which file(s) to keep. > I've seen instances where this function throws an exception due to the fact > that the newer attemptId file size is less than the older attemptId (thus > failing the query). > I think this assumption is faulty, due to various factors such as file > compression and the order in which values are written. It may be prudent to > trust that the newest attemptId is in fact the best choice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23337) Move q tests to TestMiniLlapLocal from TestCliDriver where the output is different, batch 2
[ https://issues.apache.org/jira/browse/HIVE-23337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23337: -- Attachment: HIVE-23337.04.patch > Move q tests to TestMiniLlapLocal from TestCliDriver where the output is > different, batch 2 > --- > > Key: HIVE-23337 > URL: https://issues.apache.org/jira/browse/HIVE-23337 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23337.01.patch, HIVE-23337.02.patch, > HIVE-23337.03.patch, HIVE-23337.04.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23371) StatsUtils.getConstValue may log misleading exception
[
https://issues.apache.org/jira/browse/HIVE-23371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101284#comment-17101284
]
Hive QA commented on HIVE-23371:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002203/HIVE-23371.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 134 failed/errored test(s), 17226 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[interval_arithmetic]
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_query_oneskew_2]
(batchId=2)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[order_by_expr_1]
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[semijoin4] (batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stat_estimate_drill]
(batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_notexists]
(batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_notexists_having]
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_notin_having]
(batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_unqualcolumnrefs]
(batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[timestamp] (batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[timestamp_literal]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[timestamptz] (batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf5] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf9] (batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_aes_decrypt]
(batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_aes_encrypt]
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_isnull_isnotnull]
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_like] (batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_struct] (batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[unionall_unbalancedppd]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_multi_value]
(batchId=28)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_expressions]
(batchId=127)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters1]
(batchId=48)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters]
(batchId=51)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=46)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
(batchId=49)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_view_delete]
(batchId=76)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[approx_distinct]
(batchId=79)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_const]
(batchId=65)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit]
(batchId=120)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_windowing_2]
(batchId=72)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_subq_exists]
(batchId=111)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_subq_not_in]
(batchId=90)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=105)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[constGby]
(batchId=58)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[constant_prop_3]
(batchId=84)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[constprog_semijoin]
(batchId=100)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[constprog_type]
(batchId=53)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[constprog_when_case]
(batchId=96)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[ctas_date]
(batchId=53)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[current_date_timestamp]
(batchId=86)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[date_1]
(batchId=113)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[date_3]
(batchId=90)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[date_4]
(batchId=88)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[date_timestamp_prec]
(batchId=120)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[date_udf]
(batchId=75)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=80)
[jira] [Commented] (HIVE-23371) StatsUtils.getConstValue may log misleading exception
[
https://issues.apache.org/jira/browse/HIVE-23371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101273#comment-17101273
]
Hive QA commented on HIVE-23371:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
52s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22168/dev-support/hive-personality.sh
|
| git revision | master / 54b8799 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22168/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> StatsUtils.getConstValue may log misleading exception
> -
>
> Key: HIVE-23371
> URL: https://issues.apache.org/jira/browse/HIVE-23371
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23371.01.patch, HIVE-23371.02.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> it logs exceptions like:
> {code}
> 2020-05-05T01:51:59,609 DEBUG [cba65ded-3bb5-4295-9b3f-cdf844a8faec main]
> stats.StatsUtils: Interpreting constant (2008-04-08) resulted in exception
> java.lang.NumberFormatException: For input string: "2008-04-08"
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
> ~[?:1.8.0_252]
> at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
> ~[?:1.8.0_252]
> at java.lang.Double.parseDouble(Double.java:538) ~[?:1.8.0_252]
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getConstValue(StatsUtils.java:1653)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.buildColStatForConstant(StatsUtils.java:1632)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getColStatisticsFromExpression(StatsUtils.java:1538)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> {code}
> however...these exceptions are not that important; might just draw attention
> away from important ones
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23318) TxnHandler should delete from MATERIALIZATION_REBUILD_LOCKS on need basis
[
https://issues.apache.org/jira/browse/HIVE-23318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-23318:
---
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master, thanks [~Marton Bod]!
> TxnHandler should delete from MATERIALIZATION_REBUILD_LOCKS on need basis
> -
>
> Key: HIVE-23318
> URL: https://issues.apache.org/jira/browse/HIVE-23318
> Project: Hive
> Issue Type: Improvement
> Components: Hive, Materialized views
>Reporter: Rajesh Balamohan
>Assignee: Marton Bod
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-23318.1.patch, HIVE-23318.2.patch,
> HIVE-23318.3.patch, HIVE-23318.4.patch
>
>
> Observed the following queries when materialized view or any of its feature
> was not used.
> TxnHandler need not clear this part of txn commit. It would help in reducing
> the sql parsing time in server side as well.
> {noformat}
> Gdelete from MATERIALIZATION_REBUILD_LOCKS where mrl_txn_id = 120398082
> Gdelete from MATERIALIZATION_REBUILD_LOCKS where mrl_txn_id = 120398084
> Gdelete from MATERIALIZATION_REBUILD_LOCKS where mrl_txn_id = 120398112
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23318) TxnHandler should delete from MATERIALIZATION_REBUILD_LOCKS on need basis
[
https://issues.apache.org/jira/browse/HIVE-23318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-23318:
---
Summary: TxnHandler should delete from MATERIALIZATION_REBUILD_LOCKS on
need basis (was: TxnHandler need not delete from MATERIALIZATION_REBUILD_LOCKS
on need basis)
> TxnHandler should delete from MATERIALIZATION_REBUILD_LOCKS on need basis
> -
>
> Key: HIVE-23318
> URL: https://issues.apache.org/jira/browse/HIVE-23318
> Project: Hive
> Issue Type: Improvement
> Components: Hive, Materialized views
>Reporter: Rajesh Balamohan
>Assignee: Marton Bod
>Priority: Minor
> Attachments: HIVE-23318.1.patch, HIVE-23318.2.patch,
> HIVE-23318.3.patch, HIVE-23318.4.patch
>
>
> Observed the following queries when materialized view or any of its feature
> was not used.
> TxnHandler need not clear this part of txn commit. It would help in reducing
> the sql parsing time in server side as well.
> {noformat}
> Gdelete from MATERIALIZATION_REBUILD_LOCKS where mrl_txn_id = 120398082
> Gdelete from MATERIALIZATION_REBUILD_LOCKS where mrl_txn_id = 120398084
> Gdelete from MATERIALIZATION_REBUILD_LOCKS where mrl_txn_id = 120398112
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23338) Bump jackson-databind version up to 2.9.10.4
[
https://issues.apache.org/jira/browse/HIVE-23338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101262#comment-17101262
]
Hive QA commented on HIVE-23338:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002198/HIVE-23338.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17226 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22167/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22167/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22167/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002198 - PreCommit-HIVE-Build
> Bump jackson-databind version up to 2.9.10.4
>
>
> Key: HIVE-23338
> URL: https://issues.apache.org/jira/browse/HIVE-23338
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23338.01.patch, HIVE-23338.01.patch,
> HIVE-23338.01.patch, HIVE-23338.01.patch, HIVE-23338.02.patch
>
>
> com.fasterxml.jackson.core:jackson-databind:2.9.9 is exploitable.
> And exclude a transitive dependency on
> com.fasterxml.jackson.core:jackson-databind:2.6.5, which is also exploitable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23318) TxnHandler need not delete from MATERIALIZATION_REBUILD_LOCKS on need basis
[
https://issues.apache.org/jira/browse/HIVE-23318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101259#comment-17101259
]
Jesus Camacho Rodriguez commented on HIVE-23318:
+1
> TxnHandler need not delete from MATERIALIZATION_REBUILD_LOCKS on need basis
> ---
>
> Key: HIVE-23318
> URL: https://issues.apache.org/jira/browse/HIVE-23318
> Project: Hive
> Issue Type: Improvement
> Components: Hive, Materialized views
>Reporter: Rajesh Balamohan
>Assignee: Marton Bod
>Priority: Minor
> Attachments: HIVE-23318.1.patch, HIVE-23318.2.patch,
> HIVE-23318.3.patch, HIVE-23318.4.patch
>
>
> Observed the following queries when materialized view or any of its feature
> was not used.
> TxnHandler need not clear this part of txn commit. It would help in reducing
> the sql parsing time in server side as well.
> {noformat}
> Gdelete from MATERIALIZATION_REBUILD_LOCKS where mrl_txn_id = 120398082
> Gdelete from MATERIALIZATION_REBUILD_LOCKS where mrl_txn_id = 120398084
> Gdelete from MATERIALIZATION_REBUILD_LOCKS where mrl_txn_id = 120398112
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23389) FilterMergeRule can lead to AssertionError
[
https://issues.apache.org/jira/browse/HIVE-23389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-23389:
---
Attachment: HIVE-23389.patch
> FilterMergeRule can lead to AssertionError
> --
>
> Key: HIVE-23389
> URL: https://issues.apache.org/jira/browse/HIVE-23389
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-23389.patch
>
>
> I have not been able to reproduce the issue in latest master. However, this
> could potentially happen since Filter creation has a check on whether the
> expression is flat
> ([here|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/core/Filter.java#L74])
> and Filter merge does not flatten an expression when it is created.
> {noformat}
> java.lang.AssertionError: AND(=($3, 100), OR(OR(null, IS NOT
> NULL(CAST(100):INTEGER)), =(CAST(100):INTEGER, CAST(200):INTEGER)))
> at org.apache.calcite.rel.core.Filter.(Filter.java:74)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFilter.(HiveFilter.java:39)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories$HiveFilterFactoryImpl.createFilter(HiveRelFactories.java:126)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelBuilder.filter(HiveRelBuilder.java:99)
> at org.apache.calcite.tools.RelBuilder.filter(RelBuilder.java:1055)
> at
> org.apache.calcite.rel.rules.FilterMergeRule.onMatch(FilterMergeRule.java:81)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23389) FilterMergeRule can lead to AssertionError
[
https://issues.apache.org/jira/browse/HIVE-23389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-23389:
---
Status: Patch Available (was: In Progress)
> FilterMergeRule can lead to AssertionError
> --
>
> Key: HIVE-23389
> URL: https://issues.apache.org/jira/browse/HIVE-23389
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> I have not been able to reproduce the issue in latest master. However, this
> could potentially happen since Filter creation has a check on whether the
> expression is flat
> ([here|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/core/Filter.java#L74])
> and Filter merge does not flatten an expression when it is created.
> {noformat}
> java.lang.AssertionError: AND(=($3, 100), OR(OR(null, IS NOT
> NULL(CAST(100):INTEGER)), =(CAST(100):INTEGER, CAST(200):INTEGER)))
> at org.apache.calcite.rel.core.Filter.(Filter.java:74)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFilter.(HiveFilter.java:39)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories$HiveFilterFactoryImpl.createFilter(HiveRelFactories.java:126)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelBuilder.filter(HiveRelBuilder.java:99)
> at org.apache.calcite.tools.RelBuilder.filter(RelBuilder.java:1055)
> at
> org.apache.calcite.rel.rules.FilterMergeRule.onMatch(FilterMergeRule.java:81)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HIVE-23389) FilterMergeRule can lead to AssertionError
[
https://issues.apache.org/jira/browse/HIVE-23389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-23389 started by Jesus Camacho Rodriguez.
--
> FilterMergeRule can lead to AssertionError
> --
>
> Key: HIVE-23389
> URL: https://issues.apache.org/jira/browse/HIVE-23389
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> I have not been able to reproduce the issue in latest master. However, this
> could potentially happen since Filter creation has a check on whether the
> expression is flat
> ([here|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/core/Filter.java#L74])
> and Filter merge does not flatten an expression when it is created.
> {noformat}
> java.lang.AssertionError: AND(=($3, 100), OR(OR(null, IS NOT
> NULL(CAST(100):INTEGER)), =(CAST(100):INTEGER, CAST(200):INTEGER)))
> at org.apache.calcite.rel.core.Filter.(Filter.java:74)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFilter.(HiveFilter.java:39)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories$HiveFilterFactoryImpl.createFilter(HiveRelFactories.java:126)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelBuilder.filter(HiveRelBuilder.java:99)
> at org.apache.calcite.tools.RelBuilder.filter(RelBuilder.java:1055)
> at
> org.apache.calcite.rel.rules.FilterMergeRule.onMatch(FilterMergeRule.java:81)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23389) FilterMergeRule can lead to AssertionError
[
https://issues.apache.org/jira/browse/HIVE-23389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez reassigned HIVE-23389:
--
> FilterMergeRule can lead to AssertionError
> --
>
> Key: HIVE-23389
> URL: https://issues.apache.org/jira/browse/HIVE-23389
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> I have not been able in latest master but this could potentially happens
> since Filter creation as a check on whether the expression is flat
> ([here|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/core/Filter.java#L74])
> and Filter merge does not flatten an expression when it is created.
> {noformat}
> java.lang.AssertionError: AND(=($3, 100), OR(OR(null, IS NOT
> NULL(CAST(100):INTEGER)), =(CAST(100):INTEGER, CAST(200):INTEGER)))
> at org.apache.calcite.rel.core.Filter.(Filter.java:74)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFilter.(HiveFilter.java:39)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories$HiveFilterFactoryImpl.createFilter(HiveRelFactories.java:126)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelBuilder.filter(HiveRelBuilder.java:99)
> at org.apache.calcite.tools.RelBuilder.filter(RelBuilder.java:1055)
> at
> org.apache.calcite.rel.rules.FilterMergeRule.onMatch(FilterMergeRule.java:81)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23389) FilterMergeRule can lead to AssertionError
[
https://issues.apache.org/jira/browse/HIVE-23389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-23389:
---
Description:
I have not been able to reproduce the issue in latest master. However, this
could potentially happen since Filter creation has a check on whether the
expression is flat
([here|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/core/Filter.java#L74])
and Filter merge does not flatten an expression when it is created.
{noformat}
java.lang.AssertionError: AND(=($3, 100), OR(OR(null, IS NOT
NULL(CAST(100):INTEGER)), =(CAST(100):INTEGER, CAST(200):INTEGER)))
at org.apache.calcite.rel.core.Filter.(Filter.java:74)
at
org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFilter.(HiveFilter.java:39)
at
org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories$HiveFilterFactoryImpl.createFilter(HiveRelFactories.java:126)
at
org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelBuilder.filter(HiveRelBuilder.java:99)
at org.apache.calcite.tools.RelBuilder.filter(RelBuilder.java:1055)
at
org.apache.calcite.rel.rules.FilterMergeRule.onMatch(FilterMergeRule.java:81)
{noformat}
was:
I have not been able in latest master but this could potentially happens since
Filter creation as a check on whether the expression is flat
([here|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/core/Filter.java#L74])
and Filter merge does not flatten an expression when it is created.
{noformat}
java.lang.AssertionError: AND(=($3, 100), OR(OR(null, IS NOT
NULL(CAST(100):INTEGER)), =(CAST(100):INTEGER, CAST(200):INTEGER)))
at org.apache.calcite.rel.core.Filter.(Filter.java:74)
at
org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFilter.(HiveFilter.java:39)
at
org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories$HiveFilterFactoryImpl.createFilter(HiveRelFactories.java:126)
at
org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelBuilder.filter(HiveRelBuilder.java:99)
at org.apache.calcite.tools.RelBuilder.filter(RelBuilder.java:1055)
at
org.apache.calcite.rel.rules.FilterMergeRule.onMatch(FilterMergeRule.java:81)
{noformat}
> FilterMergeRule can lead to AssertionError
> --
>
> Key: HIVE-23389
> URL: https://issues.apache.org/jira/browse/HIVE-23389
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> I have not been able to reproduce the issue in latest master. However, this
> could potentially happen since Filter creation has a check on whether the
> expression is flat
> ([here|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/core/Filter.java#L74])
> and Filter merge does not flatten an expression when it is created.
> {noformat}
> java.lang.AssertionError: AND(=($3, 100), OR(OR(null, IS NOT
> NULL(CAST(100):INTEGER)), =(CAST(100):INTEGER, CAST(200):INTEGER)))
> at org.apache.calcite.rel.core.Filter.(Filter.java:74)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFilter.(HiveFilter.java:39)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories$HiveFilterFactoryImpl.createFilter(HiveRelFactories.java:126)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelBuilder.filter(HiveRelBuilder.java:99)
> at org.apache.calcite.tools.RelBuilder.filter(RelBuilder.java:1055)
> at
> org.apache.calcite.rel.rules.FilterMergeRule.onMatch(FilterMergeRule.java:81)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23338) Bump jackson-databind version up to 2.9.10.4
[
https://issues.apache.org/jira/browse/HIVE-23338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101241#comment-17101241
]
Hive QA commented on HIVE-23338:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
38s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22167/dev-support/hive-personality.sh
|
| git revision | master / c3007d4 |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22167/yetus/patch-asflicense-problems.txt
|
| modules | C: standalone-metastore upgrade-acid/pre-upgrade testutils/ptest2
U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22167/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Bump jackson-databind version up to 2.9.10.4
>
>
> Key: HIVE-23338
> URL: https://issues.apache.org/jira/browse/HIVE-23338
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23338.01.patch, HIVE-23338.01.patch,
> HIVE-23338.01.patch, HIVE-23338.01.patch, HIVE-23338.02.patch
>
>
> com.fasterxml.jackson.core:jackson-databind:2.9.9 is exploitable.
> And exclude a transitive dependency on
> com.fasterxml.jackson.core:jackson-databind:2.6.5, which is also exploitable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23388) CTAS queries should use target's location for staging.
[ https://issues.apache.org/jira/browse/HIVE-23388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam reassigned HIVE-23388: > CTAS queries should use target's location for staging. > -- > > Key: HIVE-23388 > URL: https://issues.apache.org/jira/browse/HIVE-23388 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > > In cloud based storage systems, renaming files across different root level > buckets seem to be disallowed. The S3AFileSystem throws the following > exception. This appears to be bug in S3FS impl. > Failed with exception Wrong FS > s3a://hive-managed/clusters/env-x/warehouse--/warehouse/tablespace/managed/hive/tpch.db/customer/delta_001_001_ > -expected s3a://hive-external > 2020-04-27T19:34:27,573 INFO [Thread-6] jdbc.TestDriver: > java.lang.IllegalArgumentException: Wrong FS > s3a://hive-managed//clusters/env-/warehouse--/warehouse/tablespace/managed/hive/tpch.db/customer/delta_001_001_ > -expected s3a://hive-external > But we should fix our query plans to use the target table's directory for > staging as well. That should resolve this issue and it is the right thing to > do as well (in case there are different encryption zones/keys for these > buckets). > Fix in HIVE-22995 probably changed this behavior. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23337) Move q tests to TestMiniLlapLocal from TestCliDriver where the output is different, batch 2
[
https://issues.apache.org/jira/browse/HIVE-23337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101225#comment-17101225
]
Hive QA commented on HIVE-23337:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002200/HIVE-23337.03.patch
{color:green}SUCCESS:{color} +1 due to 17 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 17188 tests
executed
*Failed tests:*
{noformat}
TestHiveKVResultCache - did not produce a TEST-*.xml file (likely timed out)
(batchId=289)
TestHiveSparkClient - did not produce a TEST-*.xml file (likely timed out)
(batchId=289)
TestLocalHiveSparkClient - did not produce a TEST-*.xml file (likely timed out)
(batchId=289)
TestSmallTableCache - did not produce a TEST-*.xml file (likely timed out)
(batchId=289)
TestSparkInvalidFileFormat - did not produce a TEST-*.xml file (likely timed
out) (batchId=289)
TestSparkJobMonitor - did not produce a TEST-*.xml file (likely timed out)
(batchId=289)
TestSparkPlan - did not produce a TEST-*.xml file (likely timed out)
(batchId=289)
TestSparkSessionManagerImpl - did not produce a TEST-*.xml file (likely timed
out) (batchId=289)
TestSparkTask - did not produce a TEST-*.xml file (likely timed out)
(batchId=289)
TestSparkUtilities - did not produce a TEST-*.xml file (likely timed out)
(batchId=289)
TestStatsReplicationScenariosACID - did not produce a TEST-*.xml file (likely
timed out) (batchId=183)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby_grouping_sets_limit]
(batchId=58)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=75)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testReplTimeouts
(batchId=243)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22166/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22166/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22166/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002200 - PreCommit-HIVE-Build
> Move q tests to TestMiniLlapLocal from TestCliDriver where the output is
> different, batch 2
> ---
>
> Key: HIVE-23337
> URL: https://issues.apache.org/jira/browse/HIVE-23337
> Project: Hive
> Issue Type: Sub-task
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-23337.01.patch, HIVE-23337.02.patch,
> HIVE-23337.03.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23387) Flip the Warehouse.getDefaultTablePath() to return path from ext warehouse
[ https://issues.apache.org/jira/browse/HIVE-23387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-23387: - Attachment: HIVE-23387.patch > Flip the Warehouse.getDefaultTablePath() to return path from ext warehouse > -- > > Key: HIVE-23387 > URL: https://issues.apache.org/jira/browse/HIVE-23387 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-23387.patch > > > For backward compatibility, initial fix returned path that was set on db. It > could have been either from managed warehouse or external depending on what > was set. There were tests relying on certain paths to be returned. This fix > is to address the tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23387) Flip the Warehouse.getDefaultTablePath() to return path from ext warehouse
[ https://issues.apache.org/jira/browse/HIVE-23387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-23387: - Status: Patch Available (was: Open) > Flip the Warehouse.getDefaultTablePath() to return path from ext warehouse > -- > > Key: HIVE-23387 > URL: https://issues.apache.org/jira/browse/HIVE-23387 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-23387.patch > > > For backward compatibility, initial fix returned path that was set on db. It > could have been either from managed warehouse or external depending on what > was set. There were tests relying on certain paths to be returned. This fix > is to address the tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23387) Flip the Warehouse.getDefaultTablePath() to return path from ext warehouse
[ https://issues.apache.org/jira/browse/HIVE-23387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam reassigned HIVE-23387: > Flip the Warehouse.getDefaultTablePath() to return path from ext warehouse > -- > > Key: HIVE-23387 > URL: https://issues.apache.org/jira/browse/HIVE-23387 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > > For backward compatibility, initial fix returned path that was set on db. It > could have been either from managed warehouse or external depending on what > was set. There were tests relying on certain paths to be returned. This fix > is to address the tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23337) Move q tests to TestMiniLlapLocal from TestCliDriver where the output is different, batch 2
[ https://issues.apache.org/jira/browse/HIVE-23337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23337: -- Attachment: HIVE-23337.03.patch > Move q tests to TestMiniLlapLocal from TestCliDriver where the output is > different, batch 2 > --- > > Key: HIVE-23337 > URL: https://issues.apache.org/jira/browse/HIVE-23337 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23337.01.patch, HIVE-23337.02.patch, > HIVE-23337.03.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23337) Move q tests to TestMiniLlapLocal from TestCliDriver where the output is different, batch 2
[ https://issues.apache.org/jira/browse/HIVE-23337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23337: -- Attachment: (was: HIVE-23337.03.patch) > Move q tests to TestMiniLlapLocal from TestCliDriver where the output is > different, batch 2 > --- > > Key: HIVE-23337 > URL: https://issues.apache.org/jira/browse/HIVE-23337 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23337.01.patch, HIVE-23337.02.patch, > HIVE-23337.03.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23350) Upgrade DBCP To DBCP 2.7.0
[
https://issues.apache.org/jira/browse/HIVE-23350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23350:
--
Attachment: HIVE-23350.1.patch
> Upgrade DBCP To DBCP 2.7.0
> --
>
> Key: HIVE-23350
> URL: https://issues.apache.org/jira/browse/HIVE-23350
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23350.1.patch, HIVE-23350.1.patch
>
>
> In {{hive-standalone-metastore-common}} the current version is 1.4 which is
> very antiquated and apparently (emphasis mine):
> bq. DBCP 1.4 compiles and runs under Java 6 *only* (JDBC 4)
> Hive is on Java 8
> {code:xml}
> 2.6.0
> 1.4
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23359) "show tables like" support for SQL wildcard characters (% and _)
[ https://issues.apache.org/jira/browse/HIVE-23359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23359: -- Attachment: HIVE-23359.05.patch > "show tables like" support for SQL wildcard characters (% and _) > > > Key: HIVE-23359 > URL: https://issues.apache.org/jira/browse/HIVE-23359 > Project: Hive > Issue Type: Improvement > Components: SQL >Affects Versions: 2.3.7 >Reporter: Yuming Wang >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23359.01.patch, HIVE-23359.02.patch, > HIVE-23359.03.patch, HIVE-23359.04.patch, HIVE-23359.05.patch > > > https://docs.snowflake.com/en/sql-reference/sql/show-tables.html > https://clickhouse.tech/docs/en/sql-reference/statements/show/ > https://www.mysqltutorial.org/mysql-show-tables/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23359) "show tables like" support for SQL wildcard characters (% and _)
[ https://issues.apache.org/jira/browse/HIVE-23359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23359: -- Attachment: (was: HIVE-23359.05.patch) > "show tables like" support for SQL wildcard characters (% and _) > > > Key: HIVE-23359 > URL: https://issues.apache.org/jira/browse/HIVE-23359 > Project: Hive > Issue Type: Improvement > Components: SQL >Affects Versions: 2.3.7 >Reporter: Yuming Wang >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23359.01.patch, HIVE-23359.02.patch, > HIVE-23359.03.patch, HIVE-23359.04.patch, HIVE-23359.05.patch > > > https://docs.snowflake.com/en/sql-reference/sql/show-tables.html > https://clickhouse.tech/docs/en/sql-reference/statements/show/ > https://www.mysqltutorial.org/mysql-show-tables/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23307) Cache ColumnIndex in HiveBaseResultSet
[
https://issues.apache.org/jira/browse/HIVE-23307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101198#comment-17101198
]
Naveen Gangam commented on HIVE-23307:
--
Change looks good to me. +1 for me. Thanks for the patch.
> Cache ColumnIndex in HiveBaseResultSet
> --
>
> Key: HIVE-23307
> URL: https://issues.apache.org/jira/browse/HIVE-23307
> Project: Hive
> Issue Type: Improvement
> Components: JDBC
>Affects Versions: 3.1.2
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23307.1.patch
>
>
> {code:java}
> public int findColumn(String columnName) throws SQLException {
> int columnIndex = 0;
> boolean findColumn = false;
> for (String normalizedColumnName : normalizedColumnNames) {
> ++columnIndex;
> String[] names = normalizedColumnName.split("\\.");
> String name = names[names.length -1];
> if (name.equalsIgnoreCase(columnName) ||
> normalizedColumnName.equalsIgnoreCase(columnName)) {
> findColumn = true;
> break;
> }
> }
> if (!findColumn) {
> throw new SQLException("Could not find " + columnName + " in " +
> normalizedColumnNames);
> } else {
> return columnIndex;
> }
> }
> {code}
> Cache the column name to column index map to avoid having to look it up again
> and again for each row O(n^2)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23337) Move q tests to TestMiniLlapLocal from TestCliDriver where the output is different, batch 2
[
https://issues.apache.org/jira/browse/HIVE-23337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101187#comment-17101187
]
Hive QA commented on HIVE-23337:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
2s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 3m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22166/dev-support/hive-personality.sh
|
| git revision | master / c3007d4 |
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22166/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Move q tests to TestMiniLlapLocal from TestCliDriver where the output is
> different, batch 2
> ---
>
> Key: HIVE-23337
> URL: https://issues.apache.org/jira/browse/HIVE-23337
> Project: Hive
> Issue Type: Sub-task
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-23337.01.patch, HIVE-23337.02.patch,
> HIVE-23337.03.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23359) "show tables like" support for SQL wildcard characters (% and _)
[
https://issues.apache.org/jira/browse/HIVE-23359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101177#comment-17101177
]
Hive QA commented on HIVE-23359:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002188/HIVE-23359.05.patch
{color:green}SUCCESS:{color} +1 due to 19 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17213 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.security.TestZookeeperTokenStoreSSLEnabled.org.apache.hadoop.hive.metastore.security.TestZookeeperTokenStoreSSLEnabled
(batchId=174)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22165/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22165/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22165/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002188 - PreCommit-HIVE-Build
> "show tables like" support for SQL wildcard characters (% and _)
>
>
> Key: HIVE-23359
> URL: https://issues.apache.org/jira/browse/HIVE-23359
> Project: Hive
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 2.3.7
>Reporter: Yuming Wang
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-23359.01.patch, HIVE-23359.02.patch,
> HIVE-23359.03.patch, HIVE-23359.04.patch, HIVE-23359.05.patch
>
>
> https://docs.snowflake.com/en/sql-reference/sql/show-tables.html
> https://clickhouse.tech/docs/en/sql-reference/statements/show/
> https://www.mysqltutorial.org/mysql-show-tables/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23301) Optimising privilege synchroniser: UDF for updating privileges
[ https://issues.apache.org/jira/browse/HIVE-23301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simhadri G updated HIVE-23301: -- Attachment: hive-23301.6.patch Status: Patch Available (was: In Progress) > Optimising privilege synchroniser: UDF for updating privileges > -- > > Key: HIVE-23301 > URL: https://issues.apache.org/jira/browse/HIVE-23301 > Project: Hive > Issue Type: Improvement > Components: Metastore, UDF >Affects Versions: 3.1.1 >Reporter: Simhadri G >Assignee: Simhadri G >Priority: Minor > Attachments: UDFSplitMapPrivs.patch, UDFSplitMapPrivs.patch, > hive-23301-2.patch, hive-23301.3.patch, hive-23301.4.patch, > hive-23301.5.patch, hive-23301.6.patch > > > Privilege synchronizer pulls the list of databases, tables and columns from > the Hive Metastore. For each of these objects it fetches the privilege > information and invokes HMS API to refresh the privilege information in HMS. > The current UDF Maps a bit string to a privilege based on if the privilege > is granted or not. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23301) Optimising privilege synchroniser: UDF for updating privileges
[ https://issues.apache.org/jira/browse/HIVE-23301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simhadri G updated HIVE-23301: -- Status: In Progress (was: Patch Available) > Optimising privilege synchroniser: UDF for updating privileges > -- > > Key: HIVE-23301 > URL: https://issues.apache.org/jira/browse/HIVE-23301 > Project: Hive > Issue Type: Improvement > Components: Metastore, UDF >Affects Versions: 3.1.1 >Reporter: Simhadri G >Assignee: Simhadri G >Priority: Minor > Attachments: UDFSplitMapPrivs.patch, UDFSplitMapPrivs.patch, > hive-23301-2.patch, hive-23301.3.patch, hive-23301.4.patch, hive-23301.5.patch > > > Privilege synchronizer pulls the list of databases, tables and columns from > the Hive Metastore. For each of these objects it fetches the privilege > information and invokes HMS API to refresh the privilege information in HMS. > The current UDF Maps a bit string to a privilege based on if the privilege > is granted or not. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23359) "show tables like" support for SQL wildcard characters (% and _)
[
https://issues.apache.org/jira/browse/HIVE-23359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101138#comment-17101138
]
Hive QA commented on HIVE-23359:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
37s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22165/dev-support/hive-personality.sh
|
| git revision | master / 1caa309 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22165/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> "show tables like" support for SQL wildcard characters (% and _)
>
>
> Key: HIVE-23359
> URL: https://issues.apache.org/jira/browse/HIVE-23359
> Project: Hive
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 2.3.7
>Reporter: Yuming Wang
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-23359.01.patch, HIVE-23359.02.patch,
> HIVE-23359.03.patch, HIVE-23359.04.patch, HIVE-23359.05.patch
>
>
> https://docs.snowflake.com/en/sql-reference/sql/show-tables.html
> https://clickhouse.tech/docs/en/sql-reference/statements/show/
> https://www.mysqltutorial.org/mysql-show-tables/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-19064) Add mode to support delimited identifiers enclosed within double quotation
[
https://issues.apache.org/jira/browse/HIVE-19064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-19064:
---
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Add mode to support delimited identifiers enclosed within double quotation
> --
>
> Key: HIVE-19064
> URL: https://issues.apache.org/jira/browse/HIVE-19064
> Project: Hive
> Issue Type: Improvement
> Components: Parser, SQL
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-19064.01.patch, HIVE-19064.02.patch,
> HIVE-19064.03.patch, HIVE-19064.10.patch, HIVE-19064.4.patch,
> HIVE-19064.5.patch, HIVE-19064.6.patch, HIVE-19064.7.patch,
> HIVE-19064.7.patch, HIVE-19064.8.patch, HIVE-19064.9.patch
>
>
> As per SQL standard. Hive currently uses `` (backticks). Default will
> continue being backticks, but we will support identifiers within double
> quotation via configuration parameter.
> This issue will also extends support for arbitrary char sequences, e.g.,
> containing {{~ ! @ # $ % ^ & * () , < >}}, in database and table names.
> Currently, special characters are only supported for column names.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-19064) Add mode to support delimited identifiers enclosed within double quotation
[
https://issues.apache.org/jira/browse/HIVE-19064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101128#comment-17101128
]
Jesus Camacho Rodriguez commented on HIVE-19064:
+1
Let's address any remaining issues in follow-up JIRAs.
> Add mode to support delimited identifiers enclosed within double quotation
> --
>
> Key: HIVE-19064
> URL: https://issues.apache.org/jira/browse/HIVE-19064
> Project: Hive
> Issue Type: Improvement
> Components: Parser, SQL
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-19064.01.patch, HIVE-19064.02.patch,
> HIVE-19064.03.patch, HIVE-19064.10.patch, HIVE-19064.4.patch,
> HIVE-19064.5.patch, HIVE-19064.6.patch, HIVE-19064.7.patch,
> HIVE-19064.7.patch, HIVE-19064.8.patch, HIVE-19064.9.patch
>
>
> As per SQL standard. Hive currently uses `` (backticks). Default will
> continue being backticks, but we will support identifiers within double
> quotation via configuration parameter.
> This issue will also extends support for arbitrary char sequences, e.g.,
> containing {{~ ! @ # $ % ^ & * () , < >}}, in database and table names.
> Currently, special characters are only supported for column names.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23318) TxnHandler need not delete from MATERIALIZATION_REBUILD_LOCKS on need basis
[
https://issues.apache.org/jira/browse/HIVE-23318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101119#comment-17101119
]
Hive QA commented on HIVE-23318:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002185/HIVE-23318.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17222 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22164/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22164/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22164/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002185 - PreCommit-HIVE-Build
> TxnHandler need not delete from MATERIALIZATION_REBUILD_LOCKS on need basis
> ---
>
> Key: HIVE-23318
> URL: https://issues.apache.org/jira/browse/HIVE-23318
> Project: Hive
> Issue Type: Improvement
> Components: Hive, Materialized views
>Reporter: Rajesh Balamohan
>Assignee: Marton Bod
>Priority: Minor
> Attachments: HIVE-23318.1.patch, HIVE-23318.2.patch,
> HIVE-23318.3.patch, HIVE-23318.4.patch
>
>
> Observed the following queries when materialized view or any of its feature
> was not used.
> TxnHandler need not clear this part of txn commit. It would help in reducing
> the sql parsing time in server side as well.
> {noformat}
> Gdelete from MATERIALIZATION_REBUILD_LOCKS where mrl_txn_id = 120398082
> Gdelete from MATERIALIZATION_REBUILD_LOCKS where mrl_txn_id = 120398084
> Gdelete from MATERIALIZATION_REBUILD_LOCKS where mrl_txn_id = 120398112
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23318) TxnHandler need not delete from MATERIALIZATION_REBUILD_LOCKS on need basis
[
https://issues.apache.org/jira/browse/HIVE-23318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101107#comment-17101107
]
Hive QA commented on HIVE-23318:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
36s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
14s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
50s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22164/dev-support/hive-personality.sh
|
| git revision | master / 1caa309 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22164/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TxnHandler need not delete from MATERIALIZATION_REBUILD_LOCKS on need basis
> ---
>
> Key: HIVE-23318
> URL: https://issues.apache.org/jira/browse/HIVE-23318
> Project: Hive
> Issue Type: Improvement
> Components: Hive, Materialized views
>Reporter: Rajesh Balamohan
>Assignee: Marton Bod
>Priority: Minor
> Attachments: HIVE-23318.1.patch, HIVE-23318.2.patch,
> HIVE-23318.3.patch, HIVE-23318.4.patch
>
>
> Observed the following queries when materialized view or any of its feature
> was not used.
> TxnHandler need not clear this part of txn commit. It would help in reducing
> the sql parsing time in server side as well.
> {noformat}
> Gdelete from MATERIALIZATION_REBUILD_LOCKS where mrl_txn_id = 120398082
> Gdelete from MATERIALIZATION_REBUILD_LOCKS where mrl_txn_id
[jira] [Commented] (HIVE-23266) Remove QueryWrapper from ObjectStore
[
https://issues.apache.org/jira/browse/HIVE-23266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101098#comment-17101098
]
David Mollitor commented on HIVE-23266:
---
Added a new patch which is a rebase on master.
> Remove QueryWrapper from ObjectStore
>
>
> Key: HIVE-23266
> URL: https://issues.apache.org/jira/browse/HIVE-23266
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23266.1.patch, HIVE-23266.10.patch,
> HIVE-23266.11.patch, HIVE-23266.2.patch, HIVE-23266.2.patch,
> HIVE-23266.3.patch, HIVE-23266.4.patch, HIVE-23266.5.patch,
> HIVE-23266.6.patch, HIVE-23266.7.patch, HIVE-23266.8.patch, HIVE-23266.9.patch
>
>
> There is currently a utility called {{QueryWrapper}} that makes a normal
> {{Query}} auto-closable. However, {{Query}} is now in fact already
> auto-closing, so there is no need for this class. In trying to remove it, I
> realized that this wrapper was being passed around in pretty convoluted ways
> and also it was sometimes being created in a {{try-with-resources}} block but
> then never actually used in any way.
> Remove the {{QueryWrapper}} from the class and simplify some of the DB
> interactions.
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L178
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23266) Remove QueryWrapper from ObjectStore
[
https://issues.apache.org/jira/browse/HIVE-23266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23266:
--
Attachment: HIVE-23266.11.patch
> Remove QueryWrapper from ObjectStore
>
>
> Key: HIVE-23266
> URL: https://issues.apache.org/jira/browse/HIVE-23266
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23266.1.patch, HIVE-23266.10.patch,
> HIVE-23266.11.patch, HIVE-23266.2.patch, HIVE-23266.2.patch,
> HIVE-23266.3.patch, HIVE-23266.4.patch, HIVE-23266.5.patch,
> HIVE-23266.6.patch, HIVE-23266.7.patch, HIVE-23266.8.patch, HIVE-23266.9.patch
>
>
> There is currently a utility called {{QueryWrapper}} that makes a normal
> {{Query}} auto-closable. However, {{Query}} is now in fact already
> auto-closing, so there is no need for this class. In trying to remove it, I
> realized that this wrapper was being passed around in pretty convoluted ways
> and also it was sometimes being created in a {{try-with-resources}} block but
> then never actually used in any way.
> Remove the {{QueryWrapper}} from the class and simplify some of the DB
> interactions.
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L178
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23234) Optimize TxnHandler::allocateTableWriteIds
[
https://issues.apache.org/jira/browse/HIVE-23234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101072#comment-17101072
]
Hive QA commented on HIVE-23234:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002182/HIVE-23234.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17216 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestPartitionManagement.testPartitionDiscoveryTransactionalTable
(batchId=151)
org.apache.hive.minikdc.TestSSLWithMiniKdc.testConnection (batchId=234)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22163/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22163/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22163/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002182 - PreCommit-HIVE-Build
> Optimize TxnHandler::allocateTableWriteIds
> --
>
> Key: HIVE-23234
> URL: https://issues.apache.org/jira/browse/HIVE-23234
> Project: Hive
> Issue Type: Improvement
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Attachments: HIVE-23234.1.patch, HIVE-23234.2.patch,
> HIVE-23234.3.patch
>
>
> Table write id allocation should be examined and optimized. One low hanging
> fruit is batching all the PreparedStatement inserts, but there might be other
> opportunities as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23234) Optimize TxnHandler::allocateTableWriteIds
[
https://issues.apache.org/jira/browse/HIVE-23234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101041#comment-17101041
]
Hive QA commented on HIVE-23234:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
14s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 1 new + 527 unchanged - 3 fixed = 528 total (was 530) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
25s{color} | {color:red} standalone-metastore/metastore-server generated 1 new
+ 187 unchanged - 0 fixed = 188 total (was 187) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | Dead store to updateCount in
org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.setHadoopJobId(String,
long) At
CompactionTxnHandler.java:org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.setHadoopJobId(String,
long) At CompactionTxnHandler.java:[line 1122] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22163/dev-support/hive-personality.sh
|
| git revision | master / 1caa309 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22163/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22163/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22163/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Optimize TxnHandler::allocateTableWriteIds
> --
>
> Key: HIVE-23234
> URL: https://issues.apache.org/jira/browse/HIVE-23234
> Project: Hive
> Issue Type: Improvement
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Attachments: HIVE-23234.1.patch, HIVE-23234.2.patch,
> HIVE-23234.3.patch
>
>
> Table write id allocation should be examined and optimized. One low hanging
> fruit is batching all the PreparedStatement inserts, but there might be other
> opportunities as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23325) Consolidate acid-related cleanup tasks
[ https://issues.apache.org/jira/browse/HIVE-23325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod updated HIVE-23325: -- Attachment: HIVE-23325.8.patch > Consolidate acid-related cleanup tasks > -- > > Key: HIVE-23325 > URL: https://issues.apache.org/jira/browse/HIVE-23325 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Attachments: HIVE-23325.1.patch, HIVE-23325.2.patch, > HIVE-23325.2.patch, HIVE-23325.2.patch, HIVE-23325.3.patch, > HIVE-23325.4.patch, HIVE-23325.5.patch, HIVE-23325.5.patch, > HIVE-23325.6.patch, HIVE-23325.6.patch, HIVE-23325.7.patch, > HIVE-23325.8.patch, HIVE-23325.8.patch > > > Currently there are multiple ways to clean up ACID related stuff: > * AcidHouseKeeperService > * AcidWriteSetService > * AcidCompactionHistoryService > * Initiator > * Cleaner > * etc. > We should consolidate them where possible and improve logging. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23325) Consolidate acid-related cleanup tasks
[
https://issues.apache.org/jira/browse/HIVE-23325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101003#comment-17101003
]
Hive QA commented on HIVE-23325:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002181/HIVE-23325.8.patch
{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17205 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=49)
[intersect_all.q,unionDistinct_1.q,orc_ppd_schema_evol_3a.q,import_exported_table.q,tez_union_dynamic_partition.q,vector_offset_limit.q,orc_merge10.q,tez_union_dynamic_partition_2.q,temp_table_external.q,llap_udf.q,schemeAuthority.q,cte_2.q,rcfile_createas1.q,cttl.q,remote_script.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22162/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22162/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22162/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002181 - PreCommit-HIVE-Build
> Consolidate acid-related cleanup tasks
> --
>
> Key: HIVE-23325
> URL: https://issues.apache.org/jira/browse/HIVE-23325
> Project: Hive
> Issue Type: Improvement
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Attachments: HIVE-23325.1.patch, HIVE-23325.2.patch,
> HIVE-23325.2.patch, HIVE-23325.2.patch, HIVE-23325.3.patch,
> HIVE-23325.4.patch, HIVE-23325.5.patch, HIVE-23325.5.patch,
> HIVE-23325.6.patch, HIVE-23325.6.patch, HIVE-23325.7.patch, HIVE-23325.8.patch
>
>
> Currently there are multiple ways to clean up ACID related stuff:
> * AcidHouseKeeperService
> * AcidWriteSetService
> * AcidCompactionHistoryService
> * Initiator
> * Cleaner
> * etc.
> We should consolidate them where possible and improve logging.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23325) Consolidate acid-related cleanup tasks
[
https://issues.apache.org/jira/browse/HIVE-23325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100971#comment-17100971
]
Hive QA commented on HIVE-23325:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
3s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
35s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
17s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
38s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch metastore-common passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} standalone-metastore/metastore-server: The patch
generated 0 new + 895 unchanged - 3 fixed = 895 total (was 898) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} ql: The patch generated 0 new + 716 unchanged - 4
fixed = 716 total (was 720) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22162/dev-support/hive-personality.sh
|
| git revision | master / 1caa309 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-common common
standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22162/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Consolidate acid-related cleanup tasks
> --
>
> Key: HIVE-23325
> URL: https://issues.apache.org/jira/browse/HIVE-23325
[jira] [Updated] (HIVE-23371) StatsUtils.getConstValue may log misleading exception
[
https://issues.apache.org/jira/browse/HIVE-23371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-23371:
--
Labels: pull-request-available (was: )
> StatsUtils.getConstValue may log misleading exception
> -
>
> Key: HIVE-23371
> URL: https://issues.apache.org/jira/browse/HIVE-23371
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23371.01.patch, HIVE-23371.02.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> it logs exceptions like:
> {code}
> 2020-05-05T01:51:59,609 DEBUG [cba65ded-3bb5-4295-9b3f-cdf844a8faec main]
> stats.StatsUtils: Interpreting constant (2008-04-08) resulted in exception
> java.lang.NumberFormatException: For input string: "2008-04-08"
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
> ~[?:1.8.0_252]
> at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
> ~[?:1.8.0_252]
> at java.lang.Double.parseDouble(Double.java:538) ~[?:1.8.0_252]
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getConstValue(StatsUtils.java:1653)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.buildColStatForConstant(StatsUtils.java:1632)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getColStatisticsFromExpression(StatsUtils.java:1538)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> {code}
> however...these exceptions are not that important; might just draw attention
> away from important ones
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23371) StatsUtils.getConstValue may log misleading exception
[
https://issues.apache.org/jira/browse/HIVE-23371?focusedWorklogId=431293=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-431293
]
ASF GitHub Bot logged work on HIVE-23371:
-
Author: ASF GitHub Bot
Created on: 06/May/20 16:24
Start Date: 06/May/20 16:24
Worklog Time Spent: 10m
Work Description: kgyrtkirk opened a new pull request #1006:
URL: https://github.com/apache/hive/pull/1006
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 431293)
Remaining Estimate: 0h
Time Spent: 10m
> StatsUtils.getConstValue may log misleading exception
> -
>
> Key: HIVE-23371
> URL: https://issues.apache.org/jira/browse/HIVE-23371
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-23371.01.patch, HIVE-23371.02.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> it logs exceptions like:
> {code}
> 2020-05-05T01:51:59,609 DEBUG [cba65ded-3bb5-4295-9b3f-cdf844a8faec main]
> stats.StatsUtils: Interpreting constant (2008-04-08) resulted in exception
> java.lang.NumberFormatException: For input string: "2008-04-08"
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
> ~[?:1.8.0_252]
> at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
> ~[?:1.8.0_252]
> at java.lang.Double.parseDouble(Double.java:538) ~[?:1.8.0_252]
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getConstValue(StatsUtils.java:1653)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.buildColStatForConstant(StatsUtils.java:1632)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getColStatisticsFromExpression(StatsUtils.java:1538)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> {code}
> however...these exceptions are not that important; might just draw attention
> away from important ones
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23371) StatsUtils.getConstValue may log misleading exception
[
https://issues.apache.org/jira/browse/HIVE-23371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-23371:
Attachment: HIVE-23371.02.patch
> StatsUtils.getConstValue may log misleading exception
> -
>
> Key: HIVE-23371
> URL: https://issues.apache.org/jira/browse/HIVE-23371
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-23371.01.patch, HIVE-23371.02.patch
>
>
> it logs exceptions like:
> {code}
> 2020-05-05T01:51:59,609 DEBUG [cba65ded-3bb5-4295-9b3f-cdf844a8faec main]
> stats.StatsUtils: Interpreting constant (2008-04-08) resulted in exception
> java.lang.NumberFormatException: For input string: "2008-04-08"
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
> ~[?:1.8.0_252]
> at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
> ~[?:1.8.0_252]
> at java.lang.Double.parseDouble(Double.java:538) ~[?:1.8.0_252]
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getConstValue(StatsUtils.java:1653)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.buildColStatForConstant(StatsUtils.java:1632)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getColStatisticsFromExpression(StatsUtils.java:1538)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> {code}
> however...these exceptions are not that important; might just draw attention
> away from important ones
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23242) Fix flaky tests testHouseKeepingThreadExistence in TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader
[
https://issues.apache.org/jira/browse/HIVE-23242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100920#comment-17100920
]
Hive QA commented on HIVE-23242:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002176/HIVE-23242.1.patch
{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17203 tests
executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=44)
[update_orig_table.q,explainanalyze_3.q,orc_vectorization_ppd.q,vector_non_string_partition.q,multi_count_distinct.q,explainanalyze_1.q,explainuser_3.q,hybridgrace_hashjoin_1.q,tez_union_udtf.q,delete_orig_table.q,tez_complextype_with_null.q,hybridgrace_hashjoin_2.q,tez-tag.q,explainanalyze_5.q,vector_join_part_col_char.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22161/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22161/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22161/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002176 - PreCommit-HIVE-Build
> Fix flaky tests testHouseKeepingThreadExistence in
> TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader
> -
>
> Key: HIVE-23242
> URL: https://issues.apache.org/jira/browse/HIVE-23242
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Peter Varga
>Priority: Major
> Attachments: HIVE-23242.1.patch
>
>
> Tests were ignored, see https://issues.apache.org/jira/browse/HIVE-23221 for
> details.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23282) Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
[ https://issues.apache.org/jira/browse/HIVE-23282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23282: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks, Attila! > Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal > - > > Key: HIVE-23282 > URL: https://issues.apache.org/jira/browse/HIVE-23282 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23282.1.patch, HIVE-23282.3.patch, > HIVE-23282.5.patch, HIVE-23282.6.patch, HIVE-23282.7.patch, > HIVE-23282.8.patch, image-2020-04-23-14-07-06-077.png > > > ObjectStore::getPartitionsByExprInternal internally uses Table information > for getting partitionKeys, table, catalog name. > > For this, it ends up populating entire table data from DB (including skew > column, parameters, sort, bucket cols etc). This makes it a lot more > expensive call. It would be good to check if MTable itself can be used > instead of Table. > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3327] > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3669] > > !image-2020-04-23-14-07-06-077.png|width=665,height=592! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23337) Move q tests to TestMiniLlapLocal from TestCliDriver where the output is different, batch 2
[ https://issues.apache.org/jira/browse/HIVE-23337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23337: -- Attachment: HIVE-23337.03.patch > Move q tests to TestMiniLlapLocal from TestCliDriver where the output is > different, batch 2 > --- > > Key: HIVE-23337 > URL: https://issues.apache.org/jira/browse/HIVE-23337 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23337.01.patch, HIVE-23337.02.patch, > HIVE-23337.03.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23338) Bump jackson-databind version up to 2.9.10.4
[ https://issues.apache.org/jira/browse/HIVE-23338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-23338: - Attachment: HIVE-23338.02.patch Status: Patch Available (was: Open) > Bump jackson-databind version up to 2.9.10.4 > > > Key: HIVE-23338 > URL: https://issues.apache.org/jira/browse/HIVE-23338 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-23338.01.patch, HIVE-23338.01.patch, > HIVE-23338.01.patch, HIVE-23338.01.patch, HIVE-23338.02.patch > > > com.fasterxml.jackson.core:jackson-databind:2.9.9 is exploitable. > And exclude a transitive dependency on > com.fasterxml.jackson.core:jackson-databind:2.6.5, which is also exploitable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23338) Bump jackson-databind version up to 2.9.10.4
[ https://issues.apache.org/jira/browse/HIVE-23338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-23338: - Status: Open (was: Patch Available) > Bump jackson-databind version up to 2.9.10.4 > > > Key: HIVE-23338 > URL: https://issues.apache.org/jira/browse/HIVE-23338 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-23338.01.patch, HIVE-23338.01.patch, > HIVE-23338.01.patch, HIVE-23338.01.patch > > > com.fasterxml.jackson.core:jackson-databind:2.9.9 is exploitable. > And exclude a transitive dependency on > com.fasterxml.jackson.core:jackson-databind:2.6.5, which is also exploitable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23242) Fix flaky tests testHouseKeepingThreadExistence in TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader
[
https://issues.apache.org/jira/browse/HIVE-23242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100873#comment-17100873
]
Hive QA commented on HIVE-23242:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22161/dev-support/hive-personality.sh
|
| git revision | master / 48c0110 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: itests/hive-unit U: itests/hive-unit |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22161/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix flaky tests testHouseKeepingThreadExistence in
> TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader
> -
>
> Key: HIVE-23242
> URL: https://issues.apache.org/jira/browse/HIVE-23242
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Peter Varga
>Priority: Major
> Attachments: HIVE-23242.1.patch
>
>
> Tests were ignored, see https://issues.apache.org/jira/browse/HIVE-23221 for
> details.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23351) Ranger Replication Scheduling
[
https://issues.apache.org/jira/browse/HIVE-23351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100854#comment-17100854
]
Hive QA commented on HIVE-23351:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002177/HIVE-23351.04.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 235 failed/errored test(s), 17216 tests
executed
*Failed tests:*
{noformat}
TestRangerRestClient - did not produce a TEST-*.xml file (likely timed out)
(batchId=247)
org.apache.hadoop.hive.ql.parse.TestCopyUtils.testPrivilegedDistCpWithSameUserAsCurrentDoesNotTryToImpersonate
(batchId=208)
org.apache.hadoop.hive.ql.parse.TestMetaStoreEventListenerInRepl.testReplEvents
(batchId=183)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesBootstrapWithJsonMessage.retryIncBootstrapAcidFromDifferentDumpWithoutCleanTablesConfig
(batchId=185)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesBootstrapWithJsonMessage.testAcidTablesBootstrapDuringIncremental
(batchId=185)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesBootstrapWithJsonMessage.testAcidTablesBootstrapDuringIncrementalWithOpenTxnsTimeout
(batchId=185)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesBootstrapWithJsonMessage.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites
(batchId=185)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesBootstrapWithJsonMessage.testRetryAcidTablesBootstrapFromDifferentDump
(batchId=185)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testAcidBootstrapReplLoadRetryAfterFailure
(batchId=202)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testAcidTablesBootstrap
(batchId=202)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testAcidTablesBootstrapWithConcurrentWrites
(batchId=202)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testAcidTablesBootstrapWithOpenTxnsTimeout
(batchId=202)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testAcidTablesMoveOptimizationBootStrap
(batchId=202)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testCheckPointingBootstrapDuringIncrementalRegularCopy
(batchId=202)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testCheckPointingDataDumpFailure
(batchId=202)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testCheckPointingDataDumpFailureBootstrapDuringIncremental
(batchId=202)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testCheckPointingDataDumpFailureORCTableDistcpCopy
(batchId=202)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testCheckPointingDataDumpFailureORCTableRegularCopy
(batchId=202)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testCheckPointingDataDumpFailureRegularCopy
(batchId=202)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testCheckPointingMetadataDumpFailure
(batchId=202)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testCheckPointingWithNewTablesAdded
(batchId=202)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testCheckPointingWithSourceTableDataInserted
(batchId=202)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testCheckPointingWithSourceTableDeleted
(batchId=202)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testCheckpointingIncrWithTableDrop
(batchId=202)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testCheckpointingOnFirstEventDump
(batchId=202)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testHdfsMaxDirItemsLimitDuringIncremental
(batchId=202)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testIncrementalDumpCheckpointing
(batchId=202)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testIncrementalResumeDump
(batchId=202)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testIncrementalResumeDumpFromInvalidEventDumpFile
(batchId=202)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testMultiDBTxn
(batchId=202)
org.apache.hadoop.hive.ql.parse.TestReplAcrossInstancesWithJsonMessageFormat.testBootstrapLoadRetryAfterFailureForAlterTable
(batchId=194)
org.apache.hadoop.hive.ql.parse.TestReplAcrossInstancesWithJsonMessageFormat.testBootstrapReplLoadRetryAfterFailureForPartitions
(batchId=194)
org.apache.hadoop.hive.ql.parse.TestReplAcrossInstancesWithJsonMessageFormat.testBootstrapReplLoadRetryAfterFailureForTablesAndConstraints
(batchId=194)
org.apache.hadoop.hive.ql.parse.TestReplAcrossInstancesWithJsonMessageFormat.testIfBootstrapReplLoadFailWhenRetryAfterBootstrapComplete
(batchId=194)
org.apache.hadoop.hive.ql.parse.TestReplAcrossInstancesWithJsonMessageFormat.testIfCkptAndSourceOfReplPropsIgnoredByReplDump
(batchId=194)
[jira] [Updated] (HIVE-23381) Investigate why the column stats are gone from groupby_sort_1_23.q, louter_join_ppr.q
[ https://issues.apache.org/jira/browse/HIVE-23381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23381: -- Summary: Investigate why the column stats are gone from groupby_sort_1_23.q, louter_join_ppr.q (was: Investigate why the column stats are gone for groupby_sort_1_23.q) > Investigate why the column stats are gone from groupby_sort_1_23.q, > louter_join_ppr.q > - > > Key: HIVE-23381 > URL: https://issues.apache.org/jira/browse/HIVE-23381 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Priority: Major > > Check [https://reviews.apache.org/r/72466/#comment309109] for details. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23325) Consolidate acid-related cleanup tasks
[ https://issues.apache.org/jira/browse/HIVE-23325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod updated HIVE-23325: -- Summary: Consolidate acid-related cleanup tasks (was: Clean up cleanup tasks for TxnHandler/CompactionTxnHandler) > Consolidate acid-related cleanup tasks > -- > > Key: HIVE-23325 > URL: https://issues.apache.org/jira/browse/HIVE-23325 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Attachments: HIVE-23325.1.patch, HIVE-23325.2.patch, > HIVE-23325.2.patch, HIVE-23325.2.patch, HIVE-23325.3.patch, > HIVE-23325.4.patch, HIVE-23325.5.patch, HIVE-23325.5.patch, > HIVE-23325.6.patch, HIVE-23325.6.patch, HIVE-23325.7.patch, HIVE-23325.8.patch > > > Currently there are multiple ways to clean up ACID related stuff: > * AcidHouseKeeperService > * AcidWriteSetService > * AcidCompactionHistoryService > * Initiator > * Cleaner > * etc. > We should consolidate them where possible and improve logging. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23325) Clean up cleanup tasks for TxnHandler/CompactionTxnHandler
[ https://issues.apache.org/jira/browse/HIVE-23325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100838#comment-17100838 ] Denys Kuzmenko commented on HIVE-23325: --- +1, pending tests > Clean up cleanup tasks for TxnHandler/CompactionTxnHandler > -- > > Key: HIVE-23325 > URL: https://issues.apache.org/jira/browse/HIVE-23325 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Attachments: HIVE-23325.1.patch, HIVE-23325.2.patch, > HIVE-23325.2.patch, HIVE-23325.2.patch, HIVE-23325.3.patch, > HIVE-23325.4.patch, HIVE-23325.5.patch, HIVE-23325.5.patch, > HIVE-23325.6.patch, HIVE-23325.6.patch, HIVE-23325.7.patch, HIVE-23325.8.patch > > > Currently there are multiple ways to clean up ACID related stuff: > * AcidHouseKeeperService > * AcidWriteSetService > * AcidCompactionHistoryService > * Initiator > * Cleaner > * etc. > We should consolidate them where possible and improve logging. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23351) Ranger Replication Scheduling
[
https://issues.apache.org/jira/browse/HIVE-23351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100836#comment-17100836
]
Hive QA commented on HIVE-23351:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
6s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
39s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 11 new + 52 unchanged - 0
fixed = 63 total (was 52) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22160/dev-support/hive-personality.sh
|
| git revision | master / 48c0110 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22160/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22160/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Ranger Replication Scheduling
> -
>
> Key: HIVE-23351
> URL: https://issues.apache.org/jira/browse/HIVE-23351
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23351.01.patch, HIVE-23351.02.patch,
> HIVE-23351.03.patch, HIVE-23351.04.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23344) Bump org.scala-lang:scala-library version to 2.12.14
[ https://issues.apache.org/jira/browse/HIVE-23344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-23344: - Description: org.scala-lang:scala-library:jar:2.11.8 has security vulnerabilities, e.g. https://snyk.io/vuln/SNYK-JAVA-ORGSCALALANG-31592 (was: Lots of org.scala-lang:scala-compiler:jar:2.11.0:compile -> org.scala-lang:scalap:jar:2.11.0:compile -> org.json4s:json4s-core_2.11:jar:3.2.11:compile -> org.json4s:json4s-jackson_2.11:jar:3.2.11:compile -> org.apache.spark:spark-core_2.11:jar:2.3.0:compile Where scala-compiler 2.11.0 is a security vulnerability. Excluding org.json4s:json4s-jackson_2.11:jar:3.2.11 from spark-core_2 dependencies.) > Bump org.scala-lang:scala-library version to 2.12.14 > > > Key: HIVE-23344 > URL: https://issues.apache.org/jira/browse/HIVE-23344 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-23344.01.patch, HIVE-23344.01.patch > > > org.scala-lang:scala-library:jar:2.11.8 has security vulnerabilities, e.g. > https://snyk.io/vuln/SNYK-JAVA-ORGSCALALANG-31592 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23344) Bump org.scala-lang:scala-library version to 2.12.14
[ https://issues.apache.org/jira/browse/HIVE-23344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-23344: - Summary: Bump org.scala-lang:scala-library version to 2.12.14 (was: Remove org.scala-lang:scala-compiler 2.11.0 transitive dependency) > Bump org.scala-lang:scala-library version to 2.12.14 > > > Key: HIVE-23344 > URL: https://issues.apache.org/jira/browse/HIVE-23344 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-23344.01.patch, HIVE-23344.01.patch > > > Lots of > org.scala-lang:scala-compiler:jar:2.11.0:compile -> > org.scala-lang:scalap:jar:2.11.0:compile -> > org.json4s:json4s-core_2.11:jar:3.2.11:compile -> > org.json4s:json4s-jackson_2.11:jar:3.2.11:compile -> > org.apache.spark:spark-core_2.11:jar:2.3.0:compile > Where scala-compiler 2.11.0 is a security vulnerability. > Excluding org.json4s:json4s-jackson_2.11:jar:3.2.11 from spark-core_2 > dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23359) "show tables like" support for SQL wildcard characters (% and _)
[ https://issues.apache.org/jira/browse/HIVE-23359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23359: -- Attachment: HIVE-23359.05.patch > "show tables like" support for SQL wildcard characters (% and _) > > > Key: HIVE-23359 > URL: https://issues.apache.org/jira/browse/HIVE-23359 > Project: Hive > Issue Type: Improvement > Components: SQL >Affects Versions: 2.3.7 >Reporter: Yuming Wang >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23359.01.patch, HIVE-23359.02.patch, > HIVE-23359.03.patch, HIVE-23359.04.patch, HIVE-23359.05.patch > > > https://docs.snowflake.com/en/sql-reference/sql/show-tables.html > https://clickhouse.tech/docs/en/sql-reference/statements/show/ > https://www.mysqltutorial.org/mysql-show-tables/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23318) TxnHandler need not delete from MATERIALIZATION_REBUILD_LOCKS on need basis
[
https://issues.apache.org/jira/browse/HIVE-23318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Bod updated HIVE-23318:
--
Attachment: HIVE-23318.4.patch
> TxnHandler need not delete from MATERIALIZATION_REBUILD_LOCKS on need basis
> ---
>
> Key: HIVE-23318
> URL: https://issues.apache.org/jira/browse/HIVE-23318
> Project: Hive
> Issue Type: Improvement
> Components: Hive, Materialized views
>Reporter: Rajesh Balamohan
>Assignee: Marton Bod
>Priority: Minor
> Attachments: HIVE-23318.1.patch, HIVE-23318.2.patch,
> HIVE-23318.3.patch, HIVE-23318.4.patch
>
>
> Observed the following queries when materialized view or any of its feature
> was not used.
> TxnHandler need not clear this part of txn commit. It would help in reducing
> the sql parsing time in server side as well.
> {noformat}
> Gdelete from MATERIALIZATION_REBUILD_LOCKS where mrl_txn_id = 120398082
> Gdelete from MATERIALIZATION_REBUILD_LOCKS where mrl_txn_id = 120398084
> Gdelete from MATERIALIZATION_REBUILD_LOCKS where mrl_txn_id = 120398112
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23307) Cache ColumnIndex in HiveBaseResultSet
[
https://issues.apache.org/jira/browse/HIVE-23307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23307:
--
Issue Type: Improvement (was: New Feature)
> Cache ColumnIndex in HiveBaseResultSet
> --
>
> Key: HIVE-23307
> URL: https://issues.apache.org/jira/browse/HIVE-23307
> Project: Hive
> Issue Type: Improvement
> Components: JDBC
>Affects Versions: 3.1.2
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23307.1.patch
>
>
> {code:java}
> public int findColumn(String columnName) throws SQLException {
> int columnIndex = 0;
> boolean findColumn = false;
> for (String normalizedColumnName : normalizedColumnNames) {
> ++columnIndex;
> String[] names = normalizedColumnName.split("\\.");
> String name = names[names.length -1];
> if (name.equalsIgnoreCase(columnName) ||
> normalizedColumnName.equalsIgnoreCase(columnName)) {
> findColumn = true;
> break;
> }
> }
> if (!findColumn) {
> throw new SQLException("Could not find " + columnName + " in " +
> normalizedColumnNames);
> } else {
> return columnIndex;
> }
> }
> {code}
> Cache the column name to column index map to avoid having to look it up again
> and again for each row O(n^2)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23307) Cache ColumnIndex in HiveBaseResultSet
[
https://issues.apache.org/jira/browse/HIVE-23307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100805#comment-17100805
]
David Mollitor commented on HIVE-23307:
---
[~ngangam] Any spare cycles on this one? :)
> Cache ColumnIndex in HiveBaseResultSet
> --
>
> Key: HIVE-23307
> URL: https://issues.apache.org/jira/browse/HIVE-23307
> Project: Hive
> Issue Type: New Feature
> Components: JDBC
>Affects Versions: 3.1.2
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23307.1.patch
>
>
> {code:java}
> public int findColumn(String columnName) throws SQLException {
> int columnIndex = 0;
> boolean findColumn = false;
> for (String normalizedColumnName : normalizedColumnNames) {
> ++columnIndex;
> String[] names = normalizedColumnName.split("\\.");
> String name = names[names.length -1];
> if (name.equalsIgnoreCase(columnName) ||
> normalizedColumnName.equalsIgnoreCase(columnName)) {
> findColumn = true;
> break;
> }
> }
> if (!findColumn) {
> throw new SQLException("Could not find " + columnName + " in " +
> normalizedColumnNames);
> } else {
> return columnIndex;
> }
> }
> {code}
> Cache the column name to column index map to avoid having to look it up again
> and again for each row O(n^2)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23359) "show tables like" support for SQL wildcard characters (% and _)
[
https://issues.apache.org/jira/browse/HIVE-23359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100804#comment-17100804
]
Hive QA commented on HIVE-23359:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002164/HIVE-23359.04.patch
{color:green}SUCCESS:{color} +1 due to 19 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17216 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.parse.TestScheduledReplicationScenarios.testAcidTablesReplLoadBootstrapIncr
(batchId=208)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22159/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22159/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22159/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002164 - PreCommit-HIVE-Build
> "show tables like" support for SQL wildcard characters (% and _)
>
>
> Key: HIVE-23359
> URL: https://issues.apache.org/jira/browse/HIVE-23359
> Project: Hive
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 2.3.7
>Reporter: Yuming Wang
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-23359.01.patch, HIVE-23359.02.patch,
> HIVE-23359.03.patch, HIVE-23359.04.patch
>
>
> https://docs.snowflake.com/en/sql-reference/sql/show-tables.html
> https://clickhouse.tech/docs/en/sql-reference/statements/show/
> https://www.mysqltutorial.org/mysql-show-tables/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23234) Optimize TxnHandler::allocateTableWriteIds
[ https://issues.apache.org/jira/browse/HIVE-23234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod updated HIVE-23234: -- Attachment: HIVE-23234.3.patch > Optimize TxnHandler::allocateTableWriteIds > -- > > Key: HIVE-23234 > URL: https://issues.apache.org/jira/browse/HIVE-23234 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Attachments: HIVE-23234.1.patch, HIVE-23234.2.patch, > HIVE-23234.3.patch > > > Table write id allocation should be examined and optimized. One low hanging > fruit is batching all the PreparedStatement inserts, but there might be other > opportunities as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23325) Clean up cleanup tasks for TxnHandler/CompactionTxnHandler
[ https://issues.apache.org/jira/browse/HIVE-23325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod updated HIVE-23325: -- Attachment: HIVE-23325.8.patch > Clean up cleanup tasks for TxnHandler/CompactionTxnHandler > -- > > Key: HIVE-23325 > URL: https://issues.apache.org/jira/browse/HIVE-23325 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Attachments: HIVE-23325.1.patch, HIVE-23325.2.patch, > HIVE-23325.2.patch, HIVE-23325.2.patch, HIVE-23325.3.patch, > HIVE-23325.4.patch, HIVE-23325.5.patch, HIVE-23325.5.patch, > HIVE-23325.6.patch, HIVE-23325.6.patch, HIVE-23325.7.patch, HIVE-23325.8.patch > > > Currently there are multiple ways to clean up ACID related stuff: > * AcidHouseKeeperService > * AcidWriteSetService > * AcidCompactionHistoryService > * Initiator > * Cleaner > * etc. > We should consolidate them where possible and improve logging. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23234) Optimize TxnHandler::allocateTableWriteIds
[ https://issues.apache.org/jira/browse/HIVE-23234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod updated HIVE-23234: -- Attachment: HIVE-23234.2.patch > Optimize TxnHandler::allocateTableWriteIds > -- > > Key: HIVE-23234 > URL: https://issues.apache.org/jira/browse/HIVE-23234 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Attachments: HIVE-23234.1.patch, HIVE-23234.2.patch > > > Table write id allocation should be examined and optimized. One low hanging > fruit is batching all the PreparedStatement inserts, but there might be other > opportunities as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23242) Fix flaky tests testHouseKeepingThreadExistence in TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader
[ https://issues.apache.org/jira/browse/HIVE-23242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23242: --- Status: Patch Available (was: Open) The problem was with the timing of the tests. The metastore threads were starting with 7 sec delay and the test was waiting for 10 sec before checking. But the metastore threads are starting asynchronously, and it seems the startup time sometimes was getting slower. In the attached log the missing thread scheduled after 5 second, and would start at 12 second, so the check was running before. Modified the threads to start only with 2 sec delay. > Fix flaky tests testHouseKeepingThreadExistence in > TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader > - > > Key: HIVE-23242 > URL: https://issues.apache.org/jira/browse/HIVE-23242 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Miklos Gergely >Assignee: Peter Varga >Priority: Major > Attachments: HIVE-23242.1.patch > > > Tests were ignored, see https://issues.apache.org/jira/browse/HIVE-23221 for > details. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23351) Ranger Replication Scheduling
[ https://issues.apache.org/jira/browse/HIVE-23351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23351: --- Status: In Progress (was: Patch Available) > Ranger Replication Scheduling > - > > Key: HIVE-23351 > URL: https://issues.apache.org/jira/browse/HIVE-23351 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23351.01.patch, HIVE-23351.02.patch, > HIVE-23351.03.patch, HIVE-23351.04.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23351) Ranger Replication Scheduling
[ https://issues.apache.org/jira/browse/HIVE-23351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23351: --- Attachment: HIVE-23351.04.patch Status: Patch Available (was: In Progress) > Ranger Replication Scheduling > - > > Key: HIVE-23351 > URL: https://issues.apache.org/jira/browse/HIVE-23351 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23351.01.patch, HIVE-23351.02.patch, > HIVE-23351.03.patch, HIVE-23351.04.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23242) Fix flaky tests testHouseKeepingThreadExistence in TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader
[ https://issues.apache.org/jira/browse/HIVE-23242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23242: --- Attachment: HIVE-23242.1.patch > Fix flaky tests testHouseKeepingThreadExistence in > TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader > - > > Key: HIVE-23242 > URL: https://issues.apache.org/jira/browse/HIVE-23242 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Miklos Gergely >Assignee: Peter Varga >Priority: Major > Attachments: HIVE-23242.1.patch > > > Tests were ignored, see https://issues.apache.org/jira/browse/HIVE-23221 for > details. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23359) "show tables like" support for SQL wildcard characters (% and _)
[
https://issues.apache.org/jira/browse/HIVE-23359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100771#comment-17100771
]
Hive QA commented on HIVE-23359:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
43s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22159/dev-support/hive-personality.sh
|
| git revision | master / 48c0110 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22159/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> "show tables like" support for SQL wildcard characters (% and _)
>
>
> Key: HIVE-23359
> URL: https://issues.apache.org/jira/browse/HIVE-23359
> Project: Hive
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 2.3.7
>Reporter: Yuming Wang
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-23359.01.patch, HIVE-23359.02.patch,
> HIVE-23359.03.patch, HIVE-23359.04.patch
>
>
> https://docs.snowflake.com/en/sql-reference/sql/show-tables.html
> https://clickhouse.tech/docs/en/sql-reference/statements/show/
> https://www.mysqltutorial.org/mysql-show-tables/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23382) Investigate why the results have changed for groupby_sort_1_23.q
[ https://issues.apache.org/jira/browse/HIVE-23382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23382: -- Description: Check [https://reviews.apache.org/r/72466/#comment309109] for details. > Investigate why the results have changed for groupby_sort_1_23.q > > > Key: HIVE-23382 > URL: https://issues.apache.org/jira/browse/HIVE-23382 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Priority: Major > > Check [https://reviews.apache.org/r/72466/#comment309109] for details. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23318) TxnHandler need not delete from MATERIALIZATION_REBUILD_LOCKS on need basis
[
https://issues.apache.org/jira/browse/HIVE-23318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100747#comment-17100747
]
Hive QA commented on HIVE-23318:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002140/HIVE-23318.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17218 tests
executed
*Failed tests:*
{noformat}
TestStatsReplicationScenariosACID - did not produce a TEST-*.xml file (likely
timed out) (batchId=187)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22158/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22158/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22158/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002140 - PreCommit-HIVE-Build
> TxnHandler need not delete from MATERIALIZATION_REBUILD_LOCKS on need basis
> ---
>
> Key: HIVE-23318
> URL: https://issues.apache.org/jira/browse/HIVE-23318
> Project: Hive
> Issue Type: Improvement
> Components: Hive, Materialized views
>Reporter: Rajesh Balamohan
>Assignee: Marton Bod
>Priority: Minor
> Attachments: HIVE-23318.1.patch, HIVE-23318.2.patch,
> HIVE-23318.3.patch
>
>
> Observed the following queries when materialized view or any of its feature
> was not used.
> TxnHandler need not clear this part of txn commit. It would help in reducing
> the sql parsing time in server side as well.
> {noformat}
> Gdelete from MATERIALIZATION_REBUILD_LOCKS where mrl_txn_id = 120398082
> Gdelete from MATERIALIZATION_REBUILD_LOCKS where mrl_txn_id = 120398084
> Gdelete from MATERIALIZATION_REBUILD_LOCKS where mrl_txn_id = 120398112
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23351) Ranger Replication Scheduling
[
https://issues.apache.org/jira/browse/HIVE-23351?focusedWorklogId=431200=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-431200
]
ASF GitHub Bot logged work on HIVE-23351:
-
Author: ASF GitHub Bot
Created on: 06/May/20 12:17
Start Date: 06/May/20 12:17
Worklog Time Spent: 10m
Work Description: aasha commented on a change in pull request #1004:
URL: https://github.com/apache/hive/pull/1004#discussion_r420744102
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##
@@ -170,6 +175,31 @@ public int execute() {
return 0;
}
+ private void initiateAuthorizationDumpTask(Path currentDumpPath) throws
SemanticException, IOException {
+if
(RANGER_AUTHORIZER.equalsIgnoreCase(conf.getVar(HiveConf.ConfVars.REPL_AUTHORIZATION_PROVIDER_SERVICE)))
{
+ Path rangerDumpRoot = new Path(currentDumpPath,
ReplUtils.REPL_RANGER_BASE_DIR);
+ FileSystem fs = rangerDumpRoot.getFileSystem(conf);
+ if (fs.exists(new Path(rangerDumpRoot,
ReplAck.RANGER_DUMP_ACKNOWLEDGEMENT.toString( {
+LOG.info("Ranger Authorization Metadata is already exported at {} ",
rangerDumpRoot);
+ } else {
+LOG.info("Exporting Authorization Metadata at {} ", rangerDumpRoot);
+RangerDumpWork rangerDumpWork = new RangerDumpWork(rangerDumpRoot,
work.dbNameOrPattern);
+Task rangerDumpTask = TaskFactory.get(rangerDumpWork,
conf);
+if (childTasks == null) {
+ childTasks = new ArrayList<>();
+}
+childTasks.add(rangerDumpTask);
Review comment:
Its getting added before the data copy tasks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 431200)
Time Spent: 0.5h (was: 20m)
> Ranger Replication Scheduling
> -
>
> Key: HIVE-23351
> URL: https://issues.apache.org/jira/browse/HIVE-23351
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23351.01.patch, HIVE-23351.02.patch,
> HIVE-23351.03.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23351) Ranger Replication Scheduling
[
https://issues.apache.org/jira/browse/HIVE-23351?focusedWorklogId=431199=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-431199
]
ASF GitHub Bot logged work on HIVE-23351:
-
Author: ASF GitHub Bot
Created on: 06/May/20 12:16
Start Date: 06/May/20 12:16
Worklog Time Spent: 10m
Work Description: aasha commented on a change in pull request #1004:
URL: https://github.com/apache/hive/pull/1004#discussion_r420743627
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplAck.java
##
@@ -23,7 +23,9 @@
public enum ReplAck {
DUMP_ACKNOWLEDGEMENT("_finished_dump"),
EVENTS_DUMP("_events_dump"),
-LOAD_ACKNOWLEDGEMENT("_finished_load");
+LOAD_ACKNOWLEDGEMENT("_finished_load"),
+RANGER_DUMP_ACKNOWLEDGEMENT("_finished_ranger_dump"),
Review comment:
As discussed we will remove the Acks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 431199)
Time Spent: 20m (was: 10m)
> Ranger Replication Scheduling
> -
>
> Key: HIVE-23351
> URL: https://issues.apache.org/jira/browse/HIVE-23351
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23351.01.patch, HIVE-23351.02.patch,
> HIVE-23351.03.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23242) Fix flaky tests testHouseKeepingThreadExistence in TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader
[ https://issues.apache.org/jira/browse/HIVE-23242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga reassigned HIVE-23242: -- Assignee: Peter Varga > Fix flaky tests testHouseKeepingThreadExistence in > TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader > - > > Key: HIVE-23242 > URL: https://issues.apache.org/jira/browse/HIVE-23242 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Miklos Gergely >Assignee: Peter Varga >Priority: Major > > Tests were ignored, see https://issues.apache.org/jira/browse/HIVE-23221 for > details. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23318) TxnHandler need not delete from MATERIALIZATION_REBUILD_LOCKS on need basis
[
https://issues.apache.org/jira/browse/HIVE-23318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100731#comment-17100731
]
Hive QA commented on HIVE-23318:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
34s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
17s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
40s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22158/dev-support/hive-personality.sh
|
| git revision | master / 48c0110 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22158/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TxnHandler need not delete from MATERIALIZATION_REBUILD_LOCKS on need basis
> ---
>
> Key: HIVE-23318
> URL: https://issues.apache.org/jira/browse/HIVE-23318
> Project: Hive
> Issue Type: Improvement
> Components: Hive, Materialized views
>Reporter: Rajesh Balamohan
>Assignee: Marton Bod
>Priority: Minor
> Attachments: HIVE-23318.1.patch, HIVE-23318.2.patch,
> HIVE-23318.3.patch
>
>
> Observed the following queries when materialized view or any of its feature
> was not used.
> TxnHandler need not clear this part of txn commit. It would help in reducing
> the sql parsing time in server side as well.
> {noformat}
> Gdelete from MATERIALIZATION_REBUILD_LOCKS where mrl_txn_id = 120398082
> Gdelete from MATERIALIZATION_REBUILD_LOCKS where mrl_txn_id = 120398084
>
[jira] [Commented] (HIVE-23375) Track MJ HashTable Load time
[ https://issues.apache.org/jira/browse/HIVE-23375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100696#comment-17100696 ] Panagiotis Garefalakis commented on HIVE-23375: --- Hey [~kgyrtkirk] thanks for the comments! Adding a memory estimation counter would make sense indeed, can work on that as a follow-up. Currently, these counters are aggregated in vertex level so yes it would be the cumulative load_time which gives you some information but not the whole picture I agree. Going task level would be too much so let me dig a bit deeper on Tez level to check how we can introduce a counter that tracks a distribution of values. > Track MJ HashTable Load time > > > Key: HIVE-23375 > URL: https://issues.apache.org/jira/browse/HIVE-23375 > Project: Hive > Issue Type: Improvement >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23375.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Introduce TezCounter to track MJ HashTable Load time -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23375) Track MJ HashTable Load time
[
https://issues.apache.org/jira/browse/HIVE-23375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100679#comment-17100679
]
Zoltan Haindrich commented on HIVE-23375:
-
what happens in case there are 2 tasks loading the same hashtable? IIRC these
counters are aggregated; so we will see {{loadTime_1+loadTime_2}} is that ok?
...hmm...I'm wondering if it would be possible to introduce a tez counter which
could report the min/max/avg/var for the actual value - which could give
insight into what kind of values it have aggregated
> Track MJ HashTable Load time
>
>
> Key: HIVE-23375
> URL: https://issues.apache.org/jira/browse/HIVE-23375
> Project: Hive
> Issue Type: Improvement
>Reporter: Panagiotis Garefalakis
>Assignee: Panagiotis Garefalakis
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-23375.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Introduce TezCounter to track MJ HashTable Load time
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23375) Track MJ HashTable Load time
[ https://issues.apache.org/jira/browse/HIVE-23375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-23375: -- Attachment: HIVE-23375.01.patch > Track MJ HashTable Load time > > > Key: HIVE-23375 > URL: https://issues.apache.org/jira/browse/HIVE-23375 > Project: Hive > Issue Type: Improvement >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23375.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Introduce TezCounter to track MJ HashTable Load time -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23375) Track MJ HashTable Load time
[ https://issues.apache.org/jira/browse/HIVE-23375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100677#comment-17100677 ] Zoltan Haindrich commented on HIVE-23375: - [~pgaref]: if you could also add a counter for the used space estimation; that would be extremely usefull in some cases > Track MJ HashTable Load time > > > Key: HIVE-23375 > URL: https://issues.apache.org/jira/browse/HIVE-23375 > Project: Hive > Issue Type: Improvement >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23375.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Introduce TezCounter to track MJ HashTable Load time -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23375) Track MJ HashTable Load time
[ https://issues.apache.org/jira/browse/HIVE-23375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-23375: -- Attachment: HIVE-23375.01.patch > Track MJ HashTable Load time > > > Key: HIVE-23375 > URL: https://issues.apache.org/jira/browse/HIVE-23375 > Project: Hive > Issue Type: Improvement >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Introduce TezCounter to track MJ HashTable Load time -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23375) Track MJ HashTable Load time
[ https://issues.apache.org/jira/browse/HIVE-23375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-23375: -- Attachment: (was: HIVE-23375.01.patch) > Track MJ HashTable Load time > > > Key: HIVE-23375 > URL: https://issues.apache.org/jira/browse/HIVE-23375 > Project: Hive > Issue Type: Improvement >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Introduce TezCounter to track MJ HashTable Load time -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23375) Track MJ HashTable Load time
[ https://issues.apache.org/jira/browse/HIVE-23375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-23375: -- Labels: pull-request-available (was: ) > Track MJ HashTable Load time > > > Key: HIVE-23375 > URL: https://issues.apache.org/jira/browse/HIVE-23375 > Project: Hive > Issue Type: Improvement >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Introduce TezCounter to track MJ HashTable Load time -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23375) Track MJ HashTable Load time
[ https://issues.apache.org/jira/browse/HIVE-23375?focusedWorklogId=431168=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-431168 ] ASF GitHub Bot logged work on HIVE-23375: - Author: ASF GitHub Bot Created on: 06/May/20 11:02 Start Date: 06/May/20 11:02 Worklog Time Spent: 10m Work Description: pgaref opened a new pull request #1005: URL: https://github.com/apache/hive/pull/1005 Change-Id: I277696aa46b8a6343f53c36f773ee53575f513ac This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 431168) Remaining Estimate: 0h Time Spent: 10m > Track MJ HashTable Load time > > > Key: HIVE-23375 > URL: https://issues.apache.org/jira/browse/HIVE-23375 > Project: Hive > Issue Type: Improvement >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > Introduce TezCounter to track MJ HashTable Load time -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23375) Track MJ HashTable Load time
[ https://issues.apache.org/jira/browse/HIVE-23375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis reassigned HIVE-23375: - > Track MJ HashTable Load time > > > Key: HIVE-23375 > URL: https://issues.apache.org/jira/browse/HIVE-23375 > Project: Hive > Issue Type: Improvement >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Minor > > Introduce TezCounter to track MJ HashTable Load time -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23325) Clean up cleanup tasks for TxnHandler/CompactionTxnHandler
[
https://issues.apache.org/jira/browse/HIVE-23325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100671#comment-17100671
]
Hive QA commented on HIVE-23325:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002135/HIVE-23325.7.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22157/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22157/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22157/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-05-06 10:59:52.288
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-22157/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-05-06 10:59:52.291
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 48c0110 HIVE-23309: Lazy Initialization of Hadoop Shims (Aasha
Medhi, reviewed by Anishek Agarwal)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 48c0110 HIVE-23309: Lazy Initialization of Hadoop Shims (Aasha
Medhi, reviewed by Anishek Agarwal)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-05-06 10:59:53.011
+ rm -rf ../yetus_PreCommit-HIVE-Build-22157
+ mkdir ../yetus_PreCommit-HIVE-Build-22157
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-22157
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-22157/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java:110
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java' with
conflicts.
error: patch failed:
ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestInitiator.java:201
Falling back to three-way merge...
Applied patch to
'ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestInitiator.java' with
conflicts.
Going to apply patch with: git apply -p0
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java:110
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java' with
conflicts.
error: patch failed:
ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestInitiator.java:201
Falling back to three-way merge...
Applied patch to
'ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestInitiator.java' with
conflicts.
U ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java
U ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestInitiator.java
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-22157
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002135 - PreCommit-HIVE-Build
> Clean up cleanup tasks for TxnHandler/CompactionTxnHandler
> --
>
> Key: HIVE-23325
> URL: https://issues.apache.org/jira/browse/HIVE-23325
> Project: Hive
> Issue Type: Improvement
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Attachments: HIVE-23325.1.patch, HIVE-23325.2.patch,
> HIVE-23325.2.patch, HIVE-23325.2.patch, HIVE-23325.3.patch,
> HIVE-23325.4.patch, HIVE-23325.5.patch, HIVE-23325.5.patch,
> HIVE-23325.6.patch,
[jira] [Commented] (HIVE-23338) Bump jackson-databind version up to 2.9.10.4
[
https://issues.apache.org/jira/browse/HIVE-23338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100667#comment-17100667
]
Hive QA commented on HIVE-23338:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002134/HIVE-23338.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17216 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22156/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22156/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22156/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002134 - PreCommit-HIVE-Build
> Bump jackson-databind version up to 2.9.10.4
>
>
> Key: HIVE-23338
> URL: https://issues.apache.org/jira/browse/HIVE-23338
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23338.01.patch, HIVE-23338.01.patch,
> HIVE-23338.01.patch, HIVE-23338.01.patch
>
>
> com.fasterxml.jackson.core:jackson-databind:2.9.9 is exploitable.
> And exclude a transitive dependency on
> com.fasterxml.jackson.core:jackson-databind:2.6.5, which is also exploitable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23351) Ranger Replication Scheduling
[
https://issues.apache.org/jira/browse/HIVE-23351?focusedWorklogId=431151=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-431151
]
ASF GitHub Bot logged work on HIVE-23351:
-
Author: ASF GitHub Bot
Created on: 06/May/20 10:13
Start Date: 06/May/20 10:13
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on a change in pull request #1004:
URL: https://github.com/apache/hive/pull/1004#discussion_r420680172
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##
@@ -144,6 +145,11 @@ public int execute() {
if (shouldDump(previousValidHiveDumpPath)) {
Path currentDumpPath = getCurrentDumpPath(dumpRoot, isBootstrap);
Path hiveDumpRoot = new Path(currentDumpPath,
ReplUtils.REPL_HIVE_BASE_DIR);
+ work.setCurrentDumpPath(currentDumpPath);
+ if (shouldDumpAuthorizationMetadata()) {
+LOG.info("Dumping authorization data");
+initiateAuthorizationDumpTask(currentDumpPath);
Review comment:
If you don't return here, this will run only after current repl dump
execution?
What is the order of execution in the current case?
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##
@@ -170,6 +175,31 @@ public int execute() {
return 0;
}
+ private void initiateAuthorizationDumpTask(Path currentDumpPath) throws
SemanticException, IOException {
+if
(RANGER_AUTHORIZER.equalsIgnoreCase(conf.getVar(HiveConf.ConfVars.REPL_AUTHORIZATION_PROVIDER_SERVICE)))
{
+ Path rangerDumpRoot = new Path(currentDumpPath,
ReplUtils.REPL_RANGER_BASE_DIR);
+ FileSystem fs = rangerDumpRoot.getFileSystem(conf);
+ if (fs.exists(new Path(rangerDumpRoot,
ReplAck.RANGER_DUMP_ACKNOWLEDGEMENT.toString( {
+LOG.info("Ranger Authorization Metadata is already exported at {} ",
rangerDumpRoot);
+ } else {
+LOG.info("Exporting Authorization Metadata at {} ", rangerDumpRoot);
+RangerDumpWork rangerDumpWork = new RangerDumpWork(rangerDumpRoot,
work.dbNameOrPattern);
+Task rangerDumpTask = TaskFactory.get(rangerDumpWork,
conf);
+if (childTasks == null) {
+ childTasks = new ArrayList<>();
+}
+childTasks.add(rangerDumpTask);
Review comment:
Not getting added to the DAG, how does it get executed?
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplAck.java
##
@@ -23,7 +23,9 @@
public enum ReplAck {
DUMP_ACKNOWLEDGEMENT("_finished_dump"),
EVENTS_DUMP("_events_dump"),
-LOAD_ACKNOWLEDGEMENT("_finished_load");
+LOAD_ACKNOWLEDGEMENT("_finished_load"),
+RANGER_DUMP_ACKNOWLEDGEMENT("_finished_ranger_dump"),
Review comment:
We can use same DUMP_ACKNOWLEDGEMENT LOAD_ACKNOWLEDGEMENT
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ranger/RangerBaseModelObject.java
##
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.repl.ranger;
+
+import org.codehaus.jackson.annotate.JsonAutoDetect;
+import org.codehaus.jackson.annotate.JsonAutoDetect.Visibility;
+import org.codehaus.jackson.annotate.JsonIgnoreProperties;
+import org.codehaus.jackson.map.annotate.JsonSerialize;
+
+import javax.xml.bind.annotation.XmlAccessType;
+import javax.xml.bind.annotation.XmlAccessorType;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Date;
+
+/** RangerBaseModelObject class to contain common attributes of Ranger Base
object.
+*
+*/
+@JsonAutoDetect(getterVisibility=Visibility.NONE,
setterVisibility=Visibility.NONE, fieldVisibility=Visibility.ANY)
+@JsonSerialize(include=JsonSerialize.Inclusion.NON_NULL)
+@JsonIgnoreProperties(ignoreUnknown=true)
+@XmlRootElement
+@XmlAccessorType(XmlAccessType.FIELD)
+public class RangerBaseModelObject implements java.io.Serializable {
+private static final long serialVersionUID = 1L;
+
+private Longid;
+private String guid;
+private Boolean isEnabled;
+private String createdBy;
+
