[jira] [Commented] (HIVE-22753) Fix gradual mem leak: Operationlog related appenders should be cleared up on errors
[ https://issues.apache.org/jira/browse/HIVE-22753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020863#comment-17020863 ] Rajesh Balamohan commented on HIVE-22753: - Another way could be to add "IdlePurgePolicy" to "RoutingAppender" to get rid of unused appenders. > Fix gradual mem leak: Operationlog related appenders should be cleared up on > errors > > > Key: HIVE-22753 > URL: https://issues.apache.org/jira/browse/HIVE-22753 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-22753.1.patch, image-2020-01-21-11-14-37-911.png, > image-2020-01-21-11-17-59-279.png, image-2020-01-21-11-18-37-294.png > > > In case of exception in SQLOperation, operational log does not get cleared > up. This causes gradual build up of HushableRandomAccessFileAppender causing > HS2 to OOM after some time. > !image-2020-01-21-11-14-37-911.png|width=431,height=267! > > Allocation tree > !image-2020-01-21-11-18-37-294.png|width=425,height=178! > > Prod instance mem > !image-2020-01-21-11-17-59-279.png|width=698,height=209! > > Each HushableRandomAccessFileAppender holds internal ref to > RandomAccessFileAppender which holds a 256 KB bytebuffer, causing the mem > leak. > Related ticket: HIVE-18820 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22736) Support replication across multiple encryption zones
[
https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020857#comment-17020857
]
Hive QA commented on HIVE-22736:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991417/HIVE-22736.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 460 failed/errored test(s), 15828 tests
executed
*Failed tests:*
{noformat}
TestAddPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestAlterPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestAppendPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestCatalogNonDefaultClient - did not produce a TEST-*.xml file (likely timed
out) (batchId=227)
TestCatalogOldClient - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestCheckConstraint - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestDataSourceProviderFactory - did not produce a TEST-*.xml file (likely timed
out) (batchId=238)
TestDatabases - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestDefaultConstraint - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestDropPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestExchangePartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestFilterHooks - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestForeignKey - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestFunctions - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestGetTableMeta - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestHLLNoBias - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestHLLSerialization - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestHiveAlterHandler - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestHiveMetaStorePartitionSpecs - did not produce a TEST-*.xml file (likely
timed out) (batchId=229)
TestHiveMetaStoreTimeout - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestHiveMetastoreCli - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestHyperLogLogMerge - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestHyperLogLogSparse - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestLdapAtnProviderWithMiniDS - did not produce a TEST-*.xml file (likely timed
out) (batchId=238)
TestListPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestLockRequestBuilder - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestMetaStoreEndFunctionListener - did not produce a TEST-*.xml file (likely
timed out) (batchId=229)
TestMetaStoreEventListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=234)
TestMetaStoreEventListenerWithOldConf - did not produce a TEST-*.xml file
(likely timed out) (batchId=238)
TestMetastoreConf - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestMetastoreScheduledQueries - did not produce a TEST-*.xml file (likely timed
out) (batchId=229)
TestMetrics - did not produce a TEST-*.xml file (likely timed out) (batchId=229)
TestNotNullConstraint - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestObjectStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestObjectStoreSchemaMethods - did not produce a TEST-*.xml file (likely timed
out) (batchId=238)
TestObjectStoreStatementVerify - did not produce a TEST-*.xml file (likely
timed out) (batchId=229)
TestPartitionManagement - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestPartitionNameWhitelistValidation - did not produce a TEST-*.xml file
(likely timed out) (batchId=229)
TestPartitionProjectionEvaluator - did not produce a TEST-*.xml file (likely
timed out) (batchId=238)
TestPrimaryKey - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestRawStoreProxy - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestRemoteHiveMetaStoreZKBindHost - did not produce a TEST-*.xml file (likely
timed out) (batchId=238)
TestRetriesInRetryingHMSHandler - did not produce a TEST-*.xml file (likely
timed out) (batchId=238)
TestRuntimeStats - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestStats - did not produce a TEST-*.xml file (likely timed out) (batchId=229)
TestTablesCreateDropAlterTruncate - did not produce a TEST-*.xml file (likely
timed out) (batchId=227)
TestTablesList - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestTxnHandlerNegative - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
[jira] [Updated] (HIVE-22666) Introduce TopNKey operator for PTF Reduce Sink
[
https://issues.apache.org/jira/browse/HIVE-22666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22666:
--
Status: Patch Available (was: Open)
> Introduce TopNKey operator for PTF Reduce Sink
> --
>
> Key: HIVE-22666
> URL: https://issues.apache.org/jira/browse/HIVE-22666
> Project: Hive
> Issue Type: Improvement
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-22666.1.patch, HIVE-22666.2.patch,
> HIVE-22666.3.patch, HIVE-22666.3.patch, HIVE-22666.4.patch,
> HIVE-22666.4.patch, HIVE-22666.4.patch, HIVE-22666.4.patch, HIVE-22666.5.patch
>
>
> {code}
> EXPLAIN EXTENDED
> SELECT s_state, ranking
> FROM (
> SELECT s_state AS s_state,
> rank() OVER (PARTITION BY s_state ORDER BY ss_net_profit) AS ranking
> FROM testtable_n1000) tmp1
> WHERE ranking <= 3;
> {code}
> {code}
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Tez
> A masked pattern was here
> Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> A masked pattern was here
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: testtable_n1000
> Statistics: Num rows: 10 Data size: 940 Basic stats:
> COMPLETE Column stats: COMPLETE
> GatherStats: false
> Reduce Output Operator
> key expressions: s_state (type: string), ss_net_profit
> (type: double)
> null sort order: az
> sort order: ++
> Map-reduce partition columns: s_state (type: string)
> Statistics: Num rows: 10 Data size: 940 Basic stats:
> COMPLETE Column stats: COMPLETE
> tag: -1
> TopN: 4
> TopN Hash Memory Usage: 0.1
> auto parallelism: true
> Execution mode: vectorized, llap
> LLAP IO: no inputs
> Path -> Alias:
> A masked pattern was here
> Path -> Partition:
> A masked pattern was here
> Partition
> base file name: testtable_n1000
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> properties:
> COLUMN_STATS_ACCURATE
> {"BASIC_STATS":"true","COLUMN_STATS":{"s_state":"true","ss_net_profit":"true"}}
> bucket_count -1
> bucketing_version 2
> column.name.delimiter ,
> columns s_state,ss_net_profit
> columns.comments
> columns.types string:double
> A masked pattern was here
> name default.testtable_n1000
> numFiles 1
> numRows 10
> rawDataSize 80
> serialization.ddl struct testtable_n1000 { string
> s_state, double ss_net_profit}
> serialization.format 1
> serialization.lib
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> totalSize 90
> A masked pattern was here
> serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
>
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> properties:
> COLUMN_STATS_ACCURATE
> {"BASIC_STATS":"true","COLUMN_STATS":{"s_state":"true","ss_net_profit":"true"}}
> bucket_count -1
> bucketing_version 2
> column.name.delimiter ,
> columns s_state,ss_net_profit
> columns.comments
> columns.types string:double
> A masked pattern was here
> name default.testtable_n1000
> numFiles 1
> numRows 10
> rawDataSize 80
> serialization.ddl struct testtable_n1000 { string
> s_state, double ss_net_profit}
> serialization.format 1
> serialization.lib
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> totalSize 90
> A masked pattern was here
> serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> name: default.testtable_n1000
> name:
[jira] [Updated] (HIVE-22666) Introduce TopNKey operator for PTF Reduce Sink
[
https://issues.apache.org/jira/browse/HIVE-22666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22666:
--
Attachment: HIVE-22666.5.patch
> Introduce TopNKey operator for PTF Reduce Sink
> --
>
> Key: HIVE-22666
> URL: https://issues.apache.org/jira/browse/HIVE-22666
> Project: Hive
> Issue Type: Improvement
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-22666.1.patch, HIVE-22666.2.patch,
> HIVE-22666.3.patch, HIVE-22666.3.patch, HIVE-22666.4.patch,
> HIVE-22666.4.patch, HIVE-22666.4.patch, HIVE-22666.4.patch, HIVE-22666.5.patch
>
>
> {code}
> EXPLAIN EXTENDED
> SELECT s_state, ranking
> FROM (
> SELECT s_state AS s_state,
> rank() OVER (PARTITION BY s_state ORDER BY ss_net_profit) AS ranking
> FROM testtable_n1000) tmp1
> WHERE ranking <= 3;
> {code}
> {code}
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Tez
> A masked pattern was here
> Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> A masked pattern was here
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: testtable_n1000
> Statistics: Num rows: 10 Data size: 940 Basic stats:
> COMPLETE Column stats: COMPLETE
> GatherStats: false
> Reduce Output Operator
> key expressions: s_state (type: string), ss_net_profit
> (type: double)
> null sort order: az
> sort order: ++
> Map-reduce partition columns: s_state (type: string)
> Statistics: Num rows: 10 Data size: 940 Basic stats:
> COMPLETE Column stats: COMPLETE
> tag: -1
> TopN: 4
> TopN Hash Memory Usage: 0.1
> auto parallelism: true
> Execution mode: vectorized, llap
> LLAP IO: no inputs
> Path -> Alias:
> A masked pattern was here
> Path -> Partition:
> A masked pattern was here
> Partition
> base file name: testtable_n1000
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> properties:
> COLUMN_STATS_ACCURATE
> {"BASIC_STATS":"true","COLUMN_STATS":{"s_state":"true","ss_net_profit":"true"}}
> bucket_count -1
> bucketing_version 2
> column.name.delimiter ,
> columns s_state,ss_net_profit
> columns.comments
> columns.types string:double
> A masked pattern was here
> name default.testtable_n1000
> numFiles 1
> numRows 10
> rawDataSize 80
> serialization.ddl struct testtable_n1000 { string
> s_state, double ss_net_profit}
> serialization.format 1
> serialization.lib
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> totalSize 90
> A masked pattern was here
> serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
>
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> properties:
> COLUMN_STATS_ACCURATE
> {"BASIC_STATS":"true","COLUMN_STATS":{"s_state":"true","ss_net_profit":"true"}}
> bucket_count -1
> bucketing_version 2
> column.name.delimiter ,
> columns s_state,ss_net_profit
> columns.comments
> columns.types string:double
> A masked pattern was here
> name default.testtable_n1000
> numFiles 1
> numRows 10
> rawDataSize 80
> serialization.ddl struct testtable_n1000 { string
> s_state, double ss_net_profit}
> serialization.format 1
> serialization.lib
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> totalSize 90
> A masked pattern was here
> serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> name: default.testtable_n1000
> name: default.testtable_n1000
[jira] [Updated] (HIVE-22666) Introduce TopNKey operator for PTF Reduce Sink
[
https://issues.apache.org/jira/browse/HIVE-22666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22666:
--
Status: Open (was: Patch Available)
> Introduce TopNKey operator for PTF Reduce Sink
> --
>
> Key: HIVE-22666
> URL: https://issues.apache.org/jira/browse/HIVE-22666
> Project: Hive
> Issue Type: Improvement
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-22666.1.patch, HIVE-22666.2.patch,
> HIVE-22666.3.patch, HIVE-22666.3.patch, HIVE-22666.4.patch,
> HIVE-22666.4.patch, HIVE-22666.4.patch, HIVE-22666.4.patch
>
>
> {code}
> EXPLAIN EXTENDED
> SELECT s_state, ranking
> FROM (
> SELECT s_state AS s_state,
> rank() OVER (PARTITION BY s_state ORDER BY ss_net_profit) AS ranking
> FROM testtable_n1000) tmp1
> WHERE ranking <= 3;
> {code}
> {code}
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Tez
> A masked pattern was here
> Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> A masked pattern was here
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: testtable_n1000
> Statistics: Num rows: 10 Data size: 940 Basic stats:
> COMPLETE Column stats: COMPLETE
> GatherStats: false
> Reduce Output Operator
> key expressions: s_state (type: string), ss_net_profit
> (type: double)
> null sort order: az
> sort order: ++
> Map-reduce partition columns: s_state (type: string)
> Statistics: Num rows: 10 Data size: 940 Basic stats:
> COMPLETE Column stats: COMPLETE
> tag: -1
> TopN: 4
> TopN Hash Memory Usage: 0.1
> auto parallelism: true
> Execution mode: vectorized, llap
> LLAP IO: no inputs
> Path -> Alias:
> A masked pattern was here
> Path -> Partition:
> A masked pattern was here
> Partition
> base file name: testtable_n1000
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> properties:
> COLUMN_STATS_ACCURATE
> {"BASIC_STATS":"true","COLUMN_STATS":{"s_state":"true","ss_net_profit":"true"}}
> bucket_count -1
> bucketing_version 2
> column.name.delimiter ,
> columns s_state,ss_net_profit
> columns.comments
> columns.types string:double
> A masked pattern was here
> name default.testtable_n1000
> numFiles 1
> numRows 10
> rawDataSize 80
> serialization.ddl struct testtable_n1000 { string
> s_state, double ss_net_profit}
> serialization.format 1
> serialization.lib
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> totalSize 90
> A masked pattern was here
> serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
>
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> properties:
> COLUMN_STATS_ACCURATE
> {"BASIC_STATS":"true","COLUMN_STATS":{"s_state":"true","ss_net_profit":"true"}}
> bucket_count -1
> bucketing_version 2
> column.name.delimiter ,
> columns s_state,ss_net_profit
> columns.comments
> columns.types string:double
> A masked pattern was here
> name default.testtable_n1000
> numFiles 1
> numRows 10
> rawDataSize 80
> serialization.ddl struct testtable_n1000 { string
> s_state, double ss_net_profit}
> serialization.format 1
> serialization.lib
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> totalSize 90
> A masked pattern was here
> serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> name: default.testtable_n1000
> name: default.testtable_n1000
>
[jira] [Commented] (HIVE-22755) Cleaner/Compaction can skip the read locks and use the min open txn id
[ https://issues.apache.org/jira/browse/HIVE-22755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020832#comment-17020832 ] Peter Vary commented on HIVE-22755: --- I have 2 ideas: * Time based approach: use the start time of the minOpenTxn to filter the files to clean * Instead of skipping every read lock, create a read lock per table instead of per partition. This would fix some failures created by the previous patch with the added benefit of knowing what tables are under read. CC: [~lpinter], [~dkuzmenko] > Cleaner/Compaction can skip the read locks and use the min open txn id > -- > > Key: HIVE-22755 > URL: https://issues.apache.org/jira/browse/HIVE-22755 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Slim Bouguerra >Priority: Major > Fix For: 4.0.0 > > > The minOpenTxnId is used by the Cleaner here > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java#L154 > This currently converts it to open write-ids to clean appropriately. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22736) Support replication across multiple encryption zones
[
https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020826#comment-17020826
]
Hive QA commented on HIVE-22736:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
17s{color} | {color:blue} shims/common in master has 6 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
20s{color} | {color:blue} shims/0.23 in master has 7 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
19s{color} | {color:blue} standalone-metastore/metastore-common in master has
37 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
13s{color} | {color:blue} standalone-metastore/metastore-server in master has
181 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
50s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} shims/common: The patch generated 2 new + 94 unchanged
- 0 fixed = 96 total (was 94) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 3 new + 255 unchanged - 0
fixed = 258 total (was 255) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} itests/hive-unit: The patch generated 12 new + 0
unchanged - 0 fixed = 12 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
15s{color} | {color:red} standalone-metastore/metastore-server generated 1 new
+ 181 unchanged - 0 fixed = 182 total (was 181) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 9s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| |
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_database_core(RawStore,
String, String, boolean, boolean) makes inefficient use of keySet iterator
instead of entrySet iterator At HiveMetaStore.java:makes inefficient use of
keySet iterator instead of entrySet iterator At HiveMetaStore.java:[line 1774]
|
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
|
[jira] [Commented] (HIVE-22753) Fix gradual mem leak: Operationlog related appenders should be cleared up on errors
[
https://issues.apache.org/jira/browse/HIVE-22753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020794#comment-17020794
]
Rajesh Balamohan commented on HIVE-22753:
-
Did some more debugging.
1. https://issues.apache.org/jira/browse/HIVE-22733 does not fix this issue.
Observed mem leak with this fix as well.
2. HushableRandomAccessFileAppender stop() is getting invoked correctly as part
of "Operation.cleanupOperationLog --> LogUtils.stopQueryAppender".
However, due to some residual message in BatchEventProcessor,
"HushableRandomAccessFileAppender" with same filename gets immediately
recreated. This happens immediately after stop() is invoked. E.g
{noformat}
at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:136)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:958)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:898)
at
org.apache.logging.log4j.core.appender.routing.RoutingAppender.createAppender(RoutingAppender.java:271)
at
org.apache.logging.log4j.core.appender.routing.RoutingAppender.getControl(RoutingAppender.java:255)
at
org.apache.logging.log4j.core.appender.routing.RoutingAppender.append(RoutingAppender.java:225)
at
org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
at
org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
at
org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
at
org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
at
org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:448)
at
org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:433)
at
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417)
at
org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:79)
at
org.apache.logging.log4j.core.async.AsyncLogger.actualAsyncLog(AsyncLogger.java:380)
at
org.apache.logging.log4j.core.async.RingBufferLogEvent.execute(RingBufferLogEvent.java:152)
at
org.apache.logging.log4j.core.async.RingBufferLogEventHandler.onEvent(RingBufferLogEventHandler.java:45)
at
org.apache.logging.log4j.core.async.RingBufferLogEventHandler.onEvent(RingBufferLogEventHandler.java:29)
at
com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:129)
{noformat}
So this leaves the object forever in the map, causing the memory leak. Yet to
check how to prevent this from reinstantiated immediately.
> Fix gradual mem leak: Operationlog related appenders should be cleared up on
> errors
>
>
> Key: HIVE-22753
> URL: https://issues.apache.org/jira/browse/HIVE-22753
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-22753.1.patch, image-2020-01-21-11-14-37-911.png,
> image-2020-01-21-11-17-59-279.png, image-2020-01-21-11-18-37-294.png
>
>
> In case of exception in SQLOperation, operational log does not get cleared
> up. This causes gradual build up of HushableRandomAccessFileAppender causing
> HS2 to OOM after some time.
> !image-2020-01-21-11-14-37-911.png|width=431,height=267!
>
> Allocation tree
> !image-2020-01-21-11-18-37-294.png|width=425,height=178!
>
> Prod instance mem
> !image-2020-01-21-11-17-59-279.png|width=698,height=209!
>
> Each HushableRandomAccessFileAppender holds internal ref to
> RandomAccessFileAppender which holds a 256 KB bytebuffer, causing the mem
> leak.
> Related ticket: HIVE-18820
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-22736: --- Status: Patch Available (was: In Progress) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-22736: --- Status: In Progress (was: Patch Available) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-22736: --- Status: Patch Available (was: Open) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-22736: --- Status: Open (was: Patch Available) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera reassigned HIVE-22736: -- Assignee: mahesh kumar behera (was: Aasha Medhi) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22758) Create database with permission error when doas set to true
[
https://issues.apache.org/jira/browse/HIVE-22758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chiran Ravani updated HIVE-22758:
-
Issue Type: Bug (was: Improvement)
> Create database with permission error when doas set to true
> ---
>
> Key: HIVE-22758
> URL: https://issues.apache.org/jira/browse/HIVE-22758
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Chiran Ravani
>Assignee: Chiran Ravani
>Priority: Critical
> Attachments: HIVE-22758.1.patch
>
>
> With doAs set to true, running create database on external location fails due
> to permission denied to write on directory specified for hive user (User with
> HMS is running).
> Steps to reproduce the issue:
> 1. Turn on, Hive run as end-user to true.
> 2. Connect to hive as some user other than admin, eg:- chiran
> 3. Create a database with external location
> {code}
> create database externaldbexample location '/user/chiran/externaldbexample'
> {code}
> The above statement fails with HDFS write permission denied error as below.
> {code}
> > create database externaldbexample location '/user/chiran/externaldbexample';
> INFO : Compiling
> command(queryId=hive_20200122043626_5c95e1fd-ce00-45fd-b58d-54f5e579f87d):
> create database externaldbexample location '/user/chiran/externaldbexample'
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO : Completed compiling
> command(queryId=hive_20200122043626_5c95e1fd-ce00-45fd-b58d-54f5e579f87d);
> Time taken: 1.377 seconds
> INFO : Executing
> command(queryId=hive_20200122043626_5c95e1fd-ce00-45fd-b58d-54f5e579f87d):
> create database externaldbexample location '/user/chiran/externaldbexample'
> INFO : Starting task [Stage-0:DDL] in serial mode
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask.
> MetaException(message:java.lang.reflect.UndeclaredThrowableException)
> INFO : Completed executing
> command(queryId=hive_20200122043626_5c95e1fd-ce00-45fd-b58d-54f5e579f87d);
> Time taken: 0.238 seconds
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask.
> MetaException(message:java.lang.reflect.UndeclaredThrowableException)
> (state=08S01,code=1)
> {code}
> From Hive Metastore service log, below is seen.
> {code}
> 2020-01-22T04:36:27,870 WARN [pool-6-thread-6]: metastore.ObjectStore
> (ObjectStore.java:getDatabase(1010)) - Failed to get database
> hive.externaldbexample, returning NoSuchObjectExcept
> ion
> 2020-01-22T04:36:27,898 INFO [pool-6-thread-6]: metastore.HiveMetaStore
> (HiveMetaStore.java:run(1339)) - Creating database path in managed directory
> hdfs://c470-node2.squadron.support.
> hortonworks.com:8020/user/chiran/externaldbexample
> 2020-01-22T04:36:27,903 INFO [pool-6-thread-6]: utils.FileUtils
> (FileUtils.java:mkdir(170)) - Creating directory if it doesn't exist:
> hdfs://namenodeaddress:8020/user/chiran/externaldbexample
> 2020-01-22T04:36:27,932 ERROR [pool-6-thread-6]: utils.MetaStoreUtils
> (MetaStoreUtils.java:logAndThrowMetaException(169)) - Got exception:
> org.apache.hadoop.security.AccessControlException Permission denied:
> user=hive, access=WRITE, inode="/user/chiran":chiran:chiran:drwxr-xr-x
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:399)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:255)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:193)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1859)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1843)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1802)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:59)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3150)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1126)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:707)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
>
[jira] [Updated] (HIVE-22758) Create database with permission error when doas set to true
[
https://issues.apache.org/jira/browse/HIVE-22758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chiran Ravani updated HIVE-22758:
-
Attachment: HIVE-22758.1.patch
Status: Patch Available (was: Open)
> Create database with permission error when doas set to true
> ---
>
> Key: HIVE-22758
> URL: https://issues.apache.org/jira/browse/HIVE-22758
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.0, 3.0.0
>Reporter: Chiran Ravani
>Assignee: Chiran Ravani
>Priority: Critical
> Attachments: HIVE-22758.1.patch
>
>
> With doAs set to true, running create database on external location fails due
> to permission denied to write on directory specified for hive user (User with
> HMS is running).
> Steps to reproduce the issue:
> 1. Turn on, Hive run as end-user to true.
> 2. Connect to hive as some user other than admin, eg:- chiran
> 3. Create a database with external location
> {code}
> create database externaldbexample location '/user/chiran/externaldbexample'
> {code}
> The above statement fails with HDFS write permission denied error as below.
> {code}
> > create database externaldbexample location '/user/chiran/externaldbexample';
> INFO : Compiling
> command(queryId=hive_20200122043626_5c95e1fd-ce00-45fd-b58d-54f5e579f87d):
> create database externaldbexample location '/user/chiran/externaldbexample'
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO : Completed compiling
> command(queryId=hive_20200122043626_5c95e1fd-ce00-45fd-b58d-54f5e579f87d);
> Time taken: 1.377 seconds
> INFO : Executing
> command(queryId=hive_20200122043626_5c95e1fd-ce00-45fd-b58d-54f5e579f87d):
> create database externaldbexample location '/user/chiran/externaldbexample'
> INFO : Starting task [Stage-0:DDL] in serial mode
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask.
> MetaException(message:java.lang.reflect.UndeclaredThrowableException)
> INFO : Completed executing
> command(queryId=hive_20200122043626_5c95e1fd-ce00-45fd-b58d-54f5e579f87d);
> Time taken: 0.238 seconds
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask.
> MetaException(message:java.lang.reflect.UndeclaredThrowableException)
> (state=08S01,code=1)
> {code}
> From Hive Metastore service log, below is seen.
> {code}
> 2020-01-22T04:36:27,870 WARN [pool-6-thread-6]: metastore.ObjectStore
> (ObjectStore.java:getDatabase(1010)) - Failed to get database
> hive.externaldbexample, returning NoSuchObjectExcept
> ion
> 2020-01-22T04:36:27,898 INFO [pool-6-thread-6]: metastore.HiveMetaStore
> (HiveMetaStore.java:run(1339)) - Creating database path in managed directory
> hdfs://c470-node2.squadron.support.
> hortonworks.com:8020/user/chiran/externaldbexample
> 2020-01-22T04:36:27,903 INFO [pool-6-thread-6]: utils.FileUtils
> (FileUtils.java:mkdir(170)) - Creating directory if it doesn't exist:
> hdfs://namenodeaddress:8020/user/chiran/externaldbexample
> 2020-01-22T04:36:27,932 ERROR [pool-6-thread-6]: utils.MetaStoreUtils
> (MetaStoreUtils.java:logAndThrowMetaException(169)) - Got exception:
> org.apache.hadoop.security.AccessControlException Permission denied:
> user=hive, access=WRITE, inode="/user/chiran":chiran:chiran:drwxr-xr-x
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:399)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:255)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:193)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1859)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1843)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1802)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:59)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3150)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1126)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:707)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
>
[jira] [Updated] (HIVE-22758) Create database with permission error when doas set to true
[
https://issues.apache.org/jira/browse/HIVE-22758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chiran Ravani updated HIVE-22758:
-
Attachment: HIVE-22758.1.patch
> Create database with permission error when doas set to true
> ---
>
> Key: HIVE-22758
> URL: https://issues.apache.org/jira/browse/HIVE-22758
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Chiran Ravani
>Assignee: Chiran Ravani
>Priority: Critical
>
> With doAs set to true, running create database on external location fails due
> to permission denied to write on directory specified for hive user (User with
> HMS is running).
> Steps to reproduce the issue:
> 1. Turn on, Hive run as end-user to true.
> 2. Connect to hive as some user other than admin, eg:- chiran
> 3. Create a database with external location
> {code}
> create database externaldbexample location '/user/chiran/externaldbexample'
> {code}
> The above statement fails with HDFS write permission denied error as below.
> {code}
> > create database externaldbexample location '/user/chiran/externaldbexample';
> INFO : Compiling
> command(queryId=hive_20200122043626_5c95e1fd-ce00-45fd-b58d-54f5e579f87d):
> create database externaldbexample location '/user/chiran/externaldbexample'
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO : Completed compiling
> command(queryId=hive_20200122043626_5c95e1fd-ce00-45fd-b58d-54f5e579f87d);
> Time taken: 1.377 seconds
> INFO : Executing
> command(queryId=hive_20200122043626_5c95e1fd-ce00-45fd-b58d-54f5e579f87d):
> create database externaldbexample location '/user/chiran/externaldbexample'
> INFO : Starting task [Stage-0:DDL] in serial mode
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask.
> MetaException(message:java.lang.reflect.UndeclaredThrowableException)
> INFO : Completed executing
> command(queryId=hive_20200122043626_5c95e1fd-ce00-45fd-b58d-54f5e579f87d);
> Time taken: 0.238 seconds
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask.
> MetaException(message:java.lang.reflect.UndeclaredThrowableException)
> (state=08S01,code=1)
> {code}
> From Hive Metastore service log, below is seen.
> {code}
> 2020-01-22T04:36:27,870 WARN [pool-6-thread-6]: metastore.ObjectStore
> (ObjectStore.java:getDatabase(1010)) - Failed to get database
> hive.externaldbexample, returning NoSuchObjectExcept
> ion
> 2020-01-22T04:36:27,898 INFO [pool-6-thread-6]: metastore.HiveMetaStore
> (HiveMetaStore.java:run(1339)) - Creating database path in managed directory
> hdfs://c470-node2.squadron.support.
> hortonworks.com:8020/user/chiran/externaldbexample
> 2020-01-22T04:36:27,903 INFO [pool-6-thread-6]: utils.FileUtils
> (FileUtils.java:mkdir(170)) - Creating directory if it doesn't exist:
> hdfs://namenodeaddress:8020/user/chiran/externaldbexample
> 2020-01-22T04:36:27,932 ERROR [pool-6-thread-6]: utils.MetaStoreUtils
> (MetaStoreUtils.java:logAndThrowMetaException(169)) - Got exception:
> org.apache.hadoop.security.AccessControlException Permission denied:
> user=hive, access=WRITE, inode="/user/chiran":chiran:chiran:drwxr-xr-x
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:399)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:255)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:193)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1859)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1843)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1802)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:59)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3150)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1126)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:707)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at
[jira] [Updated] (HIVE-22758) Create database with permission error when doas set to true
[
https://issues.apache.org/jira/browse/HIVE-22758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chiran Ravani updated HIVE-22758:
-
Attachment: (was: HIVE-22758.1.patch)
> Create database with permission error when doas set to true
> ---
>
> Key: HIVE-22758
> URL: https://issues.apache.org/jira/browse/HIVE-22758
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Chiran Ravani
>Assignee: Chiran Ravani
>Priority: Critical
>
> With doAs set to true, running create database on external location fails due
> to permission denied to write on directory specified for hive user (User with
> HMS is running).
> Steps to reproduce the issue:
> 1. Turn on, Hive run as end-user to true.
> 2. Connect to hive as some user other than admin, eg:- chiran
> 3. Create a database with external location
> {code}
> create database externaldbexample location '/user/chiran/externaldbexample'
> {code}
> The above statement fails with HDFS write permission denied error as below.
> {code}
> > create database externaldbexample location '/user/chiran/externaldbexample';
> INFO : Compiling
> command(queryId=hive_20200122043626_5c95e1fd-ce00-45fd-b58d-54f5e579f87d):
> create database externaldbexample location '/user/chiran/externaldbexample'
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO : Completed compiling
> command(queryId=hive_20200122043626_5c95e1fd-ce00-45fd-b58d-54f5e579f87d);
> Time taken: 1.377 seconds
> INFO : Executing
> command(queryId=hive_20200122043626_5c95e1fd-ce00-45fd-b58d-54f5e579f87d):
> create database externaldbexample location '/user/chiran/externaldbexample'
> INFO : Starting task [Stage-0:DDL] in serial mode
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask.
> MetaException(message:java.lang.reflect.UndeclaredThrowableException)
> INFO : Completed executing
> command(queryId=hive_20200122043626_5c95e1fd-ce00-45fd-b58d-54f5e579f87d);
> Time taken: 0.238 seconds
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask.
> MetaException(message:java.lang.reflect.UndeclaredThrowableException)
> (state=08S01,code=1)
> {code}
> From Hive Metastore service log, below is seen.
> {code}
> 2020-01-22T04:36:27,870 WARN [pool-6-thread-6]: metastore.ObjectStore
> (ObjectStore.java:getDatabase(1010)) - Failed to get database
> hive.externaldbexample, returning NoSuchObjectExcept
> ion
> 2020-01-22T04:36:27,898 INFO [pool-6-thread-6]: metastore.HiveMetaStore
> (HiveMetaStore.java:run(1339)) - Creating database path in managed directory
> hdfs://c470-node2.squadron.support.
> hortonworks.com:8020/user/chiran/externaldbexample
> 2020-01-22T04:36:27,903 INFO [pool-6-thread-6]: utils.FileUtils
> (FileUtils.java:mkdir(170)) - Creating directory if it doesn't exist:
> hdfs://namenodeaddress:8020/user/chiran/externaldbexample
> 2020-01-22T04:36:27,932 ERROR [pool-6-thread-6]: utils.MetaStoreUtils
> (MetaStoreUtils.java:logAndThrowMetaException(169)) - Got exception:
> org.apache.hadoop.security.AccessControlException Permission denied:
> user=hive, access=WRITE, inode="/user/chiran":chiran:chiran:drwxr-xr-x
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:399)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:255)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:193)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1859)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1843)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1802)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:59)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3150)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1126)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:707)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
>
[jira] [Assigned] (HIVE-22758) Create database with permission error when doas set to true
[
https://issues.apache.org/jira/browse/HIVE-22758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chiran Ravani reassigned HIVE-22758:
> Create database with permission error when doas set to true
> ---
>
> Key: HIVE-22758
> URL: https://issues.apache.org/jira/browse/HIVE-22758
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.0, 3.0.0
>Reporter: Chiran Ravani
>Assignee: Chiran Ravani
>Priority: Critical
>
> With doAs set to true, running create database on external location fails due
> to permission denied to write on directory specified for hive user (User with
> HMS is running).
> Steps to reproduce the issue:
> 1. Turn on, Hive run as end-user to true.
> 2. Connect to hive as some user other than admin, eg:- chiran
> 3. Create a database with external location
> {code}
> create database externaldbexample location '/user/chiran/externaldbexample'
> {code}
> The above statement fails with HDFS write permission denied error as below.
> {code}
> > create database externaldbexample location '/user/chiran/externaldbexample';
> INFO : Compiling
> command(queryId=hive_20200122043626_5c95e1fd-ce00-45fd-b58d-54f5e579f87d):
> create database externaldbexample location '/user/chiran/externaldbexample'
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO : Completed compiling
> command(queryId=hive_20200122043626_5c95e1fd-ce00-45fd-b58d-54f5e579f87d);
> Time taken: 1.377 seconds
> INFO : Executing
> command(queryId=hive_20200122043626_5c95e1fd-ce00-45fd-b58d-54f5e579f87d):
> create database externaldbexample location '/user/chiran/externaldbexample'
> INFO : Starting task [Stage-0:DDL] in serial mode
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask.
> MetaException(message:java.lang.reflect.UndeclaredThrowableException)
> INFO : Completed executing
> command(queryId=hive_20200122043626_5c95e1fd-ce00-45fd-b58d-54f5e579f87d);
> Time taken: 0.238 seconds
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask.
> MetaException(message:java.lang.reflect.UndeclaredThrowableException)
> (state=08S01,code=1)
> {code}
> From Hive Metastore service log, below is seen.
> {code}
> 2020-01-22T04:36:27,870 WARN [pool-6-thread-6]: metastore.ObjectStore
> (ObjectStore.java:getDatabase(1010)) - Failed to get database
> hive.externaldbexample, returning NoSuchObjectExcept
> ion
> 2020-01-22T04:36:27,898 INFO [pool-6-thread-6]: metastore.HiveMetaStore
> (HiveMetaStore.java:run(1339)) - Creating database path in managed directory
> hdfs://c470-node2.squadron.support.
> hortonworks.com:8020/user/chiran/externaldbexample
> 2020-01-22T04:36:27,903 INFO [pool-6-thread-6]: utils.FileUtils
> (FileUtils.java:mkdir(170)) - Creating directory if it doesn't exist:
> hdfs://namenodeaddress:8020/user/chiran/externaldbexample
> 2020-01-22T04:36:27,932 ERROR [pool-6-thread-6]: utils.MetaStoreUtils
> (MetaStoreUtils.java:logAndThrowMetaException(169)) - Got exception:
> org.apache.hadoop.security.AccessControlException Permission denied:
> user=hive, access=WRITE, inode="/user/chiran":chiran:chiran:drwxr-xr-x
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:399)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:255)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:193)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1859)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1843)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1802)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:59)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3150)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1126)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:707)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at
[jira] [Commented] (HIVE-21334) Eviction of blocks is major source of blockage for allocation request. Allocation path need to be lock-free.
[
https://issues.apache.org/jira/browse/HIVE-21334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020750#comment-17020750
]
Slim Bouguerra commented on HIVE-21334:
---
The idea to amortize this operation is keep an IO thread local buffer stash of
the most frequent allocations.
In practice we should take advantage that all the allocation from a given split
will be the same size since the orc file buffer sizes is the same for all the
tiles.
> Eviction of blocks is major source of blockage for allocation request.
> Allocation path need to be lock-free.
>
>
> Key: HIVE-21334
> URL: https://issues.apache.org/jira/browse/HIVE-21334
> Project: Hive
> Issue Type: Improvement
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: lock_profile.png
>
>
> Eviction is getting in the way of memory allocation when the query fragment
> has no cache entry.
> This is cause major bottleneck and waist lot of cpu cycles.
> To fix this is first we can batch the evictions to avoid taking the lock
> multiple times.
> The memory manager need to be able to anticipate such issue and keep some
> spare space for queries that do not have any hit.
> {code}
> IO-Elevator-Thread-12 Blocked CPU usage on sample: 692ms
>
> org.apache.hadoop.hive.llap.cache.LowLevelLrfuCachePolicy.evictSomeBlocks(long)
> LowLevelLrfuCachePolicy.java:264
>
> org.apache.hadoop.hive.llap.cache.CacheContentsTracker.evictSomeBlocks(long)
> CacheContentsTracker.java:194
>
> org.apache.hadoop.hive.llap.cache.LowLevelCacheMemoryManager.reserveMemory(long,
> boolean, AtomicBoolean) LowLevelCacheMemoryManager.java:87
>
> org.apache.hadoop.hive.llap.cache.LowLevelCacheMemoryManager.reserveMemory(long,
> AtomicBoolean) LowLevelCacheMemoryManager.java:63
>
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(MemoryBuffer[],
> int, Allocator$BufferObjectFactory, AtomicBoolean) BuddyAllocator.java:263
>
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.allocateMultiple(MemoryBuffer[],
> int) EncodedReaderImpl.java:1295
>
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedStream(long,
> DiskRangeList, long, long, EncodedColumnBatch$ColumnStreamData, long, long,
> IdentityHashMap) EncodedReaderImpl.java:923
>
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedColumns(int,
> StripeInformation, OrcProto$RowIndex[], List, List, boolean[], boolean[],
> Consumer) EncodedReaderImpl.java:501
>
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead()
> OrcEncodedDataReader.java:407
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run()
> OrcEncodedDataReader.java:266
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run()
> OrcEncodedDataReader.java:263
> java.security.AccessController.doPrivileged(PrivilegedExceptionAction,
> AccessControlContext) AccessController.java (native)
> javax.security.auth.Subject.doAs(Subject, PrivilegedExceptionAction)
> Subject.java:422
>
> org.apache.hadoop.security.UserGroupInformation.doAs(PrivilegedExceptionAction)
> UserGroupInformation.java:1688
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal()
> OrcEncodedDataReader.java:263
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal()
> OrcEncodedDataReader.java:110
> org.apache.tez.common.CallableWithNdc.call() CallableWithNdc.java:36
>
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call()
> StatsRecordingThreadPool.java:110
> java.util.concurrent.FutureTask.run() FutureTask.java:266
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker)
> ThreadPoolExecutor.java:1142
> java.util.concurrent.ThreadPoolExecutor$Worker.run()
> ThreadPoolExecutor.java:617
> java.lang.Thread.run() Thread.java:745
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21334) Eviction of blocks is major source of blockage for allocation request. Allocation path need to be lock-free.
[
https://issues.apache.org/jira/browse/HIVE-21334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Slim Bouguerra updated HIVE-21334:
--
Component/s: llap
> Eviction of blocks is major source of blockage for allocation request.
> Allocation path need to be lock-free.
>
>
> Key: HIVE-21334
> URL: https://issues.apache.org/jira/browse/HIVE-21334
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: lock_profile.png
>
>
> Eviction is getting in the way of memory allocation when the query fragment
> has no cache entry.
> This is cause major bottleneck and waist lot of cpu cycles.
> To fix this is first we can batch the evictions to avoid taking the lock
> multiple times.
> The memory manager need to be able to anticipate such issue and keep some
> spare space for queries that do not have any hit.
> {code}
> IO-Elevator-Thread-12 Blocked CPU usage on sample: 692ms
>
> org.apache.hadoop.hive.llap.cache.LowLevelLrfuCachePolicy.evictSomeBlocks(long)
> LowLevelLrfuCachePolicy.java:264
>
> org.apache.hadoop.hive.llap.cache.CacheContentsTracker.evictSomeBlocks(long)
> CacheContentsTracker.java:194
>
> org.apache.hadoop.hive.llap.cache.LowLevelCacheMemoryManager.reserveMemory(long,
> boolean, AtomicBoolean) LowLevelCacheMemoryManager.java:87
>
> org.apache.hadoop.hive.llap.cache.LowLevelCacheMemoryManager.reserveMemory(long,
> AtomicBoolean) LowLevelCacheMemoryManager.java:63
>
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(MemoryBuffer[],
> int, Allocator$BufferObjectFactory, AtomicBoolean) BuddyAllocator.java:263
>
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.allocateMultiple(MemoryBuffer[],
> int) EncodedReaderImpl.java:1295
>
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedStream(long,
> DiskRangeList, long, long, EncodedColumnBatch$ColumnStreamData, long, long,
> IdentityHashMap) EncodedReaderImpl.java:923
>
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedColumns(int,
> StripeInformation, OrcProto$RowIndex[], List, List, boolean[], boolean[],
> Consumer) EncodedReaderImpl.java:501
>
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead()
> OrcEncodedDataReader.java:407
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run()
> OrcEncodedDataReader.java:266
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run()
> OrcEncodedDataReader.java:263
> java.security.AccessController.doPrivileged(PrivilegedExceptionAction,
> AccessControlContext) AccessController.java (native)
> javax.security.auth.Subject.doAs(Subject, PrivilegedExceptionAction)
> Subject.java:422
>
> org.apache.hadoop.security.UserGroupInformation.doAs(PrivilegedExceptionAction)
> UserGroupInformation.java:1688
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal()
> OrcEncodedDataReader.java:263
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal()
> OrcEncodedDataReader.java:110
> org.apache.tez.common.CallableWithNdc.call() CallableWithNdc.java:36
>
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call()
> StatsRecordingThreadPool.java:110
> java.util.concurrent.FutureTask.run() FutureTask.java:266
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker)
> ThreadPoolExecutor.java:1142
> java.util.concurrent.ThreadPoolExecutor$Worker.run()
> ThreadPoolExecutor.java:617
> java.lang.Thread.run() Thread.java:745
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22757) NullPointerException when executing SQLs
[
https://issues.apache.org/jira/browse/HIVE-22757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deegue updated HIVE-22757:
--
Attachment: HIVE-22757.patch
> NullPointerException when executing SQLs
>
>
> Key: HIVE-22757
> URL: https://issues.apache.org/jira/browse/HIVE-22757
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.6
>Reporter: Deegue
>Priority: Major
> Attachments: HIVE-22757.patch
>
>
> When executing SQL:
> {code:java}
> insert overwrite table ods.ods_1 partition(stat_day='20191209')
> select
> id
> ,user_id
> ,teacher_user_id
> ,partner_user_id
> ,order_id
> ,barcode
> ,sub_order_id
> ,item_id
> ,sales
> ,refund
> ,teacher_profit
> ,partner_profit
> ,teacher_refund_profit
> ,partner_refund_profit
> ,teacher_commission_value
> ,partner_commission_value
> ,biz_type
> ,pay_time
> ,item_profit_type
> ,black_mark
> ,is_deleted
> ,create_time
> ,modify_time
> from src.src_1
> where partition_date='20191209'
> union all
> select
> t1.id
> ,t1.user_id
> ,t1.teacher_user_id
> ,t1.partner_user_id
> ,t1.order_id
> ,t1.barcode
> ,t1.sub_order_id
> ,t1.item_id
> ,t1.sales
> ,t1.refund
> ,t1.teacher_profit
> ,t1.partner_profit
> ,t1.teacher_refund_profit
> ,t1.partner_refund_profit
> ,t1.teacher_commission_value
> ,t1.partner_commission_value
> ,t1.biz_type
> ,t1.pay_time
> ,t1.item_profit_type
> ,t1.black_mark
> ,t1.is_deleted
> ,t1.create_time
> ,t1.modify_time
> from
> (select *
> from ods.ods_1
> where stat_day='20191208'
> ) t1
> left join
> ( select order_id
> ,sub_order_id
> from src.src_1
> where partition_date='20191209'
> ) t2
> on t1.order_id=t2.order_id
> and t1.sub_order_id=t2.sub_order_id
> where t2.order_id is null
> {code}
> `java.lang.NullPointerException` thrown because the array list
> `neededNestedColumnPaths` haven't been inited when `addAll` method is invoked.
> {code:java}
> Launching Job 5 out of 5
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_1566481621886_4925755, Tracking URL =
> http://TXIDC65-bigdata-resourcemanager1:8042/proxy/application_1566481621886_4925755/
> Kill Command = /usr/local/yunji/hadoop/bin/hadoop job -kill
> job_1566481621886_4925755
> Hadoop job information for Stage-4: number of mappers: 1; number of reducers: > 0
> 2019-12-24 16:00:40,584 Stage-4 map = 0%, reduce = 0%
> 2019-12-24 16:01:40,956 Stage-4 map = 0%, reduce = 0%
> 2019-12-24 16:02:41,451 Stage-4 map = 0%, reduce = 0%
> 2019-12-24 16:02:45,550 Stage-4 map = 100%, reduce = 0%
> Ended Job = job_1566481621886_4925755 with errors
> Error during job, obtaining debugging information...
> Examining task ID: task_1566481621886_4925755_m_00 (and more) from job
> job_1566481621886_4925755
> Task with the most failures(4):
> -
> Task ID:
> task_1566481621886_4925755_m_00
> URL:
>
> http://TXIDC65-bigdata-resourcemanager1:8088/taskdetails.jsp?jobid=job_1566481621886_4925755=task_1566481621886_4925755_m_00
> -
> Diagnostic Messages for this Task:
> Error: java.io.IOException: java.lang.reflect.InvocationTargetException
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:271)
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.(HadoopShimsSecure.java:217)
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:345)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:695)
> at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.(MapTask.java:169)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:438)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at
[jira] [Commented] (HIVE-22753) Fix gradual mem leak: Operationlog related appenders should be cleared up on errors
[ https://issues.apache.org/jira/browse/HIVE-22753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020723#comment-17020723 ] Rajesh Balamohan commented on HIVE-22753: - [~maheshk114]: BTW, I have tried HIVE-22733 and issue persists. > Fix gradual mem leak: Operationlog related appenders should be cleared up on > errors > > > Key: HIVE-22753 > URL: https://issues.apache.org/jira/browse/HIVE-22753 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-22753.1.patch, image-2020-01-21-11-14-37-911.png, > image-2020-01-21-11-17-59-279.png, image-2020-01-21-11-18-37-294.png > > > In case of exception in SQLOperation, operational log does not get cleared > up. This causes gradual build up of HushableRandomAccessFileAppender causing > HS2 to OOM after some time. > !image-2020-01-21-11-14-37-911.png|width=431,height=267! > > Allocation tree > !image-2020-01-21-11-18-37-294.png|width=425,height=178! > > Prod instance mem > !image-2020-01-21-11-17-59-279.png|width=698,height=209! > > Each HushableRandomAccessFileAppender holds internal ref to > RandomAccessFileAppender which holds a 256 KB bytebuffer, causing the mem > leak. > Related ticket: HIVE-18820 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22754) Trim some extra HDFS find file name calls that can be deduced using current TXN watermark
[ https://issues.apache.org/jira/browse/HIVE-22754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-22754: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks, Slim! > Trim some extra HDFS find file name calls that can be deduced using current > TXN watermark > - > > Key: HIVE-22754 > URL: https://issues.apache.org/jira/browse/HIVE-22754 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Slim Bouguerra >Assignee: Slim Bouguerra >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22754.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22366) Multiple metastore calls for same table and constraints during planning
[
https://issues.apache.org/jira/browse/HIVE-22366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-22366:
---
Status: Patch Available (was: Open)
> Multiple metastore calls for same table and constraints during planning
> ---
>
> Key: HIVE-22366
> URL: https://issues.apache.org/jira/browse/HIVE-22366
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22366.1.patch, HIVE-22366.2.patch,
> HIVE-22366.3.patch, HIVE-22366.4.patch, HIVE-22366.5.patch
>
>
> e.g. for a simple query like
> {code:sql}
> explain select count(i) from test2 group by j limit 3;
> {code}
> planner makes the following calls:
> {code}
> 2019-10-17T22:44:49,892 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table :
> tbl=hive.default.test2
> 2019-10-17T22:44:49,908 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_primary_keys : tbl=hive.default.test2
> 2019-10-17T22:44:49,910 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default
> foreigntbl=test2
> 2019-10-17T22:44:49,911 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_unique_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:49,913 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_not_null_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:49,979 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_partitions : tbl=hive.default.test2
> 2019-10-17T22:44:49,997 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_aggr_stats_for: table=hive.default.test2
> 2019-10-17T22:44:50,014 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table :
> tbl=hive.default.test2
> 2019-10-17T22:44:50,023 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_primary_keys : tbl=hive.default.test2
> 2019-10-17T22:44:50,024 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default
> foreigntbl=test2
> 2019-10-17T22:44:50,025 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_unique_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:50,026 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_not_null_constraints : tbl=hive.default.test2
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22366) Multiple metastore calls for same table and constraints during planning
[
https://issues.apache.org/jira/browse/HIVE-22366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-22366:
---
Attachment: HIVE-22366.5.patch
> Multiple metastore calls for same table and constraints during planning
> ---
>
> Key: HIVE-22366
> URL: https://issues.apache.org/jira/browse/HIVE-22366
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22366.1.patch, HIVE-22366.2.patch,
> HIVE-22366.3.patch, HIVE-22366.4.patch, HIVE-22366.5.patch
>
>
> e.g. for a simple query like
> {code:sql}
> explain select count(i) from test2 group by j limit 3;
> {code}
> planner makes the following calls:
> {code}
> 2019-10-17T22:44:49,892 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table :
> tbl=hive.default.test2
> 2019-10-17T22:44:49,908 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_primary_keys : tbl=hive.default.test2
> 2019-10-17T22:44:49,910 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default
> foreigntbl=test2
> 2019-10-17T22:44:49,911 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_unique_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:49,913 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_not_null_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:49,979 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_partitions : tbl=hive.default.test2
> 2019-10-17T22:44:49,997 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_aggr_stats_for: table=hive.default.test2
> 2019-10-17T22:44:50,014 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table :
> tbl=hive.default.test2
> 2019-10-17T22:44:50,023 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_primary_keys : tbl=hive.default.test2
> 2019-10-17T22:44:50,024 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default
> foreigntbl=test2
> 2019-10-17T22:44:50,025 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_unique_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:50,026 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_not_null_constraints : tbl=hive.default.test2
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22366) Multiple metastore calls for same table and constraints during planning
[
https://issues.apache.org/jira/browse/HIVE-22366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-22366:
---
Status: Open (was: Patch Available)
> Multiple metastore calls for same table and constraints during planning
> ---
>
> Key: HIVE-22366
> URL: https://issues.apache.org/jira/browse/HIVE-22366
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22366.1.patch, HIVE-22366.2.patch,
> HIVE-22366.3.patch, HIVE-22366.4.patch, HIVE-22366.5.patch
>
>
> e.g. for a simple query like
> {code:sql}
> explain select count(i) from test2 group by j limit 3;
> {code}
> planner makes the following calls:
> {code}
> 2019-10-17T22:44:49,892 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table :
> tbl=hive.default.test2
> 2019-10-17T22:44:49,908 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_primary_keys : tbl=hive.default.test2
> 2019-10-17T22:44:49,910 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default
> foreigntbl=test2
> 2019-10-17T22:44:49,911 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_unique_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:49,913 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_not_null_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:49,979 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_partitions : tbl=hive.default.test2
> 2019-10-17T22:44:49,997 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_aggr_stats_for: table=hive.default.test2
> 2019-10-17T22:44:50,014 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table :
> tbl=hive.default.test2
> 2019-10-17T22:44:50,023 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_primary_keys : tbl=hive.default.test2
> 2019-10-17T22:44:50,024 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default
> foreigntbl=test2
> 2019-10-17T22:44:50,025 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_unique_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:50,026 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_not_null_constraints : tbl=hive.default.test2
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22754) Trim some extra HDFS find file name calls that can be deduced using current TXN watermark
[
https://issues.apache.org/jira/browse/HIVE-22754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020699#comment-17020699
]
Hive QA commented on HIVE-22754:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991476/HIVE-22754.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17934 tests
executed
*Failed tests:*
{noformat}
TestTriggersMoveWorkloadManager - did not produce a TEST-*.xml file (likely
timed out) (batchId=288)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20266/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20266/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20266/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991476 - PreCommit-HIVE-Build
> Trim some extra HDFS find file name calls that can be deduced using current
> TXN watermark
> -
>
> Key: HIVE-22754
> URL: https://issues.apache.org/jira/browse/HIVE-22754
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22754.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22754) Trim some extra HDFS find file name calls that can be deduced using current TXN watermark
[
https://issues.apache.org/jira/browse/HIVE-22754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020679#comment-17020679
]
Hive QA commented on HIVE-22754:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
30s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20266/dev-support/hive-personality.sh
|
| git revision | master / 6b43e36 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20266/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Trim some extra HDFS find file name calls that can be deduced using current
> TXN watermark
> -
>
> Key: HIVE-22754
> URL: https://issues.apache.org/jira/browse/HIVE-22754
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22754.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22745) Config option to turn off read locks
[ https://issues.apache.org/jira/browse/HIVE-22745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-22745: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks, Slim! > Config option to turn off read locks > > > Key: HIVE-22745 > URL: https://issues.apache.org/jira/browse/HIVE-22745 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Ashutosh Chauhan >Assignee: Slim Bouguerra >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22745.2.patch, HIVE-22745.3.patch, > HIVE-22745.4.patch, HIVE-22745.patch > > > Although its not recommended but in perf critical scenario this option may be > exercised. We have observed lock acquisition to take long time in heavily > loaded system. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22745) Config option to turn off read locks
[
https://issues.apache.org/jira/browse/HIVE-22745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020659#comment-17020659
]
Hive QA commented on HIVE-22745:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991475/HIVE-22745.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17938 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20265/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20265/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20265/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991475 - PreCommit-HIVE-Build
> Config option to turn off read locks
>
>
> Key: HIVE-22745
> URL: https://issues.apache.org/jira/browse/HIVE-22745
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Ashutosh Chauhan
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22745.2.patch, HIVE-22745.3.patch,
> HIVE-22745.4.patch, HIVE-22745.patch
>
>
> Although its not recommended but in perf critical scenario this option may be
> exercised. We have observed lock acquisition to take long time in heavily
> loaded system.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22745) Config option to turn off read locks
[ https://issues.apache.org/jira/browse/HIVE-22745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020643#comment-17020643 ] Slim Bouguerra commented on HIVE-22745: --- [~gopalv] done thanks. > Config option to turn off read locks > > > Key: HIVE-22745 > URL: https://issues.apache.org/jira/browse/HIVE-22745 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Ashutosh Chauhan >Assignee: Slim Bouguerra >Priority: Major > Attachments: HIVE-22745.2.patch, HIVE-22745.3.patch, > HIVE-22745.4.patch, HIVE-22745.patch > > > Although its not recommended but in perf critical scenario this option may be > exercised. We have observed lock acquisition to take long time in heavily > loaded system. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22745) Config option to turn off read locks
[ https://issues.apache.org/jira/browse/HIVE-22745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Slim Bouguerra updated HIVE-22745: -- Attachment: HIVE-22745.4.patch > Config option to turn off read locks > > > Key: HIVE-22745 > URL: https://issues.apache.org/jira/browse/HIVE-22745 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Ashutosh Chauhan >Assignee: Slim Bouguerra >Priority: Major > Attachments: HIVE-22745.2.patch, HIVE-22745.3.patch, > HIVE-22745.4.patch, HIVE-22745.patch > > > Although its not recommended but in perf critical scenario this option may be > exercised. We have observed lock acquisition to take long time in heavily > loaded system. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22745) Config option to turn off read locks
[
https://issues.apache.org/jira/browse/HIVE-22745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020633#comment-17020633
]
Hive QA commented on HIVE-22745:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
17s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20265/dev-support/hive-personality.sh
|
| git revision | master / 6b43e36 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20265/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Config option to turn off read locks
>
>
> Key: HIVE-22745
> URL: https://issues.apache.org/jira/browse/HIVE-22745
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Ashutosh Chauhan
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22745.2.patch, HIVE-22745.3.patch, HIVE-22745.patch
>
>
> Although its not recommended but in perf critical scenario this option may be
> exercised. We have observed lock acquisition to take long time in heavily
> loaded system.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22745) Config option to turn off read locks
[
https://issues.apache.org/jira/browse/HIVE-22745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020616#comment-17020616
]
Gopal Vijayaraghavan commented on HIVE-22745:
-
[~bslim]: use {{hive.txn.read.locks}} instead of hive.read.locks? (TxnManager
specific config)
> Config option to turn off read locks
>
>
> Key: HIVE-22745
> URL: https://issues.apache.org/jira/browse/HIVE-22745
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Ashutosh Chauhan
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22745.2.patch, HIVE-22745.3.patch, HIVE-22745.patch
>
>
> Although its not recommended but in perf critical scenario this option may be
> exercised. We have observed lock acquisition to take long time in heavily
> loaded system.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22755) Cleaner/Compaction can skip the read locks and use the min open txn id
[ https://issues.apache.org/jira/browse/HIVE-22755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal Vijayaraghavan updated HIVE-22755: Description: The minOpenTxnId is used by the Cleaner here https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java#L154 This currently converts it to open write-ids to clean appropriately. > Cleaner/Compaction can skip the read locks and use the min open txn id > -- > > Key: HIVE-22755 > URL: https://issues.apache.org/jira/browse/HIVE-22755 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Slim Bouguerra >Priority: Major > Fix For: 4.0.0 > > > The minOpenTxnId is used by the Cleaner here > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java#L154 > This currently converts it to open write-ids to clean appropriately. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22745) Config option to turn off read locks
[
https://issues.apache.org/jira/browse/HIVE-22745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020611#comment-17020611
]
Hive QA commented on HIVE-22745:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991387/HIVE-22745.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17937 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20264/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20264/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20264/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991387 - PreCommit-HIVE-Build
> Config option to turn off read locks
>
>
> Key: HIVE-22745
> URL: https://issues.apache.org/jira/browse/HIVE-22745
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Ashutosh Chauhan
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22745.2.patch, HIVE-22745.3.patch, HIVE-22745.patch
>
>
> Although its not recommended but in perf critical scenario this option may be
> exercised. We have observed lock acquisition to take long time in heavily
> loaded system.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22756) Drop XAttrs options for distcp when dest is a blobstore
[ https://issues.apache.org/jira/browse/HIVE-22756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22756: Attachment: HIVE-22756.1.patch Status: Patch Available (was: Open) > Drop XAttrs options for distcp when dest is a blobstore > --- > > Key: HIVE-22756 > URL: https://issues.apache.org/jira/browse/HIVE-22756 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22756.1.patch > > > Drop XAttrs options for distcp when dest is a blobstore -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22756) Drop XAttrs options for distcp when dest is a blobstore
[ https://issues.apache.org/jira/browse/HIVE-22756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan reassigned HIVE-22756: --- > Drop XAttrs options for distcp when dest is a blobstore > --- > > Key: HIVE-22756 > URL: https://issues.apache.org/jira/browse/HIVE-22756 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > > Drop XAttrs options for distcp when dest is a blobstore -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22755) Cleaner/Compaction can skip the read locks and use the min open txn id
[ https://issues.apache.org/jira/browse/HIVE-22755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020594#comment-17020594 ] Slim Bouguerra commented on HIVE-22755: --- cc [~t3rmin4t0r] please feel free to add more insights about your idea on how the Cleaner can skip the read lock. > Cleaner/Compaction can skip the read locks and use the min open txn id > -- > > Key: HIVE-22755 > URL: https://issues.apache.org/jira/browse/HIVE-22755 > Project: Hive > Issue Type: Sub-task >Reporter: Slim Bouguerra >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22755) Cleaner/Compaction can skip the read locks and use the min open txn id
[ https://issues.apache.org/jira/browse/HIVE-22755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Slim Bouguerra updated HIVE-22755: -- Fix Version/s: 4.0.0 > Cleaner/Compaction can skip the read locks and use the min open txn id > -- > > Key: HIVE-22755 > URL: https://issues.apache.org/jira/browse/HIVE-22755 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Slim Bouguerra >Priority: Major > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22755) Cleaner/Compaction can skip the read locks and use the min open txn id
[ https://issues.apache.org/jira/browse/HIVE-22755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Slim Bouguerra updated HIVE-22755: -- Component/s: Transactions > Cleaner/Compaction can skip the read locks and use the min open txn id > -- > > Key: HIVE-22755 > URL: https://issues.apache.org/jira/browse/HIVE-22755 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Slim Bouguerra >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22745) Config option to turn off read locks
[ https://issues.apache.org/jira/browse/HIVE-22745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020588#comment-17020588 ] Ashutosh Chauhan commented on HIVE-22745: - +1 [~bslim] can you please create and link follow-up jira for not to acquire locks for compaction? > Config option to turn off read locks > > > Key: HIVE-22745 > URL: https://issues.apache.org/jira/browse/HIVE-22745 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Ashutosh Chauhan >Assignee: Slim Bouguerra >Priority: Major > Attachments: HIVE-22745.2.patch, HIVE-22745.3.patch, HIVE-22745.patch > > > Although its not recommended but in perf critical scenario this option may be > exercised. We have observed lock acquisition to take long time in heavily > loaded system. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22754) Trim some extra HDFS find file name calls that can be deduced using current TXN watermark
[ https://issues.apache.org/jira/browse/HIVE-22754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020586#comment-17020586 ] Ashutosh Chauhan commented on HIVE-22754: - +1 > Trim some extra HDFS find file name calls that can be deduced using current > TXN watermark > - > > Key: HIVE-22754 > URL: https://issues.apache.org/jira/browse/HIVE-22754 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Slim Bouguerra >Assignee: Slim Bouguerra >Priority: Major > Attachments: HIVE-22754.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22745) Config option to turn off read locks
[
https://issues.apache.org/jira/browse/HIVE-22745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020576#comment-17020576
]
Hive QA commented on HIVE-22745:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
57s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20264/dev-support/hive-personality.sh
|
| git revision | master / 53c9c6a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20264/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Config option to turn off read locks
>
>
> Key: HIVE-22745
> URL: https://issues.apache.org/jira/browse/HIVE-22745
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Ashutosh Chauhan
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22745.2.patch, HIVE-22745.3.patch, HIVE-22745.patch
>
>
> Although its not recommended but in perf critical scenario this option may be
> exercised. We have observed lock acquisition to take long time in heavily
> loaded system.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22754) Trim some extra HDFS find file name calls that can be deduced using current TXN watermark
[ https://issues.apache.org/jira/browse/HIVE-22754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Slim Bouguerra updated HIVE-22754: -- Attachment: HIVE-22754.patch > Trim some extra HDFS find file name calls that can be deduced using current > TXN watermark > - > > Key: HIVE-22754 > URL: https://issues.apache.org/jira/browse/HIVE-22754 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Slim Bouguerra >Assignee: Slim Bouguerra >Priority: Major > Attachments: HIVE-22754.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22754) Trim some extra HDFS find file name calls that can be deduced using current TXN watermark
[ https://issues.apache.org/jira/browse/HIVE-22754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Slim Bouguerra updated HIVE-22754: -- Status: Patch Available (was: Open) > Trim some extra HDFS find file name calls that can be deduced using current > TXN watermark > - > > Key: HIVE-22754 > URL: https://issues.apache.org/jira/browse/HIVE-22754 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Slim Bouguerra >Assignee: Slim Bouguerra >Priority: Major > Attachments: HIVE-22754.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22754) Trim some extra HDFS find file name calls that can be deduced using current TXN watermark
[ https://issues.apache.org/jira/browse/HIVE-22754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Slim Bouguerra updated HIVE-22754: -- Summary: Trim some extra HDFS find file name calls that can be deduced using current TXN watermark (was: Trim some extra HDFS find file name calls that can be deduced using current TX watermark) > Trim some extra HDFS find file name calls that can be deduced using current > TXN watermark > - > > Key: HIVE-22754 > URL: https://issues.apache.org/jira/browse/HIVE-22754 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Slim Bouguerra >Assignee: Slim Bouguerra >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22754) Trim some extra HDFS find file name calls that can be deduced using current TX watermark
[ https://issues.apache.org/jira/browse/HIVE-22754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Slim Bouguerra reassigned HIVE-22754: - > Trim some extra HDFS find file name calls that can be deduced using current > TX watermark > > > Key: HIVE-22754 > URL: https://issues.apache.org/jira/browse/HIVE-22754 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Slim Bouguerra >Assignee: Slim Bouguerra >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22745) Config option to turn off read locks
[ https://issues.apache.org/jira/browse/HIVE-22745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Slim Bouguerra updated HIVE-22745: -- Attachment: HIVE-22745.3.patch > Config option to turn off read locks > > > Key: HIVE-22745 > URL: https://issues.apache.org/jira/browse/HIVE-22745 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Ashutosh Chauhan >Assignee: Slim Bouguerra >Priority: Major > Attachments: HIVE-22745.2.patch, HIVE-22745.3.patch, HIVE-22745.patch > > > Although its not recommended but in perf critical scenario this option may be > exercised. We have observed lock acquisition to take long time in heavily > loaded system. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22733) After disable operation log property in hive, still HS2 saving the operation log
[
https://issues.apache.org/jira/browse/HIVE-22733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-22733:
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master. Thanks, Mahesh!
> After disable operation log property in hive, still HS2 saving the operation
> log
>
>
> Key: HIVE-22733
> URL: https://issues.apache.org/jira/browse/HIVE-22733
> Project: Hive
> Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22733.01.patch, HIVE-22733.02.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> There are few issues in this area.
> 1. If logging is disabled using hive.server2.logging.operation.enabled, then
> operation logs for the queries should not be generated. But the
> registerLoggingContext method in LogUtils, registers the logging context
> even if the operation log is disabled. This causes the logs to be added by
> logger. The registration of query context should be done only if operation
> logging is enabled.
> {code:java}
> public static void registerLoggingContext(Configuration conf) {
> -MDC.put(SESSIONID_LOG_KEY, HiveConf.getVar(conf,
> HiveConf.ConfVars.HIVESESSIONID));
> -MDC.put(QUERYID_LOG_KEY, HiveConf.getVar(conf,
> HiveConf.ConfVars.HIVEQUERYID));
> if (HiveConf.getBoolVar(conf,
> HiveConf.ConfVars.HIVE_SERVER2_LOGGING_OPERATION_ENABLED)) {
> + MDC.put(SESSIONID_LOG_KEY, HiveConf.getVar(conf,
> HiveConf.ConfVars.HIVESESSIONID));
> + MDC.put(QUERYID_LOG_KEY, HiveConf.getVar(conf,
> HiveConf.ConfVars.HIVEQUERYID));
>MDC.put(OPERATIONLOG_LEVEL_KEY, HiveConf.getVar(conf,
> HiveConf.ConfVars.HIVE_SERVER2_LOGGING_OPERATION_LEVEL));{code}
>
> 2. In case of failed query, we close the operations and that deletes the
> logging context (appender and route) from logger for that query. But if any
> log is added after that, the query logs are getting added and new operation
> log file is getting generated for the query. This looks like issue with MCD
> clear. MCD clear is not removing the keys from the map. If remove is used
> instead of clear, its working fine.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22733) After disable operation log property in hive, still HS2 saving the operation log
[
https://issues.apache.org/jira/browse/HIVE-22733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020557#comment-17020557
]
Hive QA commented on HIVE-22733:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991337/HIVE-22733.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17938 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20263/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20263/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20263/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991337 - PreCommit-HIVE-Build
> After disable operation log property in hive, still HS2 saving the operation
> log
>
>
> Key: HIVE-22733
> URL: https://issues.apache.org/jira/browse/HIVE-22733
> Project: Hive
> Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22733.01.patch, HIVE-22733.02.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> There are few issues in this area.
> 1. If logging is disabled using hive.server2.logging.operation.enabled, then
> operation logs for the queries should not be generated. But the
> registerLoggingContext method in LogUtils, registers the logging context
> even if the operation log is disabled. This causes the logs to be added by
> logger. The registration of query context should be done only if operation
> logging is enabled.
> {code:java}
> public static void registerLoggingContext(Configuration conf) {
> -MDC.put(SESSIONID_LOG_KEY, HiveConf.getVar(conf,
> HiveConf.ConfVars.HIVESESSIONID));
> -MDC.put(QUERYID_LOG_KEY, HiveConf.getVar(conf,
> HiveConf.ConfVars.HIVEQUERYID));
> if (HiveConf.getBoolVar(conf,
> HiveConf.ConfVars.HIVE_SERVER2_LOGGING_OPERATION_ENABLED)) {
> + MDC.put(SESSIONID_LOG_KEY, HiveConf.getVar(conf,
> HiveConf.ConfVars.HIVESESSIONID));
> + MDC.put(QUERYID_LOG_KEY, HiveConf.getVar(conf,
> HiveConf.ConfVars.HIVEQUERYID));
>MDC.put(OPERATIONLOG_LEVEL_KEY, HiveConf.getVar(conf,
> HiveConf.ConfVars.HIVE_SERVER2_LOGGING_OPERATION_LEVEL));{code}
>
> 2. In case of failed query, we close the operations and that deletes the
> logging context (appender and route) from logger for that query. But if any
> log is added after that, the query logs are getting added and new operation
> log file is getting generated for the query. This looks like issue with MCD
> clear. MCD clear is not removing the keys from the map. If remove is used
> instead of clear, its working fine.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22733) After disable operation log property in hive, still HS2 saving the operation log
[
https://issues.apache.org/jira/browse/HIVE-22733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020524#comment-17020524
]
Hive QA commented on HIVE-22733:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} itests/hive-unit: The patch generated 2 new + 4
unchanged - 0 fixed = 6 total (was 4) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20263/dev-support/hive-personality.sh
|
| git revision | master / 53c9c6a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20263/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: common itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20263/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> After disable operation log property in hive, still HS2 saving the operation
> log
>
>
> Key: HIVE-22733
> URL: https://issues.apache.org/jira/browse/HIVE-22733
> Project: Hive
> Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22733.01.patch, HIVE-22733.02.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> There are few issues in this area.
> 1. If logging is disabled using hive.server2.logging.operation.enabled, then
> operation logs for the queries should not be generated. But the
> registerLoggingContext method in LogUtils, registers the logging context
> even if the operation log is disabled. This causes the logs to be added by
> logger. The registration of query context should be done only if operation
> logging is enabled.
>
[jira] [Updated] (HIVE-22629) AST Node Children can be quite expensive to build due to List resizing
[ https://issues.apache.org/jira/browse/HIVE-22629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Slim Bouguerra updated HIVE-22629: -- Attachment: HIVE-22629.3.patch > AST Node Children can be quite expensive to build due to List resizing > -- > > Key: HIVE-22629 > URL: https://issues.apache.org/jira/browse/HIVE-22629 > Project: Hive > Issue Type: Improvement >Reporter: Slim Bouguerra >Assignee: Slim Bouguerra >Priority: Major > Attachments: HIVE-22629.1.patch, HIVE-22629.2.patch, > HIVE-22629.3.patch, HIVE-22629.patch, > noETLs_ETLs_profile-kc-hdp-mstr06-p.servicemanagement.com-interactive-166620-t-e-cpu-1576029590.svg > > > As per the attached profile, The AST Node can be a major source of CPU and > memory churn, due to the ArrayList resizing and copy. > In my Opinion this can be amortized by providing the actual size. > [~jcamachorodriguez] / [~vgarg] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22558) Metastore: Passwords jceks should be read lazily, in case of connection pools
[ https://issues.apache.org/jira/browse/HIVE-22558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-22558: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks, Slim! > Metastore: Passwords jceks should be read lazily, in case of connection pools > - > > Key: HIVE-22558 > URL: https://issues.apache.org/jira/browse/HIVE-22558 > Project: Hive > Issue Type: Bug > Components: Metastore, Standalone Metastore >Reporter: Gopal Vijayaraghavan >Assignee: Slim Bouguerra >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22558.1.patch, HIVE-22558.2.patch, > getDatabase-password-md5-hotpath.png > > > The jceks file is parsed for every instance of the metastore conf to populate > the password in plain-text, which is irrelevant for the scenario where the DB > connection pool is already active. > !getDatabase-password-md5-hotpath.png|width=640! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22753) Fix gradual mem leak: Operationlog related appenders should be cleared up on errors
[ https://issues.apache.org/jira/browse/HIVE-22753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020492#comment-17020492 ] Slim Bouguerra commented on HIVE-22753: --- For the record this is a duplicate of this https://issues.apache.org/jira/browse/HIVE-22127. > Fix gradual mem leak: Operationlog related appenders should be cleared up on > errors > > > Key: HIVE-22753 > URL: https://issues.apache.org/jira/browse/HIVE-22753 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-22753.1.patch, image-2020-01-21-11-14-37-911.png, > image-2020-01-21-11-17-59-279.png, image-2020-01-21-11-18-37-294.png > > > In case of exception in SQLOperation, operational log does not get cleared > up. This causes gradual build up of HushableRandomAccessFileAppender causing > HS2 to OOM after some time. > !image-2020-01-21-11-14-37-911.png|width=431,height=267! > > Allocation tree > !image-2020-01-21-11-18-37-294.png|width=425,height=178! > > Prod instance mem > !image-2020-01-21-11-17-59-279.png|width=698,height=209! > > Each HushableRandomAccessFileAppender holds internal ref to > RandomAccessFileAppender which holds a 256 KB bytebuffer, causing the mem > leak. > Related ticket: HIVE-18820 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22745) Config option to turn off read locks
[ https://issues.apache.org/jira/browse/HIVE-22745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020475#comment-17020475 ] Gopal Vijayaraghavan commented on HIVE-22745: - HIVE-20801 is pretty much complete and has a +1, needs a rebase + commit. This config name needs to be switched over to indicate it relies on the Cleaner (let me get a WIP for that today & send it to you). So these two tie to this config https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java#L258 + https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java#L335 So even without readlocks we can have a cleaner work safely by using long minUncommittedTxnId = findMinOpenTxnGLB(stmt); to slow the cleaner pass. > Config option to turn off read locks > > > Key: HIVE-22745 > URL: https://issues.apache.org/jira/browse/HIVE-22745 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Ashutosh Chauhan >Assignee: Slim Bouguerra >Priority: Major > Attachments: HIVE-22745.2.patch, HIVE-22745.patch > > > Although its not recommended but in perf critical scenario this option may be > exercised. We have observed lock acquisition to take long time in heavily > loaded system. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[ https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal Vijayaraghavan reassigned HIVE-20801: --- Assignee: Gopal Vijayaraghavan (was: Gopal V) > ACID: Allow DbTxnManager to ignore non-ACID table locking > - > > Key: HIVE-20801 > URL: https://issues.apache.org/jira/browse/HIVE-20801 > Project: Hive > Issue Type: Bug > Components: Locking, Transactions >Affects Versions: 4.0.0 >Reporter: Gopal Vijayaraghavan >Assignee: Gopal Vijayaraghavan >Priority: Major > Labels: Branch3Candidate, TODOC > Attachments: HIVE-20801.1.patch, HIVE-20801.2.patch, > HIVE-20801.2.patch, HIVE-20801.3.patch, HIVE-20801.3.patch > > > Enabling ACIDv1 on a cluster produces a central locking bottleneck for all > table types, which is not always the intention. > The Hive locking for non-acid tables are advisory (i.e a client can > write/read without locking), which means that the implementation does not > offer strong consistency despite the lock manager consuming resources > centrally. > Disabling this lock acquisition would improve the performance of non-ACID > tables co-existing with a globally configured DbTxnManager implementation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22666) Introduce TopNKey operator for PTF Reduce Sink
[
https://issues.apache.org/jira/browse/HIVE-22666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020407#comment-17020407
]
Hive QA commented on HIVE-22666:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991418/HIVE-22666.4.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 17943 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query44]
(batchId=303)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[mv_query44]
(batchId=303)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query44]
(batchId=303)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20262/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20262/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20262/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991418 - PreCommit-HIVE-Build
> Introduce TopNKey operator for PTF Reduce Sink
> --
>
> Key: HIVE-22666
> URL: https://issues.apache.org/jira/browse/HIVE-22666
> Project: Hive
> Issue Type: Improvement
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-22666.1.patch, HIVE-22666.2.patch,
> HIVE-22666.3.patch, HIVE-22666.3.patch, HIVE-22666.4.patch,
> HIVE-22666.4.patch, HIVE-22666.4.patch, HIVE-22666.4.patch
>
>
> {code}
> EXPLAIN EXTENDED
> SELECT s_state, ranking
> FROM (
> SELECT s_state AS s_state,
> rank() OVER (PARTITION BY s_state ORDER BY ss_net_profit) AS ranking
> FROM testtable_n1000) tmp1
> WHERE ranking <= 3;
> {code}
> {code}
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Tez
> A masked pattern was here
> Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> A masked pattern was here
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: testtable_n1000
> Statistics: Num rows: 10 Data size: 940 Basic stats:
> COMPLETE Column stats: COMPLETE
> GatherStats: false
> Reduce Output Operator
> key expressions: s_state (type: string), ss_net_profit
> (type: double)
> null sort order: az
> sort order: ++
> Map-reduce partition columns: s_state (type: string)
> Statistics: Num rows: 10 Data size: 940 Basic stats:
> COMPLETE Column stats: COMPLETE
> tag: -1
> TopN: 4
> TopN Hash Memory Usage: 0.1
> auto parallelism: true
> Execution mode: vectorized, llap
> LLAP IO: no inputs
> Path -> Alias:
> A masked pattern was here
> Path -> Partition:
> A masked pattern was here
> Partition
> base file name: testtable_n1000
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> properties:
> COLUMN_STATS_ACCURATE
> {"BASIC_STATS":"true","COLUMN_STATS":{"s_state":"true","ss_net_profit":"true"}}
> bucket_count -1
> bucketing_version 2
> column.name.delimiter ,
> columns s_state,ss_net_profit
> columns.comments
> columns.types string:double
> A masked pattern was here
> name default.testtable_n1000
> numFiles 1
> numRows 10
> rawDataSize 80
> serialization.ddl struct testtable_n1000 { string
> s_state, double ss_net_profit}
> serialization.format 1
> serialization.lib
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> totalSize 90
> A masked pattern was here
> serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
>
> input format:
[jira] [Updated] (HIVE-22731) Probe MapJoin hashtables for row level filtering
[ https://issues.apache.org/jira/browse/HIVE-22731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-22731: -- Labels: pull-request-available (was: ) > Probe MapJoin hashtables for row level filtering > > > Key: HIVE-22731 > URL: https://issues.apache.org/jira/browse/HIVE-22731 > Project: Hive > Issue Type: Improvement > Components: Hive, llap >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22731.1.patch, HIVE-22731.WIP.patch, > decode_time_bars.pdf > > > Currently, RecordReaders such as ORC support filtering at coarser-grained > levels, namely: File, Stripe (64 to 256mb), and Row group (10k row) level. > They only filter sets of rows if they can guarantee that none of the rows can > pass a filter (usually given as searchable argument). > However, a significant amount of time can be spend decoding rows with > multiple columns that are not even used in the final result. See figure where > original is what happens today and in LazyDecode we skip decoding rows that > do not match the key. > To enable a more fine-grained filtering in the particular case of a MapJoin > we could utilize the key HashTable created from the smaller table to skip > deserializing row columns at the larger table that do not match any key and > thus save CPU time. > This Jira investigates this direction. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22731) Probe MapJoin hashtables for row level filtering
[ https://issues.apache.org/jira/browse/HIVE-22731?focusedWorklogId=375098=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-375098 ] ASF GitHub Bot logged work on HIVE-22731: - Author: ASF GitHub Bot Created on: 21/Jan/20 17:27 Start Date: 21/Jan/20 17:27 Worklog Time Spent: 10m Work Description: pgaref commented on pull request #884: HIVE-22731 Probe decode initial patch URL: https://github.com/apache/hive/pull/884 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 375098) Remaining Estimate: 0h Time Spent: 10m > Probe MapJoin hashtables for row level filtering > > > Key: HIVE-22731 > URL: https://issues.apache.org/jira/browse/HIVE-22731 > Project: Hive > Issue Type: Improvement > Components: Hive, llap >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22731.1.patch, HIVE-22731.WIP.patch, > decode_time_bars.pdf > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, RecordReaders such as ORC support filtering at coarser-grained > levels, namely: File, Stripe (64 to 256mb), and Row group (10k row) level. > They only filter sets of rows if they can guarantee that none of the rows can > pass a filter (usually given as searchable argument). > However, a significant amount of time can be spend decoding rows with > multiple columns that are not even used in the final result. See figure where > original is what happens today and in LazyDecode we skip decoding rows that > do not match the key. > To enable a more fine-grained filtering in the particular case of a MapJoin > we could utilize the key HashTable created from the smaller table to skip > deserializing row columns at the larger table that do not match any key and > thus save CPU time. > This Jira investigates this direction. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22666) Introduce TopNKey operator for PTF Reduce Sink
[
https://issues.apache.org/jira/browse/HIVE-22666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020358#comment-17020358
]
Hive QA commented on HIVE-22666:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
2s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 3 new + 7 unchanged - 0 fixed
= 10 total (was 7) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20262/dev-support/hive-personality.sh
|
| git revision | master / ba7b3ac |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20262/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20262/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Introduce TopNKey operator for PTF Reduce Sink
> --
>
> Key: HIVE-22666
> URL: https://issues.apache.org/jira/browse/HIVE-22666
> Project: Hive
> Issue Type: Improvement
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-22666.1.patch, HIVE-22666.2.patch,
> HIVE-22666.3.patch, HIVE-22666.3.patch, HIVE-22666.4.patch,
> HIVE-22666.4.patch, HIVE-22666.4.patch, HIVE-22666.4.patch
>
>
> {code}
> EXPLAIN EXTENDED
> SELECT s_state, ranking
> FROM (
> SELECT s_state AS s_state,
> rank() OVER (PARTITION BY s_state ORDER BY ss_net_profit) AS ranking
> FROM testtable_n1000) tmp1
> WHERE ranking <= 3;
> {code}
> {code}
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Tez
> A masked pattern was here
> Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> A masked pattern was here
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: testtable_n1000
> Statistics: Num rows: 10 Data size: 940 Basic stats:
> COMPLETE Column stats: COMPLETE
> GatherStats: false
> Reduce Output Operator
> key expressions: s_state (type: string), ss_net_profit
> (type: double)
> null sort order: az
> sort order: ++
>
[jira] [Commented] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020329#comment-17020329
]
Hive QA commented on HIVE-22729:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991438/HIVE-22729.05.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 19 failed/errored test(s), 17929 tests
executed
*Failed tests:*
{noformat}
TestCLIAuthzSessionContext - did not produce a TEST-*.xml file (likely timed
out) (batchId=291)
TestJdbcWithLocalClusterSpark - did not produce a TEST-*.xml file (likely timed
out) (batchId=291)
TestMultiSessionsHS2WithLocalClusterSpark - did not produce a TEST-*.xml file
(likely timed out) (batchId=291)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testMultipleTriggers1
(batchId=291)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testMultipleTriggers2
(batchId=291)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitions
(batchId=291)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitionsMultiInsert
(batchId=291)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=291)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedFiles
(batchId=291)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomNonExistent
(batchId=291)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomReadOps
(batchId=291)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighBytesRead
(batchId=291)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes
(batchId=291)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerShortQueryElapsedTime
(batchId=291)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerSlowQueryElapsedTime
(batchId=291)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerSlowQueryExecutionTime
(batchId=291)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerTotalTasks
(batchId=291)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerVertexRawInputSplitsKill
(batchId=291)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerVertexRawInputSplitsNoKill
(batchId=291)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20261/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20261/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20261/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 19 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991438 - PreCommit-HIVE-Build
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch,
> HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22741) Speed up ObjectStore method getTableMeta
[ https://issues.apache.org/jira/browse/HIVE-22741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020321#comment-17020321 ] Naveen Gangam commented on HIVE-22741: -- Thank you [~pvary] for looping us in. [~zchovan] for the patch. Seems the performance benefits are significant, especially given clients could be running this periodically to refresh metadata in the UI or something. > Speed up ObjectStore method getTableMeta > - > > Key: HIVE-22741 > URL: https://issues.apache.org/jira/browse/HIVE-22741 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22741.patch, no_patch_datanucleus.log, > patched_datanucleus.log > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22640) Decimal64ColumnVector: ClassCastException when partition column type is Decimal
[
https://issues.apache.org/jira/browse/HIVE-22640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020312#comment-17020312
]
Ádám Szita commented on HIVE-22640:
---
Committed to master. Thanks for the patch Denys, and also thanks to our
reviewers.
> Decimal64ColumnVector: ClassCastException when partition column type is
> Decimal
> ---
>
> Key: HIVE-22640
> URL: https://issues.apache.org/jira/browse/HIVE-22640
> Project: Hive
> Issue Type: Bug
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22640.1.patch, HIVE-22640.2.patch,
> HIVE-22640.3.patch, HIVE-22640.4.patch, HIVE-22640.5.patch
>
>
> Problem statement:
> {code}
> CREATE EXTERNAL TABLE output (id DECIMAL(4,0), foo VARCHAR(10))
> PARTITIONED BY (nr_bank *decimal*(4,0))
> STORED AS ORC TBLPROPERTIES ('orc.compress'='ZLIB')
> insert into output PARTITION (nr_bank=88) values (1,'test');
> insert into output PARTITION (nr_bank=8801) values (1,'8801');
> select count(*), nr_bank from sdlint.output group by nr_bank;
> {code}
> The above query fails with:
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.FuncLongToDecimal.evaluate(FuncLongToDecimal.java:58)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22640) Decimal64ColumnVector: ClassCastException when partition column type is Decimal
[
https://issues.apache.org/jira/browse/HIVE-22640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ádám Szita updated HIVE-22640:
--
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Decimal64ColumnVector: ClassCastException when partition column type is
> Decimal
> ---
>
> Key: HIVE-22640
> URL: https://issues.apache.org/jira/browse/HIVE-22640
> Project: Hive
> Issue Type: Bug
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22640.1.patch, HIVE-22640.2.patch,
> HIVE-22640.3.patch, HIVE-22640.4.patch, HIVE-22640.5.patch
>
>
> Problem statement:
> {code}
> CREATE EXTERNAL TABLE output (id DECIMAL(4,0), foo VARCHAR(10))
> PARTITIONED BY (nr_bank *decimal*(4,0))
> STORED AS ORC TBLPROPERTIES ('orc.compress'='ZLIB')
> insert into output PARTITION (nr_bank=88) values (1,'test');
> insert into output PARTITION (nr_bank=8801) values (1,'8801');
> select count(*), nr_bank from sdlint.output group by nr_bank;
> {code}
> The above query fails with:
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.FuncLongToDecimal.evaluate(FuncLongToDecimal.java:58)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020305#comment-17020305
]
Hive QA commented on HIVE-22729:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
29s{color} | {color:blue} standalone-metastore/metastore-common in master has
37 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
12s{color} | {color:blue} standalone-metastore/metastore-server in master has
181 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
2s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
24s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 1 new + 676 unchanged - 7 fixed = 677 total (was 683) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 1 new + 45 unchanged - 0 fixed
= 46 total (was 45) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 2 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20261/dev-support/hive-personality.sh
|
| git revision | master / b408300 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20261/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20261/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20261/yetus/whitespace-tabs.txt
|
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20261/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>
[jira] [Commented] (HIVE-21487) COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
[
https://issues.apache.org/jira/browse/HIVE-21487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020264#comment-17020264
]
Hive QA commented on HIVE-21487:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991436/HIVE-21847.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17936 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestPartitionManagement.testPartitionDiscoveryTransactionalTable
(batchId=227)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20260/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20260/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20260/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991436 - PreCommit-HIVE-Build
> COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
>
>
> Key: HIVE-21487
> URL: https://issues.apache.org/jira/browse/HIVE-21487
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Todd Lipcon
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-21847.01.patch
>
>
> Looking at a MySQL install where HMS is pointed on Hive 3.1, I see a constant
> stream of queries of the form:
> {code}
> select CC_STATE from COMPLETED_COMPACTIONS where CC_DATABASE =
> 'tpcds_orc_exact_1000' and CC_TABLE = 'catalog_returns' and CC_PARTITION =
> 'cr_returned_date_sk=2452851' and CC_STATE != 'a' order by CC_ID desc;
> {code}
> but the COMPLETED_COMPACTIONS table has no index. In this case it's resulting
> in a full table scan over 115k rows, which takes around 100ms.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-19369) Locks: Add new lock implementations for always zero-wait readers
[
https://issues.apache.org/jira/browse/HIVE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019564#comment-17019564
]
Denys Kuzmenko edited comment on HIVE-19369 at 1/21/20 2:23 PM:
Hi [~gopalv], if there was a schema change (i.e. EXCL_DROP) - all
blocked/dependant locks (any type, not only SHARED_READ) should fail fast,
right?
Following scenario (current logic):
{code}alter table acid add columns (b);{code} - acquires EXCLUSIVE lock;
{code}insert into table acid values (1,2,3);{code} - SHARED_READ, waits for
EXCLUSIVE to be released and later throws IndexOutOfBoundsException: Index: 3,
Size: 3
Currently SHARED_WRITE/SEMI_SHARED lock allows only SHARED_READs to go through
and blocks the rest, used for update/delete. If it's an insert (no overwrite) -
SHARED_READ lock is used.
was (Author: dkuzmenko):
Hi [~gopalv], if there was a schema change (i.e. EXCL_DROP) - all
blocked/dependant locks (any type, not only SHARED_READ) should fail fast,
right?
Following scenario (current logic):
{code}alter table acid add columns (b);{code} - acquires EXCLUSIVE lock;
{code}insert into table acid values (1,2,3);{code} - SHARED_READ, waits for
EXCLUSIVE to be released and later throws IndexOutOfBoundsException: Index: 3,
Size: 3
> Locks: Add new lock implementations for always zero-wait readers
>
>
> Key: HIVE-19369
> URL: https://issues.apache.org/jira/browse/HIVE-19369
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal Vijayaraghavan
>Assignee: Denys Kuzmenko
>Priority: Major
>
> Hive Locking with Micro-managed and full-ACID tables needs a better locking
> implementation which allows for no-wait readers always.
> EXCL_DROP
> EXCL_WRITE
> SHARED_WRITE
> SHARED_READ
> Short write-up
> EXCL_DROP is a "drop partition" or "drop table" and waits for all others to
> exit
> EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to
> exit.
> SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an
> EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different
> threads).
> SHARED_READ does not wait for any lock - it fails fast for a pending
> EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending,
> there's no semantic reason to wait for them to succeed before going ahead
> with a SHARED_WRITE.
> a select * => SHARED_READ
> an insert into => SHARED_WRITE
> an insert overwrite or MERGE => EXCL_WRITE
> a drop table => EXCL_DROP
> TODO:
> The fate of the compactor needs to be added to this before it is a complete
> description.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22704) Distribution package incorrectly ships the upgrade.order files from the metastore module
[
https://issues.apache.org/jira/browse/HIVE-22704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-22704:
Fix Version/s: 3.1.3
3.2.0
pushed to branch-3 and branch-3.1
note: on branch-3 ; for 3.2 this issue would have caused an incorrectly
initialized metastore
> Distribution package incorrectly ships the upgrade.order files from the
> metastore module
>
>
> Key: HIVE-22704
> URL: https://issues.apache.org/jira/browse/HIVE-22704
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Fix For: 4.0.0, 3.2.0, 3.1.3
>
> Attachments: HIVE-22704.01.patch, HIVE-22704.01.patch
>
>
> Expected would be to take the upgrade.order from the standalon-metastore but
> HIVE-19938 have broke that when it [introduced a
> regex|https://github.infra.cloudera.com/CDH/hive/commit/bae4c08408720d56159847603dfb8d9dab9fe1b7#diff-44c56fadf216c807d6828b60825e43baR219]
> {code}
> $ echo asd > ./metastore/scripts/upgrade/postgres/upgrade.order.postgres
> $ mvn install -pl packaging/ -Pdist -am -DskipTests -q
> $ cat
> ./packaging/target/apache-hive-4.0.0-SNAPSHOT-bin/apache-hive-4.0.0-SNAPSHOT-bin/scripts/metastore/upgrade/postgres/upgrade.order.postgres
> asd
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22255) Hive don't trigger Major Compaction automatically if table contains only base files
[
https://issues.apache.org/jira/browse/HIVE-22255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020240#comment-17020240
]
Laszlo Pinter commented on HIVE-22255:
--
[~Rajkumar Singh] Could you please provide a review board/pull request where I
can post my remarks? Thanks.
> Hive don't trigger Major Compaction automatically if table contains only base
> files
>
>
> Key: HIVE-22255
> URL: https://issues.apache.org/jira/browse/HIVE-22255
> Project: Hive
> Issue Type: Bug
> Components: Hive, Transactions
>Affects Versions: 3.1.2
> Environment: Hive-3.1.1
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-22255.01.patch, HIVE-22255.patch
>
>
> user may run into the issue if the table consists of all base files but no
> delta, then the following condition will yield false and automatic major
> compaction will be skipped.
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L313]
>
> Steps to Reproduce:
> # create Acid table
> {code:java}
> // create table myacid(id int);
> {code}
> # Run multiple insert table
> {code:java}
> // insert overwrite table myacid values(1);insert overwrite table myacid
> values(2),(3),(4){code}
> # DFS ls output
> {code:java}
> // dfs -ls -R /warehouse/tablespace/managed/hive/myacid;
> ++
> | DFS Output |
> ++
> | drwxrwx---+ - hive hadoop 0 2019-09-27 16:42
> /warehouse/tablespace/managed/hive/myacid/base_001 |
> | -rw-rw+ 3 hive hadoop 1 2019-09-27 16:42
> /warehouse/tablespace/managed/hive/myacid/base_001/_orc_acid_version |
> | -rw-rw+ 3 hive hadoop 610 2019-09-27 16:42
> /warehouse/tablespace/managed/hive/myacid/base_001/bucket_0 |
> | drwxrwx---+ - hive hadoop 0 2019-09-27 16:43
> /warehouse/tablespace/managed/hive/myacid/base_002 |
> | -rw-rw+ 3 hive hadoop 1 2019-09-27 16:43
> /warehouse/tablespace/managed/hive/myacid/base_002/_orc_acid_version |
> | -rw-rw+ 3 hive hadoop 633 2019-09-27 16:43
> /warehouse/tablespace/managed/hive/myacid/base_002/bucket_0 |
> ++{code}
>
> you will see that Major compaction will not be trigger until you run alter
> table compact MAJOR.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21487) COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
[
https://issues.apache.org/jira/browse/HIVE-21487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020230#comment-17020230
]
Hive QA commented on HIVE-21487:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
15s{color} | {color:blue} standalone-metastore/metastore-server in master has
181 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20260/dev-support/hive-personality.sh
|
| git revision | master / b408300 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20260/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
>
>
> Key: HIVE-21487
> URL: https://issues.apache.org/jira/browse/HIVE-21487
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Todd Lipcon
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-21847.01.patch
>
>
> Looking at a MySQL install where HMS is pointed on Hive 3.1, I see a constant
> stream of queries of the form:
> {code}
> select CC_STATE from COMPLETED_COMPACTIONS where CC_DATABASE =
> 'tpcds_orc_exact_1000' and CC_TABLE = 'catalog_returns' and CC_PARTITION =
> 'cr_returned_date_sk=2452851' and CC_STATE != 'a' order by CC_ID desc;
> {code}
> but the COMPLETED_COMPACTIONS table has no index. In this case it's resulting
> in a full table scan over 115k rows, which takes around 100ms.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22518) SQLStdHiveAuthorizerFactoryForTest doesn't work correctly for llap tests
[ https://issues.apache.org/jira/browse/HIVE-22518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-22518: Attachment: HIVE-22518.05.patch > SQLStdHiveAuthorizerFactoryForTest doesn't work correctly for llap tests > > > Key: HIVE-22518 > URL: https://issues.apache.org/jira/browse/HIVE-22518 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-22518.01.patch, HIVE-22518.01.patch, > HIVE-22518.02.patch, HIVE-22518.02.patch, HIVE-22518.03.patch, > HIVE-22518.03.patch, HIVE-22518.04.patch, HIVE-22518.05.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22518) SQLStdHiveAuthorizerFactoryForTest doesn't work correctly for llap tests
[ https://issues.apache.org/jira/browse/HIVE-22518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-22518: Attachment: HIVE-22518.04.patch > SQLStdHiveAuthorizerFactoryForTest doesn't work correctly for llap tests > > > Key: HIVE-22518 > URL: https://issues.apache.org/jira/browse/HIVE-22518 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-22518.01.patch, HIVE-22518.01.patch, > HIVE-22518.02.patch, HIVE-22518.02.patch, HIVE-22518.03.patch, > HIVE-22518.03.patch, HIVE-22518.04.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22705) LLAP cache is polluted by query-based compactor
[ https://issues.apache.org/jira/browse/HIVE-22705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020223#comment-17020223 ] Peter Vary commented on HIVE-22705: --- +1 > LLAP cache is polluted by query-based compactor > --- > > Key: HIVE-22705 > URL: https://issues.apache.org/jira/browse/HIVE-22705 > Project: Hive > Issue Type: Improvement >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Attachments: HIVE-22705.0.patch, HIVE-22705.1.patch, > HIVE-22705.2.patch > > > One of the steps that query-based compaction does is the verification of ACID > sort order by using the _validate_acid_sort_order_ UDF. This is a > prerequisite before the actual compaction can happen, and is done by a [query > that reads the whole table > content|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MajorQueryCompactor.java#L161-L167]. > This results in the whole table content being populated into the cache. The > problem is that this content is not useful and will rather pollute the cache > space, as it can never be used again: cache content binds to files (file IDs) > that obviously will be changed in this case by compaction. > I propose we disable LLAP caching in the session of query-based compaction's > queries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22729:
-
Attachment: HIVE-22729.05.patch
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch,
> HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22741) Speed up ObjectStore method getTableMeta
[ https://issues.apache.org/jira/browse/HIVE-22741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-22741: -- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks for the patch [~zchovan]! Little late, but you might be interested in [~ngangam], [~thejas]... > Speed up ObjectStore method getTableMeta > - > > Key: HIVE-22741 > URL: https://issues.apache.org/jira/browse/HIVE-22741 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22741.patch, no_patch_datanucleus.log, > patched_datanucleus.log > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21487) COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
[
https://issues.apache.org/jira/browse/HIVE-21487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-21487:
-
Status: Patch Available (was: Open)
> COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
>
>
> Key: HIVE-21487
> URL: https://issues.apache.org/jira/browse/HIVE-21487
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Todd Lipcon
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-21847.01.patch
>
>
> Looking at a MySQL install where HMS is pointed on Hive 3.1, I see a constant
> stream of queries of the form:
> {code}
> select CC_STATE from COMPLETED_COMPACTIONS where CC_DATABASE =
> 'tpcds_orc_exact_1000' and CC_TABLE = 'catalog_returns' and CC_PARTITION =
> 'cr_returned_date_sk=2452851' and CC_STATE != 'a' order by CC_ID desc;
> {code}
> but the COMPLETED_COMPACTIONS table has no index. In this case it's resulting
> in a full table scan over 115k rows, which takes around 100ms.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22741) Speed up ObjectStore method getTableMeta
[
https://issues.apache.org/jira/browse/HIVE-22741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020212#comment-17020212
]
Zoltan Chovan commented on HIVE-22741:
--
Ran some benchmarks with
[hclient|https://github.com/akolb1/hclient/blob/master/metastore-cli/README.md]
(thanks [~pvary] for the help):
Benchmark with before patch:
{code:java}
$ java -jar hbench-jar-with-dependencies.jar -H localhost -N 100 -W 1 -S
"getTableMeta.*" -d "A6" -L 10
13:53:52.928 [main] INFO com.akolb.HMSBenchmark - using args -H localhost -N
100 -W 1 -S getTableMeta.* -d A6 -L 10
13:53:52.929 [main] INFO com.akolb.HMSBenchmark - Using table 'A6.bench_table
13:53:52.939 [main] INFO com.akolb.Util - Connecting to localhost:9083
0[main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
13:53:53.494 [main] INFO com.akolb.HMSBenchmark - Using 100 object instances
warmup 1 spin 10 threads 2
13:53:53.494 [main] INFO com.akolb.HMSBenchmark - Using 100 object instances
13:53:53.520 [main] INFO com.akolb.BenchmarkSuite - Running benchmark
getTableMeta.100
Run time: 248
Run time: 276
Run time: 295
Run time: 338
Run time: 285
Run time: 286
Run time: 271
Run time: 285
Run time: 270
Run time: 268
Operation AMeanMean Med Min Max Err%
getTableMeta.100 0.000282.8281.4248.2338.4
8.320
{code}
Benchmark after patch:
{code:java}
$ java -jar hbench-jar-with-dependencies.jar -H localhost -N 100 -W 1 -S
"getTableMeta.*" -d "A7" -L 10
13:56:22.312 [main] INFO com.akolb.HMSBenchmark - using args -H localhost -N
100 -W 1 -S getTableMeta.* -d A7 -L 10
13:56:22.314 [main] INFO com.akolb.HMSBenchmark - Using table 'A7.bench_table
13:56:22.339 [main] INFO com.akolb.Util - Connecting to localhost:9083
0[main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
13:56:23.722 [main] INFO com.akolb.HMSBenchmark - Using 100 object instances
warmup 1 spin 10 threads 2
13:56:23.722 [main] INFO com.akolb.HMSBenchmark - Using 100 object instances
13:56:23.754 [main] INFO com.akolb.BenchmarkSuite - Running benchmark
getTableMeta.100
Run time: 16
Run time: 19
Run time: 19
Run time: 23
Run time: 21
Run time: 14
Run time: 17
Run time: 16
Run time: 20
Run time: 13
Operation AMeanMean Med Min Max Err%
getTableMeta.100 0.00018.1818.4413.2523.03
17.86
{code}
Based on the mean values the speedup is ~15x
> Speed up ObjectStore method getTableMeta
> -
>
> Key: HIVE-22741
> URL: https://issues.apache.org/jira/browse/HIVE-22741
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Major
> Attachments: HIVE-22741.patch, no_patch_datanucleus.log,
> patched_datanucleus.log
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020202#comment-17020202
]
Hive QA commented on HIVE-22729:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991429/HIVE-22729.04.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20258/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20258/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20258/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-01-21 12:49:09.582
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-20258/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-01-21 12:49:09.585
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
bbc67d5..6686813 master -> origin/master
+ git reset --hard HEAD
HEAD is now at bbc67d5 HIVE-16690: Configure Tez cartesian product edge based
on LLAP cluster size (Zhiyuan Yang reviewed by Sergey Shelukhin, Siddharth
Seth, Ashutosh Chauhan)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 6686813 HIVE-22568: Process compaction candidates in parallel by
the Initiator (Denys Kuzmenko reviewed by Laszlo Pinter and Peter Vary)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-01-21 12:49:10.742
+ rm -rf ../yetus_PreCommit-HIVE-Build-20258
+ mkdir ../yetus_PreCommit-HIVE-Build-20258
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-20258
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-20258/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
error:
a/ql/src/java/org/apache/hadoop/hive/ql/ddl/process/show/compactions/ShowCompactionsDesc.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/ddl/process/show/compactions/ShowCompactionsOperation.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java: does
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java: does
not exist in index
error:
a/ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java:
does not exist in index
error: a/ql/src/test/results/clientpositive/dbtxnmgr_showlocks.q.out: does not
exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowCompactResponseElement.java:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift:
does not exist in index
error:
[jira] [Commented] (HIVE-22753) Fix gradual mem leak: Operationlog related appenders should be cleared up on errors
[ https://issues.apache.org/jira/browse/HIVE-22753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020197#comment-17020197 ] Peter Vary commented on HIVE-22753: --- CC: [~zchovan] > Fix gradual mem leak: Operationlog related appenders should be cleared up on > errors > > > Key: HIVE-22753 > URL: https://issues.apache.org/jira/browse/HIVE-22753 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-22753.1.patch, image-2020-01-21-11-14-37-911.png, > image-2020-01-21-11-17-59-279.png, image-2020-01-21-11-18-37-294.png > > > In case of exception in SQLOperation, operational log does not get cleared > up. This causes gradual build up of HushableRandomAccessFileAppender causing > HS2 to OOM after some time. > !image-2020-01-21-11-14-37-911.png|width=431,height=267! > > Allocation tree > !image-2020-01-21-11-18-37-294.png|width=425,height=178! > > Prod instance mem > !image-2020-01-21-11-17-59-279.png|width=698,height=209! > > Each HushableRandomAccessFileAppender holds internal ref to > RandomAccessFileAppender which holds a 256 KB bytebuffer, causing the mem > leak. > Related ticket: HIVE-18820 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21487) COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
[
https://issues.apache.org/jira/browse/HIVE-21487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-21487:
-
Attachment: HIVE-21847.01.patch
> COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
>
>
> Key: HIVE-21487
> URL: https://issues.apache.org/jira/browse/HIVE-21487
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Todd Lipcon
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-21847.01.patch
>
>
> Looking at a MySQL install where HMS is pointed on Hive 3.1, I see a constant
> stream of queries of the form:
> {code}
> select CC_STATE from COMPLETED_COMPACTIONS where CC_DATABASE =
> 'tpcds_orc_exact_1000' and CC_TABLE = 'catalog_returns' and CC_PARTITION =
> 'cr_returned_date_sk=2452851' and CC_STATE != 'a' order by CC_ID desc;
> {code}
> but the COMPLETED_COMPACTIONS table has no index. In this case it's resulting
> in a full table scan over 115k rows, which takes around 100ms.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22741) Speed up ObjectStore method getTableMeta
[
https://issues.apache.org/jira/browse/HIVE-22741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020181#comment-17020181
]
Zoltan Chovan edited comment on HIVE-22741 at 1/21/20 12:22 PM:
Based on the DataNucelus logging, the following queries were generated for a
call in TestGetTableMeta.testGetTableMeta()
{code:java}
DEBUG [DataNucleus.Datastore.Native] - SELECT DISTINCT
'org.apache.hadoop.hive.metastore.model.MTable' AS
NUCLEUS_TYPE,A0.CREATE_TIME,C0.CTLG_NAME,C0.CREATE_TIME,C0."DESC",C0.DB_LOCATION_URI,C0."NAME",C0.OWNER_NAME,C0.OWNER_TYPE,C0.DB_ID,A0.TBL_ID,A0.LAST_ACCESS_TIME,A0.OWNER,A0.OWNER_TYPE,A0.RETENTION,A0.IS_REWRITE_ENABLED,A0.TBL_NAME,A0.TBL_TYPE,A0.WRITE_ID
FROM TBLS A0 LEFT OUTER JOIN DBS B0 ON A0.DB_ID = B0.DB_ID LEFT OUTER JOIN DBS
C0 ON A0.DB_ID = C0.DB_ID WHERE B0.CTLG_NAME = <'hive'> AND LOWER(B0."NAME")
LIKE '%one' ESCAPE '\'
DEBUG [DataNucleus.Datastore.Native] - SELECT A0.PARAM_KEY,A0.PARAM_VALUE FROM
TABLE_PARAMS A0 WHERE A0.TBL_ID = <1> AND A0.PARAM_KEY IS NOT NULL
DEBUG [DataNucleus.Datastore.Native] - SELECT A0.PARAM_KEY,A0.PARAM_VALUE FROM
TABLE_PARAMS A0 WHERE A0.TBL_ID = <2> AND A0.PARAM_KEY IS NOT NULL
DEBUG [DataNucleus.Datastore.Native] - SELECT A0.PARAM_KEY,A0.PARAM_VALUE FROM
TABLE_PARAMS A0 WHERE A0.TBL_ID = <3> AND A0.PARAM_KEY IS NOT NULL
{code}
With the patch applied the generated query is as follows:
{code:java}
DEBUG [DataNucleus.Datastore.Native] - SELECT
B0."NAME",A0.TBL_NAME,A0.TBL_TYPE,(SELECT A0_SUB.PARAM_VALUE FROM TABLE_PARAMS
A0_SUB WHERE A0_SUB.TBL_ID = A0.TBL_ID AND A0_SUB.PARAM_KEY = 'comment') FROM
TBLS A0 LEFT OUTER JOIN DBS B0 ON A0.DB_ID = B0.DB_ID WHERE B0.CTLG_NAME =
<'hive'> AND LOWER(B0."NAME") LIKE '%one' ESCAPE '\'
{code}
Attached the full logs:
[^patched_datanucleus.log]
[^no_patch_datanucleus.log]
Running
{noformat}
cat | cut -d" " -f3- > stripped.log{noformat}
on the log files allows for running diff for comparison.
was (Author: zchovan):
Based on the DataNucelus logging, the following queries were generated for a
call in TestGetTableMeta.testGetTableMeta()
{code:java}
DEBUG [DataNucleus.Datastore.Native] - SELECT DISTINCT
'org.apache.hadoop.hive.metastore.model.MTable' AS
NUCLEUS_TYPE,A0.CREATE_TIME,C0.CTLG_NAME,C0.CREATE_TIME,C0."DESC",C0.DB_LOCATION_URI,C0."NAME",C0.OWNER_NAME,C0.OWNER_TYPE,C0.DB_ID,A0.TBL_ID,A0.LAST_ACCESS_TIME,A0.OWNER,A0.OWNER_TYPE,A0.RETENTION,A0.IS_REWRITE_ENABLED,A0.TBL_NAME,A0.TBL_TYPE,A0.WRITE_ID
FROM TBLS A0 LEFT OUTER JOIN DBS B0 ON A0.DB_ID = B0.DB_ID LEFT OUTER JOIN DBS
C0 ON A0.DB_ID = C0.DB_ID WHERE B0.CTLG_NAME = <'hive'> AND LOWER(B0."NAME")
LIKE '%one' ESCAPE '\'
DEBUG [DataNucleus.Datastore.Native] - SELECT A0.PARAM_KEY,A0.PARAM_VALUE FROM
TABLE_PARAMS A0 WHERE A0.TBL_ID = <1> AND A0.PARAM_KEY IS NOT NULL
DEBUG [DataNucleus.Datastore.Native] - SELECT A0.PARAM_KEY,A0.PARAM_VALUE FROM
TABLE_PARAMS A0 WHERE A0.TBL_ID = <2> AND A0.PARAM_KEY IS NOT NULL
DEBUG [DataNucleus.Datastore.Native] - SELECT A0.PARAM_KEY,A0.PARAM_VALUE FROM
TABLE_PARAMS A0 WHERE A0.TBL_ID = <3> AND A0.PARAM_KEY IS NOT NULL
{code}
With the patch applied the generated query is as follows:
{code:java}
DEBUG [DataNucleus.Datastore.Native] - SELECT
B0."NAME",A0.TBL_NAME,A0.TBL_TYPE,(SELECT A0_SUB.PARAM_VALUE FROM TABLE_PARAMS
A0_SUB WHERE A0_SUB.TBL_ID = A0.TBL_ID AND A0_SUB.PARAM_KEY = 'comment') FROM
TBLS A0 LEFT OUTER JOIN DBS B0 ON A0.DB_ID = B0.DB_ID WHERE B0.CTLG_NAME =
<'hive'> AND LOWER(B0."NAME") LIKE '%one' ESCAPE '\'
{code}
> Speed up ObjectStore method getTableMeta
> -
>
> Key: HIVE-22741
> URL: https://issues.apache.org/jira/browse/HIVE-22741
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Major
> Attachments: HIVE-22741.patch, no_patch_datanucleus.log,
> patched_datanucleus.log
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22741) Speed up ObjectStore method getTableMeta
[ https://issues.apache.org/jira/browse/HIVE-22741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-22741: - Attachment: patched_datanucleus.log no_patch_datanucleus.log > Speed up ObjectStore method getTableMeta > - > > Key: HIVE-22741 > URL: https://issues.apache.org/jira/browse/HIVE-22741 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-22741.patch, no_patch_datanucleus.log, > patched_datanucleus.log > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22741) Speed up ObjectStore method getTableMeta
[
https://issues.apache.org/jira/browse/HIVE-22741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020181#comment-17020181
]
Zoltan Chovan commented on HIVE-22741:
--
Based on the DataNucelus logging, the following queries were generated for a
call in TestGetTableMeta.testGetTableMeta()
{code:java}
DEBUG [DataNucleus.Datastore.Native] - SELECT DISTINCT
'org.apache.hadoop.hive.metastore.model.MTable' AS
NUCLEUS_TYPE,A0.CREATE_TIME,C0.CTLG_NAME,C0.CREATE_TIME,C0."DESC",C0.DB_LOCATION_URI,C0."NAME",C0.OWNER_NAME,C0.OWNER_TYPE,C0.DB_ID,A0.TBL_ID,A0.LAST_ACCESS_TIME,A0.OWNER,A0.OWNER_TYPE,A0.RETENTION,A0.IS_REWRITE_ENABLED,A0.TBL_NAME,A0.TBL_TYPE,A0.WRITE_ID
FROM TBLS A0 LEFT OUTER JOIN DBS B0 ON A0.DB_ID = B0.DB_ID LEFT OUTER JOIN DBS
C0 ON A0.DB_ID = C0.DB_ID WHERE B0.CTLG_NAME = <'hive'> AND LOWER(B0."NAME")
LIKE '%one' ESCAPE '\'
DEBUG [DataNucleus.Datastore.Native] - SELECT A0.PARAM_KEY,A0.PARAM_VALUE FROM
TABLE_PARAMS A0 WHERE A0.TBL_ID = <1> AND A0.PARAM_KEY IS NOT NULL
DEBUG [DataNucleus.Datastore.Native] - SELECT A0.PARAM_KEY,A0.PARAM_VALUE FROM
TABLE_PARAMS A0 WHERE A0.TBL_ID = <2> AND A0.PARAM_KEY IS NOT NULL
DEBUG [DataNucleus.Datastore.Native] - SELECT A0.PARAM_KEY,A0.PARAM_VALUE FROM
TABLE_PARAMS A0 WHERE A0.TBL_ID = <3> AND A0.PARAM_KEY IS NOT NULL
{code}
With the patch applied the generated query is as follows:
{code:java}
DEBUG [DataNucleus.Datastore.Native] - SELECT
B0."NAME",A0.TBL_NAME,A0.TBL_TYPE,(SELECT A0_SUB.PARAM_VALUE FROM TABLE_PARAMS
A0_SUB WHERE A0_SUB.TBL_ID = A0.TBL_ID AND A0_SUB.PARAM_KEY = 'comment') FROM
TBLS A0 LEFT OUTER JOIN DBS B0 ON A0.DB_ID = B0.DB_ID WHERE B0.CTLG_NAME =
<'hive'> AND LOWER(B0."NAME") LIKE '%one' ESCAPE '\'
{code}
> Speed up ObjectStore method getTableMeta
> -
>
> Key: HIVE-22741
> URL: https://issues.apache.org/jira/browse/HIVE-22741
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Major
> Attachments: HIVE-22741.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22568) Process compaction candidates in parallel by the Initiator
[ https://issues.apache.org/jira/browse/HIVE-22568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020170#comment-17020170 ] Denys Kuzmenko commented on HIVE-22568: --- [~pvary], [~lpinter] thank you for review and commit! > Process compaction candidates in parallel by the Initiator > -- > > Key: HIVE-22568 > URL: https://issues.apache.org/jira/browse/HIVE-22568 > Project: Hive > Issue Type: Improvement >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22568.1.patch, HIVE-22568.10.patch, > HIVE-22568.11.patch, HIVE-22568.12.patch, HIVE-22568.2.patch, > HIVE-22568.3.patch, HIVE-22568.4.patch, HIVE-22568.5.patch, > HIVE-22568.6.patch, HIVE-22568.7.patch, HIVE-22568.8.patch, HIVE-22568.9.patch > > > `checkForCompaction` includes many file metadata checks and may be expensive. > Therefore, make sense using a thread pool here and running > `checkForCompactions` in parallel. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22568) Process compaction candidates in parallel by the Initiator
[ https://issues.apache.org/jira/browse/HIVE-22568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-22568: -- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks for the patch [~dkuzmenko] and [~lpinter] for the review! > Process compaction candidates in parallel by the Initiator > -- > > Key: HIVE-22568 > URL: https://issues.apache.org/jira/browse/HIVE-22568 > Project: Hive > Issue Type: Improvement >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22568.1.patch, HIVE-22568.10.patch, > HIVE-22568.11.patch, HIVE-22568.12.patch, HIVE-22568.2.patch, > HIVE-22568.3.patch, HIVE-22568.4.patch, HIVE-22568.5.patch, > HIVE-22568.6.patch, HIVE-22568.7.patch, HIVE-22568.8.patch, HIVE-22568.9.patch > > > `checkForCompaction` includes many file metadata checks and may be expensive. > Therefore, make sense using a thread pool here and running > `checkForCompactions` in parallel. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22568) Process compaction candidates in parallel by the Initiator
[
https://issues.apache.org/jira/browse/HIVE-22568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020163#comment-17020163
]
Hive QA commented on HIVE-22568:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991423/HIVE-22568.12.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17936 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20255/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20255/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20255/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991423 - PreCommit-HIVE-Build
> Process compaction candidates in parallel by the Initiator
> --
>
> Key: HIVE-22568
> URL: https://issues.apache.org/jira/browse/HIVE-22568
> Project: Hive
> Issue Type: Improvement
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22568.1.patch, HIVE-22568.10.patch,
> HIVE-22568.11.patch, HIVE-22568.12.patch, HIVE-22568.2.patch,
> HIVE-22568.3.patch, HIVE-22568.4.patch, HIVE-22568.5.patch,
> HIVE-22568.6.patch, HIVE-22568.7.patch, HIVE-22568.8.patch, HIVE-22568.9.patch
>
>
> `checkForCompaction` includes many file metadata checks and may be expensive.
> Therefore, make sense using a thread pool here and running
> `checkForCompactions` in parallel.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22729:
-
Attachment: HIVE-22729.04.patch
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch,
> HIVE-22729.03.patch, HIVE-22729.04.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22712) ReExec Driver execute submit the query in default queue irrespective of user defined queue
[ https://issues.apache.org/jira/browse/HIVE-22712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020142#comment-17020142 ] Zoltan Haindrich commented on HIVE-22712: - please don't change the existing testcase; add it at the end... and you also have to update the q.out without that it wont pass the tests... > ReExec Driver execute submit the query in default queue irrespective of user > defined queue > -- > > Key: HIVE-22712 > URL: https://issues.apache.org/jira/browse/HIVE-22712 > Project: Hive > Issue Type: Bug > Components: Hive, HiveServer2 >Affects Versions: 3.1.2 > Environment: Hive-3 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-22712.01.patch, HIVE-22712.02.patch, > HIVE-22712.03.patch, HIVE-22712.04.patch, HIVE-22712.05.patch, > HIVE-22712.06.patch, HIVE-22712.06.patch, HIVE-22712.patch > > > we unset the queue name intentionally in > TezSessionState#startSessionAndContainers, > as a result reexec create a new session in the default queue and create a > problem, its a cumbersome to add reexec.overlay.tez.queue.name at session > level. > I could not find a better way of setting the queue name (I am open for the > suggestion here) since it can create a conflict with the Global queue name > vs user-defined queue that's why setting while initialization of > ReExecutionOverlayPlugin. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22568) Process compaction candidates in parallel by the Initiator
[
https://issues.apache.org/jira/browse/HIVE-22568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020138#comment-17020138
]
Hive QA commented on HIVE-22568:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
59s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
2s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} ql: The patch generated 0 new + 17 unchanged - 5
fixed = 17 total (was 22) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20255/dev-support/hive-personality.sh
|
| git revision | master / bbc67d5 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20255/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Process compaction candidates in parallel by the Initiator
> --
>
> Key: HIVE-22568
> URL: https://issues.apache.org/jira/browse/HIVE-22568
> Project: Hive
> Issue Type: Improvement
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22568.1.patch, HIVE-22568.10.patch,
> HIVE-22568.11.patch, HIVE-22568.12.patch, HIVE-22568.2.patch,
> HIVE-22568.3.patch, HIVE-22568.4.patch, HIVE-22568.5.patch,
> HIVE-22568.6.patch, HIVE-22568.7.patch, HIVE-22568.8.patch, HIVE-22568.9.patch
>
>
> `checkForCompaction` includes many file metadata checks and may be expensive.
> Therefore, make sense using a thread pool here and running
> `checkForCompactions` in parallel.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22640) Decimal64ColumnVector: ClassCastException when partition column type is Decimal
[
https://issues.apache.org/jira/browse/HIVE-22640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020132#comment-17020132
]
Denys Kuzmenko commented on HIVE-22640:
---
fix qfile.out due to null order change HIVE-22489
> Decimal64ColumnVector: ClassCastException when partition column type is
> Decimal
> ---
>
> Key: HIVE-22640
> URL: https://issues.apache.org/jira/browse/HIVE-22640
> Project: Hive
> Issue Type: Bug
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22640.1.patch, HIVE-22640.2.patch,
> HIVE-22640.3.patch, HIVE-22640.4.patch, HIVE-22640.5.patch
>
>
> Problem statement:
> {code}
> CREATE EXTERNAL TABLE output (id DECIMAL(4,0), foo VARCHAR(10))
> PARTITIONED BY (nr_bank *decimal*(4,0))
> STORED AS ORC TBLPROPERTIES ('orc.compress'='ZLIB')
> insert into output PARTITION (nr_bank=88) values (1,'test');
> insert into output PARTITION (nr_bank=8801) values (1,'8801');
> select count(*), nr_bank from sdlint.output group by nr_bank;
> {code}
> The above query fails with:
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.FuncLongToDecimal.evaluate(FuncLongToDecimal.java:58)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22640) Decimal64ColumnVector: ClassCastException when partition column type is Decimal
[
https://issues.apache.org/jira/browse/HIVE-22640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-22640:
--
Attachment: HIVE-22640.5.patch
> Decimal64ColumnVector: ClassCastException when partition column type is
> Decimal
> ---
>
> Key: HIVE-22640
> URL: https://issues.apache.org/jira/browse/HIVE-22640
> Project: Hive
> Issue Type: Bug
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22640.1.patch, HIVE-22640.2.patch,
> HIVE-22640.3.patch, HIVE-22640.4.patch, HIVE-22640.5.patch
>
>
> Problem statement:
> {code}
> CREATE EXTERNAL TABLE output (id DECIMAL(4,0), foo VARCHAR(10))
> PARTITIONED BY (nr_bank *decimal*(4,0))
> STORED AS ORC TBLPROPERTIES ('orc.compress'='ZLIB')
> insert into output PARTITION (nr_bank=88) values (1,'test');
> insert into output PARTITION (nr_bank=8801) values (1,'8801');
> select count(*), nr_bank from sdlint.output group by nr_bank;
> {code}
> The above query fails with:
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.FuncLongToDecimal.evaluate(FuncLongToDecimal.java:58)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22733) After disable operation log property in hive, still HS2 saving the operation log
[
https://issues.apache.org/jira/browse/HIVE-22733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mahesh kumar behera updated HIVE-22733:
---
Status: Patch Available (was: In Progress)
> After disable operation log property in hive, still HS2 saving the operation
> log
>
>
> Key: HIVE-22733
> URL: https://issues.apache.org/jira/browse/HIVE-22733
> Project: Hive
> Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22733.01.patch, HIVE-22733.02.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> There are few issues in this area.
> 1. If logging is disabled using hive.server2.logging.operation.enabled, then
> operation logs for the queries should not be generated. But the
> registerLoggingContext method in LogUtils, registers the logging context
> even if the operation log is disabled. This causes the logs to be added by
> logger. The registration of query context should be done only if operation
> logging is enabled.
> {code:java}
> public static void registerLoggingContext(Configuration conf) {
> -MDC.put(SESSIONID_LOG_KEY, HiveConf.getVar(conf,
> HiveConf.ConfVars.HIVESESSIONID));
> -MDC.put(QUERYID_LOG_KEY, HiveConf.getVar(conf,
> HiveConf.ConfVars.HIVEQUERYID));
> if (HiveConf.getBoolVar(conf,
> HiveConf.ConfVars.HIVE_SERVER2_LOGGING_OPERATION_ENABLED)) {
> + MDC.put(SESSIONID_LOG_KEY, HiveConf.getVar(conf,
> HiveConf.ConfVars.HIVESESSIONID));
> + MDC.put(QUERYID_LOG_KEY, HiveConf.getVar(conf,
> HiveConf.ConfVars.HIVEQUERYID));
>MDC.put(OPERATIONLOG_LEVEL_KEY, HiveConf.getVar(conf,
> HiveConf.ConfVars.HIVE_SERVER2_LOGGING_OPERATION_LEVEL));{code}
>
> 2. In case of failed query, we close the operations and that deletes the
> logging context (appender and route) from logger for that query. But if any
> log is added after that, the query logs are getting added and new operation
> log file is getting generated for the query. This looks like issue with MCD
> clear. MCD clear is not removing the keys from the map. If remove is used
> instead of clear, its working fine.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22733) After disable operation log property in hive, still HS2 saving the operation log
[
https://issues.apache.org/jira/browse/HIVE-22733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mahesh kumar behera updated HIVE-22733:
---
Status: In Progress (was: Patch Available)
> After disable operation log property in hive, still HS2 saving the operation
> log
>
>
> Key: HIVE-22733
> URL: https://issues.apache.org/jira/browse/HIVE-22733
> Project: Hive
> Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22733.01.patch, HIVE-22733.02.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> There are few issues in this area.
> 1. If logging is disabled using hive.server2.logging.operation.enabled, then
> operation logs for the queries should not be generated. But the
> registerLoggingContext method in LogUtils, registers the logging context
> even if the operation log is disabled. This causes the logs to be added by
> logger. The registration of query context should be done only if operation
> logging is enabled.
> {code:java}
> public static void registerLoggingContext(Configuration conf) {
> -MDC.put(SESSIONID_LOG_KEY, HiveConf.getVar(conf,
> HiveConf.ConfVars.HIVESESSIONID));
> -MDC.put(QUERYID_LOG_KEY, HiveConf.getVar(conf,
> HiveConf.ConfVars.HIVEQUERYID));
> if (HiveConf.getBoolVar(conf,
> HiveConf.ConfVars.HIVE_SERVER2_LOGGING_OPERATION_ENABLED)) {
> + MDC.put(SESSIONID_LOG_KEY, HiveConf.getVar(conf,
> HiveConf.ConfVars.HIVESESSIONID));
> + MDC.put(QUERYID_LOG_KEY, HiveConf.getVar(conf,
> HiveConf.ConfVars.HIVEQUERYID));
>MDC.put(OPERATIONLOG_LEVEL_KEY, HiveConf.getVar(conf,
> HiveConf.ConfVars.HIVE_SERVER2_LOGGING_OPERATION_LEVEL));{code}
>
> 2. In case of failed query, we close the operations and that deletes the
> logging context (appender and route) from logger for that query. But if any
> log is added after that, the query logs are getting added and new operation
> log file is getting generated for the query. This looks like issue with MCD
> clear. MCD clear is not removing the keys from the map. If remove is used
> instead of clear, its working fine.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22722) timestamptz_2 test failure
[ https://issues.apache.org/jira/browse/HIVE-22722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020127#comment-17020127 ] Zoltan Haindrich commented on HIVE-22722: - I've run this test ~200 times on my machine while I was working on it and doing stuff...it didn't failed even once...I went thru the logs - but nothing stood out... ptest executors run 2 tests on the same node at the same time; that might be connected... > timestamptz_2 test failure > -- > > Key: HIVE-22722 > URL: https://issues.apache.org/jira/browse/HIVE-22722 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > > the min/max value seems to be off in some cases; this was highly non > deterministic; and hard to reproduce - but in the recent QA runs it started > failing more -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22705) LLAP cache is polluted by query-based compactor
[
https://issues.apache.org/jira/browse/HIVE-22705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020115#comment-17020115
]
Hive QA commented on HIVE-22705:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991414/HIVE-22705.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17936 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20254/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20254/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20254/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991414 - PreCommit-HIVE-Build
> LLAP cache is polluted by query-based compactor
> ---
>
> Key: HIVE-22705
> URL: https://issues.apache.org/jira/browse/HIVE-22705
> Project: Hive
> Issue Type: Improvement
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Attachments: HIVE-22705.0.patch, HIVE-22705.1.patch,
> HIVE-22705.2.patch
>
>
> One of the steps that query-based compaction does is the verification of ACID
> sort order by using the _validate_acid_sort_order_ UDF. This is a
> prerequisite before the actual compaction can happen, and is done by a [query
> that reads the whole table
> content|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MajorQueryCompactor.java#L161-L167].
> This results in the whole table content being populated into the cache. The
> problem is that this content is not useful and will rather pollute the cache
> space, as it can never be used again: cache content binds to files (file IDs)
> that obviously will be changed in this case by compaction.
> I propose we disable LLAP caching in the session of query-based compaction's
> queries.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22729:
-
Attachment: HIVE-22729.03.patch
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch,
> HIVE-22729.03.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22729:
-
Attachment: (was: HIVE-22729.03.patch)
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22568) Process compaction candidates in parallel by the Initiator
[ https://issues.apache.org/jira/browse/HIVE-22568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020109#comment-17020109 ] Denys Kuzmenko commented on HIVE-22568: --- rebased > Process compaction candidates in parallel by the Initiator > -- > > Key: HIVE-22568 > URL: https://issues.apache.org/jira/browse/HIVE-22568 > Project: Hive > Issue Type: Improvement >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-22568.1.patch, HIVE-22568.10.patch, > HIVE-22568.11.patch, HIVE-22568.12.patch, HIVE-22568.2.patch, > HIVE-22568.3.patch, HIVE-22568.4.patch, HIVE-22568.5.patch, > HIVE-22568.6.patch, HIVE-22568.7.patch, HIVE-22568.8.patch, HIVE-22568.9.patch > > > `checkForCompaction` includes many file metadata checks and may be expensive. > Therefore, make sense using a thread pool here and running > `checkForCompactions` in parallel. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22568) Process compaction candidates in parallel by the Initiator
[ https://issues.apache.org/jira/browse/HIVE-22568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-22568: -- Attachment: HIVE-22568.12.patch > Process compaction candidates in parallel by the Initiator > -- > > Key: HIVE-22568 > URL: https://issues.apache.org/jira/browse/HIVE-22568 > Project: Hive > Issue Type: Improvement >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-22568.1.patch, HIVE-22568.10.patch, > HIVE-22568.11.patch, HIVE-22568.12.patch, HIVE-22568.2.patch, > HIVE-22568.3.patch, HIVE-22568.4.patch, HIVE-22568.5.patch, > HIVE-22568.6.patch, HIVE-22568.7.patch, HIVE-22568.8.patch, HIVE-22568.9.patch > > > `checkForCompaction` includes many file metadata checks and may be expensive. > Therefore, make sense using a thread pool here and running > `checkForCompactions` in parallel. -- This message was sent by Atlassian Jira (v8.3.4#803005)
