[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=215247=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-215247
]
ASF GitHub Bot logged work on HIVE-21446:
-
Author: ASF GitHub Bot
Created on: 19/Mar/19 05:44
Start Date: 19/Mar/19 05:44
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #569: HIVE-21446 :
Hive Server going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266738404
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
##
@@ -57,30 +57,37 @@
private final long maxNumberOfFiles;
private final boolean hiveInTest;
private final String copyAsUser;
+ private FileSystem destinationFs;
- public CopyUtils(String distCpDoAsUser, HiveConf hiveConf) {
+ public CopyUtils(String distCpDoAsUser, HiveConf hiveConf, FileSystem
destinationFs) {
this.hiveConf = hiveConf;
maxNumberOfFiles =
hiveConf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXNUMFILES);
maxCopyFileSize =
hiveConf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXSIZE);
hiveInTest = hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_IN_TEST);
this.copyAsUser = distCpDoAsUser;
+this.destinationFs = destinationFs;
}
// Used by replication, copy files from source to destination. It is
possible source file is
// changed/removed during copy, so double check the checksum after copy,
// if not match, copy again from cm
- public void copyAndVerify(FileSystem destinationFs, Path destRoot,
+ public void copyAndVerify(Path destRoot,
List srcFiles) throws
IOException, LoginException, HiveFatalException {
Map>> map =
fsToFileMap(srcFiles, destRoot);
UserGroupInformation proxyUser = getProxyUser();
try {
for (Map.Entry>>
entry : map.entrySet()) {
-FileSystem sourceFs = entry.getKey();
Map> destMap = entry.getValue();
for (Map.Entry> destMapEntry :
destMap.entrySet()) {
Path destination = destMapEntry.getKey();
List fileInfoList =
destMapEntry.getValue();
- boolean useRegularCopy = regularCopy(destinationFs, sourceFs,
fileInfoList);
+ if (fileInfoList.isEmpty()) {
Review comment:
In that case, can we just change it to assert? and add a comment for this
assumption.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 215247)
Time Spent: 2h 10m (was: 2h)
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch, HIVE-21446.02.patch
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21430) INSERT into a dynamically partitioned table with hive.stats.autogather = false throws a MetaException
[
https://issues.apache.org/jira/browse/HIVE-21430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Bapat updated HIVE-21430:
--
Attachment: HIVE-21430.03.patch
Status: Patch Available (was: In Progress)
The test failure in the last ptest run seems unrelated to the patch.
Reattaching same patch as 02 again to trigger ptests.
> INSERT into a dynamically partitioned table with hive.stats.autogather =
> false throws a MetaException
> -
>
> Key: HIVE-21430
> URL: https://issues.apache.org/jira/browse/HIVE-21430
> Project: Hive
> Issue Type: Bug
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21430.01.patch, HIVE-21430.02.patch,
> HIVE-21430.03.patch, metaexception_repro.patch,
> org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread-output.txt
>
> Original Estimate: 48h
> Time Spent: 50m
> Remaining Estimate: 47h 10m
>
> When the test TestStatsUpdaterThread#testTxnDynamicPartitions added in the
> attached patch is run it throws exception (full logs attached.)
> org.apache.hadoop.hive.metastore.api.MetaException: Cannot change stats state
> for a transactional table default.simple_stats without providing the
> transactional write state for verification (new write ID 5, valid write IDs
> null; current state \{"BASIC_STATS":"true","COLUMN_STATS":{"s":"true"}}; new
> state null
> at
> org.apache.hadoop.hive.metastore.ObjectStore.alterPartitionNoTxn(ObjectStore.java:4328)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21430) INSERT into a dynamically partitioned table with hive.stats.autogather = false throws a MetaException
[
https://issues.apache.org/jira/browse/HIVE-21430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Bapat updated HIVE-21430:
--
Status: In Progress (was: Patch Available)
> INSERT into a dynamically partitioned table with hive.stats.autogather =
> false throws a MetaException
> -
>
> Key: HIVE-21430
> URL: https://issues.apache.org/jira/browse/HIVE-21430
> Project: Hive
> Issue Type: Bug
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21430.01.patch, HIVE-21430.02.patch,
> metaexception_repro.patch,
> org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread-output.txt
>
> Original Estimate: 48h
> Time Spent: 50m
> Remaining Estimate: 47h 10m
>
> When the test TestStatsUpdaterThread#testTxnDynamicPartitions added in the
> attached patch is run it throws exception (full logs attached.)
> org.apache.hadoop.hive.metastore.api.MetaException: Cannot change stats state
> for a transactional table default.simple_stats without providing the
> transactional write state for verification (new write ID 5, valid write IDs
> null; current state \{"BASIC_STATS":"true","COLUMN_STATS":{"s":"true"}}; new
> state null
> at
> org.apache.hadoop.hive.metastore.ObjectStore.alterPartitionNoTxn(ObjectStore.java:4328)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21466) Increase Default Size of SPLIT_MAXSIZE
[
https://issues.apache.org/jira/browse/HIVE-21466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795707#comment-16795707
]
Hive QA commented on HIVE-21466:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962897/HIVE-21466.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 15833 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_invalid_place]
(batchId=99)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query28]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query44]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query8]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[mv_query44]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query28]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query44]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query8]
(batchId=275)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16566/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16566/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16566/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962897 - PreCommit-HIVE-Build
> Increase Default Size of SPLIT_MAXSIZE
> --
>
> Key: HIVE-21466
> URL: https://issues.apache.org/jira/browse/HIVE-21466
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-21466.1.patch, HIVE-21466.1.patch
>
>
> {code:java}
> MAPREDMAXSPLITSIZE(FileInputFormat.SPLIT_MAXSIZE, 25600L, "", true),
> {code}
> [https://github.com/apache/hive/blob/8d4300a02691777fc96f33861ed27e64fed72f2c/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L682]
> This field specifies a maximum size for each MR (maybe other?) splits.
> This number should be a multiple of the HDFS Block size. The way that this
> maximum is implemented, is that each block is added to the split, and if the
> split grows to be larger than the maximum allowed, the split is submitted to
> the cluster and a new split is opened.
> So, imagine the following scenario:
> * HDFS block size of 16 bytes
> * Maximum size of 40 bytes
> This will produce a split with 3 blocks. (2x16) = 32; another block will be
> inserted, (3x16) = 48 bytes in the split. So, while many operators would
> assume a split of 2 blocks, the actual is 3 blocks. Setting the maximum split
> size to a multiple of the HDFS block size will make this behavior less
> confusing.
> The current setting is ~256MB and when this was introduced, the default HDFS
> block size was 64MB. That is a factor of 4x. However, now HDFS block sizes
> are 128MB by default, so I propose setting this to 4x128MB. The larger
> splits (fewer tasks) should give a nice performance boost for modern hardware.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=215240=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-215240
]
ASF GitHub Bot logged work on HIVE-21446:
-
Author: ASF GitHub Bot
Created on: 19/Mar/19 05:26
Start Date: 19/Mar/19 05:26
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #569: HIVE-21446 :
Hive Server going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266735716
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
##
@@ -57,30 +57,37 @@
private final long maxNumberOfFiles;
private final boolean hiveInTest;
private final String copyAsUser;
+ private FileSystem destinationFs;
- public CopyUtils(String distCpDoAsUser, HiveConf hiveConf) {
+ public CopyUtils(String distCpDoAsUser, HiveConf hiveConf, FileSystem
destinationFs) {
this.hiveConf = hiveConf;
maxNumberOfFiles =
hiveConf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXNUMFILES);
maxCopyFileSize =
hiveConf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXSIZE);
hiveInTest = hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_IN_TEST);
this.copyAsUser = distCpDoAsUser;
+this.destinationFs = destinationFs;
}
// Used by replication, copy files from source to destination. It is
possible source file is
// changed/removed during copy, so double check the checksum after copy,
// if not match, copy again from cm
- public void copyAndVerify(FileSystem destinationFs, Path destRoot,
+ public void copyAndVerify(Path destRoot,
List srcFiles) throws
IOException, LoginException, HiveFatalException {
Map>> map =
fsToFileMap(srcFiles, destRoot);
UserGroupInformation proxyUser = getProxyUser();
try {
for (Map.Entry>>
entry : map.entrySet()) {
-FileSystem sourceFs = entry.getKey();
Map> destMap = entry.getValue();
for (Map.Entry> destMapEntry :
destMap.entrySet()) {
Path destination = destMapEntry.getKey();
List fileInfoList =
destMapEntry.getValue();
- boolean useRegularCopy = regularCopy(destinationFs, sourceFs,
fileInfoList);
+ if (fileInfoList.isEmpty()) {
Review comment:
currently no ..but if some filters are added and the files to be copied are
removed then this could lead to empty list
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 215240)
Time Spent: 2h (was: 1h 50m)
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch, HIVE-21446.02.patch
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21430) INSERT into a dynamically partitioned table with hive.stats.autogather = false throws a MetaException
[
https://issues.apache.org/jira/browse/HIVE-21430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795699#comment-16795699
]
mahesh kumar behera commented on HIVE-21430:
[^HIVE-21430.02.patch] code changes looks good to me
> INSERT into a dynamically partitioned table with hive.stats.autogather =
> false throws a MetaException
> -
>
> Key: HIVE-21430
> URL: https://issues.apache.org/jira/browse/HIVE-21430
> Project: Hive
> Issue Type: Bug
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21430.01.patch, HIVE-21430.02.patch,
> metaexception_repro.patch,
> org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread-output.txt
>
> Original Estimate: 48h
> Time Spent: 50m
> Remaining Estimate: 47h 10m
>
> When the test TestStatsUpdaterThread#testTxnDynamicPartitions added in the
> attached patch is run it throws exception (full logs attached.)
> org.apache.hadoop.hive.metastore.api.MetaException: Cannot change stats state
> for a transactional table default.simple_stats without providing the
> transactional write state for verification (new write ID 5, valid write IDs
> null; current state \{"BASIC_STATS":"true","COLUMN_STATS":{"s":"true"}}; new
> state null
> at
> org.apache.hadoop.hive.metastore.ObjectStore.alterPartitionNoTxn(ObjectStore.java:4328)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21466) Increase Default Size of SPLIT_MAXSIZE
[
https://issues.apache.org/jira/browse/HIVE-21466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795672#comment-16795672
]
Hive QA commented on HIVE-21466:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16566/dev-support/hive-personality.sh
|
| git revision | master / 36bd89d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16566/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Increase Default Size of SPLIT_MAXSIZE
> --
>
> Key: HIVE-21466
> URL: https://issues.apache.org/jira/browse/HIVE-21466
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-21466.1.patch, HIVE-21466.1.patch
>
>
> {code:java}
> MAPREDMAXSPLITSIZE(FileInputFormat.SPLIT_MAXSIZE, 25600L, "", true),
> {code}
> [https://github.com/apache/hive/blob/8d4300a02691777fc96f33861ed27e64fed72f2c/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L682]
> This field specifies a maximum size for each MR (maybe other?) splits.
> This number should be a multiple of the HDFS Block size. The way that this
> maximum is implemented, is that each block is added to the split, and if the
> split grows to be larger than the maximum allowed, the split is submitted to
> the cluster and a new split is opened.
> So, imagine the following scenario:
> * HDFS block size of 16 bytes
> * Maximum size of 40 bytes
> This will produce a split with 3 blocks. (2x16) = 32; another block will be
> inserted, (3x16) = 48 bytes in the split. So, while many operators would
> assume a split of 2 blocks, the actual is 3 blocks. Setting the maximum split
> size to a multiple of the HDFS block size will make this behavior less
> confusing.
> The current setting is ~256MB and when this was introduced, the default HDFS
> block size was 64MB. That is a factor of 4x. However, now HDFS block sizes
> are 128MB by default, so I propose
[jira] [Comment Edited] (HIVE-13479) Relax sorting requirement in ACID tables
[ https://issues.apache.org/jira/browse/HIVE-13479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795678#comment-16795678 ] Abhishek Somani edited comment on HIVE-13479 at 3/19/19 5:01 AM: - [~ekoifman] [~vgumashta] [~gopalv] Do we have any plans to work on this? As I understand it, this might not be a restriction on Insert Only tables as there are no row_ids there, but will need work on Orc readers because of the assumption of records sorted on row_ids for a sort merge of delete events. Is that correct? was (Author: asomani): [~ekoifman] [~vgumashta] [~gopalv] Do we have any plans to work on this? > Relax sorting requirement in ACID tables > > > Key: HIVE-13479 > URL: https://issues.apache.org/jira/browse/HIVE-13479 > Project: Hive > Issue Type: New Feature > Components: Transactions >Affects Versions: 1.2.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Original Estimate: 160h > Remaining Estimate: 160h > > Currently ACID tables require data to be sorted according to internal primary > key. This is that base + delta files can be efficiently sort/merged to > produce the snapshot for current transaction. > This prevents the user to make the table sorted based on any other criteria > which can be useful. One example is using dynamic partition insert (which > also occurs for update/delete SQL). This may create lots of writers > (buckets*partitions) and tax cluster resources. > The usual solution is hive.optimize.sort.dynamic.partition=true which won't > be honored for ACID tables. > We could rely on hash table based algorithm to merge delta files and then not > require any particular sort on Acid tables. One way to do that is to treat > each update event as an Insert (new internal PK) + delete (old PK). Delete > events are very small since they just need to contain PKs. So the hash table > would just need to contain Delete events and be reasonably memory efficient. > This is a significant amount of work but worth doing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-13479) Relax sorting requirement in ACID tables
[ https://issues.apache.org/jira/browse/HIVE-13479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795678#comment-16795678 ] Abhishek Somani commented on HIVE-13479: [~ekoifman] [~vgumashta] [~gopalv] Do we have any plans to work on this? > Relax sorting requirement in ACID tables > > > Key: HIVE-13479 > URL: https://issues.apache.org/jira/browse/HIVE-13479 > Project: Hive > Issue Type: New Feature > Components: Transactions >Affects Versions: 1.2.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Original Estimate: 160h > Remaining Estimate: 160h > > Currently ACID tables require data to be sorted according to internal primary > key. This is that base + delta files can be efficiently sort/merged to > produce the snapshot for current transaction. > This prevents the user to make the table sorted based on any other criteria > which can be useful. One example is using dynamic partition insert (which > also occurs for update/delete SQL). This may create lots of writers > (buckets*partitions) and tax cluster resources. > The usual solution is hive.optimize.sort.dynamic.partition=true which won't > be honored for ACID tables. > We could rely on hash table based algorithm to merge delta files and then not > require any particular sort on Acid tables. One way to do that is to treat > each update event as an Insert (new internal PK) + delete (old PK). Delete > events are very small since they just need to contain PKs. So the hash table > would just need to contain Delete events and be reasonably memory efficient. > This is a significant amount of work but worth doing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21456) Hive Metastore HTTP Thrift
[ https://issues.apache.org/jira/browse/HIVE-21456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Khanna updated HIVE-21456: --- Attachment: HIVE-21456.3.patch > Hive Metastore HTTP Thrift > -- > > Key: HIVE-21456 > URL: https://issues.apache.org/jira/browse/HIVE-21456 > Project: Hive > Issue Type: New Feature > Components: Metastore, Standalone Metastore >Reporter: Amit Khanna >Assignee: Amit Khanna >Priority: Major > Attachments: HIVE-21456.2.patch, HIVE-21456.3.patch, HIVE-21456.patch > > > Hive Metastore currently doesn't have support for HTTP transport because of > which it is not possible to access it via Knox. Adding support for Thrift > over HTTP transport will allow the clients to access via Knox -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21165) ACID: pass query hint to the writers to write hive.acid.key.index
[ https://issues.apache.org/jira/browse/HIVE-21165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795663#comment-16795663 ] Abhishek Somani commented on HIVE-21165: Got it. I was confused between hive.acid.key.index and hive.acid.stats. Is it perhaps then that hive.acid.stats is not required anymore as it just counts types of events? > ACID: pass query hint to the writers to write hive.acid.key.index > - > > Key: HIVE-21165 > URL: https://issues.apache.org/jira/browse/HIVE-21165 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > > For the query based compactor from HIVE-20699, the compaction runs as a sql > query. However, this mechanism skips over writing hive.acid.key.index for > each stripe, which is used to skip over stripes that are not supposed to be > read. We need a way to pass a query hint to the writer so that it can write > this index data, when invoked from a sql query. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=215223=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-215223
]
ASF GitHub Bot logged work on HIVE-21446:
-
Author: ASF GitHub Bot
Created on: 19/Mar/19 04:24
Start Date: 19/Mar/19 04:24
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #569: HIVE-21446 :
Hive Server going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266726914
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
##
@@ -57,30 +57,37 @@
private final long maxNumberOfFiles;
private final boolean hiveInTest;
private final String copyAsUser;
+ private FileSystem destinationFs;
- public CopyUtils(String distCpDoAsUser, HiveConf hiveConf) {
+ public CopyUtils(String distCpDoAsUser, HiveConf hiveConf, FileSystem
destinationFs) {
this.hiveConf = hiveConf;
maxNumberOfFiles =
hiveConf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXNUMFILES);
maxCopyFileSize =
hiveConf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXSIZE);
hiveInTest = hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_IN_TEST);
this.copyAsUser = distCpDoAsUser;
+this.destinationFs = destinationFs;
}
// Used by replication, copy files from source to destination. It is
possible source file is
// changed/removed during copy, so double check the checksum after copy,
// if not match, copy again from cm
- public void copyAndVerify(FileSystem destinationFs, Path destRoot,
+ public void copyAndVerify(Path destRoot,
List srcFiles) throws
IOException, LoginException, HiveFatalException {
Map>> map =
fsToFileMap(srcFiles, destRoot);
UserGroupInformation proxyUser = getProxyUser();
try {
for (Map.Entry>>
entry : map.entrySet()) {
-FileSystem sourceFs = entry.getKey();
Map> destMap = entry.getValue();
for (Map.Entry> destMapEntry :
destMap.entrySet()) {
Path destination = destMapEntry.getKey();
List fileInfoList =
destMapEntry.getValue();
- boolean useRegularCopy = regularCopy(destinationFs, sourceFs,
fileInfoList);
+ if (fileInfoList.isEmpty()) {
Review comment:
In what scenario, fileInfoList can be empty?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 215223)
Time Spent: 1h 50m (was: 1h 40m)
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch, HIVE-21446.02.patch
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21165) ACID: pass query hint to the writers to write hive.acid.key.index
[ https://issues.apache.org/jira/browse/HIVE-21165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795644#comment-16795644 ] Eugene Koifman commented on HIVE-21165: --- The logic in HIVE-20738 relies on this index. > ACID: pass query hint to the writers to write hive.acid.key.index > - > > Key: HIVE-21165 > URL: https://issues.apache.org/jira/browse/HIVE-21165 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > > For the query based compactor from HIVE-20699, the compaction runs as a sql > query. However, this mechanism skips over writing hive.acid.key.index for > each stripe, which is used to skip over stripes that are not supposed to be > read. We need a way to pass a query hint to the writer so that it can write > this index data, when invoked from a sql query. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20580) OrcInputFormat.isOriginal() should not rely on hive.acid.key.index
[
https://issues.apache.org/jira/browse/HIVE-20580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795638#comment-16795638
]
Eugene Koifman commented on HIVE-20580:
---
isOriginal should mean files w/o acid metadata columns in them. You can have
them in a transactional table table because it started out as a flat table and
was ALTER TABLE'd to a transactional or they were added via LOAD DATA for
example. I think you said the wrong version of the isOriginal() is not used -
I'd get rid of it.
> OrcInputFormat.isOriginal() should not rely on hive.acid.key.index
> --
>
> Key: HIVE-20580
> URL: https://issues.apache.org/jira/browse/HIVE-20580
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-20580.2.patch, HIVE-20580.3.patch,
> HIVE-20580.4.patch, HIVE-20580.5.patch, HIVE-20580.6.patch, HIVE-20580.patch
>
>
> {{org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.isOriginal()}} is checking
> for presence of {{hive.acid.key.index}} in the footer. This is only created
> when the file is written by {{OrcRecordUpdater}}. It should instead check
> for presence of Acid metadata columns so that a file can be produced by
> something other than {{OrcRecordUpater}}.
> Also, {{hive.acid.key.index}} counts number of different type of events which
> is not really useful for Acid V2 (as of Hive 3) since each file only has 1
> type of event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21409) Initial SessionState ClassLoader Reused For Subsequent Sessions
[ https://issues.apache.org/jira/browse/HIVE-21409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795587#comment-16795587 ] Shawn Weeks commented on HIVE-21409: I've verified this bug in Hive 3.1 as well. A combination of Add Jar and implicitly added UDF Jars pollute the session classloader. > Initial SessionState ClassLoader Reused For Subsequent Sessions > --- > > Key: HIVE-21409 > URL: https://issues.apache.org/jira/browse/HIVE-21409 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Shawn Weeks >Priority: Minor > Attachments: create_class.sql, run.sql, setup.sql > > > It appears that the first ClassLoader attached to a SessionState Static > Instance is being reused as the parent for all future sessions. This causes > any libraries added to the class path on the initial session to be added to > future sessions. It also appears that further sessions may be adding jars to > this initial ClassLoader as well leading to the class path getting more and > more polluted. This occurring on a build including HIVE-11878. I've included > some examples that greatly exaggerate the problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21456) Hive Metastore HTTP Thrift
[
https://issues.apache.org/jira/browse/HIVE-21456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795583#comment-16795583
]
Hive QA commented on HIVE-21456:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962888/HIVE-21456.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 15834 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_invalid_place]
(batchId=99)
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.org.apache.hadoop.hive.ql.TestWarehouseExternalDir
(batchId=243)
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.testExternalDefaultPaths
(batchId=243)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16565/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16565/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16565/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962888 - PreCommit-HIVE-Build
> Hive Metastore HTTP Thrift
> --
>
> Key: HIVE-21456
> URL: https://issues.apache.org/jira/browse/HIVE-21456
> Project: Hive
> Issue Type: New Feature
> Components: Metastore, Standalone Metastore
>Reporter: Amit Khanna
>Assignee: Amit Khanna
>Priority: Major
> Attachments: HIVE-21456.2.patch, HIVE-21456.patch
>
>
> Hive Metastore currently doesn't have support for HTTP transport because of
> which it is not possible to access it via Knox. Adding support for Thrift
> over HTTP transport will allow the clients to access via Knox
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21468) Case sensitivity in identifier names for JDBC storage handler
[ https://issues.apache.org/jira/browse/HIVE-21468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-21468: --- Attachment: HIVE-21468.01.patch > Case sensitivity in identifier names for JDBC storage handler > - > > Key: HIVE-21468 > URL: https://issues.apache.org/jira/browse/HIVE-21468 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-21468.01.patch, HIVE-21468.patch > > > Currently, when Calcite generates the SQL query for the JDBC storage handler, > it will ignore capitalization for the identifiers names, which can lead to > errors at execution time (though the query is properly generated). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-21387) Wrong result for UNION query with GROUP BY consisting of PK columns
[
https://issues.apache.org/jira/browse/HIVE-21387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg resolved HIVE-21387.
Resolution: Fixed
Fix Version/s: 4.0.0
Fixed in HIVE-21382
> Wrong result for UNION query with GROUP BY consisting of PK columns
> ---
>
> Key: HIVE-21387
> URL: https://issues.apache.org/jira/browse/HIVE-21387
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Critical
> Fix For: 4.0.0
>
>
> *Reproducer*
> {code:sql}
> create table t1(i int primary key disable rely, j int);
> insert into t1 values(1,100),(2,200);
> create table t2(i int primary key disable rely, j int);
> insert into t2 values(2,1000),(4,500);
> select i from (select i, j from t1 union all select i,j from t2) subq group
> by i,j;
> {code}
> *Expected Result*
> {noformat}
> 2
> 2
> 4
> 1
> {noformat}
> *Actual Result*
> {noformat}
> 1
> 2
> 4
> {noformat}
> *CBO Plan*
> {code:sql}
> HiveAggregate(group=[{0}])
> HiveProject(i=[$0], j=[$1])
> HiveUnion(all=[true])
> HiveProject(i=[$0], j=[$1])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> HiveProject(i=[$0], j=[$1])
> HiveTableScan(table=[[default, t2]], table:alias=[t2])
> {code}
> This is due to Group by reduction logic reducing keys when it shouldn't.
> Because of UNION relative cardinality of the group by keys are changed (they
> are not PK/UNIQUE anymore). Therefore we shouldn't be trying to reduce group
> by keys at all.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21386) Extend the fetch task enhancement done in HIVE-21279 to make it work with query result cache
[ https://issues.apache.org/jira/browse/HIVE-21386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795578#comment-16795578 ] Vineet Garg commented on HIVE-21386: Initial patch to get a test run > Extend the fetch task enhancement done in HIVE-21279 to make it work with > query result cache > > > Key: HIVE-21386 > URL: https://issues.apache.org/jira/browse/HIVE-21386 > Project: Hive > Issue Type: Improvement >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21386.1.patch > > > The improvement done in HIVE-21279 is disabled for query cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21386) Extend the fetch task enhancement done in HIVE-21279 to make it work with query result cache
[ https://issues.apache.org/jira/browse/HIVE-21386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21386: --- Attachment: HIVE-21386.1.patch > Extend the fetch task enhancement done in HIVE-21279 to make it work with > query result cache > > > Key: HIVE-21386 > URL: https://issues.apache.org/jira/browse/HIVE-21386 > Project: Hive > Issue Type: Improvement >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21386.1.patch > > > The improvement done in HIVE-21279 is disabled for query cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21456) Hive Metastore HTTP Thrift
[
https://issues.apache.org/jira/browse/HIVE-21456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795556#comment-16795556
]
Hive QA commented on HIVE-21456:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
35s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
16s{color} | {color:blue} standalone-metastore/metastore-server in master has
179 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16565/dev-support/hive-personality.sh
|
| git revision | master / 36bd89d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16565/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16565/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive Metastore HTTP Thrift
> --
>
> Key: HIVE-21456
> URL: https://issues.apache.org/jira/browse/HIVE-21456
> Project: Hive
> Issue Type: New Feature
> Components: Metastore, Standalone Metastore
>Reporter: Amit Khanna
>Assignee: Amit Khanna
>Priority: Major
> Attachments: HIVE-21456.2.patch, HIVE-21456.patch
>
>
> Hive Metastore currently doesn't have support for HTTP transport because of
> which it is not possible to access it via Knox. Adding support for Thrift
> over HTTP transport will allow the clients to access via Knox
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21469) Review of ZooKeeperHiveLockManager
[
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21469:
--
Description:
A lot of sins in this class to resolve:
{code:java}
@Override
public void setContext(HiveLockManagerCtx ctx) throws LockException {
try {
curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
try{
curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/" +
parent, new byte[0]);
} catch (Exception e) {
// ignore if the parent already exists
if (!(e instanceof KeeperException) || ((KeeperException)e).code() !=
KeeperException.Code.NODEEXISTS) {
LOG.warn("Unexpected ZK exception when creating parent node /" +
parent, e);
}
}
{code}
Every time a new session is created and this {{setContext}} method is called,
it attempts to create the root node. I have seen that, even though the root
node exists, an create node action is written to the ZK logs. Check first if
the node exists before trying to create it.
{code:java}
try {
curatorFramework.delete().forPath(zLock.getPath());
} catch (InterruptedException ie) {
curatorFramework.delete().forPath(zLock.getPath());
}
{code}
There has historically been a quite a few bugs regarding leaked locks. The
Driver will signal the session {{Thread}} by performing an interrupt. That
interrupt can happen any time and it can kill a create/delete action within the
ZK framework. We can see one example of workaround for this. If the ZK action
is interrupted, simply do it again. Well, what if it's interrupted yet again?
The lock will be leaked. Also, when the {{InterruptedException}} is caught in
the try block, the thread's interrupted flag is cleared. The flag is not reset
in this code and therefore we lose the fact that this thread has been
interrupted. This flag should be preserved so that other code paths will know
that it's time to exit.
{code:java}
if (tryNum > 1) {
Thread.sleep(sleepTime);
}
unlockPrimitive(hiveLock, parent, curatorFramework);
break;
} catch (Exception e) {
if (tryNum >= numRetriesForUnLock) {
String name = ((ZooKeeperHiveLock)hiveLock).getPath();
throw new LockException("Node " + name + " can not be deleted after "
+ numRetriesForUnLock + " attempts.",
e);
}
}
{code}
... related... the sleep here may be interrupted, but we still need to delete
the lock (again, for fear of leaking it). This sleep should be
uninterruptible. If we need to get the lock deleted, and there's a problem,
interrupting the sleep will cause the code to eventually exit and locks will be
leaked.
It also requires a bunch more TLC.
was:
A lot of sins in this class to resolve:
{code:java}
@Override
public void setContext(HiveLockManagerCtx ctx) throws LockException {
try {
curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
try{
curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/" +
parent, new byte[0]);
} catch (Exception e) {
// ignore if the parent already exists
if (!(e instanceof KeeperException) || ((KeeperException)e).code() !=
KeeperException.Code.NODEEXISTS) {
LOG.warn("Unexpected ZK exception when creating parent node /" +
parent, e);
}
}
{code}
Every time a new session is created and this {{setContext}} method is called,
it attempts to create the root node. I have seen that, even though the root
node exists, an create node action is written to the ZK logs. Check first if
the node exists before trying to create it.
{code:java}
try {
curatorFramework.delete().forPath(zLock.getPath());
} catch (InterruptedException ie) {
curatorFramework.delete().forPath(zLock.getPath());
}
{code}
There has historically been a quite a few bugs regarding leaked locks. The
Driver will signal the session {{Thread}} by performing an interrupt. That
interrupt can happen any time and it can kill a create/delete action within the
ZK framework. We can see one example of workaround for this. If the ZK action
is interrupted, simply do it again. Well, what if it's interrupted yet again?
The lock will be leaked anyway. Also, when the {{InterruptedException}} is
caught in the try block, the thread's interrupted flag is cleared. The flag is
not reset in this code and therefore we lose the fact that this thread has been
interrupted.
{code:java}
if (tryNum > 1) {
Thread.sleep(sleepTime);
}
unlockPrimitive(hiveLock, parent, curatorFramework);
break;
} catch (Exception e) {
if
[jira] [Updated] (HIVE-21469) Review of ZooKeeperHiveLockManager
[
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21469:
--
Status: Patch Available (was: Open)
> Review of ZooKeeperHiveLockManager
> --
>
> Key: HIVE-21469
> URL: https://issues.apache.org/jira/browse/HIVE-21469
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-21469.1.patch
>
>
> A lot of sins in this class to resolve:
> {code:java}
> @Override
> public void setContext(HiveLockManagerCtx ctx) throws LockException {
> try {
> curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
> parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
> try{
> curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/"
> + parent, new byte[0]);
> } catch (Exception e) {
> // ignore if the parent already exists
> if (!(e instanceof KeeperException) || ((KeeperException)e).code() !=
> KeeperException.Code.NODEEXISTS) {
> LOG.warn("Unexpected ZK exception when creating parent node /" +
> parent, e);
> }
> }
> {code}
> Every time a new session is created and this {{setContext}} method is called,
> it attempts to create the root node. I have seen that, even though the root
> node exists, an create node action is written to the ZK logs. Check first if
> the node exists before trying to create it.
> {code:java}
> try {
> curatorFramework.delete().forPath(zLock.getPath());
> } catch (InterruptedException ie) {
> curatorFramework.delete().forPath(zLock.getPath());
> }
> {code}
> There has historically been a quite a few bugs regarding leaked locks. The
> Driver will signal the session {{Thread}} by performing an interrupt. That
> interrupt can happen any time and it can kill a create/delete action within
> the ZK framework. We can see one example of workaround for this. If the ZK
> action is interrupted, simply do it again. Well, what if it's interrupted
> yet again? The lock will be leaked anyway. Also, when the
> {{InterruptedException}} is caught in the try block, the thread's interrupted
> flag is cleared. The flag is not reset in this code and therefore we lose
> the fact that this thread has been interrupted.
> {code:java}
> if (tryNum > 1) {
> Thread.sleep(sleepTime);
> }
> unlockPrimitive(hiveLock, parent, curatorFramework);
> break;
> } catch (Exception e) {
> if (tryNum >= numRetriesForUnLock) {
> String name = ((ZooKeeperHiveLock)hiveLock).getPath();
> throw new LockException("Node " + name + " can not be deleted after
> " + numRetriesForUnLock + " attempts.",
> e);
> }
> }
> {code}
> ... related... the sleep here may be interrupted, but we still need to delete
> the lock (again, for fear of leaking it). This sleep should be
> uninterruptible. If we need to get the lock deleted, and there's a problem,
> interrupting the sleep will cause the code to eventually exit and locks will
> be leaked.
> It also requires a bunch more TLC.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21469) Review of ZooKeeperHiveLockManager
[
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor reassigned HIVE-21469:
-
> Review of ZooKeeperHiveLockManager
> --
>
> Key: HIVE-21469
> URL: https://issues.apache.org/jira/browse/HIVE-21469
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-21469.1.patch
>
>
> A lot of sins in this class to resolve:
> {code:java}
> @Override
> public void setContext(HiveLockManagerCtx ctx) throws LockException {
> try {
> curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
> parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
> try{
> curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/"
> + parent, new byte[0]);
> } catch (Exception e) {
> // ignore if the parent already exists
> if (!(e instanceof KeeperException) || ((KeeperException)e).code() !=
> KeeperException.Code.NODEEXISTS) {
> LOG.warn("Unexpected ZK exception when creating parent node /" +
> parent, e);
> }
> }
> {code}
> Every time a new session is created and this {{setContext}} method is called,
> it attempts to create the root node. I have seen that, even though the root
> node exists, an create node action is written to the ZK logs. Check first if
> the node exists before trying to create it.
> {code:java}
> try {
> curatorFramework.delete().forPath(zLock.getPath());
> } catch (InterruptedException ie) {
> curatorFramework.delete().forPath(zLock.getPath());
> }
> {code}
> There has historically been a quite a few bugs regarding leaked locks. The
> Driver will signal the session {{Thread}} by performing an interrupt. That
> interrupt can happen any time and it can kill a create/delete action within
> the ZK framework. We can see one example of workaround for this. If the ZK
> action is interrupted, simply do it again. Well, what if it's interrupted
> yet again? The lock will be leaked anyway. Also, when the
> {{InterruptedException}} is caught in the try block, the thread's interrupted
> flag is cleared. The flag is not reset in this code and therefore we lose
> the fact that this thread has been interrupted.
> {code:java}
> if (tryNum > 1) {
> Thread.sleep(sleepTime);
> }
> unlockPrimitive(hiveLock, parent, curatorFramework);
> break;
> } catch (Exception e) {
> if (tryNum >= numRetriesForUnLock) {
> String name = ((ZooKeeperHiveLock)hiveLock).getPath();
> throw new LockException("Node " + name + " can not be deleted after
> " + numRetriesForUnLock + " attempts.",
> e);
> }
> }
> {code}
> ... related... the sleep here may be interrupted, but we still need to delete
> the lock (again, for fear of leaking it). This sleep should be
> uninterruptible. If we need to get the lock deleted, and there's a problem,
> interrupting the sleep will cause the code to eventually exit and locks will
> be leaked.
> It also requires a bunch more TLC.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21469) Review of ZooKeeperHiveLockManager
[
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21469:
--
Attachment: HIVE-21469.1.patch
> Review of ZooKeeperHiveLockManager
> --
>
> Key: HIVE-21469
> URL: https://issues.apache.org/jira/browse/HIVE-21469
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-21469.1.patch
>
>
> A lot of sins in this class to resolve:
> {code:java}
> @Override
> public void setContext(HiveLockManagerCtx ctx) throws LockException {
> try {
> curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
> parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
> try{
> curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/"
> + parent, new byte[0]);
> } catch (Exception e) {
> // ignore if the parent already exists
> if (!(e instanceof KeeperException) || ((KeeperException)e).code() !=
> KeeperException.Code.NODEEXISTS) {
> LOG.warn("Unexpected ZK exception when creating parent node /" +
> parent, e);
> }
> }
> {code}
> Every time a new session is created and this {{setContext}} method is called,
> it attempts to create the root node. I have seen that, even though the root
> node exists, an create node action is written to the ZK logs. Check first if
> the node exists before trying to create it.
> {code:java}
> try {
> curatorFramework.delete().forPath(zLock.getPath());
> } catch (InterruptedException ie) {
> curatorFramework.delete().forPath(zLock.getPath());
> }
> {code}
> There has historically been a quite a few bugs regarding leaked locks. The
> Driver will signal the session {{Thread}} by performing an interrupt. That
> interrupt can happen any time and it can kill a create/delete action within
> the ZK framework. We can see one example of workaround for this. If the ZK
> action is interrupted, simply do it again. Well, what if it's interrupted
> yet again? The lock will be leaked anyway. Also, when the
> {{InterruptedException}} is caught in the try block, the thread's interrupted
> flag is cleared. The flag is not reset in this code and therefore we lose
> the fact that this thread has been interrupted.
> {code:java}
> if (tryNum > 1) {
> Thread.sleep(sleepTime);
> }
> unlockPrimitive(hiveLock, parent, curatorFramework);
> break;
> } catch (Exception e) {
> if (tryNum >= numRetriesForUnLock) {
> String name = ((ZooKeeperHiveLock)hiveLock).getPath();
> throw new LockException("Node " + name + " can not be deleted after
> " + numRetriesForUnLock + " attempts.",
> e);
> }
> }
> {code}
> ... related... the sleep here may be interrupted, but we still need to delete
> the lock (again, for fear of leaking it). This sleep should be
> uninterruptible. If we need to get the lock deleted, and there's a problem,
> interrupting the sleep will cause the code to eventually exit and locks will
> be leaked.
> It also requires a bunch more TLC.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21466) Increase Default Size of SPLIT_MAXSIZE
[
https://issues.apache.org/jira/browse/HIVE-21466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21466:
--
Status: Open (was: Patch Available)
> Increase Default Size of SPLIT_MAXSIZE
> --
>
> Key: HIVE-21466
> URL: https://issues.apache.org/jira/browse/HIVE-21466
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-21466.1.patch, HIVE-21466.1.patch
>
>
> {code:java}
> MAPREDMAXSPLITSIZE(FileInputFormat.SPLIT_MAXSIZE, 25600L, "", true),
> {code}
> [https://github.com/apache/hive/blob/8d4300a02691777fc96f33861ed27e64fed72f2c/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L682]
> This field specifies a maximum size for each MR (maybe other?) splits.
> This number should be a multiple of the HDFS Block size. The way that this
> maximum is implemented, is that each block is added to the split, and if the
> split grows to be larger than the maximum allowed, the split is submitted to
> the cluster and a new split is opened.
> So, imagine the following scenario:
> * HDFS block size of 16 bytes
> * Maximum size of 40 bytes
> This will produce a split with 3 blocks. (2x16) = 32; another block will be
> inserted, (3x16) = 48 bytes in the split. So, while many operators would
> assume a split of 2 blocks, the actual is 3 blocks. Setting the maximum split
> size to a multiple of the HDFS block size will make this behavior less
> confusing.
> The current setting is ~256MB and when this was introduced, the default HDFS
> block size was 64MB. That is a factor of 4x. However, now HDFS block sizes
> are 128MB by default, so I propose setting this to 4x128MB. The larger
> splits (fewer tasks) should give a nice performance boost for modern hardware.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21466) Increase Default Size of SPLIT_MAXSIZE
[
https://issues.apache.org/jira/browse/HIVE-21466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21466:
--
Attachment: HIVE-21466.1.patch
> Increase Default Size of SPLIT_MAXSIZE
> --
>
> Key: HIVE-21466
> URL: https://issues.apache.org/jira/browse/HIVE-21466
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-21466.1.patch, HIVE-21466.1.patch
>
>
> {code:java}
> MAPREDMAXSPLITSIZE(FileInputFormat.SPLIT_MAXSIZE, 25600L, "", true),
> {code}
> [https://github.com/apache/hive/blob/8d4300a02691777fc96f33861ed27e64fed72f2c/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L682]
> This field specifies a maximum size for each MR (maybe other?) splits.
> This number should be a multiple of the HDFS Block size. The way that this
> maximum is implemented, is that each block is added to the split, and if the
> split grows to be larger than the maximum allowed, the split is submitted to
> the cluster and a new split is opened.
> So, imagine the following scenario:
> * HDFS block size of 16 bytes
> * Maximum size of 40 bytes
> This will produce a split with 3 blocks. (2x16) = 32; another block will be
> inserted, (3x16) = 48 bytes in the split. So, while many operators would
> assume a split of 2 blocks, the actual is 3 blocks. Setting the maximum split
> size to a multiple of the HDFS block size will make this behavior less
> confusing.
> The current setting is ~256MB and when this was introduced, the default HDFS
> block size was 64MB. That is a factor of 4x. However, now HDFS block sizes
> are 128MB by default, so I propose setting this to 4x128MB. The larger
> splits (fewer tasks) should give a nice performance boost for modern hardware.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21466) Increase Default Size of SPLIT_MAXSIZE
[
https://issues.apache.org/jira/browse/HIVE-21466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21466:
--
Status: Patch Available (was: Open)
> Increase Default Size of SPLIT_MAXSIZE
> --
>
> Key: HIVE-21466
> URL: https://issues.apache.org/jira/browse/HIVE-21466
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-21466.1.patch, HIVE-21466.1.patch
>
>
> {code:java}
> MAPREDMAXSPLITSIZE(FileInputFormat.SPLIT_MAXSIZE, 25600L, "", true),
> {code}
> [https://github.com/apache/hive/blob/8d4300a02691777fc96f33861ed27e64fed72f2c/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L682]
> This field specifies a maximum size for each MR (maybe other?) splits.
> This number should be a multiple of the HDFS Block size. The way that this
> maximum is implemented, is that each block is added to the split, and if the
> split grows to be larger than the maximum allowed, the split is submitted to
> the cluster and a new split is opened.
> So, imagine the following scenario:
> * HDFS block size of 16 bytes
> * Maximum size of 40 bytes
> This will produce a split with 3 blocks. (2x16) = 32; another block will be
> inserted, (3x16) = 48 bytes in the split. So, while many operators would
> assume a split of 2 blocks, the actual is 3 blocks. Setting the maximum split
> size to a multiple of the HDFS block size will make this behavior less
> confusing.
> The current setting is ~256MB and when this was introduced, the default HDFS
> block size was 64MB. That is a factor of 4x. However, now HDFS block sizes
> are 128MB by default, so I propose setting this to 4x128MB. The larger
> splits (fewer tasks) should give a nice performance boost for modern hardware.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21468) Case sensitivity in identifier names for JDBC storage handler
[
https://issues.apache.org/jira/browse/HIVE-21468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795538#comment-16795538
]
Hive QA commented on HIVE-21468:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962886/HIVE-21468.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 15834 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[jdbc_handler]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[strict_managed_tables_sysdb]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=169)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[external_jdbc_negative]
(batchId=100)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_invalid_place]
(batchId=99)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.lockConflictDbTable
(batchId=326)
org.apache.hive.service.server.TestInformationSchemaWithPrivilege.test
(batchId=259)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16564/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16564/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16564/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962886 - PreCommit-HIVE-Build
> Case sensitivity in identifier names for JDBC storage handler
> -
>
> Key: HIVE-21468
> URL: https://issues.apache.org/jira/browse/HIVE-21468
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21468.patch
>
>
> Currently, when Calcite generates the SQL query for the JDBC storage handler,
> it will ignore capitalization for the identifiers names, which can lead to
> errors at execution time (though the query is properly generated).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21468) Case sensitivity in identifier names for JDBC storage handler
[
https://issues.apache.org/jira/browse/HIVE-21468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795527#comment-16795527
]
Hive QA commented on HIVE-21468:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
15s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} jdbc-handler in master has 12 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 2 new + 15 unchanged - 0 fixed
= 17 total (was 15) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
24s{color} | {color:green} ql in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s{color} | {color:green} jdbc-handler generated 0 new + 11 unchanged - 1
fixed = 11 total (was 12) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16564/dev-support/hive-personality.sh
|
| git revision | master / 36bd89d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16564/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql jdbc-handler itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16564/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Case sensitivity in identifier names for JDBC storage handler
> -
>
> Key: HIVE-21468
> URL: https://issues.apache.org/jira/browse/HIVE-21468
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21468.patch
>
>
> Currently, when Calcite generates the SQL query for the JDBC storage handler,
> it will ignore capitalization for the identifiers names, which can lead to
> errors at execution time (though the query is properly generated).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21456) Hive Metastore HTTP Thrift
[ https://issues.apache.org/jira/browse/HIVE-21456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Khanna updated HIVE-21456: --- Issue Type: New Feature (was: Bug) > Hive Metastore HTTP Thrift > -- > > Key: HIVE-21456 > URL: https://issues.apache.org/jira/browse/HIVE-21456 > Project: Hive > Issue Type: New Feature > Components: Metastore, Standalone Metastore >Reporter: Amit Khanna >Assignee: Amit Khanna >Priority: Major > Attachments: HIVE-21456.2.patch, HIVE-21456.patch > > > Hive Metastore currently doesn't have support for HTTP transport because of > which it is not possible to access it via Knox. Adding support for Thrift > over HTTP transport will allow the clients to access via Knox -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21422) Add metrics to LRFU cache policy
[
https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795513#comment-16795513
]
Hive QA commented on HIVE-21422:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962876/HIVE-21422.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15833 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_invalid_place]
(batchId=99)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16563/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16563/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16563/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962876 - PreCommit-HIVE-Build
> Add metrics to LRFU cache policy
>
>
> Key: HIVE-21422
> URL: https://issues.apache.org/jira/browse/HIVE-21422
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Affects Versions: 4.0.0
>Reporter: Oliver Draese
>Assignee: Oliver Draese
>Priority: Major
> Labels: llap
> Fix For: 4.0.0
>
> Attachments: HIVE-21422.1.patch, HIVE-21422.patch
>
>
> The LRFU cache policy for the LLAP data cache doesn't provide enough insight
> to figure out, what is cached and why something might get evicted. This
> ticket is used to add Hadoop metrics 2 information (accessible via JMX) to
> the LRFU policy, providing following information:
> * How much memory is cached for data buffers
> * How much memory is cached for meta data buffers
> * How large is the min-heap of the cache policy
> * How long is the eviction short list (linked list)
> * How much memory is currently "locked" (buffers with positive reference
> count) and therefore in use by a query
> These new counters are found in the MX bean, following this path:
> Hadoop/LlapDaemon/LowLevelLrfuCachePolicy-
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21456) Hive Metastore HTTP Thrift
[ https://issues.apache.org/jira/browse/HIVE-21456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Khanna updated HIVE-21456: --- Attachment: HIVE-21456.2.patch > Hive Metastore HTTP Thrift > -- > > Key: HIVE-21456 > URL: https://issues.apache.org/jira/browse/HIVE-21456 > Project: Hive > Issue Type: Bug > Components: Metastore, Standalone Metastore >Reporter: Amit Khanna >Assignee: Amit Khanna >Priority: Major > Attachments: HIVE-21456.2.patch, HIVE-21456.patch > > > Hive Metastore currently doesn't have support for HTTP transport because of > which it is not possible to access it via Knox. Adding support for Thrift > over HTTP transport will allow the clients to access via Knox -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21468) Case sensitivity in identifier names for JDBC storage handler
[ https://issues.apache.org/jira/browse/HIVE-21468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-21468: --- Status: Patch Available (was: In Progress) > Case sensitivity in identifier names for JDBC storage handler > - > > Key: HIVE-21468 > URL: https://issues.apache.org/jira/browse/HIVE-21468 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > Currently, when Calcite generates the SQL query for the JDBC storage handler, > it will ignore capitalization for the identifiers names, which can lead to > errors at execution time (though the query is properly generated). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21422) Add metrics to LRFU cache policy
[
https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795484#comment-16795484
]
Hive QA commented on HIVE-21422:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
46s{color} | {color:blue} llap-server in master has 79 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} llap-server: The patch generated 21 new + 21 unchanged
- 3 fixed = 42 total (was 24) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16563/dev-support/hive-personality.sh
|
| git revision | master / 36bd89d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16563/yetus/diff-checkstyle-llap-server.txt
|
| modules | C: llap-server U: llap-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16563/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add metrics to LRFU cache policy
>
>
> Key: HIVE-21422
> URL: https://issues.apache.org/jira/browse/HIVE-21422
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Affects Versions: 4.0.0
>Reporter: Oliver Draese
>Assignee: Oliver Draese
>Priority: Major
> Labels: llap
> Fix For: 4.0.0
>
> Attachments: HIVE-21422.1.patch, HIVE-21422.patch
>
>
> The LRFU cache policy for the LLAP data cache doesn't provide enough insight
> to figure out, what is cached and why something might get evicted. This
> ticket is used to add Hadoop metrics 2 information (accessible via JMX) to
> the LRFU policy, providing following information:
> * How much memory is cached for data buffers
> * How much memory is cached for meta data buffers
> * How large is the min-heap of the cache policy
> * How long is the eviction short list (linked list)
> * How much memory is currently "locked" (buffers with positive reference
> count) and therefore in use by a query
> These new counters are found in the MX bean, following this path:
> Hadoop/LlapDaemon/LowLevelLrfuCachePolicy-
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21468) Case sensitivity in identifier names for JDBC storage handler
[ https://issues.apache.org/jira/browse/HIVE-21468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-21468: --- Attachment: HIVE-21468.patch > Case sensitivity in identifier names for JDBC storage handler > - > > Key: HIVE-21468 > URL: https://issues.apache.org/jira/browse/HIVE-21468 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-21468.patch > > > Currently, when Calcite generates the SQL query for the JDBC storage handler, > it will ignore capitalization for the identifiers names, which can lead to > errors at execution time (though the query is properly generated). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-21468) Case sensitivity in identifier names for JDBC storage handler
[ https://issues.apache.org/jira/browse/HIVE-21468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-21468 started by Jesus Camacho Rodriguez. -- > Case sensitivity in identifier names for JDBC storage handler > - > > Key: HIVE-21468 > URL: https://issues.apache.org/jira/browse/HIVE-21468 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > Currently, when Calcite generates the SQL query for the JDBC storage handler, > it will ignore capitalization for the identifiers names, which can lead to > errors at execution time (though the query is properly generated). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21468) Case sensitivity in identifier names for JDBC storage handler
[ https://issues.apache.org/jira/browse/HIVE-21468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-21468: -- > Case sensitivity in identifier names for JDBC storage handler > - > > Key: HIVE-21468 > URL: https://issues.apache.org/jira/browse/HIVE-21468 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > Currently, when Calcite generates the SQL query for the JDBC storage handler, > it will ignore capitalization for the identifiers names, which can lead to > errors at execution time (though the query is properly generated). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21467) Remove deprecated junit.framework.Assert imports
[
https://issues.apache.org/jira/browse/HIVE-21467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795475#comment-16795475
]
Hive QA commented on HIVE-21467:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
7s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} storage-api in master has 48 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} serde in master has 197 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
21s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} beeline in master has 44 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} hcatalog/core in master has 28 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} hcatalog/hcatalog-pig-adapter in master has 2 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} hcatalog/webhcat/svr in master has 96 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} hcatalog/streaming in master has 11 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
45s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
51s{color} | {color:blue} itests/util in master has 48 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} testutils/ptest2 in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
41s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 11s{color}
| {color:red} storage-api generated 2 new + 0 unchanged - 2 fixed = 2 total
(was 2) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
25s{color} | {color:red} patch/storage-api cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
36s{color} | {color:red} patch/common cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
42s{color} | {color:red} patch/serde cannot
[jira] [Commented] (HIVE-21467) Remove deprecated junit.framework.Assert imports
[
https://issues.apache.org/jira/browse/HIVE-21467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795467#comment-16795467
]
Hive QA commented on HIVE-21467:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962867/HIVE-21467.01.patch
{color:green}SUCCESS:{color} +1 due to 96 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 15833 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_invalid_place]
(batchId=99)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testRoundToDecimalPlaces
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorACos
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorASin
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorATan
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorAbs
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorCos
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorDegrees
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorExp
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorLn
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorLog10
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorLog2
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorRadians
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorRound
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorSign
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorSin
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorSqrt
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorTan
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorTypeCasts.testCastDoubleToTimestamp
(batchId=313)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorTypeCasts.testVectorCastLongToDouble
(batchId=313)
org.apache.hive.hcatalog.mapreduce.TestHCatPartitioned.testHCatPartitionedTable[1]
(batchId=209)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16562/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16562/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16562/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 21 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962867 - PreCommit-HIVE-Build
> Remove deprecated junit.framework.Assert imports
>
>
> Key: HIVE-21467
> URL: https://issues.apache.org/jira/browse/HIVE-21467
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Minor
> Labels: newbie
> Attachments: HIVE-21467.01.patch
>
>
> These imports trigger lots of warnings in ide, which could be annoying, and
> it can be replaced easily to org.junit.Assert, the signature and behavior are
> the same, so the tests should pass.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21466) Increase Default Size of SPLIT_MAXSIZE
[
https://issues.apache.org/jira/browse/HIVE-21466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795423#comment-16795423
]
Hive QA commented on HIVE-21466:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962854/HIVE-21466.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 15833 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_invalid_place]
(batchId=99)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query28]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query44]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query8]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[mv_query44]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query28]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query44]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query8]
(batchId=275)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16561/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16561/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16561/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962854 - PreCommit-HIVE-Build
> Increase Default Size of SPLIT_MAXSIZE
> --
>
> Key: HIVE-21466
> URL: https://issues.apache.org/jira/browse/HIVE-21466
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-21466.1.patch
>
>
> {code:java}
> MAPREDMAXSPLITSIZE(FileInputFormat.SPLIT_MAXSIZE, 25600L, "", true),
> {code}
> [https://github.com/apache/hive/blob/8d4300a02691777fc96f33861ed27e64fed72f2c/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L682]
> This field specifies a maximum size for each MR (maybe other?) splits.
> This number should be a multiple of the HDFS Block size. The way that this
> maximum is implemented, is that each block is added to the split, and if the
> split grows to be larger than the maximum allowed, the split is submitted to
> the cluster and a new split is opened.
> So, imagine the following scenario:
> * HDFS block size of 16 bytes
> * Maximum size of 40 bytes
> This will produce a split with 3 blocks. (2x16) = 32; another block will be
> inserted, (3x16) = 48 bytes in the split. So, while many operators would
> assume a split of 2 blocks, the actual is 3 blocks. Setting the maximum split
> size to a multiple of the HDFS block size will make this behavior less
> confusing.
> The current setting is ~256MB and when this was introduced, the default HDFS
> block size was 64MB. That is a factor of 4x. However, now HDFS block sizes
> are 128MB by default, so I propose setting this to 4x128MB. The larger
> splits (fewer tasks) should give a nice performance boost for modern hardware.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21422) Add metrics to LRFU cache policy
[ https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795410#comment-16795410 ] Gopal V commented on HIVE-21422: LGTM - +1 tests pending > Add metrics to LRFU cache policy > > > Key: HIVE-21422 > URL: https://issues.apache.org/jira/browse/HIVE-21422 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 4.0.0 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > Fix For: 4.0.0 > > Attachments: HIVE-21422.1.patch, HIVE-21422.patch > > > The LRFU cache policy for the LLAP data cache doesn't provide enough insight > to figure out, what is cached and why something might get evicted. This > ticket is used to add Hadoop metrics 2 information (accessible via JMX) to > the LRFU policy, providing following information: > * How much memory is cached for data buffers > * How much memory is cached for meta data buffers > * How large is the min-heap of the cache policy > * How long is the eviction short list (linked list) > * How much memory is currently "locked" (buffers with positive reference > count) and therefore in use by a query > These new counters are found in the MX bean, following this path: > Hadoop/LlapDaemon/LowLevelLrfuCachePolicy- > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21422) Add metrics to LRFU cache policy
[ https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795399#comment-16795399 ] Oliver Draese commented on HIVE-21422: -- Added pull request: https://github.com/apache/hive/pull/574 > Add metrics to LRFU cache policy > > > Key: HIVE-21422 > URL: https://issues.apache.org/jira/browse/HIVE-21422 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 4.0.0 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > Fix For: 4.0.0 > > Attachments: HIVE-21422.1.patch, HIVE-21422.patch > > > The LRFU cache policy for the LLAP data cache doesn't provide enough insight > to figure out, what is cached and why something might get evicted. This > ticket is used to add Hadoop metrics 2 information (accessible via JMX) to > the LRFU policy, providing following information: > * How much memory is cached for data buffers > * How much memory is cached for meta data buffers > * How large is the min-heap of the cache policy > * How long is the eviction short list (linked list) > * How much memory is currently "locked" (buffers with positive reference > count) and therefore in use by a query > These new counters are found in the MX bean, following this path: > Hadoop/LlapDaemon/LowLevelLrfuCachePolicy- > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (HIVE-21422) Add metrics to LRFU cache policy
[ https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21422: - Comment: was deleted (was: Added pull request for this patch: https://github.com/apache/hive/pull/563/files) > Add metrics to LRFU cache policy > > > Key: HIVE-21422 > URL: https://issues.apache.org/jira/browse/HIVE-21422 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 4.0.0 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > Fix For: 4.0.0 > > Attachments: HIVE-21422.1.patch, HIVE-21422.patch > > > The LRFU cache policy for the LLAP data cache doesn't provide enough insight > to figure out, what is cached and why something might get evicted. This > ticket is used to add Hadoop metrics 2 information (accessible via JMX) to > the LRFU policy, providing following information: > * How much memory is cached for data buffers > * How much memory is cached for meta data buffers > * How large is the min-heap of the cache policy > * How long is the eviction short list (linked list) > * How much memory is currently "locked" (buffers with positive reference > count) and therefore in use by a query > These new counters are found in the MX bean, following this path: > Hadoop/LlapDaemon/LowLevelLrfuCachePolicy- > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21466) Increase Default Size of SPLIT_MAXSIZE
[
https://issues.apache.org/jira/browse/HIVE-21466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795389#comment-16795389
]
Hive QA commented on HIVE-21466:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16561/dev-support/hive-personality.sh
|
| git revision | master / 36bd89d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16561/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Increase Default Size of SPLIT_MAXSIZE
> --
>
> Key: HIVE-21466
> URL: https://issues.apache.org/jira/browse/HIVE-21466
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-21466.1.patch
>
>
> {code:java}
> MAPREDMAXSPLITSIZE(FileInputFormat.SPLIT_MAXSIZE, 25600L, "", true),
> {code}
> [https://github.com/apache/hive/blob/8d4300a02691777fc96f33861ed27e64fed72f2c/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L682]
> This field specifies a maximum size for each MR (maybe other?) splits.
> This number should be a multiple of the HDFS Block size. The way that this
> maximum is implemented, is that each block is added to the split, and if the
> split grows to be larger than the maximum allowed, the split is submitted to
> the cluster and a new split is opened.
> So, imagine the following scenario:
> * HDFS block size of 16 bytes
> * Maximum size of 40 bytes
> This will produce a split with 3 blocks. (2x16) = 32; another block will be
> inserted, (3x16) = 48 bytes in the split. So, while many operators would
> assume a split of 2 blocks, the actual is 3 blocks. Setting the maximum split
> size to a multiple of the HDFS block size will make this behavior less
> confusing.
> The current setting is ~256MB and when this was introduced, the default HDFS
> block size was 64MB. That is a factor of 4x. However, now HDFS block sizes
> are 128MB by default, so I propose setting this to
[jira] [Commented] (HIVE-21422) Add metrics to LRFU cache policy
[ https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795387#comment-16795387 ] Oliver Draese commented on HIVE-21422: -- Added new counter output also to LLAP server's iomem web endpoint > Add metrics to LRFU cache policy > > > Key: HIVE-21422 > URL: https://issues.apache.org/jira/browse/HIVE-21422 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 4.0.0 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > Fix For: 4.0.0 > > Attachments: HIVE-21422.1.patch, HIVE-21422.patch > > > The LRFU cache policy for the LLAP data cache doesn't provide enough insight > to figure out, what is cached and why something might get evicted. This > ticket is used to add Hadoop metrics 2 information (accessible via JMX) to > the LRFU policy, providing following information: > * How much memory is cached for data buffers > * How much memory is cached for meta data buffers > * How large is the min-heap of the cache policy > * How long is the eviction short list (linked list) > * How much memory is currently "locked" (buffers with positive reference > count) and therefore in use by a query > These new counters are found in the MX bean, following this path: > Hadoop/LlapDaemon/LowLevelLrfuCachePolicy- > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21422) Add metrics to LRFU cache policy
[ https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21422: - Attachment: HIVE-21422.1.patch > Add metrics to LRFU cache policy > > > Key: HIVE-21422 > URL: https://issues.apache.org/jira/browse/HIVE-21422 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 4.0.0 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > Fix For: 4.0.0 > > Attachments: HIVE-21422.1.patch, HIVE-21422.patch > > > The LRFU cache policy for the LLAP data cache doesn't provide enough insight > to figure out, what is cached and why something might get evicted. This > ticket is used to add Hadoop metrics 2 information (accessible via JMX) to > the LRFU policy, providing following information: > * How much memory is cached for data buffers > * How much memory is cached for meta data buffers > * How large is the min-heap of the cache policy > * How long is the eviction short list (linked list) > * How much memory is currently "locked" (buffers with positive reference > count) and therefore in use by a query > These new counters are found in the MX bean, following this path: > Hadoop/LlapDaemon/LowLevelLrfuCachePolicy- > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21406) Add .factorypath files to .gitignore
[
https://issues.apache.org/jira/browse/HIVE-21406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795373#comment-16795373
]
Hive QA commented on HIVE-21406:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962850/HIVE-21406.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15833 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_invalid_place]
(batchId=99)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16560/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16560/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16560/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962850 - PreCommit-HIVE-Build
> Add .factorypath files to .gitignore
>
>
> Key: HIVE-21406
> URL: https://issues.apache.org/jira/browse/HIVE-21406
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Minor
> Attachments: HIVE-21406.01.patch, Screen Shot 2019-03-07 at 2.02.10
> PM.png
>
>
> .factorypath files are generated by eclipse and should be ignored
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21406) Add .factorypath files to .gitignore
[
https://issues.apache.org/jira/browse/HIVE-21406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795341#comment-16795341
]
Hive QA commented on HIVE-21406:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
52s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16560/dev-support/hive-personality.sh
|
| git revision | master / 36bd89d |
| modules | C: . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16560/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add .factorypath files to .gitignore
>
>
> Key: HIVE-21406
> URL: https://issues.apache.org/jira/browse/HIVE-21406
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Minor
> Attachments: HIVE-21406.01.patch, Screen Shot 2019-03-07 at 2.02.10
> PM.png
>
>
> .factorypath files are generated by eclipse and should be ignored
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21465) Introduce a base abstract VectorExpression to eliminate further code duplications
[
https://issues.apache.org/jira/browse/HIVE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795327#comment-16795327
]
Hive QA commented on HIVE-21465:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962847/HIVE-21465.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 125 failed/errored test(s), 15807 tests
executed
*Failed tests:*
{noformat}
TestCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=24)
[authorization_cli_stdconfigauth.q,vector_fullouter_mapjoin_1_fast.q,timestamp_comparison.q,parquet_vectorization_10.q,vector_left_outer_join.q,vectorization_parquet_ppd_decimal.q,stats10.q,vector_join_part_col_char.q,nonblock_op_deduplicate.q,udf_testlength2.q,column_pruner_multiple_children.q,udf_stddev.q,select_dummy_source.q,alter_varchar2.q,tez_multi_union.q,orc_diff_part_cols.q,partition_boolean.q,default_file_format.q,udf_to_utc_timestamp.q,windowing_ntile.q,autoColumnStats_1.q,alter_rename_partition.q,semijoin.q,authorization_create_func1.q,smb_mapjoin_3.q,udf_notop.q,input_testxpath3.q,pointlookup5.q,vector_null_map.q,array_map_access_nonconstant.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ansi_sql_arithmetic]
(batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cast_tinyint_to_double]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[convert_decimal64_to_decimal]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[decimal_5] (batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[decimal_udf] (batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_const_type]
(batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[outer_reference_windowed]
(batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_13]
(batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_17]
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_3]
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_div0]
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_not]
(batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[timestamptz_2]
(batchId=86)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_all_types]
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_case_when_1]
(batchId=92)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_1]
(batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_5]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_aggregate]
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_cast]
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_expressions]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_mapjoin]
(batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_math_funcs]
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_round]
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_udf2]
(batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_10]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_13]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_17]
(batchId=95)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_3]
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_not]
(batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_case]
(batchId=63)
org.apache.hadoop.hive.cli.TestCompareCliDriver.testCliDriver[llap_0]
(batchId=271)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[convert_decimal64_to_decimal]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_1]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_decimal64_reader]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_rebuild_dummy]
(batchId=166)
[jira] [Updated] (HIVE-21467) Remove deprecated junit.framework.Assert imports
[ https://issues.apache.org/jira/browse/HIVE-21467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21467: Status: Patch Available (was: Open) > Remove deprecated junit.framework.Assert imports > > > Key: HIVE-21467 > URL: https://issues.apache.org/jira/browse/HIVE-21467 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Minor > Labels: newbie > Attachments: HIVE-21467.01.patch > > > These imports trigger lots of warnings in ide, which could be annoying, and > it can be replaced easily to org.junit.Assert, the signature and behavior are > the same, so the tests should pass. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21467) Remove deprecated junit.framework.Assert imports
[ https://issues.apache.org/jira/browse/HIVE-21467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21467: Description: These imports trigger lots of warning in ide, which could be annoying, and it can be replaced easily to org.junit.Assert, the signature and behavior are the same, so the tests should pass. > Remove deprecated junit.framework.Assert imports > > > Key: HIVE-21467 > URL: https://issues.apache.org/jira/browse/HIVE-21467 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Priority: Minor > Labels: newbie > Attachments: HIVE-21467.01.patch > > > These imports trigger lots of warning in ide, which could be annoying, and it > can be replaced easily to org.junit.Assert, the signature and behavior are > the same, so the tests should pass. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21467) Remove deprecated junit.framework.Assert imports
[ https://issues.apache.org/jira/browse/HIVE-21467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21467: Description: These imports trigger lots of warnings in ide, which could be annoying, and it can be replaced easily to org.junit.Assert, the signature and behavior are the same, so the tests should pass. was: These imports trigger lots of warning in ide, which could be annoying, and it can be replaced easily to org.junit.Assert, the signature and behavior are the same, so the tests should pass. > Remove deprecated junit.framework.Assert imports > > > Key: HIVE-21467 > URL: https://issues.apache.org/jira/browse/HIVE-21467 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Minor > Labels: newbie > Attachments: HIVE-21467.01.patch > > > These imports trigger lots of warnings in ide, which could be annoying, and > it can be replaced easily to org.junit.Assert, the signature and behavior are > the same, so the tests should pass. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21467) Remove deprecated junit.framework.Assert imports
[ https://issues.apache.org/jira/browse/HIVE-21467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21467: Attachment: HIVE-21467.01.patch > Remove deprecated junit.framework.Assert imports > > > Key: HIVE-21467 > URL: https://issues.apache.org/jira/browse/HIVE-21467 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Minor > Labels: newbie > Attachments: HIVE-21467.01.patch > > > These imports trigger lots of warning in ide, which could be annoying, and it > can be replaced easily to org.junit.Assert, the signature and behavior are > the same, so the tests should pass. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21467) Remove deprecated junit.framework.Assert imports
[ https://issues.apache.org/jira/browse/HIVE-21467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor reassigned HIVE-21467: --- Assignee: Laszlo Bodor > Remove deprecated junit.framework.Assert imports > > > Key: HIVE-21467 > URL: https://issues.apache.org/jira/browse/HIVE-21467 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Minor > Labels: newbie > Attachments: HIVE-21467.01.patch > > > These imports trigger lots of warning in ide, which could be annoying, and it > can be replaced easily to org.junit.Assert, the signature and behavior are > the same, so the tests should pass. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21467) Remove deprecated junit.framework.Assert imports
[ https://issues.apache.org/jira/browse/HIVE-21467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21467: Priority: Minor (was: Major) > Remove deprecated junit.framework.Assert imports > > > Key: HIVE-21467 > URL: https://issues.apache.org/jira/browse/HIVE-21467 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21467) Remove deprecated junit.framework.Assert imports
[ https://issues.apache.org/jira/browse/HIVE-21467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21467: Labels: newbie (was: ) > Remove deprecated junit.framework.Assert imports > > > Key: HIVE-21467 > URL: https://issues.apache.org/jira/browse/HIVE-21467 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Priority: Minor > Labels: newbie > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-13517) Hive logs in Spark Executor and Driver should show thread-id.
[ https://issues.apache.org/jira/browse/HIVE-13517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharathkrishna Guruvayoor Murali reassigned HIVE-13517: --- Assignee: (was: Bharathkrishna Guruvayoor Murali) > Hive logs in Spark Executor and Driver should show thread-id. > - > > Key: HIVE-13517 > URL: https://issues.apache.org/jira/browse/HIVE-13517 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 1.2.1, 2.0.0 >Reporter: Szehon Ho >Priority: Major > Attachments: HIVE-13517.1.patch, HIVE-13517.2.patch, > executor-driver-log.PNG > > > In Spark, there might be more than one task running in one executor. > Similarly, there may be more than one thread running in Driver. > This makes debugging through the logs a nightmare. It would be great if there > could be thread-ids in the logs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21465) Introduce a base abstract VectorExpression to eliminate further code duplications
[
https://issues.apache.org/jira/browse/HIVE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795303#comment-16795303
]
Hive QA commented on HIVE-21465:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
21s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 3 new + 9 unchanged - 23 fixed
= 12 total (was 32) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
17s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16559/dev-support/hive-personality.sh
|
| git revision | master / 36bd89d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16559/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16559/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16559/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Introduce a base abstract VectorExpression to eliminate further code
> duplications
> -
>
> Key: HIVE-21465
> URL: https://issues.apache.org/jira/browse/HIVE-21465
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21465.01.patch
>
>
> In vectorized expressions, from time to time, I see a similar/same logic to
> be repeated, which always consists of some stuff like:
> 1. some inner loops (even the comments are the same)
> {code}
> grep -iRH "Set isNull before call in case it changes it mind" | grep
> "ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/"
> {code}
> 2. call to an actual "business logic" function ==> 1 line which could be
> abstracted... and it has been abstracted in hive codebase in n different
> cases.
> However, an abstract BaseVectorExpression would not force anybody to extend
> it instead of VectorExpression and not copy the code again, but it could be a
> good start and an example.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21430) INSERT into a dynamically partitioned table with hive.stats.autogather = false throws a MetaException
[
https://issues.apache.org/jira/browse/HIVE-21430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795264#comment-16795264
]
Hive QA commented on HIVE-21430:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962843/HIVE-21430.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15834 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_invalid_place]
(batchId=99)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16558/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16558/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16558/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962843 - PreCommit-HIVE-Build
> INSERT into a dynamically partitioned table with hive.stats.autogather =
> false throws a MetaException
> -
>
> Key: HIVE-21430
> URL: https://issues.apache.org/jira/browse/HIVE-21430
> Project: Hive
> Issue Type: Bug
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21430.01.patch, HIVE-21430.02.patch,
> metaexception_repro.patch,
> org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread-output.txt
>
> Original Estimate: 48h
> Time Spent: 50m
> Remaining Estimate: 47h 10m
>
> When the test TestStatsUpdaterThread#testTxnDynamicPartitions added in the
> attached patch is run it throws exception (full logs attached.)
> org.apache.hadoop.hive.metastore.api.MetaException: Cannot change stats state
> for a transactional table default.simple_stats without providing the
> transactional write state for verification (new write ID 5, valid write IDs
> null; current state \{"BASIC_STATS":"true","COLUMN_STATS":{"s":"true"}}; new
> state null
> at
> org.apache.hadoop.hive.metastore.ObjectStore.alterPartitionNoTxn(ObjectStore.java:4328)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21466) Increase Default Size of SPLIT_MAXSIZE
[
https://issues.apache.org/jira/browse/HIVE-21466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor reassigned HIVE-21466:
-
> Increase Default Size of SPLIT_MAXSIZE
> --
>
> Key: HIVE-21466
> URL: https://issues.apache.org/jira/browse/HIVE-21466
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-21466.1.patch
>
>
> {code:java}
> MAPREDMAXSPLITSIZE(FileInputFormat.SPLIT_MAXSIZE, 25600L, "", true),
> {code}
> [https://github.com/apache/hive/blob/8d4300a02691777fc96f33861ed27e64fed72f2c/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L682]
> This field specifies a maximum size for each MR (maybe other?) splits.
> This number should be a multiple of the HDFS Block size. The way that this
> maximum is implemented, is that each block is added to the split, and if the
> split grows to be larger than the maximum allowed, the split is submitted to
> the cluster and a new split is opened.
> So, imagine the following scenario:
> * HDFS block size of 16 bytes
> * Maximum size of 40 bytes
> This will produce a split with 3 blocks. (2x16) = 32; another block will be
> inserted, (3x16) = 48 bytes in the split. So, while many operators would
> assume a split of 2 blocks, the actual is 3 blocks. Setting the maximum split
> size to a multiple of the HDFS block size will make this behavior less
> confusing.
> The current setting is ~256MB and when this was introduced, the default HDFS
> block size was 64MB. That is a factor of 4x. However, now HDFS block sizes
> are 128MB by default, so I propose setting this to 4x128MB. The larger
> splits (fewer tasks) should give a nice performance boost for modern hardware.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21466) Increase Default Size of SPLIT_MAXSIZE
[
https://issues.apache.org/jira/browse/HIVE-21466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21466:
--
Status: Patch Available (was: Open)
> Increase Default Size of SPLIT_MAXSIZE
> --
>
> Key: HIVE-21466
> URL: https://issues.apache.org/jira/browse/HIVE-21466
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-21466.1.patch
>
>
> {code:java}
> MAPREDMAXSPLITSIZE(FileInputFormat.SPLIT_MAXSIZE, 25600L, "", true),
> {code}
> [https://github.com/apache/hive/blob/8d4300a02691777fc96f33861ed27e64fed72f2c/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L682]
> This field specifies a maximum size for each MR (maybe other?) splits.
> This number should be a multiple of the HDFS Block size. The way that this
> maximum is implemented, is that each block is added to the split, and if the
> split grows to be larger than the maximum allowed, the split is submitted to
> the cluster and a new split is opened.
> So, imagine the following scenario:
> * HDFS block size of 16 bytes
> * Maximum size of 40 bytes
> This will produce a split with 3 blocks. (2x16) = 32; another block will be
> inserted, (3x16) = 48 bytes in the split. So, while many operators would
> assume a split of 2 blocks, the actual is 3 blocks. Setting the maximum split
> size to a multiple of the HDFS block size will make this behavior less
> confusing.
> The current setting is ~256MB and when this was introduced, the default HDFS
> block size was 64MB. That is a factor of 4x. However, now HDFS block sizes
> are 128MB by default, so I propose setting this to 4x128MB. The larger
> splits (fewer tasks) should give a nice performance boost for modern hardware.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21466) Increase Default Size of SPLIT_MAXSIZE
[
https://issues.apache.org/jira/browse/HIVE-21466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21466:
--
Attachment: HIVE-21466.1.patch
> Increase Default Size of SPLIT_MAXSIZE
> --
>
> Key: HIVE-21466
> URL: https://issues.apache.org/jira/browse/HIVE-21466
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-21466.1.patch
>
>
> {code:java}
> MAPREDMAXSPLITSIZE(FileInputFormat.SPLIT_MAXSIZE, 25600L, "", true),
> {code}
> [https://github.com/apache/hive/blob/8d4300a02691777fc96f33861ed27e64fed72f2c/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L682]
> This field specifies a maximum size for each MR (maybe other?) splits.
> This number should be a multiple of the HDFS Block size. The way that this
> maximum is implemented, is that each block is added to the split, and if the
> split grows to be larger than the maximum allowed, the split is submitted to
> the cluster and a new split is opened.
> So, imagine the following scenario:
> * HDFS block size of 16 bytes
> * Maximum size of 40 bytes
> This will produce a split with 3 blocks. (2x16) = 32; another block will be
> inserted, (3x16) = 48 bytes in the split. So, while many operators would
> assume a split of 2 blocks, the actual is 3 blocks. Setting the maximum split
> size to a multiple of the HDFS block size will make this behavior less
> confusing.
> The current setting is ~256MB and when this was introduced, the default HDFS
> block size was 64MB. That is a factor of 4x. However, now HDFS block sizes
> are 128MB by default, so I propose setting this to 4x128MB. The larger
> splits (fewer tasks) should give a nice performance boost for modern hardware.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21430) INSERT into a dynamically partitioned table with hive.stats.autogather = false throws a MetaException
[
https://issues.apache.org/jira/browse/HIVE-21430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795236#comment-16795236
]
Hive QA commented on HIVE-21430:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
18s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16558/dev-support/hive-personality.sh
|
| git revision | master / 36bd89d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16558/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> INSERT into a dynamically partitioned table with hive.stats.autogather =
> false throws a MetaException
> -
>
> Key: HIVE-21430
> URL: https://issues.apache.org/jira/browse/HIVE-21430
> Project: Hive
> Issue Type: Bug
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21430.01.patch, HIVE-21430.02.patch,
> metaexception_repro.patch,
> org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread-output.txt
>
> Original Estimate: 48h
> Time Spent: 50m
> Remaining Estimate: 47h 10m
>
> When the test TestStatsUpdaterThread#testTxnDynamicPartitions added in the
> attached patch is run it throws exception (full logs attached.)
> org.apache.hadoop.hive.metastore.api.MetaException: Cannot change stats state
> for a transactional table default.simple_stats without providing the
> transactional write state for verification (new write ID 5, valid write IDs
> null; current state \{"BASIC_STATS":"true","COLUMN_STATS":{"s":"true"}}; new
> state null
> at
> org.apache.hadoop.hive.metastore.ObjectStore.alterPartitionNoTxn(ObjectStore.java:4328)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21406) Add .factorypath files to .gitignore
[ https://issues.apache.org/jira/browse/HIVE-21406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21406: Attachment: HIVE-21406.01.patch > Add .factorypath files to .gitignore > > > Key: HIVE-21406 > URL: https://issues.apache.org/jira/browse/HIVE-21406 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Minor > Attachments: HIVE-21406.01.patch, Screen Shot 2019-03-07 at 2.02.10 > PM.png > > > .factorypath files are generated by eclipse and should be ignored -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21406) Add .factorypath files to .gitignore
[ https://issues.apache.org/jira/browse/HIVE-21406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21406: Attachment: (was: HIVE-21406.01.patch) > Add .factorypath files to .gitignore > > > Key: HIVE-21406 > URL: https://issues.apache.org/jira/browse/HIVE-21406 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Minor > Attachments: HIVE-21406.01.patch, Screen Shot 2019-03-07 at 2.02.10 > PM.png > > > .factorypath files are generated by eclipse and should be ignored -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-21465) Introduce a base abstract VectorExpression to eliminate further code duplications
[
https://issues.apache.org/jira/browse/HIVE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795201#comment-16795201
]
Laszlo Bodor edited comment on HIVE-21465 at 3/18/19 4:51 PM:
--
uploaded an initial version into review board
https://reviews.apache.org/r/70231/
this introduces generic BaseVectorExpression, which can ensure correct input /
output columnvector types at compile time
beforeLoop is an example for an optionally extendable method (which should be
optimized by JIT in case it's a noop I guess)
was (Author: abstractdog):
uploaded an initial version into review board
https://reviews.apache.org/r/70231/
this introduces generic BaseVectorExpression, which can ensure correct input /
output columnvector types at compile time
> Introduce a base abstract VectorExpression to eliminate further code
> duplications
> -
>
> Key: HIVE-21465
> URL: https://issues.apache.org/jira/browse/HIVE-21465
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21465.01.patch
>
>
> In vectorized expressions, from time to time, I see a similar/same logic to
> be repeated, which always consists of some stuff like:
> 1. some inner loops (even the comments are the same)
> {code}
> grep -iRH "Set isNull before call in case it changes it mind" | grep
> "ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/"
> {code}
> 2. call to an actual "business logic" function ==> 1 line which could be
> abstracted... and it has been abstracted in hive codebase in n different
> cases.
> However, an abstract BaseVectorExpression would not force anybody to extend
> it instead of VectorExpression and not copy the code again, but it could be a
> good start and an example.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21462) Upgrading SQL server backed metastore when changing data type of a column with constraints
[ https://issues.apache.org/jira/browse/HIVE-21462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795214#comment-16795214 ] Daniel Dai commented on HIVE-21462: --- +1 > Upgrading SQL server backed metastore when changing data type of a column > with constraints > -- > > Key: HIVE-21462 > URL: https://issues.apache.org/jira/browse/HIVE-21462 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21462.01.patch > > Original Estimate: 24h > Time Spent: 10m > Remaining Estimate: 23h 50m > > SQL server does not allow changing data type of a column which has a > constraint or an index on it. The constraint or the index needs to be dropped > before changing the data type and needs to be recreated after that. Metastore > upgrade scripts aren't doing this and thus upgrade fails. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-21465) Introduce a base abstract VectorExpression to eliminate further code duplications
[
https://issues.apache.org/jira/browse/HIVE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795201#comment-16795201
]
Laszlo Bodor edited comment on HIVE-21465 at 3/18/19 4:56 PM:
--
uploaded an initial version into review board
https://reviews.apache.org/r/70231/
this introduces generic BaseVectorExpression, which can ensure correct input /
output columnvector types at compile time + a some func refactors, which could
show how this base class could prevent code repetition
beforeLoop is an example for an optionally extendable method (which should be
optimized by JIT in case it's a noop I guess)
was (Author: abstractdog):
uploaded an initial version into review board
https://reviews.apache.org/r/70231/
this introduces generic BaseVectorExpression, which can ensure correct input /
output columnvector types at compile time
beforeLoop is an example for an optionally extendable method (which should be
optimized by JIT in case it's a noop I guess)
> Introduce a base abstract VectorExpression to eliminate further code
> duplications
> -
>
> Key: HIVE-21465
> URL: https://issues.apache.org/jira/browse/HIVE-21465
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21465.01.patch
>
>
> In vectorized expressions, from time to time, I see a similar/same logic to
> be repeated, which always consists of some stuff like:
> 1. some inner loops (even the comments are the same)
> {code}
> grep -iRH "Set isNull before call in case it changes it mind" | grep
> "ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/"
> {code}
> 2. call to an actual "business logic" function ==> 1 line which could be
> abstracted... and it has been abstracted in hive codebase in n different
> cases.
> However, an abstract BaseVectorExpression would not force anybody to extend
> it instead of VectorExpression and not copy the code again, but it could be a
> good start and an example.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21406) Add .factorypath files to .gitignore
[
https://issues.apache.org/jira/browse/HIVE-21406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795205#comment-16795205
]
Hive QA commented on HIVE-21406:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962823/HIVE-21406.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15833 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[timestamptz_2]
(batchId=86)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_invalid_place]
(batchId=99)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16557/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16557/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16557/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962823 - PreCommit-HIVE-Build
> Add .factorypath files to .gitignore
>
>
> Key: HIVE-21406
> URL: https://issues.apache.org/jira/browse/HIVE-21406
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Minor
> Attachments: HIVE-21406.01.patch, Screen Shot 2019-03-07 at 2.02.10
> PM.png
>
>
> .factorypath files are generated by eclipse and should be ignored
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21465) Introduce a base abstract VectorExpression to eliminate further code duplications
[
https://issues.apache.org/jira/browse/HIVE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-21465:
Attachment: HIVE-21465.01.patch
> Introduce a base abstract VectorExpression to eliminate further code
> duplications
> -
>
> Key: HIVE-21465
> URL: https://issues.apache.org/jira/browse/HIVE-21465
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21465.01.patch
>
>
> From time to time I see a similar/same logic to be repeated, which always
> consists of some stuff like:
> 1. some inner loops (even the comments are the same)
> {code}
> grep -iRH "Set isNull before call in case it changes it mind" | grep
> "ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/"
> {code}
> 2. call to an actual "business logic" function ==> 1 line which could be
> abstracted... and it has been abstracted in hive codebase in n different
> cases.
> However, an abstract BaseVectorExpression would not force anybody to extend
> it instead of VectorExpression and not copy the code again, but it could be a
> good start and an example.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21465) Introduce a base abstract VectorExpression to eliminate further code duplications
[
https://issues.apache.org/jira/browse/HIVE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795201#comment-16795201
]
Laszlo Bodor commented on HIVE-21465:
-
uploaded an initial version into review board
https://reviews.apache.org/r/70231/
this introduces generic BaseVectorExpression, which can ensure correct input /
output columnvector types at compile time
> Introduce a base abstract VectorExpression to eliminate further code
> duplications
> -
>
> Key: HIVE-21465
> URL: https://issues.apache.org/jira/browse/HIVE-21465
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21465.01.patch
>
>
> In vectorized expressions, from time to time, I see a similar/same logic to
> be repeated, which always consists of some stuff like:
> 1. some inner loops (even the comments are the same)
> {code}
> grep -iRH "Set isNull before call in case it changes it mind" | grep
> "ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/"
> {code}
> 2. call to an actual "business logic" function ==> 1 line which could be
> abstracted... and it has been abstracted in hive codebase in n different
> cases.
> However, an abstract BaseVectorExpression would not force anybody to extend
> it instead of VectorExpression and not copy the code again, but it could be a
> good start and an example.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21465) Introduce a base abstract VectorExpression to eliminate further code duplications
[
https://issues.apache.org/jira/browse/HIVE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-21465:
Description:
In vectorized expressions, from time to time, I see a similar/same logic to be
repeated, which always consists of some stuff like:
1. some inner loops (even the comments are the same)
{code}
grep -iRH "Set isNull before call in case it changes it mind" | grep
"ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/"
{code}
2. call to an actual "business logic" function ==> 1 line which could be
abstracted... and it has been abstracted in hive codebase in n different cases.
However, an abstract BaseVectorExpression would not force anybody to extend it
instead of VectorExpression and not copy the code again, but it could be a good
start and an example.
was:
>From time to time I see a similar/same logic to be repeated, which always
>consists of some stuff like:
1. some inner loops (even the comments are the same)
{code}
grep -iRH "Set isNull before call in case it changes it mind" | grep
"ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/"
{code}
2. call to an actual "business logic" function ==> 1 line which could be
abstracted... and it has been abstracted in hive codebase in n different cases.
However, an abstract BaseVectorExpression would not force anybody to extend it
instead of VectorExpression and not copy the code again, but it could be a good
start and an example.
> Introduce a base abstract VectorExpression to eliminate further code
> duplications
> -
>
> Key: HIVE-21465
> URL: https://issues.apache.org/jira/browse/HIVE-21465
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21465.01.patch
>
>
> In vectorized expressions, from time to time, I see a similar/same logic to
> be repeated, which always consists of some stuff like:
> 1. some inner loops (even the comments are the same)
> {code}
> grep -iRH "Set isNull before call in case it changes it mind" | grep
> "ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/"
> {code}
> 2. call to an actual "business logic" function ==> 1 line which could be
> abstracted... and it has been abstracted in hive codebase in n different
> cases.
> However, an abstract BaseVectorExpression would not force anybody to extend
> it instead of VectorExpression and not copy the code again, but it could be a
> good start and an example.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21465) Introduce a base abstract VectorExpression to eliminate further code duplications
[
https://issues.apache.org/jira/browse/HIVE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-21465:
Status: Patch Available (was: Open)
> Introduce a base abstract VectorExpression to eliminate further code
> duplications
> -
>
> Key: HIVE-21465
> URL: https://issues.apache.org/jira/browse/HIVE-21465
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21465.01.patch
>
>
> From time to time I see a similar/same logic to be repeated, which always
> consists of some stuff like:
> 1. some inner loops (even the comments are the same)
> {code}
> grep -iRH "Set isNull before call in case it changes it mind" | grep
> "ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/"
> {code}
> 2. call to an actual "business logic" function ==> 1 line which could be
> abstracted... and it has been abstracted in hive codebase in n different
> cases.
> However, an abstract BaseVectorExpression would not force anybody to extend
> it instead of VectorExpression and not copy the code again, but it could be a
> good start and an example.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21465) Introduce a base abstract VectorExpression to eliminate further code duplications
[
https://issues.apache.org/jira/browse/HIVE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-21465:
Attachment: (was: HIVE-21465.01.patch)
> Introduce a base abstract VectorExpression to eliminate further code
> duplications
> -
>
> Key: HIVE-21465
> URL: https://issues.apache.org/jira/browse/HIVE-21465
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21465.01.patch
>
>
> From time to time I see a similar/same logic to be repeated, which always
> consists of some stuff like:
> 1. some inner loops (even the comments are the same)
> {code}
> grep -iRH "Set isNull before call in case it changes it mind" | grep
> "ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/"
> {code}
> 2. call to an actual "business logic" function ==> 1 line which could be
> abstracted... and it has been abstracted in hive codebase in n different
> cases.
> However, an abstract BaseVectorExpression would not force anybody to extend
> it instead of VectorExpression and not copy the code again, but it could be a
> good start and an example.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21465) Introduce a base abstract VectorExpression to eliminate further code duplications
[
https://issues.apache.org/jira/browse/HIVE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-21465:
Attachment: HIVE-21465.01.patch
> Introduce a base abstract VectorExpression to eliminate further code
> duplications
> -
>
> Key: HIVE-21465
> URL: https://issues.apache.org/jira/browse/HIVE-21465
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21465.01.patch
>
>
> From time to time I see a similar/same logic to be repeated, which always
> consists of some stuff like:
> 1. some inner loops (even the comments are the same)
> {code}
> grep -iRH "Set isNull before call in case it changes it mind" | grep
> "ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/"
> {code}
> 2. call to an actual "business logic" function ==> 1 line which could be
> abstracted... and it has been abstracted in hive codebase in n different
> cases.
> However, an abstract BaseVectorExpression would not force anybody to extend
> it instead of VectorExpression and not copy the code again, but it could be a
> good start and an example.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21430) INSERT into a dynamically partitioned table with hive.stats.autogather = false throws a MetaException
[
https://issues.apache.org/jira/browse/HIVE-21430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795181#comment-16795181
]
Ashutosh Bapat commented on HIVE-21430:
---
[~maheshk114], I have updated PR addressing your comments in a separate commit.
Please review.
> INSERT into a dynamically partitioned table with hive.stats.autogather =
> false throws a MetaException
> -
>
> Key: HIVE-21430
> URL: https://issues.apache.org/jira/browse/HIVE-21430
> Project: Hive
> Issue Type: Bug
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21430.01.patch, HIVE-21430.02.patch,
> metaexception_repro.patch,
> org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread-output.txt
>
> Original Estimate: 48h
> Time Spent: 50m
> Remaining Estimate: 47h 10m
>
> When the test TestStatsUpdaterThread#testTxnDynamicPartitions added in the
> attached patch is run it throws exception (full logs attached.)
> org.apache.hadoop.hive.metastore.api.MetaException: Cannot change stats state
> for a transactional table default.simple_stats without providing the
> transactional write state for verification (new write ID 5, valid write IDs
> null; current state \{"BASIC_STATS":"true","COLUMN_STATS":{"s":"true"}}; new
> state null
> at
> org.apache.hadoop.hive.metastore.ObjectStore.alterPartitionNoTxn(ObjectStore.java:4328)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21430) INSERT into a dynamically partitioned table with hive.stats.autogather = false throws a MetaException
[
https://issues.apache.org/jira/browse/HIVE-21430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Bapat updated HIVE-21430:
--
Attachment: HIVE-21430.02.patch
Status: Patch Available (was: In Progress)
> INSERT into a dynamically partitioned table with hive.stats.autogather =
> false throws a MetaException
> -
>
> Key: HIVE-21430
> URL: https://issues.apache.org/jira/browse/HIVE-21430
> Project: Hive
> Issue Type: Bug
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21430.01.patch, HIVE-21430.02.patch,
> metaexception_repro.patch,
> org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread-output.txt
>
> Original Estimate: 48h
> Time Spent: 50m
> Remaining Estimate: 47h 10m
>
> When the test TestStatsUpdaterThread#testTxnDynamicPartitions added in the
> attached patch is run it throws exception (full logs attached.)
> org.apache.hadoop.hive.metastore.api.MetaException: Cannot change stats state
> for a transactional table default.simple_stats without providing the
> transactional write state for verification (new write ID 5, valid write IDs
> null; current state \{"BASIC_STATS":"true","COLUMN_STATS":{"s":"true"}}; new
> state null
> at
> org.apache.hadoop.hive.metastore.ObjectStore.alterPartitionNoTxn(ObjectStore.java:4328)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21430) INSERT into a dynamically partitioned table with hive.stats.autogather = false throws a MetaException
[
https://issues.apache.org/jira/browse/HIVE-21430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Bapat updated HIVE-21430:
--
Status: In Progress (was: Patch Available)
> INSERT into a dynamically partitioned table with hive.stats.autogather =
> false throws a MetaException
> -
>
> Key: HIVE-21430
> URL: https://issues.apache.org/jira/browse/HIVE-21430
> Project: Hive
> Issue Type: Bug
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21430.01.patch, metaexception_repro.patch,
> org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread-output.txt
>
> Original Estimate: 48h
> Time Spent: 50m
> Remaining Estimate: 47h 10m
>
> When the test TestStatsUpdaterThread#testTxnDynamicPartitions added in the
> attached patch is run it throws exception (full logs attached.)
> org.apache.hadoop.hive.metastore.api.MetaException: Cannot change stats state
> for a transactional table default.simple_stats without providing the
> transactional write state for verification (new write ID 5, valid write IDs
> null; current state \{"BASIC_STATS":"true","COLUMN_STATS":{"s":"true"}}; new
> state null
> at
> org.apache.hadoop.hive.metastore.ObjectStore.alterPartitionNoTxn(ObjectStore.java:4328)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21430) INSERT into a dynamically partitioned table with hive.stats.autogather = false throws a MetaException
[
https://issues.apache.org/jira/browse/HIVE-21430?focusedWorklogId=214856=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-214856
]
ASF GitHub Bot logged work on HIVE-21430:
-
Author: ASF GitHub Bot
Created on: 18/Mar/19 16:18
Start Date: 18/Mar/19 16:18
Worklog Time Spent: 10m
Work Description: ashutosh-bapat commented on pull request #572:
HIVE-21430 : INSERT into a dynamically partitioned table with autogather stats
disabled throws a MetaException
URL: https://github.com/apache/hive/pull/572#discussion_r266524121
##
File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
##
@@ -2377,9 +2382,15 @@ private void setStatsPropAndAlterPartitions(boolean
resetStatistics, Table tbl,
LOG.debug(sb.toString());
}
+String validWriteIdList = null;
+Long writeId = 0L;
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 214856)
Time Spent: 50m (was: 40m)
Remaining Estimate: 47h 10m (was: 47h 20m)
> INSERT into a dynamically partitioned table with hive.stats.autogather =
> false throws a MetaException
> -
>
> Key: HIVE-21430
> URL: https://issues.apache.org/jira/browse/HIVE-21430
> Project: Hive
> Issue Type: Bug
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21430.01.patch, metaexception_repro.patch,
> org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread-output.txt
>
> Original Estimate: 48h
> Time Spent: 50m
> Remaining Estimate: 47h 10m
>
> When the test TestStatsUpdaterThread#testTxnDynamicPartitions added in the
> attached patch is run it throws exception (full logs attached.)
> org.apache.hadoop.hive.metastore.api.MetaException: Cannot change stats state
> for a transactional table default.simple_stats without providing the
> transactional write state for verification (new write ID 5, valid write IDs
> null; current state \{"BASIC_STATS":"true","COLUMN_STATS":{"s":"true"}}; new
> state null
> at
> org.apache.hadoop.hive.metastore.ObjectStore.alterPartitionNoTxn(ObjectStore.java:4328)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21430) INSERT into a dynamically partitioned table with hive.stats.autogather = false throws a MetaException
[
https://issues.apache.org/jira/browse/HIVE-21430?focusedWorklogId=214855=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-214855
]
ASF GitHub Bot logged work on HIVE-21430:
-
Author: ASF GitHub Bot
Created on: 18/Mar/19 16:18
Start Date: 18/Mar/19 16:18
Worklog Time Spent: 10m
Work Description: ashutosh-bapat commented on pull request #572:
HIVE-21430 : INSERT into a dynamically partitioned table with autogather stats
disabled throws a MetaException
URL: https://github.com/apache/hive/pull/572#discussion_r266524070
##
File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
##
@@ -2626,6 +2637,7 @@ private void constructOneLBLocationMap(FileStatus fSta,
}
boolean isTxnTable = AcidUtils.isTransactionalTable(tbl);
+AcidUtils.TableSnapshot tableSnapshot = getTableSnapshot(tbl, writeId);
Review comment:
Thanks for catching this. Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 214855)
Time Spent: 40m (was: 0.5h)
Remaining Estimate: 47h 20m (was: 47.5h)
> INSERT into a dynamically partitioned table with hive.stats.autogather =
> false throws a MetaException
> -
>
> Key: HIVE-21430
> URL: https://issues.apache.org/jira/browse/HIVE-21430
> Project: Hive
> Issue Type: Bug
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21430.01.patch, metaexception_repro.patch,
> org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread-output.txt
>
> Original Estimate: 48h
> Time Spent: 40m
> Remaining Estimate: 47h 20m
>
> When the test TestStatsUpdaterThread#testTxnDynamicPartitions added in the
> attached patch is run it throws exception (full logs attached.)
> org.apache.hadoop.hive.metastore.api.MetaException: Cannot change stats state
> for a transactional table default.simple_stats without providing the
> transactional write state for verification (new write ID 5, valid write IDs
> null; current state \{"BASIC_STATS":"true","COLUMN_STATS":{"s":"true"}}; new
> state null
> at
> org.apache.hadoop.hive.metastore.ObjectStore.alterPartitionNoTxn(ObjectStore.java:4328)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21455) Too verbose logging in AvroGenericRecordReader
[
https://issues.apache.org/jira/browse/HIVE-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795125#comment-16795125
]
Miklos Szurap commented on HIVE-21455:
--
To add to that, the fix is trivial so I've not included tests.
I suggest to:
* keep the "Found the Avro schema in the job" messages there (so we can see
how many files/splits are processed)
* in case DEBUG level logging is enabled, we are logging the schema too - if
that's needed for troubleshooting.
> Too verbose logging in AvroGenericRecordReader
> --
>
> Key: HIVE-21455
> URL: https://issues.apache.org/jira/browse/HIVE-21455
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Miklos Szurap
>Assignee: Miklos Szurap
>Priority: Minor
> Attachments: HIVE-21455.patch
>
>
> {{AvroGenericRecordReader}} logs the Avro schema for each datafile. It is too
> verbose, likely we don't need to log that on INFO level.
> For example a table:
> {noformat}
> create table avro_tbl (c1 string, c2 int, c3 float) stored as avro;
> {noformat}
> and querying it with a select star - with 3 datafiles HiveServer2 logs the
> following:
> {noformat}
> 2019-03-15 09:18:35,999 INFO org.apache.hadoop.mapred.FileInputFormat:
> [HiveServer2-Handler-Pool: Thread-64]: Total input paths to process : 3
> 2019-03-15 09:18:35,999 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,004 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,010 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> {noformat}
> This has a huge performance and storage penalty on a table with big schema
> and thousands of datafiles.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21455) Too verbose logging in AvroGenericRecordReader
[
https://issues.apache.org/jira/browse/HIVE-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szurap updated HIVE-21455:
-
Attachment: HIVE-21455.patch
> Too verbose logging in AvroGenericRecordReader
> --
>
> Key: HIVE-21455
> URL: https://issues.apache.org/jira/browse/HIVE-21455
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Miklos Szurap
>Priority: Minor
> Attachments: HIVE-21455.patch
>
>
> {{AvroGenericRecordReader}} logs the Avro schema for each datafile. It is too
> verbose, likely we don't need to log that on INFO level.
> For example a table:
> {noformat}
> create table avro_tbl (c1 string, c2 int, c3 float) stored as avro;
> {noformat}
> and querying it with a select star - with 3 datafiles HiveServer2 logs the
> following:
> {noformat}
> 2019-03-15 09:18:35,999 INFO org.apache.hadoop.mapred.FileInputFormat:
> [HiveServer2-Handler-Pool: Thread-64]: Total input paths to process : 3
> 2019-03-15 09:18:35,999 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,004 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,010 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> {noformat}
> This has a huge performance and storage penalty on a table with big schema
> and thousands of datafiles.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21455) Too verbose logging in AvroGenericRecordReader
[
https://issues.apache.org/jira/browse/HIVE-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szurap reassigned HIVE-21455:
Assignee: Miklos Szurap
> Too verbose logging in AvroGenericRecordReader
> --
>
> Key: HIVE-21455
> URL: https://issues.apache.org/jira/browse/HIVE-21455
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Miklos Szurap
>Assignee: Miklos Szurap
>Priority: Minor
> Attachments: HIVE-21455.patch
>
>
> {{AvroGenericRecordReader}} logs the Avro schema for each datafile. It is too
> verbose, likely we don't need to log that on INFO level.
> For example a table:
> {noformat}
> create table avro_tbl (c1 string, c2 int, c3 float) stored as avro;
> {noformat}
> and querying it with a select star - with 3 datafiles HiveServer2 logs the
> following:
> {noformat}
> 2019-03-15 09:18:35,999 INFO org.apache.hadoop.mapred.FileInputFormat:
> [HiveServer2-Handler-Pool: Thread-64]: Total input paths to process : 3
> 2019-03-15 09:18:35,999 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,004 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,010 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> {noformat}
> This has a huge performance and storage penalty on a table with big schema
> and thousands of datafiles.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21455) Too verbose logging in AvroGenericRecordReader
[
https://issues.apache.org/jira/browse/HIVE-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795123#comment-16795123
]
Miklos Szurap commented on HIVE-21455:
--
Hi [~pvary], I've created a patch, could you review it? Thx.
> Too verbose logging in AvroGenericRecordReader
> --
>
> Key: HIVE-21455
> URL: https://issues.apache.org/jira/browse/HIVE-21455
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Miklos Szurap
>Assignee: Miklos Szurap
>Priority: Minor
> Attachments: HIVE-21455.patch
>
>
> {{AvroGenericRecordReader}} logs the Avro schema for each datafile. It is too
> verbose, likely we don't need to log that on INFO level.
> For example a table:
> {noformat}
> create table avro_tbl (c1 string, c2 int, c3 float) stored as avro;
> {noformat}
> and querying it with a select star - with 3 datafiles HiveServer2 logs the
> following:
> {noformat}
> 2019-03-15 09:18:35,999 INFO org.apache.hadoop.mapred.FileInputFormat:
> [HiveServer2-Handler-Pool: Thread-64]: Total input paths to process : 3
> 2019-03-15 09:18:35,999 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,004 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,010 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> {noformat}
> This has a huge performance and storage penalty on a table with big schema
> and thousands of datafiles.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21465) Introduce a base abstract VectorExpression to eliminate further code duplications
[
https://issues.apache.org/jira/browse/HIVE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-21465:
Description:
>From time to time I see a similar/same logic to be repeated, which always
>consists of some stuff like:
1. some inner loops (even the comments are the same)
{code}
grep -iRH "Set isNull before call in case it changes it mind" | grep
"ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/"
{code}
2. call to an actual "business logic" function ==> 1 line which could be
abstracted... and it has been abstracted in hive codebase in n different cases.
However, an abstract BaseVectorExpression would not force anybody to extend it
instead of VectorExpression and not copy the code again, but it could be a good
start and an example.
> Introduce a base abstract VectorExpression to eliminate further code
> duplications
> -
>
> Key: HIVE-21465
> URL: https://issues.apache.org/jira/browse/HIVE-21465
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
>
> From time to time I see a similar/same logic to be repeated, which always
> consists of some stuff like:
> 1. some inner loops (even the comments are the same)
> {code}
> grep -iRH "Set isNull before call in case it changes it mind" | grep
> "ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/"
> {code}
> 2. call to an actual "business logic" function ==> 1 line which could be
> abstracted... and it has been abstracted in hive codebase in n different
> cases.
> However, an abstract BaseVectorExpression would not force anybody to extend
> it instead of VectorExpression and not copy the code again, but it could be a
> good start and an example.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21462) Upgrading SQL server backed metastore when changing data type of a column with constraints
[
https://issues.apache.org/jira/browse/HIVE-21462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795110#comment-16795110
]
Hive QA commented on HIVE-21462:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962818/HIVE-21462.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15833 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_invalid_place]
(batchId=99)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16556/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16556/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16556/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962818 - PreCommit-HIVE-Build
> Upgrading SQL server backed metastore when changing data type of a column
> with constraints
> --
>
> Key: HIVE-21462
> URL: https://issues.apache.org/jira/browse/HIVE-21462
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21462.01.patch
>
> Original Estimate: 24h
> Time Spent: 10m
> Remaining Estimate: 23h 50m
>
> SQL server does not allow changing data type of a column which has a
> constraint or an index on it. The constraint or the index needs to be dropped
> before changing the data type and needs to be recreated after that. Metastore
> upgrade scripts aren't doing this and thus upgrade fails.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21465) Introduce a base abstract VectorExpression to eliminate further code duplications
[ https://issues.apache.org/jira/browse/HIVE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor reassigned HIVE-21465: --- Assignee: Laszlo Bodor > Introduce a base abstract VectorExpression to eliminate further code > duplications > - > > Key: HIVE-21465 > URL: https://issues.apache.org/jira/browse/HIVE-21465 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21462) Upgrading SQL server backed metastore when changing data type of a column with constraints
[
https://issues.apache.org/jira/browse/HIVE-21462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795086#comment-16795086
]
Hive QA commented on HIVE-21462:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
36s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 21 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16556/dev-support/hive-personality.sh
|
| git revision | master / 36bd89d |
| Default Java | 1.8.0_111 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16556/yetus/whitespace-tabs.txt
|
| modules | C: standalone-metastore/metastore-server . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16556/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrading SQL server backed metastore when changing data type of a column
> with constraints
> --
>
> Key: HIVE-21462
> URL: https://issues.apache.org/jira/browse/HIVE-21462
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21462.01.patch
>
> Original Estimate: 24h
> Time Spent: 10m
> Remaining Estimate: 23h 50m
>
> SQL server does not allow changing data type of a column which has a
> constraint or an index on it. The constraint or the index needs to be dropped
> before changing the data type and needs to be recreated after that. Metastore
> upgrade scripts aren't doing this and thus upgrade fails.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21406) Add .factorypath files to .gitignore
[ https://issues.apache.org/jira/browse/HIVE-21406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21406: Attachment: HIVE-21406.01.patch > Add .factorypath files to .gitignore > > > Key: HIVE-21406 > URL: https://issues.apache.org/jira/browse/HIVE-21406 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Minor > Attachments: HIVE-21406.01.patch, Screen Shot 2019-03-07 at 2.02.10 > PM.png > > > .factorypath files are generated by eclipse and should be ignored -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21406) Add .factorypath files to .gitignore
[ https://issues.apache.org/jira/browse/HIVE-21406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21406: Attachment: (was: HIVE-21406.01.patch) > Add .factorypath files to .gitignore > > > Key: HIVE-21406 > URL: https://issues.apache.org/jira/browse/HIVE-21406 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Minor > Attachments: HIVE-21406.01.patch, Screen Shot 2019-03-07 at 2.02.10 > PM.png > > > .factorypath files are generated by eclipse and should be ignored -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795056#comment-16795056
]
Hive QA commented on HIVE-21446:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962808/HIVE-21446.02.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15833 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_groupby_reduce]
(batchId=61)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_invalid_place]
(batchId=99)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16555/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16555/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16555/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962808 - PreCommit-HIVE-Build
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch, HIVE-21446.02.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21372) Use Apache Commons IO To Read Stream To String
[ https://issues.apache.org/jira/browse/HIVE-21372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795040#comment-16795040 ] Ashutosh Chauhan commented on HIVE-21372: - +1 > Use Apache Commons IO To Read Stream To String > -- > > Key: HIVE-21372 > URL: https://issues.apache.org/jira/browse/HIVE-21372 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Trivial > Fix For: 4.0.0 > > Attachments: HIVE-21372.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21462) Upgrading SQL server backed metastore when changing data type of a column with constraints
[ https://issues.apache.org/jira/browse/HIVE-21462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21462: -- Attachment: HIVE-21462.01.patch Status: Patch Available (was: Open) > Upgrading SQL server backed metastore when changing data type of a column > with constraints > -- > > Key: HIVE-21462 > URL: https://issues.apache.org/jira/browse/HIVE-21462 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21462.01.patch > > Original Estimate: 24h > Time Spent: 10m > Remaining Estimate: 23h 50m > > SQL server does not allow changing data type of a column which has a > constraint or an index on it. The constraint or the index needs to be dropped > before changing the data type and needs to be recreated after that. Metastore > upgrade scripts aren't doing this and thus upgrade fails. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795028#comment-16795028
]
Hive QA commented on HIVE-21446:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
21s{color} | {color:blue} shims/common in master has 6 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} shims/0.23 in master has 7 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
8s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} shims/common: The patch generated 0 new + 94
unchanged - 1 fixed = 94 total (was 95) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch 0.23 passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 2 new + 19 unchanged - 2 fixed
= 21 total (was 21) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16555/dev-support/hive-personality.sh
|
| git revision | master / 36bd89d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16555/yetus/diff-checkstyle-ql.txt
|
| modules | C: shims/common shims/0.23 common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16555/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
>
[jira] [Updated] (HIVE-21264) Improvements Around CharTypeInfo
[
https://issues.apache.org/jira/browse/HIVE-21264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21264:
--
Attachment: (was: HIVE-21264.3.patch)
> Improvements Around CharTypeInfo
>
>
> Key: HIVE-21264
> URL: https://issues.apache.org/jira/browse/HIVE-21264
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21264.1.patch, HIVE-21264.2.patch,
> HIVE-21264.3.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The {{CharTypeInfo}} stores the type name of the data type (char/varchar) and
> the length (1-255). {{CharTypeInfo}} objects are often getting cached once
> they are created.
> The {{hashcode()}} and {{equals()}} of its sub-classes varchar and char are
> inconsistent.
> * Make hashcode and equals consistent (and fast)
> * Simplify the {{getQualifiedName}} implementation and reduce the scope to
> protected
> * Other related nits
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21073) Remove Extra String Object
[
https://issues.apache.org/jira/browse/HIVE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21073:
--
Attachment: (was: HIVE-21073.1.patch)
> Remove Extra String Object
> --
>
> Key: HIVE-21073
> URL: https://issues.apache.org/jira/browse/HIVE-21073
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.1.1
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Trivial
> Fix For: 4.0.0
>
> Attachments: HIVE-21073.1.patch
>
>
> {code}
> public static String generatePath(Path baseURI, String filename) {
> String path = new String(baseURI + Path.SEPARATOR + filename);
> return path;
> }
> public static String generateFileName(Byte tag, String bigBucketFileName) {
> String fileName = new String("MapJoin-" + tag + "-" + bigBucketFileName +
> suffix);
> return fileName;
> }
> {code}
> It's a bit odd to be performing string concatenation and then wrapping the
> results in a new string. This is creating superfluous String objects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21264) Improvements Around CharTypeInfo
[
https://issues.apache.org/jira/browse/HIVE-21264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21264:
--
Attachment: (was: HIVE-21264.3.patch)
> Improvements Around CharTypeInfo
>
>
> Key: HIVE-21264
> URL: https://issues.apache.org/jira/browse/HIVE-21264
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21264.1.patch, HIVE-21264.2.patch,
> HIVE-21264.3.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The {{CharTypeInfo}} stores the type name of the data type (char/varchar) and
> the length (1-255). {{CharTypeInfo}} objects are often getting cached once
> they are created.
> The {{hashcode()}} and {{equals()}} of its sub-classes varchar and char are
> inconsistent.
> * Make hashcode and equals consistent (and fast)
> * Simplify the {{getQualifiedName}} implementation and reduce the scope to
> protected
> * Other related nits
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21371) Make NonSyncByteArrayOutputStream Overflow Conscious
[
https://issues.apache.org/jira/browse/HIVE-21371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21371:
--
Attachment: (was: HIVE-21371.2.patch)
> Make NonSyncByteArrayOutputStream Overflow Conscious
> -
>
> Key: HIVE-21371
> URL: https://issues.apache.org/jira/browse/HIVE-21371
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-21371.1.patch, HIVE-21371.2.patch
>
>
> {code:java|title=NonSyncByteArrayOutputStream}
> private int enLargeBuffer(int increment) {
> int temp = count + increment;
> int newLen = temp;
> if (temp > buf.length) {
> if ((buf.length << 1) > temp) {
> newLen = buf.length << 1;
> }
> byte newbuf[] = new byte[newLen];
> System.arraycopy(buf, 0, newbuf, 0, count);
> buf = newbuf;
> }
> return newLen;
> }
> {code}
> This will fail if the array is 2GB or larger because it will double the size
> every time without consideration for the 4GB limit on arrays.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21371) Make NonSyncByteArrayOutputStream Overflow Conscious
[
https://issues.apache.org/jira/browse/HIVE-21371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21371:
--
Attachment: (was: HIVE-21371.2.patch)
> Make NonSyncByteArrayOutputStream Overflow Conscious
> -
>
> Key: HIVE-21371
> URL: https://issues.apache.org/jira/browse/HIVE-21371
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-21371.1.patch, HIVE-21371.2.patch
>
>
> {code:java|title=NonSyncByteArrayOutputStream}
> private int enLargeBuffer(int increment) {
> int temp = count + increment;
> int newLen = temp;
> if (temp > buf.length) {
> if ((buf.length << 1) > temp) {
> newLen = buf.length << 1;
> }
> byte newbuf[] = new byte[newLen];
> System.arraycopy(buf, 0, newbuf, 0, count);
> buf = newbuf;
> }
> return newLen;
> }
> {code}
> This will fail if the array is 2GB or larger because it will double the size
> every time without consideration for the 4GB limit on arrays.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
