[jira] [Updated] (HIVE-22645) Jline can break bash terminal behavior
[ https://issues.apache.org/jira/browse/HIVE-22645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-22645: Attachment: HIVE-22645.03.patch > Jline can break bash terminal behavior > -- > > Key: HIVE-22645 > URL: https://issues.apache.org/jira/browse/HIVE-22645 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22645.01.patch, HIVE-22645.01.patch, > HIVE-22645.02.patch, HIVE-22645.03.patch, HIVE-22645.03.patch > > > After HIVE-21954, running a qtest can break bash terminal (but not zsh), in a > way that, e.g. enter key doesn't break line... > As identified by [~kgyrtkirk], this line could cause this: > https://github.com/apache/hive/commit/d645d827d95de36175194407bd1e2f6725362aff#diff-c64cf2b501f04e6710bbc3fcd079156fR84 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-18325) Config to do case unaware schema evolution to ORC reader.
[ https://issues.apache.org/jira/browse/HIVE-18325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003921#comment-17003921 ] Jiayi Liu commented on HIVE-18325: -- Hi, this JIRA can solve this problem https://issues.apache.org/jira/browse/HIVE-20126 > Config to do case unaware schema evolution to ORC reader. > - > > Key: HIVE-18325 > URL: https://issues.apache.org/jira/browse/HIVE-18325 > Project: Hive > Issue Type: Bug > Components: ORC >Reporter: piyush mukati >Priority: Critical > > in case of orc data reader schema passed by hive are all small cases and if > the column name stored in the file has any uppercase, it will return null > values for those columns even if the data is present in the file. > Column name matching while schema evolution should be case unaware. > we need to pass config for same from hive. the > config(orc.schema.evolution.case.sensitive) in orc will be exposed by > https://issues.apache.org/jira/browse/ORC-264 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22670) ArrayIndexOutOfBoundsException when vectorized reader is used for reading a parquet file
[
https://issues.apache.org/jira/browse/HIVE-22670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003859#comment-17003859
]
Ganesha Shreedhara commented on HIVE-22670:
---
[~Ferd] , [~csun] Please review the patch.
> ArrayIndexOutOfBoundsException when vectorized reader is used for reading a
> parquet file
>
>
> Key: HIVE-22670
> URL: https://issues.apache.org/jira/browse/HIVE-22670
> Project: Hive
> Issue Type: Bug
> Components: Parquet, Vectorization
>Affects Versions: 3.1.2, 2.3.6
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-22670.1.patch
>
>
> ArrayIndexOutOfBoundsException is getting thrown while decoding dictionaryIds
> of a row group in parquet file with vectorization enabled.
> *Exception stack trace:*
> {code:java}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainBinaryDictionary.decodeToBinary(PlainValuesDictionary.java:122)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.ParquetDataColumnReaderFactory$DefaultParquetDataColumnReader.readString(ParquetDataColumnReaderFactory.java:95)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedPrimitiveColumnReader.decodeDictionaryIds(VectorizedPrimitiveColumnReader.java:467)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedPrimitiveColumnReader.readBatch(VectorizedPrimitiveColumnReader.java:68)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:410)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
> ... 24 more{code}
>
> This issue seems to be caused by re-using the same dictionary column vector
> while reading consecutive row groups. This looks like one of the corner case
> bug which occurs for a certain distribution of dictionary/plain encoded data
> while we read/populate the underlying bit packed dictionary data into a
> column-vector based data structure.
> Similar issue issue is reported in spark (Ref:
> https://issues.apache.org/jira/browse/SPARK-16334)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-13745) UDF current_date、current_timestamp、unix_timestamp NPE
[
https://issues.apache.org/jira/browse/HIVE-13745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003853#comment-17003853
]
Hive QA commented on HIVE-13745:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923177/HIVE-13745.2-branch-2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10756 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=227)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_tableproperty_optimize]
(batchId=22)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_parquet_types]
(batchId=155)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=117)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20024/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20024/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20024/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12923177 - PreCommit-HIVE-Build
> UDF current_date、current_timestamp、unix_timestamp NPE
> -
>
> Key: HIVE-13745
> URL: https://issues.apache.org/jira/browse/HIVE-13745
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Biao Wu
>Assignee: Biao Wu
>Priority: Major
> Attachments: HIVE-13745.1.patch, HIVE-13745.2-branch-2.patch,
> HIVE-13745.patch
>
>
> NullPointerException when current_date is used in mapreduce
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-13745) UDF current_date、current_timestamp、unix_timestamp NPE
[
https://issues.apache.org/jira/browse/HIVE-13745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003831#comment-17003831
]
caixiaofeng commented on HIVE-13745:
Problem occur in hive-3.1.0
SQL: select unix_timestamp (max('2019-09-09'),'-MM-DD') ;
ERROR LOG:
2019-12-27 09:45:56,286 INFO [main]
org.apache.hadoop.hive.ql.exec.SerializationUtilities: Deserializing ReduceWork
using kryo
2019-12-27 09:45:56,539 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities:
Deserialized plan (via FILE) - name: null size: 3.13KB
2019-12-27 09:45:56,562 INFO [main] ExecReducer:
Id =3
Id =8
<\Children>
Id = 3 null<\Parent>
<\FS>
<\Children>
<\Parent>
<\GBY>
2019-12-27 09:45:56,562 INFO [main]
org.apache.hadoop.hive.ql.exec.GroupByOperator: Initializing operator GBY[3]
2019-12-27 09:45:56,583 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.lang.RuntimeException: Error in configuring
object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:113)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:79)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:411)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:393)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:110)
... 9 more
Caused by: java.lang.RuntimeException: Reduce operator initialization failed
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:155)
... 14 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDFToUnixTimeStamp.initializeInput(GenericUDFToUnixTimeStamp.java:87)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDFUnixTimeStamp.initializeInput(GenericUDFUnixTimeStamp.java:44)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDFToUnixTimeStamp.initialize(GenericUDFToUnixTimeStamp.java:71)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:148)
at
org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:146)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.initializeOp(GroupByOperator.java:295)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:148)
... 14 more
2019-12-27 09:45:56,591 INFO [main] org.apache.hadoop.mapred.Task: Running
cleanup for the task
2019-12-27 09:45:56,697 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping ReduceTask metrics
system...
2019-12-27 09:45:56,697 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ReduceTask metrics system
stopped.
2019-12-27 09:45:56,698 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ReduceTask metrics system
shutdown complete.
> UDF current_date、current_timestamp、unix_timestamp NPE
> -
>
> Key: HIVE-13745
> URL: https://issues.apache.org/jira/browse/HIVE-13745
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Biao Wu
>Assignee: Biao Wu
>Priority: Major
> Attachments: HIVE-13745.1.patch, HIVE-13745.2-branch-2.patch,
> HIVE-13745.patch
>
>
> NullPointerException when current_date is used in mapreduce
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22645) Jline can break bash terminal behavior
[
https://issues.apache.org/jira/browse/HIVE-22645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003815#comment-17003815
]
Hive QA commented on HIVE-22645:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12989506/HIVE-22645.03.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17786 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.TestTxnCommands.testMergeDeleteUpdate (batchId=356)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20023/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20023/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20023/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12989506 - PreCommit-HIVE-Build
> Jline can break bash terminal behavior
> --
>
> Key: HIVE-22645
> URL: https://issues.apache.org/jira/browse/HIVE-22645
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22645.01.patch, HIVE-22645.01.patch,
> HIVE-22645.02.patch, HIVE-22645.03.patch
>
>
> After HIVE-21954, running a qtest can break bash terminal (but not zsh), in a
> way that, e.g. enter key doesn't break line...
> As identified by [~kgyrtkirk], this line could cause this:
> https://github.com/apache/hive/commit/d645d827d95de36175194407bd1e2f6725362aff#diff-c64cf2b501f04e6710bbc3fcd079156fR84
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22645) Jline can break bash terminal behavior
[
https://issues.apache.org/jira/browse/HIVE-22645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003813#comment-17003813
]
Hive QA commented on HIVE-22645:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
57s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
18s{color} | {color:blue} standalone-metastore/metastore-server in master has
181 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} cli in master has 9 extant Findbugs warnings. {color}
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
44s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} standalone-metastore: The patch generated 0 new + 18
unchanged - 1 fixed = 18 total (was 19) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} standalone-metastore/metastore-server: The patch
generated 0 new + 18 unchanged - 1 fixed = 18 total (was 19) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} cli: The patch generated 0 new + 7 unchanged - 2
fixed = 7 total (was 9) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
2s{color} | {color:green} root: The patch generated 0 new + 25 unchanged - 3
fixed = 25 total (was 28) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20023/dev-support/hive-personality.sh
|
| git revision | master / 6d4c37c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore standalone-metastore/metastore-server cli .
U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20023/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Jline can break bash terminal behavior
> --
>
> Key: HIVE-22645
> URL: https://issues.apache.org/jira/browse/HIVE-22645
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: László Bodor
>
[jira] [Updated] (HIVE-22645) Jline can break bash terminal behavior
[ https://issues.apache.org/jira/browse/HIVE-22645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-22645: Attachment: HIVE-22645.03.patch > Jline can break bash terminal behavior > -- > > Key: HIVE-22645 > URL: https://issues.apache.org/jira/browse/HIVE-22645 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22645.01.patch, HIVE-22645.01.patch, > HIVE-22645.02.patch, HIVE-22645.03.patch > > > After HIVE-21954, running a qtest can break bash terminal (but not zsh), in a > way that, e.g. enter key doesn't break line... > As identified by [~kgyrtkirk], this line could cause this: > https://github.com/apache/hive/commit/d645d827d95de36175194407bd1e2f6725362aff#diff-c64cf2b501f04e6710bbc3fcd079156fR84 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22649) Fix TestHiveCli: scratchdir should be writable
[
https://issues.apache.org/jira/browse/HIVE-22649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-22649:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Fix TestHiveCli: scratchdir should be writable
> --
>
> Key: HIVE-22649
> URL: https://issues.apache.org/jira/browse/HIVE-22649
> Project: Hive
> Issue Type: Sub-task
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22649.1.patch, HIVE-22649.2.patch
>
>
> Error applying authorization policy on hive configuration: The dir: /tmp/hive
> on HDFS should be writable. Current permissions are: rwxr-xr-x
> SessionState.java
> {code}
> private Path createRootHDFSDir(HiveConf conf) throws IOException {
> Path rootHDFSDirPath = new Path(HiveConf.getVar(conf,
> HiveConf.ConfVars.SCRATCHDIR));
> *Utilities.ensurePathIsWritable(rootHDFSDirPath, conf);*
> return rootHDFSDirPath;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22649) Fix TestHiveCli: scratchdir should be writable
[
https://issues.apache.org/jira/browse/HIVE-22649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003788#comment-17003788
]
László Bodor commented on HIVE-22649:
-
thanks for the patch [~dkuzmenko] and the review [~rameshkumar]
> Fix TestHiveCli: scratchdir should be writable
> --
>
> Key: HIVE-22649
> URL: https://issues.apache.org/jira/browse/HIVE-22649
> Project: Hive
> Issue Type: Sub-task
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22649.1.patch, HIVE-22649.2.patch
>
>
> Error applying authorization policy on hive configuration: The dir: /tmp/hive
> on HDFS should be writable. Current permissions are: rwxr-xr-x
> SessionState.java
> {code}
> private Path createRootHDFSDir(HiveConf conf) throws IOException {
> Path rootHDFSDirPath = new Path(HiveConf.getVar(conf,
> HiveConf.ConfVars.SCRATCHDIR));
> *Utilities.ensurePathIsWritable(rootHDFSDirPath, conf);*
> return rootHDFSDirPath;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22659) JClouds needs to be updated to 2.1.3 in ptest
[
https://issues.apache.org/jira/browse/HIVE-22659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-22659:
Summary: JClouds needs to be updated to 2.1.3 in ptest (was: JCoud needs
to be updated to 2.1.3 in ptest)
> JClouds needs to be updated to 2.1.3 in ptest
> -
>
> Key: HIVE-22659
> URL: https://issues.apache.org/jira/browse/HIVE-22659
> Project: Hive
> Issue Type: Bug
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Attachments: HIVE-22659.0.patch
>
>
> Since a couple of days ptest responded 404 to test queries coming in from
> jenkins side.
> I took a look into the issue and saw this exception on
> hiveptest-server-upstream side:
> {code:java}
> Caused by: java.lang.IllegalStateException: zone
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/us-central1-d
> not present in
> [https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/asia-east1-a
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/asia-east1-b
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/asia-east1-c
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/asia-east2-c
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/asia-east2-b
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/asia-east2-a
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/asia-northeast1-a
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/asia-northeast1-b
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/asia-northeast1-c
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/asia-northeast2-b
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/asia-northeast2-c
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/asia-northeast2-a
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/asia-south1-b
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/asia-south1-a
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/asia-south1-c
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/asia-southeast1-a
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/asia-southeast1-b
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/asia-southeast1-c
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/australia-southeast1-c
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/australia-southeast1-a
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/australia-southeast1-b
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/europe-north1-b
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/europe-north1-c
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/europe-north1-a
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/europe-west1-b
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/europe-west1-c
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/europe-west1-d
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/europe-west2-a
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/europe-west2-b
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/europe-west2-c
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/europe-west3-c
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/europe-west3-a
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/europe-west3-b
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/europe-west4-c
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/europe-west4-b
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/europe-west4-a
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/europe-west6-b
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/europe-west6-c
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/europe-west6-a
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/northamerica-northeast1-a
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/northamerica-northeast1-b
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/northamerica-northeast1-c
> https://www.googleapis.com/compute/v1/projects/gcp-hive-upstream/zones/southamerica-east1-a
>
[jira] [Updated] (HIVE-22671) TestHiveCli is flaky
[
https://issues.apache.org/jira/browse/HIVE-22671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-22671:
Resolution: Cannot Reproduce
Status: Resolved (was: Patch Available)
> TestHiveCli is flaky
>
>
> Key: HIVE-22671
> URL: https://issues.apache.org/jira/browse/HIVE-22671
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-22671.01.patch
>
>
> Fails while running scripts @Before, the actual problem is something like
> this:
> {code}
> 2019-12-24T06:11:16,065 DEBUG [main] exec.Utilities: HDFS dir: /tmp/hive with
> schema null, permission: rwxr-xr-x
> 2019-12-24T06:11:16,065 ERROR [main] cli.TestHiveCli: Failed due to the
> error:Connecting to jdbc:hive2://
> Hive Session ID = 92e1327d-4267-4efc-af7d-6bd2deeba7d7
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/hiveptest/34.70.195.43-hiveptest-0/apache-github-source-source/conf/ivysettings.xml
> will be used
> Error applying authorization policy on hive configuration: The dir: /tmp/hive
> on HDFS should be writable. Current permissions are: rwxr-xr-x
> Hive Session ID = 5d752eea-9bb5-42b4-953d-750f766fefaa
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/hiveptest/34.70.195.43-hiveptest-0/apache-github-source-source/conf/ivysettings.xml
> will be used
> Error applying authorization policy on hive configuration: The dir: /tmp/hive
> on HDFS should be writable. Current permissions are: rwxr-xr-x
> Connection is already closed.
> {code}
> locally it passes:
> {code}
> 2019-12-26T06:23:45,535 DEBUG [main] cli.CLIService: SessionHandle
> [ee1f45b2-cb74-4b0e-9c4e-5ddbc47a94ae]: closeSession()
> 2019-12-26T06:23:45,540 DEBUG [main] cli.TestHiveCli: Connecting to
> jdbc:hive2://
> Hive Session ID = feba5d41-796f-406e-a017-bce19c28fe09
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/abstractdog/apache/hive/conf/ivysettings.xml will be used
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/abstractdog/apache/hive/conf/ivysettings.xml will be used
> Connected to: Apache Hive (version 4.0.0-SNAPSHOT)
> Driver: Hive JDBC (version 4.0.0-SNAPSHOT)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> hive> create database if not exists test;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22671) TestHiveCli is flaky
[
https://issues.apache.org/jira/browse/HIVE-22671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003736#comment-17003736
]
László Bodor commented on HIVE-22671:
-
thanks [~dkuzmenko], closing this duplicate
> TestHiveCli is flaky
>
>
> Key: HIVE-22671
> URL: https://issues.apache.org/jira/browse/HIVE-22671
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-22671.01.patch
>
>
> Fails while running scripts @Before, the actual problem is something like
> this:
> {code}
> 2019-12-24T06:11:16,065 DEBUG [main] exec.Utilities: HDFS dir: /tmp/hive with
> schema null, permission: rwxr-xr-x
> 2019-12-24T06:11:16,065 ERROR [main] cli.TestHiveCli: Failed due to the
> error:Connecting to jdbc:hive2://
> Hive Session ID = 92e1327d-4267-4efc-af7d-6bd2deeba7d7
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/hiveptest/34.70.195.43-hiveptest-0/apache-github-source-source/conf/ivysettings.xml
> will be used
> Error applying authorization policy on hive configuration: The dir: /tmp/hive
> on HDFS should be writable. Current permissions are: rwxr-xr-x
> Hive Session ID = 5d752eea-9bb5-42b4-953d-750f766fefaa
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/hiveptest/34.70.195.43-hiveptest-0/apache-github-source-source/conf/ivysettings.xml
> will be used
> Error applying authorization policy on hive configuration: The dir: /tmp/hive
> on HDFS should be writable. Current permissions are: rwxr-xr-x
> Connection is already closed.
> {code}
> locally it passes:
> {code}
> 2019-12-26T06:23:45,535 DEBUG [main] cli.CLIService: SessionHandle
> [ee1f45b2-cb74-4b0e-9c4e-5ddbc47a94ae]: closeSession()
> 2019-12-26T06:23:45,540 DEBUG [main] cli.TestHiveCli: Connecting to
> jdbc:hive2://
> Hive Session ID = feba5d41-796f-406e-a017-bce19c28fe09
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/abstractdog/apache/hive/conf/ivysettings.xml will be used
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/abstractdog/apache/hive/conf/ivysettings.xml will be used
> Connected to: Apache Hive (version 4.0.0-SNAPSHOT)
> Driver: Hive JDBC (version 4.0.0-SNAPSHOT)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> hive> create database if not exists test;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22649) Fix TestHiveCli: scratchdir should be writable
[
https://issues.apache.org/jira/browse/HIVE-22649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003735#comment-17003735
]
László Bodor commented on HIVE-22649:
-
+1
I'm about to merge this shortly, test fix looks good
> Fix TestHiveCli: scratchdir should be writable
> --
>
> Key: HIVE-22649
> URL: https://issues.apache.org/jira/browse/HIVE-22649
> Project: Hive
> Issue Type: Sub-task
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22649.1.patch, HIVE-22649.2.patch
>
>
> Error applying authorization policy on hive configuration: The dir: /tmp/hive
> on HDFS should be writable. Current permissions are: rwxr-xr-x
> SessionState.java
> {code}
> private Path createRootHDFSDir(HiveConf conf) throws IOException {
> Path rootHDFSDirPath = new Path(HiveConf.getVar(conf,
> HiveConf.ConfVars.SCRATCHDIR));
> *Utilities.ensurePathIsWritable(rootHDFSDirPath, conf);*
> return rootHDFSDirPath;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22671) TestHiveCli is flaky
[
https://issues.apache.org/jira/browse/HIVE-22671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003702#comment-17003702
]
Hive QA commented on HIVE-22671:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12989497/HIVE-22671.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17762 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20022/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20022/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20022/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12989497 - PreCommit-HIVE-Build
> TestHiveCli is flaky
>
>
> Key: HIVE-22671
> URL: https://issues.apache.org/jira/browse/HIVE-22671
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-22671.01.patch
>
>
> Fails while running scripts @Before, the actual problem is something like
> this:
> {code}
> 2019-12-24T06:11:16,065 DEBUG [main] exec.Utilities: HDFS dir: /tmp/hive with
> schema null, permission: rwxr-xr-x
> 2019-12-24T06:11:16,065 ERROR [main] cli.TestHiveCli: Failed due to the
> error:Connecting to jdbc:hive2://
> Hive Session ID = 92e1327d-4267-4efc-af7d-6bd2deeba7d7
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/hiveptest/34.70.195.43-hiveptest-0/apache-github-source-source/conf/ivysettings.xml
> will be used
> Error applying authorization policy on hive configuration: The dir: /tmp/hive
> on HDFS should be writable. Current permissions are: rwxr-xr-x
> Hive Session ID = 5d752eea-9bb5-42b4-953d-750f766fefaa
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/hiveptest/34.70.195.43-hiveptest-0/apache-github-source-source/conf/ivysettings.xml
> will be used
> Error applying authorization policy on hive configuration: The dir: /tmp/hive
> on HDFS should be writable. Current permissions are: rwxr-xr-x
> Connection is already closed.
> {code}
> locally it passes:
> {code}
> 2019-12-26T06:23:45,535 DEBUG [main] cli.CLIService: SessionHandle
> [ee1f45b2-cb74-4b0e-9c4e-5ddbc47a94ae]: closeSession()
> 2019-12-26T06:23:45,540 DEBUG [main] cli.TestHiveCli: Connecting to
> jdbc:hive2://
> Hive Session ID = feba5d41-796f-406e-a017-bce19c28fe09
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/abstractdog/apache/hive/conf/ivysettings.xml will be used
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/abstractdog/apache/hive/conf/ivysettings.xml will be used
> Connected to: Apache Hive (version 4.0.0-SNAPSHOT)
> Driver: Hive JDBC (version 4.0.0-SNAPSHOT)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> hive> create database if not exists test;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22670) ArrayIndexOutOfBoundsException when vectorized reader is used for reading a parquet file
[
https://issues.apache.org/jira/browse/HIVE-22670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003688#comment-17003688
]
Ganesha Shreedhara commented on HIVE-22670:
---
Test failures are mostly due to configuration issue ( Ref:
[https://builds.apache.org/job/PreCommit-HIVE-Build/20020/testReport/org.apache.hive.beeline.cli/TestHiveCli/testUseCurrentDB1/]
)
> ArrayIndexOutOfBoundsException when vectorized reader is used for reading a
> parquet file
>
>
> Key: HIVE-22670
> URL: https://issues.apache.org/jira/browse/HIVE-22670
> Project: Hive
> Issue Type: Bug
> Components: Parquet, Vectorization
>Affects Versions: 3.1.2, 2.3.6
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-22670.1.patch
>
>
> ArrayIndexOutOfBoundsException is getting thrown while decoding dictionaryIds
> of a row group in parquet file with vectorization enabled.
> *Exception stack trace:*
> {code:java}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainBinaryDictionary.decodeToBinary(PlainValuesDictionary.java:122)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.ParquetDataColumnReaderFactory$DefaultParquetDataColumnReader.readString(ParquetDataColumnReaderFactory.java:95)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedPrimitiveColumnReader.decodeDictionaryIds(VectorizedPrimitiveColumnReader.java:467)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedPrimitiveColumnReader.readBatch(VectorizedPrimitiveColumnReader.java:68)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:410)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
> ... 24 more{code}
>
> This issue seems to be caused by re-using the same dictionary column vector
> while reading consecutive row groups. This looks like one of the corner case
> bug which occurs for a certain distribution of dictionary/plain encoded data
> while we read/populate the underlying bit packed dictionary data into a
> column-vector based data structure.
> Similar issue issue is reported in spark (Ref:
> https://issues.apache.org/jira/browse/SPARK-16334)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22671) TestHiveCli is flaky
[
https://issues.apache.org/jira/browse/HIVE-22671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003685#comment-17003685
]
Hive QA commented on HIVE-22671:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} beeline in master has 48 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} beeline: The patch generated 1 new + 26 unchanged - 1
fixed = 27 total (was 27) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20022/dev-support/hive-personality.sh
|
| git revision | master / 4dcbfb3 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20022/yetus/diff-checkstyle-beeline.txt
|
| modules | C: beeline U: beeline |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20022/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TestHiveCli is flaky
>
>
> Key: HIVE-22671
> URL: https://issues.apache.org/jira/browse/HIVE-22671
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-22671.01.patch
>
>
> Fails while running scripts @Before, the actual problem is something like
> this:
> {code}
> 2019-12-24T06:11:16,065 DEBUG [main] exec.Utilities: HDFS dir: /tmp/hive with
> schema null, permission: rwxr-xr-x
> 2019-12-24T06:11:16,065 ERROR [main] cli.TestHiveCli: Failed due to the
> error:Connecting to jdbc:hive2://
> Hive Session ID = 92e1327d-4267-4efc-af7d-6bd2deeba7d7
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/hiveptest/34.70.195.43-hiveptest-0/apache-github-source-source/conf/ivysettings.xml
> will be used
> Error applying authorization policy on hive configuration: The dir: /tmp/hive
> on HDFS should be writable. Current permissions are: rwxr-xr-x
> Hive Session ID = 5d752eea-9bb5-42b4-953d-750f766fefaa
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/hiveptest/34.70.195.43-hiveptest-0/apache-github-source-source/conf/ivysettings.xml
> will be used
> Error applying authorization policy on hive configuration: The dir: /tmp/hive
> on HDFS should be writable. Current permissions are: rwxr-xr-x
> Connection is already closed.
> {code}
> locally it passes:
> {code}
> 2019-12-26T06:23:45,535 DEBUG [main]
[jira] [Commented] (HIVE-17133) NoSuchMethodError in Hadoop FileStatus.compareTo
[
https://issues.apache.org/jira/browse/HIVE-17133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003680#comment-17003680
]
Hive QA commented on HIVE-17133:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12989496/HIVE-17133.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 17786 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.beeline.cli.TestHiveCli.testCmd (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testCommentStripping (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testDatabaseOptions (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testErrOutput (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testHelp (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testInValidCmd (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testInvalidDatabaseOptions (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testInvalidOptions (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testInvalidOptions2 (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testSetHeaderValue (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testSetPromptValue (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testSourceCmd (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testSourceCmd2 (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testSourceCmd3 (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testSourceCmd4 (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testSqlFromCmd (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testSqlFromCmdWithDBName (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testUseCurrentDB1 (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testUseCurrentDB2 (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testUseCurrentDB3 (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testUseInvalidDB (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testVariables (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testVariablesForSource (batchId=206)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20021/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20021/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20021/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 24 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12989496 - PreCommit-HIVE-Build
> NoSuchMethodError in Hadoop FileStatus.compareTo
>
>
> Key: HIVE-17133
> URL: https://issues.apache.org/jira/browse/HIVE-17133
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Major
> Attachments: HIVE-17133.1.patch
>
>
> The stack trace is:
> {noformat}
> Caused by: java.lang.NoSuchMethodError:
> org.apache.hadoop.fs.FileStatus.compareTo(Lorg/apache/hadoop/fs/FileStatus;)I
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.lambda$getAcidState$0(AcidUtils.java:931)
> at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
> at java.util.TimSort.sort(TimSort.java:234)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1454)
> at java.util.Collections.sort(Collections.java:175)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getAcidState(AcidUtils.java:929)
> {noformat}
> I'm on Hive master and using Hadoop 2.7.2. The method signature in Hadoop
> 2.7.2 is:
> https://github.com/apache/hadoop/blob/release-2.7.2-RC2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileStatus.java#L336
> In Hadoop 2.8.0 it becomes:
> https://github.com/apache/hadoop/blob/release-2.8.0-RC3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileStatus.java#L332
> I think that breaks binary compatibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22671) TestHiveCli is flaky
[
https://issues.apache.org/jira/browse/HIVE-22671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003675#comment-17003675
]
Denys Kuzmenko commented on HIVE-22671:
---
There is already fix for that: https://issues.apache.org/jira/browse/HIVE-22649
> TestHiveCli is flaky
>
>
> Key: HIVE-22671
> URL: https://issues.apache.org/jira/browse/HIVE-22671
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-22671.01.patch
>
>
> Fails while running scripts @Before, the actual problem is something like
> this:
> {code}
> 2019-12-24T06:11:16,065 DEBUG [main] exec.Utilities: HDFS dir: /tmp/hive with
> schema null, permission: rwxr-xr-x
> 2019-12-24T06:11:16,065 ERROR [main] cli.TestHiveCli: Failed due to the
> error:Connecting to jdbc:hive2://
> Hive Session ID = 92e1327d-4267-4efc-af7d-6bd2deeba7d7
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/hiveptest/34.70.195.43-hiveptest-0/apache-github-source-source/conf/ivysettings.xml
> will be used
> Error applying authorization policy on hive configuration: The dir: /tmp/hive
> on HDFS should be writable. Current permissions are: rwxr-xr-x
> Hive Session ID = 5d752eea-9bb5-42b4-953d-750f766fefaa
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/hiveptest/34.70.195.43-hiveptest-0/apache-github-source-source/conf/ivysettings.xml
> will be used
> Error applying authorization policy on hive configuration: The dir: /tmp/hive
> on HDFS should be writable. Current permissions are: rwxr-xr-x
> Connection is already closed.
> {code}
> locally it passes:
> {code}
> 2019-12-26T06:23:45,535 DEBUG [main] cli.CLIService: SessionHandle
> [ee1f45b2-cb74-4b0e-9c4e-5ddbc47a94ae]: closeSession()
> 2019-12-26T06:23:45,540 DEBUG [main] cli.TestHiveCli: Connecting to
> jdbc:hive2://
> Hive Session ID = feba5d41-796f-406e-a017-bce19c28fe09
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/abstractdog/apache/hive/conf/ivysettings.xml will be used
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/abstractdog/apache/hive/conf/ivysettings.xml will be used
> Connected to: Apache Hive (version 4.0.0-SNAPSHOT)
> Driver: Hive JDBC (version 4.0.0-SNAPSHOT)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> hive> create database if not exists test;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22671) TestHiveCli is flaky
[
https://issues.apache.org/jira/browse/HIVE-22671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-22671:
Assignee: László Bodor
Status: Patch Available (was: Open)
> TestHiveCli is flaky
>
>
> Key: HIVE-22671
> URL: https://issues.apache.org/jira/browse/HIVE-22671
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-22671.01.patch
>
>
> Fails while running scripts @Before, the actual problem is something like
> this:
> {code}
> 2019-12-24T06:11:16,065 DEBUG [main] exec.Utilities: HDFS dir: /tmp/hive with
> schema null, permission: rwxr-xr-x
> 2019-12-24T06:11:16,065 ERROR [main] cli.TestHiveCli: Failed due to the
> error:Connecting to jdbc:hive2://
> Hive Session ID = 92e1327d-4267-4efc-af7d-6bd2deeba7d7
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/hiveptest/34.70.195.43-hiveptest-0/apache-github-source-source/conf/ivysettings.xml
> will be used
> Error applying authorization policy on hive configuration: The dir: /tmp/hive
> on HDFS should be writable. Current permissions are: rwxr-xr-x
> Hive Session ID = 5d752eea-9bb5-42b4-953d-750f766fefaa
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/hiveptest/34.70.195.43-hiveptest-0/apache-github-source-source/conf/ivysettings.xml
> will be used
> Error applying authorization policy on hive configuration: The dir: /tmp/hive
> on HDFS should be writable. Current permissions are: rwxr-xr-x
> Connection is already closed.
> {code}
> locally it passes:
> {code}
> 2019-12-26T06:23:45,535 DEBUG [main] cli.CLIService: SessionHandle
> [ee1f45b2-cb74-4b0e-9c4e-5ddbc47a94ae]: closeSession()
> 2019-12-26T06:23:45,540 DEBUG [main] cli.TestHiveCli: Connecting to
> jdbc:hive2://
> Hive Session ID = feba5d41-796f-406e-a017-bce19c28fe09
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/abstractdog/apache/hive/conf/ivysettings.xml will be used
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/abstractdog/apache/hive/conf/ivysettings.xml will be used
> Connected to: Apache Hive (version 4.0.0-SNAPSHOT)
> Driver: Hive JDBC (version 4.0.0-SNAPSHOT)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> hive> create database if not exists test;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22671) TestHiveCli is flaky
[
https://issues.apache.org/jira/browse/HIVE-22671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-22671:
Attachment: HIVE-22671.01.patch
> TestHiveCli is flaky
>
>
> Key: HIVE-22671
> URL: https://issues.apache.org/jira/browse/HIVE-22671
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Priority: Major
> Attachments: HIVE-22671.01.patch
>
>
> Fails while running scripts @Before, the actual problem is something like
> this:
> {code}
> 2019-12-24T06:11:16,065 DEBUG [main] exec.Utilities: HDFS dir: /tmp/hive with
> schema null, permission: rwxr-xr-x
> 2019-12-24T06:11:16,065 ERROR [main] cli.TestHiveCli: Failed due to the
> error:Connecting to jdbc:hive2://
> Hive Session ID = 92e1327d-4267-4efc-af7d-6bd2deeba7d7
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/hiveptest/34.70.195.43-hiveptest-0/apache-github-source-source/conf/ivysettings.xml
> will be used
> Error applying authorization policy on hive configuration: The dir: /tmp/hive
> on HDFS should be writable. Current permissions are: rwxr-xr-x
> Hive Session ID = 5d752eea-9bb5-42b4-953d-750f766fefaa
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/hiveptest/34.70.195.43-hiveptest-0/apache-github-source-source/conf/ivysettings.xml
> will be used
> Error applying authorization policy on hive configuration: The dir: /tmp/hive
> on HDFS should be writable. Current permissions are: rwxr-xr-x
> Connection is already closed.
> {code}
> locally it passes:
> {code}
> 2019-12-26T06:23:45,535 DEBUG [main] cli.CLIService: SessionHandle
> [ee1f45b2-cb74-4b0e-9c4e-5ddbc47a94ae]: closeSession()
> 2019-12-26T06:23:45,540 DEBUG [main] cli.TestHiveCli: Connecting to
> jdbc:hive2://
> Hive Session ID = feba5d41-796f-406e-a017-bce19c28fe09
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/abstractdog/apache/hive/conf/ivysettings.xml will be used
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/abstractdog/apache/hive/conf/ivysettings.xml will be used
> Connected to: Apache Hive (version 4.0.0-SNAPSHOT)
> Driver: Hive JDBC (version 4.0.0-SNAPSHOT)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> hive> create database if not exists test;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22671) TestHiveCli is flaky
[
https://issues.apache.org/jira/browse/HIVE-22671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003665#comment-17003665
]
László Bodor edited comment on HIVE-22671 at 12/26/19 2:50 PM:
---
seems like an environment problem in ptest runs, but I'll try to find the first
failing precommit run (however, due to green run policy, this issue should not
have been caused by a commit)
UPDATE: seems like it's not persistent, but very flaky, most of the recent runs
fail due to this
was (Author: abstractdog):
seems like an environment problem in ptest runs, but I'll try to find the first
failing precommit run (however, due to green run policy, this issue should not
have been caused by a commit)
> TestHiveCli is flaky
>
>
> Key: HIVE-22671
> URL: https://issues.apache.org/jira/browse/HIVE-22671
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Priority: Major
>
> Fails while running scripts @Before, the actual problem is something like
> this:
> {code}
> 2019-12-24T06:11:16,065 DEBUG [main] exec.Utilities: HDFS dir: /tmp/hive with
> schema null, permission: rwxr-xr-x
> 2019-12-24T06:11:16,065 ERROR [main] cli.TestHiveCli: Failed due to the
> error:Connecting to jdbc:hive2://
> Hive Session ID = 92e1327d-4267-4efc-af7d-6bd2deeba7d7
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/hiveptest/34.70.195.43-hiveptest-0/apache-github-source-source/conf/ivysettings.xml
> will be used
> Error applying authorization policy on hive configuration: The dir: /tmp/hive
> on HDFS should be writable. Current permissions are: rwxr-xr-x
> Hive Session ID = 5d752eea-9bb5-42b4-953d-750f766fefaa
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/hiveptest/34.70.195.43-hiveptest-0/apache-github-source-source/conf/ivysettings.xml
> will be used
> Error applying authorization policy on hive configuration: The dir: /tmp/hive
> on HDFS should be writable. Current permissions are: rwxr-xr-x
> Connection is already closed.
> {code}
> locally it passes:
> {code}
> 2019-12-26T06:23:45,535 DEBUG [main] cli.CLIService: SessionHandle
> [ee1f45b2-cb74-4b0e-9c4e-5ddbc47a94ae]: closeSession()
> 2019-12-26T06:23:45,540 DEBUG [main] cli.TestHiveCli: Connecting to
> jdbc:hive2://
> Hive Session ID = feba5d41-796f-406e-a017-bce19c28fe09
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/abstractdog/apache/hive/conf/ivysettings.xml will be used
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/abstractdog/apache/hive/conf/ivysettings.xml will be used
> Connected to: Apache Hive (version 4.0.0-SNAPSHOT)
> Driver: Hive JDBC (version 4.0.0-SNAPSHOT)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> hive> create database if not exists test;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22671) TestHiveCli is flaky
[
https://issues.apache.org/jira/browse/HIVE-22671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-22671:
Summary: TestHiveCli is flaky (was: TestHiveCli tests fail)
> TestHiveCli is flaky
>
>
> Key: HIVE-22671
> URL: https://issues.apache.org/jira/browse/HIVE-22671
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Priority: Major
>
> Fails while running scripts @Before, the actual problem is something like
> this:
> {code}
> 2019-12-24T06:11:16,065 DEBUG [main] exec.Utilities: HDFS dir: /tmp/hive with
> schema null, permission: rwxr-xr-x
> 2019-12-24T06:11:16,065 ERROR [main] cli.TestHiveCli: Failed due to the
> error:Connecting to jdbc:hive2://
> Hive Session ID = 92e1327d-4267-4efc-af7d-6bd2deeba7d7
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/hiveptest/34.70.195.43-hiveptest-0/apache-github-source-source/conf/ivysettings.xml
> will be used
> Error applying authorization policy on hive configuration: The dir: /tmp/hive
> on HDFS should be writable. Current permissions are: rwxr-xr-x
> Hive Session ID = 5d752eea-9bb5-42b4-953d-750f766fefaa
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/hiveptest/34.70.195.43-hiveptest-0/apache-github-source-source/conf/ivysettings.xml
> will be used
> Error applying authorization policy on hive configuration: The dir: /tmp/hive
> on HDFS should be writable. Current permissions are: rwxr-xr-x
> Connection is already closed.
> {code}
> locally it passes:
> {code}
> 2019-12-26T06:23:45,535 DEBUG [main] cli.CLIService: SessionHandle
> [ee1f45b2-cb74-4b0e-9c4e-5ddbc47a94ae]: closeSession()
> 2019-12-26T06:23:45,540 DEBUG [main] cli.TestHiveCli: Connecting to
> jdbc:hive2://
> Hive Session ID = feba5d41-796f-406e-a017-bce19c28fe09
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/abstractdog/apache/hive/conf/ivysettings.xml will be used
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/abstractdog/apache/hive/conf/ivysettings.xml will be used
> Connected to: Apache Hive (version 4.0.0-SNAPSHOT)
> Driver: Hive JDBC (version 4.0.0-SNAPSHOT)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> hive> create database if not exists test;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22645) Jline can break bash terminal behavior
[ https://issues.apache.org/jira/browse/HIVE-22645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003666#comment-17003666 ] László Bodor commented on HIVE-22645: - TestHiveCli failures are not related, found in another runs, created jira: HIVE-22671 > Jline can break bash terminal behavior > -- > > Key: HIVE-22645 > URL: https://issues.apache.org/jira/browse/HIVE-22645 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22645.01.patch, HIVE-22645.01.patch, > HIVE-22645.02.patch > > > After HIVE-21954, running a qtest can break bash terminal (but not zsh), in a > way that, e.g. enter key doesn't break line... > As identified by [~kgyrtkirk], this line could cause this: > https://github.com/apache/hive/commit/d645d827d95de36175194407bd1e2f6725362aff#diff-c64cf2b501f04e6710bbc3fcd079156fR84 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22671) TestHiveCli tests fail
[
https://issues.apache.org/jira/browse/HIVE-22671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003665#comment-17003665
]
László Bodor commented on HIVE-22671:
-
seems like an environment problem in ptest runs, but I'll try to find the first
failing precommit run (however, due to green run policy, this issue should not
have been caused by a commit)
> TestHiveCli tests fail
> --
>
> Key: HIVE-22671
> URL: https://issues.apache.org/jira/browse/HIVE-22671
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Priority: Major
>
> Fails while running scripts @Before, the actual problem is something like
> this:
> {code}
> 2019-12-24T06:11:16,065 DEBUG [main] exec.Utilities: HDFS dir: /tmp/hive with
> schema null, permission: rwxr-xr-x
> 2019-12-24T06:11:16,065 ERROR [main] cli.TestHiveCli: Failed due to the
> error:Connecting to jdbc:hive2://
> Hive Session ID = 92e1327d-4267-4efc-af7d-6bd2deeba7d7
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/hiveptest/34.70.195.43-hiveptest-0/apache-github-source-source/conf/ivysettings.xml
> will be used
> Error applying authorization policy on hive configuration: The dir: /tmp/hive
> on HDFS should be writable. Current permissions are: rwxr-xr-x
> Hive Session ID = 5d752eea-9bb5-42b4-953d-750f766fefaa
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/hiveptest/34.70.195.43-hiveptest-0/apache-github-source-source/conf/ivysettings.xml
> will be used
> Error applying authorization policy on hive configuration: The dir: /tmp/hive
> on HDFS should be writable. Current permissions are: rwxr-xr-x
> Connection is already closed.
> {code}
> locally it passes:
> {code}
> 2019-12-26T06:23:45,535 DEBUG [main] cli.CLIService: SessionHandle
> [ee1f45b2-cb74-4b0e-9c4e-5ddbc47a94ae]: closeSession()
> 2019-12-26T06:23:45,540 DEBUG [main] cli.TestHiveCli: Connecting to
> jdbc:hive2://
> Hive Session ID = feba5d41-796f-406e-a017-bce19c28fe09
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/abstractdog/apache/hive/conf/ivysettings.xml will be used
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/abstractdog/apache/hive/conf/ivysettings.xml will be used
> Connected to: Apache Hive (version 4.0.0-SNAPSHOT)
> Driver: Hive JDBC (version 4.0.0-SNAPSHOT)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> hive> create database if not exists test;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22671) TestHiveCli tests fail
[
https://issues.apache.org/jira/browse/HIVE-22671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-22671:
Description:
Fails while running scripts @Before, the actual problem is something like this:
{code}
2019-12-24T06:11:16,065 DEBUG [main] exec.Utilities: HDFS dir: /tmp/hive with
schema null, permission: rwxr-xr-x
2019-12-24T06:11:16,065 ERROR [main] cli.TestHiveCli: Failed due to the
error:Connecting to jdbc:hive2://
Hive Session ID = 92e1327d-4267-4efc-af7d-6bd2deeba7d7
ivysettings.xml file not found in HIVE_HOME or
HIVE_CONF_DIR,/home/hiveptest/34.70.195.43-hiveptest-0/apache-github-source-source/conf/ivysettings.xml
will be used
Error applying authorization policy on hive configuration: The dir: /tmp/hive
on HDFS should be writable. Current permissions are: rwxr-xr-x
Hive Session ID = 5d752eea-9bb5-42b4-953d-750f766fefaa
ivysettings.xml file not found in HIVE_HOME or
HIVE_CONF_DIR,/home/hiveptest/34.70.195.43-hiveptest-0/apache-github-source-source/conf/ivysettings.xml
will be used
Error applying authorization policy on hive configuration: The dir: /tmp/hive
on HDFS should be writable. Current permissions are: rwxr-xr-x
Connection is already closed.
{code}
locally it passes:
{code}
2019-12-26T06:23:45,535 DEBUG [main] cli.CLIService: SessionHandle
[ee1f45b2-cb74-4b0e-9c4e-5ddbc47a94ae]: closeSession()
2019-12-26T06:23:45,540 DEBUG [main] cli.TestHiveCli: Connecting to
jdbc:hive2://
Hive Session ID = feba5d41-796f-406e-a017-bce19c28fe09
ivysettings.xml file not found in HIVE_HOME or
HIVE_CONF_DIR,/home/abstractdog/apache/hive/conf/ivysettings.xml will be used
ivysettings.xml file not found in HIVE_HOME or
HIVE_CONF_DIR,/home/abstractdog/apache/hive/conf/ivysettings.xml will be used
Connected to: Apache Hive (version 4.0.0-SNAPSHOT)
Driver: Hive JDBC (version 4.0.0-SNAPSHOT)
Transaction isolation: TRANSACTION_REPEATABLE_READ
hive> create database if not exists test;
{code}
> TestHiveCli tests fail
> --
>
> Key: HIVE-22671
> URL: https://issues.apache.org/jira/browse/HIVE-22671
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Priority: Major
>
> Fails while running scripts @Before, the actual problem is something like
> this:
> {code}
> 2019-12-24T06:11:16,065 DEBUG [main] exec.Utilities: HDFS dir: /tmp/hive with
> schema null, permission: rwxr-xr-x
> 2019-12-24T06:11:16,065 ERROR [main] cli.TestHiveCli: Failed due to the
> error:Connecting to jdbc:hive2://
> Hive Session ID = 92e1327d-4267-4efc-af7d-6bd2deeba7d7
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/hiveptest/34.70.195.43-hiveptest-0/apache-github-source-source/conf/ivysettings.xml
> will be used
> Error applying authorization policy on hive configuration: The dir: /tmp/hive
> on HDFS should be writable. Current permissions are: rwxr-xr-x
> Hive Session ID = 5d752eea-9bb5-42b4-953d-750f766fefaa
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/hiveptest/34.70.195.43-hiveptest-0/apache-github-source-source/conf/ivysettings.xml
> will be used
> Error applying authorization policy on hive configuration: The dir: /tmp/hive
> on HDFS should be writable. Current permissions are: rwxr-xr-x
> Connection is already closed.
> {code}
> locally it passes:
> {code}
> 2019-12-26T06:23:45,535 DEBUG [main] cli.CLIService: SessionHandle
> [ee1f45b2-cb74-4b0e-9c4e-5ddbc47a94ae]: closeSession()
> 2019-12-26T06:23:45,540 DEBUG [main] cli.TestHiveCli: Connecting to
> jdbc:hive2://
> Hive Session ID = feba5d41-796f-406e-a017-bce19c28fe09
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/abstractdog/apache/hive/conf/ivysettings.xml will be used
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,/home/abstractdog/apache/hive/conf/ivysettings.xml will be used
> Connected to: Apache Hive (version 4.0.0-SNAPSHOT)
> Driver: Hive JDBC (version 4.0.0-SNAPSHOT)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> hive> create database if not exists test;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-17133) NoSuchMethodError in Hadoop FileStatus.compareTo
[
https://issues.apache.org/jira/browse/HIVE-17133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003662#comment-17003662
]
Hive QA commented on HIVE-17133:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
2s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20021/dev-support/hive-personality.sh
|
| git revision | master / 4dcbfb3 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20021/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> NoSuchMethodError in Hadoop FileStatus.compareTo
>
>
> Key: HIVE-17133
> URL: https://issues.apache.org/jira/browse/HIVE-17133
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Major
> Attachments: HIVE-17133.1.patch
>
>
> The stack trace is:
> {noformat}
> Caused by: java.lang.NoSuchMethodError:
> org.apache.hadoop.fs.FileStatus.compareTo(Lorg/apache/hadoop/fs/FileStatus;)I
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.lambda$getAcidState$0(AcidUtils.java:931)
> at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
> at java.util.TimSort.sort(TimSort.java:234)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1454)
> at java.util.Collections.sort(Collections.java:175)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getAcidState(AcidUtils.java:929)
> {noformat}
> I'm on Hive master and using Hadoop 2.7.2. The method signature in Hadoop
> 2.7.2 is:
> https://github.com/apache/hadoop/blob/release-2.7.2-RC2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileStatus.java#L336
> In Hadoop 2.8.0 it becomes:
> https://github.com/apache/hadoop/blob/release-2.8.0-RC3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileStatus.java#L332
> I think that breaks binary compatibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22670) ArrayIndexOutOfBoundsException when vectorized reader is used for reading a parquet file
[
https://issues.apache.org/jira/browse/HIVE-22670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ganesha Shreedhara updated HIVE-22670:
--
Component/s: Vectorization
Parquet
> ArrayIndexOutOfBoundsException when vectorized reader is used for reading a
> parquet file
>
>
> Key: HIVE-22670
> URL: https://issues.apache.org/jira/browse/HIVE-22670
> Project: Hive
> Issue Type: Bug
> Components: Parquet, Vectorization
>Affects Versions: 3.1.2, 2.3.6
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-22670.1.patch
>
>
> ArrayIndexOutOfBoundsException is getting thrown while decoding dictionaryIds
> of a row group in parquet file with vectorization enabled.
> *Exception stack trace:*
> {code:java}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainBinaryDictionary.decodeToBinary(PlainValuesDictionary.java:122)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.ParquetDataColumnReaderFactory$DefaultParquetDataColumnReader.readString(ParquetDataColumnReaderFactory.java:95)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedPrimitiveColumnReader.decodeDictionaryIds(VectorizedPrimitiveColumnReader.java:467)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedPrimitiveColumnReader.readBatch(VectorizedPrimitiveColumnReader.java:68)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:410)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
> ... 24 more{code}
>
> This issue seems to be caused by re-using the same dictionary column vector
> while reading consecutive row groups. This looks like one of the corner case
> bug which occurs for a certain distribution of dictionary/plain encoded data
> while we read/populate the underlying bit packed dictionary data into a
> column-vector based data structure.
> Similar issue issue is reported in spark (Ref:
> https://issues.apache.org/jira/browse/SPARK-16334)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-17133) NoSuchMethodError in Hadoop FileStatus.compareTo
[
https://issues.apache.org/jira/browse/HIVE-17133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003635#comment-17003635
]
Rui Li commented on HIVE-17133:
---
While HADOOP-14683 has been fixed, it seems we still have to fix on our side.
[~sershe] [~xuefuz] Would you mind have a look at the patch?
> NoSuchMethodError in Hadoop FileStatus.compareTo
>
>
> Key: HIVE-17133
> URL: https://issues.apache.org/jira/browse/HIVE-17133
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Major
> Attachments: HIVE-17133.1.patch
>
>
> The stack trace is:
> {noformat}
> Caused by: java.lang.NoSuchMethodError:
> org.apache.hadoop.fs.FileStatus.compareTo(Lorg/apache/hadoop/fs/FileStatus;)I
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.lambda$getAcidState$0(AcidUtils.java:931)
> at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
> at java.util.TimSort.sort(TimSort.java:234)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1454)
> at java.util.Collections.sort(Collections.java:175)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getAcidState(AcidUtils.java:929)
> {noformat}
> I'm on Hive master and using Hadoop 2.7.2. The method signature in Hadoop
> 2.7.2 is:
> https://github.com/apache/hadoop/blob/release-2.7.2-RC2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileStatus.java#L336
> In Hadoop 2.8.0 it becomes:
> https://github.com/apache/hadoop/blob/release-2.8.0-RC3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileStatus.java#L332
> I think that breaks binary compatibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-17133) NoSuchMethodError in Hadoop FileStatus.compareTo
[
https://issues.apache.org/jira/browse/HIVE-17133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-17133:
--
Status: Patch Available (was: Open)
> NoSuchMethodError in Hadoop FileStatus.compareTo
>
>
> Key: HIVE-17133
> URL: https://issues.apache.org/jira/browse/HIVE-17133
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Major
> Attachments: HIVE-17133.1.patch
>
>
> The stack trace is:
> {noformat}
> Caused by: java.lang.NoSuchMethodError:
> org.apache.hadoop.fs.FileStatus.compareTo(Lorg/apache/hadoop/fs/FileStatus;)I
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.lambda$getAcidState$0(AcidUtils.java:931)
> at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
> at java.util.TimSort.sort(TimSort.java:234)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1454)
> at java.util.Collections.sort(Collections.java:175)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getAcidState(AcidUtils.java:929)
> {noformat}
> I'm on Hive master and using Hadoop 2.7.2. The method signature in Hadoop
> 2.7.2 is:
> https://github.com/apache/hadoop/blob/release-2.7.2-RC2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileStatus.java#L336
> In Hadoop 2.8.0 it becomes:
> https://github.com/apache/hadoop/blob/release-2.8.0-RC3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileStatus.java#L332
> I think that breaks binary compatibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-17133) NoSuchMethodError in Hadoop FileStatus.compareTo
[
https://issues.apache.org/jira/browse/HIVE-17133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li reassigned HIVE-17133:
-
Assignee: Rui Li
> NoSuchMethodError in Hadoop FileStatus.compareTo
>
>
> Key: HIVE-17133
> URL: https://issues.apache.org/jira/browse/HIVE-17133
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Major
> Attachments: HIVE-17133.1.patch
>
>
> The stack trace is:
> {noformat}
> Caused by: java.lang.NoSuchMethodError:
> org.apache.hadoop.fs.FileStatus.compareTo(Lorg/apache/hadoop/fs/FileStatus;)I
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.lambda$getAcidState$0(AcidUtils.java:931)
> at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
> at java.util.TimSort.sort(TimSort.java:234)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1454)
> at java.util.Collections.sort(Collections.java:175)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getAcidState(AcidUtils.java:929)
> {noformat}
> I'm on Hive master and using Hadoop 2.7.2. The method signature in Hadoop
> 2.7.2 is:
> https://github.com/apache/hadoop/blob/release-2.7.2-RC2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileStatus.java#L336
> In Hadoop 2.8.0 it becomes:
> https://github.com/apache/hadoop/blob/release-2.8.0-RC3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileStatus.java#L332
> I think that breaks binary compatibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-17133) NoSuchMethodError in Hadoop FileStatus.compareTo
[
https://issues.apache.org/jira/browse/HIVE-17133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-17133:
--
Attachment: HIVE-17133.1.patch
> NoSuchMethodError in Hadoop FileStatus.compareTo
>
>
> Key: HIVE-17133
> URL: https://issues.apache.org/jira/browse/HIVE-17133
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Major
> Attachments: HIVE-17133.1.patch
>
>
> The stack trace is:
> {noformat}
> Caused by: java.lang.NoSuchMethodError:
> org.apache.hadoop.fs.FileStatus.compareTo(Lorg/apache/hadoop/fs/FileStatus;)I
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.lambda$getAcidState$0(AcidUtils.java:931)
> at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
> at java.util.TimSort.sort(TimSort.java:234)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1454)
> at java.util.Collections.sort(Collections.java:175)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getAcidState(AcidUtils.java:929)
> {noformat}
> I'm on Hive master and using Hadoop 2.7.2. The method signature in Hadoop
> 2.7.2 is:
> https://github.com/apache/hadoop/blob/release-2.7.2-RC2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileStatus.java#L336
> In Hadoop 2.8.0 it becomes:
> https://github.com/apache/hadoop/blob/release-2.8.0-RC3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileStatus.java#L332
> I think that breaks binary compatibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22670) ArrayIndexOutOfBoundsException when vectorized reader is used for reading a parquet file
[
https://issues.apache.org/jira/browse/HIVE-22670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003590#comment-17003590
]
Hive QA commented on HIVE-22670:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12989488/HIVE-22670.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 17786 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.beeline.cli.TestHiveCli.testCmd (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testCommentStripping (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testDatabaseOptions (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testErrOutput (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testHelp (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testInValidCmd (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testInvalidDatabaseOptions (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testInvalidOptions (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testInvalidOptions2 (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testSetHeaderValue (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testSetPromptValue (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testSourceCmd (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testSourceCmd2 (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testSourceCmd3 (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testSourceCmd4 (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testSqlFromCmd (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testSqlFromCmdWithDBName (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testUseCurrentDB1 (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testUseCurrentDB2 (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testUseCurrentDB3 (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testUseInvalidDB (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testVariables (batchId=206)
org.apache.hive.beeline.cli.TestHiveCli.testVariablesForSource (batchId=206)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20020/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20020/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20020/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 24 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12989488 - PreCommit-HIVE-Build
> ArrayIndexOutOfBoundsException when vectorized reader is used for reading a
> parquet file
>
>
> Key: HIVE-22670
> URL: https://issues.apache.org/jira/browse/HIVE-22670
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.2, 2.3.6
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-22670.1.patch
>
>
> ArrayIndexOutOfBoundsException is getting thrown while decoding dictionaryIds
> of a row group in parquet file with vectorization enabled.
> *Exception stack trace:*
> {code:java}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainBinaryDictionary.decodeToBinary(PlainValuesDictionary.java:122)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.ParquetDataColumnReaderFactory$DefaultParquetDataColumnReader.readString(ParquetDataColumnReaderFactory.java:95)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedPrimitiveColumnReader.decodeDictionaryIds(VectorizedPrimitiveColumnReader.java:467)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedPrimitiveColumnReader.readBatch(VectorizedPrimitiveColumnReader.java:68)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:410)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
> ... 24 more{code}
>
> This issue seems to be caused by re-using the same dictionary column vector
> while reading consecutive row groups. This looks like one of
[jira] [Commented] (HIVE-22670) ArrayIndexOutOfBoundsException when vectorized reader is used for reading a parquet file
[
https://issues.apache.org/jira/browse/HIVE-22670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003567#comment-17003567
]
Hive QA commented on HIVE-22670:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
4s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 1 new + 2 unchanged - 0 fixed
= 3 total (was 2) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20020/dev-support/hive-personality.sh
|
| git revision | master / 4dcbfb3 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20020/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20020/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ArrayIndexOutOfBoundsException when vectorized reader is used for reading a
> parquet file
>
>
> Key: HIVE-22670
> URL: https://issues.apache.org/jira/browse/HIVE-22670
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.2, 2.3.6
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-22670.1.patch
>
>
> ArrayIndexOutOfBoundsException is getting thrown while decoding dictionaryIds
> of a row group in parquet file with vectorization enabled.
> *Exception stack trace:*
> {code:java}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainBinaryDictionary.decodeToBinary(PlainValuesDictionary.java:122)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.ParquetDataColumnReaderFactory$DefaultParquetDataColumnReader.readString(ParquetDataColumnReaderFactory.java:95)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedPrimitiveColumnReader.decodeDictionaryIds(VectorizedPrimitiveColumnReader.java:467)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedPrimitiveColumnReader.readBatch(VectorizedPrimitiveColumnReader.java:68)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:410)
> at
>
[jira] [Commented] (HIVE-21213) Acid table bootstrap replication needs to handle directory created by compaction with txn id
[
https://issues.apache.org/jira/browse/HIVE-21213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003555#comment-17003555
]
Hive QA commented on HIVE-21213:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12989486/HIVE-21213.04.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20019/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20019/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20019/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-12-26 09:54:08.890
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-20019/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-12-26 09:54:08.893
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 4dcbfb3 HIVE-22657 Add log message when stats have to to
computed during calcite (Miklos Gergely)
+ git clean -f -d
Removing ${project.basedir}/
Removing itests/${project.basedir}/
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 4dcbfb3 HIVE-22657 Add log message when stats have to to
computed during calcite (Miklos Gergely)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-12-26 09:54:09.607
+ rm -rf ../yetus_PreCommit-HIVE-Build-20019
+ mkdir ../yetus_PreCommit-HIVE-Build-20019
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-20019
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-20019/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
error:
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/BaseReplicationScenariosAcidTables.java:
does not exist in index
error:
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java: does
not exist in index
error: a/ql/src/test/org/apache/hadoop/hive/ql/io/TestAcidUtils.java: does not
exist in index
error:
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
does not exist in index
Trying to apply the patch with -p1
Going to apply patch with: git apply -p1
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc7472711545627594298.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc7472711545627594298.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
protoc-jar: executing: [/tmp/protoc5921096948735462635.exe, --version]
libprotoc 2.5.0
ANTLR Parser Generator Version 3.5.2
Output file
[jira] [Updated] (HIVE-22670) ArrayIndexOutOfBoundsException when vectorized reader is used for reading a parquet file
[
https://issues.apache.org/jira/browse/HIVE-22670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ganesha Shreedhara updated HIVE-22670:
--
Description:
ArrayIndexOutOfBoundsException is getting thrown while decoding dictionaryIds
of a row group in parquet file with vectorization enabled.
*Exception stack trace:*
{code:java}
Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
at
org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainBinaryDictionary.decodeToBinary(PlainValuesDictionary.java:122)
at
org.apache.hadoop.hive.ql.io.parquet.vector.ParquetDataColumnReaderFactory$DefaultParquetDataColumnReader.readString(ParquetDataColumnReaderFactory.java:95)
at
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedPrimitiveColumnReader.decodeDictionaryIds(VectorizedPrimitiveColumnReader.java:467)
at
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedPrimitiveColumnReader.readBatch(VectorizedPrimitiveColumnReader.java:68)
at
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:410)
at
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
at
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
... 24 more{code}
This issue seems to be caused by re-using the same dictionary column vector
while reading consecutive row groups. This looks like one of the corner case
bug which occurs for a certain distribution of dictionary/plain encoded data
while we read/populate the underlying bit packed dictionary data into a
column-vector based data structure.
Similar issue issue is reported in spark (Ref:
https://issues.apache.org/jira/browse/SPARK-16334)
was:
ArrayIndexOutOfBoundsException is getting thrown while decoding dictionaryIds
of a row group in parquet file with vectorization enabled.
*Exception stack trace:*
{code:java}
Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
at
org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainBinaryDictionary.decodeToBinary(PlainValuesDictionary.java:122)
at
org.apache.hadoop.hive.ql.io.parquet.vector.ParquetDataColumnReaderFactory$DefaultParquetDataColumnReader.readString(ParquetDataColumnReaderFactory.java:95)
at
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedPrimitiveColumnReader.decodeDictionaryIds(VectorizedPrimitiveColumnReader.java:467)
at
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedPrimitiveColumnReader.readBatch(VectorizedPrimitiveColumnReader.java:68)
at
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:410)
at
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
at
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
... 24 more{code}
This issue seems to be caused by re-using the same dictionary column vector
while reading consecutive row groups. This looks like one of the corner case
bug which occurs for a certain distribution of dictionary/plain encoded data
while we read/populate the underlying bit packed dictionary data into a
column-vector based data structure.
> ArrayIndexOutOfBoundsException when vectorized reader is used for reading a
> parquet file
>
>
> Key: HIVE-22670
> URL: https://issues.apache.org/jira/browse/HIVE-22670
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.2, 2.3.6
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-22670.1.patch
>
>
> ArrayIndexOutOfBoundsException is getting thrown while decoding dictionaryIds
> of a row group in parquet file with vectorization enabled.
> *Exception stack trace:*
> {code:java}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainBinaryDictionary.decodeToBinary(PlainValuesDictionary.java:122)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.ParquetDataColumnReaderFactory$DefaultParquetDataColumnReader.readString(ParquetDataColumnReaderFactory.java:95)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedPrimitiveColumnReader.decodeDictionaryIds(VectorizedPrimitiveColumnReader.java:467)
> at
>
[jira] [Updated] (HIVE-22670) ArrayIndexOutOfBoundsException when vectorized reader is used for reading a parquet file
[
https://issues.apache.org/jira/browse/HIVE-22670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ganesha Shreedhara updated HIVE-22670:
--
Status: Patch Available (was: In Progress)
> ArrayIndexOutOfBoundsException when vectorized reader is used for reading a
> parquet file
>
>
> Key: HIVE-22670
> URL: https://issues.apache.org/jira/browse/HIVE-22670
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.6, 3.1.2
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-22670.1.patch
>
>
> ArrayIndexOutOfBoundsException is getting thrown while decoding dictionaryIds
> of a row group in parquet file with vectorization enabled.
> *Exception stack trace:*
> {code:java}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainBinaryDictionary.decodeToBinary(PlainValuesDictionary.java:122)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.ParquetDataColumnReaderFactory$DefaultParquetDataColumnReader.readString(ParquetDataColumnReaderFactory.java:95)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedPrimitiveColumnReader.decodeDictionaryIds(VectorizedPrimitiveColumnReader.java:467)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedPrimitiveColumnReader.readBatch(VectorizedPrimitiveColumnReader.java:68)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:410)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
> ... 24 more{code}
>
> This issue seems to be caused by re-using the same dictionary column vector
> while reading consecutive row groups. This looks like one of the corner case
> bug which occurs for a certain distribution of dictionary/plain encoded data
> while we read/populate the underlying bit packed dictionary data into a
> column-vector based data structure.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22670) ArrayIndexOutOfBoundsException when vectorized reader is used for reading a parquet file
[
https://issues.apache.org/jira/browse/HIVE-22670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ganesha Shreedhara updated HIVE-22670:
--
Attachment: HIVE-22670.1.patch
> ArrayIndexOutOfBoundsException when vectorized reader is used for reading a
> parquet file
>
>
> Key: HIVE-22670
> URL: https://issues.apache.org/jira/browse/HIVE-22670
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.2, 2.3.6
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-22670.1.patch
>
>
> ArrayIndexOutOfBoundsException is getting thrown while decoding dictionaryIds
> of a row group in parquet file with vectorization enabled.
> *Exception stack trace:*
> {code:java}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainBinaryDictionary.decodeToBinary(PlainValuesDictionary.java:122)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.ParquetDataColumnReaderFactory$DefaultParquetDataColumnReader.readString(ParquetDataColumnReaderFactory.java:95)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedPrimitiveColumnReader.decodeDictionaryIds(VectorizedPrimitiveColumnReader.java:467)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedPrimitiveColumnReader.readBatch(VectorizedPrimitiveColumnReader.java:68)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:410)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
> ... 24 more{code}
>
> This issue seems to be caused by re-using the same dictionary column vector
> while reading consecutive row groups. This looks like one of the corner case
> bug which occurs for a certain distribution of dictionary/plain encoded data
> while we read/populate the underlying bit packed dictionary data into a
> column-vector based data structure.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HIVE-22670) ArrayIndexOutOfBoundsException when vectorized reader is used for reading a parquet file
[
https://issues.apache.org/jira/browse/HIVE-22670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-22670 started by Ganesha Shreedhara.
-
> ArrayIndexOutOfBoundsException when vectorized reader is used for reading a
> parquet file
>
>
> Key: HIVE-22670
> URL: https://issues.apache.org/jira/browse/HIVE-22670
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.2, 2.3.6
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
>
> ArrayIndexOutOfBoundsException is getting thrown while decoding dictionaryIds
> of a row group in parquet file with vectorization enabled.
> *Exception stack trace:*
> {code:java}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainBinaryDictionary.decodeToBinary(PlainValuesDictionary.java:122)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.ParquetDataColumnReaderFactory$DefaultParquetDataColumnReader.readString(ParquetDataColumnReaderFactory.java:95)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedPrimitiveColumnReader.decodeDictionaryIds(VectorizedPrimitiveColumnReader.java:467)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedPrimitiveColumnReader.readBatch(VectorizedPrimitiveColumnReader.java:68)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:410)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
> ... 24 more{code}
>
> This issue seems to be caused by re-using the same dictionary column vector
> while reading consecutive row groups. This looks like one of the corner case
> bug which occurs for a certain distribution of dictionary/plain encoded data
> while we read/populate the underlying bit packed dictionary data into a
> column-vector based data structure.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22670) ArrayIndexOutOfBoundsException when vectorized reader is used for reading a parquet file
[
https://issues.apache.org/jira/browse/HIVE-22670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ganesha Shreedhara reassigned HIVE-22670:
-
> ArrayIndexOutOfBoundsException when vectorized reader is used for reading a
> parquet file
>
>
> Key: HIVE-22670
> URL: https://issues.apache.org/jira/browse/HIVE-22670
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.6, 3.1.2
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
>
> ArrayIndexOutOfBoundsException is getting thrown while decoding dictionaryIds
> of a row group in parquet file with vectorization enabled.
> *Exception stack trace:*
> {code:java}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainBinaryDictionary.decodeToBinary(PlainValuesDictionary.java:122)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.ParquetDataColumnReaderFactory$DefaultParquetDataColumnReader.readString(ParquetDataColumnReaderFactory.java:95)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedPrimitiveColumnReader.decodeDictionaryIds(VectorizedPrimitiveColumnReader.java:467)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedPrimitiveColumnReader.readBatch(VectorizedPrimitiveColumnReader.java:68)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:410)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
> ... 24 more{code}
>
> This issue seems to be caused by re-using the same dictionary column vector
> while reading consecutive row groups. This looks like one of the corner case
> bug which occurs for a certain distribution of dictionary/plain encoded data
> while we read/populate the underlying bit packed dictionary data into a
> column-vector based data structure.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21213) Acid table bootstrap replication needs to handle directory created by compaction with txn id
[ https://issues.apache.org/jira/browse/HIVE-21213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21213: --- Status: Patch Available (was: Open) > Acid table bootstrap replication needs to handle directory created by > compaction with txn id > > > Key: HIVE-21213 > URL: https://issues.apache.org/jira/browse/HIVE-21213 > Project: Hive > Issue Type: Bug > Components: Hive, HiveServer2, repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21213.01.patch, HIVE-21213.02.patch, > HIVE-21213.03.patch, HIVE-21213.04.patch > > Time Spent: 2h 10m > Remaining Estimate: 0h > > The current implementation of compaction uses the txn id in the directory > name. This is used to isolate the queries from reading the directory until > compaction has finished and to avoid the compactor marking used earlier. In > case of replication, during bootstrap , directory is copied as it is with the > same name from source to destination cluster. But the directory created by > compaction with txn id can not be copied as the txn list at target may be > different from source. The txn id which is valid at source may be an aborted > txn at target. So conversion logic is required to create a new directory with > valid txn at target and dump the data to the newly created directory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21213) Acid table bootstrap replication needs to handle directory created by compaction with txn id
[ https://issues.apache.org/jira/browse/HIVE-21213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21213: --- Status: Open (was: Patch Available) > Acid table bootstrap replication needs to handle directory created by > compaction with txn id > > > Key: HIVE-21213 > URL: https://issues.apache.org/jira/browse/HIVE-21213 > Project: Hive > Issue Type: Bug > Components: Hive, HiveServer2, repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21213.01.patch, HIVE-21213.02.patch, > HIVE-21213.03.patch, HIVE-21213.04.patch > > Time Spent: 2h 10m > Remaining Estimate: 0h > > The current implementation of compaction uses the txn id in the directory > name. This is used to isolate the queries from reading the directory until > compaction has finished and to avoid the compactor marking used earlier. In > case of replication, during bootstrap , directory is copied as it is with the > same name from source to destination cluster. But the directory created by > compaction with txn id can not be copied as the txn list at target may be > different from source. The txn id which is valid at source may be an aborted > txn at target. So conversion logic is required to create a new directory with > valid txn at target and dump the data to the newly created directory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21213) Acid table bootstrap replication needs to handle directory created by compaction with txn id
[ https://issues.apache.org/jira/browse/HIVE-21213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21213: --- Attachment: HIVE-21213.04.patch > Acid table bootstrap replication needs to handle directory created by > compaction with txn id > > > Key: HIVE-21213 > URL: https://issues.apache.org/jira/browse/HIVE-21213 > Project: Hive > Issue Type: Bug > Components: Hive, HiveServer2, repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21213.01.patch, HIVE-21213.02.patch, > HIVE-21213.03.patch, HIVE-21213.04.patch > > Time Spent: 2h 10m > Remaining Estimate: 0h > > The current implementation of compaction uses the txn id in the directory > name. This is used to isolate the queries from reading the directory until > compaction has finished and to avoid the compactor marking used earlier. In > case of replication, during bootstrap , directory is copied as it is with the > same name from source to destination cluster. But the directory created by > compaction with txn id can not be copied as the txn list at target may be > different from source. The txn id which is valid at source may be an aborted > txn at target. So conversion logic is required to create a new directory with > valid txn at target and dump the data to the newly created directory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
