[jira] [Updated] (HIVE-12247) HiveServer2 in HTTP mode should print a relevant error message when its threadpool is configured such that it cannot create appropriate acceptor threads while starting up
[
https://issues.apache.org/jira/browse/HIVE-12247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-12247:
Summary: HiveServer2 in HTTP mode should print a relevant error message
when its threadpool is configured such that it cannot create appropriate
acceptor threads while starting up (was: HiveServer2 in HTTP mode should print
a relevant error message when it's threadpool is configured such that it cannot
create appropriate acceptor threads while starting up)
> HiveServer2 in HTTP mode should print a relevant error message when its
> threadpool is configured such that it cannot create appropriate acceptor
> threads while starting up
> --
>
> Key: HIVE-12247
> URL: https://issues.apache.org/jira/browse/HIVE-12247
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.2.0, 1.1.0
>Reporter: Deepesh Khandelwal
>
> When run in HTTP mode, HS2 uses embedded jetty. The number of acceptor
> threads created are based on this formula
> (http://grepcode.com/file/repo1.maven.org/maven2/org.eclipse.jetty.aggregate/jetty-plus/7.6.0.v20120127/org/eclipse/jetty/server/nio/SelectChannelConnector.java#80):
> {code}
> (Math.max(1,(Runtime.getRuntime().availableProcessors()+3)/4));
> {code}
> In cases where {{hive.server2.thrift.max.worker.threads}} is set to a lower
> value than the above, we should print out a relevant error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12160) Hbase table query execution fails in secured cluster when hive.exec.mode.local.auto is set to true
[
https://issues.apache.org/jira/browse/HIVE-12160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-12160:
Attachment: HIVE-12160.patch
> Hbase table query execution fails in secured cluster when
> hive.exec.mode.local.auto is set to true
> --
>
> Key: HIVE-12160
> URL: https://issues.apache.org/jira/browse/HIVE-12160
> Project: Hive
> Issue Type: Bug
> Components: Security
>Affects Versions: 1.1.0, 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-12160.patch, HIVE-12160_trace.txt
>
>
> In a secured cluster with kerberos, a simple query like {{select count(*)
> from hbase_table; }} will fail with the following exception when
> hive.exec.mode.local.auto is set to true.
> {noformat}
> Error: Error while processing statement: FAILED: Execution Error, return code
> 134 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=134)
> {noformat}
> There is another scenario which may be caused by the same reason.
> Set hive.auto.convert.join to true, the join query {{select * from hbase_t1
> join hbase_t2 on hbase_t1.id = hbase_t2.id;}} also fails with the following
> exception:
> {noformat}
> Error while processing statement: FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask (state=08S01,code=2)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12160) Hbase table query execution fails in secured cluster when hive.exec.mode.local.auto is set to true
[
https://issues.apache.org/jira/browse/HIVE-12160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971511#comment-14971511
]
Aihua Xu commented on HIVE-12160:
-
Submit a patch that, for local task java process, we will login with the passed
in service credential so that it can authenticate against other service like
HDFS and HBase from there.
> Hbase table query execution fails in secured cluster when
> hive.exec.mode.local.auto is set to true
> --
>
> Key: HIVE-12160
> URL: https://issues.apache.org/jira/browse/HIVE-12160
> Project: Hive
> Issue Type: Bug
> Components: Security
>Affects Versions: 1.1.0, 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-12160.patch, HIVE-12160_trace.txt
>
>
> In a secured cluster with kerberos, a simple query like {{select count(*)
> from hbase_table; }} will fail with the following exception when
> hive.exec.mode.local.auto is set to true.
> {noformat}
> Error: Error while processing statement: FAILED: Execution Error, return code
> 134 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=134)
> {noformat}
> There is another scenario which may be caused by the same reason.
> Set hive.auto.convert.join to true, the join query {{select * from hbase_t1
> join hbase_t2 on hbase_t1.id = hbase_t2.id;}} also fails with the following
> exception:
> {noformat}
> Error while processing statement: FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask (state=08S01,code=2)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11531) Add mysql-style LIMIT support to Hive, or improve ROW_NUMBER performance-wise
[ https://issues.apache.org/jira/browse/HIVE-11531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971580#comment-14971580 ] Sergey Shelukhin commented on HIVE-11531: - Do you have more specific questions? My guess is that for these optimizers, the first thing to do is to push the total (offset+limit) in place of old limit. I.e. if you select limit 10, 20, it would push down limit 30, and then Hive logic will discard 10 rows as usual. There is probably other optimization possible as step 2, i.e. not evaluating stuff for first 10 rows in this case, but it may be more difficult. For now, the simple step should suffice. > Add mysql-style LIMIT support to Hive, or improve ROW_NUMBER performance-wise > - > > Key: HIVE-11531 > URL: https://issues.apache.org/jira/browse/HIVE-11531 > Project: Hive > Issue Type: Improvement >Reporter: Sergey Shelukhin >Assignee: Hui Zheng > Attachments: HIVE-11531.WIP.1.patch, HIVE-11531.WIP.2.patch > > > For any UIs that involve pagination, it is useful to issue queries in the > form SELECT ... LIMIT X,Y where X,Y are coordinates inside the result to be > paginated (which can be extremely large by itself). At present, ROW_NUMBER > can be used to achieve this effect, but optimizations for LIMIT such as TopN > in ReduceSink do not apply to ROW_NUMBER. We can add first class support for > "skip" to existing limit, or improve ROW_NUMBER for better performance -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11981) ORC Schema Evolution Issues (Vectorized, ACID, and Non-Vectorized)
[
https://issues.apache.org/jira/browse/HIVE-11981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971476#comment-14971476
]
Prasanth Jayachandran commented on HIVE-11981:
--
This is what I expect to see in TreeReaderFactory (please let me know if I am
missing some feature that requires more changes to reader)
{code}
Object next(Object previous) {
..
Object result = null;
if (previous == null) {
result = new OrcStruct(columnCount); // this should be from file metadata as
we don't what schema that reader what hence we use file schema column count
(hive does not use this codepath)
} else {
// here we fill all the columns from the underlying file schema. If we see
more columns in the provided OrcStruct then we just null it out
}
..
}
{code}
> ORC Schema Evolution Issues (Vectorized, ACID, and Non-Vectorized)
> --
>
> Key: HIVE-11981
> URL: https://issues.apache.org/jira/browse/HIVE-11981
> Project: Hive
> Issue Type: Bug
> Components: Hive, Transactions
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-11981.01.patch, HIVE-11981.02.patch,
> HIVE-11981.03.patch, HIVE-11981.05.patch, HIVE-11981.06.patch,
> HIVE-11981.07.patch, HIVE-11981.08.patch, HIVE-11981.09.patch, ORC Schema
> Evolution Issues.docx
>
>
> High priority issues with schema evolution for the ORC file format.
> Schema evolution here is limited to adding new columns and a few cases of
> column type-widening (e.g. int to bigint).
> Renaming columns, deleting column, moving columns and other schema evolution
> were not pursued due to lack of importance and lack of time. Also, it
> appears a much more sophisticated metadata would be needed to support them.
> The biggest issues for users have been adding new columns for ACID table
> (HIVE-11421 Support Schema evolution for ACID tables) and vectorization
> (HIVE-10598 Vectorization borks when column is added to table).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12240) HiveServer2 put username in job name when doAs=false (for viewing in RM otherwise all jobs show as 'hive')
[ https://issues.apache.org/jira/browse/HIVE-12240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated HIVE-12240: --- Summary: HiveServer2 put username in job name when doAs=false (for viewing in RM otherwise all jobs show as 'hive') (was: HiveServer2 put username in job when doAs=false (for viewing in RM otherwise all jobs show as 'hive')) > HiveServer2 put username in job name when doAs=false (for viewing in RM > otherwise all jobs show as 'hive') > -- > > Key: HIVE-12240 > URL: https://issues.apache.org/jira/browse/HIVE-12240 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 0.14.0 > Environment: HDP 2.2 >Reporter: Hari Sekhon > > When using HiveServer2 with SQLStdAuthorizer and doAs=false please put the > username in the HIVE- job submission so that we can see which user > submitted which job in the RM queues as all the jobs just show as 'hive' user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12240) HiveServer2 put username in job when doAs=false for viewing in RM
[ https://issues.apache.org/jira/browse/HIVE-12240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated HIVE-12240: --- Summary: HiveServer2 put username in job when doAs=false for viewing in RM (was: Hive when not doAs put username in job name for viewing in RM) > HiveServer2 put username in job when doAs=false for viewing in RM > - > > Key: HIVE-12240 > URL: https://issues.apache.org/jira/browse/HIVE-12240 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 0.14.0 > Environment: HDP 2.2 >Reporter: Hari Sekhon > > When using HiveServer2 with SQLStdAuthorizer and doAs=false please put the > username in the HIVE- job submission so that we can see which user > submitted which job in the RM queues as all the jobs just show as 'hive' user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12240) HiveServer2 put username in job when doAs=false (for viewing in RM otherwise all jobs show as 'hive')
[ https://issues.apache.org/jira/browse/HIVE-12240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated HIVE-12240: --- Summary: HiveServer2 put username in job when doAs=false (for viewing in RM otherwise all jobs show as 'hive') (was: HiveServer2 put username in job when doAs=false for viewing in RM) > HiveServer2 put username in job when doAs=false (for viewing in RM otherwise > all jobs show as 'hive') > - > > Key: HIVE-12240 > URL: https://issues.apache.org/jira/browse/HIVE-12240 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 0.14.0 > Environment: HDP 2.2 >Reporter: Hari Sekhon > > When using HiveServer2 with SQLStdAuthorizer and doAs=false please put the > username in the HIVE- job submission so that we can see which user > submitted which job in the RM queues as all the jobs just show as 'hive' user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12240) HiveServer2 put username in job name when doAs=false (for viewing in RM otherwise all jobs show as 'hive')
[ https://issues.apache.org/jira/browse/HIVE-12240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated HIVE-12240: --- Description: When using HiveServer2 with SQLStdAuthorizer and doAs=false please put the username in the HIVE- job submission so that we can see which user submitted which job in the RM queues as all the jobs just show as 'hive' user (this is running on Tez, so the query isn't shown in the name column making differentiating queries/job difficult). (was: When using HiveServer2 with SQLStdAuthorizer and doAs=false please put the username in the HIVE- job submission so that we can see which user submitted which job in the RM queues as all the jobs just show as 'hive' user.) > HiveServer2 put username in job name when doAs=false (for viewing in RM > otherwise all jobs show as 'hive') > -- > > Key: HIVE-12240 > URL: https://issues.apache.org/jira/browse/HIVE-12240 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 0.14.0 > Environment: HDP 2.2 >Reporter: Hari Sekhon > > When using HiveServer2 with SQLStdAuthorizer and doAs=false please put the > username in the HIVE- job submission so that we can see which user > submitted which job in the RM queues as all the jobs just show as 'hive' user > (this is running on Tez, so the query isn't shown in the name column making > differentiating queries/job difficult). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9013) Hive set command exposes metastore db password
[ https://issues.apache.org/jira/browse/HIVE-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated HIVE-9013: Attachment: HIVE-9013.5.patch Update patch, changes: # change list to map # change stripHivePasswordDetails to stripHiddenConfigurations # add 2 tests to cover changes in HiveConf and SetProcessor > Hive set command exposes metastore db password > -- > > Key: HIVE-9013 > URL: https://issues.apache.org/jira/browse/HIVE-9013 > Project: Hive > Issue Type: Bug >Affects Versions: 0.13.1 >Reporter: Binglin Chang >Assignee: Binglin Chang > Attachments: HIVE-9013.1.patch, HIVE-9013.2.patch, HIVE-9013.3.patch, > HIVE-9013.4.patch, HIVE-9013.5.patch > > > When auth is enabled, we still need set command to set some variables(e.g. > mapreduce.job.queuename), but set command alone also list all > information(including vars in restrict list), this exposes like > "javax.jdo.option.ConnectionPassword" > I think conf var in the restrict list should also excluded from dump vars > command. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11895) CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix udaf_percentile_approx_23.q
[ https://issues.apache.org/jira/browse/HIVE-11895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970678#comment-14970678 ] Lefty Leverenz commented on HIVE-11895: --- [~pxiong], this needs a Fix Version. > CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix > udaf_percentile_approx_23.q > - > > Key: HIVE-11895 > URL: https://issues.apache.org/jira/browse/HIVE-11895 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11895.01.patch, HIVE-11895.02.patch, > HIVE-11895.03.patch > > > Due to a type conversion problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12160) Hbase table query execution fails in secured cluster when hive.exec.mode.local.auto is set to true
[
https://issues.apache.org/jira/browse/HIVE-12160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-12160:
Affects Version/s: 2.0.0
> Hbase table query execution fails in secured cluster when
> hive.exec.mode.local.auto is set to true
> --
>
> Key: HIVE-12160
> URL: https://issues.apache.org/jira/browse/HIVE-12160
> Project: Hive
> Issue Type: Bug
> Components: Security
>Affects Versions: 1.1.0, 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-12160_trace.txt
>
>
> In a secured cluster with kerberos, a simple query like {{select count(*)
> from hbase_table; }} will fail with the following exception when
> hive.exec.mode.local.auto is set to true.
> {noformat}
> Error: Error while processing statement: FAILED: Execution Error, return code
> 134 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=134)
> {noformat}
> There is another scenario which may be caused by the same reason.
> Set hive.auto.convert.join to true, the join query {{select * from hbase_t1
> join hbase_t2 on hbase_t1.id = hbase_t2.id;}} also fails with the following
> exception:
> {noformat}
> Error while processing statement: FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask (state=08S01,code=2)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12235) Improve beeline logging for dynamic service discovery
[
https://issues.apache.org/jira/browse/HIVE-12235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971584#comment-14971584
]
Hive QA commented on HIVE-12235:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12768141/HIVE-12235.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 9699 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.io.orc.TestColumnStatistics.testHasNull
org.apache.hadoop.hive.ql.io.orc.TestJsonFileDump.testJsonDump
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5752/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5752/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5752/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12768141 - PreCommit-HIVE-TRUNK-Build
> Improve beeline logging for dynamic service discovery
> -
>
> Key: HIVE-12235
> URL: https://issues.apache.org/jira/browse/HIVE-12235
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 1.2.1
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-12235.patch
>
>
> It maybe nice to see which host it tried to, and ended up, connecting to.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11807) Set ORC buffer size in relation to set stripe size
[ https://issues.apache.org/jira/browse/HIVE-11807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971592#comment-14971592 ] Lefty Leverenz commented on HIVE-11807: --- Cool. Thanks. > Set ORC buffer size in relation to set stripe size > -- > > Key: HIVE-11807 > URL: https://issues.apache.org/jira/browse/HIVE-11807 > Project: Hive > Issue Type: Improvement > Components: File Formats >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-11807-branch-1.patch, HIVE-11807.patch, > HIVE-11807.patch, HIVE-11807.patch > > > A customer produced ORC files with very small stripe sizes (10k rows/stripe) > by setting a small 64MB stripe size and 256K buffer size for a 54 column > table. At that size, each of the streams only get a buffer or two before the > stripe size is reached. The current code uses the available memory instead of > the stripe size and thus doesn't shrink the buffer size if the JVM has much > more memory than the stripe size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12157) Support unicode for table/column names
[
https://issues.apache.org/jira/browse/HIVE-12157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971626#comment-14971626
]
Sergey Shelukhin commented on HIVE-12157:
-
Please make sure to test this change on all the metastore database backends,
e.g. the mysql db script still has the stuff like this:
{noformat}
`TBL_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
{noformat}
This will probably need to change and would need an upgrade script
> Support unicode for table/column names
> ---
>
> Key: HIVE-12157
> URL: https://issues.apache.org/jira/browse/HIVE-12157
> Project: Hive
> Issue Type: Bug
> Components: hpl/sql
>Affects Versions: 1.2.1
>Reporter: richard du
>Assignee: richard du
>Priority: Minor
> Attachments: HIVE-12157.01.patch, HIVE-12157.patch
>
>
> Parser will throw exception when I use alias:
> hive> desc test;
> OK
> a int
> b string
> Time taken: 0.135 seconds, Fetched: 2 row(s)
> hive> select a as 行1 from test limit 10;
> NoViableAltException(302@[134:7: ( ( ( KW_AS )? identifier ) | ( KW_AS LPAREN
> identifier ( COMMA identifier )* RPAREN ) )?])
> at org.antlr.runtime.DFA.noViableAlt(DFA.java:158)
> at org.antlr.runtime.DFA.predict(DFA.java:116)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectItem(HiveParser_SelectClauseParser.java:2915)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectList(HiveParser_SelectClauseParser.java:1373)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1128)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:45827)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41495)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1590)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1109)
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202)
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:396)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> FAILED: ParseException line 1:13 cannot recognize input near 'as' '1' 'from'
> in selection target
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11378) Remove hadoop-1 support from master branch
[ https://issues.apache.org/jira/browse/HIVE-11378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971635#comment-14971635 ] Sergey Shelukhin commented on HIVE-11378: - +1 modulo test failures, if they are related > Remove hadoop-1 support from master branch > -- > > Key: HIVE-11378 > URL: https://issues.apache.org/jira/browse/HIVE-11378 > Project: Hive > Issue Type: Task > Components: Build Infrastructure >Affects Versions: 2.0.0 >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 2.0.0 > > Attachments: HIVE-11378.2.patch, HIVE-11378.3.patch, > HIVE-11378.4.patch, HIVE-11378.patch > > > When we branched branch-1 one of the goals was the ability to remove hadoop1 > support from master. I propose to do this softly at first by removing it > from the poms removing the 20S implementation of the shims. > I am not going to remove the shim layer. That would be much more disruptive. > Also, I haven't done the homework to see if we could, as there may still be > incompatibility issues between various versions of hadoop2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12242) SQLStdAuth add ability to map Roles to existing Groups
[ https://issues.apache.org/jira/browse/HIVE-12242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated HIVE-12242: --- Summary: SQLStdAuth add ability to map Roles to existing Groups (was: SQLStdAuth map Roles to existing Groups) > SQLStdAuth add ability to map Roles to existing Groups > -- > > Key: HIVE-12242 > URL: https://issues.apache.org/jira/browse/HIVE-12242 > Project: Hive > Issue Type: New Feature > Components: SQLStandardAuthorization >Affects Versions: 0.14.0 > Environment: HDP 2.2 >Reporter: Hari Sekhon > > Feature request to be able to map Hive roles to groups so that LDAP groups > can be reused rather than having to recreate all the information in Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11100) Beeline should escape semi-colon in queries
[ https://issues.apache.org/jira/browse/HIVE-11100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970733#comment-14970733 ] Lefty Leverenz commented on HIVE-11100: --- Does this need to be documented in the wiki? (If so, please add a TODOC1.3 label.) > Beeline should escape semi-colon in queries > --- > > Key: HIVE-11100 > URL: https://issues.apache.org/jira/browse/HIVE-11100 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 1.2.0, 1.1.0 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang >Priority: Minor > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-11100.patch > > > Beeline should escape the semicolon in queries. for example, the query like > followings: > CREATE TABLE beeline_tb (c1 int, c2 string) ROW FORMAT DELIMITED FIELDS > TERMINATED BY ';' LINES TERMINATED BY '\n'; > or > CREATE TABLE beeline_tb (c1 int, c2 string) ROW FORMAT DELIMITED FIELDS > TERMINATED BY '\;' LINES TERMINATED BY '\n'; > both failed. > But the 2nd query with semicolon escaped with "\" works in CLI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12223) Filter on Grouping__ID does not work properly
[
https://issues.apache.org/jira/browse/HIVE-12223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970758#comment-14970758
]
Hive QA commented on HIVE-12223:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12768060/HIVE-12223.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 9702 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.io.orc.TestColumnStatistics.testHasNull
org.apache.hadoop.hive.ql.io.orc.TestJsonFileDump.testJsonDump
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5747/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5747/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5747/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12768060 - PreCommit-HIVE-TRUNK-Build
> Filter on Grouping__ID does not work properly
> -
>
> Key: HIVE-12223
> URL: https://issues.apache.org/jira/browse/HIVE-12223
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Fix For: 1.3.0, 2.0.0
>
> Attachments: HIVE-12223.01.patch, HIVE-12223.patch
>
>
> Consider the following query:
> {noformat}
> SELECT key, value, GROUPING__ID, count(*)
> FROM T1
> GROUP BY key, value
> GROUPING SETS ((), (key))
> HAVING GROUPING__ID = 1
> {noformat}
> This query will not return results. The reason is that a "constant"
> placeholder is introduced by SemanticAnalyzer for the GROUPING\__ID column.
> At execution time, this placeholder is replaced by the actual value of the
> GROUPING\__ID. As it is a constant, the Hive optimizer will evaluate
> statically whether the condition is met or not, leading to incorrect results.
> A possible solution is to transform the placeholder constant into a function
> over the grouping keys.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12237) Replace log4j direct references with slf4j facade
[ https://issues.apache.org/jira/browse/HIVE-12237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971151#comment-14971151 ] Prasanth Jayachandran commented on HIVE-12237: -- Is there any reason to still hold on to commons-logging? If we can remove it then we can get rid of log4j-jcl dependency as well. Once this patch gets in I will see if we can get rid of compat dependency log4j-1.2-api. Ideally, we should only have slf4j-api and log4j-slf4j-impl. > Replace log4j direct references with slf4j facade > - > > Key: HIVE-12237 > URL: https://issues.apache.org/jira/browse/HIVE-12237 > Project: Hive > Issue Type: Task > Components: Logging >Affects Versions: 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12237.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12244: -- Attachment: HIVE-12244.1.patch > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Priority: Minor > Attachments: HIVE-12244.1.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we used Path from org.apache.hadoop.fs we avoid new problems in > future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7575) GetTables thrift call is very slow
[ https://issues.apache.org/jira/browse/HIVE-7575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971197#comment-14971197 ] Navis commented on HIVE-7575: - getTables() is the first call from most BI tools but it takes so much time with 100+ databases. I think it's worth to make a dedicated API in metastore for this. > GetTables thrift call is very slow > -- > > Key: HIVE-7575 > URL: https://issues.apache.org/jira/browse/HIVE-7575 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.12.0, 0.13.0 >Reporter: Ashu Pachauri >Assignee: Navis > Attachments: HIVE-7575.1.patch.txt > > > The GetTables thrift call takes a long time when the number of table is large. > With around 5000 tables, the call takes around 80 seconds compared to a "Show > Tables" query on the same HiveServer2 instance which takes 3-7 seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-12160) Hbase table query execution fails in secured cluster when hive.exec.mode.local.auto is set to true
[
https://issues.apache.org/jira/browse/HIVE-12160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu reassigned HIVE-12160:
---
Assignee: Aihua Xu
> Hbase table query execution fails in secured cluster when
> hive.exec.mode.local.auto is set to true
> --
>
> Key: HIVE-12160
> URL: https://issues.apache.org/jira/browse/HIVE-12160
> Project: Hive
> Issue Type: Bug
> Components: Security
>Affects Versions: 1.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>
> In a secured cluster with kerberos, a simple query like {{select count(*)
> from hbase_table; }} will fail with the following exception when
> hive.exec.mode.local.auto is set to true.
> {noformat}
> Error: Error while processing statement: FAILED: Execution Error, return code
> 134 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=134)
> {noformat}
> There is another scenario which may be caused by the same reason.
> Set hive.auto.convert.join to true, the join query {{select * from hbase_t1
> join hbase_t2 on hbase_t1.id = hbase_t2.id;}} also fails with the following
> exception:
> {noformat}
> Error while processing statement: FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask (state=08S01,code=2)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11540) Too many delta files during Compaction
[
https://issues.apache.org/jira/browse/HIVE-11540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-11540:
--
Summary: Too many delta files during Compaction (was: HDP 2.3 and Flume
1.6: Hive Streaming – Too many delta files during Compaction)
> Too many delta files during Compaction
> --
>
> Key: HIVE-11540
> URL: https://issues.apache.org/jira/browse/HIVE-11540
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Nivin Mathew
>Assignee: Eugene Koifman
> Attachments: HIVE-11540.3.patch, HIVE-11540.patch
>
>
> Hello,
> I am streaming weblogs to Kafka and then to Flume 1.6 using a Hive sink, with
> an average of 20 million records a day. I have 5 compactors running at
> various times (30m/5m/5s), no matter what time I give, the compactors seem to
> run out of memory cleaning up a couple thousand delta files and ultimately
> falls behind compacting/cleaning delta files. Any suggestions on what I can
> do to improve performance? Or can Hive streaming not handle this kind of load?
> I used this post as reference:
> http://henning.kropponline.de/2015/05/19/hivesink-for-flume/
> {noformat}
> 2015-08-12 15:05:01,197 FATAL [main] org.apache.hadoop.mapred.YarnChild:
> Error running child : java.lang.OutOfMemoryError: Direct buffer memory
> Max block location exceeded for split: CompactorInputSplit{base:
> hdfs://Dev01HWNameService/user/hive/warehouse/weblogs.db/dt=15-08-12/base_1056406,
> bucket: 0, length: 6493042, deltas: [delta_1056407_1056408,
> delta_1056409_1056410, delta_1056411_1056412, delta_1056413_1056414,
> delta_1056415_1056416, delta_1056417_1056418,…
> , delta_1074039_1074040, delta_1074041_1074042, delta_1074043_1074044,
> delta_1074045_1074046, delta_1074047_1074048, delta_1074049_1074050,
> delta_1074051_1074052]} splitsize: 8772 maxsize: 10
> 2015-08-12 15:34:25,271 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(198)) - number of
> splits:3
> 2015-08-12 15:34:25,367 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.JobSubmitter (JobSubmitter.java:printTokens(287)) - Submitting
> tokens for job: job_1439397150426_0068
> 2015-08-12 15:34:25,603 INFO [upladevhwd04v.researchnow.com-18]:
> impl.YarnClientImpl (YarnClientImpl.java:submitApplication(274)) - Submitted
> application application_1439397150426_0068
> 2015-08-12 15:34:25,610 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:submit(1294)) - The url to track the job:
> http://upladevhwd02v.researchnow.com:8088/proxy/application_1439397150426_0068/
> 2015-08-12 15:34:25,611 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:monitorAndPrintJob(1339)) - Running job:
> job_1439397150426_0068
> 2015-08-12 15:34:30,170 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:34:33,756 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:monitorAndPrintJob(1360)) - Job
> job_1439397150426_0068 running in uber mode : false
> 2015-08-12 15:34:33,757 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:monitorAndPrintJob(1367)) - map 0% reduce 0%
> 2015-08-12 15:34:35,147 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:34:40,155 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:34:45,184 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:34:50,201 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:34:55,256 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:35:00,205 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:35:02,975 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:monitorAndPrintJob(1367)) - map 33% reduce 0%
> 2015-08-12 15:35:02,982 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:printTaskEvents(1406)) - Task Id :
> attempt_1439397150426_0068_m_00_0, Status : FAILED
> 2015-08-12 15:35:03,000 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:printTaskEvents(1406)) - Task Id :
> attempt_1439397150426_0068_m_01_0, Status : FAILED
> 2015-08-12 15:35:04,008 INFO
[jira] [Updated] (HIVE-11540) Too many delta files during Compaction - OOM
[
https://issues.apache.org/jira/browse/HIVE-11540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-11540:
--
Summary: Too many delta files during Compaction - OOM (was: Too many delta
files during Compaction)
> Too many delta files during Compaction - OOM
>
>
> Key: HIVE-11540
> URL: https://issues.apache.org/jira/browse/HIVE-11540
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Nivin Mathew
>Assignee: Eugene Koifman
> Attachments: HIVE-11540.3.patch, HIVE-11540.patch
>
>
> Hello,
> I am streaming weblogs to Kafka and then to Flume 1.6 using a Hive sink, with
> an average of 20 million records a day. I have 5 compactors running at
> various times (30m/5m/5s), no matter what time I give, the compactors seem to
> run out of memory cleaning up a couple thousand delta files and ultimately
> falls behind compacting/cleaning delta files. Any suggestions on what I can
> do to improve performance? Or can Hive streaming not handle this kind of load?
> I used this post as reference:
> http://henning.kropponline.de/2015/05/19/hivesink-for-flume/
> {noformat}
> 2015-08-12 15:05:01,197 FATAL [main] org.apache.hadoop.mapred.YarnChild:
> Error running child : java.lang.OutOfMemoryError: Direct buffer memory
> Max block location exceeded for split: CompactorInputSplit{base:
> hdfs://Dev01HWNameService/user/hive/warehouse/weblogs.db/dt=15-08-12/base_1056406,
> bucket: 0, length: 6493042, deltas: [delta_1056407_1056408,
> delta_1056409_1056410, delta_1056411_1056412, delta_1056413_1056414,
> delta_1056415_1056416, delta_1056417_1056418,…
> , delta_1074039_1074040, delta_1074041_1074042, delta_1074043_1074044,
> delta_1074045_1074046, delta_1074047_1074048, delta_1074049_1074050,
> delta_1074051_1074052]} splitsize: 8772 maxsize: 10
> 2015-08-12 15:34:25,271 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(198)) - number of
> splits:3
> 2015-08-12 15:34:25,367 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.JobSubmitter (JobSubmitter.java:printTokens(287)) - Submitting
> tokens for job: job_1439397150426_0068
> 2015-08-12 15:34:25,603 INFO [upladevhwd04v.researchnow.com-18]:
> impl.YarnClientImpl (YarnClientImpl.java:submitApplication(274)) - Submitted
> application application_1439397150426_0068
> 2015-08-12 15:34:25,610 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:submit(1294)) - The url to track the job:
> http://upladevhwd02v.researchnow.com:8088/proxy/application_1439397150426_0068/
> 2015-08-12 15:34:25,611 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:monitorAndPrintJob(1339)) - Running job:
> job_1439397150426_0068
> 2015-08-12 15:34:30,170 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:34:33,756 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:monitorAndPrintJob(1360)) - Job
> job_1439397150426_0068 running in uber mode : false
> 2015-08-12 15:34:33,757 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:monitorAndPrintJob(1367)) - map 0% reduce 0%
> 2015-08-12 15:34:35,147 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:34:40,155 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:34:45,184 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:34:50,201 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:34:55,256 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:35:00,205 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:35:02,975 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:monitorAndPrintJob(1367)) - map 33% reduce 0%
> 2015-08-12 15:35:02,982 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:printTaskEvents(1406)) - Task Id :
> attempt_1439397150426_0068_m_00_0, Status : FAILED
> 2015-08-12 15:35:03,000 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:printTaskEvents(1406)) - Task Id :
> attempt_1439397150426_0068_m_01_0, Status : FAILED
> 2015-08-12 15:35:04,008 INFO
[jira] [Assigned] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova reassigned HIVE-12244: - Assignee: Alina Abramova > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Attachments: HIVE-12244.1.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we used Path from org.apache.hadoop.fs we avoid new problems in > future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12244: -- Labels: patch (was: ) > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Labels: patch > Attachments: HIVE-12244.1.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we used Path from org.apache.hadoop.fs we avoid new problems in > future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971206#comment-14971206 ] Alina Abramova commented on HIVE-12244: --- https://reviews.apache.org/r/39601/ > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Labels: patch > Attachments: HIVE-12244.1.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we used Path from org.apache.hadoop.fs we avoid new problems in > future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12244: -- Component/s: Hive > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Labels: patch > Attachments: HIVE-12244.1.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we used Path from org.apache.hadoop.fs we avoid new problems in > future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12244: -- Description: In Hive often String is used for representation path and it causes new issues. We need to compare it with equals() but comparing Strings often is not right in terms comparing paths . I think if we used Path from org.apache.hadoop.fs we will avoid new problems in future. was: In Hive often String is used for representation path and it causes new issues. We need to compare it with equals() but comparing Strings often is not right in terms comparing paths . I think if we used Path from org.apache.hadoop.fs we avoid new problems in future. > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Labels: patch > Attachments: HIVE-12244.1.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we used Path from org.apache.hadoop.fs we will avoid new problems > in future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12244: -- Description: In Hive often String is used for representation path and it causes new issues. We need to compare it with equals() but comparing Strings often is not right in terms comparing paths . I think if we use Path from org.apache.hadoop.fs we will avoid new problems in future. was: In Hive often String is used for representation path and it causes new issues. We need to compare it with equals() but comparing Strings often is not right in terms comparing paths . I think if we used Path from org.apache.hadoop.fs we will avoid new problems in future. > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Labels: patch > Attachments: HIVE-12244.1.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we use Path from org.apache.hadoop.fs we will avoid new problems > in future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12160) Hbase table query execution fails in secured cluster when hive.exec.mode.local.auto is set to true
[
https://issues.apache.org/jira/browse/HIVE-12160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971224#comment-14971224
]
Diane Rivier commented on HIVE-12160:
-
The message in the example for the first problem cited applies for the second
example, too.
I will attach a stack trace as well as an environment description in the next
day or two.
> Hbase table query execution fails in secured cluster when
> hive.exec.mode.local.auto is set to true
> --
>
> Key: HIVE-12160
> URL: https://issues.apache.org/jira/browse/HIVE-12160
> Project: Hive
> Issue Type: Bug
> Components: Security
>Affects Versions: 1.1.0
>Reporter: Aihua Xu
>
> In a secured cluster with kerberos, a simple query like {{select count(*)
> from hbase_table; }} will fail with the following exception when
> hive.exec.mode.local.auto is set to true.
> {noformat}
> Error: Error while processing statement: FAILED: Execution Error, return code
> 134 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=134)
> {noformat}
> There is another scenario which may be caused by the same reason.
> Set hive.auto.convert.join to true, the join query {{select * from hbase_t1
> join hbase_t2 on hbase_t1.id = hbase_t2.id;}} also fails with the following
> exception:
> {noformat}
> Error while processing statement: FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask (state=08S01,code=2)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (HIVE-11489) Jenkins PreCommit-HIVE-SPARK-Build fails with TestCliDriver.initializationError
[
https://issues.apache.org/jira/browse/HIVE-11489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-11489:
---
Comment: was deleted
(was:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12765188/HIVE-11489.2-spark.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 6452 tests
executed
*Failed tests:*
{noformat}
TestHBaseNegativeCliDriver - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.initializationError
org.apache.hadoop.hive.cli.TestHBaseCliDriver.initializationError
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_inner_join
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.initializationError
org.apache.hadoop.hive.cli.TestMinimrCliDriver.initializationError
org.apache.hadoop.hive.cli.TestNegativeCliDriver.initializationError
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.initializationError
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchCommit_Delimited
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/968/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/968/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-968/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12765188 - PreCommit-HIVE-SPARK-Build)
> Jenkins PreCommit-HIVE-SPARK-Build fails with
> TestCliDriver.initializationError
> ---
>
> Key: HIVE-11489
> URL: https://issues.apache.org/jira/browse/HIVE-11489
> Project: Hive
> Issue Type: Task
> Components: Testing Infrastructure
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-11489.2-spark.patch
>
>
> The Jenkins job {{PreCommit-HIVE-SPARK-Build}} is failing due to many
> {{TestCliDriver.initializationError}} test results.
> {noformat}
> Error Message
> Unexpected exception java.io.FileNotFoundException:
> /data/hive-ptest/working/apache-git-source-source/itests/qtest/target/generated-test-sources/java/org/apache/hadoop/hive/cli/TestCliDriverQFileNames.txt
> (No such file or directory)
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.(FileInputStream.java:146)
> at java.io.FileReader.(FileReader.java:72)
> at
> org.apache.hadoop.hive.ql.QTestUtil.addTestsToSuiteFromQfileNames(QTestUtil.java:2019)
> at org.apache.hadoop.hive.cli.TestCliDriver.suite(TestCliDriver.java:120)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.internal.runners.SuiteMethod.testFromSuiteMethod(SuiteMethod.java:35)
> at org.junit.internal.runners.SuiteMethod.(SuiteMethod.java:24)
> at
> org.junit.internal.builders.SuiteMethodBuilder.runnerForClass(SuiteMethodBuilder.java:11)
> at
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
> at
> org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
> at
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
> at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:26)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:262)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Stacktrace
> junit.framework.AssertionFailedError: Unexpected exception
> java.io.FileNotFoundException:
>
[jira] [Commented] (HIVE-12237) Replace log4j direct references with slf4j facade
[ https://issues.apache.org/jira/browse/HIVE-12237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971225#comment-14971225 ] Ashutosh Chauhan commented on HIVE-12237: - Since no one wants to review a multi MB patch, I plan to do this in a series of patches. * Get rid of references of log4j * Get rid of references to java util logging * Get rid of commons logging This first patch which addresses first bullet point touches 30+ files. I expect second one to touch 100+ files and third one to touch 700+ files. So, its prudent to do it in series of patches. > Replace log4j direct references with slf4j facade > - > > Key: HIVE-12237 > URL: https://issues.apache.org/jira/browse/HIVE-12237 > Project: Hive > Issue Type: Task > Components: Logging >Affects Versions: 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12237.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12160) Hbase table query execution fails in secured cluster when hive.exec.mode.local.auto is set to true
[
https://issues.apache.org/jira/browse/HIVE-12160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971247#comment-14971247
]
Aihua Xu commented on HIVE-12160:
-
Thanks for also testing that out.
I'm also investigating. Will try to provide a patch. Seems we should relogin
with the service principal in the child process since otherwise, it will run as
the user of the process "hive" rather than the service principal, then it will
fail to authenticate against Hbase or HDFS.
> Hbase table query execution fails in secured cluster when
> hive.exec.mode.local.auto is set to true
> --
>
> Key: HIVE-12160
> URL: https://issues.apache.org/jira/browse/HIVE-12160
> Project: Hive
> Issue Type: Bug
> Components: Security
>Affects Versions: 1.1.0
>Reporter: Aihua Xu
>
> In a secured cluster with kerberos, a simple query like {{select count(*)
> from hbase_table; }} will fail with the following exception when
> hive.exec.mode.local.auto is set to true.
> {noformat}
> Error: Error while processing statement: FAILED: Execution Error, return code
> 134 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=134)
> {noformat}
> There is another scenario which may be caused by the same reason.
> Set hive.auto.convert.join to true, the join query {{select * from hbase_t1
> join hbase_t2 on hbase_t1.id = hbase_t2.id;}} also fails with the following
> exception:
> {noformat}
> Error while processing statement: FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask (state=08S01,code=2)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7575) GetTables thrift call is very slow
[ https://issues.apache.org/jira/browse/HIVE-7575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-7575: Attachment: HIVE-7575.1.patch.txt > GetTables thrift call is very slow > -- > > Key: HIVE-7575 > URL: https://issues.apache.org/jira/browse/HIVE-7575 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.12.0, 0.13.0 >Reporter: Ashu Pachauri > Attachments: HIVE-7575.1.patch.txt > > > The GetTables thrift call takes a long time when the number of table is large. > With around 5000 tables, the call takes around 80 seconds compared to a "Show > Tables" query on the same HiveServer2 instance which takes 3-7 seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12244: -- Flags: (was: Patch) > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Labels: patch > Attachments: HIVE-12244.1.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we used Path from org.apache.hadoop.fs we avoid new problems in > future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12237) Replace log4j direct references with slf4j facade
[ https://issues.apache.org/jira/browse/HIVE-12237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971227#comment-14971227 ] Prasanth Jayachandran commented on HIVE-12237: -- Absolutely make sense :) Make sense to split up the patch to remove appropriate deps. +1 for this patch. > Replace log4j direct references with slf4j facade > - > > Key: HIVE-12237 > URL: https://issues.apache.org/jira/browse/HIVE-12237 > Project: Hive > Issue Type: Task > Components: Logging >Affects Versions: 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12237.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12243) NPE in HiveMetaStoreClient
[
https://issues.apache.org/jira/browse/HIVE-12243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970949#comment-14970949
]
Steve Loughran commented on HIVE-12243:
---
Stack
{code}
- obtain Tokens For HiveMetastore *** FAILED ***
java.lang.reflect.InvocationTargetException:
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.obtainTokenForHiveMetastoreInner(YarnSparkHadoopUtil.scala:217)
at
org.apache.spark.deploy.yarn.YarnSparkHadoopUtilSuite$$anonfun$22.apply(YarnSparkHadoopUtilSuite.scala:255)
at
org.apache.spark.deploy.yarn.YarnSparkHadoopUtilSuite$$anonfun$22.apply(YarnSparkHadoopUtilSuite.scala:249)
at
org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
... Cause: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: Unable to instantiate
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.metadata.Hive.getDelegationToken(Hive.java:3191)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.obtainTokenForHiveMetastoreInner(YarnSparkHadoopUtil.scala:217)
at
org.apache.spark.deploy.yarn.YarnSparkHadoopUtilSuite$$anonfun$22.apply(YarnSparkHadoopUtilSuite.scala:255)
at
org.apache.spark.deploy.yarn.YarnSparkHadoopUtilSuite$$anonfun$22.apply(YarnSparkHadoopUtilSuite.scala:249)
at
org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
... Cause: java.lang.RuntimeException: Unable to instantiate
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1533)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:86)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
at
org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3043)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3062)
at org.apache.hadoop.hive.ql.metadata.Hive.getDelegationToken(Hive.java:3188)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
... Cause: java.lang.reflect.InvocationTargetException:
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1531)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:86)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
at
org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3043)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3062)
... Cause: java.lang.NullPointerException:
at org.apache.thrift.transport.TSocket.open(TSocket.java:170)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:421)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:236)
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:74)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
[jira] [Commented] (HIVE-12243) NPE in HiveMetaStoreClient
[
https://issues.apache.org/jira/browse/HIVE-12243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970950#comment-14970950
]
Steve Loughran commented on HIVE-12243:
---
and code to set up the invalid conf
{code}
hadoopConf.set("hive.metastore.kerberos.principal", "bob")
hadoopConf.set("hive.metastore.uris", "localhost:-1")
{code}
> NPE in HiveMetaStoreClient
> --
>
> Key: HIVE-12243
> URL: https://issues.apache.org/jira/browse/HIVE-12243
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.2.1
>Reporter: Steve Loughran
>Priority: Minor
>
> While writing a test for SPARK-11265, I can trigger an NPE if the principal
> is set to "bob" and the metastore uri = "localhost:-1", due to a null thrift
> transport.
> presumably one of the arguments is invalid
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12186) Upgrade Hive to Calcite 1.5
[
https://issues.apache.org/jira/browse/HIVE-12186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970988#comment-14970988
]
Hive QA commented on HIVE-12186:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12768105/HIVE-12186.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 9701 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ambiguitycheck
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_groupby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_auto_join1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_gby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_subq_not_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_grouping_sets1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lineage3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.ql.io.orc.TestColumnStatistics.testHasNull
org.apache.hadoop.hive.ql.io.orc.TestJsonFileDump.testJsonDump
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5749/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5749/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5749/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12768105 - PreCommit-HIVE-TRUNK-Build
> Upgrade Hive to Calcite 1.5
> ---
>
> Key: HIVE-12186
> URL: https://issues.apache.org/jira/browse/HIVE-12186
> Project: Hive
> Issue Type: Improvement
> Components: CBO, Logical Optimizer
>Affects Versions: 2.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Fix For: 2.0.0
>
> Attachments: HIVE-12186.01.patch, HIVE-12186.01.patch,
> HIVE-12186.patch
>
>
> CLEAR LIBRARY CACHE
> Upgrade Hive to Calcite 1.5.0-incubating.
> There is currently a snapshot release, which is close to what will be in 1.5.
> First, we will test and check any possible issues against the snapshot, so we
> can upgrade quicker once the release is out.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12229) Custom script in query cannot be executed in yarn-cluster mode [Spark Branch].
[
https://issues.apache.org/jira/browse/HIVE-12229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-12229:
--
Attachment: HIVE-12229.2-spark.patch
Patch v2 also makes custom script work for local mode.
> Custom script in query cannot be executed in yarn-cluster mode [Spark Branch].
> --
>
> Key: HIVE-12229
> URL: https://issues.apache.org/jira/browse/HIVE-12229
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Lifeng Wang
>Assignee: Rui Li
> Attachments: HIVE-12229.1-spark.patch, HIVE-12229.2-spark.patch
>
>
> Added one python script in the query and the python script cannot be found
> during execution in yarn-cluster mode.
> {noformat}
> 15/10/21 21:10:55 INFO exec.ScriptOperator: Executing [/usr/bin/python,
> q2-sessionize.py, 3600]
> 15/10/21 21:10:55 INFO exec.ScriptOperator: tablename=null
> 15/10/21 21:10:55 INFO exec.ScriptOperator: partname=null
> 15/10/21 21:10:55 INFO exec.ScriptOperator: alias=null
> 15/10/21 21:10:55 INFO spark.SparkRecordHandler: processing 10 rows: used
> memory = 324896224
> 15/10/21 21:10:55 INFO exec.ScriptOperator: ErrorStreamProcessor calling
> reporter.progress()
> /usr/bin/python: can't open file 'q2-sessionize.py': [Errno 2] No such file
> or directory
> 15/10/21 21:10:55 INFO exec.ScriptOperator: StreamThread OutputProcessor done
> 15/10/21 21:10:55 INFO exec.ScriptOperator: StreamThread ErrorProcessor done
> 15/10/21 21:10:55 INFO spark.SparkRecordHandler: processing 100 rows: used
> memory = 325619920
> 15/10/21 21:10:55 ERROR exec.ScriptOperator: Error in writing to script:
> Stream closed
> 15/10/21 21:10:55 INFO exec.ScriptOperator: The script did not consume all
> input data. This is considered as an error.
> 15/10/21 21:10:55 INFO exec.ScriptOperator: set
> hive.exec.script.allow.partial.consumption=true; to ignore it.
> 15/10/21 21:10:55 ERROR spark.SparkReduceRecordHandler: Fatal error:
> org.apache.hadoop.hive.ql.metadata.HiveException: Error while processing row
> (tag=0)
> {"key":{"reducesinkkey0":2,"reducesinkkey1":3316240655},"value":{"_col0":5529}}
> org.apache.hadoop.hive.ql.metadata.HiveException: Error while processing row
> (tag=0)
> {"key":{"reducesinkkey0":2,"reducesinkkey1":3316240655},"value":{"_col0":5529}}
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.processKeyValues(SparkReduceRecordHandler.java:340)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.processRow(SparkReduceRecordHandler.java:289)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveReduceFunctionResultList.processNextRecord(HiveReduceFunctionResultList.java:49)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveReduceFunctionResultList.processNextRecord(HiveReduceFunctionResultList.java:28)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:95)
> at
> scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
> at
> org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.insertAll(BypassMergeSortShuffleWriter.java:99)
> at
> org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:73)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
> at org.apache.spark.scheduler.Task.run(Task.scala:88)
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: [Error 20001]:
> An error occurred while reading or writing to your custom script. It may have
> crashed with an error.
> at
> org.apache.hadoop.hive.ql.exec.ScriptOperator.processOp(ScriptOperator.java:453)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.processKeyValues(SparkReduceRecordHandler.java:331)
> ... 14 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12223) Filter on Grouping__ID does not work properly
[
https://issues.apache.org/jira/browse/HIVE-12223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970916#comment-14970916
]
Jesus Camacho Rodriguez commented on HIVE-12223:
Test fails are not related. [~jpullokkaran], could you check it? Thanks
> Filter on Grouping__ID does not work properly
> -
>
> Key: HIVE-12223
> URL: https://issues.apache.org/jira/browse/HIVE-12223
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Fix For: 1.3.0, 2.0.0
>
> Attachments: HIVE-12223.01.patch, HIVE-12223.patch
>
>
> Consider the following query:
> {noformat}
> SELECT key, value, GROUPING__ID, count(*)
> FROM T1
> GROUP BY key, value
> GROUPING SETS ((), (key))
> HAVING GROUPING__ID = 1
> {noformat}
> This query will not return results. The reason is that a "constant"
> placeholder is introduced by SemanticAnalyzer for the GROUPING\__ID column.
> At execution time, this placeholder is replaced by the actual value of the
> GROUPING\__ID. As it is a constant, the Hive optimizer will evaluate
> statically whether the condition is met or not, leading to incorrect results.
> A possible solution is to transform the placeholder constant into a function
> over the grouping keys.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-6674) "show grant on all" throws NPE
[
https://issues.apache.org/jira/browse/HIVE-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated HIVE-6674:
--
Affects Version/s: (was: 0.14.0)
> "show grant on all" throws NPE
> --
>
> Key: HIVE-6674
> URL: https://issues.apache.org/jira/browse/HIVE-6674
> Project: Hive
> Issue Type: Bug
>Reporter: Thejas M Nair
>Assignee: Navis
> Fix For: 0.13.0
>
> Attachments: HIVE-6674.1.patch.txt, HIVE-6674.2.patch.txt
>
>
> "show grant on all" is supposed to show all the grants in the system on all
> the objects. But it fails with NPE with both SQL standard auth, and legacy
> auth.
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.ObjectStore.listPrincipalAllDBGrant(ObjectStore.java:4206)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.listPrincipalDBGrantsAll(ObjectStore.java:4169)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
> at com.sun.proxy.$Proxy6.listPrincipalDBGrantsAll(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.list_db_privileges(HiveMetaStore.java:4295)
> ... 36 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12242) SQLStdAuth map Roles to existing Groups
[ https://issues.apache.org/jira/browse/HIVE-12242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated HIVE-12242: --- Summary: SQLStdAuth map Roles to existing Groups (was: SQLStdAuth map roles to groups (ldap)) > SQLStdAuth map Roles to existing Groups > --- > > Key: HIVE-12242 > URL: https://issues.apache.org/jira/browse/HIVE-12242 > Project: Hive > Issue Type: New Feature > Components: SQLStandardAuthorization >Affects Versions: 0.14.0 > Environment: HDP 2.2 >Reporter: Hari Sekhon > > Feature request to be able to map Hive roles to groups so that LDAP groups > can be reused rather than having to recreate all the information in Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12233) NullPointerException StdSQLAuthorizer showing grants via Hive CLI
[
https://issues.apache.org/jira/browse/HIVE-12233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated HIVE-12233:
---
Component/s: SQLStandardAuthorization
> NullPointerException StdSQLAuthorizer showing grants via Hive CLI
> -
>
> Key: HIVE-12233
> URL: https://issues.apache.org/jira/browse/HIVE-12233
> Project: Hive
> Issue Type: Bug
> Components: Authorization, Hive, SQLStandardAuthorization
>Affects Versions: 0.14.0
> Environment: HDP 2.2 + Kerberos
>Reporter: Hari Sekhon
>
> When trying to see the grants in Hive CLI the following NullPointerException
> bug occurs (despite "should use beeline", an NPE is still a bug):
> {code}
> 2015-10-22 16:36:31,535 INFO [main]: ql.Driver
> (Driver.java:launchTask(1604)) - Starting task [Stage-0:DDL] in serial mode
> 2015-10-22 16:36:31,536 ERROR [main]: exec.DDLTask (DDLTask.java:failed(511))
> -
> org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthzPluginException:
> Error showing privileges: null
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLAuthorizationUtils.getPluginException(SQLAuthorizationUtils.java:419)
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessController.showPrivileges(SQLStdHiveAccessController.java:445)
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessControllerWrapper.showPrivileges(SQLStdHiveAccessControllerWrapper.java:141)
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthorizerImpl.showPrivileges(HiveAuthorizerImpl.java:96)
> at org.apache.hadoop.hive.ql.exec.DDLTask.showGrants(DDLTask.java:649)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:465)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1606)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1367)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1179)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1006)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:996)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:247)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:199)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:410)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:783)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessController.isUserAdmin(SQLStdHiveAccessController.java:561)
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessController.showPrivileges(SQLStdHiveAccessController.java:387)
> ... 23 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12241) SQLStdAuth grant on .* not recognized
[
https://issues.apache.org/jira/browse/HIVE-12241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated HIVE-12241:
---
Component/s: SQLStandardAuthorization
> SQLStdAuth grant on .* not recognized
> -
>
> Key: HIVE-12241
> URL: https://issues.apache.org/jira/browse/HIVE-12241
> Project: Hive
> Issue Type: Bug
> Components: Parser, SQLStandardAuthorization
>Affects Versions: 0.14.0
> Environment: HDP 2.2
>Reporter: Hari Sekhon
>
> Using SQLStdAuthorizer Hive doesn't recognize doing a grant on all tables
> like I've done before in RDBMS. If having a lot of tables this becomes very
> inconvenient to grant on a table-by-table basis and granting on database
> succeeds but still doesn't allow user to query tables in that database:
> {code}
> > grant all on myDB.* to user hari;
> Error: Error while compiling statement: FAILED: ParseException line 1:15
> mismatched input '.' expecting TO near 'myDB' in grant privileges
> (state=42000,code=4)
> > grant all on myDB.`*` to user hari;
> Error: Error while compiling statement: FAILED: SemanticException [Error
> 10001]: Table not found myDB.* (state=42S02,code=10001)
> > grant all on `myDB.*` to user hari;
> Error: Error while compiling statement: FAILED: SemanticException [Error
> 10001]: Table not found myDB.* (state=42S02,code=10001)
> > grant all on all to user hari;
> Error: Error while compiling statement: FAILED: SemanticException [Error
> 10001]: Table not found myDB.all (state=42S02,code=10001)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-6674) "show grant on all" throws NPE
[
https://issues.apache.org/jira/browse/HIVE-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated HIVE-6674:
--
Affects Version/s: 0.14.0
> "show grant on all" throws NPE
> --
>
> Key: HIVE-6674
> URL: https://issues.apache.org/jira/browse/HIVE-6674
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Thejas M Nair
>Assignee: Navis
> Fix For: 0.13.0
>
> Attachments: HIVE-6674.1.patch.txt, HIVE-6674.2.patch.txt
>
>
> "show grant on all" is supposed to show all the grants in the system on all
> the objects. But it fails with NPE with both SQL standard auth, and legacy
> auth.
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.ObjectStore.listPrincipalAllDBGrant(ObjectStore.java:4206)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.listPrincipalDBGrantsAll(ObjectStore.java:4169)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
> at com.sun.proxy.$Proxy6.listPrincipalDBGrantsAll(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.list_db_privileges(HiveMetaStore.java:4295)
> ... 36 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11564) HBaseSchemaTool should be able to list objects
[
https://issues.apache.org/jira/browse/HIVE-11564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970873#comment-14970873
]
Hive QA commented on HIVE-11564:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12768075/HIVE-11564.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 9719 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.io.orc.TestColumnStatistics.testHasNull
org.apache.hadoop.hive.ql.io.orc.TestJsonFileDump.testJsonDump
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5748/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5748/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5748/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12768075 - PreCommit-HIVE-TRUNK-Build
> HBaseSchemaTool should be able to list objects
> --
>
> Key: HIVE-11564
> URL: https://issues.apache.org/jira/browse/HIVE-11564
> Project: Hive
> Issue Type: Improvement
> Components: HBase Metastore
>Affects Versions: 2.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: 2.0.0
>
> Attachments: HIVE-11564.2.patch, HIVE-11564.3.patch, HIVE-11564.patch
>
>
> Current HBaseSchemaTool can only fetch objects the user already knows the
> name of. It should also be able to list available objects (e.g. list all
> databases).
> It is also very user unfriendly in terms of error handling. That needs to be
> fixed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12181) Change hive.stats.fetch.column.stats default value to true
[ https://issues.apache.org/jira/browse/HIVE-12181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971299#comment-14971299 ] Prasanth Jayachandran commented on HIVE-12181: -- [~ashutoshc] Is the exception happening in the latest patch too? > Change hive.stats.fetch.column.stats default value to true > -- > > Key: HIVE-12181 > URL: https://issues.apache.org/jira/browse/HIVE-12181 > Project: Hive > Issue Type: Improvement > Components: Statistics >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12181.1.patch, HIVE-12181.patch > > > There was a performance concern earlier, but HIVE-7587 has fixed that. We can > change the default to true now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12187) Release plan once a query is executed
[ https://issues.apache.org/jira/browse/HIVE-12187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-12187: --- Fix Version/s: (was: 1.3.0) > Release plan once a query is executed > -- > > Key: HIVE-12187 > URL: https://issues.apache.org/jira/browse/HIVE-12187 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: 2.0.0 > > Attachments: HIVE-12187.1.patch, HIVE-12187.2.patch > > > Some clients leave query operations open for a while so that they can > retrieve the query results later. That means the allocated memory will be > kept around too. We should release those resources not needed for query > execution any more once it is executed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HIVE-11489) Jenkins PreCommit-HIVE-SPARK-Build fails with TestCliDriver.initializationError
[
https://issues.apache.org/jira/browse/HIVE-11489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-11489:
---
Comment: was deleted
(was:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12765188/HIVE-11489.2-spark.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 6455 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.initializationError
org.apache.hadoop.hive.cli.TestHBaseCliDriver.initializationError
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_inner_join
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.initializationError
org.apache.hadoop.hive.cli.TestMinimrCliDriver.initializationError
org.apache.hadoop.hive.cli.TestNegativeCliDriver.initializationError
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.initializationError
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/969/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/969/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-969/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12765188 - PreCommit-HIVE-SPARK-Build)
> Jenkins PreCommit-HIVE-SPARK-Build fails with
> TestCliDriver.initializationError
> ---
>
> Key: HIVE-11489
> URL: https://issues.apache.org/jira/browse/HIVE-11489
> Project: Hive
> Issue Type: Task
> Components: Testing Infrastructure
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-11489.2-spark.patch
>
>
> The Jenkins job {{PreCommit-HIVE-SPARK-Build}} is failing due to many
> {{TestCliDriver.initializationError}} test results.
> {noformat}
> Error Message
> Unexpected exception java.io.FileNotFoundException:
> /data/hive-ptest/working/apache-git-source-source/itests/qtest/target/generated-test-sources/java/org/apache/hadoop/hive/cli/TestCliDriverQFileNames.txt
> (No such file or directory)
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.(FileInputStream.java:146)
> at java.io.FileReader.(FileReader.java:72)
> at
> org.apache.hadoop.hive.ql.QTestUtil.addTestsToSuiteFromQfileNames(QTestUtil.java:2019)
> at org.apache.hadoop.hive.cli.TestCliDriver.suite(TestCliDriver.java:120)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.internal.runners.SuiteMethod.testFromSuiteMethod(SuiteMethod.java:35)
> at org.junit.internal.runners.SuiteMethod.(SuiteMethod.java:24)
> at
> org.junit.internal.builders.SuiteMethodBuilder.runnerForClass(SuiteMethodBuilder.java:11)
> at
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
> at
> org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
> at
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
> at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:26)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:262)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Stacktrace
> junit.framework.AssertionFailedError: Unexpected exception
> java.io.FileNotFoundException:
> /data/hive-ptest/working/apache-git-source-source/itests/qtest/target/generated-test-sources/java/org/apache/hadoop/hive/cli/TestCliDriverQFileNames.txt
> (No such file or directory)
> at
[jira] [Resolved] (HIVE-1442) HOLD_DDLTIME does not change partition metadata
[ https://issues.apache.org/jira/browse/HIVE-1442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan resolved HIVE-1442. Resolution: Won't Fix See HIVE-12224 > HOLD_DDLTIME does not change partition metadata > --- > > Key: HIVE-1442 > URL: https://issues.apache.org/jira/browse/HIVE-1442 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.7.0 >Reporter: Namit Jain > > create table T1 (key string, value string) partitioned by(ds string) stored > as sequencefile; > desc extended T1; > insert overwrite table T1 partition (ds='1') select key, value from src; > insert overwrite table T1 partition (ds='2') select key, value from src; > desc extended T1 partition (ds='1'); > desc extended T1 partition (ds='2'); > alter table T1 set fileformat rcfile; > insert overwrite table T1 partition (ds='1') > select /*+ HOLD_DDLTIME*/ key, value from src; > insert overwrite table T1 partition (ds='2') > select key, value from src; > desc extended T1 partition (ds='1'); > desc extended T1 partition (ds='2'); > drop table T1; > T1/ds=1 is left as sequencefile and corrupted after the insert as HOLD_DDLTIME -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11895) CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix udaf_percentile_approx_23.q
[ https://issues.apache.org/jira/browse/HIVE-11895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11895: --- Fix Version/s: 2.0.0 > CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix > udaf_percentile_approx_23.q > - > > Key: HIVE-11895 > URL: https://issues.apache.org/jira/browse/HIVE-11895 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.0.0 > > Attachments: HIVE-11895.01.patch, HIVE-11895.02.patch, > HIVE-11895.03.patch > > > Due to a type conversion problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11973) IN operator fails when the column type is DATE
[
https://issues.apache.org/jira/browse/HIVE-11973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971359#comment-14971359
]
Hive QA commented on HIVE-11973:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12768188/HIVE-11973.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 9702 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.io.orc.TestColumnStatistics.testHasNull
org.apache.hadoop.hive.ql.io.orc.TestJsonFileDump.testJsonDump
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5751/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5751/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5751/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12768188 - PreCommit-HIVE-TRUNK-Build
> IN operator fails when the column type is DATE
> ---
>
> Key: HIVE-11973
> URL: https://issues.apache.org/jira/browse/HIVE-11973
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.0.0
>Reporter: sanjiv singh
>Assignee: Yongzhi Chen
> Attachments: HIVE-11973.1.patch, HIVE-11973.2.patch,
> HIVE-11973.3.patch
>
>
> Test DLL :
> {code}
> CREATE TABLE `date_dim`(
> `d_date_sk` int,
> `d_date_id` string,
> `d_date` date,
> `d_current_week` string,
> `d_current_month` string,
> `d_current_quarter` string,
> `d_current_year` string) ;
> {code}
> Hive query :
> {code}
> SELECT *
> FROM date_dim
> WHERE d_date IN ('2000-03-22','2001-03-22') ;
> {code}
> In 1.0.0 , the above query fails with:
> {code}
> FAILED: SemanticException [Error 10014]: Line 1:180 Wrong arguments
> ''2001-03-22'': The arguments for IN should be the same type! Types are:
> {date IN (string, string)}
> {code}
> I changed the query as given to pass the error :
> {code}
> SELECT *
> FROM date_dim
> WHERE d_date IN (CAST('2000-03-22' AS DATE) , CAST('2001-03-22' AS DATE)
> ) ;
> {code}
> But it works without casting :
> {code}
> SELECT *
> FROM date_dim
> WHERE d_date = '2000-03-22' ;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11981) ORC Schema Evolution Issues (Vectorized, ACID, and Non-Vectorized)
[ https://issues.apache.org/jira/browse/HIVE-11981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971420#comment-14971420 ] Prasanth Jayachandran commented on HIVE-11981: -- [~mmccline] As discussed, I feel like the reader side changes to ORC are intrusive and I don't think we need that many changes to null out the additional columns that are being read. Your latest patch doesn't seem to address those changes. Ideally we should have OrcInputFormat add add additional columns to OrcStruct when it creates RecordReader. RecordReaderImpl should just fill those columns with nulls (for OrcStruct reuse) when reading old files that has missing columns. > ORC Schema Evolution Issues (Vectorized, ACID, and Non-Vectorized) > -- > > Key: HIVE-11981 > URL: https://issues.apache.org/jira/browse/HIVE-11981 > Project: Hive > Issue Type: Bug > Components: Hive, Transactions >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-11981.01.patch, HIVE-11981.02.patch, > HIVE-11981.03.patch, HIVE-11981.05.patch, HIVE-11981.06.patch, > HIVE-11981.07.patch, HIVE-11981.08.patch, HIVE-11981.09.patch, ORC Schema > Evolution Issues.docx > > > High priority issues with schema evolution for the ORC file format. > Schema evolution here is limited to adding new columns and a few cases of > column type-widening (e.g. int to bigint). > Renaming columns, deleting column, moving columns and other schema evolution > were not pursued due to lack of importance and lack of time. Also, it > appears a much more sophisticated metadata would be needed to support them. > The biggest issues for users have been adding new columns for ACID table > (HIVE-11421 Support Schema evolution for ACID tables) and vectorization > (HIVE-10598 Vectorization borks when column is added to table). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12160) Hbase table query execution fails in secured cluster when hive.exec.mode.local.auto is set to true
[
https://issues.apache.org/jira/browse/HIVE-12160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Diane Rivier updated HIVE-12160:
Attachment: HIVE-12160_trace.txt
Immediately after completion of stage 2
query had inner join... inner join left outer join sequence
hive.auto.convert.join value was true
repeat with hive.auto.convert.join value false resulted in OutOfMemory mid-way
through stage 5 of 5.
> Hbase table query execution fails in secured cluster when
> hive.exec.mode.local.auto is set to true
> --
>
> Key: HIVE-12160
> URL: https://issues.apache.org/jira/browse/HIVE-12160
> Project: Hive
> Issue Type: Bug
> Components: Security
>Affects Versions: 1.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-12160_trace.txt
>
>
> In a secured cluster with kerberos, a simple query like {{select count(*)
> from hbase_table; }} will fail with the following exception when
> hive.exec.mode.local.auto is set to true.
> {noformat}
> Error: Error while processing statement: FAILED: Execution Error, return code
> 134 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=134)
> {noformat}
> There is another scenario which may be caused by the same reason.
> Set hive.auto.convert.join to true, the join query {{select * from hbase_t1
> join hbase_t2 on hbase_t1.id = hbase_t2.id;}} also fails with the following
> exception:
> {noformat}
> Error while processing statement: FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask (state=08S01,code=2)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12181) Change hive.stats.fetch.column.stats default value to true
[ https://issues.apache.org/jira/browse/HIVE-12181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971313#comment-14971313 ] Ashutosh Chauhan commented on HIVE-12181: - yup.. latest patch is just golden file updates. I havent debugged exceptions yet. > Change hive.stats.fetch.column.stats default value to true > -- > > Key: HIVE-12181 > URL: https://issues.apache.org/jira/browse/HIVE-12181 > Project: Hive > Issue Type: Improvement > Components: Statistics >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12181.1.patch, HIVE-12181.patch > > > There was a performance concern earlier, but HIVE-7587 has fixed that. We can > change the default to true now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11973) IN operator fails when the column type is DATE
[
https://issues.apache.org/jira/browse/HIVE-11973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971372#comment-14971372
]
Yongzhi Chen commented on HIVE-11973:
-
The four failures are not related. The ages are 9 and more.
> IN operator fails when the column type is DATE
> ---
>
> Key: HIVE-11973
> URL: https://issues.apache.org/jira/browse/HIVE-11973
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.0.0
>Reporter: sanjiv singh
>Assignee: Yongzhi Chen
> Attachments: HIVE-11973.1.patch, HIVE-11973.2.patch,
> HIVE-11973.3.patch
>
>
> Test DLL :
> {code}
> CREATE TABLE `date_dim`(
> `d_date_sk` int,
> `d_date_id` string,
> `d_date` date,
> `d_current_week` string,
> `d_current_month` string,
> `d_current_quarter` string,
> `d_current_year` string) ;
> {code}
> Hive query :
> {code}
> SELECT *
> FROM date_dim
> WHERE d_date IN ('2000-03-22','2001-03-22') ;
> {code}
> In 1.0.0 , the above query fails with:
> {code}
> FAILED: SemanticException [Error 10014]: Line 1:180 Wrong arguments
> ''2001-03-22'': The arguments for IN should be the same type! Types are:
> {date IN (string, string)}
> {code}
> I changed the query as given to pass the error :
> {code}
> SELECT *
> FROM date_dim
> WHERE d_date IN (CAST('2000-03-22' AS DATE) , CAST('2001-03-22' AS DATE)
> ) ;
> {code}
> But it works without casting :
> {code}
> SELECT *
> FROM date_dim
> WHERE d_date = '2000-03-22' ;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11489) Jenkins PreCommit-HIVE-SPARK-Build fails with TestCliDriver.initializationError
[
https://issues.apache.org/jira/browse/HIVE-11489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971391#comment-14971391
]
Hive QA commented on HIVE-11489:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12765188/HIVE-11489.2-spark.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 6455 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.initializationError
org.apache.hadoop.hive.cli.TestHBaseCliDriver.initializationError
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_inner_join
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.initializationError
org.apache.hadoop.hive.cli.TestMinimrCliDriver.initializationError
org.apache.hadoop.hive.cli.TestNegativeCliDriver.initializationError
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.initializationError
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/969/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/969/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-969/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12765188 - PreCommit-HIVE-SPARK-Build
> Jenkins PreCommit-HIVE-SPARK-Build fails with
> TestCliDriver.initializationError
> ---
>
> Key: HIVE-11489
> URL: https://issues.apache.org/jira/browse/HIVE-11489
> Project: Hive
> Issue Type: Task
> Components: Testing Infrastructure
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-11489.2-spark.patch
>
>
> The Jenkins job {{PreCommit-HIVE-SPARK-Build}} is failing due to many
> {{TestCliDriver.initializationError}} test results.
> {noformat}
> Error Message
> Unexpected exception java.io.FileNotFoundException:
> /data/hive-ptest/working/apache-git-source-source/itests/qtest/target/generated-test-sources/java/org/apache/hadoop/hive/cli/TestCliDriverQFileNames.txt
> (No such file or directory)
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.(FileInputStream.java:146)
> at java.io.FileReader.(FileReader.java:72)
> at
> org.apache.hadoop.hive.ql.QTestUtil.addTestsToSuiteFromQfileNames(QTestUtil.java:2019)
> at org.apache.hadoop.hive.cli.TestCliDriver.suite(TestCliDriver.java:120)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.internal.runners.SuiteMethod.testFromSuiteMethod(SuiteMethod.java:35)
> at org.junit.internal.runners.SuiteMethod.(SuiteMethod.java:24)
> at
> org.junit.internal.builders.SuiteMethodBuilder.runnerForClass(SuiteMethodBuilder.java:11)
> at
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
> at
> org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
> at
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
> at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:26)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:262)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Stacktrace
> junit.framework.AssertionFailedError: Unexpected exception
> java.io.FileNotFoundException:
> /data/hive-ptest/working/apache-git-source-source/itests/qtest/target/generated-test-sources/java/org/apache/hadoop/hive/cli/TestCliDriverQFileNames.txt
> (No such file or directory)
> at
[jira] [Commented] (HIVE-12181) Change hive.stats.fetch.column.stats default value to true
[ https://issues.apache.org/jira/browse/HIVE-12181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971317#comment-14971317 ] Ashutosh Chauhan commented on HIVE-12181: - I think we can do this in phased manner. Lets debug the only failure on MiniTezCliDriver and set default to true there. We can tackle CliDriver later. > Change hive.stats.fetch.column.stats default value to true > -- > > Key: HIVE-12181 > URL: https://issues.apache.org/jira/browse/HIVE-12181 > Project: Hive > Issue Type: Improvement > Components: Statistics >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12181.1.patch, HIVE-12181.patch > > > There was a performance concern earlier, but HIVE-7587 has fixed that. We can > change the default to true now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12234) Beeline quit tries to connect again if no existing connections
[
https://issues.apache.org/jira/browse/HIVE-12234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971346#comment-14971346
]
Szehon Ho commented on HIVE-12234:
--
Thanks for the fast response! +1
> Beeline quit tries to connect again if no existing connections
> --
>
> Key: HIVE-12234
> URL: https://issues.apache.org/jira/browse/HIVE-12234
> Project: Hive
> Issue Type: Bug
>Reporter: Szehon Ho
>Assignee: Ferdinand Xu
>Priority: Minor
> Attachments: HIVE-12234.patch
>
>
> Beeline !quit calls close(), which then does the following check:
> {noformat}
> beeLine.getDatabaseConnection().getConnection() != null
> {noformat}
> This inadvertently tries to connect again.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9013) Hive set command exposes metastore db password
[ https://issues.apache.org/jira/browse/HIVE-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972355#comment-14972355 ] Thejas M Nair commented on HIVE-9013: - bq. HiveConf instance contains hiddenList information, so make HiveConf.stripHiddenConfigurations static would add another parameter I think, is that OK? I agree, leaving it the way it is, is better. The operation is being performed on a JobConf object, and not a HiveConf object. HiveConf already has this variable initialized. > Hive set command exposes metastore db password > -- > > Key: HIVE-9013 > URL: https://issues.apache.org/jira/browse/HIVE-9013 > Project: Hive > Issue Type: Bug >Affects Versions: 0.13.1 >Reporter: Binglin Chang >Assignee: Binglin Chang > Attachments: HIVE-9013.1.patch, HIVE-9013.2.patch, HIVE-9013.3.patch, > HIVE-9013.4.patch, HIVE-9013.5.patch > > > When auth is enabled, we still need set command to set some variables(e.g. > mapreduce.job.queuename), but set command alone also list all > information(including vars in restrict list), this exposes like > "javax.jdo.option.ConnectionPassword" > I think conf var in the restrict list should also excluded from dump vars > command. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9013) Hive set command exposes metastore db password
[ https://issues.apache.org/jira/browse/HIVE-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972337#comment-14972337 ] Binglin Chang commented on HIVE-9013: - Hi Sushanth, thanks for the review. bq. If we made that call static, we can call HiveConf.stripHiddenConfigurations(job), which would be much clearer. HiveConf instance contains hiddenList information, so make HiveConf.stripHiddenConfigurations static would add another parameter I think, is that OK? > Hive set command exposes metastore db password > -- > > Key: HIVE-9013 > URL: https://issues.apache.org/jira/browse/HIVE-9013 > Project: Hive > Issue Type: Bug >Affects Versions: 0.13.1 >Reporter: Binglin Chang >Assignee: Binglin Chang > Attachments: HIVE-9013.1.patch, HIVE-9013.2.patch, HIVE-9013.3.patch, > HIVE-9013.4.patch, HIVE-9013.5.patch > > > When auth is enabled, we still need set command to set some variables(e.g. > mapreduce.job.queuename), but set command alone also list all > information(including vars in restrict list), this exposes like > "javax.jdo.option.ConnectionPassword" > I think conf var in the restrict list should also excluded from dump vars > command. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12237) Use slf4j as logging facade
[
https://issues.apache.org/jira/browse/HIVE-12237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972346#comment-14972346
]
Hive QA commented on HIVE-12237:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12768463/HIVE-12237.1.patch
{color:green}SUCCESS:{color} +1 due to 12 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 9701 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketmapjoin8
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_filter_join_breaktask
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby2_map_multi_distinct
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby2_noskew_multi_distinct
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_insert_into1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_skewjoin_noskew
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_14
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_16
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union15
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union34
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_data_types
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_9
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_nested_udf
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorized_math_funcs
org.apache.hadoop.hive.ql.io.orc.TestColumnStatistics.testHasNull
org.apache.hadoop.hive.ql.io.orc.TestFileDump.testBloomFilter
org.apache.hadoop.hive.ql.io.orc.TestFileDump.testBloomFilter2
org.apache.hadoop.hive.ql.io.orc.TestFileDump.testDataDump
org.apache.hadoop.hive.ql.io.orc.TestFileDump.testDictionaryThreshold
org.apache.hadoop.hive.ql.io.orc.TestFileDump.testDump
org.apache.hadoop.hive.ql.io.orc.TestJsonFileDump.testJsonDump
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5771/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5771/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5771/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 25 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12768463 - PreCommit-HIVE-TRUNK-Build
> Use slf4j as logging facade
> ---
>
> Key: HIVE-12237
> URL: https://issues.apache.org/jira/browse/HIVE-12237
> Project: Hive
> Issue Type: Task
> Components: Logging
>Affects Versions: 2.0.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-12237.1.patch, HIVE-12237.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12257) Enhance ORC FileDump utility to handle flush_length files
[ https://issues.apache.org/jira/browse/HIVE-12257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972290#comment-14972290 ] Prasanth Jayachandran commented on HIVE-12257: -- [~ekoifman] Can you take a look at this patch? > Enhance ORC FileDump utility to handle flush_length files > - > > Key: HIVE-12257 > URL: https://issues.apache.org/jira/browse/HIVE-12257 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-12257.1.patch > > > ORC file dump utility currently does not handle delta directories that > contain *_flush_length files. These files contains offsets to footer in the > corresponding delta file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9013) Hive set command exposes metastore db password
[ https://issues.apache.org/jira/browse/HIVE-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972338#comment-14972338 ] Binglin Chang commented on HIVE-9013: - Hi Sushanth, thanks for the review. bq. If we made that call static, we can call HiveConf.stripHiddenConfigurations(job), which would be much clearer. HiveConf instance contains hiddenList information, so make HiveConf.stripHiddenConfigurations static would add another parameter I think, is that OK? > Hive set command exposes metastore db password > -- > > Key: HIVE-9013 > URL: https://issues.apache.org/jira/browse/HIVE-9013 > Project: Hive > Issue Type: Bug >Affects Versions: 0.13.1 >Reporter: Binglin Chang >Assignee: Binglin Chang > Attachments: HIVE-9013.1.patch, HIVE-9013.2.patch, HIVE-9013.3.patch, > HIVE-9013.4.patch, HIVE-9013.5.patch > > > When auth is enabled, we still need set command to set some variables(e.g. > mapreduce.job.queuename), but set command alone also list all > information(including vars in restrict list), this exposes like > "javax.jdo.option.ConnectionPassword" > I think conf var in the restrict list should also excluded from dump vars > command. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11378) Remove hadoop-1 support from master branch
[ https://issues.apache.org/jira/browse/HIVE-11378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-11378: -- Attachment: HIVE-11378.5.patch The TestHCatClient failures were not related, but the others were. There were a couple of places I'd forgotten to remove hadoop-1 references. I also had to make changes to ql/pom.xml to make the spark tests work. IIUC, Hadoop brings along javax.servlet.api version 2.5. But because of ordering issues we were picking up version 3.0 from eclipse's version of jetty. When I moved the hadoop dependencies out of the profile that reversed the dependency order and we got 2.5 instead of 3.0, which broke the spark tests. I fixed that by having the hadoop clients exclude javax.servlet in their dependencies. It would be good if someone who understands maven well took a look at those changes. > Remove hadoop-1 support from master branch > -- > > Key: HIVE-11378 > URL: https://issues.apache.org/jira/browse/HIVE-11378 > Project: Hive > Issue Type: Task > Components: Build Infrastructure >Affects Versions: 2.0.0 >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 2.0.0 > > Attachments: HIVE-11378.2.patch, HIVE-11378.3.patch, > HIVE-11378.4.patch, HIVE-11378.5.patch, HIVE-11378.patch > > > When we branched branch-1 one of the goals was the ability to remove hadoop1 > support from master. I propose to do this softly at first by removing it > from the poms removing the 20S implementation of the shims. > I am not going to remove the shim layer. That would be much more disruptive. > Also, I haven't done the homework to see if we could, as there may still be > incompatibility issues between various versions of hadoop2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9013) Hive set command exposes metastore db password
[ https://issues.apache.org/jira/browse/HIVE-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-9013: Attachment: HIVE-9013.5.patch Uploading HIVE-9013.5.patch again to kick off tests > Hive set command exposes metastore db password > -- > > Key: HIVE-9013 > URL: https://issues.apache.org/jira/browse/HIVE-9013 > Project: Hive > Issue Type: Bug >Affects Versions: 0.13.1 >Reporter: Binglin Chang >Assignee: Binglin Chang > Attachments: HIVE-9013.1.patch, HIVE-9013.2.patch, HIVE-9013.3.patch, > HIVE-9013.4.patch, HIVE-9013.5.patch, HIVE-9013.5.patch > > > When auth is enabled, we still need set command to set some variables(e.g. > mapreduce.job.queuename), but set command alone also list all > information(including vars in restrict list), this exposes like > "javax.jdo.option.ConnectionPassword" > I think conf var in the restrict list should also excluded from dump vars > command. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-9013) Hive set command exposes metastore db password
[ https://issues.apache.org/jira/browse/HIVE-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972374#comment-14972374 ] Thejas M Nair edited comment on HIVE-9013 at 10/24/15 4:22 AM: --- Uploading HIVE-9013.5.patch again to kick off tests again. Looks like master was broken for sometime. was (Author: thejas): Uploading HIVE-9013.5.patch again to kick off tests > Hive set command exposes metastore db password > -- > > Key: HIVE-9013 > URL: https://issues.apache.org/jira/browse/HIVE-9013 > Project: Hive > Issue Type: Bug >Affects Versions: 0.13.1 >Reporter: Binglin Chang >Assignee: Binglin Chang > Attachments: HIVE-9013.1.patch, HIVE-9013.2.patch, HIVE-9013.3.patch, > HIVE-9013.4.patch, HIVE-9013.5.patch, HIVE-9013.5.patch > > > When auth is enabled, we still need set command to set some variables(e.g. > mapreduce.job.queuename), but set command alone also list all > information(including vars in restrict list), this exposes like > "javax.jdo.option.ConnectionPassword" > I think conf var in the restrict list should also excluded from dump vars > command. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12213) Investigating the test failure TestHCatClient.testTableSchemaPropagation
[
https://issues.apache.org/jira/browse/HIVE-12213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksei Statkevich updated HIVE-12213:
--
Attachment: HIVE-12213.4.patch
Reuploaded the patch to trigger the build again.
> Investigating the test failure TestHCatClient.testTableSchemaPropagation
>
>
> Key: HIVE-12213
> URL: https://issues.apache.org/jira/browse/HIVE-12213
> Project: Hive
> Issue Type: Test
> Components: Test
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aleksei Statkevich
>Priority: Minor
> Attachments: HIVE-12213.2.patch, HIVE-12213.3.patch,
> HIVE-12213.4.patch, HIVE-12213.patch, HIVE-12231.1.patch
>
>
> The test has been failing for some time with following error.
> {noformat}
> Error Message
> Table after deserialization should have been identical to sourceTable.
> expected:<[TABLE_PROPERTIES]> but was:<[]>
> Stacktrace
> java.lang.AssertionError: Table after deserialization should have been
> identical to sourceTable. expected:<[TABLE_PROPERTIES]> but was:<[]>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation(TestHCatClient.java:1065)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12239) Constants in hive.common.metrics.common.MetricsConstant are not final
[ https://issues.apache.org/jira/browse/HIVE-12239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksei Statkevich updated HIVE-12239: -- Attachment: HIVE-12239.1.patch Reupload the patch to trigger the build again. Trivial change, no unit tests needed. > Constants in hive.common.metrics.common.MetricsConstant are not final > - > > Key: HIVE-12239 > URL: https://issues.apache.org/jira/browse/HIVE-12239 > Project: Hive > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Aleksei Statkevich >Assignee: Aleksei Statkevich >Priority: Trivial > Attachments: HIVE-12239.1.patch, HIVE-12239.patch > > > Constants defined in > org.apache.hadoop.hive.common.metrics.common.MetricsConstant are not marked > as final. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11540) Too many delta files during Compaction - OOM
[
https://issues.apache.org/jira/browse/HIVE-11540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972412#comment-14972412
]
Hive QA commented on HIVE-11540:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12768469/HIVE-11540.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 9705 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.io.orc.TestColumnStatistics.testHasNull
org.apache.hadoop.hive.ql.io.orc.TestJsonFileDump.testJsonDump
org.apache.hadoop.hive.ql.txn.compactor.TestWorker2.majorNoBaseLotsOfDeltas
org.apache.hadoop.hive.ql.txn.compactor.TestWorker2.minorNoBaseLotsOfDeltas
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5772/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5772/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5772/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12768469 - PreCommit-HIVE-TRUNK-Build
> Too many delta files during Compaction - OOM
>
>
> Key: HIVE-11540
> URL: https://issues.apache.org/jira/browse/HIVE-11540
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Nivin Mathew
>Assignee: Eugene Koifman
> Attachments: HIVE-11540.3.patch, HIVE-11540.4.patch, HIVE-11540.patch
>
>
> Hello,
> I am streaming weblogs to Kafka and then to Flume 1.6 using a Hive sink, with
> an average of 20 million records a day. I have 5 compactors running at
> various times (30m/5m/5s), no matter what time I give, the compactors seem to
> run out of memory cleaning up a couple thousand delta files and ultimately
> falls behind compacting/cleaning delta files. Any suggestions on what I can
> do to improve performance? Or can Hive streaming not handle this kind of load?
> I used this post as reference:
> http://henning.kropponline.de/2015/05/19/hivesink-for-flume/
> {noformat}
> 2015-08-12 15:05:01,197 FATAL [main] org.apache.hadoop.mapred.YarnChild:
> Error running child : java.lang.OutOfMemoryError: Direct buffer memory
> Max block location exceeded for split: CompactorInputSplit{base:
> hdfs://Dev01HWNameService/user/hive/warehouse/weblogs.db/dt=15-08-12/base_1056406,
> bucket: 0, length: 6493042, deltas: [delta_1056407_1056408,
> delta_1056409_1056410, delta_1056411_1056412, delta_1056413_1056414,
> delta_1056415_1056416, delta_1056417_1056418,…
> , delta_1074039_1074040, delta_1074041_1074042, delta_1074043_1074044,
> delta_1074045_1074046, delta_1074047_1074048, delta_1074049_1074050,
> delta_1074051_1074052]} splitsize: 8772 maxsize: 10
> 2015-08-12 15:34:25,271 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(198)) - number of
> splits:3
> 2015-08-12 15:34:25,367 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.JobSubmitter (JobSubmitter.java:printTokens(287)) - Submitting
> tokens for job: job_1439397150426_0068
> 2015-08-12 15:34:25,603 INFO [upladevhwd04v.researchnow.com-18]:
> impl.YarnClientImpl (YarnClientImpl.java:submitApplication(274)) - Submitted
> application application_1439397150426_0068
> 2015-08-12 15:34:25,610 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:submit(1294)) - The url to track the job:
> http://upladevhwd02v.researchnow.com:8088/proxy/application_1439397150426_0068/
> 2015-08-12 15:34:25,611 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:monitorAndPrintJob(1339)) - Running job:
> job_1439397150426_0068
> 2015-08-12 15:34:30,170 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:34:33,756 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:monitorAndPrintJob(1360)) - Job
> job_1439397150426_0068 running in uber mode : false
> 2015-08-12 15:34:33,757 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:monitorAndPrintJob(1367)) - map 0% reduce 0%
> 2015-08-12 15:34:35,147 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
[jira] [Commented] (HIVE-11489) Jenkins PreCommit-HIVE-SPARK-Build fails with TestCliDriver.initializationError
[
https://issues.apache.org/jira/browse/HIVE-11489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971647#comment-14971647
]
Hive QA commented on HIVE-11489:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12765188/HIVE-11489.2-spark.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 6455 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.initializationError
org.apache.hadoop.hive.cli.TestHBaseCliDriver.initializationError
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_inner_join
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.initializationError
org.apache.hadoop.hive.cli.TestMinimrCliDriver.initializationError
org.apache.hadoop.hive.cli.TestNegativeCliDriver.initializationError
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.initializationError
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/970/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/970/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-970/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12765188 - PreCommit-HIVE-SPARK-Build
> Jenkins PreCommit-HIVE-SPARK-Build fails with
> TestCliDriver.initializationError
> ---
>
> Key: HIVE-11489
> URL: https://issues.apache.org/jira/browse/HIVE-11489
> Project: Hive
> Issue Type: Task
> Components: Testing Infrastructure
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-11489.2-spark.patch
>
>
> The Jenkins job {{PreCommit-HIVE-SPARK-Build}} is failing due to many
> {{TestCliDriver.initializationError}} test results.
> {noformat}
> Error Message
> Unexpected exception java.io.FileNotFoundException:
> /data/hive-ptest/working/apache-git-source-source/itests/qtest/target/generated-test-sources/java/org/apache/hadoop/hive/cli/TestCliDriverQFileNames.txt
> (No such file or directory)
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.(FileInputStream.java:146)
> at java.io.FileReader.(FileReader.java:72)
> at
> org.apache.hadoop.hive.ql.QTestUtil.addTestsToSuiteFromQfileNames(QTestUtil.java:2019)
> at org.apache.hadoop.hive.cli.TestCliDriver.suite(TestCliDriver.java:120)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.internal.runners.SuiteMethod.testFromSuiteMethod(SuiteMethod.java:35)
> at org.junit.internal.runners.SuiteMethod.(SuiteMethod.java:24)
> at
> org.junit.internal.builders.SuiteMethodBuilder.runnerForClass(SuiteMethodBuilder.java:11)
> at
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
> at
> org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
> at
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
> at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:26)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:262)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Stacktrace
> junit.framework.AssertionFailedError: Unexpected exception
> java.io.FileNotFoundException:
> /data/hive-ptest/working/apache-git-source-source/itests/qtest/target/generated-test-sources/java/org/apache/hadoop/hive/cli/TestCliDriverQFileNames.txt
> (No such file or directory)
> at
[jira] [Commented] (HIVE-12045) ClassNotFound for GenericUDF in "select distinct..." query (Hive on Spark)
[
https://issues.apache.org/jira/browse/HIVE-12045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971713#comment-14971713
]
Zsolt Tóth commented on HIVE-12045:
---
[~lirui] thanks for the hint, it works indeed with yarn-client mode. As I can
see, HIVE-9882 is fixed in Hive 1.2 but CDH5.4.2 has Hive 1.1. I'll try to run
my queries against Hive 1.2 on a different cluster to see if they're fixed.
> ClassNotFound for GenericUDF in "select distinct..." query (Hive on Spark)
> --
>
> Key: HIVE-12045
> URL: https://issues.apache.org/jira/browse/HIVE-12045
> Project: Hive
> Issue Type: Bug
> Components: Spark
> Environment: Cloudera QuickStart VM - CDH5.4.2
> beeline
>Reporter: Zsolt Tóth
> Attachments: example.jar
>
>
> If I execute the following query in beeline, I get ClassNotFoundException for
> the UDF class.
> {code}
> drop function myGenericUdf;
> create function myGenericUdf as 'org.example.myGenericUdf' using jar
> 'hdfs:///tmp/myudf.jar';
> select distinct myGenericUdf(1,2,1) from mytable;
> {code}
> In my example, myGenericUdf just looks for the 1st argument's value in the
> others and returns the index. I don't think this is related to the actual
> GenericUDF function.
> Note that:
> "select myGenericUdf(1,2,1) from mytable;" succeeds
> If I use the non-generic implementation of the same UDF, the select distinct
> call succeeds.
> StackTrace:
> {code}
> 15/10/06 05:20:25 ERROR exec.Utilities: Failed to load plan:
> hdfs://quickstart.cloudera:8020/tmp/hive/hive/f9de3f09-c12d-4528-9ee6-1f12932a14ae/hive_2015-10-06_05-20-07_438_6519207588897968406-20/-mr-10003/27cd7226-3e22-46f4-bddd-fb8fd4aa4b8d/map.xml:
> org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find
> class: org.example.myGenericUDF
> Serialization trace:
> genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
> chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
> colExprMap (org.apache.hadoop.hive.ql.exec.GroupByOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
> org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find
> class: org.example.myGenericUDF
> Serialization trace:
> genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
> chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
> colExprMap (org.apache.hadoop.hive.ql.exec.GroupByOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:138)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:656)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:99)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
> at
>
[jira] [Commented] (HIVE-12246) Orc FileDump fails with Missing CLI jar
[ https://issues.apache.org/jira/browse/HIVE-12246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971735#comment-14971735 ] Prasanth Jayachandran commented on HIVE-12246: -- [~xuefuz]/[~Ferd] HIVE-10624 broke other tools orcfiledump/rcfilecat etc. Can some of you review this patch as you guys have more context on this issue? > Orc FileDump fails with Missing CLI jar > --- > > Key: HIVE-12246 > URL: https://issues.apache.org/jira/browse/HIVE-12246 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-12246.patch > > > Running hive --orcfiledump fails with "Missing CLI jar" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12171) LLAP: BuddyAllocator failures when querying uncompressed data
[
https://issues.apache.org/jira/browse/HIVE-12171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971784#comment-14971784
]
Sergey Shelukhin commented on HIVE-12171:
-
[~prasanth_j] can you review? [~gopalv] you can also review?
> LLAP: BuddyAllocator failures when querying uncompressed data
> -
>
> Key: HIVE-12171
> URL: https://issues.apache.org/jira/browse/HIVE-12171
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
> Attachments: HIVE-12171.other.patch, HIVE-12171.patch
>
>
> {code}
> hive> select sum(l_extendedprice * l_discount) as revenue from
> testing.lineitem where l_shipdate >= '1993-01-01' and l_shipdate <
> '1994-01-01' ;
> Caused by:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 492; at 0 out of 1
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:176)
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.preReadUncompressedStream(EncodedReaderImpl.java:882)
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedColumns(EncodedReaderImpl.java:319)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:413)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:194)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:191)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:191)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:74)
> at
> org.apache.hadoop.hive.common.CallableWithNdc.call(CallableWithNdc.java:37)
> ... 4 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11523) org.apache.hadoop.hive.ql.io.orc.FileDump should handle errors
[ https://issues.apache.org/jira/browse/HIVE-11523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971807#comment-14971807 ] Prasanth Jayachandran commented on HIVE-11523: -- [~ekoifman] Can you take a look? > org.apache.hadoop.hive.ql.io.orc.FileDump should handle errors > -- > > Key: HIVE-11523 > URL: https://issues.apache.org/jira/browse/HIVE-11523 > Project: Hive > Issue Type: Bug > Components: File Formats >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Prasanth Jayachandran > Attachments: HIVE-11523.patch > > > this utility can take N files as arguments. Currently if it fails to read > some file, it fails and bails out. It should instead log the file name and > error and proceed with other files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10328) Enable new return path for cbo
[
https://issues.apache.org/jira/browse/HIVE-10328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971779#comment-14971779
]
Hive QA commented on HIVE-10328:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12768143/HIVE-10328.12.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1446 failed/errored test(s), 9700 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alias_casted_column
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_allcolref_in_udf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_change_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_coltype
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_table_cascade
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ambiguitycheck
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_analyze_table_null_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_deep_filters
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_filter
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_groupby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_groupby2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join_pkfk
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ansi_sql_arithmetic
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_array_map_access_nonconstant
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_explain
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join0
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join15
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join16
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join17
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join18
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join18_multi_distinct
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join19
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join22
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join24
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join25
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join26
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join27
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join30
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join31
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join32
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join33
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_filters
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_nulls
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_without_localtask
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_smb_mapjoin_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_14
[jira] [Commented] (HIVE-12171) LLAP: BuddyAllocator failures when querying uncompressed data
[
https://issues.apache.org/jira/browse/HIVE-12171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971797#comment-14971797
]
Gopal V commented on HIVE-12171:
[~sershe]: I'll test it & make sure it works.
> LLAP: BuddyAllocator failures when querying uncompressed data
> -
>
> Key: HIVE-12171
> URL: https://issues.apache.org/jira/browse/HIVE-12171
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
> Attachments: HIVE-12171.other.patch, HIVE-12171.patch
>
>
> {code}
> hive> select sum(l_extendedprice * l_discount) as revenue from
> testing.lineitem where l_shipdate >= '1993-01-01' and l_shipdate <
> '1994-01-01' ;
> Caused by:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 492; at 0 out of 1
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:176)
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.preReadUncompressedStream(EncodedReaderImpl.java:882)
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedColumns(EncodedReaderImpl.java:319)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:413)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:194)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:191)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:191)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:74)
> at
> org.apache.hadoop.hive.common.CallableWithNdc.call(CallableWithNdc.java:37)
> ... 4 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12171) LLAP: BuddyAllocator failures when querying uncompressed data
[
https://issues.apache.org/jira/browse/HIVE-12171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971805#comment-14971805
]
Sergey Shelukhin commented on HIVE-12171:
-
I can test it actually. I think it should also resolve task failures on small
repro case (the initial one)
> LLAP: BuddyAllocator failures when querying uncompressed data
> -
>
> Key: HIVE-12171
> URL: https://issues.apache.org/jira/browse/HIVE-12171
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
> Attachments: HIVE-12171.other.patch, HIVE-12171.patch
>
>
> {code}
> hive> select sum(l_extendedprice * l_discount) as revenue from
> testing.lineitem where l_shipdate >= '1993-01-01' and l_shipdate <
> '1994-01-01' ;
> Caused by:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 492; at 0 out of 1
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:176)
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.preReadUncompressedStream(EncodedReaderImpl.java:882)
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedColumns(EncodedReaderImpl.java:319)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:413)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:194)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:191)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:191)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:74)
> at
> org.apache.hadoop.hive.common.CallableWithNdc.call(CallableWithNdc.java:37)
> ... 4 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11523) org.apache.hadoop.hive.ql.io.orc.FileDump should handle errors
[ https://issues.apache.org/jira/browse/HIVE-11523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-11523: - Attachment: HIVE-11523.patch > org.apache.hadoop.hive.ql.io.orc.FileDump should handle errors > -- > > Key: HIVE-11523 > URL: https://issues.apache.org/jira/browse/HIVE-11523 > Project: Hive > Issue Type: Bug > Components: File Formats >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-11523.patch > > > this utility can take N files as arguments. Currently if it fails to read > some file, it fails and bails out. It should instead log the file name and > error and proceed with other files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-11523) org.apache.hadoop.hive.ql.io.orc.FileDump should handle errors
[ https://issues.apache.org/jira/browse/HIVE-11523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-11523: Assignee: Prasanth Jayachandran (was: Eugene Koifman) > org.apache.hadoop.hive.ql.io.orc.FileDump should handle errors > -- > > Key: HIVE-11523 > URL: https://issues.apache.org/jira/browse/HIVE-11523 > Project: Hive > Issue Type: Bug > Components: File Formats >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Prasanth Jayachandran > Attachments: HIVE-11523.patch > > > this utility can take N files as arguments. Currently if it fails to read > some file, it fails and bails out. It should instead log the file name and > error and proceed with other files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12246) Orc FileDump fails with Missing CLI jar
[ https://issues.apache.org/jira/browse/HIVE-12246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-12246: - Attachment: HIVE-12246.patch > Orc FileDump fails with Missing CLI jar > --- > > Key: HIVE-12246 > URL: https://issues.apache.org/jira/browse/HIVE-12246 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-12246.patch > > > Running hive --orcfiledump fails with "Missing CLI jar" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-12171) LLAP: BuddyAllocator failures when querying uncompressed data
[
https://issues.apache.org/jira/browse/HIVE-12171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971784#comment-14971784
]
Sergey Shelukhin edited comment on HIVE-12171 at 10/23/15 8:26 PM:
---
[~prasanth_j] can you review? [~gopalv] you can also review :)
was (Author: sershe):
[~prasanth_j] can you review? [~gopalv] you can also review?
> LLAP: BuddyAllocator failures when querying uncompressed data
> -
>
> Key: HIVE-12171
> URL: https://issues.apache.org/jira/browse/HIVE-12171
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
> Attachments: HIVE-12171.other.patch, HIVE-12171.patch
>
>
> {code}
> hive> select sum(l_extendedprice * l_discount) as revenue from
> testing.lineitem where l_shipdate >= '1993-01-01' and l_shipdate <
> '1994-01-01' ;
> Caused by:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 492; at 0 out of 1
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:176)
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.preReadUncompressedStream(EncodedReaderImpl.java:882)
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedColumns(EncodedReaderImpl.java:319)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:413)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:194)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:191)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:191)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:74)
> at
> org.apache.hadoop.hive.common.CallableWithNdc.call(CallableWithNdc.java:37)
> ... 4 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11523) org.apache.hadoop.hive.ql.io.orc.FileDump should handle errors
[ https://issues.apache.org/jira/browse/HIVE-11523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-11523: - Attachment: HIVE-11523-branch-1.patch > org.apache.hadoop.hive.ql.io.orc.FileDump should handle errors > -- > > Key: HIVE-11523 > URL: https://issues.apache.org/jira/browse/HIVE-11523 > Project: Hive > Issue Type: Bug > Components: File Formats >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Prasanth Jayachandran > Attachments: HIVE-11523-branch-1.patch, HIVE-11523.patch > > > this utility can take N files as arguments. Currently if it fails to read > some file, it fails and bails out. It should instead log the file name and > error and proceed with other files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11540) Too many delta files during Compaction - OOM
[
https://issues.apache.org/jira/browse/HIVE-11540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-11540:
--
Attachment: HIVE-11540.4.patch
> Too many delta files during Compaction - OOM
>
>
> Key: HIVE-11540
> URL: https://issues.apache.org/jira/browse/HIVE-11540
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Nivin Mathew
>Assignee: Eugene Koifman
> Attachments: HIVE-11540.3.patch, HIVE-11540.4.patch, HIVE-11540.patch
>
>
> Hello,
> I am streaming weblogs to Kafka and then to Flume 1.6 using a Hive sink, with
> an average of 20 million records a day. I have 5 compactors running at
> various times (30m/5m/5s), no matter what time I give, the compactors seem to
> run out of memory cleaning up a couple thousand delta files and ultimately
> falls behind compacting/cleaning delta files. Any suggestions on what I can
> do to improve performance? Or can Hive streaming not handle this kind of load?
> I used this post as reference:
> http://henning.kropponline.de/2015/05/19/hivesink-for-flume/
> {noformat}
> 2015-08-12 15:05:01,197 FATAL [main] org.apache.hadoop.mapred.YarnChild:
> Error running child : java.lang.OutOfMemoryError: Direct buffer memory
> Max block location exceeded for split: CompactorInputSplit{base:
> hdfs://Dev01HWNameService/user/hive/warehouse/weblogs.db/dt=15-08-12/base_1056406,
> bucket: 0, length: 6493042, deltas: [delta_1056407_1056408,
> delta_1056409_1056410, delta_1056411_1056412, delta_1056413_1056414,
> delta_1056415_1056416, delta_1056417_1056418,…
> , delta_1074039_1074040, delta_1074041_1074042, delta_1074043_1074044,
> delta_1074045_1074046, delta_1074047_1074048, delta_1074049_1074050,
> delta_1074051_1074052]} splitsize: 8772 maxsize: 10
> 2015-08-12 15:34:25,271 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(198)) - number of
> splits:3
> 2015-08-12 15:34:25,367 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.JobSubmitter (JobSubmitter.java:printTokens(287)) - Submitting
> tokens for job: job_1439397150426_0068
> 2015-08-12 15:34:25,603 INFO [upladevhwd04v.researchnow.com-18]:
> impl.YarnClientImpl (YarnClientImpl.java:submitApplication(274)) - Submitted
> application application_1439397150426_0068
> 2015-08-12 15:34:25,610 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:submit(1294)) - The url to track the job:
> http://upladevhwd02v.researchnow.com:8088/proxy/application_1439397150426_0068/
> 2015-08-12 15:34:25,611 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:monitorAndPrintJob(1339)) - Running job:
> job_1439397150426_0068
> 2015-08-12 15:34:30,170 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:34:33,756 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:monitorAndPrintJob(1360)) - Job
> job_1439397150426_0068 running in uber mode : false
> 2015-08-12 15:34:33,757 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:monitorAndPrintJob(1367)) - map 0% reduce 0%
> 2015-08-12 15:34:35,147 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:34:40,155 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:34:45,184 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:34:50,201 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:34:55,256 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:35:00,205 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:35:02,975 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:monitorAndPrintJob(1367)) - map 33% reduce 0%
> 2015-08-12 15:35:02,982 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:printTaskEvents(1406)) - Task Id :
> attempt_1439397150426_0068_m_00_0, Status : FAILED
> 2015-08-12 15:35:03,000 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:printTaskEvents(1406)) - Task Id :
> attempt_1439397150426_0068_m_01_0, Status : FAILED
> 2015-08-12 15:35:04,008 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job
[jira] [Commented] (HIVE-12256) Move LLAP registry into llap-client module
[
https://issues.apache.org/jira/browse/HIVE-12256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972232#comment-14972232
]
Hive QA commented on HIVE-12256:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12768464/HIVE-12256.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5770/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5770/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5770/
Messages:
{noformat}
This message was trimmed, see log for full details
symbol: variable RegistryUtils
location: class
org.apache.hadoop.hive.llap.registry.impl.LlapYarnRegistryImpl.DynamicServiceInstanceSet
[ERROR]
/data/hive-ptest/working/apache-github-source-source/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/registry/impl/LlapYarnRegistryImpl.java:[282,14]
cannot find symbol
symbol: class ServiceRecord
location: class
org.apache.hadoop.hive.llap.registry.impl.LlapYarnRegistryImpl.DynamicServiceInstanceSet
[INFO] 59 errors
[INFO] -
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] Hive .. SUCCESS [15.043s]
[INFO] Hive Shims Common . SUCCESS [19.154s]
[INFO] Hive Shims 0.20S .. SUCCESS [3.651s]
[INFO] Hive Shims 0.23 ... SUCCESS [13.340s]
[INFO] Hive Shims Scheduler .. SUCCESS [3.711s]
[INFO] Hive Shims SUCCESS [3.494s]
[INFO] Hive Storage API .. SUCCESS [4.449s]
[INFO] Hive Common ... SUCCESS [23.319s]
[INFO] Hive Serde SUCCESS [17.479s]
[INFO] Hive Metastore SUCCESS [50.392s]
[INFO] Hive Ant Utilities SUCCESS [1.824s]
[INFO] Hive Llap Client .. SUCCESS [10.645s]
[INFO] Spark Remote Client ... SUCCESS [21.495s]
[INFO] Hive Query Language ... SUCCESS [2:21.603s]
[INFO] Hive Service .. SUCCESS [9.757s]
[INFO] Hive Accumulo Handler . SUCCESS [6.606s]
[INFO] Hive JDBC . SUCCESS [19.864s]
[INFO] Hive Beeline .. SUCCESS [2.897s]
[INFO] Hive CLI .. SUCCESS [2.984s]
[INFO] Hive Contrib .. SUCCESS [2.120s]
[INFO] Hive HBase Handler SUCCESS [6.163s]
[INFO] Hive HCatalog . SUCCESS [0.823s]
[INFO] Hive HCatalog Core SUCCESS [5.953s]
[INFO] Hive HCatalog Pig Adapter . SUCCESS [2.993s]
[INFO] Hive HCatalog Server Extensions ... SUCCESS [2.548s]
[INFO] Hive HCatalog Webhcat Java Client . SUCCESS [3.408s]
[INFO] Hive HCatalog Webhcat . SUCCESS [16.036s]
[INFO] Hive HCatalog Streaming ... SUCCESS [3.408s]
[INFO] Hive HPL/SQL .. SUCCESS [16.273s]
[INFO] Hive HWI .. SUCCESS [1.708s]
[INFO] Hive ODBC . SUCCESS [0.986s]
[INFO] Hive Shims Aggregator . SUCCESS [0.187s]
[INFO] Hive TestUtils SUCCESS [0.378s]
[INFO] Hive Llap Server .. FAILURE [4.114s]
[INFO] Hive Packaging SKIPPED
[INFO]
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 7:22.078s
[INFO] Finished at: Fri Oct 23 21:16:45 EDT 2015
[INFO] Final Memory: 211M/1062M
[INFO]
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on
project hive-llap-server: Compilation failure: Compilation failure:
[ERROR]
/data/hive-ptest/working/apache-github-source-source/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/registry/impl/LlapYarnRegistryImpl.java:[39,45]
package org.apache.hadoop.registry.client.api does not exist
[ERROR]
[jira] [Updated] (HIVE-12246) Orc FileDump fails with Missing CLI jar
[ https://issues.apache.org/jira/browse/HIVE-12246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-12246: - Attachment: HIVE-12246.patch > Orc FileDump fails with Missing CLI jar > --- > > Key: HIVE-12246 > URL: https://issues.apache.org/jira/browse/HIVE-12246 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-12246.patch, HIVE-12246.patch > > > Running hive --orcfiledump fails with "Missing CLI jar" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11523) org.apache.hadoop.hive.ql.io.orc.FileDump should handle errors
[ https://issues.apache.org/jira/browse/HIVE-11523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-11523: - Attachment: HIVE-11523.patch > org.apache.hadoop.hive.ql.io.orc.FileDump should handle errors > -- > > Key: HIVE-11523 > URL: https://issues.apache.org/jira/browse/HIVE-11523 > Project: Hive > Issue Type: Bug > Components: File Formats >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Prasanth Jayachandran > Attachments: HIVE-11523-branch-1.patch, HIVE-11523.patch, > HIVE-11523.patch > > > this utility can take N files as arguments. Currently if it fails to read > some file, it fails and bails out. It should instead log the file name and > error and proceed with other files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12257) Enhance ORC FileDump utility to handle flush_length files
[ https://issues.apache.org/jira/browse/HIVE-12257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-12257: - Attachment: HIVE-12257.1.patch > Enhance ORC FileDump utility to handle flush_length files > - > > Key: HIVE-12257 > URL: https://issues.apache.org/jira/browse/HIVE-12257 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-12257.1.patch > > > ORC file dump utility currently does not handle delta directories that > contain *_flush_length files. These files contains offsets to footer in the > corresponding delta file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11807) Set ORC buffer size in relation to set stripe size
[ https://issues.apache.org/jira/browse/HIVE-11807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971110#comment-14971110 ] Prasanth Jayachandran commented on HIVE-11807: -- [~leftylev] I don't think this requires documentation as this is a self adjusting config to make sure we get better data layout. > Set ORC buffer size in relation to set stripe size > -- > > Key: HIVE-11807 > URL: https://issues.apache.org/jira/browse/HIVE-11807 > Project: Hive > Issue Type: Improvement > Components: File Formats >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-11807-branch-1.patch, HIVE-11807.patch, > HIVE-11807.patch, HIVE-11807.patch > > > A customer produced ORC files with very small stripe sizes (10k rows/stripe) > by setting a small 64MB stripe size and 256K buffer size for a 54 column > table. At that size, each of the streams only get a buffer or two before the > stripe size is reached. The current code uses the available memory instead of > the stripe size and thus doesn't shrink the buffer size if the JVM has much > more memory than the stripe size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12244: -- Affects Version/s: 0.13.0 0.14.0 1.0.0 1.2.1 > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Priority: Minor > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we used Path from org.apache.hadoop.fs we avoid new problems in > future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12237) Replace log4j direct references with slf4j facade
[ https://issues.apache.org/jira/browse/HIVE-12237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-12237: - Affects Version/s: 2.0.0 > Replace log4j direct references with slf4j facade > - > > Key: HIVE-12237 > URL: https://issues.apache.org/jira/browse/HIVE-12237 > Project: Hive > Issue Type: Task > Components: Logging >Affects Versions: 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12237.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12213) Investigating the test failure TestHCatClient.testTableSchemaPropagation
[
https://issues.apache.org/jira/browse/HIVE-12213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971112#comment-14971112
]
Aihua Xu commented on HIVE-12213:
-
+1.
> Investigating the test failure TestHCatClient.testTableSchemaPropagation
>
>
> Key: HIVE-12213
> URL: https://issues.apache.org/jira/browse/HIVE-12213
> Project: Hive
> Issue Type: Test
> Components: Test
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aleksei Statkevich
>Priority: Minor
> Attachments: HIVE-12213.2.patch, HIVE-12213.3.patch,
> HIVE-12213.patch, HIVE-12231.1.patch
>
>
> The test has been failing for some time with following error.
> {noformat}
> Error Message
> Table after deserialization should have been identical to sourceTable.
> expected:<[TABLE_PROPERTIES]> but was:<[]>
> Stacktrace
> java.lang.AssertionError: Table after deserialization should have been
> identical to sourceTable. expected:<[TABLE_PROPERTIES]> but was:<[]>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation(TestHCatClient.java:1065)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11954) Extend logic to choose side table in MapJoin Conversion algorithm
[
https://issues.apache.org/jira/browse/HIVE-11954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971145#comment-14971145
]
Hive QA commented on HIVE-11954:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12768134/HIVE-11954.12.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 9701 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.io.orc.TestColumnStatistics.testHasNull
org.apache.hadoop.hive.ql.io.orc.TestJsonFileDump.testJsonDump
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5750/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5750/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5750/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12768134 - PreCommit-HIVE-TRUNK-Build
> Extend logic to choose side table in MapJoin Conversion algorithm
> -
>
> Key: HIVE-11954
> URL: https://issues.apache.org/jira/browse/HIVE-11954
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 2.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-11954.01.patch, HIVE-11954.02.patch,
> HIVE-11954.03.patch, HIVE-11954.04.patch, HIVE-11954.05.patch,
> HIVE-11954.06.patch, HIVE-11954.07.patch, HIVE-11954.08.patch,
> HIVE-11954.09.patch, HIVE-11954.10.patch, HIVE-11954.11.patch,
> HIVE-11954.12.patch, HIVE-11954.patch, HIVE-11954.patch
>
>
> Selection of side table (in memory/hash table) in MapJoin Conversion
> algorithm needs to be more sophisticated.
> In an N way Map Join, Hive should pick an input stream as side table (in
> memory table) that has least cost in producing relation (like TS(FIL|Proj)*).
> Cost based choice needs extended cost model; without return path its going to
> be hard to do this.
> For the time being we could employ a modified cost based algorithm for side
> table selection.
> New algorithm is described below:
> 1. Identify the candidate set of inputs for side table (in memory/hash table)
> from the inputs (based on conditional task size)
> 2. For each of the input identify its cost, memory requirement. Cost is 1 for
> each heavy weight relation op (Join, GB, PTF/Windowing, TF, etc.). Cost for
> an input is the total no of heavy weight ops in its branch.
> 3. Order set from #1 on cost & memory req (ascending order)
> 4. Pick the first element from #3 as the side table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11981) ORC Schema Evolution Issues (Vectorized, ACID, and Non-Vectorized)
[ https://issues.apache.org/jira/browse/HIVE-11981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-11981: Attachment: HIVE-11981.09.patch > ORC Schema Evolution Issues (Vectorized, ACID, and Non-Vectorized) > -- > > Key: HIVE-11981 > URL: https://issues.apache.org/jira/browse/HIVE-11981 > Project: Hive > Issue Type: Bug > Components: Hive, Transactions >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-11981.01.patch, HIVE-11981.02.patch, > HIVE-11981.03.patch, HIVE-11981.05.patch, HIVE-11981.06.patch, > HIVE-11981.07.patch, HIVE-11981.08.patch, HIVE-11981.09.patch, ORC Schema > Evolution Issues.docx > > > High priority issues with schema evolution for the ORC file format. > Schema evolution here is limited to adding new columns and a few cases of > column type-widening (e.g. int to bigint). > Renaming columns, deleting column, moving columns and other schema evolution > were not pursued due to lack of importance and lack of time. Also, it > appears a much more sophisticated metadata would be needed to support them. > The biggest issues for users have been adding new columns for ACID table > (HIVE-11421 Support Schema evolution for ACID tables) and vectorization > (HIVE-10598 Vectorization borks when column is added to table). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9013) Hive set command exposes metastore db password
[ https://issues.apache.org/jira/browse/HIVE-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970527#comment-14970527 ] Thejas M Nair commented on HIVE-9013: - [~decster] Yes, I think he meant to say Utilities.stripHiddenConfigurations not Utilities.stripRestrictedConfigurations. Also, for testing "set " command case, I think a java unit test would be better than a .q file based one, as .q.out file would have many entries including some that depend on local environment, and might end up needing additional logic for masking/ignoring those during comparison. > Hive set command exposes metastore db password > -- > > Key: HIVE-9013 > URL: https://issues.apache.org/jira/browse/HIVE-9013 > Project: Hive > Issue Type: Bug >Affects Versions: 0.13.1 >Reporter: Binglin Chang >Assignee: Binglin Chang > Attachments: HIVE-9013.1.patch, HIVE-9013.2.patch, HIVE-9013.3.patch, > HIVE-9013.4.patch > > > When auth is enabled, we still need set command to set some variables(e.g. > mapreduce.job.queuename), but set command alone also list all > information(including vars in restrict list), this exposes like > "javax.jdo.option.ConnectionPassword" > I think conf var in the restrict list should also excluded from dump vars > command. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11901) StorageBasedAuthorizationProvider requires write permission on table for SELECT statements
[
https://issues.apache.org/jira/browse/HIVE-11901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970533#comment-14970533
]
Thejas M Nair commented on HIVE-11901:
--
The test failures are unrelated.
> StorageBasedAuthorizationProvider requires write permission on table for
> SELECT statements
> --
>
> Key: HIVE-11901
> URL: https://issues.apache.org/jira/browse/HIVE-11901
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 1.2.1
>Reporter: Chengbing Liu
>Assignee: Chengbing Liu
> Attachments: HIVE-11901.01.patch, HIVE-11901.02.patch,
> HIVE-11901.03.patch
>
>
> With HIVE-7895, it will require write permission on the table directory even
> for a SELECT statement.
> Looking at the stacktrace, it seems the method
> {{StorageBasedAuthorizationProvider#authorize(Table table, Partition part,
> Privilege[] readRequiredPriv, Privilege[] writeRequiredPriv)}} always treats
> a null partition as a CREATE statement, which can also be a SELECT.
> We may have to check {{readRequiredPriv}} and {{writeRequiredPriv}} first
> in order to tell which statement it is.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12232) Create external table failed when enabled StorageBasedAuthorization
[
https://issues.apache.org/jira/browse/HIVE-12232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970539#comment-14970539
]
Hive QA commented on HIVE-12232:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12768027/HIVE-12232.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 9700 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.io.orc.TestColumnStatistics.testHasNull
org.apache.hadoop.hive.ql.io.orc.TestJsonFileDump.testJsonDump
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5745/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5745/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5745/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12768027 - PreCommit-HIVE-TRUNK-Build
> Create external table failed when enabled StorageBasedAuthorization
> ---
>
> Key: HIVE-12232
> URL: https://issues.apache.org/jira/browse/HIVE-12232
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: WangMeng
>Assignee: WangMeng
> Attachments: HIVE-12232.01.patch
>
>
> Please look at the stacktrace, when enabled StorageBasedAuthorization,
> creating external table will failed with write permission about the default
> warehouse path "/user/hive/warehouse":
> > CREATE EXTERNAL TABLE test(id int) LOCATION '/tmp/wangmeng/test' ;
> Error: Error while compiling statement: FAILED: HiveException
> java.security.AccessControlException: Permission denied: user=wangmeng,
> access=WRITE, inode="/user/hive/warehouse":hive:hive:drwxr-x--t.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
