[jira] [Created] (HIVE-4904) A little more CP crossing RS boundaries
Navis created HIVE-4904: --- Summary: A little more CP crossing RS boundaries Key: HIVE-4904 URL: https://issues.apache.org/jira/browse/HIVE-4904 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Currently, CP context cannot be propagated over RS except for JOIN/EXT. A little more CP is possible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: VOTE: moving hive from forest to Apache CMS
+1 (non-binding) On Jul 21, 2013, at 8:25 PM, Mark Grover grover.markgro...@gmail.com wrote: +1 (non-binding) On Sun, Jul 21, 2013 at 11:08 AM, Jarek Jarcec Cecho jar...@apache.org wrote: +1 (non-binding) Jarcec On Sun, Jul 21, 2013 at 01:53:39PM -0400, Edward Capriolo wrote: http://hive.apache.org is generated by forest, a rather cumbersome and confusing way to run a website. Forest is difficult to maintain and publish updates with. As a nail in the coffin forest does not even work well with recent versions of java. This vote is to move the site to: Apache CMShttps://www.apache.org/dev/cms.html and away from forest. Brock Noland has offered to move the site, and I am offering to help him and look it over. Vote +1 if you support the move to Apache CMS. (This is the one case where cutting down a forest is a very good idea :) Edward -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF
[jira] [Commented] (HIVE-4827) Merge a Map-only job to its following MapReduce job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715009#comment-13715009
]
Hive QA commented on HIVE-4827:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12593451/HIVE-4827.2.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 2647 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/123/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/123/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
Merge a Map-only job to its following MapReduce job with multiple inputs
Key: HIVE-4827
URL: https://issues.apache.org/jira/browse/HIVE-4827
Project: Hive
Issue Type: Improvement
Components: Query Processor
Affects Versions: 0.12.0
Reporter: Yin Huai
Assignee: Yin Huai
Attachments: HIVE-4827.1.patch, HIVE-4827.2.patch
When hive.optimize.mapjoin.mapreduce is on, CommonJoinResolver can attach a
Map-only job (MapJoin) to its following MapReduce job. But this merge only
happens when the MapReduce job has a single input. With Correlation Optimizer
(HIVE-2206), it is possible that the MapReduce job can have multiple inputs
(for multiple operation paths). It is desired to improve CommonJoinResolver
to merge a Map-only job to the corresponding Map task of the MapReduce job.
Example:
{code:sql}
set hive.optimize.correlation=true;
set hive.auto.convert.join=true;
set hive.optimize.mapjoin.mapreduce=true;
SELECT tmp1.key, count(*)
FROM (SELECT x1.key1 AS key
FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
GROUP BY x1.key1) tmp1
JOIN (SELECT x2.key2 AS key
FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key2 = y2.key2)
GROUP BY x2.key2) tmp2
ON (tmp1.key = tmp2.key)
GROUP BY tmp1.key;
{\code}
In this query, join operations inside tmp1 and tmp2 will be converted to two
MapJoins. With Correlation Optimizer, aggregations in tmp1, tmp2, and join of
tmp1 and tmp2, and the last aggregation will be executed in the same
MapReduce job (Reduce side). Since this MapReduce job has two inputs, right
now, CommonJoinResolver cannot attach two MapJoins to the Map side of a
MapReduce job.
Another example:
{code:sql}
SELECT tmp1.key
FROM (SELECT x1.key2 AS key
FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
UNION ALL
SELECT x2.key2 AS key
FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key1 = y2.key1)) tmp1
{\code}
For this case, we will have three Map-only jobs (two for MapJoins and one for
Union). It will be good to use a single Map-only job to execute this query.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4905) In strict mode, predicate pushdown does not work on partition columns with statements using left/right join
caofangkun created HIVE-4905: Summary: In strict mode, predicate pushdown does not work on partition columns with statements using left/right join Key: HIVE-4905 URL: https://issues.apache.org/jira/browse/HIVE-4905 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.12.0 Reporter: caofangkun Assignee: caofangkun Priority: Minor set hive.mapred.mode=strict; drop table mpt3; create table mpt3 (s1 string , s2 string) partitioned by (dt string, time string); alter table mpt3 add partition (dt='1',time='2'); drop table mpt4; create table mpt4 (s1 string , s2 string) partitioned by (dt string, time string); alter table mpt4 add partition (dt='1',time='2'); Query One: works well explain select * from mpt3 a join mpt4 b on (a.s1 = b.s1) where a.dt='1' and a.time='2' and b.dt='1'; Query Two :failed hive (default) explain select a.* from mpt3 a right outer join mpt4 b on (a.s1 = b.s1) where a.dt='1' and a.time='2' and b.dt='1'; FAILED: SemanticException [Error 10041]: No partition predicate found for Alias a Table mpt3 Query Three: failed hive (default) explain select a.* from mpt3 a left outer join mpt4 b on (a.s1 = b.s1) where a.dt='1' and a.time='2' and b.dt='1'; FAILED: SemanticException [Error 10041]: No partition predicate found for Alias b Table mpt4 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3725) Add support for pulling HBase columns with prefixes
[
https://issues.apache.org/jira/browse/HIVE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715071#comment-13715071
]
Hive QA commented on HIVE-3725:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12593448/HIVE-3725.3.patch.txt
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 2647 tests executed
*Failed tests:*
{noformat}
org.apache.hcatalog.mapreduce.TestHCatInputFormat.testBadRecordHandlingPasses
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/124/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/124/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
Add support for pulling HBase columns with prefixes
---
Key: HIVE-3725
URL: https://issues.apache.org/jira/browse/HIVE-3725
Project: Hive
Issue Type: Improvement
Components: HBase Handler
Affects Versions: 0.9.0
Reporter: Swarnim Kulkarni
Assignee: Swarnim Kulkarni
Attachments: HIVE-3725.1.patch.txt, HIVE-3725.2.patch.txt,
HIVE-3725.3.patch.txt, HIVE-3725.patch.3.txt
Current HBase Hive integration supports reading many values from the same row
by specifying a column family. And specifying just the column family can pull
in all qualifiers within the family.
We should add in support to be able to specify a prefix for the qualifier and
all columns that start with the prefix would automatically get pulled in. A
wildcard support would be ideal.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
hive pull request: Branch 0.8 r2
GitHub user rajeshbnagaraju opened a pull request: https://github.com/apache/hive/pull/10 Branch 0.8 r2 You can merge this pull request into a Git repository by running: $ git pull https://github.com/apache/hive branch-0.8-r2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hive/pull/10.patch
[jira] [Created] (HIVE-4906) IO Exception for empty file not found if IP address used in core site instead of hostname
Kanwaljit Singh created HIVE-4906: - Summary: IO Exception for empty file not found if IP address used in core site instead of hostname Key: HIVE-4906 URL: https://issues.apache.org/jira/browse/HIVE-4906 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.10.0 Reporter: Kanwaljit Singh The hive server has core-site.xml with IP address for DFS namenode instead of the hostname. But it also has host ip mapping. We are getting this error. I think the host reverse mapping is not taking place while trying to access the folder on the HDFS. hive select httpurl, ptime from edr where ptime='2013-07-07 10:35:00' limit 10; Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator java.io.IOException: cannot find dir = hdfs://Namenode-VIP:9000/tmp/hive-admin/hive_2013-07-19_07-56-06_498_6777975312240173780/-mr-10002/1/emptyFile in pathToPartitionInfo: [hdfs://192.168.172.87:9000/tmp/hive-admin/hive_2013-07-19_07-56-06_498_6777975312240173780/-mr-10002/1] at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:298) at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:260) at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat$CombineHiveInputSplit.init(CombineHiveInputFormat.java:104) at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:407) at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:929) at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:921) at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:838) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:791) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Unknown Source) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:791) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:765) at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:447) at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:136) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4293) Predicates following UDTF operator are removed by PPD
[
https://issues.apache.org/jira/browse/HIVE-4293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715136#comment-13715136
]
Hive QA commented on HIVE-4293:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12593454/HIVE-4293.D9933.4.patch
{color:green}SUCCESS:{color} +1 2647 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/126/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/126/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
Predicates following UDTF operator are removed by PPD
-
Key: HIVE-4293
URL: https://issues.apache.org/jira/browse/HIVE-4293
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Assignee: Navis
Attachments: HIVE-4293.D9933.1.patch, HIVE-4293.D9933.2.patch,
HIVE-4293.D9933.3.patch, HIVE-4293.D9933.4.patch
For example,
{noformat}
explain SELECT value from (
select explode(array(key, value)) as (value) from (
select * FROM src WHERE key 200
) A
) B WHERE value 300
;
{noformat}
Makes plan like this, removing last predicates
{noformat}
TableScan
alias: src
Filter Operator
predicate:
expr: (key 200.0)
type: boolean
Select Operator
expressions:
expr: array(key,value)
type: arraystring
outputColumnNames: _col0
UDTF Operator
function name: explode
Select Operator
expressions:
expr: col
type: string
outputColumnNames: _col0
File Output Operator
compressed: false
GlobalTableId: 0
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3725) Add support for pulling HBase columns with prefixes
[ https://issues.apache.org/jira/browse/HIVE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715205#comment-13715205 ] Swarnim Kulkarni commented on HIVE-3725: Looking into the test failure, as per my understanding doesn't seem to be related to this change. Add support for pulling HBase columns with prefixes --- Key: HIVE-3725 URL: https://issues.apache.org/jira/browse/HIVE-3725 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.9.0 Reporter: Swarnim Kulkarni Assignee: Swarnim Kulkarni Attachments: HIVE-3725.1.patch.txt, HIVE-3725.2.patch.txt, HIVE-3725.3.patch.txt, HIVE-3725.patch.3.txt Current HBase Hive integration supports reading many values from the same row by specifying a column family. And specifying just the column family can pull in all qualifiers within the family. We should add in support to be able to specify a prefix for the qualifier and all columns that start with the prefix would automatically get pulled in. A wildcard support would be ideal. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-trunk-h0.21 - Build # 2213 - Still Failing
Changes for Build #2172 [hashutosh] HIVE-4781 : Adding new data files for tests. Missed in original commit. Changes for Build #2173 Changes for Build #2174 [navis] HIVE-2517 : Support group by on struct type (Ashutosh Chauhan via Navis) [hashutosh] HIVE-4406 : Missing / or /dbname in hs2 jdbc uri switches mode to embedded mode(Anandha Ranganathan via Ashutosh Chauhan) [hashutosh] HIVE-4430 : Semantic analysis fails in presence of certain literals in on clause (Kevin Wilfong via Ashutosh Chauhan) [hashutosh] HIVE-4757 : LazyTimestamp goes into irretrievable NULL mode once inited with NULL once (Gopal V via Ashutosh Chauhan) [hashutosh] HIVE-4785 : Implement isCaseSensitive for Hive JDBC driver (Robert Roland via Ashutosh Chauhan) Changes for Build #2175 [navis] HIVE-4436 : hive.exec.parallel=true doesn't work on hadoop-2 (Gopal V via Navis) Changes for Build #2176 Changes for Build #2177 [hashutosh] HIVE-4689 : For outerjoins, joinEmitInterval might make wrong result (Navis via Ashutosh Chauhan) [hashutosh] HIVE-3253 : ArrayIndexOutOfBounds exception for deeply nested structs (Thejas Nair via Ashutosh Chauhan) Changes for Build #2178 Changes for Build #2179 Changes for Build #2180 Changes for Build #2181 [hashutosh] HIVE-4089 : javax.jdo : jdo2-api dependency not in Maven Central (Navis via Ashutosh Chauhan) [ecapriolo] HIVE-4804 parallel order by fails for small datasets (Navis via egc) Submitted by: Navis Reviewed by:Edward Capriolo Changes for Build #2182 Changes for Build #2183 [hashutosh] HIVE-4814 : Adjust WebHCat e2e tests until HIVE4703 is addressed (Eugene Koifman via Ashutosh Chauhan) Changes for Build #2184 [hashutosh] HIVE-4811 : (Slightly) break up the SemanticAnalyzer monstrosity (Gunther Hagleitner via Ashutosh Chauhan) Changes for Build #2185 [hashutosh] HIVE-4251 : Indices can't be built on tables whose schema info comes from SerDe (Mark Wagner via Ashutosh Chauhan) [hashutosh] HIVE-4805 : Enhance coverage of package org.apache.hadoop.hive.ql.exec.errors (Ivan Veselovsky via Ashutosh Chauhan) Changes for Build #2186 [hashutosh] HIVE-4733 : HiveLockObjectData is not compared properly (Navis via Ashutosh Chauhan) [ecapriolo] HIVE-3475 INLINE UDTF does not convert types properly (Igor Kabiljo and Navis Ryu via egc) Submitted by: Navis Ryu and Igor Kabiljo Reviewed by:Edward Capriolo Changes for Build #2187 [hashutosh] HIVE-4802 : Fix url check for missing / or /db after hostname in jdb uri (Thejas Nair via Ashutosh Chauhan) Changes for Build #2188 [hashutosh] HIVE-4813 : Improve test coverage of package org.apache.hadoop.hive.ql.optimizer.pcr (Ivan Veselovsky via Ashutosh Chauhan) [hashutosh] HIVE-4580 : Change DDLTask to report errors using canonical error messages rather than http status codes (Eugene Koifman via Ashutosh Chauhan) [hashutosh] HIVE-4796 : Increase coverage of package org.apache.hadoop.hive.common.metrics (Ivan Veselovsky via Ashutosh Chauhan) [navis] HIVE-4812 : Logical explain plan (Gunther Hagleitner V via Navis) [hashutosh] HIVE-3810 : HiveHistory.log need to replace \r with space before writing Entry.value to historyfile (Mark Grover via Ashutosh Chauhan) Changes for Build #2189 [hashutosh] HIVE-4810 [jira] Refactor exec package (Gunther Hagleitner via Ashutosh Chauhan) Summary: HIVE-4810 The exec package contains both operators and classes used to execute the job. Moving the latter into a sub package makes the package slightly more manageable and will make it easier to provide a tez-based implementation. Test Plan: Refactoring Reviewers: ashutoshc Reviewed By: ashutoshc Differential Revision: https://reviews.facebook.net/D11625 [hashutosh] HIVE-4829 : TestWebHCatE2e checkstyle violation causes all tests to fail (Eugene Koifman via Ashutosh Chauhan) [hashutosh] HIVE-4819 : Comments in CommonJoinOperator for aliasTag is not valid (Navis via Ashutosh Chauhan) Changes for Build #2190 [hashutosh] HIVE-4807 : Hive metastore hangs (Sarvesh Sakalanaga via Ashutosh Chauhan) [hashutosh] HIVE-4833 : Fix eclipse template classpath to include the correct jdo lib (Yin Huai via Ashutosh Chauhan) [hashutosh] HIVE-4830 : Test clientnegative/nested_complex_neg.q got broken due to 4580 (Vikram Dixit via Ashutosh Chauhan) Changes for Build #2191 [hashutosh] HIVE-3691 : TestDynamicSerDe failed with IBM JDK (Bing Li Renata Ghisloti via Ashutosh Chauhan) Changes for Build #2192 Changes for Build #2193 Changes for Build #2194 Changes for Build #2195 [hashutosh] HIVE-4840 : Fix eclipse template classpath to include the BoneCP lib (Yin Huai via Ashutosh Chauhan) Changes for Build #2196 [navis] HIVE-4290 : Build profiles: Partial builds for quicker dev (Gunther Hagleitner via Navis) [navis] HIVE-4658 : Make KW_OUTER optional in outer joins (Edward Capriolo via Navis) Changes for Build #2197 Changes for Build #2198 Changes for Build #2199 [hashutosh] HIVE-4852 : -Dbuild.profile=core
Re: [ANNOUNCE] New Hive Committer - Gunther Hagleitner
Congrats Gunther! Great news! On Jul 21, 2013, at 8:52 PM, Vaibhav Gumashta wrote: Congratulations Gunther! On Sun, Jul 21, 2013 at 8:36 PM, David Schorow dscho...@hortonworks.com wrote: Cool. Congratulations Gunther. This is great news. David On Jul 21, 2013, at 9:52 AM, Yin Huai wrote: Congratulations Gunther!!! -- Forwarded message -- From: Yin Huai huaiyin@gmail.com Date: Sun, Jul 21, 2013 at 9:51 AM Subject: Fwd: [ANNOUNCE] New Hive Committer - Gunther Hagleitner To: yh...@hortonworks.com -- Forwarded message -- From: Carl Steinbach c...@apache.org Date: Sun, Jul 21, 2013 at 1:00 AM Subject: [ANNOUNCE] New Hive Committer - Gunther Hagleitner To: u...@hive.apache.org u...@hive.apache.org, dev@hive.apache.org, gunt...@apache.org The Apache Hive PMC has voted to make Gunther Hagleitner a committer on the Apache Hive project. Congratulations Gunther! Carl
[jira] [Updated] (HIVE-4907) Allow additional tests cases to be specified with -Dtestcase
[ https://issues.apache.org/jira/browse/HIVE-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-4907: --- Issue Type: Improvement (was: Bug) Allow additional tests cases to be specified with -Dtestcase Key: HIVE-4907 URL: https://issues.apache.org/jira/browse/HIVE-4907 Project: Hive Issue Type: Improvement Reporter: Brock Noland Assignee: Brock Noland Currently we only allow a single tests case to be specified with -Dtestcase. It'd be ideal if we could add on additional test cases as this would allow us to batch the unit tests in ptest2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4904) A little more CP crossing RS boundaries
[ https://issues.apache.org/jira/browse/HIVE-4904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715335#comment-13715335 ] Yin Huai commented on HIVE-4904: Hi Navis, can you attach an example at here? I am not sure I understand the problem. An example is helpful. thanks:) A little more CP crossing RS boundaries --- Key: HIVE-4904 URL: https://issues.apache.org/jira/browse/HIVE-4904 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Currently, CP context cannot be propagated over RS except for JOIN/EXT. A little more CP is possible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4827) Merge a Map-only job to its following MapReduce job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yin Huai updated HIVE-4827:
---
Status: Open (was: Patch Available)
Merge a Map-only job to its following MapReduce job with multiple inputs
Key: HIVE-4827
URL: https://issues.apache.org/jira/browse/HIVE-4827
Project: Hive
Issue Type: Improvement
Components: Query Processor
Affects Versions: 0.12.0
Reporter: Yin Huai
Assignee: Yin Huai
Attachments: HIVE-4827.1.patch, HIVE-4827.2.patch
When hive.optimize.mapjoin.mapreduce is on, CommonJoinResolver can attach a
Map-only job (MapJoin) to its following MapReduce job. But this merge only
happens when the MapReduce job has a single input. With Correlation Optimizer
(HIVE-2206), it is possible that the MapReduce job can have multiple inputs
(for multiple operation paths). It is desired to improve CommonJoinResolver
to merge a Map-only job to the corresponding Map task of the MapReduce job.
Example:
{code:sql}
set hive.optimize.correlation=true;
set hive.auto.convert.join=true;
set hive.optimize.mapjoin.mapreduce=true;
SELECT tmp1.key, count(*)
FROM (SELECT x1.key1 AS key
FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
GROUP BY x1.key1) tmp1
JOIN (SELECT x2.key2 AS key
FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key2 = y2.key2)
GROUP BY x2.key2) tmp2
ON (tmp1.key = tmp2.key)
GROUP BY tmp1.key;
{\code}
In this query, join operations inside tmp1 and tmp2 will be converted to two
MapJoins. With Correlation Optimizer, aggregations in tmp1, tmp2, and join of
tmp1 and tmp2, and the last aggregation will be executed in the same
MapReduce job (Reduce side). Since this MapReduce job has two inputs, right
now, CommonJoinResolver cannot attach two MapJoins to the Map side of a
MapReduce job.
Another example:
{code:sql}
SELECT tmp1.key
FROM (SELECT x1.key2 AS key
FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
UNION ALL
SELECT x2.key2 AS key
FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key1 = y2.key1)) tmp1
{\code}
For this case, we will have three Map-only jobs (two for MapJoins and one for
Union). It will be good to use a single Map-only job to execute this query.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4907) Allow additional tests cases to be specified with -Dtestcase
Brock Noland created HIVE-4907: -- Summary: Allow additional tests cases to be specified with -Dtestcase Key: HIVE-4907 URL: https://issues.apache.org/jira/browse/HIVE-4907 Project: Hive Issue Type: Bug Reporter: Brock Noland Assignee: Brock Noland Currently we only allow a single tests case to be specified with -Dtestcase. It'd be ideal if we could add on additional test cases as this would allow us to batch the unit tests in ptest2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4907) Allow additional tests cases to be specified with -Dtestcase
[ https://issues.apache.org/jira/browse/HIVE-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-4907: --- Component/s: Testing Infrastructure Allow additional tests cases to be specified with -Dtestcase Key: HIVE-4907 URL: https://issues.apache.org/jira/browse/HIVE-4907 Project: Hive Issue Type: Improvement Components: Testing Infrastructure Reporter: Brock Noland Assignee: Brock Noland Currently we only allow a single tests case to be specified with -Dtestcase. It'd be ideal if we could add on additional test cases as this would allow us to batch the unit tests in ptest2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4825) Separate MapredWork into MapWork and ReduceWork
[ https://issues.apache.org/jira/browse/HIVE-4825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715307#comment-13715307 ] Ashutosh Chauhan commented on HIVE-4825: As far as I can see, there is nothing Tez specific in this patch. Its welcome change to improve state of the art of Hive trunk. [~appodictic] are you -1ing this patch for trunk? Separate MapredWork into MapWork and ReduceWork --- Key: HIVE-4825 URL: https://issues.apache.org/jira/browse/HIVE-4825 Project: Hive Issue Type: Improvement Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Priority: Minor Attachments: HIVE-4825.1.patch, HIVE-4825.2.code.patch, HIVE-4825.2.testfiles.patch, HIVE-4825.3.testfiles.patch Right now all the information needed to run an MR job is captured in MapredWork. This class has aliases, tagging info, table descriptors etc. For Tez and MRR it will be useful to break this into map and reduce specific pieces. The separation is natural and I think has value in itself, it makes the code easier to understand. However, it will also allow us to reuse these abstractions in Tez where you'll have a graph of these instead of just 1M and 0-1R. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3725) Add support for pulling HBase columns with prefixes
[ https://issues.apache.org/jira/browse/HIVE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715277#comment-13715277 ] Brock Noland commented on HIVE-3725: This looks pretty good to me. I am just going to run the precommit tests again. If you have a chance I think we should add a javadoc to HBASE_COLUMNS_REGEX_MATCHING saying we don't so full regex matching at present. Not a huge deal I could add that on commit as well. Add support for pulling HBase columns with prefixes --- Key: HIVE-3725 URL: https://issues.apache.org/jira/browse/HIVE-3725 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.9.0 Reporter: Swarnim Kulkarni Assignee: Swarnim Kulkarni Attachments: HIVE-3725.1.patch.txt, HIVE-3725.2.patch.txt, HIVE-3725.3.patch.txt, HIVE-3725.patch.3.txt Current HBase Hive integration supports reading many values from the same row by specifying a column family. And specifying just the column family can pull in all qualifiers within the family. We should add in support to be able to specify a prefix for the qualifier and all columns that start with the prefix would automatically get pulled in. A wildcard support would be ideal. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3725) Add support for pulling HBase columns with prefixes
[
https://issues.apache.org/jira/browse/HIVE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715368#comment-13715368
]
Hive QA commented on HIVE-3725:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12593448/HIVE-3725.3.patch.txt
{color:green}SUCCESS:{color} +1 2647 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/127/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/127/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
Add support for pulling HBase columns with prefixes
---
Key: HIVE-3725
URL: https://issues.apache.org/jira/browse/HIVE-3725
Project: Hive
Issue Type: Improvement
Components: HBase Handler
Affects Versions: 0.9.0
Reporter: Swarnim Kulkarni
Assignee: Swarnim Kulkarni
Attachments: HIVE-3725.1.patch.txt, HIVE-3725.2.patch.txt,
HIVE-3725.3.patch.txt, HIVE-3725.patch.3.txt
Current HBase Hive integration supports reading many values from the same row
by specifying a column family. And specifying just the column family can pull
in all qualifiers within the family.
We should add in support to be able to specify a prefix for the qualifier and
all columns that start with the prefix would automatically get pulled in. A
wildcard support would be ideal.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3725) Add support for pulling HBase columns with prefixes
[ https://issues.apache.org/jira/browse/HIVE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swarnim Kulkarni updated HIVE-3725: --- Attachment: HIVE-3725.4.patch.txt Add support for pulling HBase columns with prefixes --- Key: HIVE-3725 URL: https://issues.apache.org/jira/browse/HIVE-3725 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.9.0 Reporter: Swarnim Kulkarni Assignee: Swarnim Kulkarni Attachments: HIVE-3725.1.patch.txt, HIVE-3725.2.patch.txt, HIVE-3725.3.patch.txt, HIVE-3725.4.patch.txt, HIVE-3725.patch.3.txt Current HBase Hive integration supports reading many values from the same row by specifying a column family. And specifying just the column family can pull in all qualifiers within the family. We should add in support to be able to specify a prefix for the qualifier and all columns that start with the prefix would automatically get pulled in. A wildcard support would be ideal. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request 9276: Add support for pulling HBase columns with prefixes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/9276/#review23644 --- Ship it! Ship It! - Brock Noland On July 22, 2013, 1:20 a.m., Swarnim Kulkarni wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/9276/ --- (Updated July 22, 2013, 1:20 a.m.) Review request for hive. Bugs: HIVE-3725 https://issues.apache.org/jira/browse/HIVE-3725 Repository: hive-git Description --- Added support for pulling hbase columns just by providing prefixes and a wildcard. So a query now could look something like this: CREATE EXTERNAL TABLE hive_hbase_test ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe' STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES (hbase.columns.mapping = :key,fam1:col*) TBLPROPERTIES (hbase.table.name = TEST_HBASE_TABLE); This would pull in all columns under column family fam1 which start with col. This gives a little more flexibility over pull all columns format. Diffs - hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 65c81bf hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java b550f45 hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 01938a7 hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseCellMap.java a8ba9d9 hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java 10a9207 hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java e821282 Diff: https://reviews.apache.org/r/9276/diff/ Testing --- Added unit tests to demonstrate the new functionality. Also made sure that all existing unit tests passed. Thanks, Swarnim Kulkarni
[jira] [Commented] (HIVE-3725) Add support for pulling HBase columns with prefixes
[ https://issues.apache.org/jira/browse/HIVE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715385#comment-13715385 ] Swarnim Kulkarni commented on HIVE-3725: Added the javadoc comment. Add support for pulling HBase columns with prefixes --- Key: HIVE-3725 URL: https://issues.apache.org/jira/browse/HIVE-3725 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.9.0 Reporter: Swarnim Kulkarni Assignee: Swarnim Kulkarni Attachments: HIVE-3725.1.patch.txt, HIVE-3725.2.patch.txt, HIVE-3725.3.patch.txt, HIVE-3725.4.patch.txt, HIVE-3725.patch.3.txt Current HBase Hive integration supports reading many values from the same row by specifying a column family. And specifying just the column family can pull in all qualifiers within the family. We should add in support to be able to specify a prefix for the qualifier and all columns that start with the prefix would automatically get pulled in. A wildcard support would be ideal. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4827) Merge a Map-only job to its following MapReduce job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715411#comment-13715411
]
Yin Huai commented on HIVE-4827:
Cannot reproduce the failed test case
(TestMinimrCliDriver.testCliDriver_bucketmapjoin6) in my laptop. I will use
another machine to test it.
Merge a Map-only job to its following MapReduce job with multiple inputs
Key: HIVE-4827
URL: https://issues.apache.org/jira/browse/HIVE-4827
Project: Hive
Issue Type: Improvement
Components: Query Processor
Affects Versions: 0.12.0
Reporter: Yin Huai
Assignee: Yin Huai
Attachments: HIVE-4827.1.patch, HIVE-4827.2.patch
When hive.optimize.mapjoin.mapreduce is on, CommonJoinResolver can attach a
Map-only job (MapJoin) to its following MapReduce job. But this merge only
happens when the MapReduce job has a single input. With Correlation Optimizer
(HIVE-2206), it is possible that the MapReduce job can have multiple inputs
(for multiple operation paths). It is desired to improve CommonJoinResolver
to merge a Map-only job to the corresponding Map task of the MapReduce job.
Example:
{code:sql}
set hive.optimize.correlation=true;
set hive.auto.convert.join=true;
set hive.optimize.mapjoin.mapreduce=true;
SELECT tmp1.key, count(*)
FROM (SELECT x1.key1 AS key
FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
GROUP BY x1.key1) tmp1
JOIN (SELECT x2.key2 AS key
FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key2 = y2.key2)
GROUP BY x2.key2) tmp2
ON (tmp1.key = tmp2.key)
GROUP BY tmp1.key;
{\code}
In this query, join operations inside tmp1 and tmp2 will be converted to two
MapJoins. With Correlation Optimizer, aggregations in tmp1, tmp2, and join of
tmp1 and tmp2, and the last aggregation will be executed in the same
MapReduce job (Reduce side). Since this MapReduce job has two inputs, right
now, CommonJoinResolver cannot attach two MapJoins to the Map side of a
MapReduce job.
Another example:
{code:sql}
SELECT tmp1.key
FROM (SELECT x1.key2 AS key
FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
UNION ALL
SELECT x2.key2 AS key
FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key1 = y2.key1)) tmp1
{\code}
For this case, we will have three Map-only jobs (two for MapJoins and one for
Union). It will be good to use a single Map-only job to execute this query.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4871) Apache builds fail with Target make-pom does not exist in the project hcatalog.
[ https://issues.apache.org/jira/browse/HIVE-4871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715412#comment-13715412 ] Eugene Koifman commented on HIVE-4871: -- I tested the patch with ant -Dmvn.publish.repo=local very-clean maven-build -logfile ant.log ant -Dmvn.publish.repo=local maven-publish -logfile ant.log and ant maven-publish -logfile ant.log with mvn.deploy.id=archiva.internal mvn.deploy.url=http://localhost:8080/archiva/repository/internal/ All runs clean. Apache builds fail with Target make-pom does not exist in the project hcatalog. --- Key: HIVE-4871 URL: https://issues.apache.org/jira/browse/HIVE-4871 Project: Hive Issue Type: Sub-task Components: HCatalog Reporter: Eugene Koifman Assignee: Eugene Koifman Fix For: 0.12.0 Attachments: HIVE-4871.patch Original Estimate: 168h Remaining Estimate: 168h For example, https://builds.apache.org/job/Hive-trunk-h0.21/2192/console. All unit tests pass, but deployment of build artifacts fails. HIVE-4387 provided a bandaid for 0.11. Need to figure out long term fix for this for 0.12. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4851) Fix flaky tests
[ https://issues.apache.org/jira/browse/HIVE-4851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715407#comment-13715407 ] Yin Huai commented on HIVE-4851: Hi Brock, have you seen TestMinimrCliDriver.testCliDriver_bucketmapjoin6 failed randomly? The precommit test of my 4827 patch (https://builds.apache.org/job/PreCommit-HIVE-Build/123/) failed because of this case. But I cannot reproduce it in my laptop. I will also keep an eye on it. Fix flaky tests --- Key: HIVE-4851 URL: https://issues.apache.org/jira/browse/HIVE-4851 Project: Hive Issue Type: Bug Reporter: Brock Noland Assignee: Brock Noland I see the following tests fail quite often: * TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_hadoop20 * TestOrcHCatLoader.testReadDataBasic * TestMinimrCliDriver.testCliDriver_bucketmpjoin6 * TestNotificationListener.testAMQListener This one is less often, but still fails randomly: * TestMinimrCliDriver.testCliDriver_bucket4 * TestHCatHiveCompatibility.testUnpartedReadWrite * TestHCatLoader.testReadPartitionedBasic * TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [ANNOUNCE] New Hive Committer - Gunther Hagleitner
Congrats Gunther! On Mon, Jul 22, 2013 at 8:24 AM, Vikram Dixit vik...@hortonworks.comwrote: Congrats Gunther! Great news! On Jul 21, 2013, at 8:52 PM, Vaibhav Gumashta wrote: Congratulations Gunther! On Sun, Jul 21, 2013 at 8:36 PM, David Schorow dscho...@hortonworks.com wrote: Cool. Congratulations Gunther. This is great news. David On Jul 21, 2013, at 9:52 AM, Yin Huai wrote: Congratulations Gunther!!! -- Forwarded message -- From: Yin Huai huaiyin@gmail.com Date: Sun, Jul 21, 2013 at 9:51 AM Subject: Fwd: [ANNOUNCE] New Hive Committer - Gunther Hagleitner To: yh...@hortonworks.com -- Forwarded message -- From: Carl Steinbach c...@apache.org Date: Sun, Jul 21, 2013 at 1:00 AM Subject: [ANNOUNCE] New Hive Committer - Gunther Hagleitner To: u...@hive.apache.org u...@hive.apache.org, dev@hive.apache.org, gunt...@apache.org The Apache Hive PMC has voted to make Gunther Hagleitner a committer on the Apache Hive project. Congratulations Gunther! Carl
[jira] [Commented] (HIVE-4827) Merge a Map-only job to its following MapReduce job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715424#comment-13715424
]
Brock Noland commented on HIVE-4827:
Yeah that test is flaky. I just submitted another run of the precommit.
Merge a Map-only job to its following MapReduce job with multiple inputs
Key: HIVE-4827
URL: https://issues.apache.org/jira/browse/HIVE-4827
Project: Hive
Issue Type: Improvement
Components: Query Processor
Affects Versions: 0.12.0
Reporter: Yin Huai
Assignee: Yin Huai
Attachments: HIVE-4827.1.patch, HIVE-4827.2.patch
When hive.optimize.mapjoin.mapreduce is on, CommonJoinResolver can attach a
Map-only job (MapJoin) to its following MapReduce job. But this merge only
happens when the MapReduce job has a single input. With Correlation Optimizer
(HIVE-2206), it is possible that the MapReduce job can have multiple inputs
(for multiple operation paths). It is desired to improve CommonJoinResolver
to merge a Map-only job to the corresponding Map task of the MapReduce job.
Example:
{code:sql}
set hive.optimize.correlation=true;
set hive.auto.convert.join=true;
set hive.optimize.mapjoin.mapreduce=true;
SELECT tmp1.key, count(*)
FROM (SELECT x1.key1 AS key
FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
GROUP BY x1.key1) tmp1
JOIN (SELECT x2.key2 AS key
FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key2 = y2.key2)
GROUP BY x2.key2) tmp2
ON (tmp1.key = tmp2.key)
GROUP BY tmp1.key;
{\code}
In this query, join operations inside tmp1 and tmp2 will be converted to two
MapJoins. With Correlation Optimizer, aggregations in tmp1, tmp2, and join of
tmp1 and tmp2, and the last aggregation will be executed in the same
MapReduce job (Reduce side). Since this MapReduce job has two inputs, right
now, CommonJoinResolver cannot attach two MapJoins to the Map side of a
MapReduce job.
Another example:
{code:sql}
SELECT tmp1.key
FROM (SELECT x1.key2 AS key
FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
UNION ALL
SELECT x2.key2 AS key
FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key1 = y2.key1)) tmp1
{\code}
For this case, we will have three Map-only jobs (two for MapJoins and one for
Union). It will be good to use a single Map-only job to execute this query.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4677) [HCatalog] WebHCat e2e tests fail on Hadoop 2
[
https://issues.apache.org/jira/browse/HIVE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-4677:
-
Component/s: HCatalog
Fix Version/s: 0.12.0
Assignee: Daniel Dai
[HCatalog] WebHCat e2e tests fail on Hadoop 2
-
Key: HIVE-4677
URL: https://issues.apache.org/jira/browse/HIVE-4677
Project: Hive
Issue Type: Bug
Components: HCatalog
Reporter: Daniel Dai
Assignee: Daniel Dai
Fix For: 0.12.0
Attachments: HIVE-4677-1.patch
curl
http://hor5n26.gq1.ygridcore.net:50111/templeton/v1/queue/job_1370377838831_0012?user.name=hrt_qa
{error:Does not contain a valid host:port authority: local}
Here is the detailed stacktrace from the server:
{code}
WARN | 04 Jun 2013 22:21:52,204 | org.apache.hadoop.conf.Configuration |
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
ERROR | 04 Jun 2013 22:21:52,204 |
org.apache.hcatalog.templeton.CatchallExceptionMapper | Does not contain a
valid host:port authority: local
java.lang.IllegalArgumentException: Does not contain a valid host:port
authority: local
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:211)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:163)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:152)
at
org.apache.hcatalog.templeton.TempletonDelegator.getAddress(TempletonDelegator.java:41)
at
org.apache.hcatalog.templeton.StatusDelegator.run(StatusDelegator.java:47)
at org.apache.hcatalog.templeton.Server.showQueueId(Server.java:688)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4586) [HCatalog] WebHCat should return 404 error for undefined resource
[ https://issues.apache.org/jira/browse/HIVE-4586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-4586: - Component/s: HCatalog Affects Version/s: 0.11.0 Fix Version/s: 0.12.0 Assignee: Daniel Dai [HCatalog] WebHCat should return 404 error for undefined resource - Key: HIVE-4586 URL: https://issues.apache.org/jira/browse/HIVE-4586 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.11.0 Reporter: Daniel Dai Assignee: Daniel Dai Fix For: 0.12.0 Attachments: HIVE-4586-1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4851) Fix flaky tests
[ https://issues.apache.org/jira/browse/HIVE-4851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715428#comment-13715428 ] Brock Noland commented on HIVE-4851: I have a theory that most of these are only problems with the CPU is busy. I cannot get them to reproduce on my workstation. Fix flaky tests --- Key: HIVE-4851 URL: https://issues.apache.org/jira/browse/HIVE-4851 Project: Hive Issue Type: Bug Reporter: Brock Noland Assignee: Brock Noland I see the following tests fail quite often: * TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_hadoop20 * TestOrcHCatLoader.testReadDataBasic * TestMinimrCliDriver.testCliDriver_bucketmpjoin6 * TestNotificationListener.testAMQListener This one is less often, but still fails randomly: * TestMinimrCliDriver.testCliDriver_bucket4 * TestHCatHiveCompatibility.testUnpartedReadWrite * TestHCatLoader.testReadPartitionedBasic * TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4586) [HCatalog] WebHCat should return 404 error for undefined resource
[ https://issues.apache.org/jira/browse/HIVE-4586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-4586: - Status: Patch Available (was: Open) [HCatalog] WebHCat should return 404 error for undefined resource - Key: HIVE-4586 URL: https://issues.apache.org/jira/browse/HIVE-4586 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.11.0 Reporter: Daniel Dai Assignee: Daniel Dai Fix For: 0.12.0 Attachments: HIVE-4586-1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4677) [HCatalog] WebHCat e2e tests fail on Hadoop 2
[
https://issues.apache.org/jira/browse/HIVE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-4677:
-
Status: Patch Available (was: Open)
[HCatalog] WebHCat e2e tests fail on Hadoop 2
-
Key: HIVE-4677
URL: https://issues.apache.org/jira/browse/HIVE-4677
Project: Hive
Issue Type: Bug
Components: HCatalog
Reporter: Daniel Dai
Assignee: Daniel Dai
Fix For: 0.12.0
Attachments: HIVE-4677-1.patch
curl
http://hor5n26.gq1.ygridcore.net:50111/templeton/v1/queue/job_1370377838831_0012?user.name=hrt_qa
{error:Does not contain a valid host:port authority: local}
Here is the detailed stacktrace from the server:
{code}
WARN | 04 Jun 2013 22:21:52,204 | org.apache.hadoop.conf.Configuration |
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
ERROR | 04 Jun 2013 22:21:52,204 |
org.apache.hcatalog.templeton.CatchallExceptionMapper | Does not contain a
valid host:port authority: local
java.lang.IllegalArgumentException: Does not contain a valid host:port
authority: local
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:211)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:163)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:152)
at
org.apache.hcatalog.templeton.TempletonDelegator.getAddress(TempletonDelegator.java:41)
at
org.apache.hcatalog.templeton.StatusDelegator.run(StatusDelegator.java:47)
at org.apache.hcatalog.templeton.Server.showQueueId(Server.java:688)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4586) [HCatalog] WebHCat should return 404 error for undefined resource
[ https://issues.apache.org/jira/browse/HIVE-4586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715432#comment-13715432 ] Eugene Koifman commented on HIVE-4586: -- Does this affect WebHCat e2e tests? (recorded results) [HCatalog] WebHCat should return 404 error for undefined resource - Key: HIVE-4586 URL: https://issues.apache.org/jira/browse/HIVE-4586 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.11.0 Reporter: Daniel Dai Assignee: Daniel Dai Fix For: 0.12.0 Attachments: HIVE-4586-1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4883) TestHadoop20SAuthBridge tests fail sometimes because of race condition
[ https://issues.apache.org/jira/browse/HIVE-4883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715418#comment-13715418 ] Brock Noland commented on HIVE-4883: +1 TestHadoop20SAuthBridge tests fail sometimes because of race condition -- Key: HIVE-4883 URL: https://issues.apache.org/jira/browse/HIVE-4883 Project: Hive Issue Type: Bug Components: Metastore, Tests Affects Versions: 0.11.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-4883.01.patch, HIVE-4883.02.patch, HIVE-4883.patch.dependson-HIVE-3255, HIVE-4883-reproduce.patch TestHadoop20SAuthBridge tests testSaslWithHiveMetaStore and testMetastoreProxyUser sometimes fail. I have seen this more often on mac and windows, but this can happen on linux as well. The problem is that metastore is started in a different thread and these unit tests actually rely on the metastore having initialized DelegationTokenSecretManager in HadoopThriftAuthBridge20S as part of the metastore startup (HiveMetaStore.startMetaStore ) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4225) HiveServer2 does not support SASL QOP
[ https://issues.apache.org/jira/browse/HIVE-4225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715430#comment-13715430 ] Brock Noland commented on HIVE-4225: That test might be flaky as reported over in HIVE-4883. I am going to commit that and then we'll what fails here. HiveServer2 does not support SASL QOP - Key: HIVE-4225 URL: https://issues.apache.org/jira/browse/HIVE-4225 Project: Hive Issue Type: Bug Components: HiveServer2, Shims Affects Versions: 0.11.0 Reporter: Chris Drome Assignee: Chris Drome Attachments: HIVE-4225-1.patch, HIVE-4225.D10959.1.patch, HIVE-4225.patch HiveServer2 implements Kerberos authentication through SASL framework, but does not support setting QOP. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4851) Fix flaky tests
[ https://issues.apache.org/jira/browse/HIVE-4851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715437#comment-13715437 ] Yin Huai commented on HIVE-4851: oops, I somehow missed it... In bucketmapjoin6, there are 10 buckets for every input table. So, we will have 10 mappers to do SMB join. Will the number of buckets be the problem? Fix flaky tests --- Key: HIVE-4851 URL: https://issues.apache.org/jira/browse/HIVE-4851 Project: Hive Issue Type: Bug Reporter: Brock Noland Assignee: Brock Noland I see the following tests fail quite often: * TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_hadoop20 * TestOrcHCatLoader.testReadDataBasic * TestMinimrCliDriver.testCliDriver_bucketmpjoin6 * TestNotificationListener.testAMQListener This one is less often, but still fails randomly: * TestMinimrCliDriver.testCliDriver_bucket4 * TestHCatHiveCompatibility.testUnpartedReadWrite * TestHCatLoader.testReadPartitionedBasic * TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4827) Merge a Map-only job to its following MapReduce job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715438#comment-13715438
]
Yin Huai commented on HIVE-4827:
Thanks!
Merge a Map-only job to its following MapReduce job with multiple inputs
Key: HIVE-4827
URL: https://issues.apache.org/jira/browse/HIVE-4827
Project: Hive
Issue Type: Improvement
Components: Query Processor
Affects Versions: 0.12.0
Reporter: Yin Huai
Assignee: Yin Huai
Attachments: HIVE-4827.1.patch, HIVE-4827.2.patch
When hive.optimize.mapjoin.mapreduce is on, CommonJoinResolver can attach a
Map-only job (MapJoin) to its following MapReduce job. But this merge only
happens when the MapReduce job has a single input. With Correlation Optimizer
(HIVE-2206), it is possible that the MapReduce job can have multiple inputs
(for multiple operation paths). It is desired to improve CommonJoinResolver
to merge a Map-only job to the corresponding Map task of the MapReduce job.
Example:
{code:sql}
set hive.optimize.correlation=true;
set hive.auto.convert.join=true;
set hive.optimize.mapjoin.mapreduce=true;
SELECT tmp1.key, count(*)
FROM (SELECT x1.key1 AS key
FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
GROUP BY x1.key1) tmp1
JOIN (SELECT x2.key2 AS key
FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key2 = y2.key2)
GROUP BY x2.key2) tmp2
ON (tmp1.key = tmp2.key)
GROUP BY tmp1.key;
{\code}
In this query, join operations inside tmp1 and tmp2 will be converted to two
MapJoins. With Correlation Optimizer, aggregations in tmp1, tmp2, and join of
tmp1 and tmp2, and the last aggregation will be executed in the same
MapReduce job (Reduce side). Since this MapReduce job has two inputs, right
now, CommonJoinResolver cannot attach two MapJoins to the Map side of a
MapReduce job.
Another example:
{code:sql}
SELECT tmp1.key
FROM (SELECT x1.key2 AS key
FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
UNION ALL
SELECT x2.key2 AS key
FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key1 = y2.key1)) tmp1
{\code}
For this case, we will have three Map-only jobs (two for MapJoins and one for
Union). It will be good to use a single Map-only job to execute this query.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4908) rename templeton to webhcat?
Eugene Koifman created HIVE-4908: Summary: rename templeton to webhcat? Key: HIVE-4908 URL: https://issues.apache.org/jira/browse/HIVE-4908 Project: Hive Issue Type: Sub-task Components: HCatalog Affects Versions: 0.11.0 Reporter: Eugene Koifman Fix For: 0.12.0 Should we do this since we are changing package name of all the classes? (Don't forget Templeton URL (should make it listen on old + new) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4851) Fix flaky tests
[ https://issues.apache.org/jira/browse/HIVE-4851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715419#comment-13715419 ] Brock Noland commented on HIVE-4851: Yes I have, it's in the list in the jira description :) After the stack_trace one I feel that bucketmapjoin6 is the most flaky. Fix flaky tests --- Key: HIVE-4851 URL: https://issues.apache.org/jira/browse/HIVE-4851 Project: Hive Issue Type: Bug Reporter: Brock Noland Assignee: Brock Noland I see the following tests fail quite often: * TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_hadoop20 * TestOrcHCatLoader.testReadDataBasic * TestMinimrCliDriver.testCliDriver_bucketmpjoin6 * TestNotificationListener.testAMQListener This one is less often, but still fails randomly: * TestMinimrCliDriver.testCliDriver_bucket4 * TestHCatHiveCompatibility.testUnpartedReadWrite * TestHCatLoader.testReadPartitionedBasic * TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4909) Vectorized ORC reader does not set isRepeating flag correctly when 1’s are present is the input stream.
[ https://issues.apache.org/jira/browse/HIVE-4909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarvesh Sakalanaga updated HIVE-4909: - Description: As the default value for nulls in Vectorization for int types is 1, and as non-null values can also be 1, the isRepeating logic should also check for IsNull flag when determining the isRepeating flag. was: As the default value for nulls in Vectorization for int types is 1, and as non-null values can also be 1, the isRepeating logic should also check for IsNull flag. Vectorized ORC reader does not set isRepeating flag correctly when 1’s are present is the input stream. --- Key: HIVE-4909 URL: https://issues.apache.org/jira/browse/HIVE-4909 Project: Hive Issue Type: Sub-task Reporter: Sarvesh Sakalanaga Assignee: Sarvesh Sakalanaga As the default value for nulls in Vectorization for int types is 1, and as non-null values can also be 1, the isRepeating logic should also check for IsNull flag when determining the isRepeating flag. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4909) Vectorized ORC reader does not set isRepeating flag correctly when 1’s are present is the input stream.
Sarvesh Sakalanaga created HIVE-4909: Summary: Vectorized ORC reader does not set isRepeating flag correctly when 1’s are present is the input stream. Key: HIVE-4909 URL: https://issues.apache.org/jira/browse/HIVE-4909 Project: Hive Issue Type: Sub-task Reporter: Sarvesh Sakalanaga Assignee: Sarvesh Sakalanaga As the default value for nulls in Vectorization for int types is 1, and as non-null values can also be 1, the isRepeating logic should also check for IsNull flag. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3725) Add support for pulling HBase columns with prefixes
[ https://issues.apache.org/jira/browse/HIVE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715406#comment-13715406 ] Brock Noland commented on HIVE-3725: +1 Add support for pulling HBase columns with prefixes --- Key: HIVE-3725 URL: https://issues.apache.org/jira/browse/HIVE-3725 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.9.0 Reporter: Swarnim Kulkarni Assignee: Swarnim Kulkarni Attachments: HIVE-3725.1.patch.txt, HIVE-3725.2.patch.txt, HIVE-3725.3.patch.txt, HIVE-3725.4.patch.txt, HIVE-3725.patch.3.txt Current HBase Hive integration supports reading many values from the same row by specifying a column family. And specifying just the column family can pull in all qualifiers within the family. We should add in support to be able to specify a prefix for the qualifier and all columns that start with the prefix would automatically get pulled in. A wildcard support would be ideal. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4909) Vectorized ORC reader does not set isRepeating flag correctly when 1’s are present is the input stream
[ https://issues.apache.org/jira/browse/HIVE-4909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarvesh Sakalanaga updated HIVE-4909: - Summary: Vectorized ORC reader does not set isRepeating flag correctly when 1’s are present is the input stream (was: Vectorized ORC reader does not set isRepeating flag correctly when 1’s are present is the input stream.) Vectorized ORC reader does not set isRepeating flag correctly when 1’s are present is the input stream -- Key: HIVE-4909 URL: https://issues.apache.org/jira/browse/HIVE-4909 Project: Hive Issue Type: Sub-task Reporter: Sarvesh Sakalanaga Assignee: Sarvesh Sakalanaga As the default value for nulls in Vectorization for int types is 1, and as non-null values can also be 1, the isRepeating logic should also check for IsNull flag when determining the isRepeating flag. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4907) Allow additional tests cases to be specified with -Dtestcase
[ https://issues.apache.org/jira/browse/HIVE-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-4907: --- Attachment: HIVE-4907.patch Attached is a backwards compatible patch which allows unit test batching. Allow additional tests cases to be specified with -Dtestcase Key: HIVE-4907 URL: https://issues.apache.org/jira/browse/HIVE-4907 Project: Hive Issue Type: Improvement Components: Testing Infrastructure Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-4907.patch Currently we only allow a single tests case to be specified with -Dtestcase. It'd be ideal if we could add on additional test cases as this would allow us to batch the unit tests in ptest2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4907) Allow additional tests cases to be specified with -Dtestcase
[ https://issues.apache.org/jira/browse/HIVE-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-4907: --- Status: Patch Available (was: Open) Allow additional tests cases to be specified with -Dtestcase Key: HIVE-4907 URL: https://issues.apache.org/jira/browse/HIVE-4907 Project: Hive Issue Type: Improvement Components: Testing Infrastructure Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-4907.patch Currently we only allow a single tests case to be specified with -Dtestcase. It'd be ideal if we could add on additional test cases as this would allow us to batch the unit tests in ptest2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4909) Vectorized ORC reader does not set isRepeating flag correctly when 1’s are present is the input stream
[ https://issues.apache.org/jira/browse/HIVE-4909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarvesh Sakalanaga updated HIVE-4909: - Status: Patch Available (was: Open) Vectorized ORC reader does not set isRepeating flag correctly when 1’s are present is the input stream -- Key: HIVE-4909 URL: https://issues.apache.org/jira/browse/HIVE-4909 Project: Hive Issue Type: Sub-task Reporter: Sarvesh Sakalanaga Assignee: Sarvesh Sakalanaga Attachments: Hive-4909.0.patch As the default value for nulls in Vectorization for int types is 1, and as non-null values can also be 1, the isRepeating logic should also check for IsNull flag when determining the isRepeating flag. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4611) SMB joins fail based on bigtable selection policy.
[ https://issues.apache.org/jira/browse/HIVE-4611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-4611: --- Status: Open (was: Patch Available) Nice patch! Since some tests failed I am going to remove the Patch Available state of this patch. Once those are addressed please mark it Patch Available again! SMB joins fail based on bigtable selection policy. -- Key: HIVE-4611 URL: https://issues.apache.org/jira/browse/HIVE-4611 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Fix For: 0.11.1 Attachments: HIVE-4611.2.patch, HIVE-4611.3.patch, HIVE-4611.patch The default setting for hive.auto.convert.sortmerge.join.bigtable.selection.policy will choose the big table as the one with largest average partition size. However, this can result in a query failing because this policy conflicts with the big table candidates chosen for outer joins. This policy should just be a tie breaker and not have the ultimate say in the choice of tables. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4875) hive config template is not parse-able due to angle brackets in description
[ https://issues.apache.org/jira/browse/HIVE-4875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-4875: --- Attachment: HIVE-4875-v0.patch Trivial patch, I don't think RB is necessary hive config template is not parse-able due to angle brackets in description --- Key: HIVE-4875 URL: https://issues.apache.org/jira/browse/HIVE-4875 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Trivial Attachments: HIVE-4875-v0.patch hive config template has angle brackets in description; if it is used as hive-site.xml config parsing fails -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4875) hive config template is not parse-able due to angle brackets in description
[ https://issues.apache.org/jira/browse/HIVE-4875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715457#comment-13715457 ] Sergey Shelukhin commented on HIVE-4875: [~ashutoshc] do you want to take a look? thanks hive config template is not parse-able due to angle brackets in description --- Key: HIVE-4875 URL: https://issues.apache.org/jira/browse/HIVE-4875 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Trivial Attachments: HIVE-4875-v0.patch hive config template has angle brackets in description; if it is used as hive-site.xml config parsing fails -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4876) Beeling help text do not contain -f and -e parameters
[
https://issues.apache.org/jira/browse/HIVE-4876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715456#comment-13715456
]
Brock Noland commented on HIVE-4876:
+1
Beeling help text do not contain -f and -e parameters
-
Key: HIVE-4876
URL: https://issues.apache.org/jira/browse/HIVE-4876
Project: Hive
Issue Type: Bug
Affects Versions: 0.11.0
Reporter: Jarek Jarcec Cecho
Assignee: Jarek Jarcec Cecho
Priority: Trivial
Attachments: bugHIVE-4876.patch, HIVE-4876.patch
I've noticed that beeline do have support for {{-e}} and {{-f}} parameters,
however those are not documented in the help text that can be retrieved by
calling beeline with parameter {{-h}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4900) Fix the mismatched column names in package.jdo
[ https://issues.apache.org/jira/browse/HIVE-4900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-4900: -- Attachment: HIVE-4900.2.patch Fix the mismatched column names in package.jdo -- Key: HIVE-4900 URL: https://issues.apache.org/jira/browse/HIVE-4900 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.10.0, 0.11.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Priority: Minor Fix For: 0.12.0 Attachments: HIVE-4900.1.patch, HIVE-4900.2.patch, HIVE-4900.patch There are several errors in DataNucleus O-R mapping file, package.jdo, which are not complained by the existing DN version. These errors may be subject to future DN complaint (as experienced in HIVE-3632 and HIVE-2084). However, it is still better if we fix these errors as it also creates some confusion in the community. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4909) Vectorized ORC reader does not set isRepeating flag correctly when 1’s are present is the input stream
[ https://issues.apache.org/jira/browse/HIVE-4909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarvesh Sakalanaga updated HIVE-4909: - Attachment: Hive-4909.0.patch Vectorized ORC reader does not set isRepeating flag correctly when 1’s are present is the input stream -- Key: HIVE-4909 URL: https://issues.apache.org/jira/browse/HIVE-4909 Project: Hive Issue Type: Sub-task Reporter: Sarvesh Sakalanaga Assignee: Sarvesh Sakalanaga Attachments: Hive-4909.0.patch As the default value for nulls in Vectorization for int types is 1, and as non-null values can also be 1, the isRepeating logic should also check for IsNull flag when determining the isRepeating flag. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [ANNOUNCE] New Hive Committer - Gunther Hagleitner
Congratulations! Thanks, +Vinod On Jul 21, 2013, at 1:00 AM, Carl Steinbach wrote: The Apache Hive PMC has voted to make Gunther Hagleitner a committer on the Apache Hive project. Congratulations Gunther! Carl
[jira] [Updated] (HIVE-4395) Support TFetchOrientation.FIRST for HiveServer2 FetchResults
[ https://issues.apache.org/jira/browse/HIVE-4395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-4395: --- Attachment: HIVE-4395.1.patch Re-uploading the exact patch as HIVE-4395.1.patch so the precommit tests run. Support TFetchOrientation.FIRST for HiveServer2 FetchResults Key: HIVE-4395 URL: https://issues.apache.org/jira/browse/HIVE-4395 Project: Hive Issue Type: Improvement Components: HiveServer2 Affects Versions: 0.11.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Attachments: HIVE-4395-1.patch, HIVE-4395.1.patch Currently HiveServer2 only support fetching next row (TFetchOrientation.NEXT). This ticket is to implement support for TFetchOrientation.FIRST that resets the fetch position at the begining of the resultset. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request 12790: HIVE-4892: PTest2 cleanup after merge
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/12790/ --- (Updated July 22, 2013, 6:23 p.m.) Review request for hive. Changes --- Removed and renamed the TEST-*.xml files we use for testing parsing of output files. This was causing issues with the build. Bugs: HIVE-4892 https://issues.apache.org/jira/browse/HIVE-4892 Repository: hive-git Description --- Fixes the following issues: README is out of date Need to limit the number of failed source directories we copy back from the slaves when looking for TEST-*.xml files we look at both the log directory (good) and the failed source directories (bad) therefore duplicating failures in jenkins report We need to process bad hosts in the finally block of PTest.run (HIVE-4882) Need a mechanism to clean the ivy and maven cache (HIVE-4882) PTest2 fails to publish a comment to a JIRA sometimes (HIVE-4889) Now that PTest2 is committed to the source tree it's copying in our TEST-SomeTest*.xml files Diffs (updated) - testutils/ptest2/README.md a483851 testutils/ptest2/src/main/java/org/apache/hive/ptest/api/client/PTestClient.java d6e9ce3 testutils/ptest2/src/main/java/org/apache/hive/ptest/api/request/TestStartRequest.java b41726a testutils/ptest2/src/main/java/org/apache/hive/ptest/api/server/TestExecutor.java 9b2125c testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/ExecutionPhase.java b5d7909 testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/HostExecutor.java 34a087c testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/JIRAService.java 09c6d66 testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/JUnitReportParser.java 50cce25 testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java f4f5076 testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/ReportingPhase.java 023c26b testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/QFileTestBatch.java 722312a testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/TestConfiguration.java fef2a3e testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/UnitTestBatch.java e911304 testutils/ptest2/src/main/resources/batch-exec.vm a35fcec testutils/ptest2/src/main/resources/source-prep.vm 7a92272 testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/TestExecutionPhase.java eec5560 testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/TestHostExecutor.java 6fc649a testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/TestReportParser.java 61b345d testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/TestReportingPhase.testExecute.approved.txt 19bc3e7 testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/TestScripts.java 0631996 testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/TestScripts.testBatch.approved.txt 9b4bb2f testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/TestScripts.testPrepGit.approved.txt 2decd3e testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/TestScripts.testPrepNone.approved.txt 7d8719d testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/TestScripts.testPrepSvn.approved.txt 0af317e testutils/ptest2/src/test/resources/SomeTest-failure.xml PRE-CREATION testutils/ptest2/src/test/resources/SomeTest-success.xml PRE-CREATION testutils/ptest2/src/test/resources/TEST-SomeTest-failure.xml d033cc4 testutils/ptest2/src/test/resources/test-outputs/SomeTest-truncated.xml PRE-CREATION testutils/ptest2/src/test/resources/test-outputs/TEST-SomeTest-truncated.xml 7e3fca4 testutils/ptest2/src/test/resources/test-outputs/TEST-index_auth.q-bucketcontex-ba31fb54-1d7f-4c70-a89d-477b7d155191-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml dcf4d4f testutils/ptest2/src/test/resources/test-outputs/TEST-skewjoin.q-ab8536a7-1b5c-45ed-ba29-14450f27db8b-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml c7d5e9a testutils/ptest2/src/test/resources/test-outputs/TEST-skewjoin_union_remove_1.q-6fa31776-d2b0-4e13-9761-11f750627ad1-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml fdf82c3 testutils/ptest2/src/test/resources/test-outputs/TEST-union_remove_9.q-acb9de8f-1b9c-4874-924c-b2107ca7b07c-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml f36 testutils/ptest2/src/test/resources/test-outputs/index_auth.q-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml PRE-CREATION testutils/ptest2/src/test/resources/test-outputs/index_auth.q-bucketcontex-ba31fb54-1d7f-4c70-a89d-477b7d155191-hive.log bf4e0a8 testutils/ptest2/src/test/resources/test-outputs/index_auth.q-hive.log PRE-CREATION
[jira] [Updated] (HIVE-4892) PTest2 cleanup after merge
[ https://issues.apache.org/jira/browse/HIVE-4892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-4892: --- Attachment: HIVE-4892.patch Removed and renamed the TEST-*.xml files we use for testing parsing of output files. This was causing issues with the build. These changed make the patch look much larger than it actually is. PTest2 cleanup after merge -- Key: HIVE-4892 URL: https://issues.apache.org/jira/browse/HIVE-4892 Project: Hive Issue Type: Bug Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-4892.patch, HIVE-4892.patch HIVE-4675 was merged but there are still a few minor issues we need to cleanup: * README is out of date * Need to limit the number of failed source directories we copy back from the slaves * when looking for TEST-*.xml files we look at both the log directory (good) and the failed source directories (bad) therefore duplicating failures in jenkins report * We need to process bad hosts in the finally block of PTest.run (HIVE-4882) * Need a mechanism to clean the ivy and maven cache (HIVE-4882) * PTest2 fails to publish a comment to a JIRA sometimes (HIVE-4889) * Now that PTest2 is committed to the source tree it's copying in our TEST-SomeTest*.xml files Test Properties: NO PRECOMMIT TESTS -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-4863) Fix parallel order by on hadoop2
[ https://issues.apache.org/jira/browse/HIVE-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland resolved HIVE-4863. Resolution: Fixed parallel orderby is passing after the build change: https://builds.apache.org/user/brock/my-views/view/hive/job/Hive-trunk-hadoop2-ptest/lastCompletedBuild/testReport/org.apache.hadoop.hive.cli/TestMinimrCliDriver/testCliDriver_parallel_orderby/ Fix parallel order by on hadoop2 Key: HIVE-4863 URL: https://issues.apache.org/jira/browse/HIVE-4863 Project: Hive Issue Type: Bug Reporter: Edward Capriolo Assignee: Brock Noland -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4910) Hadoop 2 archives broken
Vikram Dixit K created HIVE-4910: Summary: Hadoop 2 archives broken Key: HIVE-4910 URL: https://issues.apache.org/jira/browse/HIVE-4910 Project: Hive Issue Type: Bug Components: Query Processor, Tests Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Minor Fix For: 0.11.1 Hadoop 2 archive tests are broken. The issue stems from the fact that har uri construction does not really have a port in the URI when unit tests are run. This means that an invalid uri is constructed resulting in failures. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4910) Hadoop 2 archives broken
[ https://issues.apache.org/jira/browse/HIVE-4910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-4910: - Status: Patch Available (was: Open) Hadoop 2 archives broken Key: HIVE-4910 URL: https://issues.apache.org/jira/browse/HIVE-4910 Project: Hive Issue Type: Bug Components: Query Processor, Tests Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Minor Fix For: 0.11.1 Attachments: HIVE-4910.patch Hadoop 2 archive tests are broken. The issue stems from the fact that har uri construction does not really have a port in the URI when unit tests are run. This means that an invalid uri is constructed resulting in failures. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4910) Hadoop 2 archives broken
[ https://issues.apache.org/jira/browse/HIVE-4910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-4910: - Attachment: HIVE-4910.patch Patch addresses the test issue. Hadoop 2 archives broken Key: HIVE-4910 URL: https://issues.apache.org/jira/browse/HIVE-4910 Project: Hive Issue Type: Bug Components: Query Processor, Tests Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Minor Fix For: 0.11.1 Attachments: HIVE-4910.patch Hadoop 2 archive tests are broken. The issue stems from the fact that har uri construction does not really have a port in the URI when unit tests are run. This means that an invalid uri is constructed resulting in failures. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4910) Hadoop 2 archives broken
[ https://issues.apache.org/jira/browse/HIVE-4910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715515#comment-13715515 ] Brock Noland commented on HIVE-4910: Does this fix HIVE-3025? If so can you link and mark duplicate. Hadoop 2 archives broken Key: HIVE-4910 URL: https://issues.apache.org/jira/browse/HIVE-4910 Project: Hive Issue Type: Bug Components: Query Processor, Tests Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Minor Fix For: 0.11.1 Attachments: HIVE-4910.patch Hadoop 2 archive tests are broken. The issue stems from the fact that har uri construction does not really have a port in the URI when unit tests are run. This means that an invalid uri is constructed resulting in failures. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [ANNOUNCE] New Hive Committer - Gunther Hagleitner
Thanks everyone! This is awesome! ... alright, time for me to get back to work to hopefully prove worthy of the commit bit. Cheers, Gunther. On Mon, Jul 22, 2013 at 11:08 AM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: Congratulations! Thanks, +Vinod On Jul 21, 2013, at 1:00 AM, Carl Steinbach wrote: The Apache Hive PMC has voted to make Gunther Hagleitner a committer on the Apache Hive project. Congratulations Gunther! Carl
[jira] [Commented] (HIVE-4910) Hadoop 2 archives broken
[ https://issues.apache.org/jira/browse/HIVE-4910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715549#comment-13715549 ] Vikram Dixit K commented on HIVE-4910: -- This is related to archive failures but, the patch on that jira does not address this issue. It changes/adds hostname but this patch addresses port number related issue. Also, the title of that jira is too general. I could upload this patch there and make this a duplicate. Let me know your opinion. Thanks Vikram. -- Nothing better than when appreciated for hard work. -Mark Hadoop 2 archives broken Key: HIVE-4910 URL: https://issues.apache.org/jira/browse/HIVE-4910 Project: Hive Issue Type: Bug Components: Query Processor, Tests Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Minor Fix For: 0.11.1 Attachments: HIVE-4910.patch Hadoop 2 archive tests are broken. The issue stems from the fact that har uri construction does not really have a port in the URI when unit tests are run. This means that an invalid uri is constructed resulting in failures. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3725) Add support for pulling HBase columns with prefixes
[
https://issues.apache.org/jira/browse/HIVE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715548#comment-13715548
]
Hive QA commented on HIVE-3725:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12593553/HIVE-3725.4.patch.txt
{color:green}SUCCESS:{color} +1 2647 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/129/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/129/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
Add support for pulling HBase columns with prefixes
---
Key: HIVE-3725
URL: https://issues.apache.org/jira/browse/HIVE-3725
Project: Hive
Issue Type: Improvement
Components: HBase Handler
Affects Versions: 0.9.0
Reporter: Swarnim Kulkarni
Assignee: Swarnim Kulkarni
Attachments: HIVE-3725.1.patch.txt, HIVE-3725.2.patch.txt,
HIVE-3725.3.patch.txt, HIVE-3725.4.patch.txt, HIVE-3725.patch.3.txt
Current HBase Hive integration supports reading many values from the same row
by specifying a column family. And specifying just the column family can pull
in all qualifiers within the family.
We should add in support to be able to specify a prefix for the qualifier and
all columns that start with the prefix would automatically get pulled in. A
wildcard support would be ideal.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4909) Vectorized ORC reader does not set isRepeating flag correctly when 1’s are present is the input stream
[
https://issues.apache.org/jira/browse/HIVE-4909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715551#comment-13715551
]
Hive QA commented on HIVE-4909:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12593561/Hive-4909.0.patch
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/133/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/133/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests failed with: NonZeroExitCodeException: Command 'bash
/data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and
output '+ [[ -n '' ]]
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-133/source-prep.txt
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted
'hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java'
Reverted
'hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java'
Reverted
'hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseCellMap.java'
Reverted 'hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java'
Reverted
'hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java'
Reverted 'hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java'
++ awk '{print $2}'
++ egrep -v '^X|^Performing status on external'
++ svn status --no-ignore
+ rm -rf build hcatalog/build hcatalog/core/build
hcatalog/storage-handlers/hbase/build hcatalog/server-extensions/build
hcatalog/webhcat/svr/build hcatalog/webhcat/java-client/build
hcatalog/hcatalog-pig-adapter/build common/src/gen
+ svn update
Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1505774.
At revision 1505774.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0 to p2
+ exit 1
'
{noformat}
This message is automatically generated.
Vectorized ORC reader does not set isRepeating flag correctly when 1’s are
present is the input stream
--
Key: HIVE-4909
URL: https://issues.apache.org/jira/browse/HIVE-4909
Project: Hive
Issue Type: Sub-task
Reporter: Sarvesh Sakalanaga
Assignee: Sarvesh Sakalanaga
Attachments: Hive-4909.0.patch
As the default value for nulls in Vectorization for int types is 1, and as
non-null values can also be 1, the isRepeating logic should also check for
IsNull flag when determining the isRepeating flag.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Commented] (HIVE-4910) Hadoop 2 archives broken
Hi, Thanks for this patch! Great to see more hadoop 2 issues being resolved. It doesn't matter to me which Jira we take the issue forward with but if its related lets link the two or if this supersedes that issue lets link and close one. On Monday, July 22, 2013 at 2:16 PM, Vikram Dixit K (JIRA) wrote: [ https://issues.apache.org/jira/browse/HIVE-4910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715549#comment-13715549 ] Vikram Dixit K commented on HIVE-4910: -- This is related to archive failures but, the patch on that jira does not address this issue. It changes/adds hostname but this patch addresses port number related issue. Also, the title of that jira is too general. I could upload this patch there and make this a duplicate. Let me know your opinion. Thanks Vikram. -- Nothing better than when appreciated for hard work. -Mark Hadoop 2 archives broken Key: HIVE-4910 URL: https://issues.apache.org/jira/browse/HIVE-4910 Project: Hive Issue Type: Bug Components: Query Processor, Tests Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Minor Fix For: 0.11.1 Attachments: HIVE-4910.patch Hadoop 2 archive tests are broken. The issue stems from the fact that har uri construction does not really have a port in the URI when unit tests are run. This means that an invalid uri is constructed resulting in failures. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4836) make checkstyle ignore IntelliJ files and templeton e2e files
[ https://issues.apache.org/jira/browse/HIVE-4836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-4836: - Attachment: HIVE-4836.patch excluded files created by webhcat e2e at runtime make checkstyle ignore IntelliJ files and templeton e2e files - Key: HIVE-4836 URL: https://issues.apache.org/jira/browse/HIVE-4836 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.12.0 Reporter: Eugene Koifman Assignee: Eugene Koifman Fix For: 0.12.0 Attachments: HIVE-4836.patch hcatalog/build-support/ant/checkstyle.xml already has exclude name=src/test/e2e/hcatalog/testdist/**/ !-- Test run results -- exclude name=src/test/e2e/hcatalog/tar/**/ !-- Test build area -- exclude name=src/test/e2e/hcatalog/udfs/java/*.jar/ !-- Test build area -- exclude name=src/test/e2e/hcatalog/hcattests.tar/ !-- Test build artifact -- It needs to exclude the same under e2e/templeton/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4836) make checkstyle ignore IntelliJ files and templeton e2e files
[ https://issues.apache.org/jira/browse/HIVE-4836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-4836: - Status: Patch Available (was: Open) make checkstyle ignore IntelliJ files and templeton e2e files - Key: HIVE-4836 URL: https://issues.apache.org/jira/browse/HIVE-4836 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.12.0 Reporter: Eugene Koifman Assignee: Eugene Koifman Fix For: 0.12.0 Attachments: HIVE-4836.patch hcatalog/build-support/ant/checkstyle.xml already has exclude name=src/test/e2e/hcatalog/testdist/**/ !-- Test run results -- exclude name=src/test/e2e/hcatalog/tar/**/ !-- Test build area -- exclude name=src/test/e2e/hcatalog/udfs/java/*.jar/ !-- Test build area -- exclude name=src/test/e2e/hcatalog/hcattests.tar/ !-- Test build artifact -- It needs to exclude the same under e2e/templeton/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4911) Enable QOP configuration for Hive Server 2 thrift transport
[ https://issues.apache.org/jira/browse/HIVE-4911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arup Malakar updated HIVE-4911: --- Issue Type: New Feature (was: Bug) Enable QOP configuration for Hive Server 2 thrift transport --- Key: HIVE-4911 URL: https://issues.apache.org/jira/browse/HIVE-4911 Project: Hive Issue Type: New Feature Reporter: Arup Malakar The QoP for hive server 2 should be configurable to enable encryption. A new configuration should be exposed hive.server2.thrift.rpc.protection. This would give greater control configuring hive server 2 service. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4911) Enable QOP configuration for Hive Server 2 thrift transport
Arup Malakar created HIVE-4911: -- Summary: Enable QOP configuration for Hive Server 2 thrift transport Key: HIVE-4911 URL: https://issues.apache.org/jira/browse/HIVE-4911 Project: Hive Issue Type: Bug Reporter: Arup Malakar The QoP for hive server 2 should be configurable to enable encryption. A new configuration should be exposed hive.server2.thrift.rpc.protection. This would give greater control configuring hive server 2 service. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: VOTE: moving hive from forest to Apache CMS
+1 On Sun, Jul 21, 2013 at 10:53 AM, Edward Capriolo edlinuxg...@gmail.comwrote: http://hive.apache.org is generated by forest, a rather cumbersome and confusing way to run a website. Forest is difficult to maintain and publish updates with. As a nail in the coffin forest does not even work well with recent versions of java. This vote is to move the site to: Apache CMShttps://www.apache.org/dev/cms.html and away from forest. Brock Noland has offered to move the site, and I am offering to help him and look it over. Vote +1 if you support the move to Apache CMS. (This is the one case where cutting down a forest is a very good idea :) Edward
[jira] [Commented] (HIVE-2702) listPartitionsByFilter only supports string partitions for equals
[
https://issues.apache.org/jira/browse/HIVE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715607#comment-13715607
]
Phabricator commented on HIVE-2702:
---
ashutoshc has requested changes to the revision HIVE-2702 [jira]
listPartitionsByFilter only supports string partitions.
Some comments.
INLINE COMMENTS
metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java:161
Its better to say numeric type here, since we can also support byte and short
(both of which are valid hive types) as well.
metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java:262
As stated earlier, lets rename this method and variable as
doesOpSupportNumeric()
metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java:272
Why are you checking value instanceof Long here ? This parsing is not done via
grammar described in Filter.g
metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java:299
I presume this depends on DN upgrade, lets postpone it till than.
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:7699
HIVE-3059, I assume you want to say.
REVISION DETAIL
https://reviews.facebook.net/D11715
BRANCH
HIVE-2702
ARCANIST PROJECT
hive
To: JIRA, ashutoshc, sershe
listPartitionsByFilter only supports string partitions for equals
-
Key: HIVE-2702
URL: https://issues.apache.org/jira/browse/HIVE-2702
Project: Hive
Issue Type: Bug
Affects Versions: 0.8.1
Reporter: Aniket Mokashi
Assignee: Sergey Shelukhin
Fix For: 0.12.0
Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-2702.D2043.1.patch,
HIVE-2702.1.patch, HIVE-2702.D11715.1.patch, HIVE-2702.D11715.2.patch,
HIVE-2702-v0.patch
listPartitionsByFilter supports only non-string partitions. This is because
its explicitly specified in generateJDOFilterOverPartitions in
ExpressionTree.java.
//Can only support partitions whose types are string
if( ! table.getPartitionKeys().get(partitionColumnIndex).
getType().equals(org.apache.hadoop.hive.serde.Constants.STRING_TYPE_NAME) ) {
throw new MetaException
(Filtering is supported only on partition keys of type string);
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4912) selectStar and selStarNoCompute in SelectDesc are confusing
[ https://issues.apache.org/jira/browse/HIVE-4912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715633#comment-13715633 ] Yin Huai commented on HIVE-4912: Assign to me first. If anyone wants to work on it, feel free to take it. selectStar and selStarNoCompute in SelectDesc are confusing --- Key: HIVE-4912 URL: https://issues.apache.org/jira/browse/HIVE-4912 Project: Hive Issue Type: Improvement Reporter: Yin Huai In SelectDesc, we have two confusing booleans, selectStar and selStarNoCompute. From the code, selectStar is only really used in SimpleFetchOptimizer. selStarNoCompute is used in lots of places. The question is if selectStar and selStarNoCompute are different. If so, we need to comment the differences in the code and check if there is any place we used the wrong boolean. If not, we should remove one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4912) selectStar and selStarNoCompute in SelectDesc are confusing
Yin Huai created HIVE-4912: -- Summary: selectStar and selStarNoCompute in SelectDesc are confusing Key: HIVE-4912 URL: https://issues.apache.org/jira/browse/HIVE-4912 Project: Hive Issue Type: Improvement Reporter: Yin Huai In SelectDesc, we have two confusing booleans, selectStar and selStarNoCompute. From the code, selectStar is only really used in SimpleFetchOptimizer. selStarNoCompute is used in lots of places. The question is if selectStar and selStarNoCompute are different. If so, we need to comment the differences in the code and check if there is any place we used the wrong boolean. If not, we should remove one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4875) hive config template is not parse-able due to angle brackets in description
[ https://issues.apache.org/jira/browse/HIVE-4875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715636#comment-13715636 ] Ashutosh Chauhan commented on HIVE-4875: +1 hive config template is not parse-able due to angle brackets in description --- Key: HIVE-4875 URL: https://issues.apache.org/jira/browse/HIVE-4875 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Trivial Attachments: HIVE-4875-v0.patch hive config template has angle brackets in description; if it is used as hive-site.xml config parsing fails -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4912) selectStar and selStarNoCompute in SelectDesc are confusing
[ https://issues.apache.org/jira/browse/HIVE-4912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated HIVE-4912: --- Priority: Minor (was: Major) selectStar and selStarNoCompute in SelectDesc are confusing --- Key: HIVE-4912 URL: https://issues.apache.org/jira/browse/HIVE-4912 Project: Hive Issue Type: Improvement Reporter: Yin Huai Assignee: Yin Huai Priority: Minor In SelectDesc, we have two confusing booleans, selectStar and selStarNoCompute. From the code, selectStar is only really used in SimpleFetchOptimizer. selStarNoCompute is used in lots of places. The question is if selectStar and selStarNoCompute are different. If so, we need to comment the differences in the code and check if there is any place we used the wrong boolean. If not, we should remove one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4912) selectStar and selStarNoCompute in SelectDesc are confusing
[ https://issues.apache.org/jira/browse/HIVE-4912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated HIVE-4912: --- Assignee: Yin Huai selectStar and selStarNoCompute in SelectDesc are confusing --- Key: HIVE-4912 URL: https://issues.apache.org/jira/browse/HIVE-4912 Project: Hive Issue Type: Improvement Reporter: Yin Huai Assignee: Yin Huai In SelectDesc, we have two confusing booleans, selectStar and selStarNoCompute. From the code, selectStar is only really used in SimpleFetchOptimizer. selStarNoCompute is used in lots of places. The question is if selectStar and selStarNoCompute are different. If so, we need to comment the differences in the code and check if there is any place we used the wrong boolean. If not, we should remove one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2702) listPartitionsByFilter only supports string partitions for equals
[
https://issues.apache.org/jira/browse/HIVE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-2702:
---
Attachment: HIVE-2702.patch
attaching patch for hiveqa.
listPartitionsByFilter only supports string partitions for equals
-
Key: HIVE-2702
URL: https://issues.apache.org/jira/browse/HIVE-2702
Project: Hive
Issue Type: Bug
Affects Versions: 0.8.1
Reporter: Aniket Mokashi
Assignee: Sergey Shelukhin
Fix For: 0.12.0
Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-2702.D2043.1.patch,
HIVE-2702.1.patch, HIVE-2702.D11715.1.patch, HIVE-2702.D11715.2.patch,
HIVE-2702.patch, HIVE-2702-v0.patch
listPartitionsByFilter supports only non-string partitions. This is because
its explicitly specified in generateJDOFilterOverPartitions in
ExpressionTree.java.
//Can only support partitions whose types are string
if( ! table.getPartitionKeys().get(partitionColumnIndex).
getType().equals(org.apache.hadoop.hive.serde.Constants.STRING_TYPE_NAME) ) {
throw new MetaException
(Filtering is supported only on partition keys of type string);
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Review Request 12824: [HIVE-4911] Enable QOP configuration for Hive Server 2 thrift transport
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/12824/ --- Review request for hive. Bugs: HIVE-4911 https://issues.apache.org/jira/browse/HIVE-4911 Repository: hive-git Description --- The QoP for hive server 2 should be configurable to enable encryption. A new configuration should be exposed hive.server2.thrift.rpc.protection. This would give greater control configuring hive server 2 service. Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 11c31216495d0c4e454f2627af5c93a9f270b1fe data/conf/hive-site.xml 4e6ff16135833da1a4df12a12a6fe59ad4f870ba jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java 00f43511b478c687b7811fc8ad66af2b507a3626 service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java 1809e1b26ceee5de14a354a0e499aa8c0ab793bf service/src/java/org/apache/hive/service/auth/KerberosSaslHelper.java 379dafb8377aed55e74f0ae18407996bb9e1216f service/src/java/org/apache/hive/service/auth/SaslQOP.java PRE-CREATION shims/src/common-secure/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge20S.java 777226f8da0af2235d4294cd6a676fa8192c89e4 shims/src/common/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge.java 9b0ec0a75563b41339e6fc747556440fdf83e31e Diff: https://reviews.apache.org/r/12824/diff/ Testing --- Thanks, Arup Malakar
[jira] [Assigned] (HIVE-4911) Enable QOP configuration for Hive Server 2 thrift transport
[ https://issues.apache.org/jira/browse/HIVE-4911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arup Malakar reassigned HIVE-4911: -- Assignee: Arup Malakar Enable QOP configuration for Hive Server 2 thrift transport --- Key: HIVE-4911 URL: https://issues.apache.org/jira/browse/HIVE-4911 Project: Hive Issue Type: New Feature Reporter: Arup Malakar Assignee: Arup Malakar The QoP for hive server 2 should be configurable to enable encryption. A new configuration should be exposed hive.server2.thrift.rpc.protection. This would give greater control configuring hive server 2 service. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4911) Enable QOP configuration for Hive Server 2 thrift transport
[ https://issues.apache.org/jira/browse/HIVE-4911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arup Malakar updated HIVE-4911: --- Status: Patch Available (was: Open) Review: https://reviews.apache.org/r/12824/ Enable QOP configuration for Hive Server 2 thrift transport --- Key: HIVE-4911 URL: https://issues.apache.org/jira/browse/HIVE-4911 Project: Hive Issue Type: New Feature Reporter: Arup Malakar Assignee: Arup Malakar The QoP for hive server 2 should be configurable to enable encryption. A new configuration should be exposed hive.server2.thrift.rpc.protection. This would give greater control configuring hive server 2 service. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4911) Enable QOP configuration for Hive Server 2 thrift transport
[ https://issues.apache.org/jira/browse/HIVE-4911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arup Malakar updated HIVE-4911: --- Attachment: HIVE-4911-trunk-0.patch Enable QOP configuration for Hive Server 2 thrift transport --- Key: HIVE-4911 URL: https://issues.apache.org/jira/browse/HIVE-4911 Project: Hive Issue Type: New Feature Reporter: Arup Malakar Assignee: Arup Malakar Attachments: HIVE-4911-trunk-0.patch The QoP for hive server 2 should be configurable to enable encryption. A new configuration should be exposed hive.server2.thrift.rpc.protection. This would give greater control configuring hive server 2 service. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: VOTE: moving hive from forest to Apache CMS
+1 Thanks, Gunther. On Mon, Jul 22, 2013 at 1:25 PM, Owen O'Malley omal...@apache.org wrote: +1 On Sun, Jul 21, 2013 at 10:53 AM, Edward Capriolo edlinuxg...@gmail.com wrote: http://hive.apache.org is generated by forest, a rather cumbersome and confusing way to run a website. Forest is difficult to maintain and publish updates with. As a nail in the coffin forest does not even work well with recent versions of java. This vote is to move the site to: Apache CMShttps://www.apache.org/dev/cms.html and away from forest. Brock Noland has offered to move the site, and I am offering to help him and look it over. Vote +1 if you support the move to Apache CMS. (This is the one case where cutting down a forest is a very good idea :) Edward
Re: VOTE: moving hive from forest to Apache CMS
+1 (non binding) On Mon, Jul 22, 2013 at 2:01 PM, Gunther Hagleitner ghagleit...@hortonworks.com wrote: +1 Thanks, Gunther. On Mon, Jul 22, 2013 at 1:25 PM, Owen O'Malley omal...@apache.org wrote: +1 On Sun, Jul 21, 2013 at 10:53 AM, Edward Capriolo edlinuxg...@gmail.com wrote: http://hive.apache.org is generated by forest, a rather cumbersome and confusing way to run a website. Forest is difficult to maintain and publish updates with. As a nail in the coffin forest does not even work well with recent versions of java. This vote is to move the site to: Apache CMShttps://www.apache.org/dev/cms.html and away from forest. Brock Noland has offered to move the site, and I am offering to help him and look it over. Vote +1 if you support the move to Apache CMS. (This is the one case where cutting down a forest is a very good idea :) Edward
[jira] [Updated] (HIVE-4831) QTestUtil based test exiting abnormally on windows fails startup of other QTestUtil tests
[ https://issues.apache.org/jira/browse/HIVE-4831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-4831: Attachment: HIVE-4831.2.patch [~brocknoland] Thanks for your feedback. HIVE-4831.2.patch - I could not use Guava Files.createTempDir as it does not take a baseDir argument, so I added such a function in Utilities.java . I will mark it patch available after some more testing. QTestUtil based test exiting abnormally on windows fails startup of other QTestUtil tests - Key: HIVE-4831 URL: https://issues.apache.org/jira/browse/HIVE-4831 Project: Hive Issue Type: Bug Components: Testing Infrastructure Affects Versions: 0.11.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-4831.1.patch, HIVE-4831.2.patch QTestUtil tests start mini zookeeper cluster. If it exits abnormally (eg timeout), it fails to stop the zookeeper mini cluster. On Windows when the process is still running the files can't be deleted, and as a result the new zookeeper cluster started by a new QFileUtil based test case fails to start. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request 12690: HIVE-4870: Explain Extended to show partition info for Fetch Task
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/12690/#review23654 --- Ship it! Ship It! - Ashutosh Chauhan On July 17, 2013, 5:14 p.m., John Pullokkaran wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/12690/ --- (Updated July 17, 2013, 5:14 p.m.) Review request for hive and Ashutosh Chauhan. Repository: hive-git Description --- Explain extended does not include partition information for Fetch Task (FetchWork). Map Reduce Task (MapredWork)already does this. Patch adds Partition Description info to Fetch Task. Diffs - ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java 65c39d6 ql/src/test/results/clientpositive/auto_sortmerge_join_1.q.out 0e8f96b ql/src/test/results/clientpositive/auto_sortmerge_join_11.q.out 42e25fa ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out 47a8635 ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out c39d057 ql/src/test/results/clientpositive/auto_sortmerge_join_3.q.out bd7381f ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out 6121722 ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out e0cd848 ql/src/test/results/clientpositive/auto_sortmerge_join_8.q.out 924fbad ql/src/test/results/clientpositive/bucketcontext_1.q.out 62910fb ql/src/test/results/clientpositive/bucketcontext_2.q.out 0857c9d ql/src/test/results/clientpositive/bucketcontext_3.q.out 69dc2b2 ql/src/test/results/clientpositive/bucketcontext_4.q.out 0d79901 ql/src/test/results/clientpositive/bucketcontext_7.q.out 19ea4fa ql/src/test/results/clientpositive/bucketcontext_8.q.out 9a7aaa0 ql/src/test/results/clientpositive/bucketmapjoin1.q.out 9f8552a ql/src/test/results/clientpositive/bucketmapjoin10.q.out 1a6bc06 ql/src/test/results/clientpositive/bucketmapjoin11.q.out bd9b1fe ql/src/test/results/clientpositive/bucketmapjoin12.q.out fc161a9 ql/src/test/results/clientpositive/bucketmapjoin13.q.out 30d8925 ql/src/test/results/clientpositive/bucketmapjoin2.q.out 7f3fb3e ql/src/test/results/clientpositive/bucketmapjoin3.q.out 913e925 ql/src/test/results/clientpositive/bucketmapjoin7.q.out 8105ba4 ql/src/test/results/clientpositive/bucketmapjoin8.q.out 92c74a9 ql/src/test/results/clientpositive/bucketmapjoin9.q.out b7aec66 ql/src/test/results/clientpositive/bucketmapjoin_negative.q.out 1dd45d2 ql/src/test/results/clientpositive/bucketmapjoin_negative2.q.out 37f4a48 ql/src/test/results/clientpositive/join32.q.out 92d81b9 ql/src/test/results/clientpositive/join32_lessSize.q.out 82b3e4a ql/src/test/results/clientpositive/join33.q.out 92d81b9 ql/src/test/results/clientpositive/sort_merge_join_desc_6.q.out f6aae06 ql/src/test/results/clientpositive/sort_merge_join_desc_7.q.out dbce51a ql/src/test/results/clientpositive/stats11.q.out 57d2f9a ql/src/test/results/clientpositive/union22.q.out bec39f4 Diff: https://reviews.apache.org/r/12690/diff/ Testing --- All the hive unit tests passed. Thanks, John Pullokkaran
[jira] [Commented] (HIVE-4907) Allow additional tests cases to be specified with -Dtestcase
[
https://issues.apache.org/jira/browse/HIVE-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715685#comment-13715685
]
Hive QA commented on HIVE-4907:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12593560/HIVE-4907.patch
{color:green}SUCCESS:{color} +1 2646 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/134/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/134/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
Allow additional tests cases to be specified with -Dtestcase
Key: HIVE-4907
URL: https://issues.apache.org/jira/browse/HIVE-4907
Project: Hive
Issue Type: Improvement
Components: Testing Infrastructure
Reporter: Brock Noland
Assignee: Brock Noland
Attachments: HIVE-4907.patch
Currently we only allow a single tests case to be specified with -Dtestcase.
It'd be ideal if we could add on additional test cases as this would allow us
to batch the unit tests in ptest2.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4870) Explain Extended to show partition info for Fetch Task
[ https://issues.apache.org/jira/browse/HIVE-4870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-4870: --- Assignee: Laljo John Pullokkaran Explain Extended to show partition info for Fetch Task -- Key: HIVE-4870 URL: https://issues.apache.org/jira/browse/HIVE-4870 Project: Hive Issue Type: Bug Components: Query Processor, Tests Affects Versions: 0.11.0 Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Fix For: 0.11.1 Attachments: ep1.patch Explain extended does not include partition information for Fetch Task (FetchWork). Map Reduce Task (MapredWork)already does this. Patch includes Partition Description info to Fetch Task. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4870) Explain Extended to show partition info for Fetch Task
[ https://issues.apache.org/jira/browse/HIVE-4870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715686#comment-13715686 ] Ashutosh Chauhan commented on HIVE-4870: +1 [~jpullokkaran] Can you re-upload the patch with name HIVE-4870.patch to trigger HIVE QA build bot to trigger. Explain Extended to show partition info for Fetch Task -- Key: HIVE-4870 URL: https://issues.apache.org/jira/browse/HIVE-4870 Project: Hive Issue Type: Bug Components: Query Processor, Tests Affects Versions: 0.11.0 Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Fix For: 0.11.1 Attachments: ep1.patch Explain extended does not include partition information for Fetch Task (FetchWork). Map Reduce Task (MapredWork)already does this. Patch includes Partition Description info to Fetch Task. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HIVE-4863) Fix parallel order by on hadoop2
[ https://issues.apache.org/jira/browse/HIVE-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan reopened HIVE-4863: Compiling source with -Dhadoop.mr.rev=23 is not an acceptable solution, since that means we need to generate two different binaries for 20S and 23. Till now single hive binary works for both 20S and 23 and we need to keep it that way. We need to shim this method appropriately. Fix parallel order by on hadoop2 Key: HIVE-4863 URL: https://issues.apache.org/jira/browse/HIVE-4863 Project: Hive Issue Type: Bug Reporter: Edward Capriolo Assignee: Brock Noland -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4907) Allow additional tests cases to be specified with -Dtestcase
[ https://issues.apache.org/jira/browse/HIVE-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715696#comment-13715696 ] Ashutosh Chauhan commented on HIVE-4907: I didnt get the issue here. -Dtestcase=test1.q,test2.q works today. Allow additional tests cases to be specified with -Dtestcase Key: HIVE-4907 URL: https://issues.apache.org/jira/browse/HIVE-4907 Project: Hive Issue Type: Improvement Components: Testing Infrastructure Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-4907.patch Currently we only allow a single tests case to be specified with -Dtestcase. It'd be ideal if we could add on additional test cases as this would allow us to batch the unit tests in ptest2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HIVE-4907) Allow additional tests cases to be specified with -Dtestcase
[ https://issues.apache.org/jira/browse/HIVE-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715696#comment-13715696 ] Ashutosh Chauhan edited comment on HIVE-4907 at 7/22/13 9:40 PM: - I didnt get the issue here. -Dtestcase=TestCliDriver -Dqfile=test1.q,test2.q works today. was (Author: ashutoshc): I didnt get the issue here. -Dtestcase=test1.q,test2.q works today. Allow additional tests cases to be specified with -Dtestcase Key: HIVE-4907 URL: https://issues.apache.org/jira/browse/HIVE-4907 Project: Hive Issue Type: Improvement Components: Testing Infrastructure Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-4907.patch Currently we only allow a single tests case to be specified with -Dtestcase. It'd be ideal if we could add on additional test cases as this would allow us to batch the unit tests in ptest2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4222) Timestamp type constants cannot be deserialized in JDK 1.6 or less
[
https://issues.apache.org/jira/browse/HIVE-4222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715733#comment-13715733
]
Jason Dere commented on HIVE-4222:
--
Any update on the status of this one? I'm running into same issue with my work
on the Date type (HIVE-4055) when trying to create ExprNodeConstDesc with a
java.sql.Date value. Should I roll my own version of this solution for the Date
type?
Timestamp type constants cannot be deserialized in JDK 1.6 or less
--
Key: HIVE-4222
URL: https://issues.apache.org/jira/browse/HIVE-4222
Project: Hive
Issue Type: Bug
Components: Types
Reporter: Navis
Assignee: Navis
Attachments: HIVE-4222.D9681.1.patch
For example,
{noformat}
ExprNodeConstantDesc constant = new
ExprNodeConstantDesc(TypeInfoFactory.timestampTypeInfo, new Timestamp(100));
String serialized = Utilities.serializeExpression(constant);
ExprNodeConstantDesc deserilized = (ExprNodeConstantDesc)
Utilities.deserializeExpression(serialized, new Configuration());
{noformat}
logs error message
{noformat}
java.lang.InstantiationException: java.sql.Timestamp
Continuing ...
java.lang.RuntimeException: failed to evaluate: unbound=Class.new();
Continuing ...
{noformat}
and makes NPE in final.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4870) Explain Extended to show partition info for Fetch Task
[ https://issues.apache.org/jira/browse/HIVE-4870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-4870: - Attachment: (was: ep1.patch) Explain Extended to show partition info for Fetch Task -- Key: HIVE-4870 URL: https://issues.apache.org/jira/browse/HIVE-4870 Project: Hive Issue Type: Bug Components: Query Processor, Tests Affects Versions: 0.11.0 Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Fix For: 0.11.1 Attachments: HIVE-4870.patch Explain extended does not include partition information for Fetch Task (FetchWork). Map Reduce Task (MapredWork)already does this. Patch includes Partition Description info to Fetch Task. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4870) Explain Extended to show partition info for Fetch Task
[ https://issues.apache.org/jira/browse/HIVE-4870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-4870: - Status: Patch Available (was: Open) Explain Extended to show partition info for Fetch Task -- Key: HIVE-4870 URL: https://issues.apache.org/jira/browse/HIVE-4870 Project: Hive Issue Type: Bug Components: Query Processor, Tests Affects Versions: 0.11.0 Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Fix For: 0.11.1 Attachments: HIVE-4870.patch Explain extended does not include partition information for Fetch Task (FetchWork). Map Reduce Task (MapredWork)already does this. Patch includes Partition Description info to Fetch Task. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4870) Explain Extended to show partition info for Fetch Task
[ https://issues.apache.org/jira/browse/HIVE-4870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-4870: - Attachment: HIVE-4870.patch Explain Extended to show partition info for Fetch Task -- Key: HIVE-4870 URL: https://issues.apache.org/jira/browse/HIVE-4870 Project: Hive Issue Type: Bug Components: Query Processor, Tests Affects Versions: 0.11.0 Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Fix For: 0.11.1 Attachments: HIVE-4870.patch Explain extended does not include partition information for Fetch Task (FetchWork). Map Reduce Task (MapredWork)already does this. Patch includes Partition Description info to Fetch Task. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4870) Explain Extended to show partition info for Fetch Task
[ https://issues.apache.org/jira/browse/HIVE-4870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-4870: - Status: Open (was: Patch Available) Explain Extended to show partition info for Fetch Task -- Key: HIVE-4870 URL: https://issues.apache.org/jira/browse/HIVE-4870 Project: Hive Issue Type: Bug Components: Query Processor, Tests Affects Versions: 0.11.0 Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Fix For: 0.11.1 Attachments: HIVE-4870.patch Explain extended does not include partition information for Fetch Task (FetchWork). Map Reduce Task (MapredWork)already does this. Patch includes Partition Description info to Fetch Task. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4660) Let there be Tez
[ https://issues.apache.org/jira/browse/HIVE-4660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-4660: - Summary: Let there be Tez (was: Let there be Tez (aka mrr ftw)) Let there be Tez Key: HIVE-4660 URL: https://issues.apache.org/jira/browse/HIVE-4660 Project: Hive Issue Type: New Feature Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HiveonTez.pdf Tez is a new application framework built on Hadoop Yarn that can execute complex directed acyclic graphs of general data processing tasks. Here's the project's page: http://incubator.apache.org/projects/tez.html The interesting thing about Tez from Hive's perspective is that it will over time allow us to overcome inefficiencies in query processing due to having to express every algorithm in the map-reduce paradigm. The barrier to entry is pretty low as well: Tez can actually run unmodified MR jobs; But as a first step we can without much trouble start using more of Tez' features by taking advantage of the MRR pattern. MRR simply means that there can be any number of reduce stages following a single map stage - without having to write intermediate results to HDFS and re-read them in a new job. This is common when queries require multiple shuffles on keys without correlation (e.g.: join - grp by - window function - order by) For more details see the attached design doc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4660) Let there be Tez
[ https://issues.apache.org/jira/browse/HIVE-4660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-4660: - Attachment: (was: HiveonTez.pdf) Let there be Tez Key: HIVE-4660 URL: https://issues.apache.org/jira/browse/HIVE-4660 Project: Hive Issue Type: New Feature Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Tez is a new application framework built on Hadoop Yarn that can execute complex directed acyclic graphs of general data processing tasks. Here's the project's page: http://incubator.apache.org/projects/tez.html The interesting thing about Tez from Hive's perspective is that it will over time allow us to overcome inefficiencies in query processing due to having to express every algorithm in the map-reduce paradigm. The barrier to entry is pretty low as well: Tez can actually run unmodified MR jobs; But as a first step we can without much trouble start using more of Tez' features by taking advantage of the MRR pattern. MRR simply means that there can be any number of reduce stages following a single map stage - without having to write intermediate results to HDFS and re-read them in a new job. This is common when queries require multiple shuffles on keys without correlation (e.g.: join - grp by - window function - order by) For more details see the attached design doc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4907) Allow additional tests cases to be specified with -Dtestcase
[ https://issues.apache.org/jira/browse/HIVE-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715743#comment-13715743 ] Brock Noland commented on HIVE-4907: [~ashutoshc] this change is about batching junit tests not qfile tests. Agreed the qfile test batching works and is being used to great benefit in ptest2 :) Allow additional tests cases to be specified with -Dtestcase Key: HIVE-4907 URL: https://issues.apache.org/jira/browse/HIVE-4907 Project: Hive Issue Type: Improvement Components: Testing Infrastructure Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-4907.patch Currently we only allow a single tests case to be specified with -Dtestcase. It'd be ideal if we could add on additional test cases as this would allow us to batch the unit tests in ptest2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
