[jira] [Commented] (HIVE-8485) HMS on Oracle incompatibility
[
https://issues.apache.org/jira/browse/HIVE-8485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270263#comment-14270263
]
Sushanth Sowmyan commented on HIVE-8485:
[~ctang.ma]/[~sershe], could you please check/review this patch, and this
approach?
HMS on Oracle incompatibility
-
Key: HIVE-8485
URL: https://issues.apache.org/jira/browse/HIVE-8485
Project: Hive
Issue Type: Bug
Components: Metastore
Environment: Oracle as metastore DB
Reporter: Ryan Pridgeon
Assignee: Chaoyu Tang
Attachments: HIVE-8485.2.patch, HIVE-8485.patch
Oracle does not distinguish between empty strings and NULL,which proves
problematic for DataNucleus.
In the event a user creates a table with some property stored as an empty
string the table will no longer be accessible.
i.e. TBLPROPERTIES ('serialization.null.format'='')
If they try to select, describe, drop, etc the client prints the following
exception.
ERROR ql.Driver: FAILED: SemanticException [Error 10001]: Table not found
table name
The work around for this was to go into the hive metastore on the Oracle
database and replace NULL with some other string. Users could then drop the
tables or alter their data to use the new null format they just set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-836) Add syntax to force a new mapreduce job / transform subquery in mapper
[
https://issues.apache.org/jira/browse/HIVE-836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270390#comment-14270390
]
Adam Kramer commented on HIVE-836:
--
Oh hey there five year old task.
Workaround: Use CLUSTER BY to force a reduce phase, and a staging table to
force a map phase. Hive writes all the data to disk in every phase anyway so
the staging table isn't actually a performance hit.
Also protip: DON'T get distracted by the Hive keywords MAP and REDUCE, they
are just synonyms for TRANSFORM and do not do what anybody expects.
Add syntax to force a new mapreduce job / transform subquery in mapper
--
Key: HIVE-836
URL: https://issues.apache.org/jira/browse/HIVE-836
Project: Hive
Issue Type: Wish
Reporter: Adam Kramer
Hive currently does a lot of awesome work to figure out when my transformers
should be used in the mapper and when they should be used in the reducer.
However, sometimes I have a different plan.
For example, consider this:
{code:title=foo.sql}
SELECT TRANSFORM(a.val1, a.val2)
USING './niftyscript'
AS part1, part2, part3
FROM (
SELECT b.val AS val1, c.val AS val2
FROM tblb b JOIN tblc c on (b.key=c.key)
) a
{code}
...now, assume that the join step is very easy and 'niftyscript' is really
processor intensive. The ideal format for this is a MR task with few mappers
and few reducers, and then a second MR task with lots of mappers.
Currently, there is no way to even require the outer TRANSFORM statement
occur in a separate map phase. Implementing a hint such as /* +MAP */, akin
to /* +MAPJOIN(x) */, would be awesome.
Current workaround is to dump everything to a temporary table and then start
over, but that is not an easy to scale--the subquery structure effectively
(and easily) locks the mid-points so no other job can touch the table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-4790) MapredLocalTask task does not make virtual columns
[
https://issues.apache.org/jira/browse/HIVE-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-4790:
Attachment: HIVE-4790.14.patch.txt
MapredLocalTask task does not make virtual columns
--
Key: HIVE-4790
URL: https://issues.apache.org/jira/browse/HIVE-4790
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Minor
Attachments: D11511.3.patch, D11511.4.patch, HIVE-4790.10.patch.txt,
HIVE-4790.11.patch.txt, HIVE-4790.12.patch.txt, HIVE-4790.13.patch.txt,
HIVE-4790.14.patch.txt, HIVE-4790.5.patch.txt, HIVE-4790.6.patch.txt,
HIVE-4790.7.patch.txt, HIVE-4790.8.patch.txt, HIVE-4790.9.patch.txt,
HIVE-4790.D11511.1.patch, HIVE-4790.D11511.2.patch
From mailing list,
http://www.mail-archive.com/user@hive.apache.org/msg08264.html
{noformat}
SELECT *,b.BLOCK__OFFSET__INSIDE__FILE FROM a JOIN b ON
b.rownumber = a.number;
fails with this error:
SELECT *,b.BLOCK__OFFSET__INSIDE__FILE FROM a JOIN b ON b.rownumber =
a.number;
Automatically selecting local only mode for query
Total MapReduce jobs = 1
setting HADOOP_USER_NAMEpmarron
13/06/25 10:52:56 WARN conf.HiveConf: DEPRECATED: Configuration property
hive.metastore.local no longer has any effect. Make sure to provide a valid
value for hive.metastore.uris if you are connecting to a remote metastore.
Execution log at: /tmp/pmarron/.log
2013-06-25 10:52:56 Starting to launch local task to process map join;
maximum memory = 932118528
java.lang.RuntimeException: cannot find field block__offset__inside__file
from [0:rownumber, 1:offset]
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:366)
at
org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.getStructFieldRef(LazySimpleStructObjectInspector.java:168)
at
org.apache.hadoop.hive.serde2.objectinspector.DelegatedStructObjectInspector.getStructFieldRef(DelegatedStructObjectInspector.java:74)
at
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)
at
org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:68)
at
org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.initializeOp(HashTableSinkOperator.java:222)
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:451)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:407)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:186)
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
at
org.apache.hadoop.hive.ql.exec.MapredLocalTask.initializeOperators(MapredLocalTask.java:394)
at
org.apache.hadoop.hive.ql.exec.MapredLocalTask.executeFromChildJVM(MapredLocalTask.java:277)
at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:676)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Execution failed with exit status: 2
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7550) Extend cached evaluation to multiple expressions
[ https://issues.apache.org/jira/browse/HIVE-7550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-7550: Status: Patch Available (was: Open) Extend cached evaluation to multiple expressions Key: HIVE-7550 URL: https://issues.apache.org/jira/browse/HIVE-7550 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Trivial Attachments: HIVE-7550.1.patch.txt, HIVE-7550.2.patch.txt Currently, hive.cache.expr.evaluation caches per expression. But cache context might be shared for multiple expressions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7550) Extend cached evaluation to multiple expressions
[ https://issues.apache.org/jira/browse/HIVE-7550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-7550: Attachment: HIVE-7550.2.patch.txt Extend cached evaluation to multiple expressions Key: HIVE-7550 URL: https://issues.apache.org/jira/browse/HIVE-7550 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Trivial Attachments: HIVE-7550.1.patch.txt, HIVE-7550.2.patch.txt Currently, hive.cache.expr.evaluation caches per expression. But cache context might be shared for multiple expressions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9319) Cleanup Modified Files [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-9319: Attachment: HIVE-9319-spark.patch Cleanup Modified Files [Spark Branch] - Key: HIVE-9319 URL: https://issues.apache.org/jira/browse/HIVE-9319 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Szehon Ho Assignee: Szehon Ho Priority: Minor Attachments: HIVE-9319-spark.patch Cleanup the code that is modified based on checkstyle/TODO/warnings. It is a follow-up of HIVE-9281 which is for new files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9322) Make null-checks consistent for MapObjectInspector subclasses.
Mithun Radhakrishnan created HIVE-9322:
--
Summary: Make null-checks consistent for MapObjectInspector
subclasses.
Key: HIVE-9322
URL: https://issues.apache.org/jira/browse/HIVE-9322
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Affects Versions: 0.14.0
Reporter: Mithun Radhakrishnan
Assignee: Mithun Radhakrishnan
Priority: Minor
{{LazyBinaryMapObjectInspector}}, {{DeepParquetHiveMapInspector}}, etc. check
both the map-column value and the map-key for null, before dereferencing them.
{{OrcMapObjectInspector}} and {{LazyMapObjectInspector}} do not.
This patch brings them all in sync. Might not be a real problem, unless (for
example) the lookup key is itself a (possibly null) value from another column.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9319) Cleanup Modified Files [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270369#comment-14270369
]
Hive QA commented on HIVE-9319:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12690973/HIVE-9319-spark.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 7285 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby3_map_skew
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_windowing
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/621/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/621/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-621/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12690973 - PreCommit-HIVE-SPARK-Build
Cleanup Modified Files [Spark Branch]
-
Key: HIVE-9319
URL: https://issues.apache.org/jira/browse/HIVE-9319
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Szehon Ho
Assignee: Szehon Ho
Priority: Minor
Attachments: HIVE-9319-spark.patch
Cleanup the code that is modified based on checkstyle/TODO/warnings.
It is a follow-up of HIVE-9281 which is for new files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9290) Make some test results deterministic
[
https://issues.apache.org/jira/browse/HIVE-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270456#comment-14270456
]
Rui Li commented on HIVE-9290:
--
The failed test is not related to the patch here.
Make some test results deterministic
Key: HIVE-9290
URL: https://issues.apache.org/jira/browse/HIVE-9290
Project: Hive
Issue Type: Test
Reporter: Rui Li
Assignee: Rui Li
Attachments: HIVE-9290.1.patch, HIVE-9290.1.patch
{noformat}
limit_pushdown.q
optimize_nullscan.q
ppd_gby_join.q
vector_string_concat.q
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9307) Use metastore warehouse dir variable from conf instead of hard coded dir in encryption test
[
https://issues.apache.org/jira/browse/HIVE-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270469#comment-14270469
]
Brock Noland commented on HIVE-9307:
Do the other .q files which were modified need to be updated?
Use metastore warehouse dir variable from conf instead of hard coded dir in
encryption test
---
Key: HIVE-9307
URL: https://issues.apache.org/jira/browse/HIVE-9307
Project: Hive
Issue Type: Sub-task
Reporter: Dong Chen
Assignee: Dong Chen
Fix For: encryption-branch
Attachments: HIVE-9307.1.patch, HIVE-9307.patch
NO PRECOMMIT TESTS
Use the following variable to get the metastore directory
$\{hiveconf:hive.metastore.warehouse.dir\} in test cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-9156) Investigate CBO [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland reassigned HIVE-9156: -- Assignee: Chao Investigate CBO [Spark Branch] -- Key: HIVE-9156 URL: https://issues.apache.org/jira/browse/HIVE-9156 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Brock Noland Assignee: Chao We need to get a list of the following: * What happens when you enable the CBO with HOS * What portions of the CBO integration are tez or MR specific * Some basic idea of what our cost model will look like -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9306) Let Context.isLocalOnlyExecutionMode() return false if execution engine is Spark [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270202#comment-14270202 ] Xuefu Zhang commented on HIVE-9306: --- Test failure above, org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23.q, doesn't seem related to the patch. It didn't happen in previous run, and neither in my local run. Let Context.isLocalOnlyExecutionMode() return false if execution engine is Spark [Spark Branch] --- Key: HIVE-9306 URL: https://issues.apache.org/jira/browse/HIVE-9306 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Xuefu Zhang Assignee: Xuefu Zhang Attachments: HIVE-9306.1-spark.patch, HIVE-9306.2-spark.patch, HIVE-9306.3-spark.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9320) Add UnionEliminatorRule on cbo path
Ashutosh Chauhan created HIVE-9320: -- Summary: Add UnionEliminatorRule on cbo path Key: HIVE-9320 URL: https://issues.apache.org/jira/browse/HIVE-9320 Project: Hive Issue Type: Improvement Components: CBO, Logical Optimizer Affects Versions: 0.14.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Shorten the pipeline, where possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9290) Make some test results deterministic
[
https://issues.apache.org/jira/browse/HIVE-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-9290:
-
Attachment: HIVE-9290.1.patch
Reload patch to trigger test
Make some test results deterministic
Key: HIVE-9290
URL: https://issues.apache.org/jira/browse/HIVE-9290
Project: Hive
Issue Type: Test
Reporter: Rui Li
Assignee: Rui Li
Attachments: HIVE-9290.1.patch, HIVE-9290.1.patch
{noformat}
limit_pushdown.q
optimize_nullscan.q
ppd_gby_join.q
vector_string_concat.q
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9316) TestSqoop tests in WebHCat testsuite hardcode libdir path to hdfs
[ https://issues.apache.org/jira/browse/HIVE-9316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270244#comment-14270244 ] Eugene Koifman commented on HIVE-9316: -- +1 TestSqoop tests in WebHCat testsuite hardcode libdir path to hdfs - Key: HIVE-9316 URL: https://issues.apache.org/jira/browse/HIVE-9316 Project: Hive Issue Type: Bug Components: Tests, WebHCat Reporter: Deepesh Khandelwal Assignee: Deepesh Khandelwal Priority: Minor Fix For: 0.15.0 Attachments: HIVE-9316.1.patch Currently the TestSqoop tests in WebHCat Perl based testsuite has hdfs:// prefix in the jdbc jar path in libdir, we should remove this to enable it to run against other file systems. NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9290) Make some test results deterministic
[
https://issues.apache.org/jira/browse/HIVE-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270450#comment-14270450
]
Hive QA commented on HIVE-9290:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12690996/HIVE-9290.1.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6739 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2297/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2297/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2297/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12690996 - PreCommit-HIVE-TRUNK-Build
Make some test results deterministic
Key: HIVE-9290
URL: https://issues.apache.org/jira/browse/HIVE-9290
Project: Hive
Issue Type: Test
Reporter: Rui Li
Assignee: Rui Li
Attachments: HIVE-9290.1.patch, HIVE-9290.1.patch
{noformat}
limit_pushdown.q
optimize_nullscan.q
ppd_gby_join.q
vector_string_concat.q
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-3405) UDF initcap to obtain a string with the first letter of each word in uppercase other letters in lowercase
[ https://issues.apache.org/jira/browse/HIVE-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Pivovarov updated HIVE-3405: -- Attachment: HIVE-3405.5.patch - use WorkUtils.capitalizeFully instead of custom implementation - check arguments.length == 1 in GenericUDFInitCap.initialize UDF initcap to obtain a string with the first letter of each word in uppercase other letters in lowercase - Key: HIVE-3405 URL: https://issues.apache.org/jira/browse/HIVE-3405 Project: Hive Issue Type: New Feature Components: UDF Affects Versions: 0.8.1, 0.9.0, 0.9.1, 0.10.0, 0.11.0, 0.13.0, 0.14.0, 0.15.0, 0.14.1 Reporter: Archana Nair Assignee: Alexander Pivovarov Labels: patch Attachments: HIVE-3405.1.patch.txt, HIVE-3405.2.patch, HIVE-3405.3.patch, HIVE-3405.4.patch, HIVE-3405.5.patch Hive current releases lacks a INITCAP function which returns String with first letter of the word in uppercase.INITCAP returns String, with the first letter of each word in uppercase, all other letters in same case. Words are delimited by white space.This will be useful report generation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-3405) UDF initcap to obtain a string with the first letter of each word in uppercase other letters in lowercase
[ https://issues.apache.org/jira/browse/HIVE-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Pivovarov updated HIVE-3405: -- Status: Patch Available (was: In Progress) HIVE-3405.5.patch UDF initcap to obtain a string with the first letter of each word in uppercase other letters in lowercase - Key: HIVE-3405 URL: https://issues.apache.org/jira/browse/HIVE-3405 Project: Hive Issue Type: New Feature Components: UDF Affects Versions: 0.14.0, 0.13.0, 0.11.0, 0.10.0, 0.9.0, 0.8.1, 0.15.0, 0.14.1, 0.9.1 Reporter: Archana Nair Assignee: Alexander Pivovarov Labels: patch Attachments: HIVE-3405.1.patch.txt, HIVE-3405.2.patch, HIVE-3405.3.patch, HIVE-3405.4.patch, HIVE-3405.5.patch Hive current releases lacks a INITCAP function which returns String with first letter of the word in uppercase.INITCAP returns String, with the first letter of each word in uppercase, all other letters in same case. Words are delimited by white space.This will be useful report generation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8872) Hive view of HBase range scan intermittently returns incorrect data.
[ https://issues.apache.org/jira/browse/HIVE-8872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270500#comment-14270500 ] 杨昆 commented on HIVE-8872: -- how can I reappear this bug? [~ychena] Can you provide your code and the hive runtime configuration? Hive view of HBase range scan intermittently returns incorrect data. Key: HIVE-8872 URL: https://issues.apache.org/jira/browse/HIVE-8872 Project: Hive Issue Type: Bug Affects Versions: 0.13.1 Reporter: Yongzhi Chen Assignee: Yongzhi Chen Fix For: 0.15.0 Attachments: HIVE-8872.1.patch, HIVE-8872.2.patch This need running in cluster: 1. Create a hive external table pointing to a hbase table. 2. Create views to the hive table(for example 30 views), each view looks like following with different range check: CREATE VIEW hview_nn AS SELECT * FROM hivehbasetable WHERE (pk ='pk_nn_0' AND pk = pk_nn_A') 3. Create same number of hive new tables as views. 4. then runs several queries in parallel (30 threads): INSERT OVERWRITE TABLE hivenewtable_nn SELECT * FROM hview_nn //nn is from 01 to 30 5 After insert, check the hivenewtables, some values are not right. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9296) Need to add schema upgrade changes for queueing events in the database
[ https://issues.apache.org/jira/browse/HIVE-9296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270198#comment-14270198 ] Sushanth Sowmyan commented on HIVE-9296: +1. Thanks, Alan! Need to add schema upgrade changes for queueing events in the database -- Key: HIVE-9296 URL: https://issues.apache.org/jira/browse/HIVE-9296 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.15.0 Reporter: Alan Gates Assignee: Alan Gates Fix For: 0.15.0 Attachments: HIVE-9296.patch HIVE-9174 added the ability to queue notification events in the database, but did not include the schema upgrade scripts. Also, in the thrift changes the convention was not followed properly in naming the thrift methods. HIVE-9174 used camel case, where the thrift methods use all lower case separated by underscores. Both of these issues should be fixed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4639) Add has null flag to ORC internal index
[
https://issues.apache.org/jira/browse/HIVE-4639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270348#comment-14270348
]
Hive QA commented on HIVE-4639:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12690690/HIVE-4639.2.patch
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 6747 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.io.orc.TestOrcNullOptimization.testColumnsWithNullAndCompression
org.apache.hadoop.hive.ql.io.orc.TestOrcNullOptimization.testMultiStripeWithNull
org.apache.hadoop.hive.ql.io.orc.TestOrcNullOptimization.testMultiStripeWithoutNull
org.apache.hadoop.hive.ql.io.orc.TestOrcSerDeStats.testOrcSerDeStatsComplex
org.apache.hadoop.hive.ql.io.orc.TestOrcSerDeStats.testOrcSerDeStatsComplexOldFormat
org.apache.hadoop.hive.ql.io.orc.TestOrcSerDeStats.testSerdeStatsOldFormat
org.apache.hadoop.hive.ql.io.orc.TestOrcSerDeStats.testStringAndBinaryStatistics
org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2296/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2296/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2296/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12690690 - PreCommit-HIVE-TRUNK-Build
Add has null flag to ORC internal index
---
Key: HIVE-4639
URL: https://issues.apache.org/jira/browse/HIVE-4639
Project: Hive
Issue Type: Improvement
Components: File Formats
Reporter: Owen O'Malley
Assignee: Prasanth Jayachandran
Attachments: HIVE-4639.1.patch, HIVE-4639.2.patch
It would enable more predicate pushdown if we added a flag to the index entry
recording if there were any null values in the column for the 10k rows.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9272) Tests for utf-8 support
[ https://issues.apache.org/jira/browse/HIVE-9272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-9272: - Assignee: Aswathy Chellammal Sreekumar Tests for utf-8 support --- Key: HIVE-9272 URL: https://issues.apache.org/jira/browse/HIVE-9272 Project: Hive Issue Type: Test Components: Tests, WebHCat Reporter: Aswathy Chellammal Sreekumar Assignee: Aswathy Chellammal Sreekumar Priority: Minor Attachments: HIVE-9272.1.patch, HIVE-9272.patch Including some test cases for utf8 support in webhcat. The first four tests invoke hive, pig, mapred and streaming apis for testing the utf8 support for data processed, file names and job name. The last test case tests the filtering of job name with utf8 character -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9272) Tests for utf-8 support
[ https://issues.apache.org/jira/browse/HIVE-9272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270365#comment-14270365 ] Eugene Koifman commented on HIVE-9272: -- 1. could you add a comment to the change in deploy_e2e_artifacts.sh to indicate what set of tests the artifact is for? 2. I'm not sure how you are generating the patch, but it has some headers that may not apply. I usually use git diff --no-prefix SHA SHA1 foo.patch 3. my knowledge of Perl is very limited. Is there someone else who can review .pm changes? Tests for utf-8 support --- Key: HIVE-9272 URL: https://issues.apache.org/jira/browse/HIVE-9272 Project: Hive Issue Type: Test Components: Tests, WebHCat Reporter: Aswathy Chellammal Sreekumar Priority: Minor Attachments: HIVE-9272.1.patch, HIVE-9272.patch Including some test cases for utf8 support in webhcat. The first four tests invoke hive, pig, mapred and streaming apis for testing the utf8 support for data processed, file names and job name. The last test case tests the filtering of job name with utf8 character -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9307) Use metastore warehouse dir variable from conf instead of hard coded dir in encryption test
[
https://issues.apache.org/jira/browse/HIVE-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270398#comment-14270398
]
Dong Chen commented on HIVE-9307:
-
Update patch V1.
The variable is used in 3 statement: {{CREATE table sql}}, {{dfs command}}, and
{{CRYPTO command}}. The first 2 already can handle this variable. For CRYPTO
command, since we only use it for test, I added the handling in
{{QTestUtil.executeTestCommand()}} for simplicity.
The output of 2 cases changed by 2 lines, which are the table location in
create query in PREHOOK / POSTHOOK is masked. After double check, I think it is
OK and I misunderstood before.
Use metastore warehouse dir variable from conf instead of hard coded dir in
encryption test
---
Key: HIVE-9307
URL: https://issues.apache.org/jira/browse/HIVE-9307
Project: Hive
Issue Type: Sub-task
Reporter: Dong Chen
Assignee: Dong Chen
Attachments: HIVE-9307.patch
NO PRECOMMIT TESTS
Use the following variable to get the metastore directory
$\{hiveconf:hive.metastore.warehouse.dir\} in test cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9307) Use metastore warehouse dir variable from conf instead of hard coded dir in encryption test
[
https://issues.apache.org/jira/browse/HIVE-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9307:
Attachment: HIVE-9307.1.patch
Use metastore warehouse dir variable from conf instead of hard coded dir in
encryption test
---
Key: HIVE-9307
URL: https://issues.apache.org/jira/browse/HIVE-9307
Project: Hive
Issue Type: Sub-task
Reporter: Dong Chen
Assignee: Dong Chen
Attachments: HIVE-9307.1.patch, HIVE-9307.patch
NO PRECOMMIT TESTS
Use the following variable to get the metastore directory
$\{hiveconf:hive.metastore.warehouse.dir\} in test cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9323) Merge from trunk to spark 1/8/2015
[ https://issues.apache.org/jira/browse/HIVE-9323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-9323: Attachment: HIVE-9323-spark.patch Merge from trunk to spark 1/8/2015 -- Key: HIVE-9323 URL: https://issues.apache.org/jira/browse/HIVE-9323 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Szehon Ho Assignee: Szehon Ho Attachments: HIVE-9323-spark.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9307) Use metastore warehouse dir variable from conf instead of hard coded dir in encryption test
[
https://issues.apache.org/jira/browse/HIVE-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9307:
Fix Version/s: encryption-branch
Status: Patch Available (was: Open)
Use metastore warehouse dir variable from conf instead of hard coded dir in
encryption test
---
Key: HIVE-9307
URL: https://issues.apache.org/jira/browse/HIVE-9307
Project: Hive
Issue Type: Sub-task
Reporter: Dong Chen
Assignee: Dong Chen
Fix For: encryption-branch
Attachments: HIVE-9307.1.patch, HIVE-9307.patch
NO PRECOMMIT TESTS
Use the following variable to get the metastore directory
$\{hiveconf:hive.metastore.warehouse.dir\} in test cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9323) Merge from trunk to spark 1/8/2015
[ https://issues.apache.org/jira/browse/HIVE-9323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-9323: Status: Patch Available (was: Open) Giving this a try for precommit test. Merge from trunk to spark 1/8/2015 -- Key: HIVE-9323 URL: https://issues.apache.org/jira/browse/HIVE-9323 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Szehon Ho Assignee: Szehon Ho Attachments: HIVE-9323-spark.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-3405) UDF initcap to obtain a string with the first letter of each word in uppercase other letters in lowercase
[ https://issues.apache.org/jira/browse/HIVE-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Pivovarov updated HIVE-3405: -- Status: In Progress (was: Patch Available) UDF initcap to obtain a string with the first letter of each word in uppercase other letters in lowercase - Key: HIVE-3405 URL: https://issues.apache.org/jira/browse/HIVE-3405 Project: Hive Issue Type: New Feature Components: UDF Affects Versions: 0.14.0, 0.13.0, 0.11.0, 0.10.0, 0.9.0, 0.8.1, 0.15.0, 0.14.1, 0.9.1 Reporter: Archana Nair Assignee: Alexander Pivovarov Labels: patch Attachments: HIVE-3405.1.patch.txt, HIVE-3405.2.patch, HIVE-3405.3.patch, HIVE-3405.4.patch Hive current releases lacks a INITCAP function which returns String with first letter of the word in uppercase.INITCAP returns String, with the first letter of each word in uppercase, all other letters in same case. Words are delimited by white space.This will be useful report generation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9323) Merge from trunk to spark 1/8/2015
[
https://issues.apache.org/jira/browse/HIVE-9323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270497#comment-14270497
]
Hive QA commented on HIVE-9323:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12691015/HIVE-9323-spark.patch
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 7301 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join7
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ctas
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_cube1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join7
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_windowing
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/622/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/622/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-622/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12691015 - PreCommit-HIVE-SPARK-Build
Merge from trunk to spark 1/8/2015
--
Key: HIVE-9323
URL: https://issues.apache.org/jira/browse/HIVE-9323
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Szehon Ho
Assignee: Szehon Ho
Attachments: HIVE-9323-spark.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8966) Delta files created by hive hcatalog streaming cannot be compacted
[ https://issues.apache.org/jira/browse/HIVE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270282#comment-14270282 ] Alan Gates commented on HIVE-8966: -- The issue is that since the writer died with an unclosed batch it left the orc file in a state where it cannot be read without the length file. So removing the length file means any reader will fail when reading it. The proper solution is for the compactor to stop at that partition until it has determined all transactions in that file have committed or aborted. Then it should compact it using the length file, but properly ignore the length file. I'll work on the fix. Delta files created by hive hcatalog streaming cannot be compacted -- Key: HIVE-8966 URL: https://issues.apache.org/jira/browse/HIVE-8966 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.14.0 Environment: hive Reporter: Jihong Liu Assignee: Alan Gates Priority: Critical Fix For: 0.14.1 Attachments: HIVE-8966.2.patch, HIVE-8966.3.patch, HIVE-8966.patch hive hcatalog streaming will also create a file like bucket_n_flush_length in each delta directory. Where n is the bucket number. But the compactor.CompactorMR think this file also needs to compact. However this file of course cannot be compacted, so compactor.CompactorMR will not continue to do the compaction. Did a test, after removed the bucket_n_flush_length file, then the alter table partition compact finished successfully. If don't delete that file, nothing will be compacted. This is probably a very severity bug. Both 0.13 and 0.14 have this issue -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8966) Delta files created by hive hcatalog streaming cannot be compacted
[ https://issues.apache.org/jira/browse/HIVE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-8966: - Status: Open (was: Patch Available) Delta files created by hive hcatalog streaming cannot be compacted -- Key: HIVE-8966 URL: https://issues.apache.org/jira/browse/HIVE-8966 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.14.0 Environment: hive Reporter: Jihong Liu Assignee: Alan Gates Priority: Critical Fix For: 0.14.1 Attachments: HIVE-8966.2.patch, HIVE-8966.3.patch, HIVE-8966.patch hive hcatalog streaming will also create a file like bucket_n_flush_length in each delta directory. Where n is the bucket number. But the compactor.CompactorMR think this file also needs to compact. However this file of course cannot be compacted, so compactor.CompactorMR will not continue to do the compaction. Did a test, after removed the bucket_n_flush_length file, then the alter table partition compact finished successfully. If don't delete that file, nothing will be compacted. This is probably a very severity bug. Both 0.13 and 0.14 have this issue -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-836) Add syntax to force a new mapreduce job / transform subquery in mapper
[
https://issues.apache.org/jira/browse/HIVE-836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Kramer resolved HIVE-836.
--
Resolution: Won't Fix
Release Note: See comments for workarounds.
Add syntax to force a new mapreduce job / transform subquery in mapper
--
Key: HIVE-836
URL: https://issues.apache.org/jira/browse/HIVE-836
Project: Hive
Issue Type: Wish
Reporter: Adam Kramer
Hive currently does a lot of awesome work to figure out when my transformers
should be used in the mapper and when they should be used in the reducer.
However, sometimes I have a different plan.
For example, consider this:
{code:title=foo.sql}
SELECT TRANSFORM(a.val1, a.val2)
USING './niftyscript'
AS part1, part2, part3
FROM (
SELECT b.val AS val1, c.val AS val2
FROM tblb b JOIN tblc c on (b.key=c.key)
) a
{code}
...now, assume that the join step is very easy and 'niftyscript' is really
processor intensive. The ideal format for this is a MR task with few mappers
and few reducers, and then a second MR task with lots of mappers.
Currently, there is no way to even require the outer TRANSFORM statement
occur in a separate map phase. Implementing a hint such as /* +MAP */, akin
to /* +MAPJOIN(x) */, would be awesome.
Current workaround is to dump everything to a temporary table and then start
over, but that is not an easy to scale--the subquery structure effectively
(and easily) locks the mid-points so no other job can touch the table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-8853) Make vectorization work with Spark [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland reassigned HIVE-8853: -- Assignee: Jimmy Xiang Jimmy mentioned he was interested in this. Make vectorization work with Spark [Spark Branch] - Key: HIVE-8853 URL: https://issues.apache.org/jira/browse/HIVE-8853 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Xuefu Zhang Assignee: Jimmy Xiang In Hive to make vectorization work, the reader needs to be also vectorized, which means that the reader can read a chunk of rows (or a list of column chunks) instead of one row at a time. However, we use Spark RDD for reading, which again utilized the underlying inputformat to read. Subsequent processing also needs to hapen in batches. We need to make sure that vectorizatoin is working as expected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-4639) Add has null flag to ORC internal index
[ https://issues.apache.org/jira/browse/HIVE-4639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-4639: Attachment: HIVE-4639.3.patch I missed out few test failure diffs in previous patch. Added them in this patch. Add has null flag to ORC internal index --- Key: HIVE-4639 URL: https://issues.apache.org/jira/browse/HIVE-4639 Project: Hive Issue Type: Improvement Components: File Formats Reporter: Owen O'Malley Assignee: Prasanth Jayachandran Attachments: HIVE-4639.1.patch, HIVE-4639.2.patch, HIVE-4639.3.patch It would enable more predicate pushdown if we added a flag to the index entry recording if there were any null values in the column for the 10k rows. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9242) Many places in CBO code eat exceptions
[
https://issues.apache.org/jira/browse/HIVE-9242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270519#comment-14270519
]
Hive QA commented on HIVE-9242:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12690708/HIVE-9242.1.patch.txt
{color:green}SUCCESS:{color} +1 6739 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2298/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2298/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2298/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12690708 - PreCommit-HIVE-TRUNK-Build
Many places in CBO code eat exceptions
--
Key: HIVE-9242
URL: https://issues.apache.org/jira/browse/HIVE-9242
Project: Hive
Issue Type: Bug
Reporter: Brock Noland
Priority: Blocker
Attachments: HIVE-9242.1.patch.txt
I've noticed that there are a number of places in the CBO code which eat
exceptions. This is not acceptable. Example:
https://github.com/apache/hive/blob/357b473a354aace3bd59b522ad7108be561e9d0f/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java#L274
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9306) Let Context.isLocalOnlyExecutionMode() return false if execution engine is Spark [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-9306: -- Resolution: Fixed Fix Version/s: spark-branch Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks to Szehon for the review. Let Context.isLocalOnlyExecutionMode() return false if execution engine is Spark [Spark Branch] --- Key: HIVE-9306 URL: https://issues.apache.org/jira/browse/HIVE-9306 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Xuefu Zhang Assignee: Xuefu Zhang Fix For: spark-branch Attachments: HIVE-9306.1-spark.patch, HIVE-9306.2-spark.patch, HIVE-9306.3-spark.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9175) Add alters to list of events handled by NotificationListener
[ https://issues.apache.org/jira/browse/HIVE-9175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270216#comment-14270216 ] Alan Gates commented on HIVE-9175: -- I don't believe the TestStreaming failure is related, as it passes locally and I didn't change any code in this area. I thought I fixed the flakiness in that test but I guess not. Add alters to list of events handled by NotificationListener Key: HIVE-9175 URL: https://issues.apache.org/jira/browse/HIVE-9175 Project: Hive Issue Type: New Feature Components: HCatalog Reporter: Alan Gates Assignee: Alan Gates Fix For: 0.15.0 Attachments: HIVE-9175.patch HCatalog currently doesn't implement onAlterTable and onAlterPartition. It should. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9319) Cleanup Modified Files [Spark Branch]
Szehon Ho created HIVE-9319: --- Summary: Cleanup Modified Files [Spark Branch] Key: HIVE-9319 URL: https://issues.apache.org/jira/browse/HIVE-9319 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Szehon Ho Assignee: Szehon Ho Priority: Minor Cleanup the code that is modified based on checkstyle/TODO/warnings. It is a follow-up of HIVE-9281 which is for new files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 29733: HIVE-9319 : Cleanup Modified Files [Spark Branch]
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29733/ --- Review request for hive and Xuefu Zhang. Repository: hive-git Description --- Note that this limits cleanup to lines of code changed in spark-branch in the merge to trunk, not cleanup of all of the modified files, in order to reduce merge conflicts. Diffs - ql/src/java/org/apache/hadoop/hive/ql/Driver.java fa40082 ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java b25a639 ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java ee42f4c ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java abdb6af ql/src/java/org/apache/hadoop/hive/ql/io/HiveKey.java 33aeda4 ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 6f216c9 ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java a6d5c62 ql/src/java/org/apache/hadoop/hive/ql/optimizer/unionproc/UnionProcessor.java fec6822 ql/src/java/org/apache/hadoop/hive/ql/parse/MapReduceCompiler.java 1b6de64 ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 1efbb12 ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java 4582678 ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 076d2fa shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java f1743ae Diff: https://reviews.apache.org/r/29733/diff/ Testing --- Thanks, Szehon Ho
[jira] [Updated] (HIVE-9319) Cleanup Modified Files [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-9319: Status: Patch Available (was: Open) Cleanup Modified Files [Spark Branch] - Key: HIVE-9319 URL: https://issues.apache.org/jira/browse/HIVE-9319 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Szehon Ho Assignee: Szehon Ho Priority: Minor Attachments: HIVE-9319-spark.patch Cleanup the code that is modified based on checkstyle/TODO/warnings. It is a follow-up of HIVE-9281 which is for new files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8327) mvn site -Pfindbugs
[ https://issues.apache.org/jira/browse/HIVE-8327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270437#comment-14270437 ] Ashutosh Chauhan commented on HIVE-8327: +1 mvn site -Pfindbugs --- Key: HIVE-8327 URL: https://issues.apache.org/jira/browse/HIVE-8327 Project: Hive Issue Type: Test Components: Diagnosability Reporter: Gopal V Assignee: Gopal V Fix For: 0.15.0 Attachments: HIVE-8327.1.patch, HIVE-8327.2.patch, ql-findbugs.html HIVE-3099 originally added findbugs into the old ant build. Get basic findbugs working for the maven build. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9304) [Refactor] remove unused method in SemAly
[
https://issues.apache.org/jira/browse/HIVE-9304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-9304:
---
Resolution: Fixed
Fix Version/s: 0.15.0
Status: Resolved (was: Patch Available)
Committed to trunk.
[Refactor] remove unused method in SemAly
-
Key: HIVE-9304
URL: https://issues.apache.org/jira/browse/HIVE-9304
Project: Hive
Issue Type: Task
Components: Query Processor
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
Priority: Trivial
Fix For: 0.15.0
Attachments: HIVE-9304.patch
Seems like method {{genConversionOps}} don't serve any purpose any longer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9290) Make some test results deterministic
[
https://issues.apache.org/jira/browse/HIVE-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-9290:
--
Resolution: Fixed
Fix Version/s: 0.15.0
spark-branch
Status: Resolved (was: Patch Available)
Committed to trunk and merged to Spark branch. Thanks, Rui.
Make some test results deterministic
Key: HIVE-9290
URL: https://issues.apache.org/jira/browse/HIVE-9290
Project: Hive
Issue Type: Test
Reporter: Rui Li
Assignee: Rui Li
Fix For: spark-branch, 0.15.0
Attachments: HIVE-9290.1.patch, HIVE-9290.1.patch
{noformat}
limit_pushdown.q
optimize_nullscan.q
ppd_gby_join.q
vector_string_concat.q
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9307) Use metastore warehouse dir variable from conf instead of hard coded dir in encryption test
[
https://issues.apache.org/jira/browse/HIVE-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270486#comment-14270486
]
Dong Chen commented on HIVE-9307:
-
Not necessary. Other .q files result keep same.
The trick is that:
In {{QTestUtil.planMask}}, one is {{LOCATION}}. This cause the table location
in create query in PREHOOK / POSTHOOK masked.
In the 2 .q file of this patch, it is {{location}} so that it is not masked, as
expected.
Use metastore warehouse dir variable from conf instead of hard coded dir in
encryption test
---
Key: HIVE-9307
URL: https://issues.apache.org/jira/browse/HIVE-9307
Project: Hive
Issue Type: Sub-task
Reporter: Dong Chen
Assignee: Dong Chen
Fix For: encryption-branch
Attachments: HIVE-9307.1.patch, HIVE-9307.patch
NO PRECOMMIT TESTS
Use the following variable to get the metastore directory
$\{hiveconf:hive.metastore.warehouse.dir\} in test cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9307) Use metastore warehouse dir variable from conf instead of hard coded dir in encryption test
[
https://issues.apache.org/jira/browse/HIVE-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270488#comment-14270488
]
Brock Noland commented on HIVE-9307:
+1
Use metastore warehouse dir variable from conf instead of hard coded dir in
encryption test
---
Key: HIVE-9307
URL: https://issues.apache.org/jira/browse/HIVE-9307
Project: Hive
Issue Type: Sub-task
Reporter: Dong Chen
Assignee: Dong Chen
Fix For: encryption-branch
Attachments: HIVE-9307.1.patch, HIVE-9307.patch
NO PRECOMMIT TESTS
Use the following variable to get the metastore directory
$\{hiveconf:hive.metastore.warehouse.dir\} in test cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9310) CLI JLine does not flush history back to ~/.hivehistory
[
https://issues.apache.org/jira/browse/HIVE-9310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270515#comment-14270515
]
Navis commented on HIVE-9310:
-
Should it be called in shutdown hook or at least in signal handler either?
CLI JLine does not flush history back to ~/.hivehistory
---
Key: HIVE-9310
URL: https://issues.apache.org/jira/browse/HIVE-9310
Project: Hive
Issue Type: Bug
Components: CLI
Affects Versions: 0.15.0
Reporter: Gopal V
Assignee: Gopal V
Priority: Minor
Attachments: HIVE-9310.1.patch
Hive CLI does not seem to be saving history anymore.
In JLine with the PersistentHistory class, to keep history across sessions,
you need to do {{reader.getHistory().flush()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9242) Many places in CBO code eat exceptions
[ https://issues.apache.org/jira/browse/HIVE-9242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-9242: Resolution: Fixed Fix Version/s: 0.15.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks Brock. Many places in CBO code eat exceptions -- Key: HIVE-9242 URL: https://issues.apache.org/jira/browse/HIVE-9242 Project: Hive Issue Type: Bug Reporter: Brock Noland Priority: Blocker Fix For: 0.15.0 Attachments: HIVE-9242.1.patch.txt I've noticed that there are a number of places in the CBO code which eat exceptions. This is not acceptable. Example: https://github.com/apache/hive/blob/357b473a354aace3bd59b522ad7108be561e9d0f/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java#L274 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9323) Merge from trunk to spark 1/8/2015
[ https://issues.apache.org/jira/browse/HIVE-9323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270701#comment-14270701 ] Szehon Ho commented on HIVE-9323: - Remaining new failures are caused by merge of HIVE-9290 and should be fixed there. Merge from trunk to spark 1/8/2015 -- Key: HIVE-9323 URL: https://issues.apache.org/jira/browse/HIVE-9323 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Szehon Ho Assignee: Szehon Ho Attachments: HIVE-9323-spark.patch, HIVE-9323.2-spark.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9307) Use metastore warehouse dir variable from conf instead of hard coded dir in encryption test
[
https://issues.apache.org/jira/browse/HIVE-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland updated HIVE-9307:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thank you Dong! I have committed this to branch!
Use metastore warehouse dir variable from conf instead of hard coded dir in
encryption test
---
Key: HIVE-9307
URL: https://issues.apache.org/jira/browse/HIVE-9307
Project: Hive
Issue Type: Sub-task
Reporter: Dong Chen
Assignee: Dong Chen
Fix For: encryption-branch
Attachments: HIVE-9307.1.patch, HIVE-9307.patch
NO PRECOMMIT TESTS
Use the following variable to get the metastore directory
$\{hiveconf:hive.metastore.warehouse.dir\} in test cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9249) java.lang.ClassCastException: org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to org.apache.hadoop.hive.common.type.HiveVarchar when joining tables
[
https://issues.apache.org/jira/browse/HIVE-9249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-9249:
--
Attachment: HIVE-9249.hive-14.patch
[~mmccline]: I have regenerated the golden files for the branch. Can you verify
the mapjoin.out files with a look-see?
java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
org.apache.hadoop.hive.common.type.HiveVarchar when joining tables
---
Key: HIVE-9249
URL: https://issues.apache.org/jira/browse/HIVE-9249
Project: Hive
Issue Type: Bug
Components: Vectorization
Affects Versions: 0.14.0
Reporter: Matt McCline
Assignee: Matt McCline
Priority: Critical
Fix For: 0.15.0
Attachments: HIVE-9249.01.patch, HIVE-9249.hive-14.patch
VectorColumnAssignFactory doesn't handle HiveCharWritable /
HiveVarcharWritable objects.
Either:
HiveVarcharWritable cannot be cast to ... HiveVarchar
or
HiveCharWritable cannot be cast to ... HiveChar
{code}
Caused by: java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
org.apache.hadoop.hive.common.type.HiveVarchar
at
org.apache.hadoop.hive.ql.exec.vector.VectorColumnAssignFactory$17.assignObjectValue(VectorColumnAssignFactory.java:417)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.internalForward(VectorMapJoinOperator.java:196)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genAllOneUniqueJoinObject(CommonJoinOperator.java:670)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:748)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:299)
... 24 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-9325) Handle the case of insert overwrite statement with a qualified path that the destination path does not have a schema.
[ https://issues.apache.org/jira/browse/HIVE-9325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland resolved HIVE-9325. Resolution: Fixed Fix Version/s: encryption-branch Thank you Ferd! I have committed this to branch! Handle the case of insert overwrite statement with a qualified path that the destination path does not have a schema. - Key: HIVE-9325 URL: https://issues.apache.org/jira/browse/HIVE-9325 Project: Hive Issue Type: Sub-task Reporter: Ferdinand Xu Assignee: Ferdinand Xu Fix For: encryption-branch Attachments: HIVE-9325-encryption.patch The insert overwrite statement can pass in a path that does not have a schema specified. This will make the isSub method out of work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9323) Merge from trunk to spark 1/8/2015
[
https://issues.apache.org/jira/browse/HIVE-9323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270695#comment-14270695
]
Hive QA commented on HIVE-9323:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12691051/HIVE-9323.2-spark.patch
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 7301 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_memcheck
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_limit_pushdown
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_gby_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_string_concat
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_windowing
org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/623/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/623/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-623/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12691051 - PreCommit-HIVE-SPARK-Build
Merge from trunk to spark 1/8/2015
--
Key: HIVE-9323
URL: https://issues.apache.org/jira/browse/HIVE-9323
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Szehon Ho
Assignee: Szehon Ho
Attachments: HIVE-9323-spark.patch, HIVE-9323.2-spark.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9309) schematool fails on Postgres 8.1
[
https://issues.apache.org/jira/browse/HIVE-9309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270570#comment-14270570
]
Hive QA commented on HIVE-9309:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12690741/HIVE-9309.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6740 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2299/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2299/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2299/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12690741 - PreCommit-HIVE-TRUNK-Build
schematool fails on Postgres 8.1
Key: HIVE-9309
URL: https://issues.apache.org/jira/browse/HIVE-9309
Project: Hive
Issue Type: Bug
Components: Database/Schema
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9309.patch
Postgres upgrade scripts set {{standard_conforming_strings}} which is not
allowed in 8.1:
{code}
ERROR: parameter standard_conforming_strings cannot be changed
(state=55P02,code=0)
{code}
Postgres [8.1 Release
notes|http://www.postgresql.org/docs/8.2/static/release-8-1.html] say that
standard_conforming_strings value is read-only
Postgres [8.2
notes|http://www.postgresql.org/docs/8.2/static/release-8-2.html] say that it
can be set at runtime.
It'd be nice to address this for those still using Postgres 8.1
This patch provides a schemaTool db option postgres.filter.81 which, if
set, filters out the standard_conforming_strings statement from upgrade
scripts.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9324) Reduce side joins failing with IOException from RowContainer.nextBlock
Amareshwari Sriramadasu created HIVE-9324:
-
Summary: Reduce side joins failing with IOException from
RowContainer.nextBlock
Key: HIVE-9324
URL: https://issues.apache.org/jira/browse/HIVE-9324
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.13.1
Reporter: Amareshwari Sriramadasu
We are seeing some reduce side join mapreduce jobs failing with following
exception :
{noformat}
2014-12-14 16:58:51,296 ERROR
org.apache.hadoop.hive.ql.exec.persistence.RowContainer:
org.apache.hadoop.hive.ql.io.RCFile$KeyBuffer@42610e8 read 1 bytes, should read
27264
java.io.IOException: org.apache.hadoop.hive.ql.io.RCFile$KeyBuffer@42610e8 read
1 bytes, should read 27264
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2435)
at
org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:76)
at
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.nextBlock(RowContainer.java:360)
at
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:230)
at
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:74)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:644)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:758)
at
org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:256)
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:216)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:506)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:447)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
2014-12-14 16:58:51,334 FATAL ExecReducer:
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
org.apache.hadoop.hive.ql.io.RCFile$KeyBuffer@42610e8 read 1 bytes, should read
27264
at
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:237)
at
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:74)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:644)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:758)
at
org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:256)
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:216)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:506)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:447)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.io.IOException: org.apache.hadoop.hive.ql.io.RCFile$KeyBuffer@42610e8 read
1 bytes, should read 27264
at
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.nextBlock(RowContainer.java:385)
at
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:230)
... 12 more
Caused by: java.io.IOException:
org.apache.hadoop.hive.ql.io.RCFile$KeyBuffer@42610e8 read 1 bytes, should read
27264
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2435)
at
org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:76)
at
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.nextBlock(RowContainer.java:360)
... 13 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9323) Merge from trunk to spark 1/8/2015
[
https://issues.apache.org/jira/browse/HIVE-9323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270602#comment-14270602
]
Szehon Ho commented on HIVE-9323:
-
The ctas failure is due to HIVE-9094:
{noformat}
2015-01-08 18:43:14,025 WARN [main]: client.SparkClientImpl
(SparkClientImpl.java:init(88)) - Error while waiting for client to connect.
java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException:
Timed out waiting for client connection.
at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37)
at
org.apache.hive.spark.client.SparkClientImpl.init(SparkClientImpl.java:86)
at
org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:75)
at
org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.init(RemoteHiveSparkClient.java:82)
at
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:53)
at
org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:56)
at
org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:128)
at
org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:84)
at
org.apache.hadoop.hive.ql.optimizer.spark.SetSparkReducerParallelism.process(SetSparkReducerParallelism.java:116)
at
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
at
org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.optimizeOperatorPlan(SparkCompiler.java:133)
at
org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:99)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10053)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:189)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:224)
at
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:224)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:420)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:306)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1108)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1156)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1045)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1035)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:206)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:158)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:369)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:304)
at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:880)
at
org.apache.hadoop.hive.cli.TestSparkCliDriver.runTest(TestSparkCliDriver.java:234)
at
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ctas(TestSparkCliDriver.java:166)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at junit.framework.TestCase.runTest(TestCase.java:176)
at junit.framework.TestCase.runBare(TestCase.java:141)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:255)
at junit.framework.TestSuite.run(TestSuite.java:250)
at
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at
[jira] [Commented] (HIVE-9094) TimeoutException when trying get executor count from RSC [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270604#comment-14270604
]
Szehon Ho commented on HIVE-9094:
-
Just as FYI this happened again in latest ptest run :
[https://issues.apache.org/jira/browse/HIVE-9323?focusedCommentId=14270602page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14270602|https://issues.apache.org/jira/browse/HIVE-9323?focusedCommentId=14270602page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14270602]
TimeoutException when trying get executor count from RSC [Spark Branch]
---
Key: HIVE-9094
URL: https://issues.apache.org/jira/browse/HIVE-9094
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Xuefu Zhang
Assignee: Chengxiang Li
Labels: TODOC-SPARK
Fix For: spark-branch
Attachments: HIVE-9094.1-spark.patch, HIVE-9094.2-spark.patch
In
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/532/testReport,
join25.q failed because:
{code}
2014-12-12 19:14:50,084 ERROR [main]: ql.Driver
(SessionState.java:printError(838)) - FAILED: SemanticException Failed to get
spark memory/core info: java.util.concurrent.TimeoutException
org.apache.hadoop.hive.ql.parse.SemanticException: Failed to get spark
memory/core info: java.util.concurrent.TimeoutException
at
org.apache.hadoop.hive.ql.optimizer.spark.SetSparkReducerParallelism.process(SetSparkReducerParallelism.java:120)
at
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
at
org.apache.hadoop.hive.ql.lib.ForwardWalker.walk(ForwardWalker.java:79)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
at
org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.optimizeOperatorPlan(SparkCompiler.java:134)
at
org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:99)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10202)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
at
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:420)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:306)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1108)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1045)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1035)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:199)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:151)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:362)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:297)
at
org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:837)
at
org.apache.hadoop.hive.cli.TestSparkCliDriver.runTest(TestSparkCliDriver.java:234)
at
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join25(TestSparkCliDriver.java:162)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at junit.framework.TestCase.runTest(TestCase.java:176)
at junit.framework.TestCase.runBare(TestCase.java:141)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:255)
at junit.framework.TestSuite.run(TestSuite.java:250)
at
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
[jira] [Updated] (HIVE-9251) SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9251: - Attachment: HIVE-9251.4-spark.patch Update more golden files. SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch] --- Key: HIVE-9251 URL: https://issues.apache.org/jira/browse/HIVE-9251 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Rui Li Assignee: Rui Li Attachments: HIVE-9251.1-spark.patch, HIVE-9251.2-spark.patch, HIVE-9251.3-spark.patch, HIVE-9251.4-spark.patch This may hurt performance or even lead to task failures. For example, spark's netty-based shuffle limits the max frame size to be 2G. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8931) Test TestAccumuloCliDriver is not completing
[
https://issues.apache.org/jira/browse/HIVE-8931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270685#comment-14270685
]
Josh Elser commented on HIVE-8931:
--
Getting back to this, I'm a little stuck here. Backing up, {{hive-metastore}}
is bringing in libthrift-0.9.2 which is breaking things. The qtests ultimately
pull from $CLASSPATH to star the Accumulo minicluster (which includes stuff
from HIVE_HADOOP_TEST_CLASSPATH), that ultimately comes back to the maven test
classpath. Without getting libthrift-0.9.1 somehow on the maven classpath, I
don't know where the libthirft-0.9.1.jar even exists on the local m2 repository
(and thus can't do any trickery to substitute it in place of the
libthrift-0.9.2 dependency). My assumption is that excluding libthrift from the
hive-metastore dependency will break the other qtests (but that is only a
guess).
Assuming I can't exclude libthrift from hive-metastore, I'm not sure what I
could even do at this point aside from introducing a new maven module
specifically for the Accumulo qtests (and gives me carte blanche over the
classpath). [~brocknoland], any ideas?
Test TestAccumuloCliDriver is not completing
Key: HIVE-8931
URL: https://issues.apache.org/jira/browse/HIVE-8931
Project: Hive
Issue Type: Bug
Reporter: Brock Noland
Assignee: Josh Elser
Tests are taking 3 hours due to {{TestAccumuloCliDriver}} not finishing.
Logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1848/failed/TestAccumuloCliDriver/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9323) Merge from trunk to spark 1/8/2015
[ https://issues.apache.org/jira/browse/HIVE-9323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-9323: Resolution: Fixed Fix Version/s: spark-branch Status: Resolved (was: Patch Available) Committed to spark branch. Merge from trunk to spark 1/8/2015 -- Key: HIVE-9323 URL: https://issues.apache.org/jira/browse/HIVE-9323 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Szehon Ho Assignee: Szehon Ho Fix For: spark-branch Attachments: HIVE-9323-spark.patch, HIVE-9323.2-spark.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 29753: Support split sampling for parallel order by
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29753/ --- Review request for hive. Bugs: HIVE-9210 https://issues.apache.org/jira/browse/HIVE-9210 Repository: hive-git Description --- Random sampling cannot sure it covers whole range of input data. See HIVE-7661 Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8264b16 ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java 0ccab02 ql/src/java/org/apache/hadoop/hive/ql/exec/PartitionKeySampler.java 96f4530 ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 2227e6f ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 9f8c091 ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java ac2cc86 ql/src/test/queries/clientpositive/parallel_orderby.q 73c3940 ql/src/test/results/clientpositive/parallel_orderby.q.out 2f4ac8f Diff: https://reviews.apache.org/r/29753/diff/ Testing --- Thanks, Navis Ryu
[jira] [Commented] (HIVE-9324) Reduce side joins failing with IOException from RowContainer.nextBlock
[
https://issues.apache.org/jira/browse/HIVE-9324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270588#comment-14270588
]
Amareshwari Sriramadasu commented on HIVE-9324:
---
More task log :
{noformat}
2014-12-14 16:58:03,905 INFO org.apache.hadoop.hive.ql.exec.mr.ObjectCache:
Ignoring retrieval request: __REDUCE_PLAN__
2014-12-14 16:58:03,945 INFO org.apache.hadoop.hive.ql.log.PerfLogger: PERFLOG
method=deserializePlan from=org.apache.hadoop.hive.ql.exec.Utilities
2014-12-14 16:58:03,945 INFO org.apache.hadoop.hive.ql.exec.Utilities:
Deserializing ReduceWork via kryo
2014-12-14 16:58:04,987 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
/PERFLOG method=deserializePlan start=1418576283945 end=1418576284987
duration=1042 from=org.apache.hadoop.hive.ql.exec.Utilities
2014-12-14 16:58:04,988 INFO org.apache.hadoop.hive.ql.exec.mr.ObjectCache:
Ignoring cache key: __REDUCE_PLAN__
2014-12-14 16:58:05,327 INFO ExecReducer:
JOINId =0
Children
FSId =1
Children
\Children
ParentId = 0 null\Parent
\FS
\Children
Parent\Parent
\JOIN
2014-12-14 16:58:05,327 INFO org.apache.hadoop.hive.ql.exec.JoinOperator:
Initializing Self 0 JOIN
2014-12-14 16:58:05,377 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator:
JOIN
struct_col23:string,_col65:double,_col99:double,_col237:double,_col240:double,_col250:string,_col367:int
totalsz = 7
2014-12-14 16:58:05,377 INFO org.apache.hadoop.hive.ql.exec.JoinOperator:
Operator 0 JOIN initialized
2014-12-14 16:58:05,377 INFO org.apache.hadoop.hive.ql.exec.JoinOperator:
Initializing children of 0 JOIN
2014-12-14 16:58:05,377 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Initializing child 1 FS
2014-12-14 16:58:05,377 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Initializing Self 1 FS
2014-12-14 16:58:05,394 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Operator 1 FS initialized
2014-12-14 16:58:05,394 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Initialization Done 1 FS
2014-12-14 16:58:05,395 INFO org.apache.hadoop.hive.ql.exec.JoinOperator:
Initialization Done 0 JOIN
2014-12-14 16:58:05,401 INFO ExecReducer: ExecReducer: processing 1 rows: used
memory = 242598168
2014-12-14 16:58:05,406 INFO ExecReducer: ExecReducer: processing 10 rows: used
memory = 242759392
2014-12-14 16:58:05,437 INFO ExecReducer: ExecReducer: processing 100 rows:
used memory = 242759392
2014-12-14 16:58:05,657 INFO ExecReducer: ExecReducer: processing 1000 rows:
used memory = 243653240
2014-12-14 16:58:06,976 INFO ExecReducer: ExecReducer: processing 1 rows:
used memory = 247197944
2014-12-14 16:58:07,646 INFO ExecReducer: ExecReducer: processing 10 rows:
used memory = 277801256
2014-12-14 16:58:11,511 INFO ExecReducer: ExecReducer: processing 100 rows:
used memory = 283150744
2014-12-14 16:58:14,993 INFO ExecReducer: ExecReducer: processing 200 rows:
used memory = 293036992
2014-12-14 16:58:18,497 INFO ExecReducer: ExecReducer: processing 300 rows:
used memory = 311449488
2014-12-14 16:58:20,815 INFO ExecReducer: ExecReducer: processing 400 rows:
used memory = 285251752
2014-12-14 16:58:26,460 INFO ExecReducer: ExecReducer: processing 500 rows:
used memory = 328223864
2014-12-14 16:58:29,412 INFO ExecReducer: ExecReducer: processing 600 rows:
used memory = 263175576
2014-12-14 16:58:31,331 INFO ExecReducer: ExecReducer: processing 700 rows:
used memory = 282021320
2014-12-14 16:58:35,099 INFO ExecReducer: ExecReducer: processing 800 rows:
used memory = 299301184
2014-12-14 16:58:37,981 INFO ExecReducer: ExecReducer: processing 900 rows:
used memory = 306925648
2014-12-14 16:58:40,506 INFO ExecReducer: ExecReducer: processing 1000
rows: used memory = 307407920
2014-12-14 16:58:42,242 INFO ExecReducer: ExecReducer: processing 1100
rows: used memory = 304664048
2014-12-14 16:58:46,142 INFO ExecReducer: ExecReducer: processing 1200
rows: used memory = 298347024
2014-12-14 16:58:48,549 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator:
table 0 has 1000 rows for join key [003b9de7876541c2bcce9029ff0d3873]
2014-12-14 16:58:48,622 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator:
table 0 has 2000 rows for join key [003b9de7876541c2bcce9029ff0d3873]
2014-12-14 16:58:48,677 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator:
table 0 has 4000 rows for join key [003b9de7876541c2bcce9029ff0d3873]
2014-12-14 16:58:48,679 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Final Path: FS
hdfs://data-grill300-null.arshad.ev1.inmobi.com:8020/tmp/hive-dataqa/hive_2014-12-14_16-49-14_996_1630664550753106415-32/_tmp.-mr-10002/00_0
2014-12-14 16:58:48,680 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Writing to temp file: FS
[jira] [Commented] (HIVE-3405) UDF initcap to obtain a string with the first letter of each word in uppercase other letters in lowercase
[ https://issues.apache.org/jira/browse/HIVE-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270644#comment-14270644 ] Jason Dere commented on HIVE-3405: -- +1 if precommit tests still look good UDF initcap to obtain a string with the first letter of each word in uppercase other letters in lowercase - Key: HIVE-3405 URL: https://issues.apache.org/jira/browse/HIVE-3405 Project: Hive Issue Type: New Feature Components: UDF Affects Versions: 0.8.1, 0.9.0, 0.9.1, 0.10.0, 0.11.0, 0.13.0, 0.14.0, 0.15.0, 0.14.1 Reporter: Archana Nair Assignee: Alexander Pivovarov Labels: patch Attachments: HIVE-3405.1.patch.txt, HIVE-3405.2.patch, HIVE-3405.3.patch, HIVE-3405.4.patch, HIVE-3405.5.patch Hive current releases lacks a INITCAP function which returns String with first letter of the word in uppercase.INITCAP returns String, with the first letter of each word in uppercase, all other letters in same case. Words are delimited by white space.This will be useful report generation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9325) Handle the case of insert overwrite statement with a qualified path that the destination path does not have a schema.
Ferdinand Xu created HIVE-9325: -- Summary: Handle the case of insert overwrite statement with a qualified path that the destination path does not have a schema. Key: HIVE-9325 URL: https://issues.apache.org/jira/browse/HIVE-9325 Project: Hive Issue Type: Sub-task Reporter: Ferdinand Xu Assignee: Ferdinand Xu The insert overwrite statement can pass in a path that does not have a schema specified. This will make the isSub method out of work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9325) Handle the case of insert overwrite statement with a qualified path that the destination path does not have a schema.
[ https://issues.apache.org/jira/browse/HIVE-9325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-9325: --- Attachment: HIVE-9325-encryption.patch Handle the case of insert overwrite statement with a qualified path that the destination path does not have a schema. - Key: HIVE-9325 URL: https://issues.apache.org/jira/browse/HIVE-9325 Project: Hive Issue Type: Sub-task Reporter: Ferdinand Xu Assignee: Ferdinand Xu Attachments: HIVE-9325-encryption.patch The insert overwrite statement can pass in a path that does not have a schema specified. This will make the isSub method out of work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-9324) Reduce side joins failing with IOException from RowContainer.nextBlock
[
https://issues.apache.org/jira/browse/HIVE-9324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270588#comment-14270588
]
Amareshwari Sriramadasu edited comment on HIVE-9324 at 1/9/15 5:54 AM:
---
More task log :
{noformat}
2014-12-14 16:58:03,905 INFO org.apache.hadoop.hive.ql.exec.mr.ObjectCache:
Ignoring retrieval request: __REDUCE_PLAN__
2014-12-14 16:58:03,945 INFO org.apache.hadoop.hive.ql.log.PerfLogger: PERFLOG
method=deserializePlan from=org.apache.hadoop.hive.ql.exec.Utilities
2014-12-14 16:58:03,945 INFO org.apache.hadoop.hive.ql.exec.Utilities:
Deserializing ReduceWork via kryo
2014-12-14 16:58:04,987 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
/PERFLOG method=deserializePlan start=1418576283945 end=1418576284987
duration=1042 from=org.apache.hadoop.hive.ql.exec.Utilities
2014-12-14 16:58:04,988 INFO org.apache.hadoop.hive.ql.exec.mr.ObjectCache:
Ignoring cache key: __REDUCE_PLAN__
2014-12-14 16:58:05,327 INFO ExecReducer:
JOINId =0
Children
FSId =1
Children
\Children
ParentId = 0 null\Parent
\FS
\Children
Parent\Parent
\JOIN
2014-12-14 16:58:05,327 INFO org.apache.hadoop.hive.ql.exec.JoinOperator:
Initializing Self 0 JOIN
2014-12-14 16:58:05,377 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator:
JOIN
struct_col23:string,_col65:double,_col99:double,_col237:double,_col240:double,_col250:string,_col367:int
totalsz = 7
2014-12-14 16:58:05,377 INFO org.apache.hadoop.hive.ql.exec.JoinOperator:
Operator 0 JOIN initialized
2014-12-14 16:58:05,377 INFO org.apache.hadoop.hive.ql.exec.JoinOperator:
Initializing children of 0 JOIN
2014-12-14 16:58:05,377 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Initializing child 1 FS
2014-12-14 16:58:05,377 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Initializing Self 1 FS
2014-12-14 16:58:05,394 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Operator 1 FS initialized
2014-12-14 16:58:05,394 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Initialization Done 1 FS
2014-12-14 16:58:05,395 INFO org.apache.hadoop.hive.ql.exec.JoinOperator:
Initialization Done 0 JOIN
2014-12-14 16:58:05,401 INFO ExecReducer: ExecReducer: processing 1 rows: used
memory = 242598168
2014-12-14 16:58:05,406 INFO ExecReducer: ExecReducer: processing 10 rows: used
memory = 242759392
2014-12-14 16:58:05,437 INFO ExecReducer: ExecReducer: processing 100 rows:
used memory = 242759392
2014-12-14 16:58:05,657 INFO ExecReducer: ExecReducer: processing 1000 rows:
used memory = 243653240
2014-12-14 16:58:06,976 INFO ExecReducer: ExecReducer: processing 1 rows:
used memory = 247197944
2014-12-14 16:58:07,646 INFO ExecReducer: ExecReducer: processing 10 rows:
used memory = 277801256
2014-12-14 16:58:11,511 INFO ExecReducer: ExecReducer: processing 100 rows:
used memory = 283150744
2014-12-14 16:58:14,993 INFO ExecReducer: ExecReducer: processing 200 rows:
used memory = 293036992
2014-12-14 16:58:18,497 INFO ExecReducer: ExecReducer: processing 300 rows:
used memory = 311449488
2014-12-14 16:58:20,815 INFO ExecReducer: ExecReducer: processing 400 rows:
used memory = 285251752
2014-12-14 16:58:26,460 INFO ExecReducer: ExecReducer: processing 500 rows:
used memory = 328223864
2014-12-14 16:58:29,412 INFO ExecReducer: ExecReducer: processing 600 rows:
used memory = 263175576
2014-12-14 16:58:31,331 INFO ExecReducer: ExecReducer: processing 700 rows:
used memory = 282021320
2014-12-14 16:58:35,099 INFO ExecReducer: ExecReducer: processing 800 rows:
used memory = 299301184
2014-12-14 16:58:37,981 INFO ExecReducer: ExecReducer: processing 900 rows:
used memory = 306925648
2014-12-14 16:58:40,506 INFO ExecReducer: ExecReducer: processing 1000
rows: used memory = 307407920
2014-12-14 16:58:42,242 INFO ExecReducer: ExecReducer: processing 1100
rows: used memory = 304664048
2014-12-14 16:58:46,142 INFO ExecReducer: ExecReducer: processing 1200
rows: used memory = 298347024
2014-12-14 16:58:48,549 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator:
table 0 has 1000 rows for join key [003b9de7876541c2bcce9029ff0d3873]
2014-12-14 16:58:48,622 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator:
table 0 has 2000 rows for join key [003b9de7876541c2bcce9029ff0d3873]
2014-12-14 16:58:48,677 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator:
table 0 has 4000 rows for join key [003b9de7876541c2bcce9029ff0d3873]
2014-12-14 16:58:48,679 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Final Path: FS
hdfs://test-machine:8020/tmp/hive-dataqa/hive_2014-12-14_16-49-14_996_1630664550753106415-32/_tmp.-mr-10002/00_0
2014-12-14 16:58:48,680 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Writing to temp file: FS
[jira] [Updated] (HIVE-9323) Merge from trunk to spark 1/8/2015
[ https://issues.apache.org/jira/browse/HIVE-9323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-9323: Attachment: HIVE-9323.2-spark.patch Regenerate the golden files for auto_join7, join7, and groupby_cube1 due to various changes in the trunk. Merge from trunk to spark 1/8/2015 -- Key: HIVE-9323 URL: https://issues.apache.org/jira/browse/HIVE-9323 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Szehon Ho Assignee: Szehon Ho Attachments: HIVE-9323-spark.patch, HIVE-9323.2-spark.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9325) Handle the case of insert overwrite statement with a qualified path that the destination path does not have a schema.
[ https://issues.apache.org/jira/browse/HIVE-9325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270674#comment-14270674 ] Brock Noland commented on HIVE-9325: +1 Handle the case of insert overwrite statement with a qualified path that the destination path does not have a schema. - Key: HIVE-9325 URL: https://issues.apache.org/jira/browse/HIVE-9325 Project: Hive Issue Type: Sub-task Reporter: Ferdinand Xu Assignee: Ferdinand Xu Attachments: HIVE-9325-encryption.patch The insert overwrite statement can pass in a path that does not have a schema specified. This will make the isSub method out of work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9323) Merge from trunk to spark 1/8/2015
[
https://issues.apache.org/jira/browse/HIVE-9323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270719#comment-14270719
]
Chengxiang Li commented on HIVE-9323:
-
[~Szehon], I take a look at the hive log, the failed reason is quite strange
and a little different from HIVE-9094. HIVE-9094 failed due to get executor
count timeout because of spark cluster launch time is longer than spark client
future timeout interval(5s, and 30s after HIVE-9094), while this timeout
failure is due to RemoteDriver is not response in time(spark client wait 10s
for RemoteDriver to register).
From the hive.log, RemoteDriver processor is launched at 2015-01-08
18:43:03,938
{noformat}
2015-01-08 18:43:03,938 DEBUG [main]: client.SparkClientImpl
(SparkClientImpl.java:startDriver(298)) - Running client driver with argv:
/home/hiveptest/54.177.142.77-hiveptest-1/apache-svn-spark-source/itests/qtest-spark/../../itests/qtest-spark/target/spark/bin/spark-submit
--properties-file
/home/hiveptest/54.177.142.77-hiveptest-1/apache-svn-spark-source/itests/qtest-spark/target/tmp/spark-submit.1097041260552550316.properties
--class org.apache.hive.spark.client.RemoteDriver
/home/hiveptest/54.177.142.77-hiveptest-1/maven/org/apache/hive/hive-exec/0.15.0-SNAPSHOT/hive-exec-0.15.0-SNAPSHOT.jar
--remote-host ip-10-228-130-250.us-west-1.compute.internal --remote-port 40406
{noformat}
In spark.log, RemoteDriver register back to SparkClient at 2015-01-08
18:43:13,891 which should just more than timeout interval which is 10s.
{noformat}
2015-01-08 18:43:13,891 DEBUG [Driver-RPC-Handler-0]: rpc.RpcDispatcher
(RpcDispatcher.java:registerRpc(185)) - [DriverProtocol] Registered outstanding
rpc 0 (org.apache.hive.spark.client.rpc.Rpc$Hello).
{noformat}
The strange thing is that RemoteDriver processor is unusual slow, as it's
launched at 2015-01-08 18:43:03,938 but we get it's first debug info at
2015-01-08 18:43:13,161, RemoteDriver hardly do anything before this debug info.
{noformat}
2015-01-08 18:43:13,161 INFO [main]: client.RemoteDriver
(RemoteDriver.java:init(118)) - Connecting to:
ip-10-228-130-250.us-west-1.compute.internal:40406
{noformat}
I not sure why this happens, but this should be a quite rarely case, we can
check whether it happens again, besides expand timeout interval, i don't have a
good solution for this issue now.
Merge from trunk to spark 1/8/2015
--
Key: HIVE-9323
URL: https://issues.apache.org/jira/browse/HIVE-9323
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Szehon Ho
Assignee: Szehon Ho
Fix For: spark-branch
Attachments: HIVE-9323-spark.patch, HIVE-9323.2-spark.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9318) Add UnionMerge rule on cbo path
[ https://issues.apache.org/jira/browse/HIVE-9318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-9318: --- Component/s: Logical Optimizer CBO Add UnionMerge rule on cbo path --- Key: HIVE-9318 URL: https://issues.apache.org/jira/browse/HIVE-9318 Project: Hive Issue Type: Improvement Components: CBO, Logical Optimizer Affects Versions: 0.14.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-9318.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9318) Add UnionMerge rule on cbo path
[ https://issues.apache.org/jira/browse/HIVE-9318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-9318: --- Affects Version/s: 0.14.0 Status: Patch Available (was: Open) Add UnionMerge rule on cbo path --- Key: HIVE-9318 URL: https://issues.apache.org/jira/browse/HIVE-9318 Project: Hive Issue Type: Improvement Components: CBO, Logical Optimizer Affects Versions: 0.14.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-9318.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9322) Make null-checks consistent for MapObjectInspector subclasses.
[
https://issues.apache.org/jira/browse/HIVE-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-9322:
---
Attachment: HIVE-9322.1.patch
Make null-checks consistent for MapObjectInspector subclasses.
--
Key: HIVE-9322
URL: https://issues.apache.org/jira/browse/HIVE-9322
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Affects Versions: 0.14.0
Reporter: Mithun Radhakrishnan
Assignee: Mithun Radhakrishnan
Priority: Minor
Attachments: HIVE-9322.1.patch
{{LazyBinaryMapObjectInspector}}, {{DeepParquetHiveMapInspector}}, etc. check
both the map-column value and the map-key for null, before dereferencing
them. {{OrcMapObjectInspector}} and {{LazyMapObjectInspector}} do not.
This patch brings them all in sync. Might not be a real problem, unless (for
example) the lookup key is itself a (possibly null) value from another column.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9322) Make null-checks consistent for MapObjectInspector subclasses.
[
https://issues.apache.org/jira/browse/HIVE-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-9322:
---
Status: Patch Available (was: Open)
Make null-checks consistent for MapObjectInspector subclasses.
--
Key: HIVE-9322
URL: https://issues.apache.org/jira/browse/HIVE-9322
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Affects Versions: 0.14.0
Reporter: Mithun Radhakrishnan
Assignee: Mithun Radhakrishnan
Priority: Minor
Attachments: HIVE-9322.1.patch
{{LazyBinaryMapObjectInspector}}, {{DeepParquetHiveMapInspector}}, etc. check
both the map-column value and the map-key for null, before dereferencing
them. {{OrcMapObjectInspector}} and {{LazyMapObjectInspector}} do not.
This patch brings them all in sync. Might not be a real problem, unless (for
example) the lookup key is itself a (possibly null) value from another column.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8696) HCatClientHMSImpl doesn't use a Retrying-HiveMetastoreClient.
[ https://issues.apache.org/jira/browse/HIVE-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mithun Radhakrishnan updated HIVE-8696: --- Status: Patch Available (was: Open) HCatClientHMSImpl doesn't use a Retrying-HiveMetastoreClient. - Key: HIVE-8696 URL: https://issues.apache.org/jira/browse/HIVE-8696 Project: Hive Issue Type: Bug Components: HCatalog, Metastore Affects Versions: 0.13.1, 0.12.0 Reporter: Mithun Radhakrishnan Assignee: Mithun Radhakrishnan Attachments: HIVE-8696.1.patch The HCatClientHMSImpl doesn't use a RetryingHiveMetastoreClient. Users of the HCatClient API that log in through keytabs will fail without retry, when their TGTs expire. The fix is inbound. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9323) Merge from trunk to spark 1/8/2015
Szehon Ho created HIVE-9323: --- Summary: Merge from trunk to spark 1/8/2015 Key: HIVE-9323 URL: https://issues.apache.org/jira/browse/HIVE-9323 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Szehon Ho Assignee: Szehon Ho -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8853) Make vectorization work with Spark [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270418#comment-14270418
]
Brock Noland commented on HIVE-8853:
[~jxiang] I took some thread dumps of an executor JVM during execution with
vectorization turned on and I saw a ton of thread dumps here like the ones
below.
{noformat}
Executor task launch worker-4 daemon prio=10 tid=0x7f8394048800
nid=0x707a runnable [0x7f8457dfb000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked 0x000281c57b70 (a sun.nio.ch.Util$2)
- locked 0x000281c57b80 (a java.util.Collections$UnmodifiableSet)
- locked 0x000281c57b28 (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.readChannelFully(PacketReceiver.java:258)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:209)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:171)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:102)
at
org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:186)
at
org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:146)
- locked 0x000718424118 (a
org.apache.hadoop.hdfs.RemoteBlockReader2)
at
org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:693)
at
org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:749)
- eliminated 0x0007184169b8 (a
org.apache.hadoop.hdfs.DFSInputStream)
at
org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:806)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:847)
- locked 0x0007184169b8 (a org.apache.hadoop.hdfs.DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hive.com.esotericsoftware.kryo.io.Input.fill(Input.java:146)
at
org.apache.hive.com.esotericsoftware.kryo.io.Input.require(Input.java:178)
at
org.apache.hive.com.esotericsoftware.kryo.io.Input.readUtf8_slow(Input.java:542)
at
org.apache.hive.com.esotericsoftware.kryo.io.Input.readUtf8(Input.java:535)
at
org.apache.hive.com.esotericsoftware.kryo.io.Input.readString(Input.java:465)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$StringSerializer.read(DefaultSerializers.java:171)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$StringSerializer.read(DefaultSerializers.java:160)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at
[jira] [Commented] (HIVE-9274) STORED BY not setting input and output formats correctly
[ https://issues.apache.org/jira/browse/HIVE-9274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270096#comment-14270096 ] Thomas Friedrich commented on HIVE-9274: Looks like this behavior has changed due to HIVE-6584. In DDLTask the input and output format classes are only persisted when explicitly specified. I wonder if the documentation in https://cwiki.apache.org/confluence/display/Hive/StorageHandlers should be updated (under open issues: Names of helper classes such as input format and output format are saved into the metastore based on what the storage handler returns during CREATE TABLE; it would be better to leave these null in case they are changed later as part of a handler upgrade). STORED BY not setting input and output formats correctly Key: HIVE-9274 URL: https://issues.apache.org/jira/browse/HIVE-9274 Project: Hive Issue Type: Bug Affects Versions: 0.14.0 Reporter: Thomas Friedrich Priority: Minor When creating a new table using a storage handler via the STORED BY clause, for example using the HBaseStorageHandler, the input and output formats are set to null instead of the correct formats: CREATE TABLE hbase_table_1(key int, value string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES (hbase.columns.mapping = :key,cf1:val) TBLPROPERTIES (hbase.table.name = xyz); describe formatted hbase_table_1: # Storage Information SerDe Library: org.apache.hadoop.hive.hbase.HBaseSerDe InputFormat:null OutputFormat: null In older Hive versions, it would set the correct formats. Could it be related to the changes in HIVE-5976? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8327) mvn site -Pfindbugs
[
https://issues.apache.org/jira/browse/HIVE-8327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270113#comment-14270113
]
Gopal V commented on HIVE-8327:
---
Yes, all other targets are unaffected.
Before we run regressions on this, we need to get the warnings list to a
known-set of false-positives which we can ignore.
The most critical ones I noticed were classes which implement equals() but not
hashCode(), which when used in a Set return bad results.
{code}
org.apache.hadoop.hive.ql.hooks.ReadEntity defines equals but not hashCode
BAD_PRACTICEHE_EQUALS_NO_HASHCODE 143 Medium
{code}
mvn site -Pfindbugs
---
Key: HIVE-8327
URL: https://issues.apache.org/jira/browse/HIVE-8327
Project: Hive
Issue Type: Test
Components: Diagnosability
Reporter: Gopal V
Assignee: Gopal V
Fix For: 0.15.0
Attachments: HIVE-8327.1.patch, HIVE-8327.2.patch, ql-findbugs.html
HIVE-3099 originally added findbugs into the old ant build.
Get basic findbugs working for the maven build.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9249) java.lang.ClassCastException: org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to org.apache.hadoop.hive.common.type.HiveVarchar when joining tables
[
https://issues.apache.org/jira/browse/HIVE-9249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-9249:
--
Fix Version/s: 0.15.0
java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
org.apache.hadoop.hive.common.type.HiveVarchar when joining tables
---
Key: HIVE-9249
URL: https://issues.apache.org/jira/browse/HIVE-9249
Project: Hive
Issue Type: Bug
Components: Vectorization
Affects Versions: 0.14.0
Reporter: Matt McCline
Assignee: Matt McCline
Priority: Critical
Fix For: 0.15.0
Attachments: HIVE-9249.01.patch
VectorColumnAssignFactory doesn't handle HiveCharWritable /
HiveVarcharWritable objects.
Either:
HiveVarcharWritable cannot be cast to ... HiveVarchar
or
HiveCharWritable cannot be cast to ... HiveChar
{code}
Caused by: java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
org.apache.hadoop.hive.common.type.HiveVarchar
at
org.apache.hadoop.hive.ql.exec.vector.VectorColumnAssignFactory$17.assignObjectValue(VectorColumnAssignFactory.java:417)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.internalForward(VectorMapJoinOperator.java:196)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genAllOneUniqueJoinObject(CommonJoinOperator.java:670)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:748)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:299)
... 24 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8327) mvn site -Pfindbugs
[ https://issues.apache.org/jira/browse/HIVE-8327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270123#comment-14270123 ] Prasanth Jayachandran commented on HIVE-8327: - We have been bitten by this in other cases as well. I remember LazyMap implementing equals() but not hashCode() causing issues 5 years later. Its definitely good to catch such cases before committing patches. 4 min spent in QA better than hours later :) mvn site -Pfindbugs --- Key: HIVE-8327 URL: https://issues.apache.org/jira/browse/HIVE-8327 Project: Hive Issue Type: Test Components: Diagnosability Reporter: Gopal V Assignee: Gopal V Fix For: 0.15.0 Attachments: HIVE-8327.1.patch, HIVE-8327.2.patch, ql-findbugs.html HIVE-3099 originally added findbugs into the old ant build. Get basic findbugs working for the maven build. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9274) STORED BY not setting input and output formats correctly
[ https://issues.apache.org/jira/browse/HIVE-9274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Friedrich updated HIVE-9274: --- Priority: Minor (was: Major) STORED BY not setting input and output formats correctly Key: HIVE-9274 URL: https://issues.apache.org/jira/browse/HIVE-9274 Project: Hive Issue Type: Bug Affects Versions: 0.14.0 Reporter: Thomas Friedrich Priority: Minor When creating a new table using a storage handler via the STORED BY clause, for example using the HBaseStorageHandler, the input and output formats are set to null instead of the correct formats: CREATE TABLE hbase_table_1(key int, value string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES (hbase.columns.mapping = :key,cf1:val) TBLPROPERTIES (hbase.table.name = xyz); describe formatted hbase_table_1: # Storage Information SerDe Library: org.apache.hadoop.hive.hbase.HBaseSerDe InputFormat:null OutputFormat: null In older Hive versions, it would set the correct formats. Could it be related to the changes in HIVE-5976? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9249) java.lang.ClassCastException: org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to org.apache.hadoop.hive.common.type.HiveVarchar when joining tables
[
https://issues.apache.org/jira/browse/HIVE-9249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270119#comment-14270119
]
Gopal V commented on HIVE-9249:
---
Committed to 0.15.0.
Not committed to 0.14.1 because of plan formatting changes in the golden files.
{code}
clientpositive/vector_varchar_mapjoin1.q.out
148,150d147
condition expressions:
0 {c1}
1 {c1} {c2}
167,169d163
condition expressions:
0 {c1} {c2}
1 {c1} {c2}
244,246d237
condition expressions:
0 {c1}
1 {c1} {c2}
263,265d253
condition expressions:
0 {c1} {c2}
1 {c1} {c2}
342,344d329
condition expressions:
0 {c1} {c2}
1 {c1}
361,363d345
condition expressions:
0 {c1} {c2}
1 {c1} {c2}
{code}
Will generate a new patch, in a few mins.
java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
org.apache.hadoop.hive.common.type.HiveVarchar when joining tables
---
Key: HIVE-9249
URL: https://issues.apache.org/jira/browse/HIVE-9249
Project: Hive
Issue Type: Bug
Components: Vectorization
Affects Versions: 0.14.0
Reporter: Matt McCline
Assignee: Matt McCline
Priority: Critical
Fix For: 0.15.0
Attachments: HIVE-9249.01.patch
VectorColumnAssignFactory doesn't handle HiveCharWritable /
HiveVarcharWritable objects.
Either:
HiveVarcharWritable cannot be cast to ... HiveVarchar
or
HiveCharWritable cannot be cast to ... HiveChar
{code}
Caused by: java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
org.apache.hadoop.hive.common.type.HiveVarchar
at
org.apache.hadoop.hive.ql.exec.vector.VectorColumnAssignFactory$17.assignObjectValue(VectorColumnAssignFactory.java:417)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.internalForward(VectorMapJoinOperator.java:196)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genAllOneUniqueJoinObject(CommonJoinOperator.java:670)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:748)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:299)
... 24 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9318) Add UnionMerge rule on cbo path
[ https://issues.apache.org/jira/browse/HIVE-9318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-9318: --- Attachment: HIVE-9318.patch If there are subsequent unions, we can merge them. Add UnionMerge rule on cbo path --- Key: HIVE-9318 URL: https://issues.apache.org/jira/browse/HIVE-9318 Project: Hive Issue Type: Improvement Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-9318.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9320) Add UnionEliminatorRule on cbo path
[ https://issues.apache.org/jira/browse/HIVE-9320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-9320: --- Status: Patch Available (was: Open) Add UnionEliminatorRule on cbo path --- Key: HIVE-9320 URL: https://issues.apache.org/jira/browse/HIVE-9320 Project: Hive Issue Type: Improvement Components: CBO, Logical Optimizer Affects Versions: 0.14.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-9320.patch Shorten the pipeline, where possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9320) Add UnionEliminatorRule on cbo path
[ https://issues.apache.org/jira/browse/HIVE-9320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-9320: --- Attachment: HIVE-9320.patch Add UnionEliminatorRule on cbo path --- Key: HIVE-9320 URL: https://issues.apache.org/jira/browse/HIVE-9320 Project: Hive Issue Type: Improvement Components: CBO, Logical Optimizer Affects Versions: 0.14.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-9320.patch Shorten the pipeline, where possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 29733: HIVE-9319 : Cleanup Modified Files [Spark Branch]
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29733/#review67348 --- Ship it! Ship It! - Xuefu Zhang On Jan. 9, 2015, 12:01 a.m., Szehon Ho wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29733/ --- (Updated Jan. 9, 2015, 12:01 a.m.) Review request for hive and Xuefu Zhang. Repository: hive-git Description --- Note that this limits cleanup to lines of code changed in spark-branch in the merge to trunk, not cleanup of all of the modified files, in order to reduce merge conflicts. Diffs - ql/src/java/org/apache/hadoop/hive/ql/Driver.java fa40082 ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java b25a639 ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java ee42f4c ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java abdb6af ql/src/java/org/apache/hadoop/hive/ql/io/HiveKey.java 33aeda4 ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 6f216c9 ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java a6d5c62 ql/src/java/org/apache/hadoop/hive/ql/optimizer/unionproc/UnionProcessor.java fec6822 ql/src/java/org/apache/hadoop/hive/ql/parse/MapReduceCompiler.java 1b6de64 ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 1efbb12 ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java 4582678 ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 076d2fa shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java f1743ae Diff: https://reviews.apache.org/r/29733/diff/ Testing --- Thanks, Szehon Ho
[jira] [Commented] (HIVE-9319) Cleanup Modified Files [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270303#comment-14270303 ] Xuefu Zhang commented on HIVE-9319: --- +1 pending on test Cleanup Modified Files [Spark Branch] - Key: HIVE-9319 URL: https://issues.apache.org/jira/browse/HIVE-9319 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Szehon Ho Assignee: Szehon Ho Priority: Minor Attachments: HIVE-9319-spark.patch Cleanup the code that is modified based on checkstyle/TODO/warnings. It is a follow-up of HIVE-9281 which is for new files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9321) Notification message size can be arbitrarily long, DbNotificationListener limits to 1024
Sushanth Sowmyan created HIVE-9321: -- Summary: Notification message size can be arbitrarily long, DbNotificationListener limits to 1024 Key: HIVE-9321 URL: https://issues.apache.org/jira/browse/HIVE-9321 Project: Hive Issue Type: Bug Reporter: Sushanth Sowmyan Assignee: Alan Gates As part of HIVE-9174, a new notification listener, DbNotificationListener was introduced that stores notification messages in the metastore. Now, the message field of a Notification event can be arbitrarily long, since it is a JSON-serialized message of what event actually occurred, and an AddPartitionMessage which captured the result of an add_partitions can be arbitrarily long (as long as the number of partitions that were added). However, the message field for the storage in db is limited to VARCHAR(1024), which it will easily break. We need to fix this somehow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9319) Cleanup Modified Files [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-9319: Resolution: Fixed Fix Version/s: spark-branch Status: Resolved (was: Patch Available) Committed to spark-branch. Thanks Xuefu for review. Cleanup Modified Files [Spark Branch] - Key: HIVE-9319 URL: https://issues.apache.org/jira/browse/HIVE-9319 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Szehon Ho Assignee: Szehon Ho Priority: Minor Fix For: spark-branch Attachments: HIVE-9319-spark.patch Cleanup the code that is modified based on checkstyle/TODO/warnings. It is a follow-up of HIVE-9281 which is for new files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8988) Support advanced aggregation in Hive to Calcite path
[ https://issues.apache.org/jira/browse/HIVE-8988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-8988: - Resolution: Fixed Status: Resolved (was: Patch Available) Support advanced aggregation in Hive to Calcite path - Key: HIVE-8988 URL: https://issues.apache.org/jira/browse/HIVE-8988 Project: Hive Issue Type: Improvement Affects Versions: 0.15.0 Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Labels: grouping, logical, optiq Fix For: 0.15.0 Attachments: HIVE-8988.01.patch, HIVE-8988.02.patch, HIVE-8988.03.patch, HIVE-8988.04.patch, HIVE-8988.05.patch, HIVE-8988.05.patch, HIVE-8988.06.patch, HIVE-8988.07.patch, HIVE-8988.patch CLEAR LIBRARY CACHE To close the gap between Hive and Calcite, we need to support the translation of GroupingSets into Calcite; currently this is not implemented. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8988) Support advanced aggregation in Hive to Calcite path
[ https://issues.apache.org/jira/browse/HIVE-8988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270133#comment-14270133 ] Laljo John Pullokkaran commented on HIVE-8988: -- Committed to trunk, thanks [~jcamachorodriguez] Support advanced aggregation in Hive to Calcite path - Key: HIVE-8988 URL: https://issues.apache.org/jira/browse/HIVE-8988 Project: Hive Issue Type: Improvement Affects Versions: 0.15.0 Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Labels: grouping, logical, optiq Fix For: 0.15.0 Attachments: HIVE-8988.01.patch, HIVE-8988.02.patch, HIVE-8988.03.patch, HIVE-8988.04.patch, HIVE-8988.05.patch, HIVE-8988.05.patch, HIVE-8988.06.patch, HIVE-8988.07.patch, HIVE-8988.patch CLEAR LIBRARY CACHE To close the gap between Hive and Calcite, we need to support the translation of GroupingSets into Calcite; currently this is not implemented. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9316) TestSqoop tests in WebHCat testsuite hardcode libdir path to hdfs
Deepesh Khandelwal created HIVE-9316: Summary: TestSqoop tests in WebHCat testsuite hardcode libdir path to hdfs Key: HIVE-9316 URL: https://issues.apache.org/jira/browse/HIVE-9316 Project: Hive Issue Type: Bug Components: Tests, WebHCat Reporter: Deepesh Khandelwal Assignee: Deepesh Khandelwal Priority: Minor Fix For: 0.15.0 Currently the TestSqoop tests in WebHCat Perl based testsuite has hdfs:// prefix in the jdbc jar path in libdir, we should remove this to enable it to run against other file systems. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9316) TestSqoop tests in WebHCat testsuite hardcode libdir path to hdfs
[ https://issues.apache.org/jira/browse/HIVE-9316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepesh Khandelwal updated HIVE-9316: - Attachment: HIVE-9316.1.patch Attaching the patch that removes the prefix. Please review. TestSqoop tests in WebHCat testsuite hardcode libdir path to hdfs - Key: HIVE-9316 URL: https://issues.apache.org/jira/browse/HIVE-9316 Project: Hive Issue Type: Bug Components: Tests, WebHCat Reporter: Deepesh Khandelwal Assignee: Deepesh Khandelwal Priority: Minor Fix For: 0.15.0 Attachments: HIVE-9316.1.patch Currently the TestSqoop tests in WebHCat Perl based testsuite has hdfs:// prefix in the jdbc jar path in libdir, we should remove this to enable it to run against other file systems. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9316) TestSqoop tests in WebHCat testsuite hardcode libdir path to hdfs
[ https://issues.apache.org/jira/browse/HIVE-9316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepesh Khandelwal updated HIVE-9316: - Description: Currently the TestSqoop tests in WebHCat Perl based testsuite has hdfs:// prefix in the jdbc jar path in libdir, we should remove this to enable it to run against other file systems. NO PRECOMMIT TESTS was:Currently the TestSqoop tests in WebHCat Perl based testsuite has hdfs:// prefix in the jdbc jar path in libdir, we should remove this to enable it to run against other file systems. TestSqoop tests in WebHCat testsuite hardcode libdir path to hdfs - Key: HIVE-9316 URL: https://issues.apache.org/jira/browse/HIVE-9316 Project: Hive Issue Type: Bug Components: Tests, WebHCat Reporter: Deepesh Khandelwal Assignee: Deepesh Khandelwal Priority: Minor Fix For: 0.15.0 Attachments: HIVE-9316.1.patch Currently the TestSqoop tests in WebHCat Perl based testsuite has hdfs:// prefix in the jdbc jar path in libdir, we should remove this to enable it to run against other file systems. NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9304) [Refactor] remove unused method in SemAly
[
https://issues.apache.org/jira/browse/HIVE-9304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270152#comment-14270152
]
Hive QA commented on HIVE-9304:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12690693/HIVE-9304.patch
{color:green}SUCCESS:{color} +1 6733 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2295/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2295/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2295/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12690693 - PreCommit-HIVE-TRUNK-Build
[Refactor] remove unused method in SemAly
-
Key: HIVE-9304
URL: https://issues.apache.org/jira/browse/HIVE-9304
Project: Hive
Issue Type: Task
Components: Query Processor
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
Attachments: HIVE-9304.patch
Seems like method {{genConversionOps}} don't serve any purpose any longer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9217) UnionProcessor misses results for multi-insert when hive.optimize.union.remove=true
[ https://issues.apache.org/jira/browse/HIVE-9217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-9217: - Resolution: Fixed Status: Resolved (was: Patch Available) UnionProcessor misses results for multi-insert when hive.optimize.union.remove=true --- Key: HIVE-9217 URL: https://issues.apache.org/jira/browse/HIVE-9217 Project: Hive Issue Type: Bug Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-9217.01.patch, HIVE-9217.02.patch, HIVE-9217.03.patch To reproduce: In union_remove_6.q, just change FROM ( SELECT key, count(1) as values from inputTbl1 group by key UNION ALL SELECT key, count(1) as values from inputTbl1 group by key ) a insert overwrite table outputTbl1 select * insert overwrite table outputTbl2 select *; to FROM ( select * from( SELECT key, count(1) as values from inputTbl1 group by key UNION ALL SELECT key, count(1) as values from inputTbl1 group by key )subq ) a insert overwrite table outputTbl1 select * insert overwrite table outputTbl2 select *; select * from outputtbl2 will output nothing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9217) UnionProcessor misses results for multi-insert when hive.optimize.union.remove=true
[ https://issues.apache.org/jira/browse/HIVE-9217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270161#comment-14270161 ] Laljo John Pullokkaran commented on HIVE-9217: -- Committed to trunk; thanks [~pxiong] UnionProcessor misses results for multi-insert when hive.optimize.union.remove=true --- Key: HIVE-9217 URL: https://issues.apache.org/jira/browse/HIVE-9217 Project: Hive Issue Type: Bug Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-9217.01.patch, HIVE-9217.02.patch, HIVE-9217.03.patch To reproduce: In union_remove_6.q, just change FROM ( SELECT key, count(1) as values from inputTbl1 group by key UNION ALL SELECT key, count(1) as values from inputTbl1 group by key ) a insert overwrite table outputTbl1 select * insert overwrite table outputTbl2 select *; to FROM ( select * from( SELECT key, count(1) as values from inputTbl1 group by key UNION ALL SELECT key, count(1) as values from inputTbl1 group by key )subq ) a insert overwrite table outputTbl1 select * insert overwrite table outputTbl2 select *; select * from outputtbl2 will output nothing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9316) TestSqoop tests in WebHCat testsuite hardcode libdir path to hdfs
[ https://issues.apache.org/jira/browse/HIVE-9316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepesh Khandelwal updated HIVE-9316: - Status: Patch Available (was: Open) TestSqoop tests in WebHCat testsuite hardcode libdir path to hdfs - Key: HIVE-9316 URL: https://issues.apache.org/jira/browse/HIVE-9316 Project: Hive Issue Type: Bug Components: Tests, WebHCat Reporter: Deepesh Khandelwal Assignee: Deepesh Khandelwal Priority: Minor Fix For: 0.15.0 Attachments: HIVE-9316.1.patch Currently the TestSqoop tests in WebHCat Perl based testsuite has hdfs:// prefix in the jdbc jar path in libdir, we should remove this to enable it to run against other file systems. NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9306) Let Context.isLocalOnlyExecutionMode() return false if execution engine is Spark [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270170#comment-14270170
]
Hive QA commented on HIVE-9306:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12690946/HIVE-9306.3-spark.patch
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 7285 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_windowing
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchEmptyCommit
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/620/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/620/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-620/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12690946 - PreCommit-HIVE-SPARK-Build
Let Context.isLocalOnlyExecutionMode() return false if execution engine is
Spark [Spark Branch]
---
Key: HIVE-9306
URL: https://issues.apache.org/jira/browse/HIVE-9306
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
Attachments: HIVE-9306.1-spark.patch, HIVE-9306.2-spark.patch,
HIVE-9306.3-spark.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9304) [Refactor] remove unused method in SemAly
[
https://issues.apache.org/jira/browse/HIVE-9304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-9304:
---
Priority: Trivial (was: Major)
[Refactor] remove unused method in SemAly
-
Key: HIVE-9304
URL: https://issues.apache.org/jira/browse/HIVE-9304
Project: Hive
Issue Type: Task
Components: Query Processor
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
Priority: Trivial
Attachments: HIVE-9304.patch
Seems like method {{genConversionOps}} don't serve any purpose any longer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
