[jira] [Updated] (HIVE-9416) Get rid of Extract Operator
[
https://issues.apache.org/jira/browse/HIVE-9416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-9416:
---
Attachment: HIVE-9416.5.patch
Rebased with more golden files updates.
Get rid of Extract Operator
---
Key: HIVE-9416
URL: https://issues.apache.org/jira/browse/HIVE-9416
Project: Hive
Issue Type: Task
Components: Query Processor
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
Attachments: HIVE-9416.1.patch, HIVE-9416.2.patch, HIVE-9416.3.patch,
HIVE-9416.4.patch, HIVE-9416.5.patch, HIVE-9416.patch
{{Extract Operator}} has been there for legacy reasons. But there is no
functionality it provides which cant be provided by {{Select Operator}}
Instead of having two operators, one being subset of another we should just
get rid of {{Extract}} and simplify our codebase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9327) CBO (Calcite Return Path): Removing Row Resolvers from ParseContext
[
https://issues.apache.org/jira/browse/HIVE-9327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290828#comment-14290828
]
Hive QA commented on HIVE-9327:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12694348/HIVE-9327.06.patch
{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 7366 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby3_map
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf_streaming
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_ptf
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_ptf_streaming
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_column_access_stats
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_sort_skew_1_23
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_insert_into1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_insert_into2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapjoin_distinct
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_gby_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ptf
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ptf_streaming
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_skewjoinopt10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_cast_constant
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_groupby_3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorized_ptf
org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2511/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2511/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2511/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 22 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12694348 - PreCommit-HIVE-TRUNK-Build
CBO (Calcite Return Path): Removing Row Resolvers from ParseContext
---
Key: HIVE-9327
URL: https://issues.apache.org/jira/browse/HIVE-9327
Project: Hive
Issue Type: Sub-task
Components: CBO
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez
Fix For: 0.15.0
Attachments: HIVE-9327.01.patch, HIVE-9327.02.patch,
HIVE-9327.03.patch, HIVE-9327.04.patch, HIVE-9327.05.patch,
HIVE-9327.06.patch, HIVE-9327.patch
CLEAR LIBRARY CACHE
ParseContext includes a map of Operator to RowResolver (OpParseContext). It
would be ideal to remove this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9457) Fix obsolete parameter name in HiveConf description of hive.hashtable.initialCapacity
Lefty Leverenz created HIVE-9457:
Summary: Fix obsolete parameter name in HiveConf description of
hive.hashtable.initialCapacity
Key: HIVE-9457
URL: https://issues.apache.org/jira/browse/HIVE-9457
Project: Hive
Issue Type: Bug
Components: Documentation
Affects Versions: 0.14.0
Reporter: Lefty Leverenz
Priority: Minor
The description of *hive.hashtable.initialCapacity* in HiveConf.java refers to
a parameter that existed in an early patch for HIVE-7616
(hive.hashtable.stats.key.estimate.adjustment) but was renamed in later
patches. So change *hive.hashtable.stats.key.estimate.adjustment* to
*hive.hashtable.key.count.adjustment* in this parameter definition in
HiveConf.java:
{code}
HIVEHASHTABLETHRESHOLD(hive.hashtable.initialCapacity, 10, Initial
capacity of +
mapjoin hashtable if statistics are absent, or if
hive.hashtable.stats.key.estimate.adjustment is set to 0),
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9431) CBO (Calcite Return Path): Removing AST from ParseContext
[
https://issues.apache.org/jira/browse/HIVE-9431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290789#comment-14290789
]
Hive QA commented on HIVE-9431:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12694349/HIVE-9431.01.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 7366 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2510/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2510/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2510/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12694349 - PreCommit-HIVE-TRUNK-Build
CBO (Calcite Return Path): Removing AST from ParseContext
-
Key: HIVE-9431
URL: https://issues.apache.org/jira/browse/HIVE-9431
Project: Hive
Issue Type: Sub-task
Components: CBO
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez
Fix For: 0.15.0
Attachments: HIVE-9431.01.patch, HIVE-9431.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-6308) COLUMNS_V2 Metastore table not populated for tables created without an explicit column list.
[ https://issues.apache.org/jira/browse/HIVE-6308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-6308: --- Status: Patch Available (was: Open) COLUMNS_V2 Metastore table not populated for tables created without an explicit column list. Key: HIVE-6308 URL: https://issues.apache.org/jira/browse/HIVE-6308 Project: Hive Issue Type: Bug Components: Database/Schema Affects Versions: 0.10.0 Reporter: Alexander Behm Assignee: Yongzhi Chen Attachments: HIVE-6308.1.patch Consider this example table: CREATE TABLE avro_test ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' STORED as INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' TBLPROPERTIES ( 'avro.schema.url'='file:///path/to/the/schema/test_serializer.avsc'); When I try to run an ANALYZE TABLE for computing column stats on any of the columns, then I get: org.apache.hadoop.hive.ql.metadata.HiveException: NoSuchObjectException(message:Column o_orderpriority for which stats gathering is requested doesn't exist.) at org.apache.hadoop.hive.ql.metadata.Hive.updateTableColumnStatistics(Hive.java:2280) at org.apache.hadoop.hive.ql.exec.ColumnStatsTask.persistTableStats(ColumnStatsTask.java:331) at org.apache.hadoop.hive.ql.exec.ColumnStatsTask.execute(ColumnStatsTask.java:343) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:66) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1383) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1169) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:982) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:208) The root cause appears to be that the COLUMNS_V2 table in the Metastore isn't populated properly during the table creation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9416) Get rid of Extract Operator
[
https://issues.apache.org/jira/browse/HIVE-9416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-9416:
---
Status: Open (was: Patch Available)
Get rid of Extract Operator
---
Key: HIVE-9416
URL: https://issues.apache.org/jira/browse/HIVE-9416
Project: Hive
Issue Type: Task
Components: Query Processor
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
Attachments: HIVE-9416.1.patch, HIVE-9416.2.patch, HIVE-9416.3.patch,
HIVE-9416.4.patch, HIVE-9416.patch
{{Extract Operator}} has been there for legacy reasons. But there is no
functionality it provides which cant be provided by {{Select Operator}}
Instead of having two operators, one being subset of another we should just
get rid of {{Extract}} and simplify our codebase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-6308) COLUMNS_V2 Metastore table not populated for tables created without an explicit column list.
[ https://issues.apache.org/jira/browse/HIVE-6308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-6308: --- Attachment: HIVE-6308.1.patch Need code review COLUMNS_V2 Metastore table not populated for tables created without an explicit column list. Key: HIVE-6308 URL: https://issues.apache.org/jira/browse/HIVE-6308 Project: Hive Issue Type: Bug Components: Database/Schema Affects Versions: 0.10.0 Reporter: Alexander Behm Assignee: Yongzhi Chen Attachments: HIVE-6308.1.patch Consider this example table: CREATE TABLE avro_test ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' STORED as INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' TBLPROPERTIES ( 'avro.schema.url'='file:///path/to/the/schema/test_serializer.avsc'); When I try to run an ANALYZE TABLE for computing column stats on any of the columns, then I get: org.apache.hadoop.hive.ql.metadata.HiveException: NoSuchObjectException(message:Column o_orderpriority for which stats gathering is requested doesn't exist.) at org.apache.hadoop.hive.ql.metadata.Hive.updateTableColumnStatistics(Hive.java:2280) at org.apache.hadoop.hive.ql.exec.ColumnStatsTask.persistTableStats(ColumnStatsTask.java:331) at org.apache.hadoop.hive.ql.exec.ColumnStatsTask.execute(ColumnStatsTask.java:343) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:66) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1383) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1169) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:982) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:208) The root cause appears to be that the COLUMNS_V2 table in the Metastore isn't populated properly during the table creation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9449) Push YARN configuration to Spark while deply Spark on YARN[Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290717#comment-14290717
]
Hive QA commented on HIVE-9449:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12694362/HIVE-9449.1-spark.patch
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 7355 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/681/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/681/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-681/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12694362 - PreCommit-HIVE-SPARK-Build
Push YARN configuration to Spark while deply Spark on YARN[Spark Branch]
Key: HIVE-9449
URL: https://issues.apache.org/jira/browse/HIVE-9449
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Chengxiang Li
Assignee: Chengxiang Li
Attachments: HIVE-9449.1-spark.patch, HIVE-9449.1-spark.patch
We only push Spark configuration and RSC configuration to Spark while launch
Spark cluster now, for Spark on YARN mode, Spark need extra YARN
configuration to launch Spark cluster. Besides this, to support dynamically
configuration setting for RSC configuration/YARN configuration, we need to
recreate SparkSession while RSC configuration/YARN configuration update as
well, as they may influence the Spark cluster deployment as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-1830) mappers in group followed by joins may die OOM
[ https://issues.apache.org/jira/browse/HIVE-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290901#comment-14290901 ] Lefty Leverenz commented on HIVE-1830: -- Doc note: This added three configuration parameters to HiveConf.java, with descriptions in the template file. They are documented in the wiki. * [hive.mapjoin.followby.map.aggr.hash.percentmemory | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.mapjoin.followby.map.aggr.hash.percentmemory] * [hive.map.aggr.hash.force.flush.memory.threshold | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.map.aggr.hash.force.flush.memory.threshold] * [hive.mapjoin.followby.gby.localtask.max.memory.usage | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.mapjoin.followby.gby.localtask.max.memory.usage] mappers in group followed by joins may die OOM -- Key: HIVE-1830 URL: https://issues.apache.org/jira/browse/HIVE-1830 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Liyin Tang Fix For: 0.7.0 Attachments: hive-1830-1.patch, hive-1830-2.patch, hive-1830-3.patch, hive-1830-4.patch, hive-1830-5.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9443) ORC PPD - fix fuzzy case evaluation of IS_NULL
[
https://issues.apache.org/jira/browse/HIVE-9443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-9443:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
ORC PPD - fix fuzzy case evaluation of IS_NULL
--
Key: HIVE-9443
URL: https://issues.apache.org/jira/browse/HIVE-9443
Project: Hive
Issue Type: Bug
Components: File Formats
Affects Versions: 0.15.0
Reporter: Gopal V
Assignee: Gopal V
Fix For: 0.15.0
Attachments: HIVE-9443.1.patch, HIVE-9443.2.patch, HIVE-9443.3.patch
CLEAR LIBRARY CACHE
ORC PPD returns IS_NULL wrong for the fuzz case of some-nulls.
The code-flow in effect should be
{code}
if (min == null) {
// all nulls
return YES;
} else {
if (hasNull == true) {
// some nulls
return YES_NO; //maybe
}
// no nulls
return NO;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9445) Revert HIVE-5700 - enforce single date format for partition column storage
[
https://issues.apache.org/jira/browse/HIVE-9445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland updated HIVE-9445:
---
Attachment: HIVE-9445.1.patch
Revert HIVE-5700 - enforce single date format for partition column storage
--
Key: HIVE-9445
URL: https://issues.apache.org/jira/browse/HIVE-9445
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.13.0, 0.14.0, 0.13.1, 0.15.0, 0.14.1
Reporter: Brock Noland
Assignee: Brock Noland
Priority: Blocker
Attachments: HIVE-9445.1.patch, HIVE-9445.1.patch
HIVE-5700 has the following issues:
* HIVE-8730 - fails mysql upgrades
* Does not upgrade all metadata, e.g. {{PARTITIONS.PART_NAME}} See comments
in HIVE-5700.
* Completely corrupts postgres, see below.
With a postgres metastore on 0.12, I executed the following:
{noformat}
CREATE TABLE HIVE5700_DATE_PARTED (line string) PARTITIONED BY (ddate date);
CREATE TABLE HIVE5700_STRING_PARTED (line string) PARTITIONED BY (ddate
string);
ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='NOT_DATE');
ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='20150121');
ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='20150122');
ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='2015-01-23');
ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='NOT_DATE');
ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='20150121');
ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='20150122');
ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='2015-01-23');
LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE
HIVE5700_DATE_PARTED PARTITION (ddate='NOT_DATE');
LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE
HIVE5700_DATE_PARTED PARTITION (ddate='20150121');
LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE
HIVE5700_DATE_PARTED PARTITION (ddate='20150122');
LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE
HIVE5700_DATE_PARTED PARTITION (ddate='2015-01-23');
LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE
HIVE5700_STRING_PARTED PARTITION (ddate='NOT_DATE');
LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE
HIVE5700_STRING_PARTED PARTITION (ddate='20150121');
LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE
HIVE5700_STRING_PARTED PARTITION (ddate='20150122');
LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE
HIVE5700_STRING_PARTED PARTITION (ddate='2015-01-23');
hive show partitions HIVE5700_DATE_PARTED;
OK
ddate=20150121
ddate=20150122
ddate=2015-01-23
ddate=NOT_DATE
Time taken: 0.052 seconds, Fetched: 4 row(s)
hive show partitions HIVE5700_STRING_PARTED;
OK
ddate=20150121
ddate=20150122
ddate=2015-01-23
ddate=NOT_DATE
Time taken: 0.051 seconds, Fetched: 4 row(s)
{noformat}
I then took a dump of the database named {{postgres-pre-upgrade.sql}} and the
data in the dump looks good:
{noformat}
[root@hive5700-1-1 ~]# egrep -A9 '^COPY PARTITIONS|^COPY
PARTITION_KEY_VALS' postgres-pre-upgrade.sql
COPY PARTITIONS (PART_ID, CREATE_TIME, LAST_ACCESS_TIME, PART_NAME,
SD_ID, TBL_ID) FROM stdin;
3 1421943647 0 ddate=NOT_DATE 6 2
4 1421943647 0 ddate=20150121 7 2
5 1421943648 0 ddate=20150122 8 2
6 1421943664 0 ddate=NOT_DATE 9 3
7 1421943664 0 ddate=20150121 10 3
8 1421943665 0 ddate=20150122 11 3
9 1421943694 0 ddate=2015-01-2312 2
101421943695 0 ddate=2015-01-2313 3
\.
--
COPY PARTITION_KEY_VALS (PART_ID, PART_KEY_VAL, INTEGER_IDX) FROM
stdin;
3 NOT_DATE0
4 201501210
5 201501220
6 NOT_DATE0
7 201501210
8 201501220
9 2015-01-23 0
102015-01-23 0
\.
{noformat}
I then upgraded to 0.13 and subsequently upgraded the MS with the following
command: {{schematool -dbType postgres -upgradeSchema -verbose}}
The file {{postgres-post-upgrade.sql}} is the post-upgrade db dump. As you
can see the data is completely corrupt.
{noformat}
[root@hive5700-1-1 ~]# egrep -A9 '^COPY PARTITIONS|^COPY
PARTITION_KEY_VALS' postgres-post-upgrade.sql
COPY PARTITIONS (PART_ID, CREATE_TIME, LAST_ACCESS_TIME, PART_NAME,
SD_ID, TBL_ID) FROM stdin;
3 1421943647 0 ddate=NOT_DATE 6 2
4 1421943647 0 ddate=20150121 7 2
5 1421943648 0 ddate=20150122 8 2
6 1421943664 0 ddate=NOT_DATE 9 3
7 1421943664 0 ddate=20150121 10 3
8 1421943665 0 ddate=20150122 11 3
9 1421943694
[jira] [Commented] (HIVE-9445) Revert HIVE-5700 - enforce single date format for partition column storage
[
https://issues.apache.org/jira/browse/HIVE-9445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14291014#comment-14291014
]
Hive QA commented on HIVE-9445:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12694410/HIVE-9445.1.patch
{color:green}SUCCESS:{color} +1 7365 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2515/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2515/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2515/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12694410 - PreCommit-HIVE-TRUNK-Build
Revert HIVE-5700 - enforce single date format for partition column storage
--
Key: HIVE-9445
URL: https://issues.apache.org/jira/browse/HIVE-9445
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.13.0, 0.14.0, 0.13.1, 0.15.0, 0.14.1
Reporter: Brock Noland
Assignee: Brock Noland
Priority: Blocker
Attachments: HIVE-9445.1.patch, HIVE-9445.1.patch
HIVE-5700 has the following issues:
* HIVE-8730 - fails mysql upgrades
* Does not upgrade all metadata, e.g. {{PARTITIONS.PART_NAME}} See comments
in HIVE-5700.
* Completely corrupts postgres, see below.
With a postgres metastore on 0.12, I executed the following:
{noformat}
CREATE TABLE HIVE5700_DATE_PARTED (line string) PARTITIONED BY (ddate date);
CREATE TABLE HIVE5700_STRING_PARTED (line string) PARTITIONED BY (ddate
string);
ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='NOT_DATE');
ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='20150121');
ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='20150122');
ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='2015-01-23');
ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='NOT_DATE');
ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='20150121');
ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='20150122');
ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='2015-01-23');
LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE
HIVE5700_DATE_PARTED PARTITION (ddate='NOT_DATE');
LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE
HIVE5700_DATE_PARTED PARTITION (ddate='20150121');
LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE
HIVE5700_DATE_PARTED PARTITION (ddate='20150122');
LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE
HIVE5700_DATE_PARTED PARTITION (ddate='2015-01-23');
LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE
HIVE5700_STRING_PARTED PARTITION (ddate='NOT_DATE');
LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE
HIVE5700_STRING_PARTED PARTITION (ddate='20150121');
LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE
HIVE5700_STRING_PARTED PARTITION (ddate='20150122');
LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE
HIVE5700_STRING_PARTED PARTITION (ddate='2015-01-23');
hive show partitions HIVE5700_DATE_PARTED;
OK
ddate=20150121
ddate=20150122
ddate=2015-01-23
ddate=NOT_DATE
Time taken: 0.052 seconds, Fetched: 4 row(s)
hive show partitions HIVE5700_STRING_PARTED;
OK
ddate=20150121
ddate=20150122
ddate=2015-01-23
ddate=NOT_DATE
Time taken: 0.051 seconds, Fetched: 4 row(s)
{noformat}
I then took a dump of the database named {{postgres-pre-upgrade.sql}} and the
data in the dump looks good:
{noformat}
[root@hive5700-1-1 ~]# egrep -A9 '^COPY PARTITIONS|^COPY
PARTITION_KEY_VALS' postgres-pre-upgrade.sql
COPY PARTITIONS (PART_ID, CREATE_TIME, LAST_ACCESS_TIME, PART_NAME,
SD_ID, TBL_ID) FROM stdin;
3 1421943647 0 ddate=NOT_DATE 6 2
4 1421943647 0 ddate=20150121 7 2
5 1421943648 0 ddate=20150122 8 2
6 1421943664 0 ddate=NOT_DATE 9 3
7 1421943664 0 ddate=20150121 10 3
8 1421943665 0 ddate=20150122 11 3
9 1421943694 0 ddate=2015-01-2312 2
101421943695 0 ddate=2015-01-2313 3
\.
--
COPY PARTITION_KEY_VALS (PART_ID, PART_KEY_VAL, INTEGER_IDX) FROM
stdin;
3 NOT_DATE0
4 201501210
5 201501220
6 NOT_DATE0
7 201501210
8 201501220
9 2015-01-23 0
[jira] [Commented] (HIVE-9416) Get rid of Extract Operator
[
https://issues.apache.org/jira/browse/HIVE-9416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290927#comment-14290927
]
Hive QA commented on HIVE-9416:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12694378/HIVE-9416.5.patch
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 7366 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization2
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testSaslWithHiveMetaStore
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2513/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2513/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2513/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12694378 - PreCommit-HIVE-TRUNK-Build
Get rid of Extract Operator
---
Key: HIVE-9416
URL: https://issues.apache.org/jira/browse/HIVE-9416
Project: Hive
Issue Type: Task
Components: Query Processor
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
Attachments: HIVE-9416.1.patch, HIVE-9416.2.patch, HIVE-9416.3.patch,
HIVE-9416.4.patch, HIVE-9416.5.patch, HIVE-9416.patch
{{Extract Operator}} has been there for legacy reasons. But there is no
functionality it provides which cant be provided by {{Select Operator}}
Instead of having two operators, one being subset of another we should just
get rid of {{Extract}} and simplify our codebase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9443) ORC PPD - fix fuzzy case evaluation of IS_NULL
[
https://issues.apache.org/jira/browse/HIVE-9443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290983#comment-14290983
]
Gopal V commented on HIVE-9443:
---
Test failures are unrelated.
ORC PPD - fix fuzzy case evaluation of IS_NULL
--
Key: HIVE-9443
URL: https://issues.apache.org/jira/browse/HIVE-9443
Project: Hive
Issue Type: Bug
Components: File Formats
Affects Versions: 0.15.0
Reporter: Gopal V
Assignee: Gopal V
Fix For: 0.15.0
Attachments: HIVE-9443.1.patch, HIVE-9443.2.patch, HIVE-9443.3.patch
CLEAR LIBRARY CACHE
ORC PPD returns IS_NULL wrong for the fuzz case of some-nulls.
The code-flow in effect should be
{code}
if (min == null) {
// all nulls
return YES;
} else {
if (hasNull == true) {
// some nulls
return YES_NO; //maybe
}
// no nulls
return NO;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9454) Test failures due to new Calcite version
[
https://issues.apache.org/jira/browse/HIVE-9454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Hyde updated HIVE-9454:
--
Attachment: HIVE-9454.1.patch
The attached patch (HIVE-9454.1.patch) works around CALCITE-566. It provides an
executor so that constant-reduction rules can be executed.
The other test failures are, as far as I can tell, changes to generated plans
and are either improvements or cosmetic changes. This patch does not fix those.
Test failures due to new Calcite version
Key: HIVE-9454
URL: https://issues.apache.org/jira/browse/HIVE-9454
Project: Hive
Issue Type: Bug
Reporter: Brock Noland
Attachments: HIVE-9454.1.patch
A bunch of failures have started appearing in patches which seen unrelated. I
am thinking we've picked up a new version of Calcite. E.g.:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2488/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_auto_join12/
{noformat}
Running: diff -a
/home/hiveptest/54.147.202.89-hiveptest-1/apache-svn-trunk-source/itests/qtest/../../itests/qtest/target/qfile-results/clientpositive/auto_join12.q.out
/home/hiveptest/54.147.202.89-hiveptest-1/apache-svn-trunk-source/itests/qtest/../../ql/src/test/results/clientpositive/auto_join12.q.out
32c32
$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src
---
$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src
35c35
$hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:src
---
$hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:$hdt$_1:src
39c39
$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src
---
$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src
54c54
$hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:src
---
$hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:$hdt$_1:src
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9444) CBO (Calcite Return Path): Rewrite GlobalLimitOptimizer
[
https://issues.apache.org/jira/browse/HIVE-9444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290871#comment-14290871
]
Hive QA commented on HIVE-9444:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12694352/HIVE-9444.patch
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 7366 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2512/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2512/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2512/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12694352 - PreCommit-HIVE-TRUNK-Build
CBO (Calcite Return Path): Rewrite GlobalLimitOptimizer
---
Key: HIVE-9444
URL: https://issues.apache.org/jira/browse/HIVE-9444
Project: Hive
Issue Type: Sub-task
Components: CBO
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez
Fix For: 0.15.0
Attachments: HIVE-9444.patch
Currently, GlobalLimitOptimization relies heavily on the information
contained in QBParseInfo. The goal is to extract that information from the
operator tree so we do not need to rely on QBParseInfo.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7616) pre-size mapjoin hashtable based on statistics
[ https://issues.apache.org/jira/browse/HIVE-7616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290910#comment-14290910 ] Lefty Leverenz commented on HIVE-7616: -- Doc note: *hive.hashtable.key.count.adjustment* is documented and *hive.hashtable.initialCapacity* has a description with the correct parameter reference in the wiki. Removing TODOC14 label. * [Configuration Properties -- hive.hashtable.key.count.adjustment | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.hashtable.key.count.adjustment] * [Configuration Properties -- hive.hashtable.initialCapacity | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.hashtable.initialCapacity] pre-size mapjoin hashtable based on statistics -- Key: HIVE-7616 URL: https://issues.apache.org/jira/browse/HIVE-7616 Project: Hive Issue Type: Improvement Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.14.0 Attachments: HIVE-7616.01.patch, HIVE-7616.02.patch, HIVE-7616.03.patch, HIVE-7616.04.patch, HIVE-7616.05.patch, HIVE-7616.06.patch, HIVE-7616.07.patch, HIVE-7616.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7616) pre-size mapjoin hashtable based on statistics
[ https://issues.apache.org/jira/browse/HIVE-7616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-7616: - Labels: (was: TODOC14) pre-size mapjoin hashtable based on statistics -- Key: HIVE-7616 URL: https://issues.apache.org/jira/browse/HIVE-7616 Project: Hive Issue Type: Improvement Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.14.0 Attachments: HIVE-7616.01.patch, HIVE-7616.02.patch, HIVE-7616.03.patch, HIVE-7616.04.patch, HIVE-7616.05.patch, HIVE-7616.06.patch, HIVE-7616.07.patch, HIVE-7616.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9443) ORC PPD - fix fuzzy case evaluation of IS_NULL
[
https://issues.apache.org/jira/browse/HIVE-9443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290985#comment-14290985
]
Gopal V commented on HIVE-9443:
---
Committed to trunk, thanks [~prasanth_j].
ORC PPD - fix fuzzy case evaluation of IS_NULL
--
Key: HIVE-9443
URL: https://issues.apache.org/jira/browse/HIVE-9443
Project: Hive
Issue Type: Bug
Components: File Formats
Affects Versions: 0.15.0
Reporter: Gopal V
Assignee: Gopal V
Fix For: 0.15.0
Attachments: HIVE-9443.1.patch, HIVE-9443.2.patch, HIVE-9443.3.patch
CLEAR LIBRARY CACHE
ORC PPD returns IS_NULL wrong for the fuzz case of some-nulls.
The code-flow in effect should be
{code}
if (min == null) {
// all nulls
return YES;
} else {
if (hasNull == true) {
// some nulls
return YES_NO; //maybe
}
// no nulls
return NO;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-6308) COLUMNS_V2 Metastore table not populated for tables created without an explicit column list.
[
https://issues.apache.org/jira/browse/HIVE-6308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290964#comment-14290964
]
Hive QA commented on HIVE-6308:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12694386/HIVE-6308.1.patch
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 7366 tests executed
*Failed tests:*
{noformat}
TestCustomAuthentication - did not produce a TEST-*.xml file
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2514/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2514/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2514/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12694386 - PreCommit-HIVE-TRUNK-Build
COLUMNS_V2 Metastore table not populated for tables created without an
explicit column list.
Key: HIVE-6308
URL: https://issues.apache.org/jira/browse/HIVE-6308
Project: Hive
Issue Type: Bug
Components: Database/Schema
Affects Versions: 0.10.0
Reporter: Alexander Behm
Assignee: Yongzhi Chen
Attachments: HIVE-6308.1.patch
Consider this example table:
CREATE TABLE avro_test
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED as INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
TBLPROPERTIES (
'avro.schema.url'='file:///path/to/the/schema/test_serializer.avsc');
When I try to run an ANALYZE TABLE for computing column stats on any of the
columns, then I get:
org.apache.hadoop.hive.ql.metadata.HiveException:
NoSuchObjectException(message:Column o_orderpriority for which stats
gathering is requested doesn't exist.)
at
org.apache.hadoop.hive.ql.metadata.Hive.updateTableColumnStatistics(Hive.java:2280)
at
org.apache.hadoop.hive.ql.exec.ColumnStatsTask.persistTableStats(ColumnStatsTask.java:331)
at
org.apache.hadoop.hive.ql.exec.ColumnStatsTask.execute(ColumnStatsTask.java:343)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:66)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1383)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1169)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:982)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
The root cause appears to be that the COLUMNS_V2 table in the Metastore isn't
populated properly during the table creation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8485) HMS on Oracle incompatibility
[
https://issues.apache.org/jira/browse/HIVE-8485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290698#comment-14290698
]
Chaoyu Tang commented on HIVE-8485:
---
[~sushanth] I was initially concerned about the impacts probably caused by the
change to other dbs, thanks for the clarification!
+1 (non-binding)
HMS on Oracle incompatibility
-
Key: HIVE-8485
URL: https://issues.apache.org/jira/browse/HIVE-8485
Project: Hive
Issue Type: Bug
Components: Metastore
Environment: Oracle as metastore DB
Reporter: Ryan Pridgeon
Assignee: Chaoyu Tang
Attachments: HIVE-8485.2.patch, HIVE-8485.patch
Oracle does not distinguish between empty strings and NULL,which proves
problematic for DataNucleus.
In the event a user creates a table with some property stored as an empty
string the table will no longer be accessible.
i.e. TBLPROPERTIES ('serialization.null.format'='')
If they try to select, describe, drop, etc the client prints the following
exception.
ERROR ql.Driver: FAILED: SemanticException [Error 10001]: Table not found
table name
The work around for this was to go into the hive metastore on the Oracle
database and replace NULL with some other string. Users could then drop the
tables or alter their data to use the new null format they just set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9443) ORC PPD - fix fuzzy case evaluation of IS_NULL
[
https://issues.apache.org/jira/browse/HIVE-9443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290725#comment-14290725
]
Hive QA commented on HIVE-9443:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12694340/HIVE-9443.3.patch
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 7366 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testSaslWithHiveMetaStore
org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2509/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2509/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2509/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12694340 - PreCommit-HIVE-TRUNK-Build
ORC PPD - fix fuzzy case evaluation of IS_NULL
--
Key: HIVE-9443
URL: https://issues.apache.org/jira/browse/HIVE-9443
Project: Hive
Issue Type: Bug
Components: File Formats
Affects Versions: 0.15.0
Reporter: Gopal V
Assignee: Gopal V
Fix For: 0.15.0
Attachments: HIVE-9443.1.patch, HIVE-9443.2.patch, HIVE-9443.3.patch
CLEAR LIBRARY CACHE
ORC PPD returns IS_NULL wrong for the fuzz case of some-nulls.
The code-flow in effect should be
{code}
if (min == null) {
// all nulls
return YES;
} else {
if (hasNull == true) {
// some nulls
return YES_NO; //maybe
}
// no nulls
return NO;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9416) Get rid of Extract Operator
[
https://issues.apache.org/jira/browse/HIVE-9416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-9416:
---
Status: Patch Available (was: Open)
Get rid of Extract Operator
---
Key: HIVE-9416
URL: https://issues.apache.org/jira/browse/HIVE-9416
Project: Hive
Issue Type: Task
Components: Query Processor
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
Attachments: HIVE-9416.1.patch, HIVE-9416.2.patch, HIVE-9416.3.patch,
HIVE-9416.4.patch, HIVE-9416.5.patch, HIVE-9416.patch
{{Extract Operator}} has been there for legacy reasons. But there is no
functionality it provides which cant be provided by {{Select Operator}}
Instead of having two operators, one being subset of another we should just
get rid of {{Extract}} and simplify our codebase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Review Request 30151: Remove Extract Operator its friends from codebase.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30151/ --- Review request for hive and Navis Ryu. Bugs: HIVE-9416 https://issues.apache.org/jira/browse/HIVE-9416 Repository: hive-git Description --- Remove Extract Operator its friends from codebase. Diffs - ql/src/java/org/apache/hadoop/hive/ql/exec/ExtractOperator.java c299d3a ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java f3c382a ql/src/java/org/apache/hadoop/hive/ql/exec/PTFOperator.java 2e6a880 ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 9ed2c61 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorExtractOperator.java 7f4bb64 ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java 24ca89f ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java e16ba6c ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/CorrelationUtilities.java dc906e8 ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/ReduceSinkDeDuplication.java 3fead79 ql/src/java/org/apache/hadoop/hive/ql/optimizer/lineage/OpProcFactory.java d6a6ed6 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/BucketingSortingInferenceOptimizer.java 7954767 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/BucketingSortingOpProcFactory.java cf02bec ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 94b4621 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 4364f28 ql/src/java/org/apache/hadoop/hive/ql/plan/ExtractDesc.java 6762155 ql/src/java/org/apache/hadoop/hive/ql/plan/SelectDesc.java fa6b548 ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java 41862e6 ql/src/test/results/clientpositive/bucket1.q.out 13ec735 ql/src/test/results/clientpositive/bucket2.q.out 32a77c3 ql/src/test/results/clientpositive/bucket3.q.out ff7173e ql/src/test/results/clientpositive/bucket4.q.out b99d12f ql/src/test/results/clientpositive/bucket5.q.out 5992d6d ql/src/test/results/clientpositive/bucket6.q.out 5b23d7d ql/src/test/results/clientpositive/bucketsortoptimize_insert_1.q.out 75de953 ql/src/test/results/clientpositive/bucketsortoptimize_insert_2.q.out 599b8b9 ql/src/test/results/clientpositive/bucketsortoptimize_insert_3.q.out 7456ab0 ql/src/test/results/clientpositive/bucketsortoptimize_insert_4.q.out fd99597 ql/src/test/results/clientpositive/bucketsortoptimize_insert_5.q.out 8130ab9 ql/src/test/results/clientpositive/bucketsortoptimize_insert_6.q.out 627aba0 ql/src/test/results/clientpositive/disable_merge_for_bucketing.q.out 9b058c8 ql/src/test/results/clientpositive/dynpart_sort_opt_vectorization.q.out 32e0745 ql/src/test/results/clientpositive/dynpart_sort_optimization.q.out 494bfa3 ql/src/test/results/clientpositive/encrypted/encryption_insert_partition_dynamic.q.out b6e7b88 ql/src/test/results/clientpositive/encrypted/encryption_insert_partition_static.q.out fc6d2ae ql/src/test/results/clientpositive/load_dyn_part2.q.out 26f318a ql/src/test/results/clientpositive/ptf.q.out f678035 ql/src/test/results/clientpositive/ptf_streaming.q.out 9cf645d ql/src/test/results/clientpositive/smb_mapjoin_20.q.out 999dabd ql/src/test/results/clientpositive/smb_mapjoin_21.q.out 539b70e ql/src/test/results/clientpositive/spark/bucket2.q.out 5eb28fa ql/src/test/results/clientpositive/spark/bucket3.q.out 1b1010a ql/src/test/results/clientpositive/spark/bucket4.q.out 7dd49ac ql/src/test/results/clientpositive/spark/bucketsortoptimize_insert_2.q.out 365306e ql/src/test/results/clientpositive/spark/bucketsortoptimize_insert_4.q.out 3846de7 ql/src/test/results/clientpositive/spark/bucketsortoptimize_insert_6.q.out 5b559c4 ql/src/test/results/clientpositive/spark/bucketsortoptimize_insert_7.q.out cefc6aa ql/src/test/results/clientpositive/spark/bucketsortoptimize_insert_8.q.out ca44d7c ql/src/test/results/clientpositive/spark/disable_merge_for_bucketing.q.out 3864c44 ql/src/test/results/clientpositive/spark/load_dyn_part2.q.out a8cef34 ql/src/test/results/clientpositive/spark/ptf.q.out ad7d9d5 ql/src/test/results/clientpositive/spark/ptf_streaming.q.out 9f98933 ql/src/test/results/clientpositive/spark/smb_mapjoin_20.q.out 9815d24 ql/src/test/results/clientpositive/spark/smb_mapjoin_21.q.out 44bae6e ql/src/test/results/clientpositive/spark/stats10.q.out 4bec7fc ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out 020fdff ql/src/test/results/clientpositive/stats10.q.out 804e1b8 ql/src/test/results/clientpositive/tez/bucket2.q.out 9536ed0 ql/src/test/results/clientpositive/tez/bucket3.q.out 3427474 ql/src/test/results/clientpositive/tez/bucket4.q.out ca3584e
[jira] [Resolved] (HIVE-9455) MapJoin task shouldn't start if HashTableSink task failed [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao resolved HIVE-9455.
Resolution: Duplicate
Looks like this is a duplicate of HIVE-9428..
MapJoin task shouldn't start if HashTableSink task failed [Spark Branch]
-
Key: HIVE-9455
URL: https://issues.apache.org/jira/browse/HIVE-9455
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Chao
While playing with {{auto_join25.q}}, I noticed that even though the task for
hash table sink failed, HOS will still continue launch the task for map join.
This is not the desired result. Instead, like MR, we should abandon the
second task.
Console output:
{code}
Total jobs = 2
Launching Job 1 out of 2
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=number
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=number
In order to set a constant number of reducers:
set mapreduce.job.reduces=number
Query Hive on Spark job[0] stages:
0
Status: Running (Hive on Spark job[0])
Job Progress Format
CurrentTime StageId_StageAttemptId:
SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount
[StageCost]
2015-01-23 16:18:14,604 Stage-0_0: 0/1
2015-01-23 04:18:14 Processing rows:4 Hashtable size: 3
Memory usage: 119199408 percentage: 0.25
2015-01-23 16:18:15,611 Stage-0_0: 0(+0,-1)/1
Status: Finished successfully in 1.07 seconds
Launching Job 2 out of 2
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=number
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=number
In order to set a constant number of reducers:
set mapreduce.job.reduces=number
2015-01-23 16:22:27,854 Stage-1_0: 0(+0,-1)/1
Status: Finished successfully in 1.01 seconds
Loading data to table default.dest1
Table default.dest1 stats: [numFiles=0, numRows=0, totalSize=0, rawDataSize=0]
OK
Time taken: 311.979 seconds
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9039) Support Union Distinct
[
https://issues.apache.org/jira/browse/HIVE-9039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290685#comment-14290685
]
Hive QA commented on HIVE-9039:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12694329/HIVE-9039.21.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 7375 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testRevokeTimedOutWorkers
org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2508/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2508/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2508/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12694329 - PreCommit-HIVE-TRUNK-Build
Support Union Distinct
--
Key: HIVE-9039
URL: https://issues.apache.org/jira/browse/HIVE-9039
Project: Hive
Issue Type: New Feature
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong
Attachments: HIVE-9039.01.patch, HIVE-9039.02.patch,
HIVE-9039.03.patch, HIVE-9039.04.patch, HIVE-9039.05.patch,
HIVE-9039.06.patch, HIVE-9039.07.patch, HIVE-9039.08.patch,
HIVE-9039.09.patch, HIVE-9039.10.patch, HIVE-9039.11.patch,
HIVE-9039.12.patch, HIVE-9039.13.patch, HIVE-9039.14.patch,
HIVE-9039.15.patch, HIVE-9039.16.patch, HIVE-9039.17.patch,
HIVE-9039.18.patch, HIVE-9039.19.patch, HIVE-9039.20.patch, HIVE-9039.21.patch
CLEAR LIBRARY CACHE
Current version (Hive 0.14) does not support union (or union distinct). It
only supports union all. In this patch, we try to add this new feature by
rewriting union distinct to union all followed by group by.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9410) ClassNotFoundException occurs during hive query case execution with UDF defined [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290688#comment-14290688
]
Hive QA commented on HIVE-9410:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12694361/HIVE-9410.4-spark.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 7357 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/680/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/680/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-680/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12694361 - PreCommit-HIVE-SPARK-Build
ClassNotFoundException occurs during hive query case execution with UDF
defined [Spark Branch]
--
Key: HIVE-9410
URL: https://issues.apache.org/jira/browse/HIVE-9410
Project: Hive
Issue Type: Sub-task
Components: Spark
Environment: CentOS 6.5
JDK1.7
Reporter: Xin Hao
Assignee: Chengxiang Li
Attachments: HIVE-9410.1-spark.patch, HIVE-9410.2-spark.patch,
HIVE-9410.3-spark.patch, HIVE-9410.4-spark.patch, HIVE-9410.4-spark.patch
We have a hive query case with UDF defined (i.e. BigBench case Q10, Q18
etc.). It will be passed for default Hive (on MR) mode, while failed for Hive
On Spark mode (both Standalone and Yarn-Client).
Although we use 'add jar .jar;' to add the UDF jar explicitly, the issue
still exists.
BTW, if we put the UDF jar into $HIVE_HOME/lib dir, the case will be passed.
Detail Error Message is as below (NOTE:
de.bankmark.bigbench.queries.q10.SentimentUDF is the UDF which contained in
jar bigbenchqueriesmr.jar, and we have add command like 'add jar
/location/to/bigbenchqueriesmr.jar;' into .sql explicitly)
{code}
INFO [pool-1-thread-1]: client.RemoteDriver (RemoteDriver.java:call(316)) -
Failed to run job 8dd120cb-1a4d-4d1c-ba31-61eac648c27d
org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find
class: de.bankmark.bigbench.queries.q10.SentimentUDF
Serialization trace:
genericUDTF (org.apache.hadoop.hive.ql.plan.UDTFDesc)
conf (org.apache.hadoop.hive.ql.exec.UDTFOperator)
childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.MapJoinOperator)
childOperators (org.apache.hadoop.hive.ql.exec.FilterOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
right (org.apache.commons.lang3.tuple.ImmutablePair)
edgeProperties (org.apache.hadoop.hive.ql.plan.SparkWork)
at
org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:138)
at
org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:656)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:99)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at
[jira] [Updated] (HIVE-9410) ClassNotFoundException occurs during hive query case execution with UDF defined [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-9410:
--
Resolution: Fixed
Fix Version/s: spark-branch
Status: Resolved (was: Patch Available)
Committed to Spark branch. Thanks, Chengxiang.
ClassNotFoundException occurs during hive query case execution with UDF
defined [Spark Branch]
--
Key: HIVE-9410
URL: https://issues.apache.org/jira/browse/HIVE-9410
Project: Hive
Issue Type: Sub-task
Components: Spark
Environment: CentOS 6.5
JDK1.7
Reporter: Xin Hao
Assignee: Chengxiang Li
Fix For: spark-branch
Attachments: HIVE-9410.1-spark.patch, HIVE-9410.2-spark.patch,
HIVE-9410.3-spark.patch, HIVE-9410.4-spark.patch, HIVE-9410.4-spark.patch
We have a hive query case with UDF defined (i.e. BigBench case Q10, Q18
etc.). It will be passed for default Hive (on MR) mode, while failed for Hive
On Spark mode (both Standalone and Yarn-Client).
Although we use 'add jar .jar;' to add the UDF jar explicitly, the issue
still exists.
BTW, if we put the UDF jar into $HIVE_HOME/lib dir, the case will be passed.
Detail Error Message is as below (NOTE:
de.bankmark.bigbench.queries.q10.SentimentUDF is the UDF which contained in
jar bigbenchqueriesmr.jar, and we have add command like 'add jar
/location/to/bigbenchqueriesmr.jar;' into .sql explicitly)
{code}
INFO [pool-1-thread-1]: client.RemoteDriver (RemoteDriver.java:call(316)) -
Failed to run job 8dd120cb-1a4d-4d1c-ba31-61eac648c27d
org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find
class: de.bankmark.bigbench.queries.q10.SentimentUDF
Serialization trace:
genericUDTF (org.apache.hadoop.hive.ql.plan.UDTFDesc)
conf (org.apache.hadoop.hive.ql.exec.UDTFOperator)
childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.MapJoinOperator)
childOperators (org.apache.hadoop.hive.ql.exec.FilterOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
right (org.apache.commons.lang3.tuple.ImmutablePair)
edgeProperties (org.apache.hadoop.hive.ql.plan.SparkWork)
at
org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:138)
at
org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:656)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:99)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
...
Caused by: java.lang.ClassNotFoundException:
de.bankmark.bigbench.queries.q10.SentimentUDF
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at
org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:136)
... 55 more
{code}
--
This message was sent
[jira] [Commented] (HIVE-3280) Make HiveMetaStoreClient a public API
[
https://issues.apache.org/jira/browse/HIVE-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290518#comment-14290518
]
Hive QA commented on HIVE-3280:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12694317/HIVE-3280.1.patch
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 7366 tests executed
*Failed tests:*
{noformat}
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2505/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2505/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2505/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12694317 - PreCommit-HIVE-TRUNK-Build
Make HiveMetaStoreClient a public API
-
Key: HIVE-3280
URL: https://issues.apache.org/jira/browse/HIVE-3280
Project: Hive
Issue Type: Improvement
Components: Metastore
Reporter: Carl Steinbach
Assignee: Thejas M Nair
Labels: api-addition
Attachments: HIVE-3280.1.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9361) Intermittent NPE in SessionHiveMetaStoreClient.alterTempTable
[
https://issues.apache.org/jira/browse/HIVE-9361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290504#comment-14290504
]
Hive QA commented on HIVE-9361:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12694318/HIVE-9361.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 7366 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testSaslWithHiveMetaStore
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2504/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2504/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2504/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12694318 - PreCommit-HIVE-TRUNK-Build
Intermittent NPE in SessionHiveMetaStoreClient.alterTempTable
-
Key: HIVE-9361
URL: https://issues.apache.org/jira/browse/HIVE-9361
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.14.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman
Attachments: HIVE-9361.patch
it's happening at
{noformat}
MetaStoreUtils.updateUnpartitionedTableStatsFast(newtCopy,
wh.getFileStatusesForSD(newtCopy.getSd()), false, true);
{noformat}
other methods in this class call getWh() to get Warehouse so this likely
explains why it's intermittent.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9439) merge ORC disk ranges as we go when reading RGs
[ https://issues.apache.org/jira/browse/HIVE-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290512#comment-14290512 ] Lefty Leverenz commented on HIVE-9439: -- Acronym clarification: Does RG mean row group? merge ORC disk ranges as we go when reading RGs --- Key: HIVE-9439 URL: https://issues.apache.org/jira/browse/HIVE-9439 Project: Hive Issue Type: Improvement Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Minor Attachments: HIVE-9439.01.patch, HIVE-9439.patch Currently we get ranges for all the RGs individually, then merge them. We can do some (probably most of) the merging as we go. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9327) CBO (Calcite Return Path): Removing Row Resolvers from ParseContext
[ https://issues.apache.org/jira/browse/HIVE-9327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-9327: -- Attachment: HIVE-9327.06.patch CBO (Calcite Return Path): Removing Row Resolvers from ParseContext --- Key: HIVE-9327 URL: https://issues.apache.org/jira/browse/HIVE-9327 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: 0.15.0 Attachments: HIVE-9327.01.patch, HIVE-9327.02.patch, HIVE-9327.03.patch, HIVE-9327.04.patch, HIVE-9327.05.patch, HIVE-9327.06.patch, HIVE-9327.patch ParseContext includes a map of Operator to RowResolver (OpParseContext). It would be ideal to remove this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9327) CBO (Calcite Return Path): Removing Row Resolvers from ParseContext
[ https://issues.apache.org/jira/browse/HIVE-9327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-9327: -- Status: Open (was: Patch Available) CBO (Calcite Return Path): Removing Row Resolvers from ParseContext --- Key: HIVE-9327 URL: https://issues.apache.org/jira/browse/HIVE-9327 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: 0.15.0 Attachments: HIVE-9327.01.patch, HIVE-9327.02.patch, HIVE-9327.03.patch, HIVE-9327.04.patch, HIVE-9327.05.patch, HIVE-9327.06.patch, HIVE-9327.patch ParseContext includes a map of Operator to RowResolver (OpParseContext). It would be ideal to remove this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9327) CBO (Calcite Return Path): Removing Row Resolvers from ParseContext
[ https://issues.apache.org/jira/browse/HIVE-9327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-9327: -- Status: Patch Available (was: Open) CBO (Calcite Return Path): Removing Row Resolvers from ParseContext --- Key: HIVE-9327 URL: https://issues.apache.org/jira/browse/HIVE-9327 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: 0.15.0 Attachments: HIVE-9327.01.patch, HIVE-9327.02.patch, HIVE-9327.03.patch, HIVE-9327.04.patch, HIVE-9327.05.patch, HIVE-9327.patch CLEAR LIBRARY CACHE ParseContext includes a map of Operator to RowResolver (OpParseContext). It would be ideal to remove this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9327) CBO (Calcite Return Path): Removing Row Resolvers from ParseContext
[ https://issues.apache.org/jira/browse/HIVE-9327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-9327: -- Attachment: HIVE-9327.06.patch CBO (Calcite Return Path): Removing Row Resolvers from ParseContext --- Key: HIVE-9327 URL: https://issues.apache.org/jira/browse/HIVE-9327 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: 0.15.0 Attachments: HIVE-9327.01.patch, HIVE-9327.02.patch, HIVE-9327.03.patch, HIVE-9327.04.patch, HIVE-9327.05.patch, HIVE-9327.06.patch, HIVE-9327.patch CLEAR LIBRARY CACHE ParseContext includes a map of Operator to RowResolver (OpParseContext). It would be ideal to remove this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9327) CBO (Calcite Return Path): Removing Row Resolvers from ParseContext
[ https://issues.apache.org/jira/browse/HIVE-9327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-9327: -- Description: CLEAR LIBRARY CACHE ParseContext includes a map of Operator to RowResolver (OpParseContext). It would be ideal to remove this. was:ParseContext includes a map of Operator to RowResolver (OpParseContext). It would be ideal to remove this. CBO (Calcite Return Path): Removing Row Resolvers from ParseContext --- Key: HIVE-9327 URL: https://issues.apache.org/jira/browse/HIVE-9327 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: 0.15.0 Attachments: HIVE-9327.01.patch, HIVE-9327.02.patch, HIVE-9327.03.patch, HIVE-9327.04.patch, HIVE-9327.05.patch, HIVE-9327.patch CLEAR LIBRARY CACHE ParseContext includes a map of Operator to RowResolver (OpParseContext). It would be ideal to remove this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9327) CBO (Calcite Return Path): Removing Row Resolvers from ParseContext
[ https://issues.apache.org/jira/browse/HIVE-9327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-9327: -- Attachment: (was: HIVE-9327.06.patch) CBO (Calcite Return Path): Removing Row Resolvers from ParseContext --- Key: HIVE-9327 URL: https://issues.apache.org/jira/browse/HIVE-9327 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: 0.15.0 Attachments: HIVE-9327.01.patch, HIVE-9327.02.patch, HIVE-9327.03.patch, HIVE-9327.04.patch, HIVE-9327.05.patch, HIVE-9327.patch ParseContext includes a map of Operator to RowResolver (OpParseContext). It would be ideal to remove this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8340) Windows: HiveServer2 service doesn't stop backend jvm process, which prevents follow-up service start.
[ https://issues.apache.org/jira/browse/HIVE-8340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290528#comment-14290528 ] Lefty Leverenz commented on HIVE-8340: -- *hive.hadoop.classpath* is documented in the wiki: * [Configuration Properties -- HiveServer2 -- hive.hadoop.classpath | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.hadoop.classpath] Should it also be documented in Setting Up HiveServer2? * [Setting Up HiveServer2 -- How to Configure | https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2#SettingUpHiveServer2-HowtoConfigure] Windows: HiveServer2 service doesn't stop backend jvm process, which prevents follow-up service start. -- Key: HIVE-8340 URL: https://issues.apache.org/jira/browse/HIVE-8340 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.14.0 Environment: Windows Reporter: Xiaobing Zhou Assignee: Xiaobing Zhou Priority: Critical Labels: TODOC14 Fix For: 0.14.0 Attachments: HIVE-8340.1.patch, HIVE-8340.2.patch, HIVE-8340.3.patch, HIVE-8340.4.patch On stopping the HS2 service from the services tab, it only kills the root process and does not kill the child java process. As a result resources are not freed and this throws an error on restarting from command line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9431) CBO (Calcite Return Path): Removing AST from ParseContext
[ https://issues.apache.org/jira/browse/HIVE-9431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-9431: -- Status: Patch Available (was: Open) CBO (Calcite Return Path): Removing AST from ParseContext - Key: HIVE-9431 URL: https://issues.apache.org/jira/browse/HIVE-9431 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: 0.15.0 Attachments: HIVE-9431.01.patch, HIVE-9431.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9431) CBO (Calcite Return Path): Removing AST from ParseContext
[ https://issues.apache.org/jira/browse/HIVE-9431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-9431: -- Status: Open (was: Patch Available) CBO (Calcite Return Path): Removing AST from ParseContext - Key: HIVE-9431 URL: https://issues.apache.org/jira/browse/HIVE-9431 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: 0.15.0 Attachments: HIVE-9431.01.patch, HIVE-9431.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9431) CBO (Calcite Return Path): Removing AST from ParseContext
[ https://issues.apache.org/jira/browse/HIVE-9431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-9431: -- Attachment: HIVE-9431.01.patch CBO (Calcite Return Path): Removing AST from ParseContext - Key: HIVE-9431 URL: https://issues.apache.org/jira/browse/HIVE-9431 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: 0.15.0 Attachments: HIVE-9431.01.patch, HIVE-9431.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9444) CBO (Calcite Return Path): Rewrite GlobalLimitOptimizer
[ https://issues.apache.org/jira/browse/HIVE-9444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-9444: -- Attachment: HIVE-9444.patch CBO (Calcite Return Path): Rewrite GlobalLimitOptimizer --- Key: HIVE-9444 URL: https://issues.apache.org/jira/browse/HIVE-9444 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: 0.15.0 Attachments: HIVE-9444.patch Currently, GlobalLimitOptimization relies heavily on the information contained in QBParseInfo. The goal is to extract that information from the operator tree so we do not need to rely on QBParseInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9444) CBO (Calcite Return Path): Rewrite GlobalLimitOptimizer
[ https://issues.apache.org/jira/browse/HIVE-9444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-9444: -- Status: Patch Available (was: In Progress) CBO (Calcite Return Path): Rewrite GlobalLimitOptimizer --- Key: HIVE-9444 URL: https://issues.apache.org/jira/browse/HIVE-9444 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: 0.15.0 Attachments: HIVE-9444.patch Currently, GlobalLimitOptimization relies heavily on the information contained in QBParseInfo. The goal is to extract that information from the operator tree so we do not need to rely on QBParseInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8966) Delta files created by hive hcatalog streaming cannot be compacted
[
https://issues.apache.org/jira/browse/HIVE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290583#comment-14290583
]
Hive QA commented on HIVE-8966:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12694321/HIVE-8966.6.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 7370 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-parallel_join1.q-avro_joins.q-groupby_ppr.q-and-12-more -
did not produce a TEST-*.xml file
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2506/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2506/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2506/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12694321 - PreCommit-HIVE-TRUNK-Build
Delta files created by hive hcatalog streaming cannot be compacted
--
Key: HIVE-8966
URL: https://issues.apache.org/jira/browse/HIVE-8966
Project: Hive
Issue Type: Bug
Components: HCatalog
Affects Versions: 0.14.0
Environment: hive
Reporter: Jihong Liu
Assignee: Alan Gates
Priority: Critical
Fix For: 0.14.1
Attachments: HIVE-8966.2.patch, HIVE-8966.3.patch, HIVE-8966.4.patch,
HIVE-8966.5.patch, HIVE-8966.6.patch, HIVE-8966.patch
hive hcatalog streaming will also create a file like bucket_n_flush_length in
each delta directory. Where n is the bucket number. But the
compactor.CompactorMR think this file also needs to compact. However this
file of course cannot be compacted, so compactor.CompactorMR will not
continue to do the compaction.
Did a test, after removed the bucket_n_flush_length file, then the alter
table partition compact finished successfully. If don't delete that file,
nothing will be compacted.
This is probably a very severity bug. Both 0.13 and 0.14 have this issue
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-6617) Reduce ambiguity in grammar
[
https://issues.apache.org/jira/browse/HIVE-6617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290626#comment-14290626
]
Hive QA commented on HIVE-6617:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12694328/HIVE-6617.08.patch
{color:red}ERROR:{color} -1 due to 62 failed/errored test(s), 7367 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters_overlap
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lateral_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lateral_view_cp
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lateral_view_noalias
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lateral_view_outer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lateral_view_ppd
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_leadlag
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_limit_pushdown_negative
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part15
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonmr_fetch
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonreserved_keywords_input37
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonreserved_keywords_insert_into1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_field_garbage
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf_matchpath
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf_rcfile
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf_register_tblfn
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf_seqfile
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf_streaming
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_mapjoin6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_covar_pop
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_explode
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_inline
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udtf_explode
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udtf_json_tuple
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udtf_parse_url_tuple
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udtf_posexplode
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udtf_stack
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union26
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_lateralview
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_distinct_gby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_ptf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_navfn
org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver_lateral_view_explode2
org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver_udtf_output_on_close
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_lvj_mapjoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_ptf
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_ptf_streaming
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_distinct_gby
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_ptf
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_cannot_create_none_role
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_lateral_view_alias
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_lateral_view_join
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_ptf_negative_HavingLeadWithPTF
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_ptf_negative_JoinWithAmbigousAlias
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udf_assert_true
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udf_assert_true2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_filters_overlap
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part15
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ptf
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ptf_matchpath
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ptf_rcfile
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ptf_register_tblfn
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ptf_seqfile
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ptf_streaming
[jira] [Updated] (HIVE-9449) Push YARN configuration to Spark while deply Spark on YARN[Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-9449: -- Attachment: HIVE-9449.1-spark.patch Reattached the same patch to have another test run. Push YARN configuration to Spark while deply Spark on YARN[Spark Branch] Key: HIVE-9449 URL: https://issues.apache.org/jira/browse/HIVE-9449 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Chengxiang Li Assignee: Chengxiang Li Attachments: HIVE-9449.1-spark.patch, HIVE-9449.1-spark.patch We only push Spark configuration and RSC configuration to Spark while launch Spark cluster now, for Spark on YARN mode, Spark need extra YARN configuration to launch Spark cluster. Besides this, to support dynamically configuration setting for RSC configuration/YARN configuration, we need to recreate SparkSession while RSC configuration/YARN configuration update as well, as they may influence the Spark cluster deployment as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9428) LocalSparkJobStatus may return failed job as successful [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290636#comment-14290636 ] Xuefu Zhang commented on HIVE-9428: --- [~lirui], It's fine to check both isDone() and get(). As to the exception, since we eat the exception, I think it makes sense to log it here. Something like: LOG.log(Failed to ..., ex); LocalSparkJobStatus may return failed job as successful [Spark Branch] -- Key: HIVE-9428 URL: https://issues.apache.org/jira/browse/HIVE-9428 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Rui Li Assignee: Rui Li Priority: Minor Attachments: HIVE-9428.1-spark.patch, HIVE-9428.2-spark.patch Future is done doesn't necessarily mean the job is successful. We should rely on SparkJobInfo to get job status whenever it's available. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9409) Avoid ser/de loggers as logging framework can be incompatible on driver and workers
[ https://issues.apache.org/jira/browse/HIVE-9409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-9409: -- Resolution: Fixed Fix Version/s: 0.15.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks, Rui. Avoid ser/de loggers as logging framework can be incompatible on driver and workers --- Key: HIVE-9409 URL: https://issues.apache.org/jira/browse/HIVE-9409 Project: Hive Issue Type: Sub-task Components: Spark Environment: CentOS6.5 Java version: 1.7.0_67 Reporter: Xin Hao Assignee: Rui Li Fix For: 0.15.0 Attachments: HIVE-9409.1.patch, HIVE-9409.1.patch, HIVE-9409.1.patch When we use current [Spark Branch] to build hive package. deploy it on our cluster and execute hive queries (e.g. BigBench case Q10, Q18, Q19, Q27) by default mode (i.e. just Hive on MR, not HiveOnSpark), Error 'java.lang.ClassNotFoundException: org.apache.commons.logging.impl.SLF4JLocationAwareLog' will occurs. For other released apache or CDH hive version(e.g. apache hive 0.14), there is no this issue. By the way, if we use 'add jar /location/to/jcl-over-slf4j-1.7.5.jar' before hive query execution, the issue will be workaround. The detail diagnostic messages are as below: == Diagnostic Messages for this Task: Error: java.lang.RuntimeException: Failed to load plan: hdfs://bhx1:8020/tmp/hive/root/4a4cbeb2-cf42-4eb7-a78a-7ecea6af2aff/hive_2015-01-17_10-45-51_360_5581900288096206774-1/-mr-10004/1c6c4667-8b81-41ed-a42e-fe099ae3379f/map.xml: org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find class: org.apache.commons.logging.impl.SLF4JLocationAwareLog Serialization trace: LOG (org.apache.hadoop.hive.ql.exec.UDTFOperator) childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator) childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator) aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork) at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:431) at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:287) at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:268) at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:484) at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:477) at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:657) at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.init(MapTask.java:169) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find cl Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find class: org.apache.commons.logging.impl.SLF4JLocationAwareLog Serialization trace: LOG (org.apache.hadoop.hive.ql.exec.UDTFOperator) childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator) childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator) aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork) at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:138) at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115) at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:656) at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:99) at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776) at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112) at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18) at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) at
[jira] [Commented] (HIVE-9448) Merge spark to trunk 1/23/15
[ https://issues.apache.org/jira/browse/HIVE-9448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290623#comment-14290623 ] Xuefu Zhang commented on HIVE-9448: --- +1 to the code changes. However, it seems that a few qtests need updated output. Merge spark to trunk 1/23/15 Key: HIVE-9448 URL: https://issues.apache.org/jira/browse/HIVE-9448 Project: Hive Issue Type: Bug Components: Spark Affects Versions: 0.15.0 Reporter: Szehon Ho Assignee: Szehon Ho Attachments: HIVE-9448.2.patch, HIVE-9448.patch Merging latest spark changes to trunk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9410) ClassNotFoundException occurs during hive query case execution with UDF defined [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-9410:
--
Attachment: HIVE-9410.4-spark.patch
Attach the same patch for another test run.
ClassNotFoundException occurs during hive query case execution with UDF
defined [Spark Branch]
--
Key: HIVE-9410
URL: https://issues.apache.org/jira/browse/HIVE-9410
Project: Hive
Issue Type: Sub-task
Components: Spark
Environment: CentOS 6.5
JDK1.7
Reporter: Xin Hao
Assignee: Chengxiang Li
Attachments: HIVE-9410.1-spark.patch, HIVE-9410.2-spark.patch,
HIVE-9410.3-spark.patch, HIVE-9410.4-spark.patch, HIVE-9410.4-spark.patch
We have a hive query case with UDF defined (i.e. BigBench case Q10, Q18
etc.). It will be passed for default Hive (on MR) mode, while failed for Hive
On Spark mode (both Standalone and Yarn-Client).
Although we use 'add jar .jar;' to add the UDF jar explicitly, the issue
still exists.
BTW, if we put the UDF jar into $HIVE_HOME/lib dir, the case will be passed.
Detail Error Message is as below (NOTE:
de.bankmark.bigbench.queries.q10.SentimentUDF is the UDF which contained in
jar bigbenchqueriesmr.jar, and we have add command like 'add jar
/location/to/bigbenchqueriesmr.jar;' into .sql explicitly)
{code}
INFO [pool-1-thread-1]: client.RemoteDriver (RemoteDriver.java:call(316)) -
Failed to run job 8dd120cb-1a4d-4d1c-ba31-61eac648c27d
org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find
class: de.bankmark.bigbench.queries.q10.SentimentUDF
Serialization trace:
genericUDTF (org.apache.hadoop.hive.ql.plan.UDTFDesc)
conf (org.apache.hadoop.hive.ql.exec.UDTFOperator)
childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.MapJoinOperator)
childOperators (org.apache.hadoop.hive.ql.exec.FilterOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
right (org.apache.commons.lang3.tuple.ImmutablePair)
edgeProperties (org.apache.hadoop.hive.ql.plan.SparkWork)
at
org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:138)
at
org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:656)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:99)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
...
Caused by: java.lang.ClassNotFoundException:
de.bankmark.bigbench.queries.q10.SentimentUDF
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at
org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:136)
... 55 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (HIVE-9449) Push YARN configuration to Spark while deply Spark on YARN[Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-9449:
--
Comment: was deleted
(was:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12694107/HIVE-9449.1-spark.patch
{color:red}ERROR:{color} -1 due to 65 failed/errored test(s), 7355 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_excludeHadoop20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_multi
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join27
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_simple_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_subq_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_column_access_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_logical
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataOnlyOptimizer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataonly1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_gby2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join_filter
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_union_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_vc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rcfile_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_25
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in_having
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin_having
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_views
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_table_access_keys_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union24
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union28
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union30
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_null
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_6_subq
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_mapjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_mapjoin_reduce
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_simple_select
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_subq_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_filter_join_breaktask
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_filter_join_breaktask2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mrr
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_mapjoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_mapjoin_reduce
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
[jira] [Commented] (HIVE-9449) Push YARN configuration to Spark while deply Spark on YARN[Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290631#comment-14290631 ] Xuefu Zhang commented on HIVE-9449: --- +1 pending on tests. Push YARN configuration to Spark while deply Spark on YARN[Spark Branch] Key: HIVE-9449 URL: https://issues.apache.org/jira/browse/HIVE-9449 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Chengxiang Li Assignee: Chengxiang Li Attachments: HIVE-9449.1-spark.patch, HIVE-9449.1-spark.patch We only push Spark configuration and RSC configuration to Spark while launch Spark cluster now, for Spark on YARN mode, Spark need extra YARN configuration to launch Spark cluster. Besides this, to support dynamically configuration setting for RSC configuration/YARN configuration, we need to recreate SparkSession while RSC configuration/YARN configuration update as well, as they may influence the Spark cluster deployment as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7616) pre-size mapjoin hashtable based on statistics
[
https://issues.apache.org/jira/browse/HIVE-7616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290853#comment-14290853
]
Lefty Leverenz commented on HIVE-7616:
--
Doc error: The description of *hive.hashtable.initialCapacity* refers to a
parameter that existed in patch 2
(hive.hashtable.stats.key.estimate.adjustment) but was renamed
*hive.hashtable.key.count.adjustment* in patch 3.
{quote}
+HIVEHASHTABLEKEYCOUNTADJUSTMENT(hive.hashtable.key.count.adjustment,
1.0f,
+Adjustment to mapjoin hashtable size derived from table and column
statistics; the estimate +
+ of the number of keys is divided by this value. If the value is 0,
statistics are not used +
+and hive.hashtable.initialCapacity is used instead.),
+HIVEHASHTABLETHRESHOLD(hive.hashtable.initialCapacity, 10, Initial
capacity of +
+mapjoin hashtable if statistics are absent, or if
hive.hashtable.stats.key.estimate.adjustment is set to 0),
{quote}
Opened HIVE-9457 to fix this.
pre-size mapjoin hashtable based on statistics
--
Key: HIVE-7616
URL: https://issues.apache.org/jira/browse/HIVE-7616
Project: Hive
Issue Type: Improvement
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Labels: TODOC14
Fix For: 0.14.0
Attachments: HIVE-7616.01.patch, HIVE-7616.02.patch,
HIVE-7616.03.patch, HIVE-7616.04.patch, HIVE-7616.05.patch,
HIVE-7616.06.patch, HIVE-7616.07.patch, HIVE-7616.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
