[jira] [Updated] (HIVE-6083) User provided table properties are not assigned to the TableDesc of the FileSinkDesc in a CTAS query
[
https://issues.apache.org/jira/browse/HIVE-6083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yin Huai updated HIVE-6083:
---
Status: Open (was: Patch Available)
User provided table properties are not assigned to the TableDesc of the
FileSinkDesc in a CTAS query
Key: HIVE-6083
URL: https://issues.apache.org/jira/browse/HIVE-6083
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0, 0.13.0
Reporter: Yin Huai
Assignee: Yin Huai
Attachments: HIVE-6083.1.patch.txt, HIVE-6083.2.patch.txt
I was trying to use a CTAS query to create a table stored with ORC and
orc.compress was set to SNAPPY. However, the table was still compressed as
ZLIB (although the result of DESCRIBE still shows that this table is
compressed by SNAPPY). For a CTAS query, SemanticAnalyzer.genFileSinkPlan
uses CreateTableDesc to generate the TableDesc for the FileSinkDesc by
calling PlanUtils.getTableDesc. However, in PlanUtils.getTableDesc, I do not
see user provided table properties are assigned to the returned TableDesc
(CreateTableDesc.getTblProps was not called in this method ).
btw, I only checked the code of 0.12 and trunk.
Two examples:
* Snappy compression
{code}
create table web_sales_wrong_orc_snappy
stored as orc tblproperties (orc.compress=SNAPPY)
as select * from web_sales;
{code}
{code}
describe formatted web_sales_wrong_orc_snappy;
Location:
hdfs://localhost:54310/user/hive/warehouse/web_sales_wrong_orc_snappy
Table Type: MANAGED_TABLE
Table Parameters:
COLUMN_STATS_ACCURATE true
numFiles1
numRows 719384
orc.compressSNAPPY
rawDataSize 97815412
totalSize 40625243
transient_lastDdlTime 1387566015
{code}
{code}
bin/hive --orcfiledump
/user/hive/warehouse/web_sales_wrong_orc_snappy/00_0
Rows: 719384
Compression: ZLIB
Compression size: 262144
...
{code}
* No compression
{code}
create table web_sales_wrong_orc_none
stored as orc tblproperties (orc.compress=NONE)
as select * from web_sales;
{code}
{code}
describe formatted web_sales_wrong_orc_none;
Location:
hdfs://localhost:54310/user/hive/warehouse/web_sales_wrong_orc_none
Table Type: MANAGED_TABLE
Table Parameters:
COLUMN_STATS_ACCURATE true
numFiles1
numRows 719384
orc.compressNONE
rawDataSize 97815412
totalSize 40625243
transient_lastDdlTime 1387566064
{code}
{code}
bin/hive --orcfiledump /user/hive/warehouse/web_sales_wrong_orc_none/00_0
Rows: 719384
Compression: ZLIB
Compression size: 262144
...
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HIVE-6083) User provided table properties are not assigned to the TableDesc of the FileSinkDesc in a CTAS query
[
https://issues.apache.org/jira/browse/HIVE-6083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yin Huai updated HIVE-6083:
---
Status: Patch Available (was: Open)
User provided table properties are not assigned to the TableDesc of the
FileSinkDesc in a CTAS query
Key: HIVE-6083
URL: https://issues.apache.org/jira/browse/HIVE-6083
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0, 0.13.0
Reporter: Yin Huai
Assignee: Yin Huai
Attachments: HIVE-6083.1.patch.txt, HIVE-6083.2.patch.txt
I was trying to use a CTAS query to create a table stored with ORC and
orc.compress was set to SNAPPY. However, the table was still compressed as
ZLIB (although the result of DESCRIBE still shows that this table is
compressed by SNAPPY). For a CTAS query, SemanticAnalyzer.genFileSinkPlan
uses CreateTableDesc to generate the TableDesc for the FileSinkDesc by
calling PlanUtils.getTableDesc. However, in PlanUtils.getTableDesc, I do not
see user provided table properties are assigned to the returned TableDesc
(CreateTableDesc.getTblProps was not called in this method ).
btw, I only checked the code of 0.12 and trunk.
Two examples:
* Snappy compression
{code}
create table web_sales_wrong_orc_snappy
stored as orc tblproperties (orc.compress=SNAPPY)
as select * from web_sales;
{code}
{code}
describe formatted web_sales_wrong_orc_snappy;
Location:
hdfs://localhost:54310/user/hive/warehouse/web_sales_wrong_orc_snappy
Table Type: MANAGED_TABLE
Table Parameters:
COLUMN_STATS_ACCURATE true
numFiles1
numRows 719384
orc.compressSNAPPY
rawDataSize 97815412
totalSize 40625243
transient_lastDdlTime 1387566015
{code}
{code}
bin/hive --orcfiledump
/user/hive/warehouse/web_sales_wrong_orc_snappy/00_0
Rows: 719384
Compression: ZLIB
Compression size: 262144
...
{code}
* No compression
{code}
create table web_sales_wrong_orc_none
stored as orc tblproperties (orc.compress=NONE)
as select * from web_sales;
{code}
{code}
describe formatted web_sales_wrong_orc_none;
Location:
hdfs://localhost:54310/user/hive/warehouse/web_sales_wrong_orc_none
Table Type: MANAGED_TABLE
Table Parameters:
COLUMN_STATS_ACCURATE true
numFiles1
numRows 719384
orc.compressNONE
rawDataSize 97815412
totalSize 40625243
transient_lastDdlTime 1387566064
{code}
{code}
bin/hive --orcfiledump /user/hive/warehouse/web_sales_wrong_orc_none/00_0
Rows: 719384
Compression: ZLIB
Compression size: 262144
...
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
Hive-trunk-hadoop2 - Build # 636 - Still Failing
Changes for Build #591 [xuefu] HIVE-5872: Make UDAFs such as GenericUDAFSum report accurate precision/scale for decimal types (reviewed by Sergey Shelukhin) [hashutosh] HIVE-5978 : Rollups not supported in vector mode. (Jitendra Nath Pandey via Ashutosh Chauhan) [hashutosh] HIVE-5830 : SubQuery: Not In subqueries should check if subquery contains nulls in matching column (Harish Butani via Ashutosh Chauhan) [hashutosh] HIVE-5598 : Remove dummy new line at the end of non-sql commands (Navis via Ashutosh Chauhan) Changes for Build #592 [hashutosh] HIVE-5982 : Remove redundant filesystem operations and methods in FileSink (Ashutosh Chauhan via Thejas Nair) [navis] HIVE-5955 : decimal_precision.q test case fails in trunk (Prasanth J via Navis) [brock] HIVE-5983 - Fix name of ColumnProjectionUtils.appendReadColumnIDs (Brock Noland reviewed by Navis) Changes for Build #593 [omalley] HIVE-5580. Predicate pushdown predicates with an and-operator between non-SARGable predicates cause a NPE. (omalley) Changes for Build #594 [gunther] HIVE-6000: Hive build broken on hadoop2 (Vikram Dixit K via Gunther Hagleitner [gunther] HIVE-2093: UPDATE - add two missing files from previous commit (Gunther Hagleitner) [thejas] HIVE-2093 : create/drop database should populate inputs/outputs and check concurrency and user permission (Navis via Thejas Nair) [hashutosh] HIVE-6016 : Hadoop23Shims has a bug in listLocatedStatus impl. (Prasanth J via Ashutosh Chauhan) [hashutosh] HIVE-5994 : ORC RLEv2 encodes wrongly for large negative BIGINTs (64 bits ) (Prasanth J via Owen Omalley) [hashutosh] HIVE-5991 : ORC RLEv2 fails with ArrayIndexOutOfBounds exception for PATCHED_BLOB encoding (Prasanth J via Owen Omalley) [prasadm] HIVE-4395: Support TFetchOrientation.FIRST for HiveServer2 FetchResults (Prasad Mujumdar reviewed by Thejas Nair) [ehans] HIVE-5756: Implement vectorized support for IF conditional expression (Eric Hanson) [hashutosh] HIVE-6018 : FetchTask should not reference metastore classes (Navis via Prasad Mujumdar) [hashutosh] HIVE-5979. Failure in cast to timestamps. (Jitendra Pandey) [hashutosh] HIVE-5897 : Fix hadoop2 execution environment Milestone 2 (Vikram Dixit via Brock Noland) Changes for Build #595 Changes for Build #596 [hashutosh] HIVE-6027 : non-vectorized log10 has rounding issue (Sergey Shelukhin via Ashutosh Chauhan) [prasadm] HIVE-5993: JDBC Driver should not hard-code the database name (Szehon Ho via Prasad Mujumdar) Changes for Build #597 [hashutosh] HIVE-6004 : Fix statistics annotation related test failures in hadoop2 (Prasanth J via Ashutosh Chauhan) Changes for Build #598 [navis] HIVE-5985 : Make qfile_regex to accept multiple patterns (Navis reviewed by Ashutosh Chauhan) Changes for Build #599 Changes for Build #600 Changes for Build #601 [navis] HIVE-5276 : Skip redundant string encoding/decoding for hiveserver2 (Navis Reviewed by Carl Steinbach) Changes for Build #602 [xuefu] HIVE-6022: Load statements with incorrect order of partitions put input files to unreadable places (Teruyoshi Zenmyo via Xuefu) Changes for Build #603 Changes for Build #604 [thejas] HIVE-5975 : [WebHCat] templeton mapreduce job failed if provide define parameters (Shanyu Zhao via Thejas Nair) Changes for Build #605 [prasadm] HIVE-1466: Add NULL DEFINED AS to ROW FORMAT specification (Prasad Mujumdar reviewed by Xuefu Zhang) Changes for Build #606 [jitendra] HIVE-5521 : Remove CommonRCFileInputFormat. (hashutosh via jitendra) [rhbutani] HIVE-5973 SMB joins produce incorrect results with multiple partitions and buckets (Vikram Dixit via Harish Butani) [ehans] HIVE-6015: vectorized logarithm produces results for 0 that are different from a non-vectorized one (Sergey Shelukhin via Eric Hanson) Changes for Build #607 [brock] HIVE-5812 - HiveServer2 SSL connection transport binds to loopback address by default (Prasad Mujumdar via Brock Noland) Changes for Build #608 [hashutosh] HIVE-5936 : analyze command failing to collect stats with counter mechanism (Navis via Ashutosh Chauhan) Changes for Build #609 [thejas] HIVE-5230 : Better error reporting by async threads in HiveServer2 (Vaibhav Gumashta via Thejas Nair) Changes for Build #610 [navis] HIVE-5879 : Fix spelling errors in hive-default.xml.template (Lefty Leverenz via Navis) Changes for Build #611 Changes for Build #612 [xuefu] HIVE-6021: Problem in GroupByOperator for handling distinct aggrgations (Sun Rui via Xuefu) Changes for Build #613 [prasadm] HIVE-6036: A test case for embedded beeline - with URL jdbc:hive2:///default (Anandha L Ranganathan via Prasad Mujumdar) [prasadm] HIVE-4256: JDBC2 HiveConnection does not use the specified database (Anandha L Ranganathan via Prasad Mujumdar) Changes for Build #614 [brock] HIVE-5966 - Fix eclipse:eclipse post shim aggregation changes (Szehon Ho via Brock Noland) Changes for Build #615 [daijy] HIVE-5540: webhcat e2e test failures: Expect
Hive-trunk-h0.21 - Build # 2535 - Still Failing
Changes for Build #2493 [xuefu] HIVE-5872: Make UDAFs such as GenericUDAFSum report accurate precision/scale for decimal types (reviewed by Sergey Shelukhin) [hashutosh] HIVE-5978 : Rollups not supported in vector mode. (Jitendra Nath Pandey via Ashutosh Chauhan) [hashutosh] HIVE-5830 : SubQuery: Not In subqueries should check if subquery contains nulls in matching column (Harish Butani via Ashutosh Chauhan) [hashutosh] HIVE-5598 : Remove dummy new line at the end of non-sql commands (Navis via Ashutosh Chauhan) Changes for Build #2494 [hashutosh] HIVE-5982 : Remove redundant filesystem operations and methods in FileSink (Ashutosh Chauhan via Thejas Nair) [navis] HIVE-5955 : decimal_precision.q test case fails in trunk (Prasanth J via Navis) [brock] HIVE-5983 - Fix name of ColumnProjectionUtils.appendReadColumnIDs (Brock Noland reviewed by Navis) Changes for Build #2495 [omalley] HIVE-5580. Predicate pushdown predicates with an and-operator between non-SARGable predicates cause a NPE. (omalley) Changes for Build #2496 [gunther] HIVE-6000: Hive build broken on hadoop2 (Vikram Dixit K via Gunther Hagleitner [gunther] HIVE-2093: UPDATE - add two missing files from previous commit (Gunther Hagleitner) [thejas] HIVE-2093 : create/drop database should populate inputs/outputs and check concurrency and user permission (Navis via Thejas Nair) [hashutosh] HIVE-6016 : Hadoop23Shims has a bug in listLocatedStatus impl. (Prasanth J via Ashutosh Chauhan) [hashutosh] HIVE-5994 : ORC RLEv2 encodes wrongly for large negative BIGINTs (64 bits ) (Prasanth J via Owen Omalley) [hashutosh] HIVE-5991 : ORC RLEv2 fails with ArrayIndexOutOfBounds exception for PATCHED_BLOB encoding (Prasanth J via Owen Omalley) [prasadm] HIVE-4395: Support TFetchOrientation.FIRST for HiveServer2 FetchResults (Prasad Mujumdar reviewed by Thejas Nair) [ehans] HIVE-5756: Implement vectorized support for IF conditional expression (Eric Hanson) [hashutosh] HIVE-6018 : FetchTask should not reference metastore classes (Navis via Prasad Mujumdar) [hashutosh] HIVE-5979. Failure in cast to timestamps. (Jitendra Pandey) [hashutosh] HIVE-5897 : Fix hadoop2 execution environment Milestone 2 (Vikram Dixit via Brock Noland) Changes for Build #2497 Changes for Build #2498 [hashutosh] HIVE-6004 : Fix statistics annotation related test failures in hadoop2 (Prasanth J via Ashutosh Chauhan) [hashutosh] HIVE-6027 : non-vectorized log10 has rounding issue (Sergey Shelukhin via Ashutosh Chauhan) [prasadm] HIVE-5993: JDBC Driver should not hard-code the database name (Szehon Ho via Prasad Mujumdar) Changes for Build #2499 [navis] HIVE-5985 : Make qfile_regex to accept multiple patterns (Navis reviewed by Ashutosh Chauhan) Changes for Build #2500 Changes for Build #2501 Changes for Build #2502 [navis] HIVE-5276 : Skip redundant string encoding/decoding for hiveserver2 (Navis Reviewed by Carl Steinbach) Changes for Build #2503 [xuefu] HIVE-6022: Load statements with incorrect order of partitions put input files to unreadable places (Teruyoshi Zenmyo via Xuefu) Changes for Build #2504 Changes for Build #2505 [thejas] HIVE-5975 : [WebHCat] templeton mapreduce job failed if provide define parameters (Shanyu Zhao via Thejas Nair) Changes for Build #2506 [prasadm] HIVE-1466: Add NULL DEFINED AS to ROW FORMAT specification (Prasad Mujumdar reviewed by Xuefu Zhang) Changes for Build #2507 [jitendra] HIVE-5521 : Remove CommonRCFileInputFormat. (hashutosh via jitendra) [rhbutani] HIVE-5973 SMB joins produce incorrect results with multiple partitions and buckets (Vikram Dixit via Harish Butani) [ehans] HIVE-6015: vectorized logarithm produces results for 0 that are different from a non-vectorized one (Sergey Shelukhin via Eric Hanson) Changes for Build #2508 [brock] HIVE-5812 - HiveServer2 SSL connection transport binds to loopback address by default (Prasad Mujumdar via Brock Noland) Changes for Build #2509 [hashutosh] HIVE-5936 : analyze command failing to collect stats with counter mechanism (Navis via Ashutosh Chauhan) Changes for Build #2510 [thejas] HIVE-5230 : Better error reporting by async threads in HiveServer2 (Vaibhav Gumashta via Thejas Nair) Changes for Build #2511 [navis] HIVE-5879 : Fix spelling errors in hive-default.xml.template (Lefty Leverenz via Navis) Changes for Build #2512 Changes for Build #2513 [xuefu] HIVE-6021: Problem in GroupByOperator for handling distinct aggrgations (Sun Rui via Xuefu) Changes for Build #2514 [prasadm] HIVE-6036: A test case for embedded beeline - with URL jdbc:hive2:///default (Anandha L Ranganathan via Prasad Mujumdar) [prasadm] HIVE-4256: JDBC2 HiveConnection does not use the specified database (Anandha L Ranganathan via Prasad Mujumdar) Changes for Build #2515 [brock] HIVE-5966 - Fix eclipse:eclipse post shim aggregation changes (Szehon Ho via Brock Noland) Changes for Build #2516 [daijy] HIVE-5540: webhcat e2e test failures: Expect
[jira] [Updated] (HIVE-6116) Use Paths consistently III
[ https://issues.apache.org/jira/browse/HIVE-6116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6116: --- Resolution: Fixed Fix Version/s: 0.13.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks, Xuefu for review! Use Paths consistently III -- Key: HIVE-6116 URL: https://issues.apache.org/jira/browse/HIVE-6116 Project: Hive Issue Type: Improvement Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.13.0 Attachments: HIVE-6116.2.patch, HIVE-6116.3.patch, HIVE-6116.patch Another one in patch series to make use of Paths consistently. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
Re: Review Request 16526: more usage of paths
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16526/ --- (Updated Dec. 31, 2013, 5:23 p.m.) Review request for hive, Xuefu Zhang and Xuefu Zhang. Changes --- * Rebased after HIVE-6116 went in trunk. * Whitespace changes for indentation. * Updated .q.out for failed tests. * No other code changes. Bugs: HIVE-6121 https://issues.apache.org/jira/browse/HIVE-6121 Repository: hive Description --- Refactoring patch. Diffs (updated) - trunk/hcatalog/core/src/test/java/org/apache/hcatalog/mapreduce/TestHCatMultiOutputFormat.java 1554560 trunk/hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatMultiOutputFormat.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/MergeWork.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/stats/PartialScanTask.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/stats/PartialScanWork.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/truncate/ColumnTruncateMapper.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/truncate/ColumnTruncateTask.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/truncate/ColumnTruncateWork.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SortMergeJoinTaskDispatcher.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/AlterTablePartMergeFilesDesc.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ArchiveWork.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CopyWork.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExplainWork.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/LoadMultiFilesDesc.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MoveWork.java 1554560 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TruncateTableDesc.java 1554560 trunk/ql/src/test/results/clientpositive/binary_output_format.q.out 1554560 trunk/ql/src/test/results/clientpositive/bucket1.q.out 1554560 trunk/ql/src/test/results/clientpositive/bucket2.q.out 1554560 trunk/ql/src/test/results/clientpositive/bucket3.q.out 1554560 trunk/ql/src/test/results/clientpositive/bucket4.q.out 1554560 trunk/ql/src/test/results/clientpositive/bucket5.q.out 1554560 trunk/ql/src/test/results/clientpositive/bucketmapjoin1.q.out 1554560 trunk/ql/src/test/results/clientpositive/bucketmapjoin2.q.out 1554560 trunk/ql/src/test/results/clientpositive/bucketmapjoin3.q.out 1554560 trunk/ql/src/test/results/clientpositive/bucketmapjoin4.q.out 1554560 trunk/ql/src/test/results/clientpositive/bucketmapjoin5.q.out 1554560 trunk/ql/src/test/results/clientpositive/bucketmapjoin_negative.q.out 1554560 trunk/ql/src/test/results/clientpositive/bucketmapjoin_negative2.q.out 1554560 trunk/ql/src/test/results/clientpositive/disable_merge_for_bucketing.q.out 1554560 trunk/ql/src/test/results/clientpositive/groupby_map_ppr.q.out 1554560 trunk/ql/src/test/results/clientpositive/groupby_map_ppr_multi_distinct.q.out 1554560 trunk/ql/src/test/results/clientpositive/groupby_ppr.q.out 1554560 trunk/ql/src/test/results/clientpositive/groupby_ppr_multi_distinct.q.out 1554560
[jira] [Updated] (HIVE-6121) Use Paths Consistently - IV
[ https://issues.apache.org/jira/browse/HIVE-6121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6121: --- Status: Open (was: Patch Available) Use Paths Consistently - IV --- Key: HIVE-6121 URL: https://issues.apache.org/jira/browse/HIVE-6121 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-6121.2.patch, HIVE-6121.3.patch, HIVE-6121.patch Next one in patch series to fix Hive to use paths consistently. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6121) Use Paths Consistently - IV
[ https://issues.apache.org/jira/browse/HIVE-6121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6121: --- Status: Patch Available (was: Open) Use Paths Consistently - IV --- Key: HIVE-6121 URL: https://issues.apache.org/jira/browse/HIVE-6121 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-6121.2.patch, HIVE-6121.3.patch, HIVE-6121.patch Next one in patch series to fix Hive to use paths consistently. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6121) Use Paths Consistently - IV
[ https://issues.apache.org/jira/browse/HIVE-6121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6121: --- Attachment: HIVE-6121.3.patch * Rebased after HIVE-6116 went in trunk. * Whitespace changes for indentation. * Updated .q.out for failed tests. * No other code changes. Use Paths Consistently - IV --- Key: HIVE-6121 URL: https://issues.apache.org/jira/browse/HIVE-6121 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-6121.2.patch, HIVE-6121.3.patch, HIVE-6121.patch Next one in patch series to fix Hive to use paths consistently. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6121) Use Paths Consistently - IV
[ https://issues.apache.org/jira/browse/HIVE-6121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859583#comment-13859583 ] Xuefu Zhang commented on HIVE-6121: --- +1 Use Paths Consistently - IV --- Key: HIVE-6121 URL: https://issues.apache.org/jira/browse/HIVE-6121 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-6121.2.patch, HIVE-6121.3.patch, HIVE-6121.patch Next one in patch series to fix Hive to use paths consistently. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
Hive-trunk-hadoop2 - Build # 637 - Still Failing
Changes for Build #593 [omalley] HIVE-5580. Predicate pushdown predicates with an and-operator between non-SARGable predicates cause a NPE. (omalley) Changes for Build #594 [gunther] HIVE-6000: Hive build broken on hadoop2 (Vikram Dixit K via Gunther Hagleitner [gunther] HIVE-2093: UPDATE - add two missing files from previous commit (Gunther Hagleitner) [thejas] HIVE-2093 : create/drop database should populate inputs/outputs and check concurrency and user permission (Navis via Thejas Nair) [hashutosh] HIVE-6016 : Hadoop23Shims has a bug in listLocatedStatus impl. (Prasanth J via Ashutosh Chauhan) [hashutosh] HIVE-5994 : ORC RLEv2 encodes wrongly for large negative BIGINTs (64 bits ) (Prasanth J via Owen Omalley) [hashutosh] HIVE-5991 : ORC RLEv2 fails with ArrayIndexOutOfBounds exception for PATCHED_BLOB encoding (Prasanth J via Owen Omalley) [prasadm] HIVE-4395: Support TFetchOrientation.FIRST for HiveServer2 FetchResults (Prasad Mujumdar reviewed by Thejas Nair) [ehans] HIVE-5756: Implement vectorized support for IF conditional expression (Eric Hanson) [hashutosh] HIVE-6018 : FetchTask should not reference metastore classes (Navis via Prasad Mujumdar) [hashutosh] HIVE-5979. Failure in cast to timestamps. (Jitendra Pandey) [hashutosh] HIVE-5897 : Fix hadoop2 execution environment Milestone 2 (Vikram Dixit via Brock Noland) Changes for Build #595 Changes for Build #596 [hashutosh] HIVE-6027 : non-vectorized log10 has rounding issue (Sergey Shelukhin via Ashutosh Chauhan) [prasadm] HIVE-5993: JDBC Driver should not hard-code the database name (Szehon Ho via Prasad Mujumdar) Changes for Build #597 [hashutosh] HIVE-6004 : Fix statistics annotation related test failures in hadoop2 (Prasanth J via Ashutosh Chauhan) Changes for Build #598 [navis] HIVE-5985 : Make qfile_regex to accept multiple patterns (Navis reviewed by Ashutosh Chauhan) Changes for Build #599 Changes for Build #600 Changes for Build #601 [navis] HIVE-5276 : Skip redundant string encoding/decoding for hiveserver2 (Navis Reviewed by Carl Steinbach) Changes for Build #602 [xuefu] HIVE-6022: Load statements with incorrect order of partitions put input files to unreadable places (Teruyoshi Zenmyo via Xuefu) Changes for Build #603 Changes for Build #604 [thejas] HIVE-5975 : [WebHCat] templeton mapreduce job failed if provide define parameters (Shanyu Zhao via Thejas Nair) Changes for Build #605 [prasadm] HIVE-1466: Add NULL DEFINED AS to ROW FORMAT specification (Prasad Mujumdar reviewed by Xuefu Zhang) Changes for Build #606 [jitendra] HIVE-5521 : Remove CommonRCFileInputFormat. (hashutosh via jitendra) [rhbutani] HIVE-5973 SMB joins produce incorrect results with multiple partitions and buckets (Vikram Dixit via Harish Butani) [ehans] HIVE-6015: vectorized logarithm produces results for 0 that are different from a non-vectorized one (Sergey Shelukhin via Eric Hanson) Changes for Build #607 [brock] HIVE-5812 - HiveServer2 SSL connection transport binds to loopback address by default (Prasad Mujumdar via Brock Noland) Changes for Build #608 [hashutosh] HIVE-5936 : analyze command failing to collect stats with counter mechanism (Navis via Ashutosh Chauhan) Changes for Build #609 [thejas] HIVE-5230 : Better error reporting by async threads in HiveServer2 (Vaibhav Gumashta via Thejas Nair) Changes for Build #610 [navis] HIVE-5879 : Fix spelling errors in hive-default.xml.template (Lefty Leverenz via Navis) Changes for Build #611 Changes for Build #612 [xuefu] HIVE-6021: Problem in GroupByOperator for handling distinct aggrgations (Sun Rui via Xuefu) Changes for Build #613 [prasadm] HIVE-6036: A test case for embedded beeline - with URL jdbc:hive2:///default (Anandha L Ranganathan via Prasad Mujumdar) [prasadm] HIVE-4256: JDBC2 HiveConnection does not use the specified database (Anandha L Ranganathan via Prasad Mujumdar) Changes for Build #614 [brock] HIVE-5966 - Fix eclipse:eclipse post shim aggregation changes (Szehon Ho via Brock Noland) Changes for Build #615 [daijy] HIVE-5540: webhcat e2e test failures: Expect 1 jobs in logs, but get 1 [hashutosh] HIVE-6041 : Incorrect task dependency graph for skewed join optimization (Navis via Ashutosh Chauhan) Changes for Build #616 [hashutosh] HIVE-6013 : Supporting Quoted Identifiers in Column Names (Harish Butani via Ashutosh Chauhan) [hashutosh] HIVE-6052 : metastore JDO filter pushdown for integers may produce unexpected results with non-normalized integer columns (Sergey Shelukhin via Ashutosh Chauhan) Changes for Build #617 [hashutosh] HIVE-6034 : vectorized % doesn't handle zeroes the same way as non-vectorized (Sergey Shelukhin via Eric Hanson) Changes for Build #618 Changes for Build #619 [xuefu] HIVE-5992: Hive inconsistently converts timestamp in AVG and SUM UDAF's (reviewed by Prasad) [xuefu] HIVE-6045: Beeline hivevars is broken for more than one hivevar (Szehon via Xuefu) [hashutosh] HIVE-5558 :
Hive-trunk-h0.21 - Build # 2536 - Still Failing
Changes for Build #2495 [omalley] HIVE-5580. Predicate pushdown predicates with an and-operator between non-SARGable predicates cause a NPE. (omalley) Changes for Build #2496 [gunther] HIVE-6000: Hive build broken on hadoop2 (Vikram Dixit K via Gunther Hagleitner [gunther] HIVE-2093: UPDATE - add two missing files from previous commit (Gunther Hagleitner) [thejas] HIVE-2093 : create/drop database should populate inputs/outputs and check concurrency and user permission (Navis via Thejas Nair) [hashutosh] HIVE-6016 : Hadoop23Shims has a bug in listLocatedStatus impl. (Prasanth J via Ashutosh Chauhan) [hashutosh] HIVE-5994 : ORC RLEv2 encodes wrongly for large negative BIGINTs (64 bits ) (Prasanth J via Owen Omalley) [hashutosh] HIVE-5991 : ORC RLEv2 fails with ArrayIndexOutOfBounds exception for PATCHED_BLOB encoding (Prasanth J via Owen Omalley) [prasadm] HIVE-4395: Support TFetchOrientation.FIRST for HiveServer2 FetchResults (Prasad Mujumdar reviewed by Thejas Nair) [ehans] HIVE-5756: Implement vectorized support for IF conditional expression (Eric Hanson) [hashutosh] HIVE-6018 : FetchTask should not reference metastore classes (Navis via Prasad Mujumdar) [hashutosh] HIVE-5979. Failure in cast to timestamps. (Jitendra Pandey) [hashutosh] HIVE-5897 : Fix hadoop2 execution environment Milestone 2 (Vikram Dixit via Brock Noland) Changes for Build #2497 Changes for Build #2498 [hashutosh] HIVE-6004 : Fix statistics annotation related test failures in hadoop2 (Prasanth J via Ashutosh Chauhan) [hashutosh] HIVE-6027 : non-vectorized log10 has rounding issue (Sergey Shelukhin via Ashutosh Chauhan) [prasadm] HIVE-5993: JDBC Driver should not hard-code the database name (Szehon Ho via Prasad Mujumdar) Changes for Build #2499 [navis] HIVE-5985 : Make qfile_regex to accept multiple patterns (Navis reviewed by Ashutosh Chauhan) Changes for Build #2500 Changes for Build #2501 Changes for Build #2502 [navis] HIVE-5276 : Skip redundant string encoding/decoding for hiveserver2 (Navis Reviewed by Carl Steinbach) Changes for Build #2503 [xuefu] HIVE-6022: Load statements with incorrect order of partitions put input files to unreadable places (Teruyoshi Zenmyo via Xuefu) Changes for Build #2504 Changes for Build #2505 [thejas] HIVE-5975 : [WebHCat] templeton mapreduce job failed if provide define parameters (Shanyu Zhao via Thejas Nair) Changes for Build #2506 [prasadm] HIVE-1466: Add NULL DEFINED AS to ROW FORMAT specification (Prasad Mujumdar reviewed by Xuefu Zhang) Changes for Build #2507 [jitendra] HIVE-5521 : Remove CommonRCFileInputFormat. (hashutosh via jitendra) [rhbutani] HIVE-5973 SMB joins produce incorrect results with multiple partitions and buckets (Vikram Dixit via Harish Butani) [ehans] HIVE-6015: vectorized logarithm produces results for 0 that are different from a non-vectorized one (Sergey Shelukhin via Eric Hanson) Changes for Build #2508 [brock] HIVE-5812 - HiveServer2 SSL connection transport binds to loopback address by default (Prasad Mujumdar via Brock Noland) Changes for Build #2509 [hashutosh] HIVE-5936 : analyze command failing to collect stats with counter mechanism (Navis via Ashutosh Chauhan) Changes for Build #2510 [thejas] HIVE-5230 : Better error reporting by async threads in HiveServer2 (Vaibhav Gumashta via Thejas Nair) Changes for Build #2511 [navis] HIVE-5879 : Fix spelling errors in hive-default.xml.template (Lefty Leverenz via Navis) Changes for Build #2512 Changes for Build #2513 [xuefu] HIVE-6021: Problem in GroupByOperator for handling distinct aggrgations (Sun Rui via Xuefu) Changes for Build #2514 [prasadm] HIVE-6036: A test case for embedded beeline - with URL jdbc:hive2:///default (Anandha L Ranganathan via Prasad Mujumdar) [prasadm] HIVE-4256: JDBC2 HiveConnection does not use the specified database (Anandha L Ranganathan via Prasad Mujumdar) Changes for Build #2515 [brock] HIVE-5966 - Fix eclipse:eclipse post shim aggregation changes (Szehon Ho via Brock Noland) Changes for Build #2516 [daijy] HIVE-5540: webhcat e2e test failures: Expect 1 jobs in logs, but get 1 [hashutosh] HIVE-6041 : Incorrect task dependency graph for skewed join optimization (Navis via Ashutosh Chauhan) Changes for Build #2517 [hashutosh] HIVE-6013 : Supporting Quoted Identifiers in Column Names (Harish Butani via Ashutosh Chauhan) [hashutosh] HIVE-6052 : metastore JDO filter pushdown for integers may produce unexpected results with non-normalized integer columns (Sergey Shelukhin via Ashutosh Chauhan) Changes for Build #2518 [hashutosh] HIVE-6034 : vectorized % doesn't handle zeroes the same way as non-vectorized (Sergey Shelukhin via Eric Hanson) Changes for Build #2519 [xuefu] HIVE-5992: Hive inconsistently converts timestamp in AVG and SUM UDAF's (reviewed by Prasad) [xuefu] HIVE-6045: Beeline hivevars is broken for more than one hivevar (Szehon via Xuefu) [hashutosh] HIVE-5558 : Support alternate join
[jira] [Commented] (HIVE-5414) The result of show grant is not visible via JDBC
[ https://issues.apache.org/jira/browse/HIVE-5414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859590#comment-13859590 ] Thejas M Nair commented on HIVE-5414: - +1 The result of show grant is not visible via JDBC Key: HIVE-5414 URL: https://issues.apache.org/jira/browse/HIVE-5414 Project: Hive Issue Type: Bug Components: Authorization, JDBC Reporter: Navis Assignee: Navis Priority: Trivial Attachments: D13209.1.patch, D13209.2.patch, D13209.3.patch, D13209.4.patch, HIVE-5414.4.patch.txt, HIVE-5414.5.patch.txt Currently, show grant / show role grant does not make fetch task, which provides the result schema for jdbc clients. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5414) The result of show grant is not visible via JDBC
[ https://issues.apache.org/jira/browse/HIVE-5414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859592#comment-13859592 ] Phabricator commented on HIVE-5414: --- thejas has commented on the revision HIVE-5414 [jira] The result of show grant is not visible via JDBC. +1 REVISION DETAIL https://reviews.facebook.net/D13209 To: JIRA, navis Cc: thejas The result of show grant is not visible via JDBC Key: HIVE-5414 URL: https://issues.apache.org/jira/browse/HIVE-5414 Project: Hive Issue Type: Bug Components: Authorization, JDBC Reporter: Navis Assignee: Navis Priority: Trivial Attachments: D13209.1.patch, D13209.2.patch, D13209.3.patch, D13209.4.patch, HIVE-5414.4.patch.txt, HIVE-5414.5.patch.txt Currently, show grant / show role grant does not make fetch task, which provides the result schema for jdbc clients. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6115) Remove redundant code in HiveHBaseStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-6115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859606#comment-13859606 ] Brock Noland commented on HIVE-6115: Ashutosh, any idea when Sushanth will be around to weigh-in? Remove redundant code in HiveHBaseStorageHandler Key: HIVE-6115 URL: https://issues.apache.org/jira/browse/HIVE-6115 Project: Hive Issue Type: Improvement Affects Versions: 0.12.0 Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-6115.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-5794) HiveServer2 should retry startup
[ https://issues.apache.org/jira/browse/HIVE-5794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-5794: --- Attachment: HIVE-5794.patch Rebasing the patch on trunk and making it retry only for 30 minutes. HiveServer2 should retry startup Key: HIVE-5794 URL: https://issues.apache.org/jira/browse/HIVE-5794 Project: Hive Issue Type: Improvement Affects Versions: 0.13.0 Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-5794.patch, HIVE-5794.patch, HIVE-5794.patch On startup the CLIService attempts to create scratch directories. If that fails, e.g. because the nn is also starting, HS2 fails to start. We should retry this operation this a few times with sleeps. It might be preferable todo this directly in HS2 main so that we can catch all errors. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-5380) Non-default OI constructors should be supported for backwards compatibility
[ https://issues.apache.org/jira/browse/HIVE-5380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-5380: --- Attachment: HIVE-5380.patch Rebased on trunk. Non-default OI constructors should be supported for backwards compatibility --- Key: HIVE-5380 URL: https://issues.apache.org/jira/browse/HIVE-5380 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Brock Noland Assignee: Brock Noland Priority: Minor Attachments: HIVE-5380.patch, HIVE-5380.patch, HIVE-5380.patch In HIVE-5263 we started serializing OI's when cloning the plan. This was a great boost in speed for many queries. In the future we'd like to stop copying the OI's, perhaps in HIVE-4396. Until then Custom Serdes will not work on trunk. This is a fix to allow custom serdes such as the Hive JSon Serde work until we address the fact we don't want to have to copy the OI's. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (HIVE-5776) PTest2 doesn't honor max text limit when posint to jira
[ https://issues.apache.org/jira/browse/HIVE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland resolved HIVE-5776. Resolution: Duplicate I fixed this elsewhere. PTest2 doesn't honor max text limit when posint to jira --- Key: HIVE-5776 URL: https://issues.apache.org/jira/browse/HIVE-5776 Project: Hive Issue Type: Bug Reporter: Brock Noland Assignee: Brock Noland It looks like the output of source prep is just one big string -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-4851) Fix flaky tests
[ https://issues.apache.org/jira/browse/HIVE-4851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859626#comment-13859626 ] Brock Noland commented on HIVE-4851: It seems these days most of the flaky tests have been taking care of? Fix flaky tests --- Key: HIVE-4851 URL: https://issues.apache.org/jira/browse/HIVE-4851 Project: Hive Issue Type: Bug Reporter: Brock Noland Assignee: Brock Noland I see the following tests fail quite often: * TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_hadoop20 * TestOrcHCatLoader.testReadDataBasic * TestMinimrCliDriver.testCliDriver_bucketmpjoin6 * TestNotificationListener.testAMQListener This one is less often, but still fails randomly: * TestMinimrCliDriver.testCliDriver_bucket4 * TestHCatHiveCompatibility.testUnpartedReadWrite * TestHCatLoader.testReadPartitionedBasic * TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat * TestOrcDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6115) Remove redundant code in HiveHBaseStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-6115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859629#comment-13859629 ] Ashutosh Chauhan commented on HIVE-6115: I think he should be around later this week at latest. [~viraj] may also have an idea. Remove redundant code in HiveHBaseStorageHandler Key: HIVE-6115 URL: https://issues.apache.org/jira/browse/HIVE-6115 Project: Hive Issue Type: Improvement Affects Versions: 0.12.0 Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-6115.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Comment Edited] (HIVE-6115) Remove redundant code in HiveHBaseStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-6115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859629#comment-13859629 ] Ashutosh Chauhan edited comment on HIVE-6115 at 12/31/13 6:54 PM: -- I think he should be around later this week at latest. [~viraj] may also have an idea for this piece of code. was (Author: ashutoshc): I think he should be around later this week at latest. [~viraj] may also have an idea. Remove redundant code in HiveHBaseStorageHandler Key: HIVE-6115 URL: https://issues.apache.org/jira/browse/HIVE-6115 Project: Hive Issue Type: Improvement Affects Versions: 0.12.0 Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-6115.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-5757) Implement vectorized support for CASE
[ https://issues.apache.org/jira/browse/HIVE-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Hanson updated HIVE-5757: -- Attachment: HIVE-5757.2.patch re-based, and converted to Unix format Implement vectorized support for CASE - Key: HIVE-5757 URL: https://issues.apache.org/jira/browse/HIVE-5757 Project: Hive Issue Type: Sub-task Reporter: Eric Hanson Assignee: Eric Hanson Attachments: HIVE-5757.1.patch, HIVE-5757.2.patch Implement full, end-to-end support for CASE in vectorized mode, including new VectorExpression class(es), VectorizationContext translation to a VectorExpression, and unit tests for these, as well as end-to-end ad hoc testing. An end-to-end .q test is recommended but optional. This is high priority because CASE is a fairly popular expression. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-5757) Implement vectorized support for CASE
[ https://issues.apache.org/jira/browse/HIVE-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Hanson updated HIVE-5757: -- Attachment: HIVE-5757.3.patch Added .q.out file to patch Implement vectorized support for CASE - Key: HIVE-5757 URL: https://issues.apache.org/jira/browse/HIVE-5757 Project: Hive Issue Type: Sub-task Reporter: Eric Hanson Assignee: Eric Hanson Attachments: HIVE-5757.1.patch, HIVE-5757.2.patch, HIVE-5757.3.patch Implement full, end-to-end support for CASE in vectorized mode, including new VectorExpression class(es), VectorizationContext translation to a VectorExpression, and unit tests for these, as well as end-to-end ad hoc testing. An end-to-end .q test is recommended but optional. This is high priority because CASE is a fairly popular expression. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-5757) Implement vectorized support for CASE
[ https://issues.apache.org/jira/browse/HIVE-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Hanson updated HIVE-5757: -- Affects Version/s: 0.13.0 Status: Patch Available (was: In Progress) Implement vectorized support for CASE - Key: HIVE-5757 URL: https://issues.apache.org/jira/browse/HIVE-5757 Project: Hive Issue Type: Sub-task Affects Versions: 0.13.0 Reporter: Eric Hanson Assignee: Eric Hanson Attachments: HIVE-5757.1.patch, HIVE-5757.2.patch, HIVE-5757.3.patch Implement full, end-to-end support for CASE in vectorized mode, including new VectorExpression class(es), VectorizationContext translation to a VectorExpression, and unit tests for these, as well as end-to-end ad hoc testing. An end-to-end .q test is recommended but optional. This is high priority because CASE is a fairly popular expression. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6121) Use Paths Consistently - IV
[
https://issues.apache.org/jira/browse/HIVE-6121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859640#comment-13859640
]
Hive QA commented on HIVE-6121:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12620949/HIVE-6121.3.patch
{color:green}SUCCESS:{color} +1 4818 tests passed
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/776/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/776/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12620949
Use Paths Consistently - IV
---
Key: HIVE-6121
URL: https://issues.apache.org/jira/browse/HIVE-6121
Project: Hive
Issue Type: Improvement
Components: Query Processor
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
Attachments: HIVE-6121.2.patch, HIVE-6121.3.patch, HIVE-6121.patch
Next one in patch series to fix Hive to use paths consistently.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-5794) HiveServer2 should retry startup
[
https://issues.apache.org/jira/browse/HIVE-5794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859695#comment-13859695
]
Hive QA commented on HIVE-5794:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12620964/HIVE-5794.patch
{color:green}SUCCESS:{color} +1 4818 tests passed
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/777/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/777/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12620964
HiveServer2 should retry startup
Key: HIVE-5794
URL: https://issues.apache.org/jira/browse/HIVE-5794
Project: Hive
Issue Type: Improvement
Affects Versions: 0.13.0
Reporter: Brock Noland
Assignee: Brock Noland
Attachments: HIVE-5794.patch, HIVE-5794.patch, HIVE-5794.patch
On startup the CLIService attempts to create scratch directories. If that
fails, e.g. because the nn is also starting, HS2 fails to start. We should
retry this operation this a few times with sleeps.
It might be preferable todo this directly in HS2 main so that we can catch
all errors.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-5757) Implement vectorized support for CASE
[ https://issues.apache.org/jira/browse/HIVE-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859721#comment-13859721 ] Eric Hanson commented on HIVE-5757: --- Review at https://reviews.apache.org/r/16549/ Implement vectorized support for CASE - Key: HIVE-5757 URL: https://issues.apache.org/jira/browse/HIVE-5757 Project: Hive Issue Type: Sub-task Affects Versions: 0.13.0 Reporter: Eric Hanson Assignee: Eric Hanson Attachments: HIVE-5757.1.patch, HIVE-5757.2.patch, HIVE-5757.3.patch Implement full, end-to-end support for CASE in vectorized mode, including new VectorExpression class(es), VectorizationContext translation to a VectorExpression, and unit tests for these, as well as end-to-end ad hoc testing. An end-to-end .q test is recommended but optional. This is high priority because CASE is a fairly popular expression. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5794) HiveServer2 should retry startup
[ https://issues.apache.org/jira/browse/HIVE-5794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859726#comment-13859726 ] Prasad Mujumdar commented on HIVE-5794: --- +1 Looks fine to me. HiveServer2 should retry startup Key: HIVE-5794 URL: https://issues.apache.org/jira/browse/HIVE-5794 Project: Hive Issue Type: Improvement Affects Versions: 0.13.0 Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-5794.patch, HIVE-5794.patch, HIVE-5794.patch On startup the CLIService attempts to create scratch directories. If that fails, e.g. because the nn is also starting, HS2 fails to start. We should retry this operation this a few times with sleeps. It might be preferable todo this directly in HS2 main so that we can catch all errors. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-5794) HiveServer2 should retry startup
[ https://issues.apache.org/jira/browse/HIVE-5794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-5794: --- Resolution: Fixed Fix Version/s: 0.13.0 Status: Resolved (was: Patch Available) Thanks! Since Thejas had +1'ed the earlier patch and considering it's a holiday I committed the patch. Thank you! HiveServer2 should retry startup Key: HIVE-5794 URL: https://issues.apache.org/jira/browse/HIVE-5794 Project: Hive Issue Type: Improvement Affects Versions: 0.13.0 Reporter: Brock Noland Assignee: Brock Noland Fix For: 0.13.0 Attachments: HIVE-5794.patch, HIVE-5794.patch, HIVE-5794.patch On startup the CLIService attempts to create scratch directories. If that fails, e.g. because the nn is also starting, HS2 fails to start. We should retry this operation this a few times with sleeps. It might be preferable todo this directly in HS2 main so that we can catch all errors. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5155) Support secure proxy user access to HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859731#comment-13859731 ] Shivaraju Gowda commented on HIVE-5155: --- I am trying to accomplish the same task, i.e end user wants to access Hive via middleware server and the middleware server doesn't have access to the credentials. I looked at it from a different angle though using Subject.doAs and could get it working with a little hack on the client side(no changes to hive binaries). I Wanted to check if there are any issues with this approach. Please see the attached file TestKERBEROS_Hive_JDBC.java. Obviously, the hack is not a good way to go, longer term it needs to be addressed in the Hive/Hadoop code, but wanted to point what is missing in the Hive code to accomplish support for multi-user kerberos support through middleware server. Support secure proxy user access to HiveServer2 --- Key: HIVE-5155 URL: https://issues.apache.org/jira/browse/HIVE-5155 Project: Hive Issue Type: Improvement Components: Authentication, HiveServer2, JDBC Affects Versions: 0.12.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Attachments: HIVE-5155-1-nothrift.patch, HIVE-5155-noThrift.2.patch, HIVE-5155-noThrift.4.patch, HIVE-5155-noThrift.5.patch, HIVE-5155-noThrift.6.patch, HIVE-5155.1.patch, HIVE-5155.2.patch, HIVE-5155.3.patch, ProxyAuth.java, ProxyAuth.out The HiveServer2 can authenticate a client using via Kerberos and impersonate the connecting user with underlying secure hadoop. This becomes a gateway for a remote client to access secure hadoop cluster. Now this works fine for when the client obtains Kerberos ticket and directly connects to HiveServer2. There's another big use case for middleware tools where the end user wants to access Hive via another server. For example Oozie action or Hue submitting queries or a BI tool server accessing to HiveServer2. In these cases, the third party server doesn't have end user's Kerberos credentials and hence it can't submit queries to HiveServer2 on behalf of the end user. This ticket is for enabling proxy access to HiveServer2 for third party tools on behalf of end users. There are two parts of the solution proposed in this ticket: 1) Delegation token based connection for Oozie (OOZIE-1457) This is the common mechanism for Hadoop ecosystem components. Hive Remote Metastore and HCatalog already support this. This is suitable for tool like Oozie that submits the MR jobs as actions on behalf of its client. Oozie already uses similar mechanism for Metastore/HCatalog access. 2) Direct proxy access for privileged hadoop users The delegation token implementation can be a challenge for non-hadoop (especially non-java) components. This second part enables a privileged user to directly specify an alternate session user during the connection. If the connecting user has hadoop level privilege to impersonate the requested userid, then HiveServer2 will run the session as that requested user. For example, user Hue is allowed to impersonate user Bob (via core-site.xml proxy user configuration). Then user Hue can connect to HiveServer2 and specify Bob as session user via a session property. HiveServer2 will verify Hue's proxy user privilege and then impersonate user Bob instead of Hue. This will enable any third party tool to impersonate alternate userid without having to implement delegation token connection. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-5155) Support secure proxy user access to HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaraju Gowda updated HIVE-5155: -- Attachment: TestKERBEROS_Hive_JDBC.java Test Script for Subject.doAs(). Support secure proxy user access to HiveServer2 --- Key: HIVE-5155 URL: https://issues.apache.org/jira/browse/HIVE-5155 Project: Hive Issue Type: Improvement Components: Authentication, HiveServer2, JDBC Affects Versions: 0.12.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Attachments: HIVE-5155-1-nothrift.patch, HIVE-5155-noThrift.2.patch, HIVE-5155-noThrift.4.patch, HIVE-5155-noThrift.5.patch, HIVE-5155-noThrift.6.patch, HIVE-5155.1.patch, HIVE-5155.2.patch, HIVE-5155.3.patch, ProxyAuth.java, ProxyAuth.out, TestKERBEROS_Hive_JDBC.java The HiveServer2 can authenticate a client using via Kerberos and impersonate the connecting user with underlying secure hadoop. This becomes a gateway for a remote client to access secure hadoop cluster. Now this works fine for when the client obtains Kerberos ticket and directly connects to HiveServer2. There's another big use case for middleware tools where the end user wants to access Hive via another server. For example Oozie action or Hue submitting queries or a BI tool server accessing to HiveServer2. In these cases, the third party server doesn't have end user's Kerberos credentials and hence it can't submit queries to HiveServer2 on behalf of the end user. This ticket is for enabling proxy access to HiveServer2 for third party tools on behalf of end users. There are two parts of the solution proposed in this ticket: 1) Delegation token based connection for Oozie (OOZIE-1457) This is the common mechanism for Hadoop ecosystem components. Hive Remote Metastore and HCatalog already support this. This is suitable for tool like Oozie that submits the MR jobs as actions on behalf of its client. Oozie already uses similar mechanism for Metastore/HCatalog access. 2) Direct proxy access for privileged hadoop users The delegation token implementation can be a challenge for non-hadoop (especially non-java) components. This second part enables a privileged user to directly specify an alternate session user during the connection. If the connecting user has hadoop level privilege to impersonate the requested userid, then HiveServer2 will run the session as that requested user. For example, user Hue is allowed to impersonate user Bob (via core-site.xml proxy user configuration). Then user Hue can connect to HiveServer2 and specify Bob as session user via a session property. HiveServer2 will verify Hue's proxy user privilege and then impersonate user Bob instead of Hue. This will enable any third party tool to impersonate alternate userid without having to implement delegation token connection. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6121) Use Paths Consistently - IV
[ https://issues.apache.org/jira/browse/HIVE-6121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6121: --- Resolution: Fixed Fix Version/s: 0.13.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks, Xuefu for review! Use Paths Consistently - IV --- Key: HIVE-6121 URL: https://issues.apache.org/jira/browse/HIVE-6121 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.13.0 Attachments: HIVE-6121.2.patch, HIVE-6121.3.patch, HIVE-6121.patch Next one in patch series to fix Hive to use paths consistently. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5380) Non-default OI constructors should be supported for backwards compatibility
[
https://issues.apache.org/jira/browse/HIVE-5380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859745#comment-13859745
]
Hive QA commented on HIVE-5380:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12620965/HIVE-5380.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 4818 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/778/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/778/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12620965
Non-default OI constructors should be supported for backwards compatibility
---
Key: HIVE-5380
URL: https://issues.apache.org/jira/browse/HIVE-5380
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0
Reporter: Brock Noland
Assignee: Brock Noland
Priority: Minor
Attachments: HIVE-5380.patch, HIVE-5380.patch, HIVE-5380.patch
In HIVE-5263 we started serializing OI's when cloning the plan. This was a
great boost in speed for many queries. In the future we'd like to stop
copying the OI's, perhaps in HIVE-4396.
Until then Custom Serdes will not work on trunk. This is a fix to allow
custom serdes such as the Hive JSon Serde work until we address the fact we
don't want to have to copy the OI's.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HIVE-5795) Hive should be able to skip header and footer rows when reading data file for a table
[
https://issues.apache.org/jira/browse/HIVE-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shuaishuai Nie updated HIVE-5795:
-
Attachment: HIVE-5795.5.patch
Hive should be able to skip header and footer rows when reading data file for
a table
-
Key: HIVE-5795
URL: https://issues.apache.org/jira/browse/HIVE-5795
Project: Hive
Issue Type: Bug
Reporter: Shuaishuai Nie
Assignee: Shuaishuai Nie
Attachments: HIVE-5795.1.patch, HIVE-5795.2.patch, HIVE-5795.3.patch,
HIVE-5795.4.patch, HIVE-5795.5.patch
Hive should be able to skip header and footer lines when reading data file
from table. In this way, user don't need to processing data which generated
by other application with a header or footer and directly use the file for
table operations.
To implement this, the idea is adding new properties in table descriptions to
define the number of lines in header and footer and skip them when reading
the record from record reader. An DDL example for creating a table with
header and footer should be like this:
{code}
Create external table testtable (name string, message string) row format
delimited fields terminated by '\t' lines terminated by '\n' location
'/testtable' tblproperties (skip.header.line.count=1,
skip.footer.line.count=2);
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-3746) Fix HS2 ResultSet Serialization Performance Regression
[ https://issues.apache.org/jira/browse/HIVE-3746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859755#comment-13859755 ] Carl Steinbach commented on HIVE-3746: -- +1 I'm fine with committing the patch in its current state, but there's one thing I think we definitely need to fix ASAP in a followup patch. Up to this point we have managed to avoid polluting the client and service class interfaces ( i.e. CLIService and CLIServiceClient) with direct references to the Thrift serialization layer. This patch breaks that rule by exposing TProtocolVersion in the public methods of CliService. Only ThriftCLIService should need to know that the client is using a specific version of the Thrift serialization layer. Fix HS2 ResultSet Serialization Performance Regression -- Key: HIVE-3746 URL: https://issues.apache.org/jira/browse/HIVE-3746 Project: Hive Issue Type: Sub-task Components: HiveServer2, Server Infrastructure Reporter: Carl Steinbach Assignee: Navis Labels: HiveServer2, jdbc, thrift Attachments: HIVE-3746.1.patch.txt, HIVE-3746.2.patch.txt, HIVE-3746.3.patch.txt, HIVE-3746.4.patch.txt, HIVE-3746.5.patch.txt, HIVE-3746.6.patch.txt, HIVE-3746.7.patch.txt -- This message was sent by Atlassian JIRA (v6.1.5#6160)
Hive-trunk-hadoop2 - Build # 638 - Still Failing
Changes for Build #593 [omalley] HIVE-5580. Predicate pushdown predicates with an and-operator between non-SARGable predicates cause a NPE. (omalley) Changes for Build #594 [gunther] HIVE-6000: Hive build broken on hadoop2 (Vikram Dixit K via Gunther Hagleitner [gunther] HIVE-2093: UPDATE - add two missing files from previous commit (Gunther Hagleitner) [thejas] HIVE-2093 : create/drop database should populate inputs/outputs and check concurrency and user permission (Navis via Thejas Nair) [hashutosh] HIVE-6016 : Hadoop23Shims has a bug in listLocatedStatus impl. (Prasanth J via Ashutosh Chauhan) [hashutosh] HIVE-5994 : ORC RLEv2 encodes wrongly for large negative BIGINTs (64 bits ) (Prasanth J via Owen Omalley) [hashutosh] HIVE-5991 : ORC RLEv2 fails with ArrayIndexOutOfBounds exception for PATCHED_BLOB encoding (Prasanth J via Owen Omalley) [prasadm] HIVE-4395: Support TFetchOrientation.FIRST for HiveServer2 FetchResults (Prasad Mujumdar reviewed by Thejas Nair) [ehans] HIVE-5756: Implement vectorized support for IF conditional expression (Eric Hanson) [hashutosh] HIVE-6018 : FetchTask should not reference metastore classes (Navis via Prasad Mujumdar) [hashutosh] HIVE-5979. Failure in cast to timestamps. (Jitendra Pandey) [hashutosh] HIVE-5897 : Fix hadoop2 execution environment Milestone 2 (Vikram Dixit via Brock Noland) Changes for Build #595 Changes for Build #596 [hashutosh] HIVE-6027 : non-vectorized log10 has rounding issue (Sergey Shelukhin via Ashutosh Chauhan) [prasadm] HIVE-5993: JDBC Driver should not hard-code the database name (Szehon Ho via Prasad Mujumdar) Changes for Build #597 [hashutosh] HIVE-6004 : Fix statistics annotation related test failures in hadoop2 (Prasanth J via Ashutosh Chauhan) Changes for Build #598 [navis] HIVE-5985 : Make qfile_regex to accept multiple patterns (Navis reviewed by Ashutosh Chauhan) Changes for Build #599 Changes for Build #600 Changes for Build #601 [navis] HIVE-5276 : Skip redundant string encoding/decoding for hiveserver2 (Navis Reviewed by Carl Steinbach) Changes for Build #602 [xuefu] HIVE-6022: Load statements with incorrect order of partitions put input files to unreadable places (Teruyoshi Zenmyo via Xuefu) Changes for Build #603 Changes for Build #604 [thejas] HIVE-5975 : [WebHCat] templeton mapreduce job failed if provide define parameters (Shanyu Zhao via Thejas Nair) Changes for Build #605 [prasadm] HIVE-1466: Add NULL DEFINED AS to ROW FORMAT specification (Prasad Mujumdar reviewed by Xuefu Zhang) Changes for Build #606 [jitendra] HIVE-5521 : Remove CommonRCFileInputFormat. (hashutosh via jitendra) [rhbutani] HIVE-5973 SMB joins produce incorrect results with multiple partitions and buckets (Vikram Dixit via Harish Butani) [ehans] HIVE-6015: vectorized logarithm produces results for 0 that are different from a non-vectorized one (Sergey Shelukhin via Eric Hanson) Changes for Build #607 [brock] HIVE-5812 - HiveServer2 SSL connection transport binds to loopback address by default (Prasad Mujumdar via Brock Noland) Changes for Build #608 [hashutosh] HIVE-5936 : analyze command failing to collect stats with counter mechanism (Navis via Ashutosh Chauhan) Changes for Build #609 [thejas] HIVE-5230 : Better error reporting by async threads in HiveServer2 (Vaibhav Gumashta via Thejas Nair) Changes for Build #610 [navis] HIVE-5879 : Fix spelling errors in hive-default.xml.template (Lefty Leverenz via Navis) Changes for Build #611 Changes for Build #612 [xuefu] HIVE-6021: Problem in GroupByOperator for handling distinct aggrgations (Sun Rui via Xuefu) Changes for Build #613 [prasadm] HIVE-6036: A test case for embedded beeline - with URL jdbc:hive2:///default (Anandha L Ranganathan via Prasad Mujumdar) [prasadm] HIVE-4256: JDBC2 HiveConnection does not use the specified database (Anandha L Ranganathan via Prasad Mujumdar) Changes for Build #614 [brock] HIVE-5966 - Fix eclipse:eclipse post shim aggregation changes (Szehon Ho via Brock Noland) Changes for Build #615 [daijy] HIVE-5540: webhcat e2e test failures: Expect 1 jobs in logs, but get 1 [hashutosh] HIVE-6041 : Incorrect task dependency graph for skewed join optimization (Navis via Ashutosh Chauhan) Changes for Build #616 [hashutosh] HIVE-6013 : Supporting Quoted Identifiers in Column Names (Harish Butani via Ashutosh Chauhan) [hashutosh] HIVE-6052 : metastore JDO filter pushdown for integers may produce unexpected results with non-normalized integer columns (Sergey Shelukhin via Ashutosh Chauhan) Changes for Build #617 [hashutosh] HIVE-6034 : vectorized % doesn't handle zeroes the same way as non-vectorized (Sergey Shelukhin via Eric Hanson) Changes for Build #618 Changes for Build #619 [xuefu] HIVE-5992: Hive inconsistently converts timestamp in AVG and SUM UDAF's (reviewed by Prasad) [xuefu] HIVE-6045: Beeline hivevars is broken for more than one hivevar (Szehon via Xuefu) [hashutosh] HIVE-5558 :
Hive-trunk-h0.21 - Build # 2537 - Still Failing
Changes for Build #2495 [omalley] HIVE-5580. Predicate pushdown predicates with an and-operator between non-SARGable predicates cause a NPE. (omalley) Changes for Build #2496 [gunther] HIVE-6000: Hive build broken on hadoop2 (Vikram Dixit K via Gunther Hagleitner [gunther] HIVE-2093: UPDATE - add two missing files from previous commit (Gunther Hagleitner) [thejas] HIVE-2093 : create/drop database should populate inputs/outputs and check concurrency and user permission (Navis via Thejas Nair) [hashutosh] HIVE-6016 : Hadoop23Shims has a bug in listLocatedStatus impl. (Prasanth J via Ashutosh Chauhan) [hashutosh] HIVE-5994 : ORC RLEv2 encodes wrongly for large negative BIGINTs (64 bits ) (Prasanth J via Owen Omalley) [hashutosh] HIVE-5991 : ORC RLEv2 fails with ArrayIndexOutOfBounds exception for PATCHED_BLOB encoding (Prasanth J via Owen Omalley) [prasadm] HIVE-4395: Support TFetchOrientation.FIRST for HiveServer2 FetchResults (Prasad Mujumdar reviewed by Thejas Nair) [ehans] HIVE-5756: Implement vectorized support for IF conditional expression (Eric Hanson) [hashutosh] HIVE-6018 : FetchTask should not reference metastore classes (Navis via Prasad Mujumdar) [hashutosh] HIVE-5979. Failure in cast to timestamps. (Jitendra Pandey) [hashutosh] HIVE-5897 : Fix hadoop2 execution environment Milestone 2 (Vikram Dixit via Brock Noland) Changes for Build #2497 Changes for Build #2498 [hashutosh] HIVE-6004 : Fix statistics annotation related test failures in hadoop2 (Prasanth J via Ashutosh Chauhan) [hashutosh] HIVE-6027 : non-vectorized log10 has rounding issue (Sergey Shelukhin via Ashutosh Chauhan) [prasadm] HIVE-5993: JDBC Driver should not hard-code the database name (Szehon Ho via Prasad Mujumdar) Changes for Build #2499 [navis] HIVE-5985 : Make qfile_regex to accept multiple patterns (Navis reviewed by Ashutosh Chauhan) Changes for Build #2500 Changes for Build #2501 Changes for Build #2502 [navis] HIVE-5276 : Skip redundant string encoding/decoding for hiveserver2 (Navis Reviewed by Carl Steinbach) Changes for Build #2503 [xuefu] HIVE-6022: Load statements with incorrect order of partitions put input files to unreadable places (Teruyoshi Zenmyo via Xuefu) Changes for Build #2504 Changes for Build #2505 [thejas] HIVE-5975 : [WebHCat] templeton mapreduce job failed if provide define parameters (Shanyu Zhao via Thejas Nair) Changes for Build #2506 [prasadm] HIVE-1466: Add NULL DEFINED AS to ROW FORMAT specification (Prasad Mujumdar reviewed by Xuefu Zhang) Changes for Build #2507 [jitendra] HIVE-5521 : Remove CommonRCFileInputFormat. (hashutosh via jitendra) [rhbutani] HIVE-5973 SMB joins produce incorrect results with multiple partitions and buckets (Vikram Dixit via Harish Butani) [ehans] HIVE-6015: vectorized logarithm produces results for 0 that are different from a non-vectorized one (Sergey Shelukhin via Eric Hanson) Changes for Build #2508 [brock] HIVE-5812 - HiveServer2 SSL connection transport binds to loopback address by default (Prasad Mujumdar via Brock Noland) Changes for Build #2509 [hashutosh] HIVE-5936 : analyze command failing to collect stats with counter mechanism (Navis via Ashutosh Chauhan) Changes for Build #2510 [thejas] HIVE-5230 : Better error reporting by async threads in HiveServer2 (Vaibhav Gumashta via Thejas Nair) Changes for Build #2511 [navis] HIVE-5879 : Fix spelling errors in hive-default.xml.template (Lefty Leverenz via Navis) Changes for Build #2512 Changes for Build #2513 [xuefu] HIVE-6021: Problem in GroupByOperator for handling distinct aggrgations (Sun Rui via Xuefu) Changes for Build #2514 [prasadm] HIVE-6036: A test case for embedded beeline - with URL jdbc:hive2:///default (Anandha L Ranganathan via Prasad Mujumdar) [prasadm] HIVE-4256: JDBC2 HiveConnection does not use the specified database (Anandha L Ranganathan via Prasad Mujumdar) Changes for Build #2515 [brock] HIVE-5966 - Fix eclipse:eclipse post shim aggregation changes (Szehon Ho via Brock Noland) Changes for Build #2516 [daijy] HIVE-5540: webhcat e2e test failures: Expect 1 jobs in logs, but get 1 [hashutosh] HIVE-6041 : Incorrect task dependency graph for skewed join optimization (Navis via Ashutosh Chauhan) Changes for Build #2517 [hashutosh] HIVE-6013 : Supporting Quoted Identifiers in Column Names (Harish Butani via Ashutosh Chauhan) [hashutosh] HIVE-6052 : metastore JDO filter pushdown for integers may produce unexpected results with non-normalized integer columns (Sergey Shelukhin via Ashutosh Chauhan) Changes for Build #2518 [hashutosh] HIVE-6034 : vectorized % doesn't handle zeroes the same way as non-vectorized (Sergey Shelukhin via Eric Hanson) Changes for Build #2519 [xuefu] HIVE-5992: Hive inconsistently converts timestamp in AVG and SUM UDAF's (reviewed by Prasad) [xuefu] HIVE-6045: Beeline hivevars is broken for more than one hivevar (Szehon via Xuefu) [hashutosh] HIVE-5558 : Support alternate join
[jira] [Commented] (HIVE-5794) HiveServer2 should retry startup
[ https://issues.apache.org/jira/browse/HIVE-5794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859765#comment-13859765 ] Lefty Leverenz commented on HIVE-5794: -- The patch adds hive.server2.max.start.attempts to HiveConf.java and hive-default.xml.template, so I'll add it to Configuration Properties in the wiki ... but surprise, surprise, none of the HiveServer2 properties are in the wiki yet. This might take a little while. HiveServer2 should retry startup Key: HIVE-5794 URL: https://issues.apache.org/jira/browse/HIVE-5794 Project: Hive Issue Type: Improvement Affects Versions: 0.13.0 Reporter: Brock Noland Assignee: Brock Noland Fix For: 0.13.0 Attachments: HIVE-5794.patch, HIVE-5794.patch, HIVE-5794.patch On startup the CLIService attempts to create scratch directories. If that fails, e.g. because the nn is also starting, HS2 fails to start. We should retry this operation this a few times with sleeps. It might be preferable todo this directly in HS2 main so that we can catch all errors. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
Re: Review Request 16184: Hive should be able to skip header and footer rows when reading data file for a table (HIVE-5795)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16184/ --- (Updated Dec. 31, 2013, 11:04 p.m.) Review request for hive, Eric Hanson and Thejas Nair. Changes --- fixed unit test failures in TestCliDriver Bugs: hive-5795 https://issues.apache.org/jira/browse/hive-5795 Repository: hive-git Description --- Hive should be able to skip header and footer rows when reading data file for a table (follow up with review https://reviews.apache.org/r/15663/diff/#index_header) Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2ddb08f conf/hive-default.xml.template b94013a data/files/header_footer_table_1/0001.txt PRE-CREATION data/files/header_footer_table_1/0002.txt PRE-CREATION data/files/header_footer_table_1/0003.txt PRE-CREATION data/files/header_footer_table_2/2012/01/01/0001.txt PRE-CREATION data/files/header_footer_table_2/2012/01/02/0002.txt PRE-CREATION data/files/header_footer_table_2/2012/01/03/0003.txt PRE-CREATION itests/qtest/pom.xml 88e0890 ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java fc9b7e4 ql/src/java/org/apache/hadoop/hive/ql/exec/FooterBuffer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java daf4e4a ql/src/java/org/apache/hadoop/hive/ql/io/HiveContextAwareRecordReader.java dd5cb6b ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 974a5d6 ql/src/test/org/apache/hadoop/hive/ql/io/TestHiveBinarySearchRecordReader.java 85dd975 ql/src/test/org/apache/hadoop/hive/ql/io/TestSymlinkTextInputFormat.java 0686d9b ql/src/test/queries/clientnegative/file_with_header_footer_negative.q PRE-CREATION ql/src/test/queries/clientpositive/file_with_header_footer.q PRE-CREATION ql/src/test/results/clientnegative/file_with_header_footer_negative.q.out PRE-CREATION ql/src/test/results/clientpositive/file_with_header_footer.q.out PRE-CREATION serde/if/serde.thrift 2ceb572 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/serdeConstants.java 22a6168 Diff: https://reviews.apache.org/r/16184/diff/ Testing --- Thanks, Shuaishuai Nie
[jira] [Created] (HIVE-6124) Support basic Decimal arithmetic in vector mode (+, -, *)
Eric Hanson created HIVE-6124: - Summary: Support basic Decimal arithmetic in vector mode (+, -, *) Key: HIVE-6124 URL: https://issues.apache.org/jira/browse/HIVE-6124 Project: Hive Issue Type: Sub-task Affects Versions: 0.13.0 Reporter: Eric Hanson Create support for basic decimal arithmetic (+, -, * but not /, %) based on a single template. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Work started] (HIVE-6124) Support basic Decimal arithmetic in vector mode (+, -, *)
[ https://issues.apache.org/jira/browse/HIVE-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-6124 started by Eric Hanson. Support basic Decimal arithmetic in vector mode (+, -, *) - Key: HIVE-6124 URL: https://issues.apache.org/jira/browse/HIVE-6124 Project: Hive Issue Type: Sub-task Affects Versions: 0.13.0 Reporter: Eric Hanson Assignee: Eric Hanson Create support for basic decimal arithmetic (+, -, * but not /, %) based on a single template. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Assigned] (HIVE-6124) Support basic Decimal arithmetic in vector mode (+, -, *)
[ https://issues.apache.org/jira/browse/HIVE-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Hanson reassigned HIVE-6124: - Assignee: Eric Hanson Support basic Decimal arithmetic in vector mode (+, -, *) - Key: HIVE-6124 URL: https://issues.apache.org/jira/browse/HIVE-6124 Project: Hive Issue Type: Sub-task Affects Versions: 0.13.0 Reporter: Eric Hanson Assignee: Eric Hanson Create support for basic decimal arithmetic (+, -, * but not /, %) based on a single template. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
Re: [DISCUSS] Proposed Changes to the Apache Hive Project Bylaws
Okay, I'm convinced that one-year terms for the chair are reasonable. Thanks for the reassurance, Edward and Thejas. Is 24h rule is needed at all? In other projects, I've seen patches simply reverted by author (or someone else). It's a rare occurrence, and it should be possible to revert a patch if someone -1s it after commit, esp. within the same 24 hours when not many other changes are in. Sergey makes a good point, but the 24h rule seems helpful in prioritizing tasks. We're all deadline-driven, right? I'm the chief culprit of seeing patch available and ignoring it until it has been committed. Then if I find some minor typo or doc issue, I'm embarrassed at posting a comment after the commit because nobody wants to revert a patch just for documentation. -- Lefty On Sun, Dec 29, 2013 at 12:06 PM, Thejas Nair the...@hortonworks.comwrote: On Sun, Dec 29, 2013 at 12:06 AM, Lefty Leverenz leftylever...@gmail.com wrote: Let's discuss annual rotation of the PMC chair a bit more. Although I agree with the points made in favor, I wonder about frequent loss of expertise and needing to establish new relationships. What's the ramp-up time? The ramp up time is not significant, as you can see from the list of responsibilities mentioned here - http://www.apache.org/dev/pmc.html#chair . We have enough people in PMC who have been involved with Apache project for long time and are familiar with apache bylaws and way of doing things. Also, the former PMC chairs are likely to be around to help as needed. Could a current chair be chosen for another consecutive term? Could two chairs alternate years indefinitely? I would take the meaning of rotation to mean that we have a new chair for the next term. I think it should be OK to have same chair in alternative year. 2 years is a long time and it sounds reasonable given the size of the community ! :) Do many other projects have annual rotations? Yes, at least hadoop and pig project have that. I could not find by-laws pages easily for other projects. Would it be inconvenient to change chairs in the middle of a release? No. The PMC Chair position does not have any special role in a release. And now to trivialize my comments: while making other changes, let's fix this typo: Membership of the PMC can be revoked by an unanimous vote ... *(should be a unanimous ... just like a university because the rule is based on sound, not spelling)*. I think you should feel free to fix such a typos in this wiki without a vote on it ! :) -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Updated] (HIVE-6124) Support basic Decimal arithmetic in vector mode (+, -, *)
[ https://issues.apache.org/jira/browse/HIVE-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Hanson updated HIVE-6124: -- Description: Create support for basic decimal arithmetic (+, -, * but not /, %) based on templates for column-scalar, scalar-column, and column-column operations. (was: Create support for basic decimal arithmetic (+, -, * but not /, %) based on a single template.) Support basic Decimal arithmetic in vector mode (+, -, *) - Key: HIVE-6124 URL: https://issues.apache.org/jira/browse/HIVE-6124 Project: Hive Issue Type: Sub-task Affects Versions: 0.13.0 Reporter: Eric Hanson Assignee: Eric Hanson Create support for basic decimal arithmetic (+, -, * but not /, %) based on templates for column-scalar, scalar-column, and column-column operations. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5757) Implement vectorized support for CASE
[
https://issues.apache.org/jira/browse/HIVE-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859787#comment-13859787
]
Hive QA commented on HIVE-5757:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12620972/HIVE-5757.3.patch
{color:green}SUCCESS:{color} +1 4819 tests passed
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/779/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/779/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12620972
Implement vectorized support for CASE
-
Key: HIVE-5757
URL: https://issues.apache.org/jira/browse/HIVE-5757
Project: Hive
Issue Type: Sub-task
Affects Versions: 0.13.0
Reporter: Eric Hanson
Assignee: Eric Hanson
Attachments: HIVE-5757.1.patch, HIVE-5757.2.patch, HIVE-5757.3.patch
Implement full, end-to-end support for CASE in vectorized mode, including new
VectorExpression class(es), VectorizationContext translation to a
VectorExpression, and unit tests for these, as well as end-to-end ad hoc
testing. An end-to-end .q test is recommended but optional.
This is high priority because CASE is a fairly popular expression.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-5795) Hive should be able to skip header and footer rows when reading data file for a table
[
https://issues.apache.org/jira/browse/HIVE-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859817#comment-13859817
]
Hive QA commented on HIVE-5795:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12620989/HIVE-5795.5.patch
{color:green}SUCCESS:{color} +1 4818 tests passed
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/781/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/781/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12620989
Hive should be able to skip header and footer rows when reading data file for
a table
-
Key: HIVE-5795
URL: https://issues.apache.org/jira/browse/HIVE-5795
Project: Hive
Issue Type: Bug
Reporter: Shuaishuai Nie
Assignee: Shuaishuai Nie
Attachments: HIVE-5795.1.patch, HIVE-5795.2.patch, HIVE-5795.3.patch,
HIVE-5795.4.patch, HIVE-5795.5.patch
Hive should be able to skip header and footer lines when reading data file
from table. In this way, user don't need to processing data which generated
by other application with a header or footer and directly use the file for
table operations.
To implement this, the idea is adding new properties in table descriptions to
define the number of lines in header and footer and skip them when reading
the record from record reader. An DDL example for creating a table with
header and footer should be like this:
{code}
Create external table testtable (name string, message string) row format
delimited fields terminated by '\t' lines terminated by '\n' location
'/testtable' tblproperties (skip.header.line.count=1,
skip.footer.line.count=2);
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
