[jira] [Created] (HIVE-12861) Built-in time function of hour in hive

2016-01-13 Thread chunyanglu (JIRA) 
chunyanglu created HIVE-12861:
-

 Summary: Built-in time function of hour in hive 
 Key: HIVE-12861
 URL: https://issues.apache.org/jira/browse/HIVE-12861
 Project: Hive
  Issue Type: Bug
Affects Versions: 1.2.1
Reporter: chunyanglu


when to use then Built-in time function of hour in hive ,not return the right 
result ,for example:

when we run the hql "select hour('2016-01-13 1313:46:38') from dual", 
we get 17. But the date string is not satisfid the format "-mm-dd 
HH:MM:SS",  if we don`t regexp this string, what can we do to solve this 
question?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12859) MSCK Repair table gives error for higher number of partitions

2016-01-13 Thread hakki (JIRA) 
hakki created HIVE-12859:


 Summary: MSCK Repair table gives error for higher number of 
partitions
 Key: HIVE-12859
 URL: https://issues.apache.org/jira/browse/HIVE-12859
 Project: Hive
  Issue Type: Bug
  Components: CLI, Hive
Affects Versions: 1.1.0
 Environment: Env: CDH-5.4.5
Hive version: 1.1.0-cdh5.4.5
OS: OEL 6 x86_64

Reporter: hakki


Hi,
For large number of partitions (specifically, when number of present 
directories is higher than 1.000.000) msck repair table gives the error from 
hive cli:
"FAILED: Execution Error, return code 1 from 
org.apache.hadoop.hive.ql.exec.DDLTask"
The exception is taken from also beeline cli:
Getting log thread is interrupted, since query is done!
Error: Error while processing statement: FAILED: Execution Error, return code 1 
from org.apache.hadoop.hive.ql.exec.DDLTask (state=08S01, code=1)
java.sql.SQLException: Error while processing statement: FAILED: Execution 
Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12860) Add WITH HEADER option to INSERT OVERWRITE DIRECTORY

2016-01-13 Thread Elliot West (JIRA) 
Elliot West created HIVE-12860:
--

 Summary: Add WITH HEADER option to INSERT OVERWRITE DIRECTORY
 Key: HIVE-12860
 URL: https://issues.apache.org/jira/browse/HIVE-12860
 Project: Hive
  Issue Type: New Feature
  Components: Hive
Reporter: Elliot West
Assignee: Elliot West


_As a Hive user_
_I'd like the option to seamlessly write out a header row to file system based 
result sets_
_So that I can generate reports whose specification mandates a header row._

h4. Motivations
There is a significant use-case where Hive is used to construct a scheduled 
data processing pipeline that generates a report in HDFS for consumption by 
some third party (internal or external). This report may then be transferred 
out of the system for consumption by other tools or processes. It is not 
uncommon for the third party to specify that the report includes a header row 
at the start of the file. The current options for adding headers are difficult 
to use effectively and elegantly.

h4. Acceptance criteria
* {{INSERT OVERWRITE DIRECTORY}} commands can be invoked with an option to 
include a header row at the start of the result set file.
* The header row will contain the column names derived from the accompanying 
{{SELECT}} query.
* It will likely be the case that multiple tasks will be writing the final file 
of the query result set. In this event only the task writing the first chunk of 
the file should emit the header row.

h4. Proposed HQL changes
{code}
1.  INSERT OVERWRITE [LOCAL] DIRECTORY directory1
2.[ROW FORMAT row_format] [STORED AS file_format]
3.[WITH HEADER]
4.SELECT ... FROM ...
{code}
It is proposed that the {{WITH HEADER}} stanza at line 3 be introduced to 
enable this feature.
h4. Current workarounds
* It is usually suggested that users set the CLI option 
{{hive.cli.print.header=true}} and capture the result set from standard out. 
However, this does not work well in scheduled, headless environments such as 
the Oozie Hive action. This can also push the file handling into shell scripts 
and complicate the process of getting the report into HDFS.
* The keep report processing entirely within the domain of Hive some users 
{{UNION}} the result of their query with a tiny table of a single row 
containing the header names. A synthesised rank column is used with an {{ORDER 
BY}} to ensure that the header is written to the very start of the file. See 
[this example on Stack 
Overflow|http://stackoverflow.com/questions/15139561/adding-column-headers-to-hive-result-set/25214480#25214480].

h4. References
* HIVE-138: Original request for header functionality.
* [Hive Wiki: writing data into the file system from 
queries|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Writingdataintothefilesystemfromqueries].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 41928: HIVE-12777: Add capability to restore session

2016-01-13 Thread Rajat Khandelwal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41928/
---

(Updated Jan. 13, 2016, 4:57 p.m.)


Review request for hive.


Bugs: HIVE-12777
https://issues.apache.org/jira/browse/HIVE-12777


Repository: hive-git


Description
---

Extensions using Hive session handles should be able to restore the hive 
session from the handle. 

Apache Lens depends on a fork of hive and that fork has such a capability. 

Relevant commit: 
https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be


Diffs (updated)
-

  
itests/hive-unit/src/test/java/org/apache/hive/service/cli/TestEmbeddedThriftBinaryCLIService.java
 de66d9efb1cace9d32174e3020920d5e4002dc85 
  service-rpc/if/TCLIService.thrift 0aa9d13464bf284a3048092372299efb8e1d6bcc 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService.h 
3407564de27c56cf8f0ee057eee4fcfb8f6ca81e 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService.cpp 
fc82b884d9d0cae47a43f8a734689af0cf6771ef 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService_server.skeleton.cpp 
66ed6a7c7e51b212e9d6e190fb4376f06abb4098 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 
53642939829ec137db6b5ded46c6c3edef3b9e69 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 
e62e8b720e967dac6bbd4f53328a85256a07e022 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCLIService.java
 c684f895127e4a1efed53fa0e6dda5221a935c28 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementReq.java
 2eb4d091f0d500d5c03b7e803a81400726deaf67 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java
 1aa3f946727b6d4b5bf39af40acd5d3b3119949c 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TRestoreSessionReq.java
 PRE-CREATION 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TRestoreSessionResp.java
 PRE-CREATION 
  service-rpc/src/gen/thrift/gen-php/TCLIService.php 
eba62f12de0899280b9961dd82330bb149d40758 
  service-rpc/src/gen/thrift/gen-php/Types.php 
76805df4910a31636dccbd119abe91dfbb1568a1 
  service-rpc/src/gen/thrift/gen-py/TCLIService/TCLIService-remote 
56f5c5dc36d44627a2cd571c2bc1fcd1a79e62f2 
  service-rpc/src/gen/thrift/gen-py/TCLIService/TCLIService.py 
ad2d71d720376c76632d9e3a147f227b3ff67c17 
  service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py 
ef5f5f568ffec92c093bdd0904c154f8ee67606e 
  service-rpc/src/gen/thrift/gen-py/__init__.py PRE-CREATION 
  service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service.rb 
7d7f7a78a17cab50b18fa77805022bf4ca0197d9 
  service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 
f004ec40afd29ab4ed452f65f306cb1af5ff5207 
  service/src/gen/thrift/gen-cpp/ThriftHive.h PRE-CREATION 
  service/src/gen/thrift/gen-cpp/ThriftHive.cpp PRE-CREATION 
  service/src/gen/thrift/gen-cpp/ThriftHive_server.skeleton.cpp PRE-CREATION 
  service/src/gen/thrift/gen-cpp/hive_service_constants.h PRE-CREATION 
  service/src/gen/thrift/gen-cpp/hive_service_constants.cpp PRE-CREATION 
  service/src/gen/thrift/gen-cpp/hive_service_types.h PRE-CREATION 
  service/src/gen/thrift/gen-cpp/hive_service_types.cpp PRE-CREATION 
  
service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/HiveClusterStatus.java
 PRE-CREATION 
  
service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/HiveServerException.java
 PRE-CREATION 
  
service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/JobTrackerState.java
 PRE-CREATION 
  
service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/ThriftHive.java
 PRE-CREATION 
  service/src/gen/thrift/gen-php/ThriftHive.php PRE-CREATION 
  service/src/gen/thrift/gen-php/Types.php PRE-CREATION 
  service/src/gen/thrift/gen-py/__init__.py PRE-CREATION 
  service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote PRE-CREATION 
  service/src/gen/thrift/gen-py/hive_service/ThriftHive.py PRE-CREATION 
  service/src/gen/thrift/gen-py/hive_service/__init__.py PRE-CREATION 
  service/src/gen/thrift/gen-py/hive_service/constants.py PRE-CREATION 
  service/src/gen/thrift/gen-py/hive_service/ttypes.py PRE-CREATION 
  service/src/gen/thrift/gen-rb/hive_service_constants.rb PRE-CREATION 
  service/src/gen/thrift/gen-rb/hive_service_types.rb PRE-CREATION 
  service/src/gen/thrift/gen-rb/thrift_hive.rb PRE-CREATION 
  service/src/java/org/apache/hive/service/cli/CLIService.java 
0d5ae1e491bdcf98a12f5eef7b35f23c7b9dfc00 
  service/src/java/org/apache/hive/service/cli/CLIServiceClient.java 
3155c238ff688bfea16b0aaeea950599bb659b5b 
  service/src/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java 
9cad5be198c063115a8e90c67b1c2fd910ca8bc6 
  service/src/java/org/apache/hive/service/cli/ICLIService.java 
0a54bdda8067330ea358683054aa987a6c22317f

Build failed in Jenkins: HIVE-TRUNK-JAVA8 #155

2016-01-13 Thread hiveqa 
See 


Changes:

[jdere] HIVE-12772: Beeline/JDBC output of decimal values is not 0-padded, does 
not match with CLI output (Jason Dere, reviewed by Xuefu Zhang)

[harisankar] HIVE-12640 : Allow StatsOptimizer to optimize the query for 
Constant GroupBy keys (Hari Subramaniyan, reviewed by Ashutosh Chauhan)

--
[...truncated 310 lines...]
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestHostExecutor.testIsolatedFailsOnExec.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestExecutionPhase.testPassingUnitTest.approved.txt
Aptest2/src/test/java/org/apache/hive/ptest/execution/TestJIRAService.java
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestReportingPhase.testExecute.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestHostExecutor.testShutdownBeforeExec.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestScripts.testPrepGit.approved.txt
Aptest2/src/test/java/org/apache/hive/ptest/execution/ExtendedAssert.java
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestPhase.testRsyncFromLocalToRemoteInstancesWithFailureUnknown.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestHostExecutor.testParallelFailsOnExec.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestPhase.testExecHostsWithFailure.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestCleanupPhase.testExecute.approved.txt
Aptest2/src/test/java/org/apache/hive/ptest/execution/TestScripts.java
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestHostExecutor.testIsolatedFailsOnRsyncUnknown.approved.txt
Aptest2/src/test/java/org/apache/hive/ptest/execution/TestPrepPhase.java
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestScripts.testPrepNone.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestHostExecutor.testBasic.approved.txt
Aptest2/src/test/java/org/apache/hive/ptest/execution/TestLocalCommand.java
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestScripts.testAlternativeTestJVM.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestHostExecutor.testParallelFailsOnRsync.approved.txt
Aptest2/src/test/java/org/apache/hive/ptest/execution/TestPhase.java
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestLogDirectoryCleaner.java
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestScripts.testPrepSvn.approved.txt
Aptest2/src/test/java/org/apache/hive/ptest/execution/context
A
ptest2/src/test/java/org/apache/hive/ptest/execution/context/TestCloudExecutionContextProvider.java
A
ptest2/src/test/java/org/apache/hive/ptest/execution/context/TestCloudComputeService.java
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestPrepPhase.testExecute.approved.txt
Aptest2/src/test/java/org/apache/hive/ptest/execution/conf
A
ptest2/src/test/java/org/apache/hive/ptest/execution/conf/TestTestParser.java
A
ptest2/src/test/java/org/apache/hive/ptest/execution/conf/TestQFileTestBatch.java
A
ptest2/src/test/java/org/apache/hive/ptest/execution/conf/TestTestConfiguration.java
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestPhase.testRsyncFromLocalToRemoteInstancesWithFailureOne.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestExecutionPhase.testFailingUnitTest.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestHostExecutor.testIsolatedFailsOnRsyncOne.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestPhase.testExecInstancesWithFailure.approved.txt
Aptest2/src/test/java/org/apache/hive/ptest/execution/ssh
A
ptest2/src/test/java/org/apache/hive/ptest/execution/ssh/TestSSHCommandExecutor.java
A
ptest2/src/test/java/org/apache/hive/ptest/execution/ssh/TestRSyncCommandExecutor.java
Aptest2/pom.xml
Aptest2/README.md
Aptest2/.gitignore
Aptest2/conf
Aptest2/conf/example-apache-trunk.properties
Checked out revision 1724395.
+ cd ptest2
+ mvn clean package -DskipTests -Drat.numUnapprovedLicenses=1000 
-Dmaven.repo.local=
[INFO] Scanning for projects...
[INFO] 
[INFO] 
[INFO] Building hive-ptest 1.0
[INFO] 
Downloading: 
http://repo.springsource.org/milestone/net/schmizz/sshj/maven-metadata.xml
Downloading: 
http://people.apache.org/~brock/approvals-maven-repository/net/schmizz/sshj/maven-metadata.xml
Downloading: 
http://repo.maven.apache.org/maven2/net/schmizz/sshj/maven-metadata.xml

[jira] [Created] (HIVE-12865) Exchange partition does not show inputs field for post/pre execute hooks

2016-01-13 Thread Paul Yang (JIRA) 
Paul Yang created HIVE-12865:


 Summary: Exchange partition does not show inputs field for 
post/pre execute hooks
 Key: HIVE-12865
 URL: https://issues.apache.org/jira/browse/HIVE-12865
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Affects Versions: 0.12.0
Reporter: Paul Yang


The pre/post execute hook interface has fields that indicate which Hive objects 
were read / written to as a result of running the query. For the exchange 
partition operation, the read entity field is empty.

This is an important issue as the hook interface may be configured to perform 
critical warehouse operations.

See
ql/src/test/results/clientpositive/exchange_partition3.q.out

{code}
--- a/ql/src/test/results/clientpositive/exchange_partition3.q.out
+++ b/ql/src/test/results/clientpositive/exchange_partition3.q.out
@@ -65,9 +65,17 @@ ds=2013-04-05/hr=2
 PREHOOK: query: -- This will exchange both partitions hr=1 and hr=2
 ALTER TABLE exchange_part_test1 EXCHANGE PARTITION (ds='2013-04-05') WITH 
TABLE exchange_part_test2
 PREHOOK: type: ALTERTABLE_EXCHANGEPARTITION
+PREHOOK: Output: default@exchange_part_test1
+PREHOOK: Output: default@exchange_part_test2
 POSTHOOK: query: -- This will exchange both partitions hr=1 and hr=2
 ALTER TABLE exchange_part_test1 EXCHANGE PARTITION (ds='2013-04-05') WITH 
TABLE exchange_part_test2
 POSTHOOK: type: ALTERTABLE_EXCHANGEPARTITION
+POSTHOOK: Output: default@exchange_part_test1
+POSTHOOK: Output: default@exchange_part_test1@ds=2013-04-05/hr=1
+POSTHOOK: Output: default@exchange_part_test1@ds=2013-04-05/hr=2
+POSTHOOK: Output: default@exchange_part_test2
+POSTHOOK: Output: default@exchange_part_test2@ds=2013-04-05/hr=1
+POSTHOOK: Output: default@exchange_part_test2@ds=2013-04-05/hr=2
 PREHOOK: query: SHOW PARTITIONS exchange_part_test1
 PREHOOK: type: SHOWPARTITIONS
 PREHOOK: Input: default@exchange_part_test1
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 41821: HIVE-12767: Implement table property to address Parquet int96 timestamp bug

2016-01-13 Thread Sergio Pena 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41821/
---

(Updated Jan. 13, 2016, 8:36 p.m.)


Review request for hive, Ryan Blue, Mohammad Islam, Reuben Kuhnert, and Szehon 
Ho.


Bugs: HIVE-12767
https://issues.apache.org/jira/browse/HIVE-12767


Repository: hive-git


Description
---

The following exit criteria is addressed in this patch:

* Hive will read Parquet MR int96 timestamp data and adjust values using a time 
zone from a table property, if set, or using the local time zone if it is 
absent. No adjustment will be applied to data written by Impala.

* Hive will write Parquet int96 timestamps using a time zone adjustment from 
the same table property, if set, or using the local time zone if it is absent. 
This keeps the data in the table consistent.

* New tables created by Hive will set the table property to UTC if the global 
option to set the property for new tables is enabled.
  * Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set 
the property unless the global setting to do so is enabled.
  * Tables created using CREATE TABLE LIKE  will copy the property 
of the table that is copied.

To set the timezone table property, use this:
  create table tbl1 (ts timestamp) stored as parquet tblproperties 
('parquet.mr.int96.write.zone'='PST');
  
To set UTC as default timezone table property on new tables created, use this: 
  set parquet.mr.int96.enable.utc.write.zone=true;
  create table tbl2 (ts timestamp) stored as parquet;


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
1bcdc5f49e1a4a0f357842e88cf5fd359685b5ef 
  data/files/impala_int96_timestamp.parq PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
deec8bba45c130c5dfdc482522c0825a71af9d2c 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetOutputFormat.java 
bfb48a987ce89a373f3da63c9162546c6eda43a9 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ETypeConverter.java 
ec0dd818f688ab92feb46be4fb6040ede5ac756a 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/DataWritableReadSupport.java
 53f3b72b790d87a75a7cd1d77d8f011c29c41188 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/ParquetRecordReaderWrapper.java
 74a1a82047613189678716f765bfaa9ac39b7618 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetTableUtils.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/timestamp/NanoTimeUtils.java 
aace48ee7d145d199163286d21e4ee7694140d6f 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/DataWritableWriteSupport.java
 f4621e5dbb81e8d58c4572c901ec9d1a7ca8c012 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/DataWritableWriter.java 
69272dc41dbc5fe29ab4c98e730b591c28f3a297 
  ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestDataWritableWriter.java 
70491390ba2b90f32ef9963be7b19e57672241f3 
  
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/convert/TestETypeConverter.java
 PRE-CREATION 
  
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/serde/TestParquetTimestampUtils.java
 ec6def5b9ac5f12e6a7cb24c4f4998a6ca6b4a8e 
  
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/timestamp/TestParquetTimestampConverter.java
 PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_int96_timestamp.q PRE-CREATION 
  ql/src/test/results/clientpositive/parquet_int96_timestamp.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/41821/diff/


Testing
---

Added unit and q-tests:
  ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestDataWritableWriter.java
  
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/convert/TestETypeConverter.java
  
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/serde/TestParquetTimestampUtils.java
  
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/timestamp/TestParquetTimestampConverter.java
  ql/src/test/queries/clientpositive/parquet_int96_timestamp.q


Thanks,

Sergio Pena

[jira] [Created] (HIVE-12864) StackOverflowError parsing queries with very large predicates

2016-01-13 Thread Jesus Camacho Rodriguez (JIRA) 
Jesus Camacho Rodriguez created HIVE-12864:
--

 Summary: StackOverflowError parsing queries with very large 
predicates
 Key: HIVE-12864
 URL: https://issues.apache.org/jira/browse/HIVE-12864
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.0.0, 2.1.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


We have seen that queries with very large predicates might fail with the 
following stacktrace:

{noformat}
016-01-12 05:47:36,516|beaver.machine|INFO|552|5072|Thread-22|Exception in 
thread "main" java.lang.StackOverflowError

2016-01-12 05:47:36,517|beaver.machine|INFO|552|5072|Thread-22|at 
org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:145)

2016-01-12 05:47:36,517|beaver.machine|INFO|552|5072|Thread-22|at 
org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
2016-01-12 05:47:36,517|beaver.machine|INFO|552|5072|Thread-22|at 
org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
2016-01-12 05:47:36,517|beaver.machine|INFO|552|5072|Thread-22|at 
org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
2016-01-12 05:47:36,517|beaver.machine|INFO|552|5072|Thread-22|at 
org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at 
org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at 
org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at 
org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at 
org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at 
org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at 
org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at 
org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at 
org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)

2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at 
org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)

2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at 
org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at 
org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at 
org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at 
org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at 
org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at 
org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at 
org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at 
org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
2016-01-12 05:47:36,520|beaver.machine|INFO|552|5072|Thread-22|at 
org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
2016-01-12 05:47:36,520|beaver.machine|INFO|552|5072|Thread-22|at 
org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
2016-01-12 05:47:36,520|beaver.machine|INFO|552|5072|Thread-22|at 
org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
2016-01-12 05:47:36,520|beaver.machine|INFO|552|5072|Thread-22|at 
org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
2016-01-12 05:47:36,520|beaver.machine|INFO|552|5072|Thread-22|at 
org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
2016-01-12 05:47:36,520|beaver.machine|INFO|552|5072|Thread-22|at 
org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
2016-01-12 05:47:36,520|beaver.machine|INFO|552|5072|Thread-22|at 
org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
2016-01-12

[jira] [Created] (HIVE-12866) Allow ReduceSinkDeDuplication to kick in when there are constant keys

2016-01-13 Thread Hari Sankar Sivarama Subramaniyan (JIRA) 
Hari Sankar Sivarama Subramaniyan created HIVE-12866:


 Summary: Allow ReduceSinkDeDuplication to kick in when there are 
constant keys 
 Key: HIVE-12866
 URL: https://issues.apache.org/jira/browse/HIVE-12866
 Project: Hive
  Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan


Currently in ReduceSinkDeDuplication.checkExprs() we have the following check 
which prevents the optimization from kicking in as soon as we hit constant key. 
We should lift this restriction as much as possible.
{code}
private Integer checkExprs(List ckeys, List 
pkeys,
ReduceSinkOperator cRS, ReduceSinkOperator pRS) throws 
SemanticException {
  // If ckeys or pkeys have constant node expressions avoid the merge.
  for (ExprNodeDesc ck : ckeys) {
if (ck instanceof ExprNodeConstantDesc) {
  return null;
}
  }
  for (ExprNodeDesc pk : pkeys) {
if (pk instanceof ExprNodeConstantDesc) {
  return null;
}
  }
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 41482: StatsSetupConst.COLUMN_STATS_ACCURATE is not used correctly

2016-01-13 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41482/#review114265
---



common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java (line 142)


We shall also check that value of BASIC_STATS is true. Even if we assume 
our code always ensure presence of BASIC_STATS flag indicating value as true, 
since end user can change tbl properties they may assume setting it to false 
may have an effect.



common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java (line 175)


Similarily here check value.



common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java (line 184)


LOG.trace



common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java (line 192)


Better name: setBasicStatsState ?



common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java (line 205)


Add LOG.trace(e)



common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java (line 223)


LOG.trace?



common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java (line 231)


Better name: setColumnStatsState ?



common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java (line 237)


Are we guaranteed that these colNames are lowercased, or shall we do 
.toLowerCase() for them?



common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java (line 272)


LOG.trace



ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java (lines 360 - 
370)


This logic should be part of setPartitionColStats/updateTblColStats() 
metastore calls which caller of this function invokes.

In current form, it has both semantic problem (stats state is set before 
actually setting stats) and perf problem (unnecessary metastore calls).



ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java (line 4349)


Can you add comment for this change? For truncate, setting stats to 0 is 
valid, no?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/StatsOptimizer.java (lines 373 
- 375)


Now that we are making sure, in case of invalid stats, we are removing 
them, we can possibly uplift this restriction.


- Ashutosh Chauhan


On Jan. 12, 2016, 5:17 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41482/
> ---
> 
> (Updated Jan. 12, 2016, 5:17 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-12661
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 2ff76ee 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> 432f7d0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java f6fbe74 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java ac0ecd9 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java edf69fe 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java dca1198 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java f151871 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/StatsOptimizer.java 03c1c3f 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/StatsWork.java d87022d 
>   ql/src/test/queries/clientpositive/columnStatsUpdateForStatsOptimizer_1.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/columnStatsUpdateForStatsOptimizer_2.q 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/stats_partialscan_autogether.q.out 
> d03c249 
>   ql/src/test/results/clientnegative/unset_table_property.q.out 158ed38 
>   ql/src/test/results/clientpositive/alter_file_format.q.out c9e88f8 
>   ql/src/test/results/clientpositive/alter_merge_stats_orc.q.out cefe069 
>   
> ql/src/test/results/clientpositive/alter_numbuckets_partitioned_table2_h23.q.out
>  29a4c4b 
>   
> ql/src/test/results/clientpositive/alter_numbuckets_partitioned_table_h23.q.out
>  4d0f841 
>   ql/src/test/results/clientpositive/alter_partition_clusterby_sortby.q.out 
> f5e8d1f 
>   ql/src/test/results/clientpositive/alter_partition_coltype.q.out 08bd2b8 
>   ql/src/test/results/clientpositive/alter_skewed_table.q.out 03904e6 
>

Review Request 42260: HIVE-12806 : CBO: Calcite Operator To Hive Operator (Calcite Return Path): MiniTezCliDriver vector_auto_smb_mapjoin_14.q failure

2016-01-13 Thread Hari Sankar Sivarama Subramaniyan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42260/
---

Review request for hive, Ashutosh Chauhan and John Pullokkaran.


Repository: hive-git


Description
---

HIVE-12806 : CBO: Calcite Operator To Hive Operator (Calcite Return Path): 
MiniTezCliDriver vector_auto_smb_mapjoin_14.q failure


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 5e0553d 

Diff: https://reviews.apache.org/r/42260/diff/


Testing
---


Thanks,

Hari Sankar Sivarama Subramaniyan

[jira] [Created] (HIVE-12863) fix test failure for TestMiniTezCliDriver.testCliDriver_tez_union

2016-01-13 Thread Pengcheng Xiong (JIRA) 
Pengcheng Xiong created HIVE-12863:
--

 Summary: fix test failure for 
TestMiniTezCliDriver.testCliDriver_tez_union
 Key: HIVE-12863
 URL: https://issues.apache.org/jira/browse/HIVE-12863
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12871) rawDataSize is always 0 for some query with concurrency support

2016-01-13 Thread Pengcheng Xiong (JIRA) 
Pengcheng Xiong created HIVE-12871:
--

 Summary: rawDataSize is always 0 for some query with concurrency 
support
 Key: HIVE-12871
 URL: https://issues.apache.org/jira/browse/HIVE-12871
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong


{code}
set hive.stats.fetch.column.stats=true;
set hive.stats.fetch.partition.stats=true;
set hive.compute.query.using.stats=true;
set hive.mapred.mode=nonstrict;
set hive.exec.dynamic.partition.mode=nonstrict;
set hive.support.concurrency=true;
set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;

drop table calendar;

CREATE TABLE calendar (year int, month int) clustered by (month) into 2 buckets 
stored as orc TBLPROPERTIES ('transactional'='true');

insert into calendar values (2010, 10), (2011, 11), (2012, 12);

desc formatted calendar;

{code}

desc will return 
{code}
rawDataSize 0
totalSize   1242
{code}

If I commented out 
{code}
set hive.support.concurrency=true;
set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
{code}

It will return
{code}
rawDataSize 24
totalSize   547
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 41928: HIVE-12777: Add capability to restore session

2016-01-13 Thread Rajat Khandelwal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41928/
---

(Updated Jan. 14, 2016, 12:29 p.m.)


Review request for hive.


Bugs: HIVE-12777
https://issues.apache.org/jira/browse/HIVE-12777


Repository: hive-git


Description
---

Extensions using Hive session handles should be able to restore the hive 
session from the handle. 

Apache Lens depends on a fork of hive and that fork has such a capability. 

Relevant commit: 
https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be


Diffs (updated)
-

  
itests/hive-unit/src/test/java/org/apache/hive/service/cli/TestEmbeddedThriftBinaryCLIService.java
 de66d9efb1cace9d32174e3020920d5e4002dc85 
  service-rpc/if/TCLIService.thrift 0aa9d13464bf284a3048092372299efb8e1d6bcc 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService.h 
3407564de27c56cf8f0ee057eee4fcfb8f6ca81e 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService.cpp 
fc82b884d9d0cae47a43f8a734689af0cf6771ef 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService_server.skeleton.cpp 
66ed6a7c7e51b212e9d6e190fb4376f06abb4098 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 
53642939829ec137db6b5ded46c6c3edef3b9e69 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 
e62e8b720e967dac6bbd4f53328a85256a07e022 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCLIService.java
 c684f895127e4a1efed53fa0e6dda5221a935c28 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementReq.java
 2eb4d091f0d500d5c03b7e803a81400726deaf67 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java
 1aa3f946727b6d4b5bf39af40acd5d3b3119949c 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TRestoreSessionReq.java
 PRE-CREATION 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TRestoreSessionResp.java
 PRE-CREATION 
  service-rpc/src/gen/thrift/gen-php/TCLIService.php 
eba62f12de0899280b9961dd82330bb149d40758 
  service-rpc/src/gen/thrift/gen-php/Types.php 
76805df4910a31636dccbd119abe91dfbb1568a1 
  service-rpc/src/gen/thrift/gen-py/TCLIService/TCLIService-remote 
56f5c5dc36d44627a2cd571c2bc1fcd1a79e62f2 
  service-rpc/src/gen/thrift/gen-py/TCLIService/TCLIService.py 
ad2d71d720376c76632d9e3a147f227b3ff67c17 
  service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py 
ef5f5f568ffec92c093bdd0904c154f8ee67606e 
  service-rpc/src/gen/thrift/gen-py/__init__.py PRE-CREATION 
  service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service.rb 
7d7f7a78a17cab50b18fa77805022bf4ca0197d9 
  service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 
f004ec40afd29ab4ed452f65f306cb1af5ff5207 
  service/src/gen/thrift/gen-cpp/ThriftHive.h PRE-CREATION 
  service/src/gen/thrift/gen-cpp/ThriftHive.cpp PRE-CREATION 
  service/src/gen/thrift/gen-cpp/ThriftHive_server.skeleton.cpp PRE-CREATION 
  service/src/gen/thrift/gen-cpp/hive_service_constants.h PRE-CREATION 
  service/src/gen/thrift/gen-cpp/hive_service_constants.cpp PRE-CREATION 
  service/src/gen/thrift/gen-cpp/hive_service_types.h PRE-CREATION 
  service/src/gen/thrift/gen-cpp/hive_service_types.cpp PRE-CREATION 
  
service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/HiveClusterStatus.java
 PRE-CREATION 
  
service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/HiveServerException.java
 PRE-CREATION 
  
service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/JobTrackerState.java
 PRE-CREATION 
  
service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/ThriftHive.java
 PRE-CREATION 
  service/src/gen/thrift/gen-php/ThriftHive.php PRE-CREATION 
  service/src/gen/thrift/gen-php/Types.php PRE-CREATION 
  service/src/gen/thrift/gen-py/__init__.py PRE-CREATION 
  service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote PRE-CREATION 
  service/src/gen/thrift/gen-py/hive_service/ThriftHive.py PRE-CREATION 
  service/src/gen/thrift/gen-py/hive_service/__init__.py PRE-CREATION 
  service/src/gen/thrift/gen-py/hive_service/constants.py PRE-CREATION 
  service/src/gen/thrift/gen-py/hive_service/ttypes.py PRE-CREATION 
  service/src/gen/thrift/gen-rb/hive_service_constants.rb PRE-CREATION 
  service/src/gen/thrift/gen-rb/hive_service_types.rb PRE-CREATION 
  service/src/gen/thrift/gen-rb/thrift_hive.rb PRE-CREATION 
  service/src/java/org/apache/hive/service/cli/CLIService.java 
0d5ae1e491bdcf98a12f5eef7b35f23c7b9dfc00 
  service/src/java/org/apache/hive/service/cli/CLIServiceClient.java 
3155c238ff688bfea16b0aaeea950599bb659b5b 
  service/src/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java 
9cad5be198c063115a8e90c67b1c2fd910ca8bc6 
  service/src/java/org/apache/hive/service/cli/ICLIService.java 
0a54bdda8067330ea358683054aa987a6c22317f

Hive Configure Method not getting called

2016-01-13 Thread shakun grover 
Hi All,


I am new to Hive and facing an issue while writing a Generic UDF. In the
GenericUDF, the configure method is not getting called due to which I am
unable to get the MapredContext.

I have tried modifying the following properties in hive-site.xml
1) hive.execution.engine = mr,
2) hive.fetch.task.conversion = none
3) vectorization support = true.

But no luck. I am using Hive 1.2.1.2.3.2.0-2950.

Please tell me what configuration settings are required to call the
configure and close method.

Thanks in advance!!

-- 
Thanks & Regards,
Shakun Grover

[jira] [Created] (HIVE-12872) NoSuchMethodError exception Clause in Hive 1.1.1

2016-01-13 Thread fxliuwenjie (JIRA) 
fxliuwenjie created HIVE-12872:
--

 Summary: NoSuchMethodError exception Clause in Hive 1.1.1
 Key: HIVE-12872
 URL: https://issues.apache.org/jira/browse/HIVE-12872
 Project: Hive
  Issue Type: Bug
Affects Versions: 1.1.1
 Environment: Hadoop 2.6.0
Reporter: fxliuwenjie


Firstly, i created a table :
hive>crete externale table beauties(id bigint, name string, heigth double) 
partitioned by (nation string) row format delimited fields terminated by '\t' 
location '/beauty';
Then i loaded data into this table:
hive>load data loca inpath '/home/tmpdata/b.c' into table beauties 
partition(nation='China');
hive>load data local inpath '/home/tmpdata/b.j' into table beauties 
partition(nation='Japan');
Then i test to query the uploaded data:
hive>select * from beauties;
OK
1 lee 165.0 China
2 jzmb 167.0 Japan
When i tried to run the below query i faced the issue:
hive>select * from beauties where nation = 'China';
Exception in thread "main" 
java.lang.NoSuchMethodError:org.apache/hadoop.hive.ql.ppd.ExprWalkerInfo.getConvertedNode(Lorg/apache/hadoop/hive/ql/lib/Node;)Lorg/apache/hadoop/hive/ql/plan/ExprNodeDesc;
atorg.apache.hadoop.hive.ql.ppd.ExprWalkerProcFactory$GenericFuncExprProcessor.process(ExprWalkerProcFactory.java:176)
at 
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at 
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
at 
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
at 
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
at 
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
at 
org.apache.hadoop.hive.ql.ppd.ExprWalkerProcFactory.extractPushdownPreds(ExprWalkerProcFactory.java:290)
at 
org.apache.hadoop.hive.ql.ppd.ExprWalkerProcFactory.extractPushdownPreds(ExprWalkerProcFactory.java:241)
at 
org.apache.hadoop.hive.ql.ppd.OpProcFactory$FilterPPD.process(OpProcFactory.java:418)
at 
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at 
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
at 
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
at 
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
at 
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
at 
org.apache.hadoop.hive.ql.ppd.PredicatePushDown.transform(PredicatePushDown.java:135)
at 
org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:182)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10207)
at 
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:192).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12870) Vectorization: Vectorized readers for simple data types

2016-01-13 Thread Gopal V (JIRA) 
Gopal V created HIVE-12870:
--

 Summary: Vectorization: Vectorized readers for simple data types
 Key: HIVE-12870
 URL: https://issues.apache.org/jira/browse/HIVE-12870
 Project: Hive
  Issue Type: New Feature
  Components: Tez, Vectorization
Affects Versions: 2.1.0
Reporter: Gopal V
Assignee: Gopal V


Apply the DeserializeRead interface over the simpler data readers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12868) Fix empty operation-pool metrics

2016-01-13 Thread Szehon Ho (JIRA) 
Szehon Ho created HIVE-12868:


 Summary: Fix empty operation-pool metrics
 Key: HIVE-12868
 URL: https://issues.apache.org/jira/browse/HIVE-12868
 Project: Hive
  Issue Type: Bug
  Components: Diagnosability
Reporter: Szehon Ho
Assignee: Szehon Ho


The newly-added operation pool metrics (thread-pool size, queue size) are empty 
because metrics system is initialized too late.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12867) Semantic Exception Error Msg should be with in the range of "10000 to 19999"

2016-01-13 Thread Laljo John Pullokkaran (JIRA) 
Laljo John Pullokkaran created HIVE-12867:
-

 Summary: Semantic Exception Error Msg should be with in the range 
of "1 to 1"
 Key: HIVE-12867
 URL: https://issues.apache.org/jira/browse/HIVE-12867
 Project: Hive
  Issue Type: Bug
Affects Versions: 1.2.1
Reporter: Laljo John Pullokkaran
Assignee: Hari Sankar Sivarama Subramaniyan


At many places errors encountered during semantic exception is translated as 
generic error(GENERIC_ERROR, 4) msg as opposed to semantic error msg.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2016-01-13 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review114334
---



service/if/TCLIService.thrift (line 271)


is it possible to avoid changing field IDs? I think backward compat might 
depend on them. Field should be added at the end.



service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java (line 46)


actually, why is this change necessary?


- Sergey Shelukhin


On Jan. 8, 2016, 7:18 a.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated Jan. 8, 2016, 7:18 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 479fa46 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 245c6a3 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 3f9024a 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 29e6315 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 1634143 
>   service/if/TCLIService.thrift baf583f 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.h b078c99 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.cpp b852379 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TColumnDesc.java
>  b01fadb 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementResp.java
>  0b9aa0f 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java
>  c936ada 
>   service/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5 
>   service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4 
>   service/src/java/org/apache/hive/service/cli/CLIService.java adc9809 
>   service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java f0bbf14 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> d2b3f9c 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> 92135cd 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 2eaab4a 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 4f4e92d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 50c912e 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 8434965 
> 
> Diff: https://reviews.apache.org/r/40867/diff/
> 
> 
> Testing
> ---
> 
> I have tested few simple queries and they worked well. But I think there are 
> some problems for some queries. I'm going to test more queries and fix bugs. 
> I'm also going to add unit tests.
> 
> 
> Thanks,
> 
> Takanobu Asanuma
> 
>

Review Request 42281: Upgrade Hive to Calcite 1.6

2016-01-13 Thread pengcheng xiong 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42281/
---

Review request for hive and John Pullokkaran.


Repository: hive-git


Description
---

HIVE-12839


Diffs
-

  pom.xml 848432c 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
4825a61 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelFactories.java 
eeec44e 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/cost/HiveAlgorithmsUtil.java
 6840418 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/cost/HiveDefaultCostModel.java
 6669d32 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/cost/HiveOnTezCostModel.java
 61a3a64 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/cost/HiveRelMdCost.java 
90f6a7e 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveAggregate.java
 aae57e6 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveFilter.java
 2f8cc7d 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveJoin.java
 27b1e76 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveProject.java
 c5376bf 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveSemiJoin.java
 3558676 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveTableScan.java
 5788805 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateJoinTransposeRule.java
 070c7ea 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveExpandDistinctAggregatesRule.java
 7d7631b 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinPushTransitivePredicatesRule.java
 703c8c6 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
 50e139b 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java
 0af60e8 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortRemoveRule.java
 618c717 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java
 0ec8bf1 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/FilterSelectivityEstimator.java
 c04060f 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdCollation.java
 84fa518 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdDistinctRowCount.java
 1220401 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdDistribution.java
 b83f240 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdMemory.java
 bea5943 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdParallelism.java
 2f51d3b 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdPredicates.java
 b7244fd 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java
 caf8978 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdSelectivity.java
 a0eb83d 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdSize.java
 3224039 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdUniqueKeys.java
 7c22c33 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveOpConverter.java
 00f1acb 

Diff: https://reviews.apache.org/r/42281/diff/


Testing
---


Thanks,

pengcheng xiong

Review Request 42286: HIVE-12853 LLAP: localize permanent UDF jars to daemon

2016-01-13 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42286/
---

Review request for hive and Jason Dere.


Repository: hive-git


Description
---

see jira


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 081c1fe 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 0d85671 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/FunctionLocalizer.java
 PRE-CREATION 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
ddedfbf 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
 34aa5c9 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java ec755a8 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java dca1198 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapDecider.java 
af6129a 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java c38699d 
  ql/src/java/org/apache/hadoop/hive/ql/session/DependencyResolver.java 7289426 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 5f15557 
  ql/src/java/org/apache/hadoop/hive/ql/util/DependencyResolver.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/util/DosToUnix.java 4480d54 
  ql/src/java/org/apache/hadoop/hive/ql/util/ResourceDownloader.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/session/TestAddResource.java 1f07235 

Diff: https://reviews.apache.org/r/42286/diff/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-12869) HiveConf substitution for default values is done at init time

2016-01-13 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-12869:
---

 Summary: HiveConf substitution for default values is done at init 
time
 Key: HIVE-12869
 URL: https://issues.apache.org/jira/browse/HIVE-12869
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


Some defaults have session ID and stuff like that. Also e.g. for LLAP local 
directories having YARN directories as a default would be nice (right now the 
latter is enforced manually).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 41482: StatsSetupConst.COLUMN_STATS_ACCURATE is not used correctly

2016-01-13 Thread pengcheng xiong 


> On Jan. 13, 2016, 9:20 p.m., Ashutosh Chauhan wrote:
> > common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java, line 142
> > 
> >
> > We shall also check that value of BASIC_STATS is true. Even if we 
> > assume our code always ensure presence of BASIC_STATS flag indicating value 
> > as true, since end user can change tbl properties they may assume setting 
> > it to false may have an effect.

Right now user can not change the basic stats through table properties. We will 
provide the correct support in the next patch.


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41482/#review114265
---


On Jan. 12, 2016, 5:17 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41482/
> ---
> 
> (Updated Jan. 12, 2016, 5:17 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-12661
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 2ff76ee 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> 432f7d0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java f6fbe74 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java ac0ecd9 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java edf69fe 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java dca1198 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java f151871 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/StatsOptimizer.java 03c1c3f 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/StatsWork.java d87022d 
>   ql/src/test/queries/clientpositive/columnStatsUpdateForStatsOptimizer_1.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/columnStatsUpdateForStatsOptimizer_2.q 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/stats_partialscan_autogether.q.out 
> d03c249 
>   ql/src/test/results/clientnegative/unset_table_property.q.out 158ed38 
>   ql/src/test/results/clientpositive/alter_file_format.q.out c9e88f8 
>   ql/src/test/results/clientpositive/alter_merge_stats_orc.q.out cefe069 
>   
> ql/src/test/results/clientpositive/alter_numbuckets_partitioned_table2_h23.q.out
>  29a4c4b 
>   
> ql/src/test/results/clientpositive/alter_numbuckets_partitioned_table_h23.q.out
>  4d0f841 
>   ql/src/test/results/clientpositive/alter_partition_clusterby_sortby.q.out 
> f5e8d1f 
>   ql/src/test/results/clientpositive/alter_partition_coltype.q.out 08bd2b8 
>   ql/src/test/results/clientpositive/alter_skewed_table.q.out 03904e6 
>   ql/src/test/results/clientpositive/alter_table_not_sorted.q.out 40974e4 
>   ql/src/test/results/clientpositive/alter_table_serde2.q.out 3b63e7d 
>   ql/src/test/results/clientpositive/analyze_table_null_partition.q.out 
> c7254d1 
>   ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 
> 91b89d7 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_1.q.out a48c148 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_11.q.out 206a619 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out 69ae0bf 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out fbc500b 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_3.q.out 9ae6861 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out 357bc4c 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_5.q.out b763e52 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out cef3bd3 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_8.q.out 567e3cf 
>   ql/src/test/results/clientpositive/binary_output_format.q.out 30d0037 
>   ql/src/test/results/clientpositive/bucket1.q.out 8009514 
>   ql/src/test/results/clientpositive/bucket2.q.out 75de8a9 
>   ql/src/test/results/clientpositive/bucket3.q.out c459870 
>   ql/src/test/results/clientpositive/bucket4.q.out e0e5fa6 
>   ql/src/test/results/clientpositive/bucket5.q.out c1dd90d 
>   ql/src/test/results/clientpositive/bucket_many.q.out 04ec255 
>   ql/src/test/results/clientpositive/bucket_map_join_1.q.out c7a8a20 
>   ql/src/test/results/clientpositive/bucket_map_join_2.q.out 3c3793f 
>   ql/src/test/results/clientpositive/bucket_map_join_spark1.q.out 8411a9b 
>   ql/src/test/results/clientpositive/bucket_map_join_spark2.q.out 008d57a 
>   ql/src/test/results/clientpositive/bucket_map_join_spark3.q.out d5ca1ad 
>   ql/src/test/results/clientpositive/bucket_map_join_spark4.q.out 42abaa3 
>   ql/src/test/results/clientpositive/bucketcontext_1.q.out d28e582 
>

[jira] [Created] (HIVE-12862) Fix pom.xml to package JSP of HS2.

2016-01-13 Thread Shinichi Yamashita (JIRA) 
Shinichi Yamashita created HIVE-12862:
-

 Summary: Fix pom.xml to package JSP of HS2.
 Key: HIVE-12862
 URL: https://issues.apache.org/jira/browse/HIVE-12862
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Shinichi Yamashita
Assignee: Shinichi Yamashita
 Attachments: HIVE-12862.1.patch

In current master, we can confirm 404 HTTP status code when we access Web UI of 
HS2.

* Web browser
{code}
HTTP ERROR 404

Problem accessing /hiveserver2.jsp. Reason:

Servlet Not Initialized

Caused by:
javax.servlet.UnavailableException: Servlet Not Initialized
at 
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:536)
(snip)
{code}

* log file
{code}
2016-01-13T22:26:03,787 WARN  [main]: servlet.Holder (Holder.java:doStart(90)) 
- 
java.lang.ClassNotFoundException: 
org.apache.hive.generated.hiveserver2.hiveserver2_jsp
at java.net.URLClassLoader.findClass(URLClassLoader.java:381) 
~[?:1.8.0_65]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_65]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_65]
at 
org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:424)
 ~[jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
   (snip)
{code}

When service and service-rpc were separated, handling of about the build 
hiveserver2.jsp is deleted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)