[jira] [Commented] (HIVE-4067) Followup to HIVE-701: reduce ambiguity in grammar
[ https://issues.apache.org/jira/browse/HIVE-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598566#comment-13598566 ] Samuel Yuan commented on HIVE-4067: --- This is referring to the Java code in the @members section. The section is still being properly inserted into the generated HiveParser.java file, which then compiles fine, so I don't know why ANTLR is complaining. I'll look into it. Followup to HIVE-701: reduce ambiguity in grammar - Key: HIVE-4067 URL: https://issues.apache.org/jira/browse/HIVE-4067 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Samuel Yuan Assignee: Samuel Yuan Priority: Minor Attachments: HIVE-4067.D8883.1.patch After HIVE-701 the grammar has become much more ambiguous, and the compilation generates a large number of warnings. Making FROM, DISTINCT, PRESERVE, COLUMN, ALL, AND, OR, and NOT reserved keywords again reduces the number of warnings to 134, up from the original 81 warnings but down from the 565 after HIVE-701. Most of the remaining ambiguity is trivial, an example being KW_ELEM_TYPE | KW_KEY_TYPE | KW_VALUE_TYPE | identifier, and they are all correctly handled by ANTLR. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4143) Incorrect column mappings with over clause
[
https://issues.apache.org/jira/browse/HIVE-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-4143:
---
Description:
Following query generates 3 MR job and fails in reducer initialization phase of
third job.
{code}
select ts, dec, rnk
from
(select ts, dec,
rank() over (partition by ts order by item_total desc) as rnk
from
(select other.ts, other.dec, sum(other.i) as item_total
from over10k other
join over10k on (other.b = over10k.b)
group by other.ts, other.dec
) item_sales
) item_rank
where rnk = 3;
{code}
Incorrect column mappings with over clause
--
Key: HIVE-4143
URL: https://issues.apache.org/jira/browse/HIVE-4143
Project: Hive
Issue Type: Bug
Components: PTF-Windowing
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
Following query generates 3 MR job and fails in reducer initialization phase
of third job.
{code}
select ts, dec, rnk
from
(select ts, dec,
rank() over (partition by ts order by item_total desc) as rnk
from
(select other.ts, other.dec, sum(other.i) as item_total
from over10k other
join over10k on (other.b = over10k.b)
group by other.ts, other.dec
) item_sales
) item_rank
where rnk = 3;
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4143) Incorrect column mappings with over clause
[
https://issues.apache.org/jira/browse/HIVE-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-4143:
---
Assignee: Ashutosh Chauhan
Release Note: (was: Following query generates 3 MR job and fails in
reducer initialization phase of third job.
{code}
select ts, dec, rnk
from
(select ts, dec,
rank() over (partition by ts order by item_total desc) as rnk
from
(select other.ts, other.dec, sum(other.i) as item_total
from over10k other
join over10k on (other.b = over10k.b)
group by other.ts, other.dec
) item_sales
) item_rank
where rnk = 3;
{code})
Status: Patch Available (was: Open)
Patch available at https://reviews.facebook.net/D9261
Incorrect column mappings with over clause
--
Key: HIVE-4143
URL: https://issues.apache.org/jira/browse/HIVE-4143
Project: Hive
Issue Type: Bug
Components: PTF-Windowing
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4147) Slow Hive JDBC in concurrency mode to create/drop table
Alexander Alten-Lorenz created HIVE-4147: Summary: Slow Hive JDBC in concurrency mode to create/drop table Key: HIVE-4147 URL: https://issues.apache.org/jira/browse/HIVE-4147 Project: Hive Issue Type: Improvement Components: JDBC Affects Versions: 0.10.0 Reporter: Alexander Alten-Lorenz It's very slow using hive jdbc in concurrency mode to create/drop table, which is 20 times slower than using HiveMetatstoreClient. test steps: 1. create 100 different hive table one by one by using hive jdbc: create table .. 2. drop table one by one by using hive jdbc: drop table .. and timing 3. create 100 different hive table one by one by using hive jdbc: create table .. 4. drop tables one by one by using new HiveMetatstoreClient().dropTable(default, table_name) and timing results step 2 is 20 times slower than step 4. basically hive jdbc is 20 times slower than HiveMetatstoreClient not only create/table, but also the same kind of calls. Dropping tables via this low level API could cause issues if there are any clients concurrently querying the table. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4147) Slow Hive JDBC in concurrency mode to create/drop table
[ https://issues.apache.org/jira/browse/HIVE-4147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598594#comment-13598594 ] Carl Steinbach commented on HIVE-4147: -- Does concurrency mode mean you're using HiveServer2? Slow Hive JDBC in concurrency mode to create/drop table --- Key: HIVE-4147 URL: https://issues.apache.org/jira/browse/HIVE-4147 Project: Hive Issue Type: Improvement Components: JDBC Affects Versions: 0.10.0 Reporter: Alexander Alten-Lorenz It's very slow using hive jdbc in concurrency mode to create/drop table, which is 20 times slower than using HiveMetatstoreClient. test steps: 1. create 100 different hive table one by one by using hive jdbc: create table .. 2. drop table one by one by using hive jdbc: drop table .. and timing 3. create 100 different hive table one by one by using hive jdbc: create table .. 4. drop tables one by one by using new HiveMetatstoreClient().dropTable(default, table_name) and timing results step 2 is 20 times slower than step 4. basically hive jdbc is 20 times slower than HiveMetatstoreClient not only create/table, but also the same kind of calls. Dropping tables via this low level API could cause issues if there are any clients concurrently querying the table. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4147) Slow Hive JDBC in concurrency mode to create/drop table
[ https://issues.apache.org/jira/browse/HIVE-4147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598597#comment-13598597 ] Alexander Alten-Lorenz commented on HIVE-4147: -- HiveMetatstoreClient is using hive.metastore.uris=thrift://IP:9083 to connect to metastore service which is started as standonle mode using port 9083. We've used HiveServer2, yes. Slow Hive JDBC in concurrency mode to create/drop table --- Key: HIVE-4147 URL: https://issues.apache.org/jira/browse/HIVE-4147 Project: Hive Issue Type: Improvement Components: JDBC Affects Versions: 0.10.0 Reporter: Alexander Alten-Lorenz It's very slow using hive jdbc in concurrency mode to create/drop table, which is 20 times slower than using HiveMetatstoreClient. test steps: 1. create 100 different hive table one by one by using hive jdbc: create table .. 2. drop table one by one by using hive jdbc: drop table .. and timing 3. create 100 different hive table one by one by using hive jdbc: create table .. 4. drop tables one by one by using new HiveMetatstoreClient().dropTable(default, table_name) and timing results step 2 is 20 times slower than step 4. basically hive jdbc is 20 times slower than HiveMetatstoreClient not only create/table, but also the same kind of calls. Dropping tables via this low level API could cause issues if there are any clients concurrently querying the table. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4139) MiniDFS shim does not work for hadoop 2
[ https://issues.apache.org/jira/browse/HIVE-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-4139: - Status: Patch Available (was: Open) MiniDFS shim does not work for hadoop 2 --- Key: HIVE-4139 URL: https://issues.apache.org/jira/browse/HIVE-4139 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-4139.1.patch, HIVE-4139.2.patch, HIVE-4139.3.patch There's an incompatibility between hadoop 1 2 wrt to the MiniDfsCluster class. That causes the hadoop 2 line Minimr tests to fail with a MethodNotFound exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3963) Allow Hive to connect to RDBMS
[
https://issues.apache.org/jira/browse/HIVE-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxime LANCIAUX updated HIVE-3963:
--
Fix Version/s: 0.10.1
0.9.1
Release Note: First implementation using UDTF Hive and JDBC
Status: Patch Available (was: Open)
Allow Hive to connect to RDBMS
--
Key: HIVE-3963
URL: https://issues.apache.org/jira/browse/HIVE-3963
Project: Hive
Issue Type: New Feature
Components: Import/Export, JDBC, SQL, StorageHandler
Affects Versions: 0.10.0, 0.9.0, 0.9.1, 0.11.0
Reporter: Maxime LANCIAUX
Fix For: 0.9.1, 0.10.1
I am thinking about something like :
SELECT jdbcload('driver','url','user','password','sql') FROM dual;
There is already a JIRA https://issues.apache.org/jira/browse/HIVE-1555 for
JDBCStorageHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3963) Allow Hive to connect to RDBMS
[
https://issues.apache.org/jira/browse/HIVE-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxime LANCIAUX updated HIVE-3963:
--
Fix Version/s: (was: 0.9.1)
Affects Version/s: (was: 0.9.0)
Status: Open (was: Patch Available)
Allow Hive to connect to RDBMS
--
Key: HIVE-3963
URL: https://issues.apache.org/jira/browse/HIVE-3963
Project: Hive
Issue Type: New Feature
Components: Import/Export, JDBC, SQL, StorageHandler
Affects Versions: 0.10.0, 0.9.1, 0.11.0
Reporter: Maxime LANCIAUX
Fix For: 0.10.1
I am thinking about something like :
SELECT jdbcload('driver','url','user','password','sql') FROM dual;
There is already a JIRA https://issues.apache.org/jira/browse/HIVE-1555 for
JDBCStorageHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3963) Allow Hive to connect to RDBMS
[
https://issues.apache.org/jira/browse/HIVE-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxime LANCIAUX updated HIVE-3963:
--
Attachment: patchfile
This development is rather a POC using UDTF hive function but it works !,
I will focus on removing the need to the DUAL table and improve the
implementation
Allow Hive to connect to RDBMS
--
Key: HIVE-3963
URL: https://issues.apache.org/jira/browse/HIVE-3963
Project: Hive
Issue Type: New Feature
Components: Import/Export, JDBC, SQL, StorageHandler
Affects Versions: 0.10.0, 0.9.1, 0.11.0
Reporter: Maxime LANCIAUX
Fix For: 0.10.1
Attachments: patchfile
I am thinking about something like :
SELECT jdbcload('driver','url','user','password','sql') FROM dual;
There is already a JIRA https://issues.apache.org/jira/browse/HIVE-1555 for
JDBCStorageHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3963) Allow Hive to connect to RDBMS
[
https://issues.apache.org/jira/browse/HIVE-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxime LANCIAUX updated HIVE-3963:
--
Release Note:
This development is rather a POC using UDTF hive function but it works !,
I will focus on removing the need to the DUAL table and improve the
implementation
was:First implementation using UDTF Hive and JDBC
Status: Patch Available (was: Open)
Allow Hive to connect to RDBMS
--
Key: HIVE-3963
URL: https://issues.apache.org/jira/browse/HIVE-3963
Project: Hive
Issue Type: New Feature
Components: Import/Export, JDBC, SQL, StorageHandler
Affects Versions: 0.10.0, 0.9.1, 0.11.0
Reporter: Maxime LANCIAUX
Fix For: 0.10.1
Attachments: patchfile
I am thinking about something like :
SELECT jdbcload('driver','url','user','password','sql') FROM dual;
There is already a JIRA https://issues.apache.org/jira/browse/HIVE-1555 for
JDBCStorageHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4125) Expose metastore JMX metrics
[ https://issues.apache.org/jira/browse/HIVE-4125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-4125: -- Attachment: HIVE-4125.HIVE-4125.HIVE-4125.D9123.2.patch sxyuan updated the revision HIVE-4125 [jira] Expose metastore JMX metrics. Inline comments to follow. Reviewers: kevinwilfong REVISION DETAIL https://reviews.facebook.net/D9123 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D9123?vs=29235id=29523#toc AFFECTED FILES common/src/java/org/apache/hadoop/hive/common/metrics/Metrics.java To: kevinwilfong, sxyuan Cc: JIRA Expose metastore JMX metrics Key: HIVE-4125 URL: https://issues.apache.org/jira/browse/HIVE-4125 Project: Hive Issue Type: Improvement Components: Metastore Affects Versions: 0.11.0 Reporter: Samuel Yuan Assignee: Samuel Yuan Priority: Trivial Attachments: HIVE-4125.HIVE-4125.HIVE-4125.D9123.1.patch, HIVE-4125.HIVE-4125.HIVE-4125.D9123.2.patch Add a safe way to access the metrics stored for each MetricsScope, so that they can be used outside of JMX. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4125) Expose metastore JMX metrics
[ https://issues.apache.org/jira/browse/HIVE-4125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598962#comment-13598962 ] Phabricator commented on HIVE-4125: --- sxyuan has commented on the revision HIVE-4125 [jira] Expose metastore JMX metrics. INLINE COMMENTS common/src/java/org/apache/hadoop/hive/common/metrics/Metrics.java:71 I considered doing that, but it's not a counter tracked by this class, only a metric calculated from the other two. I think it would be better to have whatever needs the average time to recalculate the value instead of getting it from the MetricsMBean (and having to wait for a lock on the metrics map). common/src/java/org/apache/hadoop/hive/common/metrics/Metrics.java:197-202 Done. I originally didn't want to be throwing exceptions but I realized it's possible for a call to be made on a new scope that hasn't been closed yet. REVISION DETAIL https://reviews.facebook.net/D9123 To: kevinwilfong, sxyuan Cc: JIRA Expose metastore JMX metrics Key: HIVE-4125 URL: https://issues.apache.org/jira/browse/HIVE-4125 Project: Hive Issue Type: Improvement Components: Metastore Affects Versions: 0.11.0 Reporter: Samuel Yuan Assignee: Samuel Yuan Priority: Trivial Attachments: HIVE-4125.HIVE-4125.HIVE-4125.D9123.1.patch, HIVE-4125.HIVE-4125.HIVE-4125.D9123.2.patch Add a safe way to access the metrics stored for each MetricsScope, so that they can be used outside of JMX. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4125) Expose metastore JMX metrics
[ https://issues.apache.org/jira/browse/HIVE-4125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samuel Yuan updated HIVE-4125: -- Status: Patch Available (was: Open) Expose metastore JMX metrics Key: HIVE-4125 URL: https://issues.apache.org/jira/browse/HIVE-4125 Project: Hive Issue Type: Improvement Components: Metastore Affects Versions: 0.11.0 Reporter: Samuel Yuan Assignee: Samuel Yuan Priority: Trivial Attachments: HIVE-4125.HIVE-4125.HIVE-4125.D9123.1.patch, HIVE-4125.HIVE-4125.HIVE-4125.D9123.2.patch Add a safe way to access the metrics stored for each MetricsScope, so that they can be used outside of JMX. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3963) Allow Hive to connect to RDBMS
[
https://issues.apache.org/jira/browse/HIVE-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598976#comment-13598976
]
Maxime LANCIAUX commented on HIVE-3963:
---
BI users will like this function because they are using Hive from Tableau or
SAS (or other tools) and they don't want to exit their software and use a
command ligne tool to load a little amount of data. (even if sqoop is a great
tool)
Allow Hive to connect to RDBMS
--
Key: HIVE-3963
URL: https://issues.apache.org/jira/browse/HIVE-3963
Project: Hive
Issue Type: New Feature
Components: Import/Export, JDBC, SQL, StorageHandler
Affects Versions: 0.10.0, 0.9.1, 0.11.0
Reporter: Maxime LANCIAUX
Fix For: 0.10.1
Attachments: patchfile
I am thinking about something like :
SELECT jdbcload('driver','url','user','password','sql') FROM dual;
There is already a JIRA https://issues.apache.org/jira/browse/HIVE-1555 for
JDBCStorageHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1633) CombineHiveInputFormat fails with cannot find dir for emptyFile
[ https://issues.apache.org/jira/browse/HIVE-1633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598983#comment-13598983 ] Shuang Wu commented on HIVE-1633: - This bug seems to also show up in hive local mode, the empty temporary file path is not qualified with file:/. 2013-03-11 09:34:30,767 INFO io.CombineHiveInputFormat (CombineHiveInputFormat.java:getSplits(363)) - CombineHiveInputSplit creating pool for file:/var/folders/w7/fp4gml2n1xqg2434qdp799r2cr/T/shuang/hive_2013-03-11_09-34-29_301_2567414763209147193/-mr-1/1; using filter path file:/var/folders/w7/fp4gml2n1xqg2434qdp799r2cr/T/shuang/hive_2013-03-11_09-34-29_301_2567414763209147193/-mr-1/1 2013-03-11 09:34:30,772 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(196)) - Total input paths to process : 1 2013-03-11 09:34:30,778 INFO mapred.JobClient (JobClient.java:run(919)) - Cleaning up the staging area file:/data/hadoop/cache/analytics-mr.sv2/shuang/mapred/staging/shuang-1827099888/.staging/job_local_0001 2013-03-11 09:34:30,778 ERROR security.UserGroupInformation (UserGroupInformation.java:doAs(1180)) - PriviledgedActionException as:shuang (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: /var/folders/w7/fp4gml2n1xqg2434qdp799r2cr/T/shuang/hive_2013-03-11_09-34-29_301_2567414763209147193/-mr-1/1/emptyFile 2013-03-11 09:34:30,779 ERROR exec.ExecDriver (SessionState.java:printError(365)) - Job Submission failed with exception 'java.io.FileNotFoundException(File does not exist: /var/folders/w7/fp4gml2n1xqg2434qdp799r2cr/T/shuang/hive_2013-03-11_09-34-29_301_2567414763209147193/-mr-1/1/emptyFile)' java.io.FileNotFoundException: File does not exist: /var/folders/w7/fp4gml2n1xqg2434qdp799r2cr/T/shuang/hive_2013-03-11_09-34-29_301_2567414763209147193/-mr-1/1/emptyFile at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:562) at org.apache.hadoop.mapred.lib.CombineFileInputFormat$OneFileInfo.init(CombineFileInputFormat.java:462) at org.apache.hadoop.mapred.lib.CombineFileInputFormat.getMoreSplits(CombineFileInputFormat.java:256) at org.apache.hadoop.mapred.lib.CombineFileInputFormat.getSplits(CombineFileInputFormat.java:212) at org.apache.hadoop.hive.shims.Hadoop20SShims$CombineFileInputFormatShim.getSplits(Hadoop20SShims.java:347) at org.apache.hadoop.hive.shims.Hadoop20SShims$CombineFileInputFormatShim.getSplits(Hadoop20SShims.java:313) at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:377) at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:977) at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:969) at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:880) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:807) at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:671) at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:1092) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:197) CombineHiveInputFormat fails with cannot find dir for emptyFile - Key: HIVE-1633 URL: https://issues.apache.org/jira/browse/HIVE-1633 Project: Hive Issue Type: Bug Components: Clients Reporter: Amareshwari Sriramadasu Assignee: Sreekanth Ramakrishnan Fix For: 0.7.0 Attachments: HIVE-1633.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4125) Expose metastore JMX metrics
[ https://issues.apache.org/jira/browse/HIVE-4125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598994#comment-13598994 ] Kevin Wilfong commented on HIVE-4125: - Sounds reasonable +1 Expose metastore JMX metrics Key: HIVE-4125 URL: https://issues.apache.org/jira/browse/HIVE-4125 Project: Hive Issue Type: Improvement Components: Metastore Affects Versions: 0.11.0 Reporter: Samuel Yuan Assignee: Samuel Yuan Priority: Trivial Attachments: HIVE-4125.HIVE-4125.HIVE-4125.D9123.1.patch, HIVE-4125.HIVE-4125.HIVE-4125.D9123.2.patch Add a safe way to access the metrics stored for each MetricsScope, so that they can be used outside of JMX. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3963) Allow Hive to connect to RDBMS
[
https://issues.apache.org/jira/browse/HIVE-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599022#comment-13599022
]
Mark Grover commented on HIVE-3963:
---
Thanks Maxime. Can you also please post the patch on reviewboard (or
Phabricator) as well?
Allow Hive to connect to RDBMS
--
Key: HIVE-3963
URL: https://issues.apache.org/jira/browse/HIVE-3963
Project: Hive
Issue Type: New Feature
Components: Import/Export, JDBC, SQL, StorageHandler
Affects Versions: 0.10.0, 0.9.1, 0.11.0
Reporter: Maxime LANCIAUX
Fix For: 0.10.1
Attachments: patchfile
I am thinking about something like :
SELECT jdbcload('driver','url','user','password','sql') FROM dual;
There is already a JIRA https://issues.apache.org/jira/browse/HIVE-1555 for
JDBCStorageHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3862) testNegativeCliDriver_cascade_dbdrop fails on hadoop-1
[ https://issues.apache.org/jira/browse/HIVE-3862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599023#comment-13599023 ] Ashutosh Chauhan commented on HIVE-3862: There is no such 1.0 version. We only have 0.20, 0.20S and 0.23. Other than that looks good. testNegativeCliDriver_cascade_dbdrop fails on hadoop-1 -- Key: HIVE-3862 URL: https://issues.apache.org/jira/browse/HIVE-3862 Project: Hive Issue Type: Bug Affects Versions: 0.10.0 Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-3862.patch Actually functionality is working correctly, but incorrect include/exclude macro make cause the wrong query file to be run. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (HIVE-3958) support partial scan for analyze command
[ https://issues.apache.org/jira/browse/HIVE-3958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-3958 started by Gang Tim Liu. support partial scan for analyze command Key: HIVE-3958 URL: https://issues.apache.org/jira/browse/HIVE-3958 Project: Hive Issue Type: Improvement Reporter: Gang Tim Liu Assignee: Gang Tim Liu analyze commands allows us to collect statistics on existing tables/partitions. It works great but might be slow since it scans all files. There are 2 ways to speed it up: 1. collect stats without file scan. It may not collect all stats but good and fast enough for use case. HIVE-3917 addresses it 2. collect stats via partial file scan. It doesn't scan all content of files but part of it to get file metadata. some examples are https://cwiki.apache.org/Hive/rcfilecat.html for RCFile, ORC ( HIVE-3874 ) and HFile of Hbase This jira is targeted to address the #2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3958) support partial scan for analyze command
[ https://issues.apache.org/jira/browse/HIVE-3958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Tim Liu updated HIVE-3958: --- Description: analyze commands allows us to collect statistics on existing tables/partitions. It works great but might be slow since it scans all files. There are 2 ways to speed it up: 1. collect stats without file scan. It may not collect all stats but good and fast enough for use case. HIVE-3917 addresses it 2. collect stats via partial file scan. It doesn't scan all content of files but part of it to get file metadata. some examples are https://cwiki.apache.org/Hive/rcfilecat.html for RCFile, ORC ( HIVE-3874 ) and HFile of Hbase This jira is targeted to address the #2. More specifically RCFile format. was: analyze commands allows us to collect statistics on existing tables/partitions. It works great but might be slow since it scans all files. There are 2 ways to speed it up: 1. collect stats without file scan. It may not collect all stats but good and fast enough for use case. HIVE-3917 addresses it 2. collect stats via partial file scan. It doesn't scan all content of files but part of it to get file metadata. some examples are https://cwiki.apache.org/Hive/rcfilecat.html for RCFile, ORC ( HIVE-3874 ) and HFile of Hbase This jira is targeted to address the #2 support partial scan for analyze command Key: HIVE-3958 URL: https://issues.apache.org/jira/browse/HIVE-3958 Project: Hive Issue Type: Improvement Reporter: Gang Tim Liu Assignee: Gang Tim Liu analyze commands allows us to collect statistics on existing tables/partitions. It works great but might be slow since it scans all files. There are 2 ways to speed it up: 1. collect stats without file scan. It may not collect all stats but good and fast enough for use case. HIVE-3917 addresses it 2. collect stats via partial file scan. It doesn't scan all content of files but part of it to get file metadata. some examples are https://cwiki.apache.org/Hive/rcfilecat.html for RCFile, ORC ( HIVE-3874 ) and HFile of Hbase This jira is targeted to address the #2. More specifically RCFile format. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3958) support partial scan for analyze command
[ https://issues.apache.org/jira/browse/HIVE-3958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599050#comment-13599050 ] Gang Tim Liu commented on HIVE-3958: We will focus on using map-reduce to calculate rawdatasize faster for RCFile. Instead of reading every byte, we read block header and aggregate them for a table/partition. support partial scan for analyze command Key: HIVE-3958 URL: https://issues.apache.org/jira/browse/HIVE-3958 Project: Hive Issue Type: Improvement Reporter: Gang Tim Liu Assignee: Gang Tim Liu analyze commands allows us to collect statistics on existing tables/partitions. It works great but might be slow since it scans all files. There are 2 ways to speed it up: 1. collect stats without file scan. It may not collect all stats but good and fast enough for use case. HIVE-3917 addresses it 2. collect stats via partial file scan. It doesn't scan all content of files but part of it to get file metadata. some examples are https://cwiki.apache.org/Hive/rcfilecat.html for RCFile, ORC ( HIVE-3874 ) and HFile of Hbase This jira is targeted to address the #2. More specifically RCFile format. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4094) decimal_3.q decimal_serde.q fail on hadoop 2
[ https://issues.apache.org/jira/browse/HIVE-4094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599071#comment-13599071 ] Ashutosh Chauhan commented on HIVE-4094: +1 decimal_3.q decimal_serde.q fail on hadoop 2 -- Key: HIVE-4094 URL: https://issues.apache.org/jira/browse/HIVE-4094 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-4094.patch Some of the decimal unit tests fail on hadoop 2. The reason is unspecified order in some of the queries. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4094) decimal_3.q decimal_serde.q fail on hadoop 2
[ https://issues.apache.org/jira/browse/HIVE-4094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-4094: --- Resolution: Fixed Fix Version/s: 0.11.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks, Gunther! decimal_3.q decimal_serde.q fail on hadoop 2 -- Key: HIVE-4094 URL: https://issues.apache.org/jira/browse/HIVE-4094 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: 0.11.0 Attachments: HIVE-4094.patch Some of the decimal unit tests fail on hadoop 2. The reason is unspecified order in some of the queries. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4096) problem in hive.map.groupby.sorted with distincts
[ https://issues.apache.org/jira/browse/HIVE-4096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599089#comment-13599089 ] Kevin Wilfong commented on HIVE-4096: - +1 problem in hive.map.groupby.sorted with distincts - Key: HIVE-4096 URL: https://issues.apache.org/jira/browse/HIVE-4096 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Namit Jain Assignee: Namit Jain Attachments: hive.4096.1.patch set hive.enforce.bucketing = true; set hive.enforce.sorting = true; set hive.exec.reducers.max = 10; set hive.map.groupby.sorted=true; CREATE TABLE T1(key STRING, val STRING) PARTITIONED BY (ds string) CLUSTERED BY (key) SORTED BY (key) INTO 2 BUCKETS STORED AS TEXTFILE; LOAD DATA LOCAL INPATH '../data/files/T1.txt' INTO TABLE T1 PARTITION (ds='1'); -- perform an insert to make sure there are 2 files INSERT OVERWRITE TABLE T1 PARTITION (ds='1') select key, val from T1 where ds = '1'; CREATE TABLE outputTbl1(cnt INT); -- The plan should be converted to a map-side group by, since the -- sorting columns and grouping columns match, and all the bucketing columns -- are part of sorting columns EXPLAIN select count(distinct key) from T1; select count(distinct key) from T1; explain INSERT OVERWRITE TABLE outputTbl1 select count(distinct key) from T1; INSERT OVERWRITE TABLE outputTbl1 select count(distinct key) from T1; SELECT * FROM outputTbl1; DROP TABLE T1; The above query gives wrong results -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4045) Modify PreDropPartitionEvent to pass Table parameter
[ https://issues.apache.org/jira/browse/HIVE-4045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-4045: Resolution: Fixed Status: Resolved (was: Patch Available) Committed, thanks Li. Modify PreDropPartitionEvent to pass Table parameter Key: HIVE-4045 URL: https://issues.apache.org/jira/browse/HIVE-4045 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Li Yang Assignee: Li Yang Priority: Minor Attachments: HIVE-4045.1.patch.txt, HIVE-4045.2.patch.txt MetaStorePreEventListener which implements onEvent(PreEventContext context) sometimes needs to access Table properties when PreDropPartitionEvent is listened to. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4045) Modify PreDropPartitionEvent to pass Table parameter
[ https://issues.apache.org/jira/browse/HIVE-4045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599175#comment-13599175 ] Li Yang commented on HIVE-4045: --- Thanks Kevin! Modify PreDropPartitionEvent to pass Table parameter Key: HIVE-4045 URL: https://issues.apache.org/jira/browse/HIVE-4045 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Li Yang Assignee: Li Yang Priority: Minor Attachments: HIVE-4045.1.patch.txt, HIVE-4045.2.patch.txt MetaStorePreEventListener which implements onEvent(PreEventContext context) sometimes needs to access Table properties when PreDropPartitionEvent is listened to. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3985) Update new UDAFs introduced for Windowing to work with new Decimal Type
[ https://issues.apache.org/jira/browse/HIVE-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599216#comment-13599216 ] Ashutosh Chauhan commented on HIVE-3985: Patch looks good. However because of HIVE-4140 you need to update the test case to have alias after defining over clause. Sorry about that. Update new UDAFs introduced for Windowing to work with new Decimal Type --- Key: HIVE-3985 URL: https://issues.apache.org/jira/browse/HIVE-3985 Project: Hive Issue Type: Bug Components: PTF-Windowing Reporter: Harish Butani Assignee: Brock Noland Attachments: HIVE-3985-0.patch, HIVE-3985-1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3985) Update new UDAFs introduced for Windowing to work with new Decimal Type
[ https://issues.apache.org/jira/browse/HIVE-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599220#comment-13599220 ] Brock Noland commented on HIVE-3985: OK, will do! Update new UDAFs introduced for Windowing to work with new Decimal Type --- Key: HIVE-3985 URL: https://issues.apache.org/jira/browse/HIVE-3985 Project: Hive Issue Type: Bug Components: PTF-Windowing Reporter: Harish Butani Assignee: Brock Noland Attachments: HIVE-3985-0.patch, HIVE-3985-1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4142) Remove inferring partition specification behavior
[ https://issues.apache.org/jira/browse/HIVE-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-4142: -- Attachment: HIVE-4142.D9285.1.patch hbutani requested code review of HIVE-4142 [jira] Remove inferring partition specification behavior. Reviewers: JIRA, ashutoshc remove behavior that infers partitioning Remove behavior that: implicitly sets the partition specification of a UDAF this interferes with: the user wanting to apply a UDAF on the entire dataset is ambiguous once we support multiple partitions in 1 query. TEST PLAN updated tests REVISION DETAIL https://reviews.facebook.net/D9285 AFFECTED FILES data/files/flights_tiny.txt data/files/part.rc data/files/part.seq ql/src/java/org/apache/hadoop/hive/ql/parse/PTFTranslator.java ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingSpec.java ql/src/test/queries/clientpositive/leadlag.q ql/src/test/queries/clientpositive/leadlag_queries.q ql/src/test/queries/clientpositive/ptf.q ql/src/test/queries/clientpositive/ptf_general_queries.q ql/src/test/queries/clientpositive/ptf_rcfile.q ql/src/test/queries/clientpositive/ptf_seqfile.q ql/src/test/queries/clientpositive/windowing.q ql/src/test/queries/clientpositive/windowing_columnPruning.q ql/src/test/queries/clientpositive/windowing_expressions.q ql/src/test/results/clientpositive/leadlag.q.out ql/src/test/results/clientpositive/leadlag_queries.q.out ql/src/test/results/clientpositive/ptf.q.out ql/src/test/results/clientpositive/ptf_general_queries.q.out ql/src/test/results/clientpositive/ptf_rcfile.q.out ql/src/test/results/clientpositive/ptf_seqfile.q.out ql/src/test/results/clientpositive/windowing.q.out ql/src/test/results/clientpositive/windowing_columnPruning.q.out ql/src/test/results/clientpositive/windowing_expressions.q.out MANAGE HERALD RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/22359/ To: JIRA, ashutoshc, hbutani Remove inferring partition specification behavior - Key: HIVE-4142 URL: https://issues.apache.org/jira/browse/HIVE-4142 Project: Hive Issue Type: Bug Components: PTF-Windowing Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-4142.D9285.1.patch Remove behavior that: - implicitly sets the partition specification of a UDAF this interferes with: - the user wanting to apply a UDAF on the entire dataset - is ambiguous once we support multiple partitions in 1 query. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4142) Remove inferring partition specification behavior
[
https://issues.apache.org/jira/browse/HIVE-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599251#comment-13599251
]
Harish Butani commented on HIVE-4142:
-
Ashutosh, Prajakta, Brock: please take a look.
Had to revise a good portion of the tests.
All the tests have the same result, except 2: ptf.q: q11 and windowing.q q10
Both these have to do with support for having with windowing.
Now that a Window Function has to explicitly have a Window Spec (unless it is
the partition by constant case) it is harder to associate a fn invocation in
having with a UDAF with an over clause. You cannot say:
{noformat}
having rank() over(..) 5
{noformat}
Will address this in another Jira; probably remove the code that tries to
associate a having clause with a WindowingSpec.
Remove inferring partition specification behavior
-
Key: HIVE-4142
URL: https://issues.apache.org/jira/browse/HIVE-4142
Project: Hive
Issue Type: Bug
Components: PTF-Windowing
Reporter: Harish Butani
Assignee: Harish Butani
Attachments: HIVE-4142.D9285.1.patch
Remove behavior that:
- implicitly sets the partition specification of a UDAF
this interferes with:
- the user wanting to apply a UDAF on the entire dataset
- is ambiguous once we support multiple partitions in 1 query.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4148) Cleanup aisle ivy
Gunther Hagleitner created HIVE-4148: Summary: Cleanup aisle ivy Key: HIVE-4148 URL: https://issues.apache.org/jira/browse/HIVE-4148 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Lot's of duplicated dependencies in the modules' ivy configs. Makes compiling slow and maintenance hard. This patch cleans up these dependencies. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4148) Cleanup aisle ivy
[ https://issues.apache.org/jira/browse/HIVE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599279#comment-13599279 ] Ashutosh Chauhan commented on HIVE-4148: A pure-red diff! Nothing better than removing code. Thanks Gunther for doing this. Apart from refactoring this will help in some speedup in compilation as well. +1 Running tests. Will commit if tests pass. Cleanup aisle ivy --- Key: HIVE-4148 URL: https://issues.apache.org/jira/browse/HIVE-4148 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-4148.patch Lot's of duplicated dependencies in the modules' ivy configs. Makes compiling slow and maintenance hard. This patch cleans up these dependencies. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4149) wrong results big outer joins with array of ints
Namit Jain created HIVE-4149: Summary: wrong results big outer joins with array of ints Key: HIVE-4149 URL: https://issues.apache.org/jira/browse/HIVE-4149 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Namit Jain Attachments: hive.4149.1.patch Consider the following query: create table tinyA(a bigint, b bigint) stored as textfile; create table tinyB(a bigint, bList arrayint) stored as textfile; load data local inpath '../data/files/tiny_a' into table tinyA; load data local inpath '../data/files/tiny_b' into table tinyB; select * from tinyA; select * from tinyB; select tinyB.a, tinyB.bList from tinyB full outer join tinyA on tinyB.a = tinyA.a; The results are wrong -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4149) wrong results big outer joins with array of ints
[ https://issues.apache.org/jira/browse/HIVE-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4149: - Attachment: hive.4149.1.patch wrong results big outer joins with array of ints Key: HIVE-4149 URL: https://issues.apache.org/jira/browse/HIVE-4149 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Namit Jain Attachments: hive.4149.1.patch Consider the following query: create table tinyA(a bigint, b bigint) stored as textfile; create table tinyB(a bigint, bList arrayint) stored as textfile; load data local inpath '../data/files/tiny_a' into table tinyA; load data local inpath '../data/files/tiny_b' into table tinyB; select * from tinyA; select * from tinyB; select tinyB.a, tinyB.bList from tinyB full outer join tinyA on tinyB.a = tinyA.a; The results are wrong -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4149) wrong results big outer joins with array of ints
[ https://issues.apache.org/jira/browse/HIVE-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599320#comment-13599320 ] Namit Jain commented on HIVE-4149: -- The patch with the test is attached wrong results big outer joins with array of ints Key: HIVE-4149 URL: https://issues.apache.org/jira/browse/HIVE-4149 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Namit Jain Attachments: hive.4149.1.patch Consider the following query: create table tinyA(a bigint, b bigint) stored as textfile; create table tinyB(a bigint, bList arrayint) stored as textfile; load data local inpath '../data/files/tiny_a' into table tinyA; load data local inpath '../data/files/tiny_b' into table tinyB; select * from tinyA; select * from tinyB; select tinyB.a, tinyB.bList from tinyB full outer join tinyA on tinyB.a = tinyA.a; The results are wrong -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4142) Remove inferring partition specification behavior
[ https://issues.apache.org/jira/browse/HIVE-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599336#comment-13599336 ] Ashutosh Chauhan commented on HIVE-4142: Thinking more about it, I think we should just outright remove support for this. No other systems supports this. And this is one of the cases where we are adding semantics (allowing having without a group-by) which will be a deviation from a standard. Further, we are not getting it for free and have to write and maintain more code for it. Remove inferring partition specification behavior - Key: HIVE-4142 URL: https://issues.apache.org/jira/browse/HIVE-4142 Project: Hive Issue Type: Bug Components: PTF-Windowing Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-4142.D9285.1.patch Remove behavior that: - implicitly sets the partition specification of a UDAF this interferes with: - the user wanting to apply a UDAF on the entire dataset - is ambiguous once we support multiple partitions in 1 query. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3862) testHBaseNegativeCliDriver_cascade_dbdrop fails on hadoop-1
[ https://issues.apache.org/jira/browse/HIVE-3862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-3862: - Summary: testHBaseNegativeCliDriver_cascade_dbdrop fails on hadoop-1 (was: testNegativeCliDriver_cascade_dbdrop fails on hadoop-1) testHBaseNegativeCliDriver_cascade_dbdrop fails on hadoop-1 --- Key: HIVE-3862 URL: https://issues.apache.org/jira/browse/HIVE-3862 Project: Hive Issue Type: Bug Affects Versions: 0.10.0 Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-3862.patch Actually functionality is working correctly, but incorrect include/exclude macro make cause the wrong query file to be run. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3862) testHBaseNegativeCliDriver_cascade_dbdrop fails on hadoop-1
[ https://issues.apache.org/jira/browse/HIVE-3862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-3862: - Attachment: HIVE-3862.1.patch testHBaseNegativeCliDriver_cascade_dbdrop fails on hadoop-1 --- Key: HIVE-3862 URL: https://issues.apache.org/jira/browse/HIVE-3862 Project: Hive Issue Type: Bug Affects Versions: 0.10.0 Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-3862.1.patch, HIVE-3862.patch Actually functionality is working correctly, but incorrect include/exclude macro make cause the wrong query file to be run. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3862) testHBaseNegativeCliDriver_cascade_dbdrop fails on hadoop-1
[ https://issues.apache.org/jira/browse/HIVE-3862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-3862: - Status: Open (was: Patch Available) testHBaseNegativeCliDriver_cascade_dbdrop fails on hadoop-1 --- Key: HIVE-3862 URL: https://issues.apache.org/jira/browse/HIVE-3862 Project: Hive Issue Type: Bug Affects Versions: 0.10.0 Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-3862.1.patch, HIVE-3862.patch Actually functionality is working correctly, but incorrect include/exclude macro make cause the wrong query file to be run. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3862) testHBaseNegativeCliDriver_cascade_dbdrop fails on hadoop-1
[ https://issues.apache.org/jira/browse/HIVE-3862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-3862: - Status: Patch Available (was: Open) testHBaseNegativeCliDriver_cascade_dbdrop fails on hadoop-1 --- Key: HIVE-3862 URL: https://issues.apache.org/jira/browse/HIVE-3862 Project: Hive Issue Type: Bug Affects Versions: 0.10.0 Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-3862.1.patch, HIVE-3862.patch Actually functionality is working correctly, but incorrect include/exclude macro make cause the wrong query file to be run. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3862) testHBaseNegativeCliDriver_cascade_dbdrop fails on hadoop-1
[ https://issues.apache.org/jira/browse/HIVE-3862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599421#comment-13599421 ] Gunther Hagleitner commented on HIVE-3862: -- Addressed Ashutosh' comment in .1. testHBaseNegativeCliDriver_cascade_dbdrop fails on hadoop-1 --- Key: HIVE-3862 URL: https://issues.apache.org/jira/browse/HIVE-3862 Project: Hive Issue Type: Bug Affects Versions: 0.10.0 Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-3862.1.patch, HIVE-3862.patch Actually functionality is working correctly, but incorrect include/exclude macro make cause the wrong query file to be run. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4150) optimize queries like 'select count(1) from T where conditions on partition columns'
Namit Jain created HIVE-4150: Summary: optimize queries like 'select count(1) from T where conditions on partition columns' Key: HIVE-4150 URL: https://issues.apache.org/jira/browse/HIVE-4150 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Namit Jain If accurate stats are available in the metastore, they should be used to optimize the above query. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4151) HiveProfiler NPE with ScriptOperator
Pamela Vagata created HIVE-4151: --- Summary: HiveProfiler NPE with ScriptOperator Key: HIVE-4151 URL: https://issues.apache.org/jira/browse/HIVE-4151 Project: Hive Issue Type: Bug Reporter: Pamela Vagata Priority: Minor -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Bergeron updated HIVE-3454: Labels: newbie newdev patch (was: ) Status: Patch Available (was: Open) Using java.sql.Date instead of java.util.Date because java.sql.Date implements a timezone agnostic method of representing timestamps (always UTC, which is what a unix epoch timestamp is defined as) Problem with CAST(BIGINT as TIMESTAMP) -- Key: HIVE-3454 URL: https://issues.apache.org/jira/browse/HIVE-3454 Project: Hive Issue Type: Bug Components: Types, UDF Affects Versions: 0.9.0, 0.8.1, 0.8.0 Reporter: Ryan Harris Labels: patch, newbie, newdev Attachments: HIVE-3454.1.patch.txt Ran into an issue while working with timestamp conversion. CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current time from the BIGINT returned by unix_timestamp() Instead, however, a 1970-01-16 timestamp is returned. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Bergeron updated HIVE-3454: Attachment: HIVE-3454.1.patch.txt Problem with CAST(BIGINT as TIMESTAMP) -- Key: HIVE-3454 URL: https://issues.apache.org/jira/browse/HIVE-3454 Project: Hive Issue Type: Bug Components: Types, UDF Affects Versions: 0.8.0, 0.8.1, 0.9.0 Reporter: Ryan Harris Labels: newbie, newdev, patch Attachments: HIVE-3454.1.patch.txt Ran into an issue while working with timestamp conversion. CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current time from the BIGINT returned by unix_timestamp() Instead, however, a 1970-01-16 timestamp is returned. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3717) Hive won't compile with -Dhadoop.mr.rev=20S
[
https://issues.apache.org/jira/browse/HIVE-3717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gunther Hagleitner updated HIVE-3717:
-
Attachment: HIVE-3717.5.patch
Hive won't compile with -Dhadoop.mr.rev=20S
---
Key: HIVE-3717
URL: https://issues.apache.org/jira/browse/HIVE-3717
Project: Hive
Issue Type: Bug
Components: Build Infrastructure, Shims
Affects Versions: 0.10.0
Reporter: Gunther Hagleitner
Assignee: Gunther Hagleitner
Attachments: HIVE-3717.1-1410543.txt, HIVE-3717.2.patch,
HIVE-3717.3.patch, HIVE-3717.4.patch, HIVE-3717.5.patch
ant -Dhadoop.mr.rev=20S clean package
fails with:
{noformat}
compile:
[echo] Project: ql
[javac] Compiling 744 source files to /root/hive/build/ql/classes
[javac]
/root/hive/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFJson.java:67: cannot
find symbol
[javac] symbol : variable ALLOW_UNQUOTED_CONTROL_CHARS
[javac] location: class org.codehaus.jackson.JsonParser.Feature
[javac] JSON_FACTORY.enable(Feature.ALLOW_UNQUOTED_CONTROL_CHARS);
[javac]^
[javac]
/root/hive/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFJson.java:158: cannot
find symbol
[javac] symbol : method writeValueAsString(java.lang.Object)
[javac] location: class org.codehaus.jackson.map.ObjectMapper
[javac] result.set(MAPPER.writeValueAsString(extractObject));
[javac] ^
[javac]
/root/hive/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFJSONTuple.java:59:
cannot find symbol
[javac] symbol : variable ALLOW_UNQUOTED_CONTROL_CHARS
[javac] location: class org.codehaus.jackson.JsonParser.Feature
[javac] JSON_FACTORY.enable(Feature.ALLOW_UNQUOTED_CONTROL_CHARS);
[javac]^
[javac]
/root/hive/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFJSONTuple.java:189:
cannot find symbol
[javac] symbol : method writeValueAsString(java.lang.Object)
[javac] location: class org.codehaus.jackson.map.ObjectMapper
[javac]
retCols[i].set(MAPPER.writeValueAsString(extractObject));
[javac]^
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 4 errors
{noformat}
According to https://issues.apache.org/jira/browse/HADOOP-7470 hadoop 1.x has
been upgraded to jackson 1.8.8 but the POM file still specifies jackson 1.0.1
which doesn't work for hive (doesn't have the ALLOW_UNQUOTED_CONTROL_CHARS).
The POM for hadoop 2.0.0-alpha (-Dhadoop.mr.rev=23) has the right dependency,
hadoop 0.20.2 (-Dhadoop.mr.rev=20) doesn't depend on jackson.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2340) optimize orderby followed by a groupby
[ https://issues.apache.org/jira/browse/HIVE-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-2340: Status: Patch Available (was: Open) optimize orderby followed by a groupby -- Key: HIVE-2340 URL: https://issues.apache.org/jira/browse/HIVE-2340 Project: Hive Issue Type: Sub-task Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Labels: perfomance Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-2340.D1209.1.patch, ASF.LICENSE.NOT.GRANTED--HIVE-2340.D1209.2.patch, ASF.LICENSE.NOT.GRANTED--HIVE-2340.D1209.3.patch, ASF.LICENSE.NOT.GRANTED--HIVE-2340.D1209.4.patch, ASF.LICENSE.NOT.GRANTED--HIVE-2340.D1209.5.patch, HIVE-2340.12.patch, HIVE-2340.13.patch, HIVE-2340.14.patch, HIVE-2340.1.patch.txt, HIVE-2340.D1209.10.patch, HIVE-2340.D1209.11.patch, HIVE-2340.D1209.12.patch, HIVE-2340.D1209.13.patch, HIVE-2340.D1209.6.patch, HIVE-2340.D1209.7.patch, HIVE-2340.D1209.8.patch, HIVE-2340.D1209.9.patch, testclidriver.txt Before implementing optimizer for JOIN-GBY, try to implement RS-GBY optimizer(cluster-by following group-by). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2340) optimize orderby followed by a groupby
[ https://issues.apache.org/jira/browse/HIVE-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2340: -- Attachment: HIVE-2340.D1209.14.patch navis updated the revision HIVE-2340 [jira] optimize orderby followed by a groupby. Rebased to trunk Reviewers: JIRA REVISION DETAIL https://reviews.facebook.net/D1209 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D1209?vs=27729id=29571#toc AFFECTED FILES common/src/java/org/apache/hadoop/hive/conf/HiveConf.java conf/hive-default.xml.template ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/NonBlockingOpDeDupProc.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkDeDuplication.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinResolver.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SkewJoinProcFactory.java ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java ql/src/test/queries/clientpositive/auto_join26.q ql/src/test/queries/clientpositive/groupby_distinct_samekey.q ql/src/test/queries/clientpositive/reduce_deduplicate.q ql/src/test/queries/clientpositive/reduce_deduplicate_extended.q ql/src/test/results/clientpositive/cluster.q.out ql/src/test/results/clientpositive/groupby2.q.out ql/src/test/results/clientpositive/groupby2_map_skew.q.out ql/src/test/results/clientpositive/groupby_cube1.q.out ql/src/test/results/clientpositive/groupby_distinct_samekey.q.out ql/src/test/results/clientpositive/groupby_rollup1.q.out ql/src/test/results/clientpositive/index_bitmap3.q.out ql/src/test/results/clientpositive/index_bitmap_auto.q.out ql/src/test/results/clientpositive/infer_bucket_sort.q.out ql/src/test/results/clientpositive/ppd2.q.out ql/src/test/results/clientpositive/ppd_gby_join.q.out ql/src/test/results/clientpositive/reduce_deduplicate_extended.q.out ql/src/test/results/clientpositive/semijoin.q.out ql/src/test/results/clientpositive/union24.q.out ql/src/test/results/compiler/plan/join1.q.xml ql/src/test/results/compiler/plan/join2.q.xml ql/src/test/results/compiler/plan/join3.q.xml To: JIRA, navis Cc: hagleitn, njain optimize orderby followed by a groupby -- Key: HIVE-2340 URL: https://issues.apache.org/jira/browse/HIVE-2340 Project: Hive Issue Type: Sub-task Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Labels: perfomance Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-2340.D1209.1.patch, ASF.LICENSE.NOT.GRANTED--HIVE-2340.D1209.2.patch, ASF.LICENSE.NOT.GRANTED--HIVE-2340.D1209.3.patch, ASF.LICENSE.NOT.GRANTED--HIVE-2340.D1209.4.patch, ASF.LICENSE.NOT.GRANTED--HIVE-2340.D1209.5.patch, HIVE-2340.12.patch, HIVE-2340.13.patch, HIVE-2340.14.patch, HIVE-2340.1.patch.txt, HIVE-2340.D1209.10.patch, HIVE-2340.D1209.11.patch, HIVE-2340.D1209.12.patch, HIVE-2340.D1209.13.patch, HIVE-2340.D1209.14.patch, HIVE-2340.D1209.6.patch, HIVE-2340.D1209.7.patch, HIVE-2340.D1209.8.patch, HIVE-2340.D1209.9.patch, testclidriver.txt Before implementing optimizer for JOIN-GBY, try to implement RS-GBY optimizer(cluster-by following group-by). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4142) Remove inferring partition specification behavior
[ https://issues.apache.org/jira/browse/HIVE-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599580#comment-13599580 ] Ashutosh Chauhan commented on HIVE-4142: You have already removed functions moveaggregationExprsToWindowingSpec(), ensureWindowingSpecOnQB() etc., so as far as I see support has already been removed. Is there anything more we need to do here? Patch looks good. +1 Remove inferring partition specification behavior - Key: HIVE-4142 URL: https://issues.apache.org/jira/browse/HIVE-4142 Project: Hive Issue Type: Bug Components: PTF-Windowing Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-4142.D9285.1.patch Remove behavior that: - implicitly sets the partition specification of a UDAF this interferes with: - the user wanting to apply a UDAF on the entire dataset - is ambiguous once we support multiple partitions in 1 query. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4143) Incorrect column mappings with over clause
[
https://issues.apache.org/jira/browse/HIVE-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-4143:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to branch. Thanks Harish for the review.
Incorrect column mappings with over clause
--
Key: HIVE-4143
URL: https://issues.apache.org/jira/browse/HIVE-4143
Project: Hive
Issue Type: Bug
Components: PTF-Windowing
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
Following query generates 3 MR job and fails in reducer initialization phase
of third job.
{code}
select ts, dec, rnk
from
(select ts, dec,
rank() over (partition by ts order by item_total desc) as rnk
from
(select other.ts, other.dec, sum(other.i) as item_total
from over10k other
join over10k on (other.b = over10k.b)
group by other.ts, other.dec
) item_sales
) item_rank
where rnk = 3;
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3968) Enhance logging in TableAccessInfo
[ https://issues.apache.org/jira/browse/HIVE-3968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-3968: Attachment: HIVE-3968.5.patch.txt Enhance logging in TableAccessInfo -- Key: HIVE-3968 URL: https://issues.apache.org/jira/browse/HIVE-3968 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-3968.1.patch.txt, HIVE-3968.2.patch.txt, HIVE-3968.3.patch.txt, HIVE-3968.4.patch.txt, HIVE-3968.5.patch.txt Based on what is currently available in the TableAccessInfo we can infer when it would be a good idea to add bucketing/sorting metadata for tables. However, we can't easily tell if we're already getting the benefits of bucketing/sorting. This information can be improved by a) storing the input table/partition objects so that we can tell if the tables/partitions are already bucketed/sorted b) running the TableAccessAnalyzer after the logical optimizer, so that we can tell from the operators whether or not we are already getting benefits (bucketed/sort merge map joins or map group bys) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4152) Lock owner should be allowed to acquire lock again on EmbeddedLockManager
Navis created HIVE-4152: --- Summary: Lock owner should be allowed to acquire lock again on EmbeddedLockManager Key: HIVE-4152 URL: https://issues.apache.org/jira/browse/HIVE-4152 Project: Hive Issue Type: Bug Components: Locking Reporter: Navis Assignee: Navis Priority: Trivial If a query owns lock on a path, it should be allowed to acquire lock on it again with success. Current EmbeddedLockManager returns null in this situation and makes some query to wait infinitely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4152) Lock owner should be allowed to acquire lock again on EmbeddedLockManager
[ https://issues.apache.org/jira/browse/HIVE-4152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-4152: -- Attachment: HIVE-4152.D9303.1.patch navis requested code review of HIVE-4152 [jira] Lock owner should be allowed to acquire lock again on EmbeddedLockManager. Reviewers: JIRA HIVE-4152 Lock owner should be allowed to acquire lock again on EmbeddedLockManager If a query owns lock on a path, it should be allowed to acquire lock on it again with success. Current EmbeddedLockManager returns null in this situation and makes some query to wait infinitely. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D9303 AFFECTED FILES common/src/java/org/apache/hadoop/hive/conf/HiveConf.java conf/hive-default.xml.template ql/src/java/org/apache/hadoop/hive/ql/lockmgr/EmbeddedLockManager.java ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestEmbeddedLockManager.java MANAGE HERALD RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/22395/ To: JIRA, navis Lock owner should be allowed to acquire lock again on EmbeddedLockManager - Key: HIVE-4152 URL: https://issues.apache.org/jira/browse/HIVE-4152 Project: Hive Issue Type: Bug Components: Locking Reporter: Navis Assignee: Navis Priority: Trivial Attachments: HIVE-4152.D9303.1.patch If a query owns lock on a path, it should be allowed to acquire lock on it again with success. Current EmbeddedLockManager returns null in this situation and makes some query to wait infinitely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4152) Lock owner should be allowed to acquire lock again on EmbeddedLockManager
[ https://issues.apache.org/jira/browse/HIVE-4152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-4152: Status: Patch Available (was: Open) Lock owner should be allowed to acquire lock again on EmbeddedLockManager - Key: HIVE-4152 URL: https://issues.apache.org/jira/browse/HIVE-4152 Project: Hive Issue Type: Bug Components: Locking Reporter: Navis Assignee: Navis Priority: Trivial Attachments: HIVE-4152.D9303.1.patch If a query owns lock on a path, it should be allowed to acquire lock on it again with success. Current EmbeddedLockManager returns null in this situation and makes some query to wait infinitely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3862) testHBaseNegativeCliDriver_cascade_dbdrop fails on hadoop-1
[ https://issues.apache.org/jira/browse/HIVE-3862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599678#comment-13599678 ] Ashutosh Chauhan commented on HIVE-3862: +1 testHBaseNegativeCliDriver_cascade_dbdrop fails on hadoop-1 --- Key: HIVE-3862 URL: https://issues.apache.org/jira/browse/HIVE-3862 Project: Hive Issue Type: Bug Affects Versions: 0.10.0 Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-3862.1.patch, HIVE-3862.patch Actually functionality is working correctly, but incorrect include/exclude macro make cause the wrong query file to be run. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4108) Allow over() clause to contain an order by with no partition by
[
https://issues.apache.org/jira/browse/HIVE-4108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HIVE-4108:
--
Attachment: HIVE-4108.D9309.1.patch
hbutani requested code review of HIVE-4108 [jira] Allow over() clause to
contain an order by with no partition by.
Reviewers: JIRA, ashutoshc
allow over clause with only an order spec
HIVE-4073 allows over() to be called with no partition by and no order by. We
should allow only an order by.
From the review of HIVE-4073:
Ashutosh
Can you also add following test. This should also work.
select p_name, p_retailprice,
avg(p_retailprice) over(order by p_name)
from part
partition by p_name;
Harish
This test will not work (:
The grammar needs to be changed so:
partitioningSpec
@init { msgs.push(partitioningSpec clause); }
@after { msgs.pop(); }
:
partitionByClause orderByClause? - ^(TOK_PARTITIONINGSPEC partitionByClause
orderByClause?) |
orderByClause - ^(TOK_PARTITIONINGSPEC orderByClause) |
distributeByClause sortByClause? - ^(TOK_PARTITIONINGSPEC distributeByClause
sortByClause?) |
sortByClause? - ^(TOK_PARTITIONINGSPEC sortByClause) |
clusterByClause - ^(TOK_PARTITIONINGSPEC clusterByClause)
;
And the SemanticAnalyzer::processPTFPartitionSpec has to handle this shape of
the AST Tree. The PTFTranslator also needs changes. Do this as another Jira
TEST PLAN
added new tests with over clause containing only order spec.
REVISION DETAIL
https://reviews.facebook.net/D9309
AFFECTED FILES
ql/src/java/org/apache/hadoop/hive/ql/parse/FromClauseParser.g
ql/src/java/org/apache/hadoop/hive/ql/parse/PTFTranslator.java
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
ql/src/test/queries/clientpositive/ptf_npath.q
ql/src/test/queries/clientpositive/windowing_windowspec.q
ql/src/test/results/clientpositive/ptf_npath.q.out
ql/src/test/results/clientpositive/windowing_windowspec.q.out
MANAGE HERALD RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/22401/
To: JIRA, ashutoshc, hbutani
Allow over() clause to contain an order by with no partition by
---
Key: HIVE-4108
URL: https://issues.apache.org/jira/browse/HIVE-4108
Project: Hive
Issue Type: Bug
Components: PTF-Windowing
Reporter: Brock Noland
Assignee: Harish Butani
Attachments: HIVE-4108.D9309.1.patch
HIVE-4073 allows over() to be called with no partition by and no order by. We
should allow only an order by.
From the review of HIVE-4073:
Ashutosh
{noformat}
Can you also add following test. This should also work.
select p_name, p_retailprice,
avg(p_retailprice) over(order by p_name)
from part
partition by p_name;
{noformat}
Harish
{noformat}
This test will not work (:
The grammar needs to be changed so:
partitioningSpec
@init { msgs.push(partitioningSpec clause); }
@after { msgs.pop(); }
:
partitionByClause orderByClause? - ^(TOK_PARTITIONINGSPEC partitionByClause
orderByClause?) |
orderByClause - ^(TOK_PARTITIONINGSPEC orderByClause) |
distributeByClause sortByClause? - ^(TOK_PARTITIONINGSPEC distributeByClause
sortByClause?) |
sortByClause? - ^(TOK_PARTITIONINGSPEC sortByClause) |
clusterByClause - ^(TOK_PARTITIONINGSPEC clusterByClause)
;
And the SemanticAnalyzer::processPTFPartitionSpec has to handle this shape of
the AST Tree. The PTFTranslator also needs changes. Do this as another Jira
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4142) Remove inferring partition specification behavior
[ https://issues.apache.org/jira/browse/HIVE-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599683#comment-13599683 ] Harish Butani commented on HIVE-4142: - Ashutosh, thanks for reviewing. Will add a Jira to remove the having behavior. Remove inferring partition specification behavior - Key: HIVE-4142 URL: https://issues.apache.org/jira/browse/HIVE-4142 Project: Hive Issue Type: Bug Components: PTF-Windowing Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-4142.D9285.1.patch Remove behavior that: - implicitly sets the partition specification of a UDAF this interferes with: - the user wanting to apply a UDAF on the entire dataset - is ambiguous once we support multiple partitions in 1 query. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3862) testHBaseNegativeCliDriver_cascade_dbdrop fails on hadoop-1
[ https://issues.apache.org/jira/browse/HIVE-3862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-3862: --- Resolution: Fixed Fix Version/s: 0.11.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks, Gunther! testHBaseNegativeCliDriver_cascade_dbdrop fails on hadoop-1 --- Key: HIVE-3862 URL: https://issues.apache.org/jira/browse/HIVE-3862 Project: Hive Issue Type: Bug Affects Versions: 0.10.0 Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: 0.11.0 Attachments: HIVE-3862.1.patch, HIVE-3862.patch Actually functionality is working correctly, but incorrect include/exclude macro make cause the wrong query file to be run. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4142) Remove inferring partition specification behavior
[ https://issues.apache.org/jira/browse/HIVE-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599705#comment-13599705 ] Ashutosh Chauhan commented on HIVE-4142: All the +ve test cases passed. But following negative test cases failed: ptf_negative_NoSortNoDistByClause.q ptf_negative_IncompatibleSortClause.q ptf_negative_AggrFuncsWithNoGBYNoPartDef.q Looks like need to update .q.out files for these. Remove inferring partition specification behavior - Key: HIVE-4142 URL: https://issues.apache.org/jira/browse/HIVE-4142 Project: Hive Issue Type: Bug Components: PTF-Windowing Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-4142.D9285.1.patch Remove behavior that: - implicitly sets the partition specification of a UDAF this interferes with: - the user wanting to apply a UDAF on the entire dataset - is ambiguous once we support multiple partitions in 1 query. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4108) Allow over() clause to contain an order by with no partition by
[
https://issues.apache.org/jira/browse/HIVE-4108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599707#comment-13599707
]
Phabricator commented on HIVE-4108:
---
ashutoshc has accepted the revision HIVE-4108 [jira] Allow over() clause to
contain an order by with no partition by.
+1
REVISION DETAIL
https://reviews.facebook.net/D9309
BRANCH
HIVE-4108
ARCANIST PROJECT
hive
To: JIRA, ashutoshc, hbutani
Allow over() clause to contain an order by with no partition by
---
Key: HIVE-4108
URL: https://issues.apache.org/jira/browse/HIVE-4108
Project: Hive
Issue Type: Bug
Components: PTF-Windowing
Reporter: Brock Noland
Assignee: Harish Butani
Attachments: HIVE-4108.D9309.1.patch
HIVE-4073 allows over() to be called with no partition by and no order by. We
should allow only an order by.
From the review of HIVE-4073:
Ashutosh
{noformat}
Can you also add following test. This should also work.
select p_name, p_retailprice,
avg(p_retailprice) over(order by p_name)
from part
partition by p_name;
{noformat}
Harish
{noformat}
This test will not work (:
The grammar needs to be changed so:
partitioningSpec
@init { msgs.push(partitioningSpec clause); }
@after { msgs.pop(); }
:
partitionByClause orderByClause? - ^(TOK_PARTITIONINGSPEC partitionByClause
orderByClause?) |
orderByClause - ^(TOK_PARTITIONINGSPEC orderByClause) |
distributeByClause sortByClause? - ^(TOK_PARTITIONINGSPEC distributeByClause
sortByClause?) |
sortByClause? - ^(TOK_PARTITIONINGSPEC sortByClause) |
clusterByClause - ^(TOK_PARTITIONINGSPEC clusterByClause)
;
And the SemanticAnalyzer::processPTFPartitionSpec has to handle this shape of
the AST Tree. The PTFTranslator also needs changes. Do this as another Jira
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4151) HiveProfiler NPE with ScriptOperator
[ https://issues.apache.org/jira/browse/HIVE-4151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pamela Vagata updated HIVE-4151: Assignee: Pamela Vagata Status: Patch Available (was: Open) HiveProfiler NPE with ScriptOperator Key: HIVE-4151 URL: https://issues.apache.org/jira/browse/HIVE-4151 Project: Hive Issue Type: Bug Reporter: Pamela Vagata Assignee: Pamela Vagata Priority: Minor -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4151) HiveProfiler NPE with ScriptOperator
[ https://issues.apache.org/jira/browse/HIVE-4151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pamela Vagata updated HIVE-4151: Attachment: HIVE-4151.patch.0.txt https://reviews.facebook.net/D9315 HiveProfiler NPE with ScriptOperator Key: HIVE-4151 URL: https://issues.apache.org/jira/browse/HIVE-4151 Project: Hive Issue Type: Bug Reporter: Pamela Vagata Assignee: Pamela Vagata Priority: Minor Attachments: HIVE-4151.patch.0.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4108) Allow over() clause to contain an order by with no partition by
[
https://issues.apache.org/jira/browse/HIVE-4108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599715#comment-13599715
]
Ashutosh Chauhan commented on HIVE-4108:
Test {{windowing_windowspec.q}} failed.
Allow over() clause to contain an order by with no partition by
---
Key: HIVE-4108
URL: https://issues.apache.org/jira/browse/HIVE-4108
Project: Hive
Issue Type: Bug
Components: PTF-Windowing
Reporter: Brock Noland
Assignee: Harish Butani
Attachments: HIVE-4108.D9309.1.patch
HIVE-4073 allows over() to be called with no partition by and no order by. We
should allow only an order by.
From the review of HIVE-4073:
Ashutosh
{noformat}
Can you also add following test. This should also work.
select p_name, p_retailprice,
avg(p_retailprice) over(order by p_name)
from part
partition by p_name;
{noformat}
Harish
{noformat}
This test will not work (:
The grammar needs to be changed so:
partitioningSpec
@init { msgs.push(partitioningSpec clause); }
@after { msgs.pop(); }
:
partitionByClause orderByClause? - ^(TOK_PARTITIONINGSPEC partitionByClause
orderByClause?) |
orderByClause - ^(TOK_PARTITIONINGSPEC orderByClause) |
distributeByClause sortByClause? - ^(TOK_PARTITIONINGSPEC distributeByClause
sortByClause?) |
sortByClause? - ^(TOK_PARTITIONINGSPEC sortByClause) |
clusterByClause - ^(TOK_PARTITIONINGSPEC clusterByClause)
;
And the SemanticAnalyzer::processPTFPartitionSpec has to handle this shape of
the AST Tree. The PTFTranslator also needs changes. Do this as another Jira
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4108) Allow over() clause to contain an order by with no partition by
[
https://issues.apache.org/jira/browse/HIVE-4108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599718#comment-13599718
]
Harish Butani commented on HIVE-4108:
-
this needs the 4142 patch. Sorry forgot to mention this.
Allow over() clause to contain an order by with no partition by
---
Key: HIVE-4108
URL: https://issues.apache.org/jira/browse/HIVE-4108
Project: Hive
Issue Type: Bug
Components: PTF-Windowing
Reporter: Brock Noland
Assignee: Harish Butani
Attachments: HIVE-4108.D9309.1.patch
HIVE-4073 allows over() to be called with no partition by and no order by. We
should allow only an order by.
From the review of HIVE-4073:
Ashutosh
{noformat}
Can you also add following test. This should also work.
select p_name, p_retailprice,
avg(p_retailprice) over(order by p_name)
from part
partition by p_name;
{noformat}
Harish
{noformat}
This test will not work (:
The grammar needs to be changed so:
partitioningSpec
@init { msgs.push(partitioningSpec clause); }
@after { msgs.pop(); }
:
partitionByClause orderByClause? - ^(TOK_PARTITIONINGSPEC partitionByClause
orderByClause?) |
orderByClause - ^(TOK_PARTITIONINGSPEC orderByClause) |
distributeByClause sortByClause? - ^(TOK_PARTITIONINGSPEC distributeByClause
sortByClause?) |
sortByClause? - ^(TOK_PARTITIONINGSPEC sortByClause) |
clusterByClause - ^(TOK_PARTITIONINGSPEC clusterByClause)
;
And the SemanticAnalyzer::processPTFPartitionSpec has to handle this shape of
the AST Tree. The PTFTranslator also needs changes. Do this as another Jira
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4142) Remove inferring partition specification behavior
[ https://issues.apache.org/jira/browse/HIVE-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-4142: -- Attachment: HIVE-4142.D9285.2.patch hbutani updated the revision HIVE-4142 [jira] Remove inferring partition specification behavior. - Merge branch 'ptf' into HIVE-4142 - Merge branch 'ptf' into HIVE-4142 Reviewers: JIRA, ashutoshc REVISION DETAIL https://reviews.facebook.net/D9285 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D9285?vs=29547id=29619#toc AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/parse/PTFTranslator.java ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingSpec.java ql/src/test/queries/clientnegative/ptf_negative_IncompatibleSortClause.q ql/src/test/queries/clientnegative/ptf_negative_NoSortNoDistByClause.q ql/src/test/queries/clientpositive/leadlag.q ql/src/test/queries/clientpositive/leadlag_queries.q ql/src/test/queries/clientpositive/ptf.q ql/src/test/queries/clientpositive/ptf_general_queries.q ql/src/test/queries/clientpositive/ptf_rcfile.q ql/src/test/queries/clientpositive/ptf_seqfile.q ql/src/test/queries/clientpositive/windowing.q ql/src/test/queries/clientpositive/windowing_columnPruning.q ql/src/test/queries/clientpositive/windowing_expressions.q ql/src/test/results/clientnegative/ptf_negative_AggrFuncsWithNoGBYNoPartDef.q.out ql/src/test/results/clientnegative/ptf_negative_NoSortNoDistByClause.q.out ql/src/test/results/clientpositive/leadlag.q.out ql/src/test/results/clientpositive/leadlag_queries.q.out ql/src/test/results/clientpositive/ptf.q.out ql/src/test/results/clientpositive/ptf_general_queries.q.out ql/src/test/results/clientpositive/ptf_rcfile.q.out ql/src/test/results/clientpositive/ptf_seqfile.q.out ql/src/test/results/clientpositive/windowing.q.out ql/src/test/results/clientpositive/windowing_columnPruning.q.out ql/src/test/results/clientpositive/windowing_expressions.q.out To: JIRA, ashutoshc, hbutani Remove inferring partition specification behavior - Key: HIVE-4142 URL: https://issues.apache.org/jira/browse/HIVE-4142 Project: Hive Issue Type: Bug Components: PTF-Windowing Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-4142.D9285.1.patch, HIVE-4142.D9285.2.patch Remove behavior that: - implicitly sets the partition specification of a UDAF this interferes with: - the user wanting to apply a UDAF on the entire dataset - is ambiguous once we support multiple partitions in 1 query. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4142) Remove inferring partition specification behavior
[ https://issues.apache.org/jira/browse/HIVE-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599721#comment-13599721 ] Harish Butani commented on HIVE-4142: - removed ptf_negative_NoSortNoDistByClause.q because this is a valid query now. Remove inferring partition specification behavior - Key: HIVE-4142 URL: https://issues.apache.org/jira/browse/HIVE-4142 Project: Hive Issue Type: Bug Components: PTF-Windowing Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-4142.D9285.1.patch, HIVE-4142.D9285.2.patch Remove behavior that: - implicitly sets the partition specification of a UDAF this interferes with: - the user wanting to apply a UDAF on the entire dataset - is ambiguous once we support multiple partitions in 1 query. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
