[jira] [Updated] (HIVE-16723) Enable configurable MetaStoreSchemaInfo

[Lefty Leverenz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lefty Leverenz updated HIVE-16723:
--
Labels: TODOC3.0  (was: )

> Enable configurable MetaStoreSchemaInfo 
> 
>
> Key: HIVE-16723
> URL: https://issues.apache.org/jira/browse/HIVE-16723
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
>  Labels: TODOC3.0
> Fix For: 3.0.0
>
> Attachments: HIVE-16723.01.patch, HIVE-16723.02.patch, 
> HIVE-16723.03.patch, HIVE-16723.04.patch
>
>
> {{MetaStoreSchemaInfo}} class is used by Schema tool to get the schema 
> version information. In addition to that schema tool invokes its init and 
> upgrade methods to initialize and upgrade the metastore schema. It is 
> possible that there are minor schema changes in the metastore schema which 
> users have to maintain manually. We can potentially enhance schema tool to 
> use a custom MetaStoreSchemaInfo implementation which can be plugged in based 
> on configuration and schematool could run these customizations while running 
> the upgrade and init scripts. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15665) LLAP: OrcFileMetadata objects in cache can impact heap usage

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022349#comment-16022349
 ] 

Hive QA commented on HIVE-15665:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869571/HIVE-15665.01.patch

{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10754 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
 (batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=144)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5411/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5411/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5411/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12869571 - PreCommit-HIVE-Build

> LLAP: OrcFileMetadata objects in cache can impact heap usage
> 
>
> Key: HIVE-15665
> URL: https://issues.apache.org/jira/browse/HIVE-15665
> Project: Hive
>  Issue Type: Improvement
>  Components: llap
>Reporter: Rajesh Balamohan
>Assignee: Sergey Shelukhin
> Attachments: HIVE-15665.01.patch, HIVE-15665.patch
>
>
> OrcFileMetadata internally has filestats, stripestats etc which are allocated 
> in heap. On large data sets, this could have an impact on the heap usage and 
> the memory usage by different executors in LLAP.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16735) Please add that hive.mapred.local.mem must be set in MBs in the documentation

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022339#comment-16022339
 ] 

Lefty Leverenz commented on HIVE-16735:
---

You're right, the doc needs to be fixed so you're not missing anything (except 
the desired fix).  I'm just saying the parameter description should also be 
fixed in HiveConf.java.

If anyone code-savvy wants to take a look at this, that would be great.  If 
not, I'll just fix the docs and HiveConf.java.

(Don't all speak at once now.)

> Please add that hive.mapred.local.mem must be set in MBs in the documentation
> -
>
> Key: HIVE-16735
> URL: https://issues.apache.org/jira/browse/HIVE-16735
> Project: Hive
>  Issue Type: Task
>  Components: Documentation
>Reporter: Laurel Hale
>Assignee: Lefty Leverenz
>Priority: Minor
>
> Hi Lefty,
> One of my developers noticed this and we do not document the local mode. 
> Thought I'd let you know.
> Here's a good place where you could put this:
> https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-QueryandDDLExecution
> There's already a small subsection where hive.mapred.local.mem is documented, 
> but there isn't any information about how to set the value. My developer 
> discovered that when you set it in bytes, it throws the following error:
> 2017-03-12 09:24:31,835 ERROR [HiveServer2-Background-Pool: Thread-22163()]: 
> mr.MapredLocalTask (MapredLocalTask.java:executeInChildVM(341)) - Exception: 
> java.lang.NumberFormatException: For input string: "4294967296"
> java.lang.NumberFormatException: For input string: "4294967296"
> at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Integer.parseInt(Integer.java:495)
> at java.lang.Integer.parseInt(Integer.java:527)
> at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1259)
> at org.apache.hadoop.hive.conf.HiveConf.getIntVar(HiveConf.java:2415)
> at org.apache.hadoop.hive.conf.HiveConf.getIntVar(HiveConf.java:2424)
> at 
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInChildVM(MapredLocalTask.java:228)
> Instead, this value should be set in MBs for success.
> Thanks,
> Laurel



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16742) cap the number of reducers for LLAP at the configured value

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022332#comment-16022332
 ] 

Lefty Leverenz commented on HIVE-16742:
---

[~sershe], please update the fix version (unless you're going to commit to some 
branches too).

> cap the number of reducers for LLAP at the configured value
> ---
>
> Key: HIVE-16742
> URL: https://issues.apache.org/jira/browse/HIVE-16742
> Project: Hive
>  Issue Type: Bug
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16742.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16738) Notification ID generation in DBNotification might not be unique across HS2 instances.

[anishek (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022329#comment-16022329
 ] 

anishek commented on HIVE-16738:


[~mohitsabharwal], i am looking at use cases where we have two separate 
instances of HS2 running in different machines(say HS1 and HS2)  with both of 
them connecting to a common hive metastore and the clients re connecting to one 
of the hive server 2 machines randomly, 

client 1 connects to HS1
client 2 connects to hs2

threads on HS1 and HS2 should be able to get a lock on {{NOTIFICATION_TBL_LOCK 
}} at the same time => hence entering the critical section to update the 
notification Event which would read the notification sequence entry with 
primary key 1 in both cases and update to the same value for possibly two 
events ?

> Notification ID generation in DBNotification might not be unique across HS2 
> instances.
> --
>
> Key: HIVE-16738
> URL: https://issues.apache.org/jira/browse/HIVE-16738
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
> Fix For: 3.0.0
>
>
> Going to explain the problem in scope of "replication" feature for hive 2 
> that is being built, as it is easier to explain:
> To allow replication to work we need to set 
> "hive.metastore.transactional.event.listeners"  to DBNotificationListener. 
> For use cases where there are multiple HiveServer2 Instances running 
> {code}
>  private void process(NotificationEvent event, ListenerEvent listenerEvent) 
> throws MetaException {
> event.setMessageFormat(msgFactory.getMessageFormat());
> synchronized (NOTIFICATION_TBL_LOCK) {
>   LOG.debug("DbNotificationListener: Processing : {}:{}", 
> event.getEventId(),
>   event.getMessage());
>   HMSHandler.getMSForConf(hiveConf).addNotificationEvent(event);
> }
>   // Set the DB_NOTIFICATION_EVENT_ID for future reference by other 
> listeners.
>   if (event.isSetEventId()) {
> listenerEvent.putParameter(
> MetaStoreEventListenerConstants.DB_NOTIFICATION_EVENT_ID_KEY_NAME,
> Long.toString(event.getEventId()));
>   }
>   }
> {code}
> the above code in DBNotificationListner having the object lock wont be 
> guarantee enough to make sure that all events get a unique id. The 
> transaction isolation level at the db "read-comitted" or "repeatable-read"  
> would  also not guarantee the same, unless a lock is at the db level 
> preferably on table {{NOTIFICATION_SEQUENCE}} which only has one row.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (HIVE-16738) Notification ID generation in DBNotification might not be unique across HS2 instances.

[anishek (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022329#comment-16022329
 ] 

anishek edited comment on HIVE-16738 at 5/24/17 5:22 AM:
-

[~mohitsabharwal], i am looking at use cases where we have two separate 
instances of HS2 running in different machines(say HS1 and HS2)  with both of 
them connecting to a common hive metastore and the clients re connecting to one 
of the hive server 2 machines randomly, 

client 1 connects to HS1
client 2 connects to hs2

threads on HS1 and HS2 should be able to get a lock on 
{{NOTIFICATION_TBL_LOCK}} at the same time => hence entering the critical 
section to update the notification Event which would read the notification 
sequence entry with primary key 1 in both cases and update to the same value 
for possibly two events ?


was (Author: anishek):
[~mohitsabharwal], i am looking at use cases where we have two separate 
instances of HS2 running in different machines(say HS1 and HS2)  with both of 
them connecting to a common hive metastore and the clients re connecting to one 
of the hive server 2 machines randomly, 

client 1 connects to HS1
client 2 connects to hs2

threads on HS1 and HS2 should be able to get a lock on {{NOTIFICATION_TBL_LOCK 
}} at the same time => hence entering the critical section to update the 
notification Event which would read the notification sequence entry with 
primary key 1 in both cases and update to the same value for possibly two 
events ?

> Notification ID generation in DBNotification might not be unique across HS2 
> instances.
> --
>
> Key: HIVE-16738
> URL: https://issues.apache.org/jira/browse/HIVE-16738
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
> Fix For: 3.0.0
>
>
> Going to explain the problem in scope of "replication" feature for hive 2 
> that is being built, as it is easier to explain:
> To allow replication to work we need to set 
> "hive.metastore.transactional.event.listeners"  to DBNotificationListener. 
> For use cases where there are multiple HiveServer2 Instances running 
> {code}
>  private void process(NotificationEvent event, ListenerEvent listenerEvent) 
> throws MetaException {
> event.setMessageFormat(msgFactory.getMessageFormat());
> synchronized (NOTIFICATION_TBL_LOCK) {
>   LOG.debug("DbNotificationListener: Processing : {}:{}", 
> event.getEventId(),
>   event.getMessage());
>   HMSHandler.getMSForConf(hiveConf).addNotificationEvent(event);
> }
>   // Set the DB_NOTIFICATION_EVENT_ID for future reference by other 
> listeners.
>   if (event.isSetEventId()) {
> listenerEvent.putParameter(
> MetaStoreEventListenerConstants.DB_NOTIFICATION_EVENT_ID_KEY_NAME,
> Long.toString(event.getEventId()));
>   }
>   }
> {code}
> the above code in DBNotificationListner having the object lock wont be 
> guarantee enough to make sure that all events get a unique id. The 
> transaction isolation level at the db "read-comitted" or "repeatable-read"  
> would  also not guarantee the same, unless a lock is at the db level 
> preferably on table {{NOTIFICATION_SEQUENCE}} which only has one row.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16727) REPL DUMP for insert event should't fail if the table is already dropped.

[Sankar Hariappan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022326#comment-16022326
 ] 

Sankar Hariappan commented on HIVE-16727:
-

The test failures are irrelevant to the patch!

> REPL DUMP for insert event should't fail if the table is already dropped.
> -
>
> Key: HIVE-16727
> URL: https://issues.apache.org/jira/browse/HIVE-16727
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive, repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR, replication
> Attachments: HIVE-16727.01.patch
>
>
> Currently, insert event doesn't log the table object as part of event 
> notification message. During dump, the table object is obtained from 
> metastore which can be null if the table is already dropped and hence REPL 
> DUMP fails.
> Steps:
> 1. Bootstrap dump/load with a table.
> 2. Insert into the table.
> 3. Drop the table.
> 4. REPL DUMP (incremental).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15300) Reuse table information in SemanticAnalyzer::getMetaData to reduce compilation time

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022318#comment-16022318
 ] 

Hive QA commented on HIVE-15300:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869570/HIVE-15300.4.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10749 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_3] (batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_4] (batchId=79)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[filter_cond_pushdown2] 
(batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[lateral_view_onview] 
(batchId=55)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
 (batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=144)
org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning 
(batchId=287)
org.apache.hive.service.cli.session.TestSessionManagerMetrics.testAbandonedSessionMetrics
 (batchId=192)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5410/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5410/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5410/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12869570 - PreCommit-HIVE-Build

> Reuse table information in SemanticAnalyzer::getMetaData to reduce 
> compilation time
> ---
>
> Key: HIVE-15300
> URL: https://issues.apache.org/jira/browse/HIVE-15300
> Project: Hive
>  Issue Type: Improvement
>  Components: Parser
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-15300.1.patch, HIVE-15300.2.patch, 
> HIVE-15300.3.patch, HIVE-15300.4.patch
>
>
> E.g Q88 in tpc-ds takes lots of time to compile and it ends up getting the 
> table details for the same table repeatedly. It took 20+seconds to compile 
> the query.
> It would be good to reuse the table information in 
> SemanticAnalyzer::getMetadata.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16746) Reduce number of index lookups for same table in IndexWhereTaskDispatcher

[Rajesh Balamohan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan reassigned HIVE-16746:
---

Assignee: Rajesh Balamohan

> Reduce number of index lookups for same table in IndexWhereTaskDispatcher
> -
>
> Key: HIVE-16746
> URL: https://issues.apache.org/jira/browse/HIVE-16746
> Project: Hive
>  Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16746.1.patch
>
>
> {{IndexWhereTaskDispatcher}} is used when 
> {{hive.optimize.index.filter=true}}. It lists all indices for the table and 
> depending on the query complexity, this ends up being in the hotpath. For 
> e.g, Q14 explain plan takes 180-200 seconds and this index querying multiple 
> times for same tables take up 30-40 seconds. This function was invoked around 
> 24000 times for same set of tables.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16746) Reduce number of index lookups for same table in IndexWhereTaskDispatcher

[Rajesh Balamohan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan updated HIVE-16746:

Attachment: HIVE-16746.1.patch

With patch, response time came down from 185 seconds to 140-145 seconds. This 
benefits irrespective of store implementation. Currently CachedStore does not 
yet implement index caching.

> Reduce number of index lookups for same table in IndexWhereTaskDispatcher
> -
>
> Key: HIVE-16746
> URL: https://issues.apache.org/jira/browse/HIVE-16746
> Project: Hive
>  Issue Type: Bug
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16746.1.patch
>
>
> {{IndexWhereTaskDispatcher}} is used when 
> {{hive.optimize.index.filter=true}}. It lists all indices for the table and 
> depending on the query complexity, this ends up being in the hotpath. For 
> e.g, Q14 explain plan takes 180-200 seconds and this index querying multiple 
> times for same tables take up 30-40 seconds. This function was invoked around 
> 24000 times for same set of tables.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16746) Reduce number of index lookups for same table in IndexWhereTaskDispatcher

[Rajesh Balamohan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan updated HIVE-16746:

Status: Patch Available  (was: Open)

> Reduce number of index lookups for same table in IndexWhereTaskDispatcher
> -
>
> Key: HIVE-16746
> URL: https://issues.apache.org/jira/browse/HIVE-16746
> Project: Hive
>  Issue Type: Bug
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16746.1.patch
>
>
> {{IndexWhereTaskDispatcher}} is used when 
> {{hive.optimize.index.filter=true}}. It lists all indices for the table and 
> depending on the query complexity, this ends up being in the hotpath. For 
> e.g, Q14 explain plan takes 180-200 seconds and this index querying multiple 
> times for same tables take up 30-40 seconds. This function was invoked around 
> 24000 times for same set of tables.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16745) Syntax error in 041-HIVE-16556.mysql.sql script

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022279#comment-16022279
 ] 

Hive QA commented on HIVE-16745:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869561/HIVE-16745.01.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10749 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
 (batchId=151)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5409/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5409/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5409/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12869561 - PreCommit-HIVE-Build

> Syntax error in 041-HIVE-16556.mysql.sql script
> ---
>
> Key: HIVE-16745
> URL: https://issues.apache.org/jira/browse/HIVE-16745
> Project: Hive
>  Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-16745.01.patch
>
>
> 041-HIVE-16556.mysql.sql has a syntax error which was introduced with 
> HIVE-16711



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16675) Fix ConcurrentModificationException in SparkClientImpl#startDriver

[Sahil Takiar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022265#comment-16022265
 ] 

Sahil Takiar commented on HIVE-16675:
-

+1 LGTM

> Fix ConcurrentModificationException in SparkClientImpl#startDriver
> --
>
> Key: HIVE-16675
> URL: https://issues.apache.org/jira/browse/HIVE-16675
> Project: Hive
>  Issue Type: Bug
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Attachments: HIVE-16675.1.patch, HIVE-16675.patch
>
>
> the exception is
> {noformat}
>   2017-05-16T00:29:37,480  WARN [Driver] client.SparkClientImpl: 
> Exception while waiting for child process.
>   3926 java.util.ConcurrentModificationException
>   3927 at 
> java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901) 
> ~[?:1.8.0_91]
>   3928 at 
> java.util.ArrayList$Itr.next(ArrayList.java:851) ~[?:1.8.0_91]
>   3929 at 
> org.apache.hive.spark.client.SparkClientImpl$3.run(SparkClientImpl.java:495) 
> [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
>   3930 at java.lang.Thread.run(Thread.java:745) 
> [?:1.8.0_91]
> {noformat}
> It seems that {{SparkClientImpl.java#childErrorLog}} is read while it is 
> written. It is better to change {{SparkClientImpl.java#childErrorLog}} from 
> ArrayList to CopyOnWriteArrayList to avoid the exception.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16743) BitSet set() is not incorrectly used in TxnUtils.createValidCompactTxnList()

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022251#comment-16022251
 ] 

Eugene Koifman commented on HIVE-16743:
---

I think the issue is that the bad code is in 
TxnUtils.createValidCompactTxnList() but all tests use ValidTxnList directly - 
so it's never getting tested.  Same for TxnUtils.createValidReadTxnList() - 
both of which are a problem since they include vital logic.
Could you add a follow up ticket to address this please?

+1 this fix



> BitSet set() is not incorrectly used in TxnUtils.createValidCompactTxnList()
> 
>
> Key: HIVE-16743
> URL: https://issues.apache.org/jira/browse/HIVE-16743
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-16743.1.patch
>
>
> The second line is problematic
> {code}
> BitSet bitSet = new BitSet(exceptions.length);
> bitSet.set(0, bitSet.length()); // for ValidCompactorTxnList, everything 
> in exceptions are aborted
> {code}
> For example, exceptions' length is 2. We declare a BitSet object with initial 
> size of 2 via the first line above. But that's not the actual size of the 
> BitSet. So bitSet.length() will still return 0.
> The intention of the second line above is to set all the bits to true. This 
> was not achieved because bitSet.set(0, bitSet.length()) is equivalent to 
> bitSet.set(0, 0).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16744) LLAP index update may be broken after ORC switch

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022249#comment-16022249
 ] 

Hive QA commented on HIVE-16744:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869557/HIVE-16744.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10749 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters]
 (batchId=142)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic] 
(batchId=139)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
 (batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
 (batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=144)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5408/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5408/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5408/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12869557 - PreCommit-HIVE-Build

> LLAP index update may be broken after ORC switch
> 
>
> Key: HIVE-16744
> URL: https://issues.apache.org/jira/browse/HIVE-16744
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16744.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-14881) integrate MM tables into ACID: merge cleaner into ACID threads

[Wei Zheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei Zheng updated HIVE-14881:
-
Attachment: HIVE-14881.2.patch

patch 2 accommodates Eugene's review comment

> integrate MM tables into ACID: merge cleaner into ACID threads 
> ---
>
> Key: HIVE-14881
> URL: https://issues.apache.org/jira/browse/HIVE-14881
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Wei Zheng
> Attachments: HIVE-14881.1.patch, HIVE-14881.2.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16743) BitSet set() is not incorrectly used in TxnUtils.createValidCompactTxnList()

[Wei Zheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022234#comment-16022234
 ] 

Wei Zheng commented on HIVE-16743:
--

[~ekoifman] Can you review it please?

> BitSet set() is not incorrectly used in TxnUtils.createValidCompactTxnList()
> 
>
> Key: HIVE-16743
> URL: https://issues.apache.org/jira/browse/HIVE-16743
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-16743.1.patch
>
>
> The second line is problematic
> {code}
> BitSet bitSet = new BitSet(exceptions.length);
> bitSet.set(0, bitSet.length()); // for ValidCompactorTxnList, everything 
> in exceptions are aborted
> {code}
> For example, exceptions' length is 2. We declare a BitSet object with initial 
> size of 2 via the first line above. But that's not the actual size of the 
> BitSet. So bitSet.length() will still return 0.
> The intention of the second line above is to set all the bits to true. This 
> was not achieved because bitSet.set(0, bitSet.length()) is equivalent to 
> bitSet.set(0, 0).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16743) BitSet set() is not incorrectly used in TxnUtils.createValidCompactTxnList()

[Wei Zheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022233#comment-16022233
 ] 

Wei Zheng commented on HIVE-16743:
--

My understanding is this newly added BitSet currently has no client which is 
using it - it's only used by isTxnAborted() and isTxnRangeAborted(). Although 
readFromString() and writeToString() also use it, but the fact that it was 
constructed incorrectly doesn't impact the serialization/deserialization.

p.s. my unit test in other ticket already verified 
isTxnAborted()/isTxnRangeAborted() is working with this fix :)

> BitSet set() is not incorrectly used in TxnUtils.createValidCompactTxnList()
> 
>
> Key: HIVE-16743
> URL: https://issues.apache.org/jira/browse/HIVE-16743
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-16743.1.patch
>
>
> The second line is problematic
> {code}
> BitSet bitSet = new BitSet(exceptions.length);
> bitSet.set(0, bitSet.length()); // for ValidCompactorTxnList, everything 
> in exceptions are aborted
> {code}
> For example, exceptions' length is 2. We declare a BitSet object with initial 
> size of 2 via the first line above. But that's not the actual size of the 
> BitSet. So bitSet.length() will still return 0.
> The intention of the second line above is to set all the bits to true. This 
> was not achieved because bitSet.set(0, bitSet.length()) is equivalent to 
> bitSet.set(0, 0).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16675) Fix ConcurrentModificationException in SparkClientImpl#startDriver

[Ferdinand Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022230#comment-16022230
 ] 

Ferdinand Xu commented on HIVE-16675:
-

[~stakiar], do you have further comments?

> Fix ConcurrentModificationException in SparkClientImpl#startDriver
> --
>
> Key: HIVE-16675
> URL: https://issues.apache.org/jira/browse/HIVE-16675
> Project: Hive
>  Issue Type: Bug
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Attachments: HIVE-16675.1.patch, HIVE-16675.patch
>
>
> the exception is
> {noformat}
>   2017-05-16T00:29:37,480  WARN [Driver] client.SparkClientImpl: 
> Exception while waiting for child process.
>   3926 java.util.ConcurrentModificationException
>   3927 at 
> java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901) 
> ~[?:1.8.0_91]
>   3928 at 
> java.util.ArrayList$Itr.next(ArrayList.java:851) ~[?:1.8.0_91]
>   3929 at 
> org.apache.hive.spark.client.SparkClientImpl$3.run(SparkClientImpl.java:495) 
> [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
>   3930 at java.lang.Thread.run(Thread.java:745) 
> [?:1.8.0_91]
> {noformat}
> It seems that {{SparkClientImpl.java#childErrorLog}} is read while it is 
> written. It is better to change {{SparkClientImpl.java#childErrorLog}} from 
> ArrayList to CopyOnWriteArrayList to avoid the exception.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16675) Fix ConcurrentModificationException in SparkClientImpl#startDriver

[liyunzhang_intel (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1604#comment-1604
 ] 

liyunzhang_intel commented on HIVE-16675:
-

[~Ferd]: help review HIVE-16675.1.patch. thanks.

> Fix ConcurrentModificationException in SparkClientImpl#startDriver
> --
>
> Key: HIVE-16675
> URL: https://issues.apache.org/jira/browse/HIVE-16675
> Project: Hive
>  Issue Type: Bug
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Attachments: HIVE-16675.1.patch, HIVE-16675.patch
>
>
> the exception is
> {noformat}
>   2017-05-16T00:29:37,480  WARN [Driver] client.SparkClientImpl: 
> Exception while waiting for child process.
>   3926 java.util.ConcurrentModificationException
>   3927 at 
> java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901) 
> ~[?:1.8.0_91]
>   3928 at 
> java.util.ArrayList$Itr.next(ArrayList.java:851) ~[?:1.8.0_91]
>   3929 at 
> org.apache.hive.spark.client.SparkClientImpl$3.run(SparkClientImpl.java:495) 
> [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
>   3930 at java.lang.Thread.run(Thread.java:745) 
> [?:1.8.0_91]
> {noformat}
> It seems that {{SparkClientImpl.java#childErrorLog}} is read while it is 
> written. It is better to change {{SparkClientImpl.java#childErrorLog}} from 
> ArrayList to CopyOnWriteArrayList to avoid the exception.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15665) LLAP: OrcFileMetadata objects in cache can impact heap usage

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-15665:

Attachment: HIVE-15665.01.patch

The patch based on, and including, HIVE-16233. Tests seem to pass. Will run on 
cluster tomorrow.

> LLAP: OrcFileMetadata objects in cache can impact heap usage
> 
>
> Key: HIVE-15665
> URL: https://issues.apache.org/jira/browse/HIVE-15665
> Project: Hive
>  Issue Type: Improvement
>  Components: llap
>Reporter: Rajesh Balamohan
>Assignee: Sergey Shelukhin
> Attachments: HIVE-15665.01.patch, HIVE-15665.patch
>
>
> OrcFileMetadata internally has filestats, stripestats etc which are allocated 
> in heap. On large data sets, this could have an impact on the heap usage and 
> the memory usage by different executors in LLAP.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16737) LLAP: Shuffle handler TCP listen queue overflows

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022213#comment-16022213
 ] 

Hive QA commented on HIVE-16737:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869552/HIVE-16737.2.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10749 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
 (batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=144)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5407/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5407/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5407/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12869552 - PreCommit-HIVE-Build

> LLAP: Shuffle handler TCP listen queue overflows
> 
>
> Key: HIVE-16737
> URL: https://issues.apache.org/jira/browse/HIVE-16737
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16737.1.patch, HIVE-16737.2.patch, 
> HIVE-16737-branch-2.x.patch
>
>
> {code}
> $ netstat -s | grep "listen queue of a socket"
> localhost: 297070 times the listen queue of a socket overflowed
> {code}
> {code}
> $ ss -tl
> LISTEN 0  50 *:15551*:*
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15300) Reuse table information in SemanticAnalyzer::getMetaData to reduce compilation time

[Rajesh Balamohan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan updated HIVE-15300:

Attachment: HIVE-15300.4.patch

Rebasing the patch with minor fixes. This is mainly for ObjectStore. If 
CachedStore is used, getTable wouldn't be expensive.

> Reuse table information in SemanticAnalyzer::getMetaData to reduce 
> compilation time
> ---
>
> Key: HIVE-15300
> URL: https://issues.apache.org/jira/browse/HIVE-15300
> Project: Hive
>  Issue Type: Improvement
>  Components: Parser
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-15300.1.patch, HIVE-15300.2.patch, 
> HIVE-15300.3.patch, HIVE-15300.4.patch
>
>
> E.g Q88 in tpc-ds takes lots of time to compile and it ends up getting the 
> table details for the same table repeatedly. It took 20+seconds to compile 
> the query.
> It would be good to reuse the table information in 
> SemanticAnalyzer::getMetadata.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16745) Syntax error in 041-HIVE-16556.mysql.sql script

[Vihang Karajgaonkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vihang Karajgaonkar updated HIVE-16745:
---
Status: Patch Available  (was: Open)

> Syntax error in 041-HIVE-16556.mysql.sql script
> ---
>
> Key: HIVE-16745
> URL: https://issues.apache.org/jira/browse/HIVE-16745
> Project: Hive
>  Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-16745.01.patch
>
>
> 041-HIVE-16556.mysql.sql has a syntax error which was introduced with 
> HIVE-16711



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16745) Syntax error in 041-HIVE-16556.mysql.sql script

[Vihang Karajgaonkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vihang Karajgaonkar updated HIVE-16745:
---
Attachment: HIVE-16745.01.patch

> Syntax error in 041-HIVE-16556.mysql.sql script
> ---
>
> Key: HIVE-16745
> URL: https://issues.apache.org/jira/browse/HIVE-16745
> Project: Hive
>  Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-16745.01.patch
>
>
> 041-HIVE-16556.mysql.sql has a syntax error which was introduced with 
> HIVE-16711



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16745) Syntax error in 041-HIVE-16556.mysql.sql script

[Vihang Karajgaonkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vihang Karajgaonkar reassigned HIVE-16745:
--


> Syntax error in 041-HIVE-16556.mysql.sql script
> ---
>
> Key: HIVE-16745
> URL: https://issues.apache.org/jira/browse/HIVE-16745
> Project: Hive
>  Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
>
> 041-HIVE-16556.mysql.sql has a syntax error which was introduced with 
> HIVE-16711



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16743) BitSet set() is not incorrectly used in TxnUtils.createValidCompactTxnList()

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022178#comment-16022178
 ] 

Hive QA commented on HIVE-16743:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869541/HIVE-16743.1.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10749 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
 (batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join30]
 (batchId=149)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5406/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5406/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5406/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12869541 - PreCommit-HIVE-Build

> BitSet set() is not incorrectly used in TxnUtils.createValidCompactTxnList()
> 
>
> Key: HIVE-16743
> URL: https://issues.apache.org/jira/browse/HIVE-16743
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-16743.1.patch
>
>
> The second line is problematic
> {code}
> BitSet bitSet = new BitSet(exceptions.length);
> bitSet.set(0, bitSet.length()); // for ValidCompactorTxnList, everything 
> in exceptions are aborted
> {code}
> For example, exceptions' length is 2. We declare a BitSet object with initial 
> size of 2 via the first line above. But that's not the actual size of the 
> BitSet. So bitSet.length() will still return 0.
> The intention of the second line above is to set all the bits to true. This 
> was not achieved because bitSet.set(0, bitSet.length()) is equivalent to 
> bitSet.set(0, 0).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16744) LLAP index update may be broken after ORC switch

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022173#comment-16022173
 ] 

Prasanth Jayachandran commented on HIVE-16744:
--

+1

> LLAP index update may be broken after ORC switch
> 
>
> Key: HIVE-16744
> URL: https://issues.apache.org/jira/browse/HIVE-16744
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16744.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16744) LLAP index update is broken after ORC switch

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-16744:

Status: Patch Available  (was: Open)

> LLAP index update is broken after ORC switch
> 
>
> Key: HIVE-16744
> URL: https://issues.apache.org/jira/browse/HIVE-16744
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16744.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16744) LLAP index update may be broken after ORC switch

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-16744:

Summary: LLAP index update may be broken after ORC switch  (was: LLAP index 
update is broken after ORC switch)

> LLAP index update may be broken after ORC switch
> 
>
> Key: HIVE-16744
> URL: https://issues.apache.org/jira/browse/HIVE-16744
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16744.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16706) Bootstrap REPL DUMP shouldn't fail when a partition is dropped/renamed when dump in progress.

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022162#comment-16022162
 ] 

Thejas M Nair commented on HIVE-16706:
--

+1


> Bootstrap REPL DUMP shouldn't fail when a partition is dropped/renamed when 
> dump in progress.
> -
>
> Key: HIVE-16706
> URL: https://issues.apache.org/jira/browse/HIVE-16706
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive, repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR, replication
> Attachments: HIVE-16706.01.patch
>
>
> Currently, bootstrap REPL DUMP gets the partitions in a batch and then 
> iterate through it. If any partition is dropped/renamed during iteration, it 
> may lead to failure/exception. In this case, the partition should be skipped 
> from dump and also need to ensure no failure of REPL DUMP and the subsequent 
> incremental dump should ensure the consistent state of the table.
> This bug is related to HIVE-16684.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16742) cap the number of reducers for LLAP at the configured value

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-16742:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed to master. Thanks for the review!

> cap the number of reducers for LLAP at the configured value
> ---
>
> Key: HIVE-16742
> URL: https://issues.apache.org/jira/browse/HIVE-16742
> Project: Hive
>  Issue Type: Bug
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16742.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16744) LLAP index update is broken after ORC switch

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HIVE-16744:
---


> LLAP index update is broken after ORC switch
> 
>
> Key: HIVE-16744
> URL: https://issues.apache.org/jira/browse/HIVE-16744
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022156#comment-16022156
 ] 

Thejas M Nair commented on HIVE-15051:
--

+1
Can you also please create a follow up jira to remove the Yetus*.sh files when 
the new release is out for yetus ?


> Test framework integration with findbugs, rat checks etc.
> -
>
> Key: HIVE-15051
> URL: https://issues.apache.org/jira/browse/HIVE-15051
> Project: Hive
>  Issue Type: Sub-task
>  Components: Testing Infrastructure
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: beeline.out, HIVE-15051.patch, Interim.patch, ql.out
>
>
> Find a way to integrate code analysis tools like findbugs, rat checks to 
> PreCommit tests, thus removing the burden from reviewers to check the code 
> style and other checks which could be done by code. 
> Might worth to take a look on Yetus, but keep in mind the Hive has a specific 
> parallel test framework.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16743) BitSet set() is not incorrectly used in TxnUtils.createValidCompactTxnList()

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022146#comment-16022146
 ] 

Eugene Koifman commented on HIVE-16743:
---

Are we missing a test case that covers the user of this bitSet.  It's strange 
that no test failed in spite of this bug.

> BitSet set() is not incorrectly used in TxnUtils.createValidCompactTxnList()
> 
>
> Key: HIVE-16743
> URL: https://issues.apache.org/jira/browse/HIVE-16743
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-16743.1.patch
>
>
> The second line is problematic
> {code}
> BitSet bitSet = new BitSet(exceptions.length);
> bitSet.set(0, bitSet.length()); // for ValidCompactorTxnList, everything 
> in exceptions are aborted
> {code}
> For example, exceptions' length is 2. We declare a BitSet object with initial 
> size of 2 via the first line above. But that's not the actual size of the 
> BitSet. So bitSet.length() will still return 0.
> The intention of the second line above is to set all the bits to true. This 
> was not achieved because bitSet.set(0, bitSet.length()) is equivalent to 
> bitSet.set(0, 0).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16737) LLAP: Shuffle handler TCP listen queue overflows

[Rajesh Balamohan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022142#comment-16022142
 ] 

Rajesh Balamohan commented on HIVE-16737:
-

+1. LGTM.

> LLAP: Shuffle handler TCP listen queue overflows
> 
>
> Key: HIVE-16737
> URL: https://issues.apache.org/jira/browse/HIVE-16737
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16737.1.patch, HIVE-16737.2.patch, 
> HIVE-16737-branch-2.x.patch
>
>
> {code}
> $ netstat -s | grep "listen queue of a socket"
> localhost: 297070 times the listen queue of a socket overflowed
> {code}
> {code}
> $ ss -tl
> LISTEN 0  50 *:15551*:*
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15144) JSON.org license is now CatX

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022141#comment-16022141
 ] 

Hive QA commented on HIVE-15144:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869539/HIVE-15144.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10749 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explain_dependency2] 
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explain_dependency] 
(batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join3] 
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join4] 
(batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join6] 
(batchId=39)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
 (batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=144)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5405/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5405/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5405/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12869539 - PreCommit-HIVE-Build

> JSON.org license is now CatX
> 
>
> Key: HIVE-15144
> URL: https://issues.apache.org/jira/browse/HIVE-15144
> Project: Hive
>  Issue Type: Bug
>Reporter: Robert Kanter
>Priority: Blocker
> Fix For: 2.2.0
>
> Attachments: HIVE-15144.patch
>
>
> per [update resolved legal|http://www.apache.org/legal/resolved.html#json]:
> {quote}
> CAN APACHE PRODUCTS INCLUDE WORKS LICENSED UNDER THE JSON LICENSE?
> No. As of 2016-11-03 this has been moved to the 'Category X' license list. 
> Prior to this, use of the JSON Java library was allowed. See Debian's page 
> for a list of alternatives.
> {quote}
> I'm not sure when this dependency was first introduced, but it looks like 
> it's currently used in a few places:
> https://github.com/apache/hive/search?p=1=%22org.json%22=%E2%9C%93



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15160) Can't order by an unselected column

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022116#comment-16022116
 ] 

Ashutosh Chauhan commented on HIVE-15160:
-

+1

> Can't order by an unselected column
> ---
>
> Key: HIVE-15160
> URL: https://issues.apache.org/jira/browse/HIVE-15160
> Project: Hive
>  Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-15160.01.patch, HIVE-15160.02.patch, 
> HIVE-15160.04.patch, HIVE-15160.05.patch, HIVE-15160.06.patch, 
> HIVE-15160.07.patch, HIVE-15160.08.patch, HIVE-15160.09.patch, 
> HIVE-15160.09.patch, HIVE-15160.10.patch, HIVE-15160.11.patch, 
> HIVE-15160.12.patch, HIVE-15160.13.patch, HIVE-15160.14.patch, 
> HIVE-15160.15.patch, HIVE-15160.16.patch, HIVE-15160.17.patch
>
>
> If a grouping key hasn't been selected, Hive complains. For comparison, 
> Postgres does not.
> Example. Notice i_item_id is not selected:
> {code}
> select  i_item_desc
>,i_category
>,i_class
>,i_current_price
>,sum(cs_ext_sales_price) as itemrevenue
>,sum(cs_ext_sales_price)*100/sum(sum(cs_ext_sales_price)) over
>(partition by i_class) as revenueratio
>  from catalog_sales
>  ,item
>  ,date_dim
>  where cs_item_sk = i_item_sk
>and i_category in ('Jewelry', 'Sports', 'Books')
>and cs_sold_date_sk = d_date_sk
>  and d_date between cast('2001-01-12' as date)
>   and (cast('2001-01-12' as date) + 30 days)
>  group by i_item_id
>  ,i_item_desc
>  ,i_category
>  ,i_class
>  ,i_current_price
>  order by i_category
>  ,i_class
>  ,i_item_id
>  ,i_item_desc
>  ,revenueratio
> limit 100;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16737) LLAP: Shuffle handler TCP listen queue overflows

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-16737:
-
Attachment: HIVE-16737.2.patch

Added logging of SOMAXCONN in .2 patch

> LLAP: Shuffle handler TCP listen queue overflows
> 
>
> Key: HIVE-16737
> URL: https://issues.apache.org/jira/browse/HIVE-16737
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16737.1.patch, HIVE-16737.2.patch, 
> HIVE-16737-branch-2.x.patch
>
>
> {code}
> $ netstat -s | grep "listen queue of a socket"
> localhost: 297070 times the listen queue of a socket overflowed
> {code}
> {code}
> $ ss -tl
> LISTEN 0  50 *:15551*:*
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15160) Can't order by an unselected column

[Pengcheng Xiong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022093#comment-16022093
 ] 

Pengcheng Xiong commented on HIVE-15160:


The failures are not related. [~ashutoshc], could u take a final look? thanks.

> Can't order by an unselected column
> ---
>
> Key: HIVE-15160
> URL: https://issues.apache.org/jira/browse/HIVE-15160
> Project: Hive
>  Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-15160.01.patch, HIVE-15160.02.patch, 
> HIVE-15160.04.patch, HIVE-15160.05.patch, HIVE-15160.06.patch, 
> HIVE-15160.07.patch, HIVE-15160.08.patch, HIVE-15160.09.patch, 
> HIVE-15160.09.patch, HIVE-15160.10.patch, HIVE-15160.11.patch, 
> HIVE-15160.12.patch, HIVE-15160.13.patch, HIVE-15160.14.patch, 
> HIVE-15160.15.patch, HIVE-15160.16.patch, HIVE-15160.17.patch
>
>
> If a grouping key hasn't been selected, Hive complains. For comparison, 
> Postgres does not.
> Example. Notice i_item_id is not selected:
> {code}
> select  i_item_desc
>,i_category
>,i_class
>,i_current_price
>,sum(cs_ext_sales_price) as itemrevenue
>,sum(cs_ext_sales_price)*100/sum(sum(cs_ext_sales_price)) over
>(partition by i_class) as revenueratio
>  from catalog_sales
>  ,item
>  ,date_dim
>  where cs_item_sk = i_item_sk
>and i_category in ('Jewelry', 'Sports', 'Books')
>and cs_sold_date_sk = d_date_sk
>  and d_date between cast('2001-01-12' as date)
>   and (cast('2001-01-12' as date) + 30 days)
>  group by i_item_id
>  ,i_item_desc
>  ,i_category
>  ,i_class
>  ,i_current_price
>  order by i_category
>  ,i_class
>  ,i_item_id
>  ,i_item_desc
>  ,revenueratio
> limit 100;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16737) LLAP: Shuffle handler TCP listen queue overflows

[Rajesh Balamohan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022086#comment-16022086
 ] 

Rajesh Balamohan commented on HIVE-16737:
-

Minor: would be nice to add backlog detail in the log 
{{LOG.info("LlapShuffleHandler" + " listening on port " + port);}}

> LLAP: Shuffle handler TCP listen queue overflows
> 
>
> Key: HIVE-16737
> URL: https://issues.apache.org/jira/browse/HIVE-16737
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16737.1.patch, HIVE-16737-branch-2.x.patch
>
>
> {code}
> $ netstat -s | grep "listen queue of a socket"
> localhost: 297070 times the listen queue of a socket overflowed
> {code}
> {code}
> $ ss -tl
> LISTEN 0  50 *:15551*:*
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15160) Can't order by an unselected column

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022073#comment-16022073
 ] 

Hive QA commented on HIVE-15160:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869535/HIVE-15160.17.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10751 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
 (batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=145)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5404/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5404/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5404/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12869535 - PreCommit-HIVE-Build

> Can't order by an unselected column
> ---
>
> Key: HIVE-15160
> URL: https://issues.apache.org/jira/browse/HIVE-15160
> Project: Hive
>  Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-15160.01.patch, HIVE-15160.02.patch, 
> HIVE-15160.04.patch, HIVE-15160.05.patch, HIVE-15160.06.patch, 
> HIVE-15160.07.patch, HIVE-15160.08.patch, HIVE-15160.09.patch, 
> HIVE-15160.09.patch, HIVE-15160.10.patch, HIVE-15160.11.patch, 
> HIVE-15160.12.patch, HIVE-15160.13.patch, HIVE-15160.14.patch, 
> HIVE-15160.15.patch, HIVE-15160.16.patch, HIVE-15160.17.patch
>
>
> If a grouping key hasn't been selected, Hive complains. For comparison, 
> Postgres does not.
> Example. Notice i_item_id is not selected:
> {code}
> select  i_item_desc
>,i_category
>,i_class
>,i_current_price
>,sum(cs_ext_sales_price) as itemrevenue
>,sum(cs_ext_sales_price)*100/sum(sum(cs_ext_sales_price)) over
>(partition by i_class) as revenueratio
>  from catalog_sales
>  ,item
>  ,date_dim
>  where cs_item_sk = i_item_sk
>and i_category in ('Jewelry', 'Sports', 'Books')
>and cs_sold_date_sk = d_date_sk
>  and d_date between cast('2001-01-12' as date)
>   and (cast('2001-01-12' as date) + 30 days)
>  group by i_item_id
>  ,i_item_desc
>  ,i_category
>  ,i_class
>  ,i_current_price
>  order by i_category
>  ,i_class
>  ,i_item_id
>  ,i_item_desc
>  ,revenueratio
> limit 100;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16737) LLAP: Shuffle handler TCP listen queue overflows

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022001#comment-16022001
 ] 

Hive QA commented on HIVE-16737:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869532/HIVE-16737.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10749 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
 (batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
 (batchId=151)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker2.minorNoBaseLotsOfDeltas 
(batchId=255)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5403/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5403/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5403/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12869532 - PreCommit-HIVE-Build

> LLAP: Shuffle handler TCP listen queue overflows
> 
>
> Key: HIVE-16737
> URL: https://issues.apache.org/jira/browse/HIVE-16737
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16737.1.patch, HIVE-16737-branch-2.x.patch
>
>
> {code}
> $ netstat -s | grep "listen queue of a socket"
> localhost: 297070 times the listen queue of a socket overflowed
> {code}
> {code}
> $ ss -tl
> LISTEN 0  50 *:15551*:*
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16743) BitSet set() is not incorrectly used in TxnUtils.createValidCompactTxnList()

[Wei Zheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei Zheng updated HIVE-16743:
-
Status: Patch Available  (was: Open)

> BitSet set() is not incorrectly used in TxnUtils.createValidCompactTxnList()
> 
>
> Key: HIVE-16743
> URL: https://issues.apache.org/jira/browse/HIVE-16743
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-16743.1.patch
>
>
> The second line is problematic
> {code}
> BitSet bitSet = new BitSet(exceptions.length);
> bitSet.set(0, bitSet.length()); // for ValidCompactorTxnList, everything 
> in exceptions are aborted
> {code}
> For example, exceptions' length is 2. We declare a BitSet object with initial 
> size of 2 via the first line above. But that's not the actual size of the 
> BitSet. So bitSet.length() will still return 0.
> The intention of the second line above is to set all the bits to true. This 
> was not achieved because bitSet.set(0, bitSet.length()) is equivalent to 
> bitSet.set(0, 0).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16743) BitSet set() is not incorrectly used in TxnUtils.createValidCompactTxnList()

[Wei Zheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei Zheng updated HIVE-16743:
-
Attachment: HIVE-16743.1.patch

> BitSet set() is not incorrectly used in TxnUtils.createValidCompactTxnList()
> 
>
> Key: HIVE-16743
> URL: https://issues.apache.org/jira/browse/HIVE-16743
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-16743.1.patch
>
>
> The second line is problematic
> {code}
> BitSet bitSet = new BitSet(exceptions.length);
> bitSet.set(0, bitSet.length()); // for ValidCompactorTxnList, everything 
> in exceptions are aborted
> {code}
> For example, exceptions' length is 2. We declare a BitSet object with initial 
> size of 2 via the first line above. But that's not the actual size of the 
> BitSet. So bitSet.length() will still return 0.
> The intention of the second line above is to set all the bits to true. This 
> was not achieved because bitSet.set(0, bitSet.length()) is equivalent to 
> bitSet.set(0, 0).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-6595) Hive 0.11.0 build failure

[Shawn Lavelle (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021965#comment-16021965
 ] 

Shawn Lavelle commented on HIVE-6595:
-

For completeness, could this be accomplished by compiling with the 1.6 JDK?

> Hive 0.11.0 build failure
> -
>
> Key: HIVE-6595
> URL: https://issues.apache.org/jira/browse/HIVE-6595
> Project: Hive
>  Issue Type: Bug
>  Components: Build Infrastructure
>Affects Versions: 0.11.0
> Environment: CentOS 6.5, java version "1.7.0_45", Hadoop 2.2.0
>Reporter: Amit Anand
>Assignee: Szehon Ho
>
> I am unable to build Hive 0.11.0 from the source. I have a single node hadoop 
> 2.2.0, that I built from the source, running. 
> I followed steps given below:
> svn co http://svn.apache.org/repos/asf/hive/tags/release-0.11.0/ hive-0.11.0
> cd hive-0.11.0
> ant clean
> ant package
> I got messages given below 
> compile:
>  [echo] Project: jdbc
> [javac] Compiling 28 source files to 
> /opt/apache/source/hive-0.11.0/build/jdbc/classes
> [javac] 
> /opt/apache/source/hive-0.11.0/jdbc/src/java/org/apache/hive/jdbc/HiveCallableStatement.java:48:
>  error: HiveCallableStatement is not abstract and does not override abstract 
> method getObject(String,Class) in CallableStatement
> [javac] public class HiveCallableStatement implements 
> java.sql.CallableStatement {
> [javac]^
> [javac]   where T is a type-variable:
> [javac] T extends Object declared in method 
> getObject(String,Class)
> [javac] 
> /opt/apache/source/hive-0.11.0/jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java:65:
>  error: HiveConnection is not abstract and does not override abstract method 
> getNetworkTimeout() in Connection
> [javac] public class HiveConnection implements java.sql.Connection {
> [javac]^
> [javac] 
> /opt/apache/source/hive-0.11.0/jdbc/src/java/org/apache/hive/jdbc/HiveDataSource.java:31:
>  error: HiveDataSource is not abstract and does not override abstract method 
> getParentLogger() in CommonDataSource
> [javac] public class HiveDataSource implements DataSource {
> [javac]^
> [javac] 
> /opt/apache/source/hive-0.11.0/jdbc/src/java/org/apache/hive/jdbc/HiveDatabaseMetaData.java:56:
>  error: HiveDatabaseMetaData is not abstract and does not override abstract 
> method generatedKeyAlwaysReturned() in DatabaseMetaData
> [javac] public class HiveDatabaseMetaData implements DatabaseMetaData {
> [javac]^
> [javac] 
> /opt/apache/source/hive-0.11.0/jdbc/src/java/org/apache/hive/jdbc/HiveDatabaseMetaData.java:707:
>  error:  is not 
> abstract and does not override abstract method getObject(String,Class) 
> in ResultSet
> [javac] , null) {
> [javac] ^
> [javac]   where T is a type-variable:
> [javac] T extends Object declared in method 
> getObject(String,Class)
> [javac] 
> /opt/apache/source/hive-0.11.0/jdbc/src/java/org/apache/hive/jdbc/HiveDriver.java:35:
>  error: HiveDriver is not abstract and does not override abstract method 
> getParentLogger() in Driver
> [javac] public class HiveDriver implements Driver {
> [javac]^
> [javac] 
> /opt/apache/source/hive-0.11.0/jdbc/src/java/org/apache/hive/jdbc/HivePreparedStatement.java:56:
>  error: HivePreparedStatement is not abstract and does not override abstract 
> method isCloseOnCompletion() in Statement
> [javac] public class HivePreparedStatement implements PreparedStatement {
> [javac]^
> [javac] 
> /opt/apache/source/hive-0.11.0/jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java:48:
>  error: HiveQueryResultSet is not abstract and does not override abstract 
> method getObject(String,Class) in ResultSet
> [javac] public class HiveQueryResultSet extends HiveBaseResultSet {
> [javac]^
> [javac]   where T is a type-variable:
> [javac] T extends Object declared in method 
> getObject(String,Class)
> [javac] 
> /opt/apache/source/hive-0.11.0/jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java:42:
>  error: HiveStatement is not abstract and does not override abstract method 
> isCloseOnCompletion() in Statement
> [javac] public class HiveStatement implements java.sql.Statement {
> [javac]^
> [javac] Note: Some input files use or override a deprecated API.
> [javac] Note: Recompile with -Xlint:deprecation for details.
> [javac] Note: Some input files use unchecked or unsafe operations.
> [javac] Note: Recompile with -Xlint:unchecked for details.
> [javac] 9 errors
> BUILD FAILED
> /opt/apache/source/hive-0.11.0/build.xml:274: The following error occurred 
> while executing this line:
> /opt/apache/source/hive-0.11.0/build.xml:113: The following error occurred 
> 

[jira] [Commented] (HIVE-15144) JSON.org license is now CatX

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021959#comment-16021959
 ] 

ASF GitHub Bot commented on HIVE-15144:
---

GitHub user omalley opened a pull request:

https://github.com/apache/hive/pull/188

HIVE-15144. Remove dependence on json.org artifacts.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/omalley/hive hive-15144

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/188.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #188


commit a3ce058d1516825a8680a6aadf47f57f5dbcad2c
Author: Owen O'Malley 
Date:   2017-05-23T22:07:04Z

HIVE-15144. Remove dependence on json.org artifacts.




> JSON.org license is now CatX
> 
>
> Key: HIVE-15144
> URL: https://issues.apache.org/jira/browse/HIVE-15144
> Project: Hive
>  Issue Type: Bug
>Reporter: Robert Kanter
>Priority: Blocker
> Fix For: 2.2.0
>
> Attachments: HIVE-15144.patch
>
>
> per [update resolved legal|http://www.apache.org/legal/resolved.html#json]:
> {quote}
> CAN APACHE PRODUCTS INCLUDE WORKS LICENSED UNDER THE JSON LICENSE?
> No. As of 2016-11-03 this has been moved to the 'Category X' license list. 
> Prior to this, use of the JSON Java library was allowed. See Debian's page 
> for a list of alternatives.
> {quote}
> I'm not sure when this dependency was first introduced, but it looks like 
> it's currently used in a few places:
> https://github.com/apache/hive/search?p=1=%22org.json%22=%E2%9C%93



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15144) JSON.org license is now CatX

[Owen O'Malley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley updated HIVE-15144:
-
Status: Patch Available  (was: Open)

Here's a patch that replaces the dependence with Ted Dunning's replacement, 
except for llap-server, which needs to write json files, which I've put in 
jettison.

> JSON.org license is now CatX
> 
>
> Key: HIVE-15144
> URL: https://issues.apache.org/jira/browse/HIVE-15144
> Project: Hive
>  Issue Type: Bug
>Reporter: Robert Kanter
>Priority: Blocker
> Fix For: 2.2.0
>
> Attachments: HIVE-15144.patch
>
>
> per [update resolved legal|http://www.apache.org/legal/resolved.html#json]:
> {quote}
> CAN APACHE PRODUCTS INCLUDE WORKS LICENSED UNDER THE JSON LICENSE?
> No. As of 2016-11-03 this has been moved to the 'Category X' license list. 
> Prior to this, use of the JSON Java library was allowed. See Debian's page 
> for a list of alternatives.
> {quote}
> I'm not sure when this dependency was first introduced, but it looks like 
> it's currently used in a few places:
> https://github.com/apache/hive/search?p=1=%22org.json%22=%E2%9C%93



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15144) JSON.org license is now CatX

[Owen O'Malley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley updated HIVE-15144:
-
Attachment: HIVE-15144.patch

> JSON.org license is now CatX
> 
>
> Key: HIVE-15144
> URL: https://issues.apache.org/jira/browse/HIVE-15144
> Project: Hive
>  Issue Type: Bug
>Reporter: Robert Kanter
>Priority: Blocker
> Fix For: 2.2.0
>
> Attachments: HIVE-15144.patch
>
>
> per [update resolved legal|http://www.apache.org/legal/resolved.html#json]:
> {quote}
> CAN APACHE PRODUCTS INCLUDE WORKS LICENSED UNDER THE JSON LICENSE?
> No. As of 2016-11-03 this has been moved to the 'Category X' license list. 
> Prior to this, use of the JSON Java library was allowed. See Debian's page 
> for a list of alternatives.
> {quote}
> I'm not sure when this dependency was first introduced, but it looks like 
> it's currently used in a few places:
> https://github.com/apache/hive/search?p=1=%22org.json%22=%E2%9C%93



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16743) BitSet set() is not incorrectly used in TxnUtils.createValidCompactTxnList()

[Wei Zheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei Zheng reassigned HIVE-16743:



> BitSet set() is not incorrectly used in TxnUtils.createValidCompactTxnList()
> 
>
> Key: HIVE-16743
> URL: https://issues.apache.org/jira/browse/HIVE-16743
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
>
> The second line is problematic
> {code}
> BitSet bitSet = new BitSet(exceptions.length);
> bitSet.set(0, bitSet.length()); // for ValidCompactorTxnList, everything 
> in exceptions are aborted
> {code}
> For example, exceptions' length is 2. We declare a BitSet object with initial 
> size of 2 via the first line above. But that's not the actual size of the 
> BitSet. So bitSet.length() will still return 0.
> The intention of the second line above is to set all the bits to true. This 
> was not achieved because bitSet.set(0, bitSet.length()) is equivalent to 
> bitSet.set(0, 0).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16743) BitSet set() is not incorrectly used in TxnUtils.createValidCompactTxnList()

[Wei Zheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021931#comment-16021931
 ] 

Wei Zheng commented on HIVE-16743:
--

This is caused by HIVE-16534

> BitSet set() is not incorrectly used in TxnUtils.createValidCompactTxnList()
> 
>
> Key: HIVE-16743
> URL: https://issues.apache.org/jira/browse/HIVE-16743
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
>
> The second line is problematic
> {code}
> BitSet bitSet = new BitSet(exceptions.length);
> bitSet.set(0, bitSet.length()); // for ValidCompactorTxnList, everything 
> in exceptions are aborted
> {code}
> For example, exceptions' length is 2. We declare a BitSet object with initial 
> size of 2 via the first line above. But that's not the actual size of the 
> BitSet. So bitSet.length() will still return 0.
> The intention of the second line above is to set all the bits to true. This 
> was not achieved because bitSet.set(0, bitSet.length()) is equivalent to 
> bitSet.set(0, 0).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15160) Can't order by an unselected column

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-15160:
---
Status: Open  (was: Patch Available)

> Can't order by an unselected column
> ---
>
> Key: HIVE-15160
> URL: https://issues.apache.org/jira/browse/HIVE-15160
> Project: Hive
>  Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-15160.01.patch, HIVE-15160.02.patch, 
> HIVE-15160.04.patch, HIVE-15160.05.patch, HIVE-15160.06.patch, 
> HIVE-15160.07.patch, HIVE-15160.08.patch, HIVE-15160.09.patch, 
> HIVE-15160.09.patch, HIVE-15160.10.patch, HIVE-15160.11.patch, 
> HIVE-15160.12.patch, HIVE-15160.13.patch, HIVE-15160.14.patch, 
> HIVE-15160.15.patch, HIVE-15160.16.patch, HIVE-15160.17.patch
>
>
> If a grouping key hasn't been selected, Hive complains. For comparison, 
> Postgres does not.
> Example. Notice i_item_id is not selected:
> {code}
> select  i_item_desc
>,i_category
>,i_class
>,i_current_price
>,sum(cs_ext_sales_price) as itemrevenue
>,sum(cs_ext_sales_price)*100/sum(sum(cs_ext_sales_price)) over
>(partition by i_class) as revenueratio
>  from catalog_sales
>  ,item
>  ,date_dim
>  where cs_item_sk = i_item_sk
>and i_category in ('Jewelry', 'Sports', 'Books')
>and cs_sold_date_sk = d_date_sk
>  and d_date between cast('2001-01-12' as date)
>   and (cast('2001-01-12' as date) + 30 days)
>  group by i_item_id
>  ,i_item_desc
>  ,i_category
>  ,i_class
>  ,i_current_price
>  order by i_category
>  ,i_class
>  ,i_item_id
>  ,i_item_desc
>  ,revenueratio
> limit 100;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15160) Can't order by an unselected column

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-15160:
---
Status: Patch Available  (was: Open)

> Can't order by an unselected column
> ---
>
> Key: HIVE-15160
> URL: https://issues.apache.org/jira/browse/HIVE-15160
> Project: Hive
>  Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-15160.01.patch, HIVE-15160.02.patch, 
> HIVE-15160.04.patch, HIVE-15160.05.patch, HIVE-15160.06.patch, 
> HIVE-15160.07.patch, HIVE-15160.08.patch, HIVE-15160.09.patch, 
> HIVE-15160.09.patch, HIVE-15160.10.patch, HIVE-15160.11.patch, 
> HIVE-15160.12.patch, HIVE-15160.13.patch, HIVE-15160.14.patch, 
> HIVE-15160.15.patch, HIVE-15160.16.patch, HIVE-15160.17.patch
>
>
> If a grouping key hasn't been selected, Hive complains. For comparison, 
> Postgres does not.
> Example. Notice i_item_id is not selected:
> {code}
> select  i_item_desc
>,i_category
>,i_class
>,i_current_price
>,sum(cs_ext_sales_price) as itemrevenue
>,sum(cs_ext_sales_price)*100/sum(sum(cs_ext_sales_price)) over
>(partition by i_class) as revenueratio
>  from catalog_sales
>  ,item
>  ,date_dim
>  where cs_item_sk = i_item_sk
>and i_category in ('Jewelry', 'Sports', 'Books')
>and cs_sold_date_sk = d_date_sk
>  and d_date between cast('2001-01-12' as date)
>   and (cast('2001-01-12' as date) + 30 days)
>  group by i_item_id
>  ,i_item_desc
>  ,i_category
>  ,i_class
>  ,i_current_price
>  order by i_category
>  ,i_class
>  ,i_item_id
>  ,i_item_desc
>  ,revenueratio
> limit 100;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16737) LLAP: Shuffle handler TCP listen queue overflows

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021914#comment-16021914
 ] 

Prasanth Jayachandran commented on HIVE-16737:
--

.1 patch to trigger ptest

> LLAP: Shuffle handler TCP listen queue overflows
> 
>
> Key: HIVE-16737
> URL: https://issues.apache.org/jira/browse/HIVE-16737
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16737.1.patch, HIVE-16737-branch-2.x.patch
>
>
> {code}
> $ netstat -s | grep "listen queue of a socket"
> localhost: 297070 times the listen queue of a socket overflowed
> {code}
> {code}
> $ ss -tl
> LISTEN 0  50 *:15551*:*
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15160) Can't order by an unselected column

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-15160:
---
Attachment: HIVE-15160.17.patch

> Can't order by an unselected column
> ---
>
> Key: HIVE-15160
> URL: https://issues.apache.org/jira/browse/HIVE-15160
> Project: Hive
>  Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-15160.01.patch, HIVE-15160.02.patch, 
> HIVE-15160.04.patch, HIVE-15160.05.patch, HIVE-15160.06.patch, 
> HIVE-15160.07.patch, HIVE-15160.08.patch, HIVE-15160.09.patch, 
> HIVE-15160.09.patch, HIVE-15160.10.patch, HIVE-15160.11.patch, 
> HIVE-15160.12.patch, HIVE-15160.13.patch, HIVE-15160.14.patch, 
> HIVE-15160.15.patch, HIVE-15160.16.patch, HIVE-15160.17.patch
>
>
> If a grouping key hasn't been selected, Hive complains. For comparison, 
> Postgres does not.
> Example. Notice i_item_id is not selected:
> {code}
> select  i_item_desc
>,i_category
>,i_class
>,i_current_price
>,sum(cs_ext_sales_price) as itemrevenue
>,sum(cs_ext_sales_price)*100/sum(sum(cs_ext_sales_price)) over
>(partition by i_class) as revenueratio
>  from catalog_sales
>  ,item
>  ,date_dim
>  where cs_item_sk = i_item_sk
>and i_category in ('Jewelry', 'Sports', 'Books')
>and cs_sold_date_sk = d_date_sk
>  and d_date between cast('2001-01-12' as date)
>   and (cast('2001-01-12' as date) + 30 days)
>  group by i_item_id
>  ,i_item_desc
>  ,i_category
>  ,i_class
>  ,i_current_price
>  order by i_category
>  ,i_class
>  ,i_item_id
>  ,i_item_desc
>  ,revenueratio
> limit 100;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16737) LLAP: Shuffle handler TCP listen queue overflows

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-16737:
-
Status: Patch Available  (was: Open)

> LLAP: Shuffle handler TCP listen queue overflows
> 
>
> Key: HIVE-16737
> URL: https://issues.apache.org/jira/browse/HIVE-16737
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16737.1.patch, HIVE-16737-branch-2.x.patch
>
>
> {code}
> $ netstat -s | grep "listen queue of a socket"
> localhost: 297070 times the listen queue of a socket overflowed
> {code}
> {code}
> $ ss -tl
> LISTEN 0  50 *:15551*:*
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16737) LLAP: Shuffle handler TCP listen queue overflows

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-16737:
-
Attachment: HIVE-16737.1.patch

> LLAP: Shuffle handler TCP listen queue overflows
> 
>
> Key: HIVE-16737
> URL: https://issues.apache.org/jira/browse/HIVE-16737
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16737.1.patch, HIVE-16737-branch-2.x.patch
>
>
> {code}
> $ netstat -s | grep "listen queue of a socket"
> localhost: 297070 times the listen queue of a socket overflowed
> {code}
> {code}
> $ ss -tl
> LISTEN 0  50 *:15551*:*
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16742) cap the number of reducers for LLAP at the configured value

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021899#comment-16021899
 ] 

Hive QA commented on HIVE-16742:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869520/HIVE-16742.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10749 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
 (batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=144)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5402/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5402/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5402/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12869520 - PreCommit-HIVE-Build

> cap the number of reducers for LLAP at the configured value
> ---
>
> Key: HIVE-16742
> URL: https://issues.apache.org/jira/browse/HIVE-16742
> Project: Hive
>  Issue Type: Bug
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16742.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16737) LLAP: Shuffle handler TCP listen queue overflows

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-16737:
-
Attachment: HIVE-16737-branch-2.x.patch

Patch still uses netty3 APIs but sets backlog param from netty4 as [~gopalv] 
suggested. Ideally we want to move away from netty3 altogether. Will put up a 
patch for that. 

> LLAP: Shuffle handler TCP listen queue overflows
> 
>
> Key: HIVE-16737
> URL: https://issues.apache.org/jira/browse/HIVE-16737
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16737-branch-2.x.patch
>
>
> {code}
> $ netstat -s | grep "listen queue of a socket"
> localhost: 297070 times the listen queue of a socket overflowed
> {code}
> {code}
> $ ss -tl
> LISTEN 0  50 *:15551*:*
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16737) LLAP: Shuffle handler TCP listen queue overflows

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran reassigned HIVE-16737:


Assignee: Prasanth Jayachandran

> LLAP: Shuffle handler TCP listen queue overflows
> 
>
> Key: HIVE-16737
> URL: https://issues.apache.org/jira/browse/HIVE-16737
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
>
> {code}
> $ netstat -s | grep "listen queue of a socket"
> localhost: 297070 times the listen queue of a socket overflowed
> {code}
> {code}
> $ ss -tl
> LISTEN 0  50 *:15551*:*
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16712) StringBuffer v.s. StringBuilder

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-16712:

   Resolution: Fixed
 Assignee: BELUGA BEHR
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Committed to master. Thanks, Beluga!

> StringBuffer v.s. StringBuilder
> ---
>
> Key: HIVE-16712
> URL: https://issues.apache.org/jira/browse/HIVE-16712
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 2.1.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HIVE-16712.1.patch
>
>
> Where appropriate, replaced {{StringBuffer}} with {{StringBuilder}} to remove 
> superfluous synchronization.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16742) cap the number of reducers for LLAP at the configured value

[Gopal V (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021775#comment-16021775
 ] 

Gopal V commented on HIVE-16742:


Yup, LGTM - +1

{code}
hive> set hive.exec.reducers.max=2;

Reducer 2 llapKILLED  2  002
   0   0 
{code}

> cap the number of reducers for LLAP at the configured value
> ---
>
> Key: HIVE-16742
> URL: https://issues.apache.org/jira/browse/HIVE-16742
> Project: Hive
>  Issue Type: Bug
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16742.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16742) cap the number of reducers for LLAP at the configured value

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-16742:

Status: Patch Available  (was: Open)

> cap the number of reducers for LLAP at the configured value
> ---
>
> Key: HIVE-16742
> URL: https://issues.apache.org/jira/browse/HIVE-16742
> Project: Hive
>  Issue Type: Bug
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16742.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16742) cap the number of reducers for LLAP at the configured value

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-16742:

Attachment: HIVE-16742.patch

[~gopalv] does this make sense?

> cap the number of reducers for LLAP at the configured value
> ---
>
> Key: HIVE-16742
> URL: https://issues.apache.org/jira/browse/HIVE-16742
> Project: Hive
>  Issue Type: Bug
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16742.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16742) cap the number of reducers for LLAP at the configured value

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HIVE-16742:
---


> cap the number of reducers for LLAP at the configured value
> ---
>
> Key: HIVE-16742
> URL: https://issues.apache.org/jira/browse/HIVE-16742
> Project: Hive
>  Issue Type: Bug
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-13288) Confusing exception message in DagUtils.localizeResource

[Michael DeGuzis (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021724#comment-16021724
 ] 

Michael DeGuzis commented on HIVE-13288:


We are still getting this issue as well on Hive 1.2.1.

> Confusing exception message in DagUtils.localizeResource
> 
>
> Key: HIVE-13288
> URL: https://issues.apache.org/jira/browse/HIVE-13288
> Project: Hive
>  Issue Type: Improvement
>  Components: Clients
>Affects Versions: 1.2.1
>Reporter: Jeff Zhang
>
> I got the following exception when query through hive server2. And check the 
> source code, it it due to some error when copying data from local to hdfs. 
> But the IOException is ignored and assume that it is due to another thread is 
> also writing. I don't think it make sense to assume that, at least should log 
> the IOException. 
> {code}
> LOG.info("Localizing resource because it does not exist: " + src + " to dest: 
> " + dest);
>   try {
> destFS.copyFromLocalFile(false, false, src, dest);
>   } catch (IOException e) {
> LOG.info("Looks like another thread is writing the same file will 
> wait.");
> int waitAttempts =
> 
> conf.getInt(HiveConf.ConfVars.HIVE_LOCALIZE_RESOURCE_NUM_WAIT_ATTEMPTS.varname,
> 
> HiveConf.ConfVars.HIVE_LOCALIZE_RESOURCE_NUM_WAIT_ATTEMPTS.defaultIntVal);
> long sleepInterval = HiveConf.getTimeVar(
> conf, HiveConf.ConfVars.HIVE_LOCALIZE_RESOURCE_WAIT_INTERVAL,
> TimeUnit.MILLISECONDS);
> LOG.info("Number of wait attempts: " + waitAttempts + ". Wait 
> interval: "
> + sleepInterval);
> boolean found = false;
> {code}
> {noformat}
> 2016-03-15 11:25:39,921 INFO  [HiveServer2-Background-Pool: Thread-249]: 
> tez.DagUtils (DagUtils.java:getHiveJarDirectory(876)) - Jar dir is 
> null/directory doesn't exist. Choosing HIVE_INSTALL_DIR - /user/jeff/.hiveJars
> 2016-03-15 11:25:40,058 INFO  [HiveServer2-Background-Pool: Thread-249]: 
> tez.DagUtils (DagUtils.java:localizeResource(952)) - Localizing resource 
> because it does not exist: 
> file:/usr/hdp/2.3.2.0-2950/hive/lib/hive-exec-1.2.1.2.3.2.0-2950.jar to dest: 
> hdfs://sandbox.hortonworks.com:8020/user/jeff/.hiveJars/hive-exec-1.2.1.2.3.2.0-2950-a97c953db414a4f792d868e2b0417578a61ccfa368048016926117b641b07f34.jar
> 2016-03-15 11:25:40,063 INFO  [HiveServer2-Background-Pool: Thread-249]: 
> tez.DagUtils (DagUtils.java:localizeResource(956)) - Looks like another 
> thread is writing the same file will wait.
> 2016-03-15 11:25:40,064 INFO  [HiveServer2-Background-Pool: Thread-249]: 
> tez.DagUtils (DagUtils.java:localizeResource(963)) - Number of wait attempts: 
> 5. Wait interval: 5000
> 2016-03-15 11:25:53,548 INFO  [HiveServer2-Handler-Pool: Thread-48]: 
> thrift.ThriftCLIService (ThriftCLIService.java:OpenSession(294)) - Client 
> protocol version: HIVE_CLI_SERVICE_PROTOCOL_V8
> 2016-03-15 11:25:53,548 INFO  [HiveServer2-Handler-Pool: Thread-48]: 
> metastore.HiveMetaStore (HiveMetaStore.java:logInfo(747)) - 1: Shutting down 
> the object store...
> 2016-03-15 11:25:53,549 INFO  [HiveServer2-Handler-Pool: Thread-48]: 
> HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(372)) - 
> ugi=hive/sandbox.hortonworks@example.com   ip=unknown-ip-addr  
> cmd=Shutting down the object store...
> 2016-03-15 11:25:53,549 INFO  [HiveServer2-Handler-Pool: Thread-48]: 
> metastore.HiveMetaStore (HiveMetaStore.java:logInfo(747)) - 1: Metastore 
> shutdown complete.
> 2016-03-15 11:25:53,549 INFO  [HiveServer2-Handler-Pool: Thread-48]: 
> HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(372)) - 
> ugi=hive/sandbox.hortonworks@example.com   ip=unknown-ip-addr  
> cmd=Metastore shutdown complete.
> 2016-03-15 11:25:53,573 INFO  [HiveServer2-Handler-Pool: Thread-48]: 
> session.SessionState (SessionState.java:createPath(641)) - Created local 
> directory: /tmp/e43fbaab-a659-4331-90cb-0ea0b2098e25_resources
> 2016-03-15 11:25:53,577 INFO  [HiveServer2-Handler-Pool: Thread-48]: 
> session.SessionState (SessionState.java:createPath(641)) - Created HDFS 
> directory: /tmp/hive/ambari-qa/e43fbaab-a659-4331-90cb-0ea0b2098e25
> 2016-03-15 11:25:53,582 INFO  [HiveServer2-Handler-Pool: Thread-48]: 
> session.SessionState (SessionState.java:createPath(641)) - Created local 
> directory: /tmp/hive/e43fbaab-a659-4331-90cb-0ea0b2098e25
> 2016-03-15 11:25:53,587 INFO  [HiveServer2-Handler-Pool: Thread-48]: 
> session.SessionState (SessionState.java:createPath(641)) - Created HDFS 
> directory: 
> /tmp/hive/ambari-qa/e43fbaab-a659-4331-90cb-0ea0b2098e25/_tmp_space.db
> 2016-03-15 11:25:53,592 INFO  [HiveServer2-Handler-Pool: Thread-48]: 
> session.HiveSessionImpl (HiveSessionImpl.java:setOperationLogSessionDir(236)) 
> - Operation log 

[jira] [Commented] (HIVE-16600) Refactor SetSparkReducerParallelism#needSetParallelism to enable parallel order by in multi_insert cases

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021627#comment-16021627
 ] 

Xuefu Zhang commented on HIVE-16600:


Yeah, I agree with [~lirui] that multi-insert operator tree doesn't necessarily 
branch at SEL operator. It does if the selected columns in all insert branch 
are purely projections and require no shuffles. Nevertheless, it's safer not to 
make such an assumption.

> Refactor SetSparkReducerParallelism#needSetParallelism to enable parallel 
> order by in multi_insert cases
> 
>
> Key: HIVE-16600
> URL: https://issues.apache.org/jira/browse/HIVE-16600
> Project: Hive
>  Issue Type: Sub-task
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Attachments: HIVE-16600.1.patch, HIVE-16600.2.patch, 
> HIVE-16600.3.patch, HIVE-16600.4.patch, HIVE-16600.5.patch, 
> HIVE-16600.6.patch, HIVE-16600.7.patch, mr.explain, mr.explain.log.HIVE-16600
>
>
> multi_insert_gby.case.q
> {code}
> set hive.exec.reducers.bytes.per.reducer=256;
> set hive.optimize.sampling.orderby=true;
> drop table if exists e1;
> drop table if exists e2;
> create table e1 (key string, value string);
> create table e2 (key string);
> FROM (select key, cast(key as double) as keyD, value from src order by key) a
> INSERT OVERWRITE TABLE e1
> SELECT key, value
> INSERT OVERWRITE TABLE e2
> SELECT key;
> select * from e1;
> select * from e2;
> {code} 
> the parallelism of Sort is 1 even we enable parallel order 
> by("hive.optimize.sampling.orderby" is set as "true").  This is not 
> reasonable because the parallelism  should be calcuated by  
> [Utilities.estimateReducers|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L170]
> this is because SetSparkReducerParallelism#needSetParallelism returns false 
> when [children size of 
> RS|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L207]
>  is greater than 1.
> in this case, the children size of {{RS[2]}} is two.
> the logical plan of the case
> {code}
>TS[0]-SEL[1]-RS[2]-SEL[3]-SEL[4]-FS[5]
> -SEL[6]-FS[7]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16689) Correlated scalar subquery with comparison to constant in predicate fails

[Vineet Garg (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021585#comment-16021585
 ] 

Vineet Garg commented on HIVE-16689:


Pushed this to master. Thanks [~ashutoshc]!
Regarding signature missing in RelFieldTrimmer - RelFieldTrimmer operates only 
on RelNode but not RexNode so I don't think this is a bug but design artifact. 
result(RelNode, Mapping) is a generalized method which takes care of correlated 
variables for a given RelNode. 

> Correlated scalar subquery with comparison to constant in predicate fails
> -
>
> Key: HIVE-16689
> URL: https://issues.apache.org/jira/browse/HIVE-16689
> Project: Hive
>  Issue Type: Bug
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-16689.1.patch, HIVE-16689.2.patch
>
>
> *Reproducer*
> {code:sql}
> CREATE TABLE `item`(
>   `i_item_sk` int,
>   `i_item_id` char(16),
>   `i_rec_start_date` date,
>   `i_rec_end_date` date,
>   `i_item_desc` varchar(200),
>   `i_current_price` decimal(7,2),
>   `i_wholesale_cost` decimal(7,2),
>   `i_brand_id` int,
>   `i_brand` char(50),
>   `i_class_id` int,
>   `i_class` char(50),
>   `i_category_id` int,
>   `i_category` char(50),
>   `i_manufact_id` int,
>   `i_manufact` char(50),
>   `i_size` char(20),
>   `i_formulation` char(20),
>   `i_color` char(20),
>   `i_units` char(10),
>   `i_container` char(10),
>   `i_manager_id` int,
>   `i_product_name` char(50));
> select count(*)
>  from item i1
>  where
>(select count(*)
>from item
>where (i_manufact = i1.i_manufact)) > 0;
> {code}
> *Error stack*
> {code}
> org.apache.calcite.util.mapping.Mappings$NoElementException: source #0 has no 
> target in mapping [size=0, sourceCount=1, targetCount=1, elements=[]]
>   at 
> org.apache.calcite.util.mapping.Mappings$AbstractMapping.getTarget(Mappings.java:874)
>  ~[calcite-core-1.12.0.jar:1.12.0]
>   at 
> org.apache.calcite.sql2rel.RelFieldTrimmer$2.handle(RelFieldTrimmer.java:304) 
> ~[calcite-core-1.12.0.jar:1.12.0]
>   at 
> org.apache.calcite.sql2rel.CorrelationReferenceFinder$MyRexVisitor.visitFieldAccess(CorrelationReferenceFinder.java:59)
>  ~[calcite-core-1.12.0.jar:1.12.0]
>   at 
> org.apache.calcite.sql2rel.CorrelationReferenceFinder$MyRexVisitor.visitFieldAccess(CorrelationReferenceFinder.java:50)
>  ~[calcite-core-1.12.0.jar:1.12.0]
>   at org.apache.calcite.rex.RexFieldAccess.accept(RexFieldAccess.java:81) 
> ~[calcite-core-1.12.0.jar:1.12.0]
>   at org.apache.calcite.rex.RexShuttle.visitList(RexShuttle.java:148) 
> ~[calcite-core-1.12.0.jar:1.12.0]
>   at org.apache.calcite.rex.RexShuttle.visitCall(RexShuttle.java:97) 
> ~[calcite-core-1.12.0.jar:1.12.0]
>   at org.apache.calcite.rex.RexShuttle.visitCall(RexShuttle.java:36) 
> ~[calcite-core-1.12.0.jar:1.12.0]
>   at org.apache.calcite.rex.RexCall.accept(RexCall.java:104) 
> ~[calcite-core-1.12.0.jar:1.12.0]
>   at org.apache.calcite.rex.RexShuttle.apply(RexShuttle.java:279) 
> ~[calcite-core-1.12.0.jar:1.12.0]
>   at org.apache.calcite.rel.core.Filter.accept(Filter.java:103) 
> ~[calcite-core-1.12.0.jar:1.12.0]
>   at 
> org.apache.calcite.sql2rel.CorrelationReferenceFinder.visit(CorrelationReferenceFinder.java:44)
>  ~[calcite-core-1.12.0.jar:1.12.0]
>   at 
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFilter.accept(HiveFilter.java:116)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
>   at 
> org.apache.calcite.rel.RelShuttleImpl.visitChild(RelShuttleImpl.java:55) 
> ~[calcite-core-1.12.0.jar:1.12.0]
>   at 
> org.apache.calcite.rel.RelShuttleImpl.visitChildren(RelShuttleImpl.java:69) 
> ~[calcite-core-1.12.0.jar:1.12.0]
>   at org.apache.calcite.rel.RelShuttleImpl.visit(RelShuttleImpl.java:131) 
> ~[calcite-core-1.12.0.jar:1.12.0]
>   at 
> org.apache.calcite.sql2rel.CorrelationReferenceFinder.visit(CorrelationReferenceFinder.java:43)
>  ~[calcite-core-1.12.0.jar:1.12.0]
>   at 
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject.accept(HiveProject.java:198)
>  ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
>   at 
> org.apache.calcite.rel.RelShuttleImpl.visitChild(RelShuttleImpl.java:55) 
> ~[calcite-core-1.12.0.jar:1.12.0]
>   at 
> org.apache.calcite.rel.RelShuttleImpl.visitChildren(RelShuttleImpl.java:69) 
> ~[calcite-core-1.12.0.jar:1.12.0]
>   at org.apache.calcite.rel.RelShuttleImpl.visit(RelShuttleImpl.java:131) 
> ~[calcite-core-1.12.0.jar:1.12.0]
>   at 
> org.apache.calcite.sql2rel.CorrelationReferenceFinder.visit(CorrelationReferenceFinder.java:43)
>  ~[calcite-core-1.12.0.jar:1.12.0]
>   at 
> org.apache.calcite.rel.AbstractRelNode.accept(AbstractRelNode.java:279) 
> ~[calcite-core-1.12.0.jar:1.12.0]
>   at 
> 

[jira] [Commented] (HIVE-16706) Bootstrap REPL DUMP shouldn't fail when a partition is dropped/renamed when dump in progress.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021556#comment-16021556
 ] 

Hive QA commented on HIVE-16706:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869494/HIVE-16706.01.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10750 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=144)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join30]
 (batchId=149)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5401/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5401/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5401/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12869494 - PreCommit-HIVE-Build

> Bootstrap REPL DUMP shouldn't fail when a partition is dropped/renamed when 
> dump in progress.
> -
>
> Key: HIVE-16706
> URL: https://issues.apache.org/jira/browse/HIVE-16706
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive, repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR, replication
> Attachments: HIVE-16706.01.patch
>
>
> Currently, bootstrap REPL DUMP gets the partitions in a batch and then 
> iterate through it. If any partition is dropped/renamed during iteration, it 
> may lead to failure/exception. In this case, the partition should be skipped 
> from dump and also need to ensure no failure of REPL DUMP and the subsequent 
> incremental dump should ensure the consistent state of the table.
> This bug is related to HIVE-16684.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16735) Please add that hive.mapred.local.mem must be set in MBs in the documentation

[Laurel Hale (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021535#comment-16021535
 ] 

Laurel Hale commented on HIVE-16735:


[~leftylev] Thanks, but I think I must be missing something? I checked both 
places and neither mentions that the value should be set in MBs. I cleared my 
browser cache. This is what I see for the first link above:
--
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.mapred.local.mem

hive.mapred.local.mem
Default Value: 0
Added In: Hive 0.3.0
For local mode, memory of the mappers/reducers.
-
And here is what I see for the second link listed above:
-
https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-Hive,Map-ReduceandLocal-Mode

Note that there may be differences in the runtime environment of Hadoop server 
nodes and the machine running the Hive client (because of different jvm 
versions or different software libraries). This can cause unexpected 
behavior/errors while running in local mode. Also note that local mode 
execution is done in a separate, child jvm (of the Hive client). If the user so 
wishes, the maximum amount of memory for this child jvm can be controlled via 
the option hive.mapred.local.mem. By default, it's set to zero, in which case 
Hive lets Hadoop determine the default memory limits of the child jvm.
-
Thanks for being patient.
Best,
Laurel

> Please add that hive.mapred.local.mem must be set in MBs in the documentation
> -
>
> Key: HIVE-16735
> URL: https://issues.apache.org/jira/browse/HIVE-16735
> Project: Hive
>  Issue Type: Task
>  Components: Documentation
>Reporter: Laurel Hale
>Assignee: Lefty Leverenz
>Priority: Minor
>
> Hi Lefty,
> One of my developers noticed this and we do not document the local mode. 
> Thought I'd let you know.
> Here's a good place where you could put this:
> https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-QueryandDDLExecution
> There's already a small subsection where hive.mapred.local.mem is documented, 
> but there isn't any information about how to set the value. My developer 
> discovered that when you set it in bytes, it throws the following error:
> 2017-03-12 09:24:31,835 ERROR [HiveServer2-Background-Pool: Thread-22163()]: 
> mr.MapredLocalTask (MapredLocalTask.java:executeInChildVM(341)) - Exception: 
> java.lang.NumberFormatException: For input string: "4294967296"
> java.lang.NumberFormatException: For input string: "4294967296"
> at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Integer.parseInt(Integer.java:495)
> at java.lang.Integer.parseInt(Integer.java:527)
> at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1259)
> at org.apache.hadoop.hive.conf.HiveConf.getIntVar(HiveConf.java:2415)
> at org.apache.hadoop.hive.conf.HiveConf.getIntVar(HiveConf.java:2424)
> at 
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInChildVM(MapredLocalTask.java:228)
> Instead, this value should be set in MBs for success.
> Thanks,
> Laurel



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16723) Enable configurable MetaStoreSchemaInfo

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021530#comment-16021530
 ] 

Vihang Karajgaonkar commented on HIVE-16723:


Thanks for the review [~ngangam]

> Enable configurable MetaStoreSchemaInfo 
> 
>
> Key: HIVE-16723
> URL: https://issues.apache.org/jira/browse/HIVE-16723
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HIVE-16723.01.patch, HIVE-16723.02.patch, 
> HIVE-16723.03.patch, HIVE-16723.04.patch
>
>
> {{MetaStoreSchemaInfo}} class is used by Schema tool to get the schema 
> version information. In addition to that schema tool invokes its init and 
> upgrade methods to initialize and upgrade the metastore schema. It is 
> possible that there are minor schema changes in the metastore schema which 
> users have to maintain manually. We can potentially enhance schema tool to 
> use a custom MetaStoreSchemaInfo implementation which can be plugged in based 
> on configuration and schematool could run these customizations while running 
> the upgrade and init scripts. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16727) REPL DUMP for insert event should't fail if the table is already dropped.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021488#comment-16021488
 ] 

Hive QA commented on HIVE-16727:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869480/HIVE-16727.01.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10750 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite]
 (batchId=236)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join30]
 (batchId=149)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5400/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5400/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5400/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12869480 - PreCommit-HIVE-Build

> REPL DUMP for insert event should't fail if the table is already dropped.
> -
>
> Key: HIVE-16727
> URL: https://issues.apache.org/jira/browse/HIVE-16727
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive, repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR, replication
> Attachments: HIVE-16727.01.patch
>
>
> Currently, insert event doesn't log the table object as part of event 
> notification message. During dump, the table object is obtained from 
> metastore which can be null if the table is already dropped and hence REPL 
> DUMP fails.
> Steps:
> 1. Bootstrap dump/load with a table.
> 2. Insert into the table.
> 3. Drop the table.
> 4. REPL DUMP (incremental).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16706) Bootstrap REPL DUMP shouldn't fail when a partition is dropped/renamed when dump in progress.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-16706:

Status: Patch Available  (was: Open)

> Bootstrap REPL DUMP shouldn't fail when a partition is dropped/renamed when 
> dump in progress.
> -
>
> Key: HIVE-16706
> URL: https://issues.apache.org/jira/browse/HIVE-16706
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive, repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR, replication
> Attachments: HIVE-16706.01.patch
>
>
> Currently, bootstrap REPL DUMP gets the partitions in a batch and then 
> iterate through it. If any partition is dropped/renamed during iteration, it 
> may lead to failure/exception. In this case, the partition should be skipped 
> from dump and also need to ensure no failure of REPL DUMP and the subsequent 
> incremental dump should ensure the consistent state of the table.
> This bug is related to HIVE-16684.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16706) Bootstrap REPL DUMP shouldn't fail when a partition is dropped/renamed when dump in progress.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-16706:

Attachment: HIVE-16706.01.patch

Added 01.patch with below changes:
- No source code changes as the issue doesn't exist. 
- There are 2 metastore apis getting called by PartitionIterable during 
bootstrap dump; listPartitionNames (without part_spec) and 
getPartitionsByNames. Both these methods never fail if a table or partition is 
dropped in between. Just return an empty list.
- Added a test case to verify if bootstrap dump fails when we return empty 
string from megastore apis.

Request [~sushanth]/[~thejas] to review the patch!

> Bootstrap REPL DUMP shouldn't fail when a partition is dropped/renamed when 
> dump in progress.
> -
>
> Key: HIVE-16706
> URL: https://issues.apache.org/jira/browse/HIVE-16706
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive, repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR, replication
> Attachments: HIVE-16706.01.patch
>
>
> Currently, bootstrap REPL DUMP gets the partitions in a batch and then 
> iterate through it. If any partition is dropped/renamed during iteration, it 
> may lead to failure/exception. In this case, the partition should be skipped 
> from dump and also need to ensure no failure of REPL DUMP and the subsequent 
> incremental dump should ensure the consistent state of the table.
> This bug is related to HIVE-16684.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16706) Bootstrap REPL DUMP shouldn't fail when a partition is dropped/renamed when dump in progress.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021474#comment-16021474
 ] 

ASF GitHub Bot commented on HIVE-16706:
---

GitHub user sankarh opened a pull request:

https://github.com/apache/hive/pull/187

HIVE-16706: Bootstrap REPL DUMP shouldn't fail when a partition is 
dropped/renamed when dump in progress.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sankarh/hive HIVE-16706

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/187.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #187


commit 7e9ed26ef2bd5a89cfe6df0ad22552dad003c82b
Author: Sankar Hariappan 
Date:   2017-05-23T16:13:04Z

HIVE-16706: Bootstrap REPL DUMP shouldn't fail when a partition is 
dropped/renamed when dump in progress.




> Bootstrap REPL DUMP shouldn't fail when a partition is dropped/renamed when 
> dump in progress.
> -
>
> Key: HIVE-16706
> URL: https://issues.apache.org/jira/browse/HIVE-16706
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive, repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR, replication
>
> Currently, bootstrap REPL DUMP gets the partitions in a batch and then 
> iterate through it. If any partition is dropped/renamed during iteration, it 
> may lead to failure/exception. In this case, the partition should be skipped 
> from dump and also need to ensure no failure of REPL DUMP and the subsequent 
> incremental dump should ensure the consistent state of the table.
> This bug is related to HIVE-16684.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16741) Counting number of records in hive and hbase are different for NULL fields in hive

[Aleksey Vovchenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Vovchenko reassigned HIVE-16741:



>  Counting number of records in hive and hbase are different for NULL fields 
> in hive
> ---
>
> Key: HIVE-16741
> URL: https://issues.apache.org/jira/browse/HIVE-16741
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 2.1.0, 1.2.0
>Reporter: Aleksey Vovchenko
>Assignee: Aleksey Vovchenko
>
> Steps to reproduce:
> STEP 1.  
> hbase> create 'testTable',{NAME=>'cf'}
> STEP 2.
> put 'testTable','10','cf:Address','My Address 411002'
> put 'testTable','10','cf:contactId','653638'
> put 'testTable','10','cf:currentStatus','Awaiting'
> put 'testTable','10','cf:createdAt','1452815193'
> put 'testTable','10','cf:Id','10'
> put 'testTable','15','cf:contactId','653638'
> put 'testTable','15','cf:currentStatus','Awaiting'
> put 'testTable','15','cf:createdAt','1452815193'
> put 'testTable','15','cf:Id','15'
> (Note: Here Addrees column is not provided.It means that NULL.)
> put 'testTable','20','cf:Address','My Address 411003'
> put 'testTable','20','cf:contactId','653638'
> put 'testTable','20','cf:currentStatus','Awaiting'
> put 'testTable','20','cf:createdAt','1452815193'
> put 'testTable','20','cf:Id','20'
> put 'testTable','17','cf:Address','My Address 411003'
> put 'testTable','17','cf:currentStatus','Awaiting'
> put 'testTable','17','cf:createdAt','1452815193'
> put 'testTable','17','cf:Id','17'
> STEP 3.
> hive> CREATE external TABLE hh_testTable(Id string,Address string,contactId 
> string,currentStatus string,createdAt string) STORED BY 
> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES 
> ("hbase.columns.mapping"=":key,cf:Address,cf:contactId,cf:currentStatus,cf:createdAt")
>  TBLPROPERTIES ("hbase.table.name"="testTable");
> STEP 4.
> hive> select count(*),contactid from hh_testTable group by contactid;
> Actual result:
> OK
> 3 653638
> Expected result:
> OK
> 1 NULL
> 3 653637



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16723) Enable configurable MetaStoreSchemaInfo

[Naveen Gangam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-16723:
-
   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Fix has been committed to the master for 3.0.0. release. Thank you [~vihangk1] 
for your contribution.

> Enable configurable MetaStoreSchemaInfo 
> 
>
> Key: HIVE-16723
> URL: https://issues.apache.org/jira/browse/HIVE-16723
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HIVE-16723.01.patch, HIVE-16723.02.patch, 
> HIVE-16723.03.patch, HIVE-16723.04.patch
>
>
> {{MetaStoreSchemaInfo}} class is used by Schema tool to get the schema 
> version information. In addition to that schema tool invokes its init and 
> upgrade methods to initialize and upgrade the metastore schema. It is 
> possible that there are minor schema changes in the metastore schema which 
> users have to maintain manually. We can potentially enhance schema tool to 
> use a custom MetaStoreSchemaInfo implementation which can be plugged in based 
> on configuration and schematool could run these customizations while running 
> the upgrade and init scripts. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16706) Bootstrap REPL DUMP shouldn't fail when a partition is dropped/renamed when dump in progress.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-16706:

Summary: Bootstrap REPL DUMP shouldn't fail when a partition is 
dropped/renamed when dump in progress.  (was: Bootstrap REPL DUMP shouldn't 
fail when a partition is dropped/renamed after fetching the partitions in a 
batch.)

> Bootstrap REPL DUMP shouldn't fail when a partition is dropped/renamed when 
> dump in progress.
> -
>
> Key: HIVE-16706
> URL: https://issues.apache.org/jira/browse/HIVE-16706
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive, repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR, replication
>
> Currently, bootstrap REPL DUMP gets the partitions in a batch and then 
> iterate through it. If any partition is dropped/renamed during iteration, it 
> may lead to failure/exception. In this case, the partition should be skipped 
> from dump and also need to ensure no failure of REPL DUMP and the subsequent 
> incremental dump should ensure the consistent state of the table.
> This bug is related to HIVE-16684.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16706) Bootstrap REPL DUMP shouldn't fail when a partition is dropped/renamed after fetching the partitions in a batch.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-16706:

Description: 
Currently, bootstrap REPL DUMP gets the partitions in a batch and then iterate 
through it. If any partition is dropped/renamed during iteration, it may lead 
to failure/exception. In this case, the partition should be skipped from dump 
and also need to ensure no failure of REPL DUMP and the subsequent incremental 
dump should ensure the consistent state of the table.

This bug is related to HIVE-16684.

  was:
Currently, bootstrap REPL DUMP gets the partitions in a batch and then iterate 
through it. If any partition is dropped/renamed during iteration, it will lead 
to failure/exception. In this case, the partition should be skipped from dump 
and also need to ensure no failure of REPL DUMP and the subsequent incremental 
dump should ensure the consistent state of the table.

This bug is related to HIVE-16684.


> Bootstrap REPL DUMP shouldn't fail when a partition is dropped/renamed after 
> fetching the partitions in a batch.
> 
>
> Key: HIVE-16706
> URL: https://issues.apache.org/jira/browse/HIVE-16706
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive, repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR, replication
>
> Currently, bootstrap REPL DUMP gets the partitions in a batch and then 
> iterate through it. If any partition is dropped/renamed during iteration, it 
> may lead to failure/exception. In this case, the partition should be skipped 
> from dump and also need to ensure no failure of REPL DUMP and the subsequent 
> incremental dump should ensure the consistent state of the table.
> This bug is related to HIVE-16684.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16684) Bootstrap REPL DUMP shouldn't fail when table is dropped after fetching the table names.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-16684:

Labels: DR replication  (was: )

> Bootstrap REPL DUMP shouldn't fail when table is dropped after fetching the 
> table names.
> 
>
> Key: HIVE-16684
> URL: https://issues.apache.org/jira/browse/HIVE-16684
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive, repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-16684.01.patch, HIVE-16684.02.patch
>
>
> Currently, bootstrap dump will fail if the table does't exist when try to 
> dump. This shall occur when table is dropped after REPL DUMP fetched the 
> table names.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16732) Transactional tables should block LOAD DATA

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021386#comment-16021386
 ] 

Eugene Koifman commented on HIVE-16732:
---

you're right - I didn't realize there was a gap in the sequence

> Transactional tables should block LOAD DATA 
> 
>
> Key: HIVE-16732
> URL: https://issues.apache.org/jira/browse/HIVE-16732
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-16732.01.patch
>
>
> This has always been the design.
> see LoadSemanticAnalyzer.analyzeInternal()
> StrictChecks.checkBucketing(conf);
> Some examples (this is exposed by HIVE-16177)
> insert_values_orig_table.q
>  insert_orig_table.q
>  insert_values_orig_table_use_metadata.q



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16727) REPL DUMP for insert event should't fail if the table is already dropped.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-16727:

Status: Patch Available  (was: In Progress)

> REPL DUMP for insert event should't fail if the table is already dropped.
> -
>
> Key: HIVE-16727
> URL: https://issues.apache.org/jira/browse/HIVE-16727
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive, repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
> Attachments: HIVE-16727.01.patch
>
>
> Currently, insert event doesn't log the table object as part of event 
> notification message. During dump, the table object is obtained from 
> metastore which can be null if the table is already dropped and hence REPL 
> DUMP fails.
> Steps:
> 1. Bootstrap dump/load with a table.
> 2. Insert into the table.
> 3. Drop the table.
> 4. REPL DUMP (incremental).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16727) REPL DUMP for insert event should't fail if the table is already dropped.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-16727:

Labels: DR replication  (was: )

> REPL DUMP for insert event should't fail if the table is already dropped.
> -
>
> Key: HIVE-16727
> URL: https://issues.apache.org/jira/browse/HIVE-16727
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive, repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR, replication
> Attachments: HIVE-16727.01.patch
>
>
> Currently, insert event doesn't log the table object as part of event 
> notification message. During dump, the table object is obtained from 
> metastore which can be null if the table is already dropped and hence REPL 
> DUMP fails.
> Steps:
> 1. Bootstrap dump/load with a table.
> 2. Insert into the table.
> 3. Drop the table.
> 4. REPL DUMP (incremental).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16727) REPL DUMP for insert event should't fail if the table is already dropped.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-16727:

Attachment: HIVE-16727.01.patch

Added 01.patch with below changes.
- Added Table and Partition objects to the InsertEvent.
- InsertHandler will read table and partition objects from message itself.

Request [~sushanth]/[~thejas] to review the patch!

> REPL DUMP for insert event should't fail if the table is already dropped.
> -
>
> Key: HIVE-16727
> URL: https://issues.apache.org/jira/browse/HIVE-16727
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive, repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
> Attachments: HIVE-16727.01.patch
>
>
> Currently, insert event doesn't log the table object as part of event 
> notification message. During dump, the table object is obtained from 
> metastore which can be null if the table is already dropped and hence REPL 
> DUMP fails.
> Steps:
> 1. Bootstrap dump/load with a table.
> 2. Insert into the table.
> 3. Drop the table.
> 4. REPL DUMP (incremental).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16727) REPL DUMP for insert event should't fail if the table is already dropped.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021370#comment-16021370
 ] 

ASF GitHub Bot commented on HIVE-16727:
---

GitHub user sankarh opened a pull request:

https://github.com/apache/hive/pull/186

HIVE-16727: REPL DUMP for insert event should't fail if the table is 
already dropped.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sankarh/hive HIVE-16727

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/186.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #186


commit c06a9df4c2c1a735dc8c0139ebdf6451cc6c9017
Author: Sankar Hariappan 
Date:   2017-05-22T08:15:54Z

HIVE-16727: REPL DUMP for insert event should't fail if the table is 
already dropped.




> REPL DUMP for insert event should't fail if the table is already dropped.
> -
>
> Key: HIVE-16727
> URL: https://issues.apache.org/jira/browse/HIVE-16727
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive, repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>
> Currently, insert event doesn't log the table object as part of event 
> notification message. During dump, the table object is obtained from 
> metastore which can be null if the table is already dropped and hence REPL 
> DUMP fails.
> Steps:
> 1. Bootstrap dump/load with a table.
> 2. Insert into the table.
> 3. Drop the table.
> 4. REPL DUMP (incremental).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15160) Can't order by an unselected column

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021366#comment-16021366
 ] 

Ashutosh Chauhan commented on HIVE-15160:
-

Latest patch looks quite good. Seems like some tests are failing. Once you fix 
them, can you also update RB after that?

> Can't order by an unselected column
> ---
>
> Key: HIVE-15160
> URL: https://issues.apache.org/jira/browse/HIVE-15160
> Project: Hive
>  Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-15160.01.patch, HIVE-15160.02.patch, 
> HIVE-15160.04.patch, HIVE-15160.05.patch, HIVE-15160.06.patch, 
> HIVE-15160.07.patch, HIVE-15160.08.patch, HIVE-15160.09.patch, 
> HIVE-15160.09.patch, HIVE-15160.10.patch, HIVE-15160.11.patch, 
> HIVE-15160.12.patch, HIVE-15160.13.patch, HIVE-15160.14.patch, 
> HIVE-15160.15.patch, HIVE-15160.16.patch
>
>
> If a grouping key hasn't been selected, Hive complains. For comparison, 
> Postgres does not.
> Example. Notice i_item_id is not selected:
> {code}
> select  i_item_desc
>,i_category
>,i_class
>,i_current_price
>,sum(cs_ext_sales_price) as itemrevenue
>,sum(cs_ext_sales_price)*100/sum(sum(cs_ext_sales_price)) over
>(partition by i_class) as revenueratio
>  from catalog_sales
>  ,item
>  ,date_dim
>  where cs_item_sk = i_item_sk
>and i_category in ('Jewelry', 'Sports', 'Books')
>and cs_sold_date_sk = d_date_sk
>  and d_date between cast('2001-01-12' as date)
>   and (cast('2001-01-12' as date) + 30 days)
>  group by i_item_id
>  ,i_item_desc
>  ,i_category
>  ,i_class
>  ,i_current_price
>  order by i_category
>  ,i_class
>  ,i_item_id
>  ,i_item_desc
>  ,revenueratio
> limit 100;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16575) Support for 'UNIQUE' and 'NOT NULL' constraints

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021289#comment-16021289
 ] 

Hive QA commented on HIVE-16575:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869459/HIVE-16575.04.patch

{color:green}SUCCESS:{color} +1 due to 20 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10751 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[create_with_constraints_validate]
 (batchId=88)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5399/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5399/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5399/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12869459 - PreCommit-HIVE-Build

> Support for 'UNIQUE' and 'NOT NULL' constraints
> ---
>
> Key: HIVE-16575
> URL: https://issues.apache.org/jira/browse/HIVE-16575
> Project: Hive
>  Issue Type: New Feature
>  Components: CBO, Logical Optimizer, Parser
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-16575.01.patch, HIVE-16575.02.patch, 
> HIVE-16575.03.patch, HIVE-16575.04.patch, HIVE-16575.patch
>
>
> Follow-up on HIVE-13076.
> This issue add support for SQL 'UNIQUE' and 'NOT NULL' constraints when we 
> create a table / alter a table 
> (https://www.postgresql.org/docs/9.6/static/sql-createtable.html).
> As with PK and FK constraints, currently we do not enforce them; thus, the 
> constraints need to use the DISABLE option, but they will be stored and can 
> be enabled for rewriting/optimization using RELY.
> This patch also adds support for inlining the constraints next to the column 
> type definition, i.e., 'column constraints'.
> Some examples of the extension to the syntax included in the patch:
> {code:sql}
> CREATE TABLE table3 (x string NOT NULL DISABLE, PRIMARY KEY (x) DISABLE, 
> CONSTRAINT fk1 FOREIGN KEY (x) REFERENCES table2(a) DISABLE); 
> CREATE TABLE table4 (x string CONSTRAINT nn4_1 NOT NULL DISABLE, y string 
> CONSTRAINT nn4_2 NOT NULL DISABLE, UNIQUE (x) DISABLE, CONSTRAINT fk2 FOREIGN 
> KEY (x) REFERENCES table2(a) DISABLE, 
> CONSTRAINT fk3 FOREIGN KEY (y) REFERENCES table2(a) DISABLE);
> CREATE TABLE table12 (a STRING CONSTRAINT nn12_1 NOT NULL DISABLE NORELY, b 
> STRING);
> CREATE TABLE table13 (a STRING NOT NULL DISABLE RELY, b STRING);
> CREATE TABLE table14 (a STRING CONSTRAINT nn14_1 NOT NULL DISABLE RELY, b 
> STRING);
> CREATE TABLE table15 (a STRING REFERENCES table4(x) DISABLE, b STRING);
> CREATE TABLE table16 (a STRING CONSTRAINT nn16_1 REFERENCES table4(x) DISABLE 
> RELY, b STRING);
> ALTER TABLE table16 CHANGE a a STRING REFERENCES table4(x) DISABLE NOVALIDATE;
> ALTER TABLE table12 CHANGE COLUMN b b STRING CONSTRAINT nn12_2 NOT NULL 
> DISABLE NOVALIDATE;
> ALTER TABLE table13 CHANGE b b STRING NOT NULL DISABLE NOVALIDATE;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Work started] (HIVE-16727) REPL DUMP for insert event should't fail if the table is already dropped.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-16727 started by Sankar Hariappan.
---
> REPL DUMP for insert event should't fail if the table is already dropped.
> -
>
> Key: HIVE-16727
> URL: https://issues.apache.org/jira/browse/HIVE-16727
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive, repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>
> Currently, insert event doesn't log the table object as part of event 
> notification message. During dump, the table object is obtained from 
> metastore which can be null if the table is already dropped and hence REPL 
> DUMP fails.
> Steps:
> 1. Bootstrap dump/load with a table.
> 2. Insert into the table.
> 3. Drop the table.
> 4. REPL DUMP (incremental).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Work stopped] (HIVE-16676) RENAME TABLE and RENAME PARTITION events shall be modified as DROP+CREATE events.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-16676 stopped by Sankar Hariappan.
---
> RENAME TABLE and RENAME PARTITION events shall be modified as DROP+CREATE 
> events.
> -
>
> Key: HIVE-16676
> URL: https://issues.apache.org/jira/browse/HIVE-16676
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>
> Currently, RENAME TABLE and RENAME PARTITION events are treated as ALTER 
> events. 
> For bootstrap dump, if the table is renamed after fetching the table names, 
> then new table will be missing in the dump and so the target database doesn't 
> have both old and new table. During incremental replication, later RENAME 
> events will be noop as the old table doesn't exist in target.
> In order to make RENAME replication simple, it is suggested to treat RENAME 
> as DROP+CREATE event.
> EVENT_RENAME_TABLE = EVENT_DROP_TABLE + EVENT_CREATE_TABLE.
> EVENT_RENAME_PARTITION = EVENT_DROP_PARTITION + EVENT_ADD_PARTITION.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16674) Hive metastore JVM dumps core

[Vlad Gudikov (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021278#comment-16021278
 ] 

Vlad Gudikov commented on HIVE-16674:
-

Are these failures related to fix?

> Hive metastore JVM dumps core
> -
>
> Key: HIVE-16674
> URL: https://issues.apache.org/jira/browse/HIVE-16674
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.2.1
> Environment: Hive-1.2.1
> Kerberos enabled cluster
>Reporter: Vlad Gudikov
>Assignee: Vlad Gudikov
>Priority: Blocker
> Fix For: 1.2.1, 2.3.0
>
> Attachments: HIVE-16674.1.patch, HIVE-16674.patch
>
>
> While trying to run a Hive query on 24 partitions executed on an external 
> table with large amount of partitions (4K+). I get an error
> {code}
>  - org.apache.thrift.transport.TSaslTransport$SaslParticipant.wrap(byte[], 
> int, int) @bci=27, line=568 (Compiled frame)
>  - org.apache.thrift.transport.TSaslTransport.flush() @bci=52, line=492 
> (Compiled frame)
>  - org.apache.thrift.transport.TSaslServerTransport.flush() @bci=1, line=41 
> (Compiled frame)
>  - org.apache.thrift.ProcessFunction.process(int, 
> org.apache.thrift.protocol.TProtocol, org.apache.thrift.protocol.TProtocol, 
> java.lang.Object) @bci=236, line=55 (Compiled frame)
>  - 
> org.apache.thrift.TBaseProcessor.process(org.apache.thrift.protocol.TProtocol,
>  org.apache.thrift.protocol.TProtocol) @bci=126, line=39 (Compiled frame)
>  - 
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run()
>  @bci=15, line=690 (Compiled frame)
>  - 
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run()
>  @bci=1, line=685 (Compiled frame)
>  - 
> java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction,
>  java.security.AccessControlContext) @bci=0 (Compiled frame)
>  - javax.security.auth.Subject.doAs(javax.security.auth.Subject, 
> java.security.PrivilegedExceptionAction) @bci=42, line=422 (Compiled frame)
>  - 
> org.apache.hadoop.security.UserGroupInformation.doAs(java.security.PrivilegedExceptionAction)
>  @bci=14, line=1595 (Compiled frame)
>  - 
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(org.apache.thrift.protocol.TProtocol,
>  org.apache.thrift.protocol.TProtocol) @bci=273, line=685 (Compiled frame)
>  - org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run() @bci=151, 
> line=285 (Interpreted frame)
>  - 
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
>  @bci=95, line=1142 (Interpreted frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 
> (Interpreted frame)
>  - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16674) Hive metastore JVM dumps core

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021227#comment-16021227
 ] 

Hive QA commented on HIVE-16674:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869451/HIVE-16674.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 245 failed/errored test(s), 6687 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite]
 (batchId=236)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.org.apache.hadoop.hive.cli.TestBlobstoreCliDriver
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[create_like] 
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ctas_blobstore_to_blobstore]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ctas_blobstore_to_hdfs]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ctas_hdfs_to_blobstore]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_blobstore]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_local]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_warehouse]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_local_to_blobstore]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_blobstore]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_blobstore_nonpart]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_local]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_warehouse]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_warehouse_nonpart]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_local_to_blobstore]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_blobstore_to_blobstore]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_empty_into_blobstore]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_dynamic_partitions]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_table]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_directory]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_table]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[join2] 
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[join] 
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[map_join] 
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[map_join_on_filter]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[nested_outer_join]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_buckets] 
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_format_nonpart]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_format_part]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_nonstd_partitions_loc]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ptf_general_queries]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ptf_matchpath] 
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ptf_orcfile] 
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ptf_persistence]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ptf_rcfile] 
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ptf_seqfile] 
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[rcfile_buckets] 
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[rcfile_format_nonpart]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[rcfile_format_part]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[rcfile_nonstd_partitions_loc]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[write_final_output_blobstore]
 

[jira] [Updated] (HIVE-16575) Support for 'UNIQUE' and 'NOT NULL' constraints

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-16575:
---
Attachment: HIVE-16575.04.patch

> Support for 'UNIQUE' and 'NOT NULL' constraints
> ---
>
> Key: HIVE-16575
> URL: https://issues.apache.org/jira/browse/HIVE-16575
> Project: Hive
>  Issue Type: New Feature
>  Components: CBO, Logical Optimizer, Parser
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-16575.01.patch, HIVE-16575.02.patch, 
> HIVE-16575.03.patch, HIVE-16575.04.patch, HIVE-16575.patch
>
>
> Follow-up on HIVE-13076.
> This issue add support for SQL 'UNIQUE' and 'NOT NULL' constraints when we 
> create a table / alter a table 
> (https://www.postgresql.org/docs/9.6/static/sql-createtable.html).
> As with PK and FK constraints, currently we do not enforce them; thus, the 
> constraints need to use the DISABLE option, but they will be stored and can 
> be enabled for rewriting/optimization using RELY.
> This patch also adds support for inlining the constraints next to the column 
> type definition, i.e., 'column constraints'.
> Some examples of the extension to the syntax included in the patch:
> {code:sql}
> CREATE TABLE table3 (x string NOT NULL DISABLE, PRIMARY KEY (x) DISABLE, 
> CONSTRAINT fk1 FOREIGN KEY (x) REFERENCES table2(a) DISABLE); 
> CREATE TABLE table4 (x string CONSTRAINT nn4_1 NOT NULL DISABLE, y string 
> CONSTRAINT nn4_2 NOT NULL DISABLE, UNIQUE (x) DISABLE, CONSTRAINT fk2 FOREIGN 
> KEY (x) REFERENCES table2(a) DISABLE, 
> CONSTRAINT fk3 FOREIGN KEY (y) REFERENCES table2(a) DISABLE);
> CREATE TABLE table12 (a STRING CONSTRAINT nn12_1 NOT NULL DISABLE NORELY, b 
> STRING);
> CREATE TABLE table13 (a STRING NOT NULL DISABLE RELY, b STRING);
> CREATE TABLE table14 (a STRING CONSTRAINT nn14_1 NOT NULL DISABLE RELY, b 
> STRING);
> CREATE TABLE table15 (a STRING REFERENCES table4(x) DISABLE, b STRING);
> CREATE TABLE table16 (a STRING CONSTRAINT nn16_1 REFERENCES table4(x) DISABLE 
> RELY, b STRING);
> ALTER TABLE table16 CHANGE a a STRING REFERENCES table4(x) DISABLE NOVALIDATE;
> ALTER TABLE table12 CHANGE COLUMN b b STRING CONSTRAINT nn12_2 NOT NULL 
> DISABLE NOVALIDATE;
> ALTER TABLE table13 CHANGE b b STRING NOT NULL DISABLE NOVALIDATE;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16738) Notification ID generation in DBNotification might not be unique across HS2 instances.

[Mohit Sabharwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021190#comment-16021190
 ] 

Mohit Sabharwal commented on HIVE-16738:


[~anishek], not sure I understand. Note that ObjectStore/RawStore is a thread 
local (see HiveMetaStore.java). 

> Notification ID generation in DBNotification might not be unique across HS2 
> instances.
> --
>
> Key: HIVE-16738
> URL: https://issues.apache.org/jira/browse/HIVE-16738
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
> Fix For: 3.0.0
>
>
> Going to explain the problem in scope of "replication" feature for hive 2 
> that is being built, as it is easier to explain:
> To allow replication to work we need to set 
> "hive.metastore.transactional.event.listeners"  to DBNotificationListener. 
> For use cases where there are multiple HiveServer2 Instances running 
> {code}
>  private void process(NotificationEvent event, ListenerEvent listenerEvent) 
> throws MetaException {
> event.setMessageFormat(msgFactory.getMessageFormat());
> synchronized (NOTIFICATION_TBL_LOCK) {
>   LOG.debug("DbNotificationListener: Processing : {}:{}", 
> event.getEventId(),
>   event.getMessage());
>   HMSHandler.getMSForConf(hiveConf).addNotificationEvent(event);
> }
>   // Set the DB_NOTIFICATION_EVENT_ID for future reference by other 
> listeners.
>   if (event.isSetEventId()) {
> listenerEvent.putParameter(
> MetaStoreEventListenerConstants.DB_NOTIFICATION_EVENT_ID_KEY_NAME,
> Long.toString(event.getEventId()));
>   }
>   }
> {code}
> the above code in DBNotificationListner having the object lock wont be 
> guarantee enough to make sure that all events get a unique id. The 
> transaction isolation level at the db "read-comitted" or "repeatable-read"  
> would  also not guarantee the same, unless a lock is at the db level 
> preferably on table {{NOTIFICATION_SEQUENCE}} which only has one row.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16575) Support for 'UNIQUE' and 'NOT NULL' constraints

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021158#comment-16021158
 ] 

Hive QA commented on HIVE-16575:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869449/HIVE-16575.03.patch

{color:green}SUCCESS:{color} +1 due to 20 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10751 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite]
 (batchId=236)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join30]
 (batchId=149)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5397/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5397/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5397/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12869449 - PreCommit-HIVE-Build

> Support for 'UNIQUE' and 'NOT NULL' constraints
> ---
>
> Key: HIVE-16575
> URL: https://issues.apache.org/jira/browse/HIVE-16575
> Project: Hive
>  Issue Type: New Feature
>  Components: CBO, Logical Optimizer, Parser
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-16575.01.patch, HIVE-16575.02.patch, 
> HIVE-16575.03.patch, HIVE-16575.patch
>
>
> Follow-up on HIVE-13076.
> This issue add support for SQL 'UNIQUE' and 'NOT NULL' constraints when we 
> create a table / alter a table 
> (https://www.postgresql.org/docs/9.6/static/sql-createtable.html).
> As with PK and FK constraints, currently we do not enforce them; thus, the 
> constraints need to use the DISABLE option, but they will be stored and can 
> be enabled for rewriting/optimization using RELY.
> This patch also adds support for inlining the constraints next to the column 
> type definition, i.e., 'column constraints'.
> Some examples of the extension to the syntax included in the patch:
> {code:sql}
> CREATE TABLE table3 (x string NOT NULL DISABLE, PRIMARY KEY (x) DISABLE, 
> CONSTRAINT fk1 FOREIGN KEY (x) REFERENCES table2(a) DISABLE); 
> CREATE TABLE table4 (x string CONSTRAINT nn4_1 NOT NULL DISABLE, y string 
> CONSTRAINT nn4_2 NOT NULL DISABLE, UNIQUE (x) DISABLE, CONSTRAINT fk2 FOREIGN 
> KEY (x) REFERENCES table2(a) DISABLE, 
> CONSTRAINT fk3 FOREIGN KEY (y) REFERENCES table2(a) DISABLE);
> CREATE TABLE table12 (a STRING CONSTRAINT nn12_1 NOT NULL DISABLE NORELY, b 
> STRING);
> CREATE TABLE table13 (a STRING NOT NULL DISABLE RELY, b STRING);
> CREATE TABLE table14 (a STRING CONSTRAINT nn14_1 NOT NULL DISABLE RELY, b 
> STRING);
> CREATE TABLE table15 (a STRING REFERENCES table4(x) DISABLE, b STRING);
> CREATE TABLE table16 (a STRING CONSTRAINT nn16_1 REFERENCES table4(x) DISABLE 
> RELY, b STRING);
> ALTER TABLE table16 CHANGE a a STRING REFERENCES table4(x) DISABLE NOVALIDATE;
> ALTER TABLE table12 CHANGE COLUMN b b STRING CONSTRAINT nn12_2 NOT NULL 
> DISABLE NOVALIDATE;
> ALTER TABLE table13 CHANGE b b STRING NOT NULL DISABLE NOVALIDATE;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16674) Hive metastore JVM dumps core

[Vlad Gudikov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vlad Gudikov updated HIVE-16674:

Attachment: HIVE-16674.1.patch

> Hive metastore JVM dumps core
> -
>
> Key: HIVE-16674
> URL: https://issues.apache.org/jira/browse/HIVE-16674
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.2.1
> Environment: Hive-1.2.1
> Kerberos enabled cluster
>Reporter: Vlad Gudikov
>Assignee: Vlad Gudikov
>Priority: Blocker
> Fix For: 1.2.1, 2.3.0
>
> Attachments: HIVE-16674.1.patch, HIVE-16674.patch
>
>
> While trying to run a Hive query on 24 partitions executed on an external 
> table with large amount of partitions (4K+). I get an error
> {code}
>  - org.apache.thrift.transport.TSaslTransport$SaslParticipant.wrap(byte[], 
> int, int) @bci=27, line=568 (Compiled frame)
>  - org.apache.thrift.transport.TSaslTransport.flush() @bci=52, line=492 
> (Compiled frame)
>  - org.apache.thrift.transport.TSaslServerTransport.flush() @bci=1, line=41 
> (Compiled frame)
>  - org.apache.thrift.ProcessFunction.process(int, 
> org.apache.thrift.protocol.TProtocol, org.apache.thrift.protocol.TProtocol, 
> java.lang.Object) @bci=236, line=55 (Compiled frame)
>  - 
> org.apache.thrift.TBaseProcessor.process(org.apache.thrift.protocol.TProtocol,
>  org.apache.thrift.protocol.TProtocol) @bci=126, line=39 (Compiled frame)
>  - 
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run()
>  @bci=15, line=690 (Compiled frame)
>  - 
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run()
>  @bci=1, line=685 (Compiled frame)
>  - 
> java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction,
>  java.security.AccessControlContext) @bci=0 (Compiled frame)
>  - javax.security.auth.Subject.doAs(javax.security.auth.Subject, 
> java.security.PrivilegedExceptionAction) @bci=42, line=422 (Compiled frame)
>  - 
> org.apache.hadoop.security.UserGroupInformation.doAs(java.security.PrivilegedExceptionAction)
>  @bci=14, line=1595 (Compiled frame)
>  - 
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(org.apache.thrift.protocol.TProtocol,
>  org.apache.thrift.protocol.TProtocol) @bci=273, line=685 (Compiled frame)
>  - org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run() @bci=151, 
> line=285 (Interpreted frame)
>  - 
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
>  @bci=95, line=1142 (Interpreted frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 
> (Interpreted frame)
>  - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16575) Support for 'UNIQUE' and 'NOT NULL' constraints

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-16575:
---
Attachment: HIVE-16575.03.patch

> Support for 'UNIQUE' and 'NOT NULL' constraints
> ---
>
> Key: HIVE-16575
> URL: https://issues.apache.org/jira/browse/HIVE-16575
> Project: Hive
>  Issue Type: New Feature
>  Components: CBO, Logical Optimizer, Parser
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-16575.01.patch, HIVE-16575.02.patch, 
> HIVE-16575.03.patch, HIVE-16575.patch
>
>
> Follow-up on HIVE-13076.
> This issue add support for SQL 'UNIQUE' and 'NOT NULL' constraints when we 
> create a table / alter a table 
> (https://www.postgresql.org/docs/9.6/static/sql-createtable.html).
> As with PK and FK constraints, currently we do not enforce them; thus, the 
> constraints need to use the DISABLE option, but they will be stored and can 
> be enabled for rewriting/optimization using RELY.
> This patch also adds support for inlining the constraints next to the column 
> type definition, i.e., 'column constraints'.
> Some examples of the extension to the syntax included in the patch:
> {code:sql}
> CREATE TABLE table3 (x string NOT NULL DISABLE, PRIMARY KEY (x) DISABLE, 
> CONSTRAINT fk1 FOREIGN KEY (x) REFERENCES table2(a) DISABLE); 
> CREATE TABLE table4 (x string CONSTRAINT nn4_1 NOT NULL DISABLE, y string 
> CONSTRAINT nn4_2 NOT NULL DISABLE, UNIQUE (x) DISABLE, CONSTRAINT fk2 FOREIGN 
> KEY (x) REFERENCES table2(a) DISABLE, 
> CONSTRAINT fk3 FOREIGN KEY (y) REFERENCES table2(a) DISABLE);
> CREATE TABLE table12 (a STRING CONSTRAINT nn12_1 NOT NULL DISABLE NORELY, b 
> STRING);
> CREATE TABLE table13 (a STRING NOT NULL DISABLE RELY, b STRING);
> CREATE TABLE table14 (a STRING CONSTRAINT nn14_1 NOT NULL DISABLE RELY, b 
> STRING);
> CREATE TABLE table15 (a STRING REFERENCES table4(x) DISABLE, b STRING);
> CREATE TABLE table16 (a STRING CONSTRAINT nn16_1 REFERENCES table4(x) DISABLE 
> RELY, b STRING);
> ALTER TABLE table16 CHANGE a a STRING REFERENCES table4(x) DISABLE NOVALIDATE;
> ALTER TABLE table12 CHANGE COLUMN b b STRING CONSTRAINT nn12_2 NOT NULL 
> DISABLE NOVALIDATE;
> ALTER TABLE table13 CHANGE b b STRING NOT NULL DISABLE NOVALIDATE;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16643) BeeLine tests output should keep the PREHOOK/POSTHOOK Input/Output orderdering

[Yongzhi Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021077#comment-16021077
 ] 

Yongzhi Chen commented on HIVE-16643:
-

The patch looks good +1

> BeeLine tests output should keep the PREHOOK/POSTHOOK Input/Output orderdering
> --
>
> Key: HIVE-16643
> URL: https://issues.apache.org/jira/browse/HIVE-16643
> Project: Hive
>  Issue Type: New Feature
>  Components: Testing Infrastructure
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16643.01.patch, HIVE-16643.patch
>
>
> The {{PreExecutePrinter}} and the {{PostExecutePrinter}} prints the query 
> input and the output list in alphabetical order in {{printEntities}} method.
> Our goal is to have the same output from the BeeLine query tests, and the Cli 
> query tests. Since the BeeLine tests are using test specific databases to run 
> the tests, and only converting the results in the end to remove this specific 
> database names from the output, we have to reorder the lists after this 
> conversion.
> Raw BeeLine output:
> {code}
> [..]
> INFO  : PREHOOK: Output: create_merge_compressed@src_rc_merge_test
> INFO  : PREHOOK: Output: database:create_merge_compressed
> [..]
> {code}
> Before patch BeeLine output:
> {code}
> [..]
> PREHOOK: Output: default@src_rc_merge_test
> PREHOOK: Output: database:default
> [..]
> {code}
> Expected output:
> {code}
> [..]
> PREHOOK: Output: database:default
> PREHOOK: Output: default@src_rc_merge_test
> [..]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-2369) Minor typo in error message in HiveConnection.java (JDBC)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021036#comment-16021036
 ] 

ASF GitHub Bot commented on HIVE-2369:
--

Github user ClementNotin closed the pull request at:

https://github.com/apache/hive/pull/3


> Minor typo in error message in HiveConnection.java (JDBC)
> -
>
> Key: HIVE-2369
> URL: https://issues.apache.org/jira/browse/HIVE-2369
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 0.7.1, 0.8.0
> Environment: Linux
>Reporter: Clément Notin
>Assignee: Clément Notin
>Priority: Trivial
> Fix For: 0.8.0
>
> Attachments: HIVE-2369.patch
>
>   Original Estimate: 2m
>  Remaining Estimate: 2m
>
> There is a minor typo issue in HiveConnection.java (jdbc) :
> {code}throw new SQLException("Could not establish connecton to "
> + uri + ": " + e.getMessage(), "08S01");{code}
> It seems like there's a "i" missing.
> I know it's a very minor typo but I report it anyway. I won't attach a patch 
> because it would be too long for me to SVN checkout just for 1 letter.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16674) Hive metastore JVM dumps core

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020972#comment-16020972
 ] 

Hive QA commented on HIVE-16674:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869413/HIVE-16674.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 244 failed/errored test(s), 6687 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.org.apache.hadoop.hive.cli.TestBlobstoreCliDriver
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[create_like] 
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ctas_blobstore_to_blobstore]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ctas_blobstore_to_hdfs]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ctas_hdfs_to_blobstore]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_blobstore]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_local]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_warehouse]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_local_to_blobstore]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_blobstore]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_blobstore_nonpart]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_local]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_warehouse]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_warehouse_nonpart]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_local_to_blobstore]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_blobstore_to_blobstore]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_empty_into_blobstore]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_dynamic_partitions]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_table]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_directory]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_table]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[join2] 
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[join] 
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[map_join] 
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[map_join_on_filter]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[nested_outer_join]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_buckets] 
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_format_nonpart]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_format_part]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_nonstd_partitions_loc]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ptf_general_queries]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ptf_matchpath] 
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ptf_orcfile] 
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ptf_persistence]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ptf_rcfile] 
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ptf_seqfile] 
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[rcfile_buckets] 
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[rcfile_format_nonpart]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[rcfile_format_part]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[rcfile_nonstd_partitions_loc]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[write_final_output_blobstore]
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[zero_rows_hdfs] 
(batchId=239)

[jira] [Comment Edited] (HIVE-16600) Refactor SetSparkReducerParallelism#needSetParallelism to enable parallel order by in multi_insert cases

[Rui Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020960#comment-16020960
 ] 

Rui Li edited comment on HIVE-16600 at 5/23/17 9:45 AM:


[~kellyzly], I tried some multi insert cases and they don't necessarily branch 
at SEL. I suppose this happens if the selected columns are identical in the 
inserts. E.g.
{noformat}
from (select key from src) A 
  insert into table target1 select key
  insert into table target2 select key;

from (select key,value from src order by key) A
  insert into table target1 select key,sum(value) group by key
  insert into table target2 select key,avg(value) group by key;
{noformat}


was (Author: lirui):
[~kellyzly], I tried some multi cases and they don't necessarily branch at SEL. 
I suppose this happens if the selected columns are identical in the inserts. 
E.g.
{noformat}
from (select key from src) A 
  insert into table target1 select key
  insert into table target2 select key;

from (select key,value from src order by key) A
  insert into table target1 select key,sum(value) group by key
  insert into table target2 select key,avg(value) group by key;
{noformat}

> Refactor SetSparkReducerParallelism#needSetParallelism to enable parallel 
> order by in multi_insert cases
> 
>
> Key: HIVE-16600
> URL: https://issues.apache.org/jira/browse/HIVE-16600
> Project: Hive
>  Issue Type: Sub-task
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Attachments: HIVE-16600.1.patch, HIVE-16600.2.patch, 
> HIVE-16600.3.patch, HIVE-16600.4.patch, HIVE-16600.5.patch, 
> HIVE-16600.6.patch, HIVE-16600.7.patch, mr.explain, mr.explain.log.HIVE-16600
>
>
> multi_insert_gby.case.q
> {code}
> set hive.exec.reducers.bytes.per.reducer=256;
> set hive.optimize.sampling.orderby=true;
> drop table if exists e1;
> drop table if exists e2;
> create table e1 (key string, value string);
> create table e2 (key string);
> FROM (select key, cast(key as double) as keyD, value from src order by key) a
> INSERT OVERWRITE TABLE e1
> SELECT key, value
> INSERT OVERWRITE TABLE e2
> SELECT key;
> select * from e1;
> select * from e2;
> {code} 
> the parallelism of Sort is 1 even we enable parallel order 
> by("hive.optimize.sampling.orderby" is set as "true").  This is not 
> reasonable because the parallelism  should be calcuated by  
> [Utilities.estimateReducers|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L170]
> this is because SetSparkReducerParallelism#needSetParallelism returns false 
> when [children size of 
> RS|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L207]
>  is greater than 1.
> in this case, the children size of {{RS[2]}} is two.
> the logical plan of the case
> {code}
>TS[0]-SEL[1]-RS[2]-SEL[3]-SEL[4]-FS[5]
> -SEL[6]-FS[7]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)