[jira] [Assigned] (HIVE-2340) optimize orderby followed by a groupby
[ https://issues.apache.org/jira/browse/HIVE-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K reassigned HIVE-2340: Assignee: Navis (was: zgl) > optimize orderby followed by a groupby > -- > > Key: HIVE-2340 > URL: https://issues.apache.org/jira/browse/HIVE-2340 > Project: Hive > Issue Type: Sub-task > Components: Query Processor >Reporter: Navis >Assignee: Navis >Priority: Minor > Labels: perfomance > Fix For: 0.11.0 > > Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-2340.D1209.1.patch, > ASF.LICENSE.NOT.GRANTED--HIVE-2340.D1209.2.patch, > ASF.LICENSE.NOT.GRANTED--HIVE-2340.D1209.3.patch, > ASF.LICENSE.NOT.GRANTED--HIVE-2340.D1209.4.patch, > ASF.LICENSE.NOT.GRANTED--HIVE-2340.D1209.5.patch, HIVE-2340.1.patch.txt, > HIVE-2340.12.patch, HIVE-2340.13.patch, HIVE-2340.14.patch, > HIVE-2340.14.rebased_and_schema_clone.patch, HIVE-2340.15.patch, > HIVE-2340.D1209.10.patch, HIVE-2340.D1209.11.patch, HIVE-2340.D1209.12.patch, > HIVE-2340.D1209.13.patch, HIVE-2340.D1209.14.patch, HIVE-2340.D1209.15.patch, > HIVE-2340.D1209.6.patch, HIVE-2340.D1209.7.patch, HIVE-2340.D1209.8.patch, > HIVE-2340.D1209.9.patch, testclidriver.txt > > > Before implementing optimizer for JOIN-GBY, try to implement RS-GBY > optimizer(cluster-by following group-by). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-3952) merge map-job followed by map-reduce job
[ https://issues.apache.org/jira/browse/HIVE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K reassigned HIVE-3952: Assignee: Vinod Kumar Vavilapalli (was: zgl) > merge map-job followed by map-reduce job > > > Key: HIVE-3952 > URL: https://issues.apache.org/jira/browse/HIVE-3952 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Namit Jain >Assignee: Vinod Kumar Vavilapalli >Priority: Major > Fix For: 0.11.0 > > Attachments: HIVE-3952-20130226.txt, HIVE-3952-20130227.1.txt, > HIVE-3952-20130301.txt, HIVE-3952-20130421.txt, HIVE-3952-20130424.txt, > HIVE-3952-20130428-branch-0.11-bugfix.txt, > HIVE-3952-20130428-branch-0.11-v2.txt, HIVE-3952-20130428-branch-0.11.txt, > hive.3952.1.patch > > > Consider the query like: > select count（*）FROM > ( select idOne, idTwo, value FROM > bigTable > JOIN > > smallTableOne on (bigTable.idOne = smallTableOne.idOne) > > ) firstjoin > > JOIN > > smallTableTwo on (firstjoin.idTwo = smallTableTwo.idTwo); > where smallTableOne and smallTableTwo are smaller than > hive.auto.convert.join.noconditionaltask.size and > hive.auto.convert.join.noconditionaltask is set to true. > The joins are collapsed into mapjoins, and it leads to a map-only job > (for the map-joins) followed by a map-reduce job (for the group by). > Ideally, the map-only job should be merged with the following map-reduce job. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-4078) Delay the serialize-deserialize pair in CommonJoinTaskDispatcher
[
https://issues.apache.org/jira/browse/HIVE-4078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikram Dixit K reassigned HIVE-4078:
Assignee: Gopal V (was: zgl)
> Delay the serialize-deserialize pair in CommonJoinTaskDispatcher
>
>
> Key: HIVE-4078
> URL: https://issues.apache.org/jira/browse/HIVE-4078
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Labels: client, perfomance
> Fix For: 0.11.0
>
> Attachments: HIVE-4078-20130305.2.patch, HIVE-4078-20130305.patch,
> HIVE-4078-20130406.patch, HIVE-4078-trunk-rebase.patch
>
>
> CommonJoinProcessor tries to clone a MapredWork while attempting a conversion
> to a map-join
> {code}
> // deep copy a new mapred work from xml
> InputStream in = new ByteArrayInputStream(xml.getBytes("UTF-8"));
> MapredWork newWork = Utilities.deserializeMapRedWork(in,
> physicalContext.getConf());
> {code}
> which is a very heavy operation memory wise & cpu-wise.
> It would be better to do this only if a conditional task is required,
> resulting in a copy of the task.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K reassigned HIVE-3891: Assignee: Namit Jain (was: zgl) > physical optimizer changes for auto sort-merge join > --- > > Key: HIVE-3891 > URL: https://issues.apache.org/jira/browse/HIVE-3891 > Project: Hive > Issue Type: Bug >Reporter: Namit Jain >Assignee: Namit Jain >Priority: Major > Labels: TODOC12 > Fix For: 0.12.0 > > Attachments: HIVE-3891_8.patch, auto_sortmerge_join_1.q, > auto_sortmerge_join_1.q.out, hive.3891.1.patch, hive.3891.10.patch, > hive.3891.11.patch, hive.3891.12.patch, hive.3891.13.patch, > hive.3891.14.patch, hive.3891.15.patch, hive.3891.16.patch, > hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, > hive.3891.6.patch, hive.3891.7.patch, hive.3891.9.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-4071) Map-join outer join produces incorrect results.
[ https://issues.apache.org/jira/browse/HIVE-4071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K reassigned HIVE-4071: Assignee: Vikram Dixit K (was: zgl) > Map-join outer join produces incorrect results. > --- > > Key: HIVE-4071 > URL: https://issues.apache.org/jira/browse/HIVE-4071 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.11.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K >Priority: Major > Fix For: 0.11.0 > > Attachments: HIVE-4071.patch, HIVE-4071_2.patch, HIVE-4071_3.patch > > > For example, if one sets the size of noConditionalTask.size to 10 with > corresponding auto join configurations set to true in auto_join28.q instead > of the current smalltable.filesize configuration, we will observe different > results if a select query is run. (The test only has explain statements at > present). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-3996) Correctly enforce the memory limit on the multi-table map-join
[ https://issues.apache.org/jira/browse/HIVE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K reassigned HIVE-3996: Assignee: Vikram Dixit K (was: zgl) > Correctly enforce the memory limit on the multi-table map-join > -- > > Key: HIVE-3996 > URL: https://issues.apache.org/jira/browse/HIVE-3996 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 0.11.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K >Priority: Major > Fix For: 0.11.0 > > Attachments: HIVE-3996.patch, HIVE-3996_2.patch, HIVE-3996_3.patch, > HIVE-3996_4.patch, HIVE-3996_5.patch, HIVE-3996_6.patch, HIVE-3996_7.patch, > HIVE-3996_8.patch, HIVE-3996_9.patch, hive.3996.9.patch-nohcat > > > Currently with HIVE-3784, the joins are converted to map-joins based on > checks of the table size against the config variable: > hive.auto.convert.join.noconditionaltask.size. > However, the current implementation will also merge multiple mapjoin > operators into a single task regardless of whether the sum of the table sizes > will exceed the configured value. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-13518) Hive on Tez: Shuffle joins do not choose the right 'big' table.
[ https://issues.apache.org/jira/browse/HIVE-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13518: -- Attachment: HIVE-13518.3.patch > Hive on Tez: Shuffle joins do not choose the right 'big' table. > --- > > Key: HIVE-13518 > URL: https://issues.apache.org/jira/browse/HIVE-13518 > Project: Hive > Issue Type: Bug >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13518.1.patch, HIVE-13518.2.patch, > HIVE-13518.3.patch > > > Currently the big table is always assumed to be at position 0 but this isn't > efficient for some queries as the big table at position 1 could have a lot > more keys/skew. We already have a mechanism of choosing the big table that > can be leveraged to make the right choice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13282) GroupBy and select operator encounter ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-13282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikram Dixit K updated HIVE-13282:
--
Attachment: smb_fail_issue.patch
> GroupBy and select operator encounter ArrayIndexOutOfBoundsException
>
>
> Key: HIVE-13282
> URL: https://issues.apache.org/jira/browse/HIVE-13282
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.1, 2.0.0, 2.1.0
>Reporter: Vikram Dixit K
>Assignee: Matt McCline
> Attachments: smb_fail_issue.patch, smb_groupby.q, smb_groupby.q.out
>
>
> The group by and select operators run into the ArrayIndexOutOfBoundsException
> when they incorrectly initialize themselves with tag 0 but the incoming tag
> id is different.
> {code}
> select count(*) from
> (select rt1.id from
> (select t1.key as id, t1.value as od from tab t1 group by key, value) rt1) vt1
> join
> (select rt2.id from
> (select t2.key as id, t2.value as od from tab_part t2 group by key, value)
> rt2) vt2
> where vt1.id=vt2.id;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13282) GroupBy and select operator encounter ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-13282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikram Dixit K updated HIVE-13282:
--
Attachment: (was: smb_fail_issue.patch)
> GroupBy and select operator encounter ArrayIndexOutOfBoundsException
>
>
> Key: HIVE-13282
> URL: https://issues.apache.org/jira/browse/HIVE-13282
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.1, 2.0.0, 2.1.0
>Reporter: Vikram Dixit K
>Assignee: Matt McCline
> Attachments: smb_fail_issue.patch, smb_groupby.q, smb_groupby.q.out
>
>
> The group by and select operators run into the ArrayIndexOutOfBoundsException
> when they incorrectly initialize themselves with tag 0 but the incoming tag
> id is different.
> {code}
> select count(*) from
> (select rt1.id from
> (select t1.key as id, t1.value as od from tab t1 group by key, value) rt1) vt1
> join
> (select rt2.id from
> (select t2.key as id, t2.value as od from tab_part t2 group by key, value)
> rt2) vt2
> where vt1.id=vt2.id;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13282) GroupBy and select operator encounter ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-13282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15298795#comment-15298795
]
Vikram Dixit K commented on HIVE-13282:
---
[~mmccline] Can you try the patch I have attached above? I think that is how I
was able to repro the issue. I think with the patch attached here, it produces
a result sometimes but that is wrong (if you switch around the tables, it may
even throw an exception).
> GroupBy and select operator encounter ArrayIndexOutOfBoundsException
>
>
> Key: HIVE-13282
> URL: https://issues.apache.org/jira/browse/HIVE-13282
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.1, 2.0.0, 2.1.0
>Reporter: Vikram Dixit K
>Assignee: Matt McCline
> Attachments: smb_fail_issue.patch, smb_groupby.q, smb_groupby.q.out
>
>
> The group by and select operators run into the ArrayIndexOutOfBoundsException
> when they incorrectly initialize themselves with tag 0 but the incoming tag
> id is different.
> {code}
> select count(*) from
> (select rt1.id from
> (select t1.key as id, t1.value as od from tab t1 group by key, value) rt1) vt1
> join
> (select rt2.id from
> (select t2.key as id, t2.value as od from tab_part t2 group by key, value)
> rt2) vt2
> where vt1.id=vt2.id;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13282) GroupBy and select operator encounter ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-13282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikram Dixit K updated HIVE-13282:
--
Attachment: smb_fail_issue.patch
> GroupBy and select operator encounter ArrayIndexOutOfBoundsException
>
>
> Key: HIVE-13282
> URL: https://issues.apache.org/jira/browse/HIVE-13282
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.1, 2.0.0, 2.1.0
>Reporter: Vikram Dixit K
>Assignee: Matt McCline
> Attachments: smb_fail_issue.patch, smb_groupby.q, smb_groupby.q.out
>
>
> The group by and select operators run into the ArrayIndexOutOfBoundsException
> when they incorrectly initialize themselves with tag 0 but the incoming tag
> id is different.
> {code}
> select count(*) from
> (select rt1.id from
> (select t1.key as id, t1.value as od from tab t1 group by key, value) rt1) vt1
> join
> (select rt2.id from
> (select t2.key as id, t2.value as od from tab_part t2 group by key, value)
> rt2) vt2
> where vt1.id=vt2.id;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13518) Hive on Tez: Shuffle joins do not choose the right 'big' table.
[ https://issues.apache.org/jira/browse/HIVE-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15298769#comment-15298769 ] Vikram Dixit K commented on HIVE-13518: --- Yes. The test failures are related. I will be posting an update for addressing them. > Hive on Tez: Shuffle joins do not choose the right 'big' table. > --- > > Key: HIVE-13518 > URL: https://issues.apache.org/jira/browse/HIVE-13518 > Project: Hive > Issue Type: Bug >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13518.1.patch > > > Currently the big table is always assumed to be at position 0 but this isn't > efficient for some queries as the big table at position 1 could have a lot > more keys/skew. We already have a mechanism of choosing the big table that > can be leveraged to make the right choice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13832) Add missing license header to files
[ https://issues.apache.org/jira/browse/HIVE-13832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15298662#comment-15298662 ] Vikram Dixit K commented on HIVE-13832: --- +1 > Add missing license header to files > --- > > Key: HIVE-13832 > URL: https://issues.apache.org/jira/browse/HIVE-13832 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Vikram Dixit K > Attachments: HIVE-13832.1.patch, HIVE-13832.2.patch, HIVE-13832.patch > > > Preparing to cut the branch for 2.1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13832) Add missing license header to files
[ https://issues.apache.org/jira/browse/HIVE-13832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15298644#comment-15298644 ] Vikram Dixit K commented on HIVE-13832: --- I think in that case, a change in bin.xml is missing in the original patch. That would work too! > Add missing license header to files > --- > > Key: HIVE-13832 > URL: https://issues.apache.org/jira/browse/HIVE-13832 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Vikram Dixit K > Attachments: HIVE-13832.1.patch, HIVE-13832.patch > > > Preparing to cut the branch for 2.1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13832) Add missing license header to files
[ https://issues.apache.org/jira/browse/HIVE-13832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13832: -- Attachment: HIVE-13832.1.patch Unfortunately, ambari project depends on a specific location of this script. I have added a specific exclude for the file. If in case in the future, we end up having more such sql scripts, we can exclude that directory. > Add missing license header to files > --- > > Key: HIVE-13832 > URL: https://issues.apache.org/jira/browse/HIVE-13832 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Vikram Dixit K > Attachments: HIVE-13832.1.patch, HIVE-13832.patch > > > Preparing to cut the branch for 2.1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13832) Add missing license header to files
[ https://issues.apache.org/jira/browse/HIVE-13832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13832: -- Assignee: Vikram Dixit K (was: Jesus Camacho Rodriguez) > Add missing license header to files > --- > > Key: HIVE-13832 > URL: https://issues.apache.org/jira/browse/HIVE-13832 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Vikram Dixit K > Attachments: HIVE-13832.patch > > > Preparing to cut the branch for 2.1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13628) Support for permanent functions - error handling if no restart
[ https://issues.apache.org/jira/browse/HIVE-13628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13628: -- Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed to master. Thanks Gunther for the review. > Support for permanent functions - error handling if no restart > -- > > Key: HIVE-13628 > URL: https://issues.apache.org/jira/browse/HIVE-13628 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Fix For: 2.1.0 > > Attachments: HIVE-13628.1.patch, HIVE-13628.2.patch, > HIVE-13628.3.patch, HIVE-13628.4.patch > > > Support for permanent functions - error handling if no restart -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13628) Support for permanent functions - error handling if no restart
[ https://issues.apache.org/jira/browse/HIVE-13628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13628: -- Attachment: HIVE-13628.4.patch [~hagleitn] review please. > Support for permanent functions - error handling if no restart > -- > > Key: HIVE-13628 > URL: https://issues.apache.org/jira/browse/HIVE-13628 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13628.1.patch, HIVE-13628.2.patch, > HIVE-13628.3.patch, HIVE-13628.4.patch > > > Support for permanent functions - error handling if no restart -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13628) Support for permanent functions - error handling if no restart
[ https://issues.apache.org/jira/browse/HIVE-13628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13628: -- Attachment: HIVE-13628.3.patch > Support for permanent functions - error handling if no restart > -- > > Key: HIVE-13628 > URL: https://issues.apache.org/jira/browse/HIVE-13628 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13628.1.patch, HIVE-13628.2.patch, > HIVE-13628.3.patch > > > Support for permanent functions - error handling if no restart -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13628) Support for permanent functions - error handling if no restart
[ https://issues.apache.org/jira/browse/HIVE-13628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13628: -- Attachment: (was: HIVE-13628.3.patch) > Support for permanent functions - error handling if no restart > -- > > Key: HIVE-13628 > URL: https://issues.apache.org/jira/browse/HIVE-13628 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13628.1.patch, HIVE-13628.2.patch > > > Support for permanent functions - error handling if no restart -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13628) Support for permanent functions - error handling if no restart
[ https://issues.apache.org/jira/browse/HIVE-13628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13628: -- Attachment: HIVE-13628.3.patch > Support for permanent functions - error handling if no restart > -- > > Key: HIVE-13628 > URL: https://issues.apache.org/jira/browse/HIVE-13628 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13628.1.patch, HIVE-13628.2.patch, > HIVE-13628.3.patch > > > Support for permanent functions - error handling if no restart -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13191) DummyTable map joins mix up columns between tables
[
https://issues.apache.org/jira/browse/HIVE-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikram Dixit K updated HIVE-13191:
--
Attachment: tez.q
Along with this addition
{code}
diff --git a/itests/src/test/resources/testconfiguration.properties
b/itests/src/test/resources/testconfiguration.properties
index c891d40..a499b0a 100644
--- a/itests/src/test/resources/testconfiguration.properties
+++ b/itests/src/test/resources/testconfiguration.properties
@@ -70,6 +70,7 @@ disabled.query.files=ql_rewrite_gbtoidx.q,\
smb_mapjoin_8.q
minitez.query.files.shared=acid_globallimit.q,\
+ tez.q,\
deleteAnalyze.q,\
empty_join.q,\
alter_merge_2_orc.q,\
{code}
> DummyTable map joins mix up columns between tables
> --
>
> Key: HIVE-13191
> URL: https://issues.apache.org/jira/browse/HIVE-13191
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Gopal V
>Assignee: Vikram Dixit K
> Attachments: tez.q
>
>
> {code}
> SELECT
> a.key,
> a.a_one,
> b.b_one,
> a.a_zero,
> b.b_zero
> FROM
> (
> SELECT
> 11 key,
> 0 confuse_you,
> 1 a_one,
> 0 a_zero
> ) a
> LEFT JOIN
> (
> SELECT
> 11 key,
> 0 confuse_you,
> 1 b_one,
> 0 b_zero
> ) b
> ON a.key = b.key
> ;
> 11 1 0 0 1
> {code}
> This should be 11, 1, 1, 0, 0 instead.
> Disabling map-joins & using shuffle-joins returns the right result.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13343) Need to disable hybrid grace hash join in llap mode except for dynamically partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13343: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Need to disable hybrid grace hash join in llap mode except for dynamically > partitioned hash join > > > Key: HIVE-13343 > URL: https://issues.apache.org/jira/browse/HIVE-13343 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13343.1.patch, HIVE-13343.2.patch, > HIVE-13343.3.patch, HIVE-13343.4.patch, HIVE-13343.5.patch, > HIVE-13343.6.patch, HIVE-13343.7.patch > > > Due to performance reasons, we should disable use of hybrid grace hash join > in llap when dynamic partition hash join is not used. With dynamic partition > hash join, we need hybrid grace hash join due to the possibility of skews. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13730) hybridgrace_hashjoin_1.q test gets stuck
[ https://issues.apache.org/jira/browse/HIVE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289292#comment-15289292 ] Vikram Dixit K commented on HIVE-13730: --- +1 > hybridgrace_hashjoin_1.q test gets stuck > > > Key: HIVE-13730 > URL: https://issues.apache.org/jira/browse/HIVE-13730 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Wei Zheng >Priority: Blocker > Attachments: HIVE-13730.1.patch, HIVE-13730.2.patch, > HIVE-13730.3.patch, HIVE-13730.4.patch > > > I am seeing hybridgrace_hashjoin_1.q getting stuck on master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13343) Need to disable hybrid grace hash join in llap mode except for dynamically partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284939#comment-15284939 ] Vikram Dixit K commented on HIVE-13343: --- Test failures unrelated. > Need to disable hybrid grace hash join in llap mode except for dynamically > partitioned hash join > > > Key: HIVE-13343 > URL: https://issues.apache.org/jira/browse/HIVE-13343 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13343.1.patch, HIVE-13343.2.patch, > HIVE-13343.3.patch, HIVE-13343.4.patch, HIVE-13343.5.patch, > HIVE-13343.6.patch, HIVE-13343.7.patch > > > Due to performance reasons, we should disable use of hybrid grace hash join > in llap when dynamic partition hash join is not used. With dynamic partition > hash join, we need hybrid grace hash join due to the possibility of skews. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13755) Hybrid mapjoin allocates memory the same for multi broadcast
[
https://issues.apache.org/jira/browse/HIVE-13755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikram Dixit K updated HIVE-13755:
--
Priority: Critical (was: Major)
> Hybrid mapjoin allocates memory the same for multi broadcast
>
>
> Key: HIVE-13755
> URL: https://issues.apache.org/jira/browse/HIVE-13755
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
>Priority: Critical
>
> PROBLEM:
> When hybrid mapjoin gets the memory needed, it estimates memory needed for
> each hashtable the same. This may cause problem when there are multiple
> broadcast, as it may exceeds the memory intended to allocate to it.
> Example reducer task log attached. This task has 5 broadcast input,
> Reducer 3 <- Map 10 (BROADCAST_EDGE), Map 11 (BROADCAST_EDGE), Map 12
> (BROADCAST_EDGE), Map 8 (SIMPLE_EDGE), Map 9 (BROADCAST_EDGE), Reducer 2
> (SIMPLE_EDGE)
> excerpt of it:
> {code}
> 2016-03-15 19:23:50,811 [INFO] [pool-47-thread-1] |tez.HashTableLoader|:
> Memory manager allocates 0 bytes for the loading hashtable.
> 2016-03-15 19:23:50,811 [INFO] [pool-47-thread-1]
> |persistence.HashMapWrapper|: Key count from statistics is 210; setting map
> size to 280
> 2016-03-15 19:23:50,811 [INFO] [pool-47-thread-1]
> |persistence.HybridHashTableContainer|: Total available memory: 1968177152
> 2016-03-15 19:23:50,812 [INFO] [pool-47-thread-1]
> |persistence.HybridHashTableContainer|: Estimated small table size: 155190
> 2016-03-15 19:23:50,812 [INFO] [pool-47-thread-1]
> |persistence.HybridHashTableContainer|: Number of hash partitions to be
> created: 16
> 2016-03-15 19:23:50,812 [INFO] [pool-47-thread-1]
> |persistence.HybridHashTableContainer|: Write buffer size: 524288
> 2016-03-15 19:23:50,812 [INFO] [pool-47-thread-1]
> |persistence.HybridHashTableContainer|: Number of partitions created: 16
> 2016-03-15 19:23:50,812 [INFO] [pool-47-thread-1]
> |persistence.HybridHashTableContainer|: Number of partitions spilled directly
> to disk on creation: 0
> 2016-03-15 19:23:50,812 [INFO] [pool-47-thread-1] |tez.HashTableLoader|:
> Using tableContainer HybridHashTableContainer
> 2016-03-15 19:23:50,812 [INFO] [pool-47-thread-1]
> |persistence.HybridHashTableContainer|: Initializing container with
> org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe and
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
> 2016-03-15 19:23:50,814 [INFO] [pool-47-thread-1]
> |readers.UnorderedKVReader|: Num Records read: 20
> 2016-03-15 19:23:50,814 [INFO] [pool-47-thread-1] |log.PerfLogger|: method=LoadHashtable start=1458069830811 end=1458069830814 duration=3
> from=org.apache.hadoop.hive.ql.exec.MapJoinOperator>
> 2016-03-15 19:23:50,814 [INFO] [pool-47-thread-1] |tez.ObjectCache|: Caching
> key:
> svc-phx-efmhadoop_20160315191303_8c53ce88-e64f-4d36-bad0-846bbf096f57__HASH_MAP_MAPJOIN_126_container
> 2016-03-15 19:23:50,814 [INFO] [TezChild] |exec.HashTableDummyOperator|:
> Initializing operator HASHTABLEDUMMY[32]
> 2016-03-15 19:23:50,814 [INFO] [TezChild] |exec.MapJoinOperator|:
> Initializing operator MAPJOIN[26]
> 2016-03-15 19:23:50,816 [INFO] [TezChild] |exec.CommonJoinOperator|: JOIN
>
[jira] [Updated] (HIVE-13628) Support for permanent functions - error handling if no restart
[ https://issues.apache.org/jira/browse/HIVE-13628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13628: -- Status: Patch Available (was: Open) > Support for permanent functions - error handling if no restart > -- > > Key: HIVE-13628 > URL: https://issues.apache.org/jira/browse/HIVE-13628 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13628.1.patch, HIVE-13628.2.patch > > > Support for permanent functions - error handling if no restart -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13343) Need to disable hybrid grace hash join in llap mode except for dynamically partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13343: -- Attachment: HIVE-13343.7.patch > Need to disable hybrid grace hash join in llap mode except for dynamically > partitioned hash join > > > Key: HIVE-13343 > URL: https://issues.apache.org/jira/browse/HIVE-13343 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13343.1.patch, HIVE-13343.2.patch, > HIVE-13343.3.patch, HIVE-13343.4.patch, HIVE-13343.5.patch, > HIVE-13343.6.patch, HIVE-13343.7.patch > > > Due to performance reasons, we should disable use of hybrid grace hash join > in llap when dynamic partition hash join is not used. With dynamic partition > hash join, we need hybrid grace hash join due to the possibility of skews. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13730) hybridgrace_hashjoin_1.q test gets stuck
[ https://issues.apache.org/jira/browse/HIVE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13730: -- Priority: Blocker (was: Critical) > hybridgrace_hashjoin_1.q test gets stuck > > > Key: HIVE-13730 > URL: https://issues.apache.org/jira/browse/HIVE-13730 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Wei Zheng >Priority: Blocker > > I am seeing hybridgrace_hashjoin_1.q getting stuck on master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13342) Improve logging in llap decider and throw exception in case llap mode is all but we cannot run in llap.
[ https://issues.apache.org/jira/browse/HIVE-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13342: -- Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed to master. > Improve logging in llap decider and throw exception in case llap mode is all > but we cannot run in llap. > --- > > Key: HIVE-13342 > URL: https://issues.apache.org/jira/browse/HIVE-13342 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Fix For: 2.1.0 > > Attachments: HIVE-13342.1.patch, HIVE-13342.2.patch, > HIVE-13342.3.patch, HIVE-13342.4.patch, HIVE-13342.5.patch > > > Currently we do not log our decisions with respect to llap. Are we running > everything in llap mode or only parts of the plan. We need more logging. > Also, if llap mode is all but for some reason, we cannot run the work in llap > mode, fail and throw an exception advise the user to change the mode to auto. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13342) Improve logging in llap decider and throw exception in case llap mode is all but we cannot run in llap.
[
https://issues.apache.org/jira/browse/HIVE-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278886#comment-15278886
]
Vikram Dixit K commented on HIVE-13342:
---
The test failures are not related to this change. I ran the test successfully
on my setup.
{code}
Running org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
Tests run: 27, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 604.698 sec -
in org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
Results :
Tests run: 27, Failures: 0, Errors: 0, Skipped: 0
{code}
> Improve logging in llap decider and throw exception in case llap mode is all
> but we cannot run in llap.
> ---
>
> Key: HIVE-13342
> URL: https://issues.apache.org/jira/browse/HIVE-13342
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.1.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-13342.1.patch, HIVE-13342.2.patch,
> HIVE-13342.3.patch, HIVE-13342.4.patch, HIVE-13342.5.patch
>
>
> Currently we do not log our decisions with respect to llap. Are we running
> everything in llap mode or only parts of the plan. We need more logging.
> Also, if llap mode is all but for some reason, we cannot run the work in llap
> mode, fail and throw an exception advise the user to change the mode to auto.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (HIVE-13342) Improve logging in llap decider and throw exception in case llap mode is all but we cannot run in llap.
[ https://issues.apache.org/jira/browse/HIVE-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13342: -- Comment: was deleted (was: Updated using java 7.) > Improve logging in llap decider and throw exception in case llap mode is all > but we cannot run in llap. > --- > > Key: HIVE-13342 > URL: https://issues.apache.org/jira/browse/HIVE-13342 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13342.1.patch, HIVE-13342.2.patch, > HIVE-13342.3.patch, HIVE-13342.4.patch, HIVE-13342.5.patch > > > Currently we do not log our decisions with respect to llap. Are we running > everything in llap mode or only parts of the plan. We need more logging. > Also, if llap mode is all but for some reason, we cannot run the work in llap > mode, fail and throw an exception advise the user to change the mode to auto. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13342) Improve logging in llap decider and throw exception in case llap mode is all but we cannot run in llap.
[ https://issues.apache.org/jira/browse/HIVE-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13342: -- Attachment: HIVE-13343.6.patch Updated using java 7. > Improve logging in llap decider and throw exception in case llap mode is all > but we cannot run in llap. > --- > > Key: HIVE-13342 > URL: https://issues.apache.org/jira/browse/HIVE-13342 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13342.1.patch, HIVE-13342.2.patch, > HIVE-13342.3.patch, HIVE-13342.4.patch, HIVE-13342.5.patch > > > Currently we do not log our decisions with respect to llap. Are we running > everything in llap mode or only parts of the plan. We need more logging. > Also, if llap mode is all but for some reason, we cannot run the work in llap > mode, fail and throw an exception advise the user to change the mode to auto. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13343) Need to disable hybrid grace hash join in llap mode except for dynamically partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13343: -- Attachment: HIVE-13343.6.patch > Need to disable hybrid grace hash join in llap mode except for dynamically > partitioned hash join > > > Key: HIVE-13343 > URL: https://issues.apache.org/jira/browse/HIVE-13343 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13343.1.patch, HIVE-13343.2.patch, > HIVE-13343.3.patch, HIVE-13343.4.patch, HIVE-13343.5.patch, HIVE-13343.6.patch > > > Due to performance reasons, we should disable use of hybrid grace hash join > in llap when dynamic partition hash join is not used. With dynamic partition > hash join, we need hybrid grace hash join due to the possibility of skews. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13342) Improve logging in llap decider and throw exception in case llap mode is all but we cannot run in llap.
[ https://issues.apache.org/jira/browse/HIVE-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13342: -- Attachment: (was: HIVE-13343.6.patch) > Improve logging in llap decider and throw exception in case llap mode is all > but we cannot run in llap. > --- > > Key: HIVE-13342 > URL: https://issues.apache.org/jira/browse/HIVE-13342 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13342.1.patch, HIVE-13342.2.patch, > HIVE-13342.3.patch, HIVE-13342.4.patch, HIVE-13342.5.patch > > > Currently we do not log our decisions with respect to llap. Are we running > everything in llap mode or only parts of the plan. We need more logging. > Also, if llap mode is all but for some reason, we cannot run the work in llap > mode, fail and throw an exception advise the user to change the mode to auto. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13343) Need to disable hybrid grace hash join in llap mode except for dynamically partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278753#comment-15278753 ] Vikram Dixit K commented on HIVE-13343: --- Hmm.. It looks like the hybrid_grace_join tests are getting stuck and probably timing out the tez tests. > Need to disable hybrid grace hash join in llap mode except for dynamically > partitioned hash join > > > Key: HIVE-13343 > URL: https://issues.apache.org/jira/browse/HIVE-13343 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13343.1.patch, HIVE-13343.2.patch, > HIVE-13343.3.patch, HIVE-13343.4.patch, HIVE-13343.5.patch > > > Due to performance reasons, we should disable use of hybrid grace hash join > in llap when dynamic partition hash join is not used. With dynamic partition > hash join, we need hybrid grace hash join due to the possibility of skews. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13621) compute stats in certain cases fails with NPE
[
https://issues.apache.org/jira/browse/HIVE-13621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278431#comment-15278431
]
Vikram Dixit K commented on HIVE-13621:
---
+1 pending tests.
> compute stats in certain cases fails with NPE
> -
>
> Key: HIVE-13621
> URL: https://issues.apache.org/jira/browse/HIVE-13621
> Project: Hive
> Issue Type: Bug
> Components: HBase Metastore, Metastore
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Vikram Dixit K
>Assignee: Pengcheng Xiong
> Attachments: HIVE-13621.1.patch, HIVE-13621.2.patch
>
>
> {code}
> FAILED: NullPointerException null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getColStatistics(StatsUtils.java:693)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.convertColStats(StatsUtils.java:739)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getTableColumnStats(StatsUtils.java:728)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:183)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:136)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:124){code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13343) Need to disable hybrid grace hash join in llap mode except for dynamically partitioned hash join
[
https://issues.apache.org/jira/browse/HIVE-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15276909#comment-15276909
]
Vikram Dixit K commented on HIVE-13343:
---
It looks like many of the diffs are about slightly varying explain plans. I
don't believe they are because of my changes.
{code}
-COLUMN_STATS_ACCURATE
{"COLUMN_STATS":{"key":"true","value":"true"},"BASIC_STATS":"true"}
+COLUMN_STATS_ACCURATE
{"BASIC_STATS":"true","COLUMN_STATS":{"key":"true","value":"true"}}
{code}
> Need to disable hybrid grace hash join in llap mode except for dynamically
> partitioned hash join
>
>
> Key: HIVE-13343
> URL: https://issues.apache.org/jira/browse/HIVE-13343
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.1.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-13343.1.patch, HIVE-13343.2.patch,
> HIVE-13343.3.patch, HIVE-13343.4.patch, HIVE-13343.5.patch
>
>
> Due to performance reasons, we should disable use of hybrid grace hash join
> in llap when dynamic partition hash join is not used. With dynamic partition
> hash join, we need hybrid grace hash join due to the possibility of skews.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13628) Support for permanent functions - error handling if no restart
[ https://issues.apache.org/jira/browse/HIVE-13628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13628: -- Attachment: HIVE-13628.2.patch One more iteration addressing the minor comments. > Support for permanent functions - error handling if no restart > -- > > Key: HIVE-13628 > URL: https://issues.apache.org/jira/browse/HIVE-13628 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13628.1.patch, HIVE-13628.2.patch > > > Support for permanent functions - error handling if no restart -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13656) need to set direct memory limit higher in LlapServiceDriver for certain edge case configurations
[ https://issues.apache.org/jira/browse/HIVE-13656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15273320#comment-15273320 ] Vikram Dixit K commented on HIVE-13656: --- After talking with Sergey, it looks like this is what we have to do for max direct memory to be used properly. +1 for the patch. > need to set direct memory limit higher in LlapServiceDriver for certain edge > case configurations > > > Key: HIVE-13656 > URL: https://issues.apache.org/jira/browse/HIVE-13656 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13656.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13656) need to set direct memory limit higher in LlapServiceDriver for certain edge case configurations
[
https://issues.apache.org/jira/browse/HIVE-13656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15273319#comment-15273319
]
Vikram Dixit K commented on HIVE-13656:
---
Don't you want this
{code}
long maxDirect = (xmx > 0 && cache > 0 && xmx < cache * 1.25) ? (long)(cache *
1.25) : -1;
{code}
to be:
{code}
long maxDirect = (xmx > 0 && cache > 0 && xmx < cache * 1.25) ? (long)(cache *
1.25) : xmx;
{code}
> need to set direct memory limit higher in LlapServiceDriver for certain edge
> case configurations
>
>
> Key: HIVE-13656
> URL: https://issues.apache.org/jira/browse/HIVE-13656
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13656.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13619) Bucket map join plan is incorrect
[ https://issues.apache.org/jira/browse/HIVE-13619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13619: -- Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) > Bucket map join plan is incorrect > - > > Key: HIVE-13619 > URL: https://issues.apache.org/jira/browse/HIVE-13619 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.0.0, 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Fix For: 2.1.0 > > Attachments: HIVE-13619.1.patch > > > Same as HIVE-12992. Missed a single line check. TPCDS query 4 with bucketing > can produce this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12837) Better memory estimation/allocation for hybrid grace hash join during hash table loading
[ https://issues.apache.org/jira/browse/HIVE-12837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271764#comment-15271764 ] Vikram Dixit K commented on HIVE-12837: --- +1 LGTM. > Better memory estimation/allocation for hybrid grace hash join during hash > table loading > > > Key: HIVE-12837 > URL: https://issues.apache.org/jira/browse/HIVE-12837 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-12837.1.patch, HIVE-12837.2.patch, > HIVE-12837.3.patch, HIVE-12837.4.patch, HIVE-12837.5.patch > > > This is to avoid an edge case when the memory available is very little (less > than a single write buffer size), and we start loading the hash table. Since > the write buffer is lazily allocated, we will easily run out of memory before > even checking if we should spill any hash partition. > e.g. > Total memory available: 210 MB > Size of ref array of BytesBytesMultiHashMap for each hash partition: ~16 MB > Size of write buffer: 8 MB (lazy allocation) > Number of hash partitions: 16 > Number of hash partitions created in memory: 13 > Number of hash partitions created on disk: 3 > Available memory left after HybridHashTableContainer initialization: > 210-16*13=2MB > Now let's say a row is to be loaded into a hash partition in memory, it will > try to allocate an 8MB write buffer for it, but we only have 2MB, thus OOM. > Solution is to perform the check for possible spilling earlier so we can > spill partitions if memory is about to be full, to avoid OOM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13620) Merge llap branch work to master
[ https://issues.apache.org/jira/browse/HIVE-13620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269831#comment-15269831 ] Vikram Dixit K commented on HIVE-13620: --- +1. > Merge llap branch work to master > > > Key: HIVE-13620 > URL: https://issues.apache.org/jira/browse/HIVE-13620 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-13620.1.patch, HIVE-13620.2.patch, > HIVE-13620.3.patch, HIVE-13620.4.patch, HIVE-13620.5.patch, > HIVE-13620.6.patch, llap_master_diff.txt > > > Would like to try to merge the llap branch work for HIVE-12991 into the > master branch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13342) Improve logging in llap decider and throw exception in case llap mode is all but we cannot run in llap.
[ https://issues.apache.org/jira/browse/HIVE-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13342: -- Attachment: HIVE-13342.5.patch The new logs captured which operator failed/passed the check. I modified it so that the log is printed only in case of failure. > Improve logging in llap decider and throw exception in case llap mode is all > but we cannot run in llap. > --- > > Key: HIVE-13342 > URL: https://issues.apache.org/jira/browse/HIVE-13342 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13342.1.patch, HIVE-13342.2.patch, > HIVE-13342.3.patch, HIVE-13342.4.patch, HIVE-13342.5.patch > > > Currently we do not log our decisions with respect to llap. Are we running > everything in llap mode or only parts of the plan. We need more logging. > Also, if llap mode is all but for some reason, we cannot run the work in llap > mode, fail and throw an exception advise the user to change the mode to auto. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13343) Need to disable hybrid grace hash join in llap mode except for dynamically partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13343: -- Attachment: HIVE-13343.5.patch > Need to disable hybrid grace hash join in llap mode except for dynamically > partitioned hash join > > > Key: HIVE-13343 > URL: https://issues.apache.org/jira/browse/HIVE-13343 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13343.1.patch, HIVE-13343.2.patch, > HIVE-13343.3.patch, HIVE-13343.4.patch, HIVE-13343.5.patch > > > Due to performance reasons, we should disable use of hybrid grace hash join > in llap when dynamic partition hash join is not used. With dynamic partition > hash join, we need hybrid grace hash join due to the possibility of skews. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13343) Need to disable hybrid grace hash join in llap mode except for dynamically partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13343: -- Attachment: HIVE-13343.4.patch Golden file updates missed in the previous patch. > Need to disable hybrid grace hash join in llap mode except for dynamically > partitioned hash join > > > Key: HIVE-13343 > URL: https://issues.apache.org/jira/browse/HIVE-13343 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13343.1.patch, HIVE-13343.2.patch, > HIVE-13343.3.patch, HIVE-13343.4.patch > > > Due to performance reasons, we should disable use of hybrid grace hash join > in llap when dynamic partition hash join is not used. With dynamic partition > hash join, we need hybrid grace hash join due to the possibility of skews. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13627) When running under LLAP, for regular map joins, throw an error if memory utilization goes above what is allocated to the task
[ https://issues.apache.org/jira/browse/HIVE-13627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13627: -- Attachment: HIVE-13628.1.patch > When running under LLAP, for regular map joins, throw an error if memory > utilization goes above what is allocated to the task > - > > Key: HIVE-13627 > URL: https://issues.apache.org/jira/browse/HIVE-13627 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13628.1.patch > > > When running under LLAP, for regular map joins, throw an error if memory > utilization goes above what is allocated to the task. This way, the rest of > the dependent tasks can fail sooner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13628) Support for permanent functions - error handling if no restart
[ https://issues.apache.org/jira/browse/HIVE-13628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13628: -- Attachment: HIVE-13628.1.patch > Support for permanent functions - error handling if no restart > -- > > Key: HIVE-13628 > URL: https://issues.apache.org/jira/browse/HIVE-13628 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13628.1.patch > > > Support for permanent functions - error handling if no restart -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13343) Need to disable hybrid grace hash join in llap mode except for dynamically partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15259356#comment-15259356 ] Vikram Dixit K commented on HIVE-13343: --- Addressed comments. Created RB. > Need to disable hybrid grace hash join in llap mode except for dynamically > partitioned hash join > > > Key: HIVE-13343 > URL: https://issues.apache.org/jira/browse/HIVE-13343 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13343.1.patch, HIVE-13343.2.patch, > HIVE-13343.3.patch > > > Due to performance reasons, we should disable use of hybrid grace hash join > in llap when dynamic partition hash join is not used. With dynamic partition > hash join, we need hybrid grace hash join due to the possibility of skews. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13343) Need to disable hybrid grace hash join in llap mode except for dynamically partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13343: -- Attachment: HIVE-13343.3.patch > Need to disable hybrid grace hash join in llap mode except for dynamically > partitioned hash join > > > Key: HIVE-13343 > URL: https://issues.apache.org/jira/browse/HIVE-13343 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13343.1.patch, HIVE-13343.2.patch, > HIVE-13343.3.patch > > > Due to performance reasons, we should disable use of hybrid grace hash join > in llap when dynamic partition hash join is not used. With dynamic partition > hash join, we need hybrid grace hash join due to the possibility of skews. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13621) compute stats in certain cases fails with NPE
[
https://issues.apache.org/jira/browse/HIVE-13621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikram Dixit K updated HIVE-13621:
--
Attachment: HIVE-13621.1.patch
> compute stats in certain cases fails with NPE
> -
>
> Key: HIVE-13621
> URL: https://issues.apache.org/jira/browse/HIVE-13621
> Project: Hive
> Issue Type: Bug
> Components: HBase Metastore, Metastore
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-13621.1.patch
>
>
> {code}
> FAILED: NullPointerException null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getColStatistics(StatsUtils.java:693)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.convertColStats(StatsUtils.java:739)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getTableColumnStats(StatsUtils.java:728)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:183)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:136)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:124){code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13621) compute stats in certain cases fails with NPE
[
https://issues.apache.org/jira/browse/HIVE-13621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikram Dixit K updated HIVE-13621:
--
Status: Patch Available (was: Open)
> compute stats in certain cases fails with NPE
> -
>
> Key: HIVE-13621
> URL: https://issues.apache.org/jira/browse/HIVE-13621
> Project: Hive
> Issue Type: Bug
> Components: HBase Metastore, Metastore
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-13621.1.patch
>
>
> {code}
> FAILED: NullPointerException null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getColStatistics(StatsUtils.java:693)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.convertColStats(StatsUtils.java:739)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getTableColumnStats(StatsUtils.java:728)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:183)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:136)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:124){code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13619) Bucket map join plan is incorrect
[ https://issues.apache.org/jira/browse/HIVE-13619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15259174#comment-15259174 ] Vikram Dixit K commented on HIVE-13619: --- Yes. The method is named findSingleUpstreamOperatorJoinAccounted. We are expecting only one instance of the operator type to be returned. It is not trying to find one specific operator in the list. > Bucket map join plan is incorrect > - > > Key: HIVE-13619 > URL: https://issues.apache.org/jira/browse/HIVE-13619 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.0.0, 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13619.1.patch > > > Same as HIVE-12992. Missed a single line check. TPCDS query 4 with bucketing > can produce this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13619) Bucket map join plan is incorrect
[ https://issues.apache.org/jira/browse/HIVE-13619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13619: -- Status: Patch Available (was: Open) > Bucket map join plan is incorrect > - > > Key: HIVE-13619 > URL: https://issues.apache.org/jira/browse/HIVE-13619 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.0.0, 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13619.1.patch > > > Same as HIVE-12992. Missed a single line check. TPCDS query 4 with bucketing > can produce this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13619) Bucket map join plan is incorrect
[ https://issues.apache.org/jira/browse/HIVE-13619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13619: -- Attachment: HIVE-13619.1.patch > Bucket map join plan is incorrect > - > > Key: HIVE-13619 > URL: https://issues.apache.org/jira/browse/HIVE-13619 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.0.0, 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13619.1.patch > > > Same as HIVE-12992. Missed a single line check. TPCDS query 4 with bucketing > can produce this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13619) Bucket map join plan is incorrect
[ https://issues.apache.org/jira/browse/HIVE-13619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13619: -- Description: Same as HIVE-12992. Missed a single line check. TPCDS query 4 with bucketing can produce this issue. (was: Same as HIVE-12992. Missed a single line check.) > Bucket map join plan is incorrect > - > > Key: HIVE-13619 > URL: https://issues.apache.org/jira/browse/HIVE-13619 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.0.0, 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > > Same as HIVE-12992. Missed a single line check. TPCDS query 4 with bucketing > can produce this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13467) Show llap info on hs2 ui when available
[ https://issues.apache.org/jira/browse/HIVE-13467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254916#comment-15254916 ] Vikram Dixit K commented on HIVE-13467: --- +1. LGTM. > Show llap info on hs2 ui when available > --- > > Key: HIVE-13467 > URL: https://issues.apache.org/jira/browse/HIVE-13467 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Gunther Hagleitner >Assignee: Gunther Hagleitner > Attachments: HIVE-13467.1.patch, HIVE-13467.2.patch, > HIVE-13467.3.patch, HIVE-13467.4.patch, HIVE-13467.5.patch, > HIVE-13467.6.patch, screen-shot-llap.png, screen.png > > > When llap is on and hs2 is configured with access to an llap cluster, HS2 UI > should show some status of the daemons and provide a mechanism to click > through to their respective UIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13408) Issue appending HIVE_QUERY_ID without checking if the prefix already exists
[
https://issues.apache.org/jira/browse/HIVE-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254315#comment-15254315
]
Vikram Dixit K commented on HIVE-13408:
---
[~sershe] Unfortunately not. This one depends on updating the hadoop version
used in hive and committing of other patches (HIVE-12254) before we can commit
this to master.
> Issue appending HIVE_QUERY_ID without checking if the prefix already exists
> ---
>
> Key: HIVE-13408
> URL: https://issues.apache.org/jira/browse/HIVE-13408
> Project: Hive
> Issue Type: Bug
> Components: Shims
>Affects Versions: 2.0.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-13408.1.patch, HIVE-13408.2.patch
>
>
> {code}
> We are resetting the hadoop caller context to HIVE_QUERY_ID:HIVE_QUERY_ID:
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13467) Show llap info on hs2 ui when available
[ https://issues.apache.org/jira/browse/HIVE-13467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15250392#comment-15250392 ] Vikram Dixit K commented on HIVE-13467: --- Nit: Can you add the apache header for some of the new files or eliminate them from the rat check? Otherwise LGTM. > Show llap info on hs2 ui when available > --- > > Key: HIVE-13467 > URL: https://issues.apache.org/jira/browse/HIVE-13467 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Gunther Hagleitner >Assignee: Gunther Hagleitner > Attachments: HIVE-13467.1.patch, HIVE-13467.2.patch, > HIVE-13467.3.patch, HIVE-13467.4.patch, HIVE-13467.5.patch, > screen-shot-llap.png, screen.png > > > When llap is on and hs2 is configured with access to an llap cluster, HS2 UI > should show some status of the daemons and provide a mechanism to click > through to their respective UIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13343) Need to disable hybrid grace hash join in llap mode except for dynamically partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15248675#comment-15248675 ] Vikram Dixit K commented on HIVE-13343: --- [~hagleitn] could you review this please? > Need to disable hybrid grace hash join in llap mode except for dynamically > partitioned hash join > > > Key: HIVE-13343 > URL: https://issues.apache.org/jira/browse/HIVE-13343 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13343.1.patch, HIVE-13343.2.patch > > > Due to performance reasons, we should disable use of hybrid grace hash join > in llap when dynamic partition hash join is not used. With dynamic partition > hash join, we need hybrid grace hash join due to the possibility of skews. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13342) Improve logging in llap decider and throw exception in case llap mode is all but we cannot run in llap.
[ https://issues.apache.org/jira/browse/HIVE-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15248677#comment-15248677 ] Vikram Dixit K commented on HIVE-13342: --- [~hagleitn] could you review this please? > Improve logging in llap decider and throw exception in case llap mode is all > but we cannot run in llap. > --- > > Key: HIVE-13342 > URL: https://issues.apache.org/jira/browse/HIVE-13342 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13342.1.patch, HIVE-13342.2.patch, > HIVE-13342.3.patch, HIVE-13342.4.patch > > > Currently we do not log our decisions with respect to llap. Are we running > everything in llap mode or only parts of the plan. We need more logging. > Also, if llap mode is all but for some reason, we cannot run the work in llap > mode, fail and throw an exception advise the user to change the mode to auto. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13342) Improve logging in llap decider and throw exception in case llap mode is all but we cannot run in llap.
[ https://issues.apache.org/jira/browse/HIVE-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15248518#comment-15248518 ] Vikram Dixit K commented on HIVE-13342: --- Ping [~sershe]. Can you take a look as well please. > Improve logging in llap decider and throw exception in case llap mode is all > but we cannot run in llap. > --- > > Key: HIVE-13342 > URL: https://issues.apache.org/jira/browse/HIVE-13342 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13342.1.patch, HIVE-13342.2.patch, > HIVE-13342.3.patch, HIVE-13342.4.patch > > > Currently we do not log our decisions with respect to llap. Are we running > everything in llap mode or only parts of the plan. We need more logging. > Also, if llap mode is all but for some reason, we cannot run the work in llap > mode, fail and throw an exception advise the user to change the mode to auto. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13518) Hive on Tez: Shuffle joins do not choose the right 'big' table.
[ https://issues.apache.org/jira/browse/HIVE-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15241692#comment-15241692 ] Vikram Dixit K commented on HIVE-13518: --- Ping [~gopalv]/[~rajesh.balamohan] > Hive on Tez: Shuffle joins do not choose the right 'big' table. > --- > > Key: HIVE-13518 > URL: https://issues.apache.org/jira/browse/HIVE-13518 > Project: Hive > Issue Type: Bug >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13518.1.patch > > > Currently the big table is always assumed to be at position 0 but this isn't > efficient for some queries as the big table at position 1 could have a lot > more keys/skew. We already have a mechanism of choosing the big table that > can be leveraged to make the right choice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13518) Hive on Tez: Shuffle joins do not choose the right 'big' table.
[ https://issues.apache.org/jira/browse/HIVE-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13518: -- Attachment: HIVE-13518.1.patch > Hive on Tez: Shuffle joins do not choose the right 'big' table. > --- > > Key: HIVE-13518 > URL: https://issues.apache.org/jira/browse/HIVE-13518 > Project: Hive > Issue Type: Bug >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13518.1.patch > > > Currently the big table is always assumed to be at position 0 but this isn't > efficient for some queries as the big table at position 1 could have a lot > more keys/skew. We already have a mechanism of choosing the big table that > can be leveraged to make the right choice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13518) Hive on Tez: Shuffle joins do not choose the right 'big' table.
[ https://issues.apache.org/jira/browse/HIVE-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13518: -- Status: Patch Available (was: Open) > Hive on Tez: Shuffle joins do not choose the right 'big' table. > --- > > Key: HIVE-13518 > URL: https://issues.apache.org/jira/browse/HIVE-13518 > Project: Hive > Issue Type: Bug >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13518.1.patch > > > Currently the big table is always assumed to be at position 0 but this isn't > efficient for some queries as the big table at position 1 could have a lot > more keys/skew. We already have a mechanism of choosing the big table that > can be leveraged to make the right choice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13342) Improve logging in llap decider and throw exception in case llap mode is all but we cannot run in llap.
[ https://issues.apache.org/jira/browse/HIVE-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15240167#comment-15240167 ] Vikram Dixit K commented on HIVE-13342: --- [~sershe] I think the test fails because of the usage of a temporary function. I think that is the kind of behavior we want to enforce with this change. I have updated the test accordingly. Let me know if you have any feedback. > Improve logging in llap decider and throw exception in case llap mode is all > but we cannot run in llap. > --- > > Key: HIVE-13342 > URL: https://issues.apache.org/jira/browse/HIVE-13342 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13342.1.patch, HIVE-13342.2.patch, > HIVE-13342.3.patch, HIVE-13342.4.patch > > > Currently we do not log our decisions with respect to llap. Are we running > everything in llap mode or only parts of the plan. We need more logging. > Also, if llap mode is all but for some reason, we cannot run the work in llap > mode, fail and throw an exception advise the user to change the mode to auto. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13342) Improve logging in llap decider and throw exception in case llap mode is all but we cannot run in llap.
[ https://issues.apache.org/jira/browse/HIVE-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13342: -- Attachment: HIVE-13342.4.patch > Improve logging in llap decider and throw exception in case llap mode is all > but we cannot run in llap. > --- > > Key: HIVE-13342 > URL: https://issues.apache.org/jira/browse/HIVE-13342 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13342.1.patch, HIVE-13342.2.patch, > HIVE-13342.3.patch, HIVE-13342.4.patch > > > Currently we do not log our decisions with respect to llap. Are we running > everything in llap mode or only parts of the plan. We need more logging. > Also, if llap mode is all but for some reason, we cannot run the work in llap > mode, fail and throw an exception advise the user to change the mode to auto. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13342) Improve logging in llap decider and throw exception in case llap mode is all but we cannot run in llap.
[ https://issues.apache.org/jira/browse/HIVE-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13342: -- Summary: Improve logging in llap decider and throw exception in case llap mode is all but we cannot run in llap. (was: Improve logging in llap decider for llap) > Improve logging in llap decider and throw exception in case llap mode is all > but we cannot run in llap. > --- > > Key: HIVE-13342 > URL: https://issues.apache.org/jira/browse/HIVE-13342 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13342.1.patch, HIVE-13342.2.patch, > HIVE-13342.3.patch > > > Currently we do not log our decisions with respect to llap. Are we running > everything in llap mode or only parts of the plan. We need more logging. > Also, if llap mode is all but for some reason, we cannot run the work in llap > mode, fail and throw an exception advise the user to change the mode to auto. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13342) Improve logging in llap decider for llap
[ https://issues.apache.org/jira/browse/HIVE-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13342: -- Attachment: HIVE-13342.3.patch > Improve logging in llap decider for llap > > > Key: HIVE-13342 > URL: https://issues.apache.org/jira/browse/HIVE-13342 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13342.1.patch, HIVE-13342.2.patch, > HIVE-13342.3.patch > > > Currently we do not log our decisions with respect to llap. Are we running > everything in llap mode or only parts of the plan. We need more logging. > Also, if llap mode is all but for some reason, we cannot run the work in llap > mode, fail and throw an exception advise the user to change the mode to auto. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13485) Session id appended to thread name multiple times.
[ https://issues.apache.org/jira/browse/HIVE-13485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13485: -- Status: Patch Available (was: Open) > Session id appended to thread name multiple times. > -- > > Key: HIVE-13485 > URL: https://issues.apache.org/jira/browse/HIVE-13485 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13485.1.patch > > > HIVE-13153 addressed a portion of this issue. Follow up from there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13485) Session id appended to thread name multiple times.
[ https://issues.apache.org/jira/browse/HIVE-13485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236070#comment-15236070 ] Vikram Dixit K commented on HIVE-13485: --- Ping [~sershe] > Session id appended to thread name multiple times. > -- > > Key: HIVE-13485 > URL: https://issues.apache.org/jira/browse/HIVE-13485 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13485.1.patch > > > HIVE-13153 addressed a portion of this issue. Follow up from there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13485) Session id appended to thread name multiple times.
[ https://issues.apache.org/jira/browse/HIVE-13485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13485: -- Attachment: HIVE-13485.1.patch > Session id appended to thread name multiple times. > -- > > Key: HIVE-13485 > URL: https://issues.apache.org/jira/browse/HIVE-13485 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13485.1.patch > > > HIVE-13153 addressed a portion of this issue. Follow up from there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13438) Add a service check script for llap
[ https://issues.apache.org/jira/browse/HIVE-13438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235932#comment-15235932 ] Vikram Dixit K commented on HIVE-13438: --- Tests pass locally. Committed to master. > Add a service check script for llap > --- > > Key: HIVE-13438 > URL: https://issues.apache.org/jira/browse/HIVE-13438 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Fix For: 2.1.0 > > Attachments: HIVE-13438.1.patch, HIVE-13438.2.patch > > > We want to have a test script that can be run by an installer such as ambari > that makes sure that the service is up and running. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13438) Add a service check script for llap
[ https://issues.apache.org/jira/browse/HIVE-13438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13438: -- Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) > Add a service check script for llap > --- > > Key: HIVE-13438 > URL: https://issues.apache.org/jira/browse/HIVE-13438 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Fix For: 2.1.0 > > Attachments: HIVE-13438.1.patch, HIVE-13438.2.patch > > > We want to have a test script that can be run by an installer such as ambari > that makes sure that the service is up and running. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-13282) GroupBy and select operator encounter ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-13282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15231193#comment-15231193
]
Vikram Dixit K edited comment on HIVE-13282 at 4/7/16 10:05 PM:
Yes. We can move this out to 2.1.0. This only happens in case of reduce side
SMB in tez. We have a simple workaround right now that will address this
(disable smb join in this case). The real fix would take a lot of refactoring
the code which is more suited for master than a maintenance release.
was (Author: vikram.dixit):
Yes. We can move this out to 2.1.0. This only happens in case of reduce side
SMB in tez. We have a simple workaround right now that will address this. The
real fix would take a lot of refactoring the code which is more suited for
master than a maintenance release.
> GroupBy and select operator encounter ArrayIndexOutOfBoundsException
>
>
> Key: HIVE-13282
> URL: https://issues.apache.org/jira/browse/HIVE-13282
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.1, 2.0.0, 2.1.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
>
> The group by and select operators run into the ArrayIndexOutOfBoundsException
> when they incorrectly initialize themselves with tag 0 but the incoming tag
> id is different.
> {code}
> select count(*) from
> (select rt1.id from
> (select t1.key as id, t1.value as od from tab t1 group by key, value) rt1) vt1
> join
> (select rt2.id from
> (select t2.key as id, t2.value as od from tab_part t2 group by key, value)
> rt2) vt2
> where vt1.id=vt2.id;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13342) Improve logging in llap decider for llap
[ https://issues.apache.org/jira/browse/HIVE-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15230950#comment-15230950 ] Vikram Dixit K commented on HIVE-13342: --- [~sseth] Yes. The other log lines should tell us which operator interferes with running in llap. I changed the exception to use the right configuration variable from HiveConf. However, there is currently no way to get the values a configuration can take from code. I think it is better to not add more configuration to enable/disable the mode = all behavior. If the user is not sure if they can run in llap, they need to use mode = auto. The mode = all behavior only prevents further checking on the query if it can be run in llap. If under mode all, query cannot be run in llap because some parts of the plan cannot be run in it, it makes sense to stop the user from proceeding. If you feel strongly about needing the flag, I can add one but I am not convinced at this point in time. > Improve logging in llap decider for llap > > > Key: HIVE-13342 > URL: https://issues.apache.org/jira/browse/HIVE-13342 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13342.1.patch, HIVE-13342.2.patch > > > Currently we do not log our decisions with respect to llap. Are we running > everything in llap mode or only parts of the plan. We need more logging. > Also, if llap mode is all but for some reason, we cannot run the work in llap > mode, fail and throw an exception advise the user to change the mode to auto. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13438) Add a service check script for llap
[ https://issues.apache.org/jira/browse/HIVE-13438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15230648#comment-15230648 ] Vikram Dixit K commented on HIVE-13438: --- Fixed the error. [~sershe] Can you elaborate a little more? Are you suggesting that we could run the query as part of starting the service itself? We could add that too but we still need something to run end-to-end (from starting the shell onwards). > Add a service check script for llap > --- > > Key: HIVE-13438 > URL: https://issues.apache.org/jira/browse/HIVE-13438 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13438.1.patch, HIVE-13438.2.patch > > > We want to have a test script that can be run by an installer such as ambari > that makes sure that the service is up and running. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13438) Add a service check script for llap
[ https://issues.apache.org/jira/browse/HIVE-13438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13438: -- Attachment: HIVE-13438.2.patch > Add a service check script for llap > --- > > Key: HIVE-13438 > URL: https://issues.apache.org/jira/browse/HIVE-13438 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13438.1.patch, HIVE-13438.2.patch > > > We want to have a test script that can be run by an installer such as ambari > that makes sure that the service is up and running. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13394) Analyze table fails in tez on empty partitions/files/tables
[
https://issues.apache.org/jira/browse/HIVE-13394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikram Dixit K updated HIVE-13394:
--
Summary: Analyze table fails in tez on empty partitions/files/tables (was:
Analyze table fails in tez on empty partitions)
> Analyze table fails in tez on empty partitions/files/tables
> ---
>
> Key: HIVE-13394
> URL: https://issues.apache.org/jira/browse/HIVE-13394
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Fix For: 1.2.2, 2.1.0, 2.0.1
>
> Attachments: HIVE-13394.1.patch, HIVE-13394.2.patch
>
>
> {code}
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:352)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:237)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:252)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150)
> ... 14 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:766)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:343)
> ... 17 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hive.ql.udf.generic.NumDistinctValueEstimator.deserialize(NumDistinctValueEstimator.java:219)
> at
> org.apache.hadoop.hive.ql.udf.generic.NumDistinctValueEstimator.(NumDistinctValueEstimator.java:112)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFComputeStats$GenericUDAFNumericStatsEvaluator.merge(GenericUDAFComputeStats.java:556)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:188)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:612)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processAggr(GroupByOperator.java:851)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:695)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:761)
> ... 18 more
> ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1
> killedTasks:0, Vertex vertex_145591034_27748_1_01 [Reducer 2]
> killed/failed due to:OWN_TASK_FAILURE]DAG did not succeed due to
> VERTEX_FAILURE. failedVertices:1 killedVertices:0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13438) Add a service check script for llap
[ https://issues.apache.org/jira/browse/HIVE-13438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13438: -- Status: Patch Available (was: Open) > Add a service check script for llap > --- > > Key: HIVE-13438 > URL: https://issues.apache.org/jira/browse/HIVE-13438 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13438.1.patch > > > We want to have a test script that can be run by an installer such as ambari > that makes sure that the service is up and running. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13438) Add a service check script for llap
[ https://issues.apache.org/jira/browse/HIVE-13438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15228863#comment-15228863 ] Vikram Dixit K commented on HIVE-13438: --- [~hagleitn] Could you please take a look? Thanks! > Add a service check script for llap > --- > > Key: HIVE-13438 > URL: https://issues.apache.org/jira/browse/HIVE-13438 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13438.1.patch > > > We want to have a test script that can be run by an installer such as ambari > that makes sure that the service is up and running. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13438) Add a service check script for llap
[ https://issues.apache.org/jira/browse/HIVE-13438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13438: -- Attachment: HIVE-13438.1.patch > Add a service check script for llap > --- > > Key: HIVE-13438 > URL: https://issues.apache.org/jira/browse/HIVE-13438 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13438.1.patch > > > We want to have a test script that can be run by an installer such as ambari > that makes sure that the service is up and running. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13343) Need to disable hybrid grace hash join in llap mode except for dynamically partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13343: -- Attachment: HIVE-13343.2.patch Address comment and update golden files. > Need to disable hybrid grace hash join in llap mode except for dynamically > partitioned hash join > > > Key: HIVE-13343 > URL: https://issues.apache.org/jira/browse/HIVE-13343 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13343.1.patch, HIVE-13343.2.patch > > > Due to performance reasons, we should disable use of hybrid grace hash join > in llap when dynamic partition hash join is not used. With dynamic partition > hash join, we need hybrid grace hash join due to the possibility of skews. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13408) Issue appending HIVE_QUERY_ID without checking if the prefix already exists
[
https://issues.apache.org/jira/browse/HIVE-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikram Dixit K updated HIVE-13408:
--
Attachment: HIVE-13408.2.patch
The right file.
> Issue appending HIVE_QUERY_ID without checking if the prefix already exists
> ---
>
> Key: HIVE-13408
> URL: https://issues.apache.org/jira/browse/HIVE-13408
> Project: Hive
> Issue Type: Bug
> Components: Shims
>Affects Versions: 2.0.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-13408.1.patch, HIVE-13408.2.patch
>
>
> {code}
> We are resetting the hadoop caller context to HIVE_QUERY_ID:HIVE_QUERY_ID:
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13408) Issue appending HIVE_QUERY_ID without checking if the prefix already exists
[
https://issues.apache.org/jira/browse/HIVE-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikram Dixit K updated HIVE-13408:
--
Attachment: HIVE-13408.2.patch
Addressed [~jdere]'s review comments (given offline).
> Issue appending HIVE_QUERY_ID without checking if the prefix already exists
> ---
>
> Key: HIVE-13408
> URL: https://issues.apache.org/jira/browse/HIVE-13408
> Project: Hive
> Issue Type: Bug
> Components: Shims
>Affects Versions: 2.0.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-13408.1.patch
>
>
> {code}
> We are resetting the hadoop caller context to HIVE_QUERY_ID:HIVE_QUERY_ID:
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13408) Issue appending HIVE_QUERY_ID without checking if the prefix already exists
[
https://issues.apache.org/jira/browse/HIVE-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikram Dixit K updated HIVE-13408:
--
Attachment: (was: HIVE-13408.2.patch)
> Issue appending HIVE_QUERY_ID without checking if the prefix already exists
> ---
>
> Key: HIVE-13408
> URL: https://issues.apache.org/jira/browse/HIVE-13408
> Project: Hive
> Issue Type: Bug
> Components: Shims
>Affects Versions: 2.0.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-13408.1.patch
>
>
> {code}
> We are resetting the hadoop caller context to HIVE_QUERY_ID:HIVE_QUERY_ID:
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13417) Some vector operators return "OP" as name
[ https://issues.apache.org/jira/browse/HIVE-13417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227038#comment-15227038 ] Vikram Dixit K commented on HIVE-13417: --- +1 pending test results for the latest iteration. > Some vector operators return "OP" as name > - > > Key: HIVE-13417 > URL: https://issues.apache.org/jira/browse/HIVE-13417 > Project: Hive > Issue Type: Bug >Reporter: Gunther Hagleitner >Assignee: Gunther Hagleitner > Attachments: HIVE-13417.1.patch, HIVE-13417.2.patch, > HIVE-13417.3.patch > > > Select/Group by/Filter/etc need to return the same name whether they are the > regular or the vector operators. If they don't the regular path matching in > our optimizer code doesn't work on them. > From the code it looks an attempt was made to follow this - unfortunately > getOperatorName is static and polymorphism doesn't work on these functions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13394) Analyze table fails in tez on empty partitions
[
https://issues.apache.org/jira/browse/HIVE-13394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikram Dixit K updated HIVE-13394:
--
Fix Version/s: 2.0.1
> Analyze table fails in tez on empty partitions
> --
>
> Key: HIVE-13394
> URL: https://issues.apache.org/jira/browse/HIVE-13394
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Fix For: 1.2.2, 2.1.0, 2.0.1
>
> Attachments: HIVE-13394.1.patch, HIVE-13394.2.patch
>
>
> {code}
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:352)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:237)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:252)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150)
> ... 14 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:766)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:343)
> ... 17 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hive.ql.udf.generic.NumDistinctValueEstimator.deserialize(NumDistinctValueEstimator.java:219)
> at
> org.apache.hadoop.hive.ql.udf.generic.NumDistinctValueEstimator.(NumDistinctValueEstimator.java:112)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFComputeStats$GenericUDAFNumericStatsEvaluator.merge(GenericUDAFComputeStats.java:556)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:188)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:612)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processAggr(GroupByOperator.java:851)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:695)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:761)
> ... 18 more
> ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1
> killedTasks:0, Vertex vertex_145591034_27748_1_01 [Reducer 2]
> killed/failed due to:OWN_TASK_FAILURE]DAG did not succeed due to
> VERTEX_FAILURE. failedVertices:1 killedVertices:0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13394) Analyze table fails in tez on empty partitions
[
https://issues.apache.org/jira/browse/HIVE-13394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikram Dixit K updated HIVE-13394:
--
Resolution: Fixed
Fix Version/s: 2.1.0
1.2.2
Status: Resolved (was: Patch Available)
> Analyze table fails in tez on empty partitions
> --
>
> Key: HIVE-13394
> URL: https://issues.apache.org/jira/browse/HIVE-13394
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Fix For: 1.2.2, 2.1.0
>
> Attachments: HIVE-13394.1.patch, HIVE-13394.2.patch
>
>
> {code}
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:352)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:237)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:252)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150)
> ... 14 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:766)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:343)
> ... 17 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hive.ql.udf.generic.NumDistinctValueEstimator.deserialize(NumDistinctValueEstimator.java:219)
> at
> org.apache.hadoop.hive.ql.udf.generic.NumDistinctValueEstimator.(NumDistinctValueEstimator.java:112)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFComputeStats$GenericUDAFNumericStatsEvaluator.merge(GenericUDAFComputeStats.java:556)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:188)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:612)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processAggr(GroupByOperator.java:851)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:695)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:761)
> ... 18 more
> ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1
> killedTasks:0, Vertex vertex_145591034_27748_1_01 [Reducer 2]
> killed/failed due to:OWN_TASK_FAILURE]DAG did not succeed due to
> VERTEX_FAILURE. failedVertices:1 killedVertices:0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13408) Issue appending HIVE_QUERY_ID without checking if the prefix already exists
[
https://issues.apache.org/jira/browse/HIVE-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15222570#comment-15222570
]
Vikram Dixit K commented on HIVE-13408:
---
Ping [~jdere].
> Issue appending HIVE_QUERY_ID without checking if the prefix already exists
> ---
>
> Key: HIVE-13408
> URL: https://issues.apache.org/jira/browse/HIVE-13408
> Project: Hive
> Issue Type: Bug
> Components: Shims
>Affects Versions: 2.0.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-13408.1.patch
>
>
> {code}
> We are resetting the hadoop caller context to HIVE_QUERY_ID:HIVE_QUERY_ID:
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13408) Issue appending HIVE_QUERY_ID without checking if the prefix already exists
[
https://issues.apache.org/jira/browse/HIVE-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikram Dixit K updated HIVE-13408:
--
Attachment: HIVE-13408.1.patch
Depends on HIVE-12254.
> Issue appending HIVE_QUERY_ID without checking if the prefix already exists
> ---
>
> Key: HIVE-13408
> URL: https://issues.apache.org/jira/browse/HIVE-13408
> Project: Hive
> Issue Type: Bug
> Components: Shims
>Affects Versions: 2.0.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-13408.1.patch
>
>
> {code}
> We are resetting the hadoop caller context to HIVE_QUERY_ID:HIVE_QUERY_ID:
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13394) Analyze table fails in tez on empty partitions
[
https://issues.apache.org/jira/browse/HIVE-13394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikram Dixit K updated HIVE-13394:
--
Attachment: HIVE-13394.2.patch
Address comments.
> Analyze table fails in tez on empty partitions
> --
>
> Key: HIVE-13394
> URL: https://issues.apache.org/jira/browse/HIVE-13394
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-13394.1.patch, HIVE-13394.2.patch
>
>
> {code}
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:352)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:237)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:252)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150)
> ... 14 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:766)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:343)
> ... 17 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hive.ql.udf.generic.NumDistinctValueEstimator.deserialize(NumDistinctValueEstimator.java:219)
> at
> org.apache.hadoop.hive.ql.udf.generic.NumDistinctValueEstimator.(NumDistinctValueEstimator.java:112)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFComputeStats$GenericUDAFNumericStatsEvaluator.merge(GenericUDAFComputeStats.java:556)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:188)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:612)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processAggr(GroupByOperator.java:851)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:695)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:761)
> ... 18 more
> ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1
> killedTasks:0, Vertex vertex_145591034_27748_1_01 [Reducer 2]
> killed/failed due to:OWN_TASK_FAILURE]DAG did not succeed due to
> VERTEX_FAILURE. failedVertices:1 killedVertices:0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13394) Analyze table fails in tez on empty partitions
[
https://issues.apache.org/jira/browse/HIVE-13394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikram Dixit K updated HIVE-13394:
--
Attachment: HIVE-13394.1.patch
[~prasanth_j] review please.
> Analyze table fails in tez on empty partitions
> --
>
> Key: HIVE-13394
> URL: https://issues.apache.org/jira/browse/HIVE-13394
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-13394.1.patch
>
>
> {code}
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:352)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:237)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:252)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150)
> ... 14 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:766)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:343)
> ... 17 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hive.ql.udf.generic.NumDistinctValueEstimator.deserialize(NumDistinctValueEstimator.java:219)
> at
> org.apache.hadoop.hive.ql.udf.generic.NumDistinctValueEstimator.(NumDistinctValueEstimator.java:112)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFComputeStats$GenericUDAFNumericStatsEvaluator.merge(GenericUDAFComputeStats.java:556)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:188)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:612)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processAggr(GroupByOperator.java:851)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:695)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:761)
> ... 18 more
> ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1
> killedTasks:0, Vertex vertex_145591034_27748_1_01 [Reducer 2]
> killed/failed due to:OWN_TASK_FAILURE]DAG did not succeed due to
> VERTEX_FAILURE. failedVertices:1 killedVertices:0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13394) Analyze table fails in tez on empty partitions
[
https://issues.apache.org/jira/browse/HIVE-13394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikram Dixit K updated HIVE-13394:
--
Status: Patch Available (was: Open)
> Analyze table fails in tez on empty partitions
> --
>
> Key: HIVE-13394
> URL: https://issues.apache.org/jira/browse/HIVE-13394
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 2.0.0, 1.2.1
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-13394.1.patch
>
>
> {code}
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:352)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:237)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:252)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150)
> ... 14 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:766)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:343)
> ... 17 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hive.ql.udf.generic.NumDistinctValueEstimator.deserialize(NumDistinctValueEstimator.java:219)
> at
> org.apache.hadoop.hive.ql.udf.generic.NumDistinctValueEstimator.(NumDistinctValueEstimator.java:112)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFComputeStats$GenericUDAFNumericStatsEvaluator.merge(GenericUDAFComputeStats.java:556)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:188)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:612)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processAggr(GroupByOperator.java:851)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:695)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:761)
> ... 18 more
> ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1
> killedTasks:0, Vertex vertex_145591034_27748_1_01 [Reducer 2]
> killed/failed due to:OWN_TASK_FAILURE]DAG did not succeed due to
> VERTEX_FAILURE. failedVertices:1 killedVertices:0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12992) Hive on tez: Bucket map join plan is incorrect
[
https://issues.apache.org/jira/browse/HIVE-12992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikram Dixit K updated HIVE-12992:
--
Resolution: Fixed
Fix Version/s: 2.0.1
2.1.0
1.2.2
Status: Resolved (was: Patch Available)
> Hive on tez: Bucket map join plan is incorrect
> --
>
> Key: HIVE-12992
> URL: https://issues.apache.org/jira/browse/HIVE-12992
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Labels: tez
> Fix For: 1.2.2, 2.1.0, 2.0.1
>
> Attachments: HIVE-12992.1.patch, HIVE-12992.2.patch
>
>
> TPCH Query 9 fails when bucket map join is enabled:
> {code}
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Reducer
> 5, vertexId=vertex_1450634494433_0007_2_06, diagnostics=[Exception in
> EdgeManager, vertex=vertex_1450634494433_0007_2_06 [Reducer 5], Fail to
> sendTezEventToDestinationTasks, event:DataMovementEvent [sourceIndex=0,
> targetIndex=-1, version=0], sourceInfo:{ producerConsumerType=OUTPUT,
> taskVertexName=Map 1, edgeVertexName=Reducer 5,
> taskAttemptId=attempt_1450634494433_0007_2_05_00_0 },
> destinationInfo:null, EdgeInfo: sourceVertexName=Map 1,
> destinationVertexName=Reducer 5, java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.CustomPartitionEdge.routeDataMovementEventToDestination(CustomPartitionEdge.java:88)
> at
> org.apache.tez.dag.app.dag.impl.Edge.sendTezEventToDestinationTasks(Edge.java:458)
> at
> org.apache.tez.dag.app.dag.impl.Edge.handleCompositeDataMovementEvent(Edge.java:386)
> at
> org.apache.tez.dag.app.dag.impl.Edge.sendTezEventToDestinationTasks(Edge.java:439)
> at
> org.apache.tez.dag.app.dag.impl.VertexImpl.handleRoutedTezEvents(VertexImpl.java:4382)
> at
> org.apache.tez.dag.app.dag.impl.VertexImpl.access$4000(VertexImpl.java:202)
> at
> org.apache.tez.dag.app.dag.impl.VertexImpl$RouteEventTransition.transition(VertexImpl.java:4172)
> at
> org.apache.tez.dag.app.dag.impl.VertexImpl$RouteEventTransition.transition(VertexImpl.java:4164)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12992) Hive on tez: Bucket map join plan is incorrect
[
https://issues.apache.org/jira/browse/HIVE-12992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15214647#comment-15214647
]
Vikram Dixit K commented on HIVE-12992:
---
The bucket_map_join test failure is related. It is a golden file update that I
missed. Posting a new patch here with golden file update for it.
> Hive on tez: Bucket map join plan is incorrect
> --
>
> Key: HIVE-12992
> URL: https://issues.apache.org/jira/browse/HIVE-12992
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Labels: tez
> Attachments: HIVE-12992.1.patch, HIVE-12992.2.patch
>
>
> TPCH Query 9 fails when bucket map join is enabled:
> {code}
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Reducer
> 5, vertexId=vertex_1450634494433_0007_2_06, diagnostics=[Exception in
> EdgeManager, vertex=vertex_1450634494433_0007_2_06 [Reducer 5], Fail to
> sendTezEventToDestinationTasks, event:DataMovementEvent [sourceIndex=0,
> targetIndex=-1, version=0], sourceInfo:{ producerConsumerType=OUTPUT,
> taskVertexName=Map 1, edgeVertexName=Reducer 5,
> taskAttemptId=attempt_1450634494433_0007_2_05_00_0 },
> destinationInfo:null, EdgeInfo: sourceVertexName=Map 1,
> destinationVertexName=Reducer 5, java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.CustomPartitionEdge.routeDataMovementEventToDestination(CustomPartitionEdge.java:88)
> at
> org.apache.tez.dag.app.dag.impl.Edge.sendTezEventToDestinationTasks(Edge.java:458)
> at
> org.apache.tez.dag.app.dag.impl.Edge.handleCompositeDataMovementEvent(Edge.java:386)
> at
> org.apache.tez.dag.app.dag.impl.Edge.sendTezEventToDestinationTasks(Edge.java:439)
> at
> org.apache.tez.dag.app.dag.impl.VertexImpl.handleRoutedTezEvents(VertexImpl.java:4382)
> at
> org.apache.tez.dag.app.dag.impl.VertexImpl.access$4000(VertexImpl.java:202)
> at
> org.apache.tez.dag.app.dag.impl.VertexImpl$RouteEventTransition.transition(VertexImpl.java:4172)
> at
> org.apache.tez.dag.app.dag.impl.VertexImpl$RouteEventTransition.transition(VertexImpl.java:4164)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12992) Hive on tez: Bucket map join plan is incorrect
[
https://issues.apache.org/jira/browse/HIVE-12992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikram Dixit K updated HIVE-12992:
--
Attachment: HIVE-12992.2.patch
> Hive on tez: Bucket map join plan is incorrect
> --
>
> Key: HIVE-12992
> URL: https://issues.apache.org/jira/browse/HIVE-12992
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Labels: tez
> Attachments: HIVE-12992.1.patch, HIVE-12992.2.patch
>
>
> TPCH Query 9 fails when bucket map join is enabled:
> {code}
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Reducer
> 5, vertexId=vertex_1450634494433_0007_2_06, diagnostics=[Exception in
> EdgeManager, vertex=vertex_1450634494433_0007_2_06 [Reducer 5], Fail to
> sendTezEventToDestinationTasks, event:DataMovementEvent [sourceIndex=0,
> targetIndex=-1, version=0], sourceInfo:{ producerConsumerType=OUTPUT,
> taskVertexName=Map 1, edgeVertexName=Reducer 5,
> taskAttemptId=attempt_1450634494433_0007_2_05_00_0 },
> destinationInfo:null, EdgeInfo: sourceVertexName=Map 1,
> destinationVertexName=Reducer 5, java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.CustomPartitionEdge.routeDataMovementEventToDestination(CustomPartitionEdge.java:88)
> at
> org.apache.tez.dag.app.dag.impl.Edge.sendTezEventToDestinationTasks(Edge.java:458)
> at
> org.apache.tez.dag.app.dag.impl.Edge.handleCompositeDataMovementEvent(Edge.java:386)
> at
> org.apache.tez.dag.app.dag.impl.Edge.sendTezEventToDestinationTasks(Edge.java:439)
> at
> org.apache.tez.dag.app.dag.impl.VertexImpl.handleRoutedTezEvents(VertexImpl.java:4382)
> at
> org.apache.tez.dag.app.dag.impl.VertexImpl.access$4000(VertexImpl.java:202)
> at
> org.apache.tez.dag.app.dag.impl.VertexImpl$RouteEventTransition.transition(VertexImpl.java:4172)
> at
> org.apache.tez.dag.app.dag.impl.VertexImpl$RouteEventTransition.transition(VertexImpl.java:4164)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13343) Need to disable hybrid grace hash join in llap mode except for dynamically partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13343: -- Attachment: HIVE-13343.1.patch > Need to disable hybrid grace hash join in llap mode except for dynamically > partitioned hash join > > > Key: HIVE-13343 > URL: https://issues.apache.org/jira/browse/HIVE-13343 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13343.1.patch > > > Due to performance reasons, we should disable use of hybrid grace hash join > in llap when dynamic partition hash join is not used. With dynamic partition > hash join, we need hybrid grace hash join due to the possibility of skews. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13343) Need to disable hybrid grace hash join in llap mode except for dynamically partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13343: -- Status: Patch Available (was: Open) > Need to disable hybrid grace hash join in llap mode except for dynamically > partitioned hash join > > > Key: HIVE-13343 > URL: https://issues.apache.org/jira/browse/HIVE-13343 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13343.1.patch > > > Due to performance reasons, we should disable use of hybrid grace hash join > in llap when dynamic partition hash join is not used. With dynamic partition > hash join, we need hybrid grace hash join due to the possibility of skews. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
