[jira] [Updated] (HIVE-11351) Column Found in more than One Tables/Subqueries
[ https://issues.apache.org/jira/browse/HIVE-11351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-11351: -- Status: Patch Available (was: In Progress) > Column Found in more than One Tables/Subqueries > --- > > Key: HIVE-11351 > URL: https://issues.apache.org/jira/browse/HIVE-11351 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 > Environment: HIVE 1.1.0 >Reporter: MK >Assignee: Alina Abramova > Attachments: HIVE-11351.2-branch-1.0.patch, > HIVE-11351-branch-1.0.patch > > > when execute a script: > INSERT overwrite TABLE tmp.tmp_dim_cpttr_categ1 >SELECT DISTINCT cur.categ_id AS categ_id, >cur.categ_code AS categ_code, >cur.categ_name AS categ_name, >cur.categ_parnt_id AS categ_parnt_id, >par.categ_name AS categ_parnt_name, >cur.mc_site_id AS mc_site_id >FROM tmp.tmp_dim_cpttr_categ cur >LEFT OUTER JOIN tmp.tmp_dim_cpttr_categ par >ON cur.categ_parnt_id = par.categ_id; > error occur : SemanticException Column categ_name Found in more than One > Tables/Subqueries > when modify the alias categ_name to categ_name_cur, it will be execute > successfully. > INSERT overwrite TABLE tmp.tmp_dim_cpttr_categ1 >SELECT DISTINCT cur.categ_id AS categ_id, >cur.categ_code AS categ_code, >cur.categ_name AS categ_name_cur, >cur.categ_parnt_id AS categ_parnt_id, >par.categ_name AS categ_parnt_name, >cur.mc_site_id AS mc_site_id >FROM tmp.tmp_dim_cpttr_categ cur >LEFT OUTER JOIN tmp.tmp_dim_cpttr_categ par >ON cur.categ_parnt_id = par.categ_id; > this happen when we upgrade hive from 0.10 to 1.1.0 . -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15433) setting hive.warehouse.subdir.inherit.perms in HIVE won't overwrite it in hive configuration
[ https://issues.apache.org/jira/browse/HIVE-15433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15907499#comment-15907499 ] Alina Abramova commented on HIVE-15433: --- Hi there, Could you please rebuild thrift files so the patch will make an effect. For now it is possible to set hive.warehouse.subdir.inherit.perms for hive session, so if it is set to false the directory for table will be created keeping in mind umask, otherwise it will inherit parent directory permission. Thanks. > setting hive.warehouse.subdir.inherit.perms in HIVE won't overwrite it in > hive configuration > > > Key: HIVE-15433 > URL: https://issues.apache.org/jira/browse/HIVE-15433 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 1.0.0, 1.2.0, 2.0.0 >Reporter: Alina Abramova >Assignee: Alina Abramova > Fix For: 1.2.0 > > Attachments: HIVE-15433.1.patch, HIVE-15433-branch-1.2.patch > > > Setting hive.warehouse.subdir.inherit.perms in HIVE won't make any effect. It > will always take the default value from HiveConf until you define it in > hive-site.xml. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15433) setting hive.warehouse.subdir.inherit.perms in HIVE won't overwrite it in hive configuration
[ https://issues.apache.org/jira/browse/HIVE-15433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878381#comment-15878381 ] Alina Abramova commented on HIVE-15433: --- Hello, If it's possible please rebuild thrift files before building the project. > setting hive.warehouse.subdir.inherit.perms in HIVE won't overwrite it in > hive configuration > > > Key: HIVE-15433 > URL: https://issues.apache.org/jira/browse/HIVE-15433 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 1.0.0, 1.2.0, 2.0.0 >Reporter: Alina Abramova >Assignee: Alina Abramova > Fix For: 1.2.0 > > Attachments: HIVE-15433.1.patch, HIVE-15433-branch-1.2.patch > > > Setting hive.warehouse.subdir.inherit.perms in HIVE won't make any effect. It > will always take the default value from HiveConf until you define it in > hive-site.xml. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15433) setting hive.warehouse.subdir.inherit.perms in HIVE won't overwrite it in hive configuration
[ https://issues.apache.org/jira/browse/HIVE-15433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865727#comment-15865727 ] Alina Abramova commented on HIVE-15433: --- Hi there, Any updates? > setting hive.warehouse.subdir.inherit.perms in HIVE won't overwrite it in > hive configuration > > > Key: HIVE-15433 > URL: https://issues.apache.org/jira/browse/HIVE-15433 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 1.0.0, 1.2.0, 2.0.0 >Reporter: Alina Abramova >Assignee: Alina Abramova > Fix For: 1.2.0 > > Attachments: HIVE-15433.1.patch, HIVE-15433-branch-1.2.patch > > > Setting hive.warehouse.subdir.inherit.perms in HIVE won't make any effect. It > will always take the default value from HiveConf until you define it in > hive-site.xml. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15433) setting hive.warehouse.subdir.inherit.perms in HIVE won't overwrite it in hive configuration
[ https://issues.apache.org/jira/browse/HIVE-15433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854210#comment-15854210 ] Alina Abramova commented on HIVE-15433: --- If it's possible please do rebuild thrift files before building the project. > setting hive.warehouse.subdir.inherit.perms in HIVE won't overwrite it in > hive configuration > > > Key: HIVE-15433 > URL: https://issues.apache.org/jira/browse/HIVE-15433 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 1.0.0, 1.2.0, 2.0.0 >Reporter: Alina Abramova >Assignee: Alina Abramova > Fix For: 1.2.0 > > Attachments: HIVE-15433.1.patch, HIVE-15433-branch-1.2.patch > > > Setting hive.warehouse.subdir.inherit.perms in HIVE won't make any effect. It > will always take the default value from HiveConf until you define it in > hive-site.xml. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-11351) Column Found in more than One Tables/Subqueries
[ https://issues.apache.org/jira/browse/HIVE-11351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831631#comment-15831631 ] Alina Abramova commented on HIVE-11351: --- Yes, 1.2.1 is affected too. > Column Found in more than One Tables/Subqueries > --- > > Key: HIVE-11351 > URL: https://issues.apache.org/jira/browse/HIVE-11351 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 > Environment: HIVE 1.1.0 >Reporter: MK >Assignee: Alina Abramova > Attachments: HIVE-11351.2-branch-1.0.patch, > HIVE-11351-branch-1.0.patch > > > when execute a script: > INSERT overwrite TABLE tmp.tmp_dim_cpttr_categ1 >SELECT DISTINCT cur.categ_id AS categ_id, >cur.categ_code AS categ_code, >cur.categ_name AS categ_name, >cur.categ_parnt_id AS categ_parnt_id, >par.categ_name AS categ_parnt_name, >cur.mc_site_id AS mc_site_id >FROM tmp.tmp_dim_cpttr_categ cur >LEFT OUTER JOIN tmp.tmp_dim_cpttr_categ par >ON cur.categ_parnt_id = par.categ_id; > error occur : SemanticException Column categ_name Found in more than One > Tables/Subqueries > when modify the alias categ_name to categ_name_cur, it will be execute > successfully. > INSERT overwrite TABLE tmp.tmp_dim_cpttr_categ1 >SELECT DISTINCT cur.categ_id AS categ_id, >cur.categ_code AS categ_code, >cur.categ_name AS categ_name_cur, >cur.categ_parnt_id AS categ_parnt_id, >par.categ_name AS categ_parnt_name, >cur.mc_site_id AS mc_site_id >FROM tmp.tmp_dim_cpttr_categ cur >LEFT OUTER JOIN tmp.tmp_dim_cpttr_categ par >ON cur.categ_parnt_id = par.categ_id; > this happen when we upgrade hive from 0.10 to 1.1.0 . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11351) Column Found in more than One Tables/Subqueries
[ https://issues.apache.org/jira/browse/HIVE-11351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-11351: -- Affects Version/s: 1.2.1 Status: In Progress (was: Patch Available) > Column Found in more than One Tables/Subqueries > --- > > Key: HIVE-11351 > URL: https://issues.apache.org/jira/browse/HIVE-11351 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 > Environment: HIVE 1.1.0 >Reporter: MK >Assignee: Alina Abramova > Attachments: HIVE-11351.2-branch-1.0.patch, > HIVE-11351-branch-1.0.patch > > > when execute a script: > INSERT overwrite TABLE tmp.tmp_dim_cpttr_categ1 >SELECT DISTINCT cur.categ_id AS categ_id, >cur.categ_code AS categ_code, >cur.categ_name AS categ_name, >cur.categ_parnt_id AS categ_parnt_id, >par.categ_name AS categ_parnt_name, >cur.mc_site_id AS mc_site_id >FROM tmp.tmp_dim_cpttr_categ cur >LEFT OUTER JOIN tmp.tmp_dim_cpttr_categ par >ON cur.categ_parnt_id = par.categ_id; > error occur : SemanticException Column categ_name Found in more than One > Tables/Subqueries > when modify the alias categ_name to categ_name_cur, it will be execute > successfully. > INSERT overwrite TABLE tmp.tmp_dim_cpttr_categ1 >SELECT DISTINCT cur.categ_id AS categ_id, >cur.categ_code AS categ_code, >cur.categ_name AS categ_name_cur, >cur.categ_parnt_id AS categ_parnt_id, >par.categ_name AS categ_parnt_name, >cur.mc_site_id AS mc_site_id >FROM tmp.tmp_dim_cpttr_categ cur >LEFT OUTER JOIN tmp.tmp_dim_cpttr_categ par >ON cur.categ_parnt_id = par.categ_id; > this happen when we upgrade hive from 0.10 to 1.1.0 . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15433) setting hive.warehouse.subdir.inherit.perms in HIVE won't overwrite it in hive configuration
[ https://issues.apache.org/jira/browse/HIVE-15433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-15433: -- Issue Type: Improvement (was: Bug) > setting hive.warehouse.subdir.inherit.perms in HIVE won't overwrite it in > hive configuration > > > Key: HIVE-15433 > URL: https://issues.apache.org/jira/browse/HIVE-15433 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 1.0.0, 1.2.0, 2.0.0 >Reporter: Alina Abramova >Assignee: Alina Abramova > Fix For: 1.2.0 > > Attachments: HIVE-15433.1.patch, HIVE-15433-branch-1.2.patch > > > Setting hive.warehouse.subdir.inherit.perms in HIVE won't make any effect. It > will always take the default value from HiveConf until you define it in > hive-site.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15433) setting hive.warehouse.subdir.inherit.perms in HIVE won't overwrite it in hive configuration
[ https://issues.apache.org/jira/browse/HIVE-15433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-15433: -- Attachment: HIVE-15433-branch-1.2.patch > setting hive.warehouse.subdir.inherit.perms in HIVE won't overwrite it in > hive configuration > > > Key: HIVE-15433 > URL: https://issues.apache.org/jira/browse/HIVE-15433 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.0.0, 1.2.0, 2.0.0 >Reporter: Alina Abramova >Assignee: Alina Abramova > Fix For: 1.2.0 > > Attachments: HIVE-15433-branch-1.2.patch, HIVE-15433.1.patch > > > Setting hive.warehouse.subdir.inherit.perms in HIVE won't make any effect. It > will always take the default value from HiveConf until you define it in > hive-site.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15433) setting hive.warehouse.subdir.inherit.perms in HIVE won't overwrite it in hive configuration
[ https://issues.apache.org/jira/browse/HIVE-15433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-15433: -- Fix Version/s: (was: 2.1.0) 1.2.0 > setting hive.warehouse.subdir.inherit.perms in HIVE won't overwrite it in > hive configuration > > > Key: HIVE-15433 > URL: https://issues.apache.org/jira/browse/HIVE-15433 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.0.0, 1.2.0, 2.0.0 >Reporter: Alina Abramova >Assignee: Alina Abramova > Fix For: 1.2.0 > > Attachments: HIVE-15433-branch-1.2.patch, HIVE-15433.1.patch > > > Setting hive.warehouse.subdir.inherit.perms in HIVE won't make any effect. It > will always take the default value from HiveConf until you define it in > hive-site.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15433) setting hive.warehouse.subdir.inherit.perms in HIVE won't overwrite it in hive configuration
[ https://issues.apache.org/jira/browse/HIVE-15433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15772981#comment-15772981 ] Alina Abramova commented on HIVE-15433: --- I've changed hive_metastore.thrift and I think to make patch actually work you should recreate files that are related to this thrift file. ROOT-CAUSE: metastore server won't change hive configuration that were set for session through hive cli. Same issue with other data structures related to create functionality. > setting hive.warehouse.subdir.inherit.perms in HIVE won't overwrite it in > hive configuration > > > Key: HIVE-15433 > URL: https://issues.apache.org/jira/browse/HIVE-15433 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.0.0, 1.2.0, 2.0.0 >Reporter: Alina Abramova >Assignee: Alina Abramova > Fix For: 2.1.0 > > Attachments: HIVE-15433.1.patch > > > Setting hive.warehouse.subdir.inherit.perms in HIVE won't make any effect. It > will always take the default value from HiveConf until you define it in > hive-site.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15433) setting hive.warehouse.subdir.inherit.perms in HIVE won't overwrite it in hive configuration
[ https://issues.apache.org/jira/browse/HIVE-15433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-15433: -- Fix Version/s: 2.1.0 Status: Patch Available (was: Open) > setting hive.warehouse.subdir.inherit.perms in HIVE won't overwrite it in > hive configuration > > > Key: HIVE-15433 > URL: https://issues.apache.org/jira/browse/HIVE-15433 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.0.0, 1.2.0, 1.0.0 >Reporter: Alina Abramova >Assignee: Alina Abramova > Fix For: 2.1.0 > > Attachments: HIVE-15433.1.patch > > > Setting hive.warehouse.subdir.inherit.perms in HIVE won't make any effect. It > will always take the default value from HiveConf until you define it in > hive-site.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15433) setting hive.warehouse.subdir.inherit.perms in HIVE won't overwrite it in hive configuration
[ https://issues.apache.org/jira/browse/HIVE-15433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-15433: -- Attachment: HIVE-15433.1.patch > setting hive.warehouse.subdir.inherit.perms in HIVE won't overwrite it in > hive configuration > > > Key: HIVE-15433 > URL: https://issues.apache.org/jira/browse/HIVE-15433 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.0.0, 1.2.0, 2.0.0 >Reporter: Alina Abramova >Assignee: Alina Abramova > Attachments: HIVE-15433.1.patch > > > Setting hive.warehouse.subdir.inherit.perms in HIVE won't make any effect. It > will always take the default value from HiveConf until you define it in > hive-site.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-15433) setting hive.warehouse.subdir.inherit.perms in HIVE won't overwrite it in hive configuration
[ https://issues.apache.org/jira/browse/HIVE-15433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova reassigned HIVE-15433: - Assignee: Alina Abramova > setting hive.warehouse.subdir.inherit.perms in HIVE won't overwrite it in > hive configuration > > > Key: HIVE-15433 > URL: https://issues.apache.org/jira/browse/HIVE-15433 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.0.0, 1.2.0, 2.0.0 >Reporter: Alina Abramova >Assignee: Alina Abramova > Attachments: HIVE-15433.1.patch > > > Setting hive.warehouse.subdir.inherit.perms in HIVE won't make any effect. It > will always take the default value from HiveConf until you define it in > hive-site.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13000) Hive returns useless parsing error
[ https://issues.apache.org/jira/browse/HIVE-13000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15647490#comment-15647490 ] Alina Abramova commented on HIVE-13000: --- Test case was added > Hive returns useless parsing error > --- > > Key: HIVE-13000 > URL: https://issues.apache.org/jira/browse/HIVE-13000 > Project: Hive > Issue Type: Bug >Affects Versions: 0.13.0, 1.0.0, 1.2.1, 2.2.0 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Attachments: HIVE-13000.1.patch, HIVE-13000.2.patch, > HIVE-13000.3.patch, HIVE-13000.4.patch, HIVE-13000.5.patch > > > When I run query like these I receive unclear exception > hive> SELECT record FROM ctest GROUP BY record.instance_id; > FAILED: SemanticException Error in parsing > It will be clearer if it would be like: > hive> SELECT record FROM ctest GROUP BY record.instance_id; > FAILED: SemanticException Expression not in GROUP BY key record -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13000) Hive returns useless parsing error
[ https://issues.apache.org/jira/browse/HIVE-13000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-13000: -- Attachment: HIVE-13000.5.patch > Hive returns useless parsing error > --- > > Key: HIVE-13000 > URL: https://issues.apache.org/jira/browse/HIVE-13000 > Project: Hive > Issue Type: Bug >Affects Versions: 0.13.0, 1.0.0, 1.2.1, 2.2.0 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Attachments: HIVE-13000.1.patch, HIVE-13000.2.patch, > HIVE-13000.3.patch, HIVE-13000.4.patch, HIVE-13000.5.patch > > > When I run query like these I receive unclear exception > hive> SELECT record FROM ctest GROUP BY record.instance_id; > FAILED: SemanticException Error in parsing > It will be clearer if it would be like: > hive> SELECT record FROM ctest GROUP BY record.instance_id; > FAILED: SemanticException Expression not in GROUP BY key record -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13000) Hive returns useless parsing error
[ https://issues.apache.org/jira/browse/HIVE-13000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-13000: -- Affects Version/s: 2.2.0 > Hive returns useless parsing error > --- > > Key: HIVE-13000 > URL: https://issues.apache.org/jira/browse/HIVE-13000 > Project: Hive > Issue Type: Bug >Affects Versions: 0.13.0, 1.0.0, 1.2.1, 2.2.0 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Attachments: HIVE-13000.1.patch, HIVE-13000.2.patch, > HIVE-13000.3.patch, HIVE-13000.4.patch > > > When I run query like these I receive unclear exception > hive> SELECT record FROM ctest GROUP BY record.instance_id; > FAILED: SemanticException Error in parsing > It will be clearer if it would be like: > hive> SELECT record FROM ctest GROUP BY record.instance_id; > FAILED: SemanticException Expression not in GROUP BY key record -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13703) "msck repair" on table with non-partition subdirectories reporting partitions not in metastore
[
https://issues.apache.org/jira/browse/HIVE-13703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509475#comment-15509475
]
Alina Abramova commented on HIVE-13703:
---
Oh yeah, now I see that HIVE-14511 really fixes this issue. Thank you
This bug could be regarded as resolved
> "msck repair" on table with non-partition subdirectories reporting partitions
> not in metastore
> --
>
> Key: HIVE-13703
> URL: https://issues.apache.org/jira/browse/HIVE-13703
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 0.14.0, 1.0.0, 1.2.1
>Reporter: Ana Gillan
>Assignee: Alina Abramova
> Attachments: HIVE-13703.patch
>
>
> PROBLEM: Subdirectories created with UNION ALL are listed in {{show
> partitions}} output, but show up as {{Partitions not in metastore}} in {{msck
> repair}} output.
> STEPS TO REPRODUCE: Table created from {{CTAS ... UNION ALL}} DDL
> {code}
> hive> msck repair table meter_001;
> OK
> Partitions not in metastore: meter_001:tech_datestamp=2016-03-09/1
> meter_001:tech_datestamp=2016-03-09/2 meter_001:tech_datestamp=2016-03-10/1
> meter_001:tech_datestamp=2016-03-10/2 meter_001:tech_datestamp=2016-03-11/1
> meter_001:tech_datestamp=2016-03-11/2 meter_001:tech_datestamp=2016-03-12/1
> meter_001:tech_datestamp=2016-03-12/2 meter_001:tech_datestamp=2016-03-13/1
> meter_001:tech_datestamp=2016-03-13/2 meter_001:tech_datestamp=2016-03-14/1
> meter_001:tech_datestamp=2016-03-14/2 meter_001:tech_datestamp=2016-03-15/1
> meter_001:tech_datestamp=2016-03-15/2 meter_001:tech_datestamp=2016-03-16/1
> meter_001:tech_datestamp=2016-03-16/2 meter_001:tech_datestamp=2016-03-17/1
> meter_001:tech_datestamp=2016-03-17/2 meter_001:tech_datestamp=2016-03-18/1
> meter_001:tech_datestamp=2016-03-18/2 meter_001:tech_datestamp=2016-03-19/1
> meter_001:tech_datestamp=2016-03-19/2 meter_001:tech_datestamp=2016-03-20/1
> meter_001:tech_datestamp=2016-03-20/2 meter_001:tech_datestamp=2016-03-21/1
> meter_001:tech_datestamp=2016-03-21/2 meter_001:tech_datestamp=2016-03-22/1
> meter_001:tech_datestamp=2016-03-22/2 meter_001:tech_datestamp=2016-03-23/1
> meter_001:tech_datestamp=2016-03-23/2 meter_001:tech_datestamp=2016-03-24/1
> meter_001:tech_datestamp=2016-03-24/2 meter_001:tech_datestamp=2016-03-25/1
> meter_001:tech_datestamp=2016-03-25/2 meter_001:tech_datestamp=2016-03-26/1
> meter_001:tech_datestamp=2016-03-26/2 meter_001:tech_datestamp=2016-03-27/1
> meter_001:tech_datestamp=2016-03-27/2 meter_001:tech_datestamp=2016-03-28/1
> meter_001:tech_datestamp=2016-03-28/2 meter_001:tech_datestamp=2016-03-29/1
> meter_001:tech_datestamp=2016-03-29/2 meter_001:tech_datestamp=2016-03-30/1
> meter_001:tech_datestamp=2016-03-30/2 meter_001:tech_datestamp=2016-03-31/1
> meter_001:tech_datestamp=2016-03-31/2 meter_001:tech_datestamp=2016-04-01/1
> meter_001:tech_datestamp=2016-04-01/2 meter_001:tech_datestamp=2016-04-02/1
> meter_001:tech_datestamp=2016-04-02/2 meter_001:tech_datestamp=2016-04-03/1
> meter_001:tech_datestamp=2016-04-03/2 meter_001:tech_datestamp=2016-04-04/1
> meter_001:tech_datestamp=2016-04-04/2 meter_001:tech_datestamp=2016-04-05/1
> meter_001:tech_datestamp=2016-04-05/2 meter_001:tech_datestamp=2016-04-06/1
> meter_001:tech_datestamp=2016-04-06/2
> Time taken: 15.996 seconds, Fetched: 1 row(s)
> {code}
> {code}
> hive> show partitions meter_001;
> OK
> tech_datestamp=2016-03-09
> tech_datestamp=2016-03-10
> tech_datestamp=2016-03-11
> tech_datestamp=2016-03-12
> tech_datestamp=2016-03-13
> tech_datestamp=2016-03-14
> tech_datestamp=2016-03-15
> tech_datestamp=2016-03-16
> tech_datestamp=2016-03-17
> tech_datestamp=2016-03-18
> tech_datestamp=2016-03-19
> tech_datestamp=2016-03-20
> tech_datestamp=2016-03-21
> tech_datestamp=2016-03-22
> tech_datestamp=2016-03-23
> tech_datestamp=2016-03-24
> tech_datestamp=2016-03-25
> tech_datestamp=2016-03-26
> tech_datestamp=2016-03-27
> tech_datestamp=2016-03-28
> tech_datestamp=2016-03-29
> tech_datestamp=2016-03-30
> tech_datestamp=2016-03-31
> tech_datestamp=2016-04-01
> tech_datestamp=2016-04-02
> tech_datestamp=2016-04-03
> tech_datestamp=2016-04-04
> tech_datestamp=2016-04-05
> tech_datestamp=2016-04-06
> Time taken: 0.417 seconds, Fetched: 29 row(s)
> {code}
> Ideally msck repair should ignore subdirectory if that additional partition
> column doesn't exist.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13703) "msck repair" on table with non-partition subdirectories reporting partitions not in metastore
[
https://issues.apache.org/jira/browse/HIVE-13703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alina Abramova updated HIVE-13703:
--
Attachment: HIVE-13703.patch
This patch fixes this issue. Could somebody review it?
> "msck repair" on table with non-partition subdirectories reporting partitions
> not in metastore
> --
>
> Key: HIVE-13703
> URL: https://issues.apache.org/jira/browse/HIVE-13703
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 0.14.0, 1.0.0, 1.2.1
>Reporter: Ana Gillan
>Assignee: Alina Abramova
> Attachments: HIVE-13703.patch
>
>
> PROBLEM: Subdirectories created with UNION ALL are listed in {{show
> partitions}} output, but show up as {{Partitions not in metastore}} in {{msck
> repair}} output.
> STEPS TO REPRODUCE: Table created from {{CTAS ... UNION ALL}} DDL
> {code}
> hive> msck repair table meter_001;
> OK
> Partitions not in metastore: meter_001:tech_datestamp=2016-03-09/1
> meter_001:tech_datestamp=2016-03-09/2 meter_001:tech_datestamp=2016-03-10/1
> meter_001:tech_datestamp=2016-03-10/2 meter_001:tech_datestamp=2016-03-11/1
> meter_001:tech_datestamp=2016-03-11/2 meter_001:tech_datestamp=2016-03-12/1
> meter_001:tech_datestamp=2016-03-12/2 meter_001:tech_datestamp=2016-03-13/1
> meter_001:tech_datestamp=2016-03-13/2 meter_001:tech_datestamp=2016-03-14/1
> meter_001:tech_datestamp=2016-03-14/2 meter_001:tech_datestamp=2016-03-15/1
> meter_001:tech_datestamp=2016-03-15/2 meter_001:tech_datestamp=2016-03-16/1
> meter_001:tech_datestamp=2016-03-16/2 meter_001:tech_datestamp=2016-03-17/1
> meter_001:tech_datestamp=2016-03-17/2 meter_001:tech_datestamp=2016-03-18/1
> meter_001:tech_datestamp=2016-03-18/2 meter_001:tech_datestamp=2016-03-19/1
> meter_001:tech_datestamp=2016-03-19/2 meter_001:tech_datestamp=2016-03-20/1
> meter_001:tech_datestamp=2016-03-20/2 meter_001:tech_datestamp=2016-03-21/1
> meter_001:tech_datestamp=2016-03-21/2 meter_001:tech_datestamp=2016-03-22/1
> meter_001:tech_datestamp=2016-03-22/2 meter_001:tech_datestamp=2016-03-23/1
> meter_001:tech_datestamp=2016-03-23/2 meter_001:tech_datestamp=2016-03-24/1
> meter_001:tech_datestamp=2016-03-24/2 meter_001:tech_datestamp=2016-03-25/1
> meter_001:tech_datestamp=2016-03-25/2 meter_001:tech_datestamp=2016-03-26/1
> meter_001:tech_datestamp=2016-03-26/2 meter_001:tech_datestamp=2016-03-27/1
> meter_001:tech_datestamp=2016-03-27/2 meter_001:tech_datestamp=2016-03-28/1
> meter_001:tech_datestamp=2016-03-28/2 meter_001:tech_datestamp=2016-03-29/1
> meter_001:tech_datestamp=2016-03-29/2 meter_001:tech_datestamp=2016-03-30/1
> meter_001:tech_datestamp=2016-03-30/2 meter_001:tech_datestamp=2016-03-31/1
> meter_001:tech_datestamp=2016-03-31/2 meter_001:tech_datestamp=2016-04-01/1
> meter_001:tech_datestamp=2016-04-01/2 meter_001:tech_datestamp=2016-04-02/1
> meter_001:tech_datestamp=2016-04-02/2 meter_001:tech_datestamp=2016-04-03/1
> meter_001:tech_datestamp=2016-04-03/2 meter_001:tech_datestamp=2016-04-04/1
> meter_001:tech_datestamp=2016-04-04/2 meter_001:tech_datestamp=2016-04-05/1
> meter_001:tech_datestamp=2016-04-05/2 meter_001:tech_datestamp=2016-04-06/1
> meter_001:tech_datestamp=2016-04-06/2
> Time taken: 15.996 seconds, Fetched: 1 row(s)
> {code}
> {code}
> hive> show partitions meter_001;
> OK
> tech_datestamp=2016-03-09
> tech_datestamp=2016-03-10
> tech_datestamp=2016-03-11
> tech_datestamp=2016-03-12
> tech_datestamp=2016-03-13
> tech_datestamp=2016-03-14
> tech_datestamp=2016-03-15
> tech_datestamp=2016-03-16
> tech_datestamp=2016-03-17
> tech_datestamp=2016-03-18
> tech_datestamp=2016-03-19
> tech_datestamp=2016-03-20
> tech_datestamp=2016-03-21
> tech_datestamp=2016-03-22
> tech_datestamp=2016-03-23
> tech_datestamp=2016-03-24
> tech_datestamp=2016-03-25
> tech_datestamp=2016-03-26
> tech_datestamp=2016-03-27
> tech_datestamp=2016-03-28
> tech_datestamp=2016-03-29
> tech_datestamp=2016-03-30
> tech_datestamp=2016-03-31
> tech_datestamp=2016-04-01
> tech_datestamp=2016-04-02
> tech_datestamp=2016-04-03
> tech_datestamp=2016-04-04
> tech_datestamp=2016-04-05
> tech_datestamp=2016-04-06
> Time taken: 0.417 seconds, Fetched: 29 row(s)
> {code}
> Ideally msck repair should ignore subdirectory if that additional partition
> column doesn't exist.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-13703) "msck repair" on table with non-partition subdirectories reporting partitions not in metastore
[
https://issues.apache.org/jira/browse/HIVE-13703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alina Abramova reassigned HIVE-13703:
-
Assignee: Alina Abramova
> "msck repair" on table with non-partition subdirectories reporting partitions
> not in metastore
> --
>
> Key: HIVE-13703
> URL: https://issues.apache.org/jira/browse/HIVE-13703
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 0.14.0, 1.0.0, 1.2.1
>Reporter: Ana Gillan
>Assignee: Alina Abramova
>
> PROBLEM: Subdirectories created with UNION ALL are listed in {{show
> partitions}} output, but show up as {{Partitions not in metastore}} in {{msck
> repair}} output.
> STEPS TO REPRODUCE: Table created from {{CTAS ... UNION ALL}} DDL
> {code}
> hive> msck repair table meter_001;
> OK
> Partitions not in metastore: meter_001:tech_datestamp=2016-03-09/1
> meter_001:tech_datestamp=2016-03-09/2 meter_001:tech_datestamp=2016-03-10/1
> meter_001:tech_datestamp=2016-03-10/2 meter_001:tech_datestamp=2016-03-11/1
> meter_001:tech_datestamp=2016-03-11/2 meter_001:tech_datestamp=2016-03-12/1
> meter_001:tech_datestamp=2016-03-12/2 meter_001:tech_datestamp=2016-03-13/1
> meter_001:tech_datestamp=2016-03-13/2 meter_001:tech_datestamp=2016-03-14/1
> meter_001:tech_datestamp=2016-03-14/2 meter_001:tech_datestamp=2016-03-15/1
> meter_001:tech_datestamp=2016-03-15/2 meter_001:tech_datestamp=2016-03-16/1
> meter_001:tech_datestamp=2016-03-16/2 meter_001:tech_datestamp=2016-03-17/1
> meter_001:tech_datestamp=2016-03-17/2 meter_001:tech_datestamp=2016-03-18/1
> meter_001:tech_datestamp=2016-03-18/2 meter_001:tech_datestamp=2016-03-19/1
> meter_001:tech_datestamp=2016-03-19/2 meter_001:tech_datestamp=2016-03-20/1
> meter_001:tech_datestamp=2016-03-20/2 meter_001:tech_datestamp=2016-03-21/1
> meter_001:tech_datestamp=2016-03-21/2 meter_001:tech_datestamp=2016-03-22/1
> meter_001:tech_datestamp=2016-03-22/2 meter_001:tech_datestamp=2016-03-23/1
> meter_001:tech_datestamp=2016-03-23/2 meter_001:tech_datestamp=2016-03-24/1
> meter_001:tech_datestamp=2016-03-24/2 meter_001:tech_datestamp=2016-03-25/1
> meter_001:tech_datestamp=2016-03-25/2 meter_001:tech_datestamp=2016-03-26/1
> meter_001:tech_datestamp=2016-03-26/2 meter_001:tech_datestamp=2016-03-27/1
> meter_001:tech_datestamp=2016-03-27/2 meter_001:tech_datestamp=2016-03-28/1
> meter_001:tech_datestamp=2016-03-28/2 meter_001:tech_datestamp=2016-03-29/1
> meter_001:tech_datestamp=2016-03-29/2 meter_001:tech_datestamp=2016-03-30/1
> meter_001:tech_datestamp=2016-03-30/2 meter_001:tech_datestamp=2016-03-31/1
> meter_001:tech_datestamp=2016-03-31/2 meter_001:tech_datestamp=2016-04-01/1
> meter_001:tech_datestamp=2016-04-01/2 meter_001:tech_datestamp=2016-04-02/1
> meter_001:tech_datestamp=2016-04-02/2 meter_001:tech_datestamp=2016-04-03/1
> meter_001:tech_datestamp=2016-04-03/2 meter_001:tech_datestamp=2016-04-04/1
> meter_001:tech_datestamp=2016-04-04/2 meter_001:tech_datestamp=2016-04-05/1
> meter_001:tech_datestamp=2016-04-05/2 meter_001:tech_datestamp=2016-04-06/1
> meter_001:tech_datestamp=2016-04-06/2
> Time taken: 15.996 seconds, Fetched: 1 row(s)
> {code}
> {code}
> hive> show partitions meter_001;
> OK
> tech_datestamp=2016-03-09
> tech_datestamp=2016-03-10
> tech_datestamp=2016-03-11
> tech_datestamp=2016-03-12
> tech_datestamp=2016-03-13
> tech_datestamp=2016-03-14
> tech_datestamp=2016-03-15
> tech_datestamp=2016-03-16
> tech_datestamp=2016-03-17
> tech_datestamp=2016-03-18
> tech_datestamp=2016-03-19
> tech_datestamp=2016-03-20
> tech_datestamp=2016-03-21
> tech_datestamp=2016-03-22
> tech_datestamp=2016-03-23
> tech_datestamp=2016-03-24
> tech_datestamp=2016-03-25
> tech_datestamp=2016-03-26
> tech_datestamp=2016-03-27
> tech_datestamp=2016-03-28
> tech_datestamp=2016-03-29
> tech_datestamp=2016-03-30
> tech_datestamp=2016-03-31
> tech_datestamp=2016-04-01
> tech_datestamp=2016-04-02
> tech_datestamp=2016-04-03
> tech_datestamp=2016-04-04
> tech_datestamp=2016-04-05
> tech_datestamp=2016-04-06
> Time taken: 0.417 seconds, Fetched: 29 row(s)
> {code}
> Ideally msck repair should ignore subdirectory if that additional partition
> column doesn't exist.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13000) Hive returns useless parsing error
[ https://issues.apache.org/jira/browse/HIVE-13000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-13000: -- Attachment: HIVE-13000.4.patch > Hive returns useless parsing error > --- > > Key: HIVE-13000 > URL: https://issues.apache.org/jira/browse/HIVE-13000 > Project: Hive > Issue Type: Bug >Affects Versions: 0.13.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Attachments: HIVE-13000.1.patch, HIVE-13000.2.patch, > HIVE-13000.3.patch, HIVE-13000.4.patch > > > When I run query like these I receive unclear exception > hive> SELECT record FROM ctest GROUP BY record.instance_id; > FAILED: SemanticException Error in parsing > It will be clearer if it would be like: > hive> SELECT record FROM ctest GROUP BY record.instance_id; > FAILED: SemanticException Expression not in GROUP BY key record -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13000) Hive returns useless parsing error
[ https://issues.apache.org/jira/browse/HIVE-13000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15261769#comment-15261769 ] Alina Abramova commented on HIVE-13000: --- Please, could somebody take a look on this issue? > Hive returns useless parsing error > --- > > Key: HIVE-13000 > URL: https://issues.apache.org/jira/browse/HIVE-13000 > Project: Hive > Issue Type: Bug >Affects Versions: 0.13.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Attachments: HIVE-13000.1.patch, HIVE-13000.2.patch, > HIVE-13000.3.patch > > > When I run query like these I receive unclear exception > hive> SELECT record FROM ctest GROUP BY record.instance_id; > FAILED: SemanticException Error in parsing > It will be clearer if it would be like: > hive> SELECT record FROM ctest GROUP BY record.instance_id; > FAILED: SemanticException Expression not in GROUP BY key record -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12963) LIMIT statement with SORT BY creates additional MR job with hardcoded only one reducer
[ https://issues.apache.org/jira/browse/HIVE-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15261766#comment-15261766 ] Alina Abramova commented on HIVE-12963: --- [~sershe] Sorry, is this failed test related with the fix ? > LIMIT statement with SORT BY creates additional MR job with hardcoded only > one reducer > -- > > Key: HIVE-12963 > URL: https://issues.apache.org/jira/browse/HIVE-12963 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.0.0, 1.2.1, 0.13 >Reporter: Alina Abramova >Assignee: Alina Abramova > Attachments: HIVE-12963.1.patch, HIVE-12963.2.patch, > HIVE-12963.3.patch, HIVE-12963.4.patch, HIVE-12963.6.patch > > > I execute query: > hive> select age from test1 sort by age.age limit 10; > Total jobs = 2 > Launching Job 1 out of 2 > Number of reduce tasks not specified. Estimated from input data size: 1 > Launching Job 2 out of 2 > Number of reduce tasks determined at compile time: 1 > When I have a large number of rows then the last stage of the job takes a > long time. I think we could allow to user choose number of reducers of last > job or refuse extra MR job. > The same behavior I observed with querie: > hive> create table new_test as select age from test1 group by age.age limit > 10; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10867) ArrayIndexOutOfBoundsException LazyBinaryUtils.byteArrayToLong with Hive on Tez
[
https://issues.apache.org/jira/browse/HIVE-10867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alina Abramova updated HIVE-10867:
--
Attachment: HIVE-10867.patch
I created this patch based on https://issues.apache.org/jira/browse/HIVE-9517
I see that that fix works for Tez too
> ArrayIndexOutOfBoundsException LazyBinaryUtils.byteArrayToLong with Hive on
> Tez
> ---
>
> Key: HIVE-10867
> URL: https://issues.apache.org/jira/browse/HIVE-10867
> Project: Hive
> Issue Type: Bug
> Components: Hive, Tez
>Affects Versions: 0.14.0, 1.0.0
> Environment: Hortwonworks distribution 2.2.4-2
> Hive 0.14.0
> Tez 0.5.2.2.2.4.2-2 on cluster
> Tez 0.7.0 in local setup
>Reporter: Per Ullberg
>Assignee: Alina Abramova
> Attachments: HIVE-10867.patch
>
>
> Hi,
> The following query runs fine on map reduce engine but when setting the
> hive.exection.engine to tez it produces an ArrayIndexOutOfBoundsException.
> Query
> {code}
> create external table table_1 (id string, date string, amount bigint);
> insert into table table_1 values (305,'2013-03-02',3790);
> create external table table_2 (id string);
> insert into table table_2 VALUES (305);
> create external table table_3 (id string, date_3 string, amount_3 bigint);
> insert into table table_3 values (305,'2013-03-01',-1600);
> create external table table_4 (id bigint, str_4 string, amount_4 bigint);
> create table table_5
> as
> SELECT
> c.diff
> FROM (
> SELECT
> id AS id,
> date AS create_date,
> -amount AS diff
> FROM table_1
> UNION ALL
> SELECT
> p.id AS id,
> p.str_4 AS create_date,
> -p.amount_4 AS diff
> FROM table_4 p
> UNION ALL
> SELECT
> id,
> create_date,
> diff
> FROM (
> SELECT
> i.id AS id,
> tp.date_3 AS create_date,
> cast(amount_3 as double) AS diff
> FROM table_3 tp
> INNER JOIN table_2 i ON cast(tp.id as string) = cast(i.id as string)
> ) fees
> ) c
> INNER JOIN table_2 i ON cast(c.id as string) = cast(i.id as string);
> {code}
> Results with map reduce engine:
> {code}
> hive> select * from table_5;
> OK
> -1600.0
> -3790.0
> Time taken: 0.061 seconds, Fetched: 2 row(s)
> {code}
> Exception with tez engine:
> {code}
> Status: Failed
> Vertex failed, vertexName=Reducer 4, vertexId=vertex_1432809678493_0891_4_06,
> diagnostics=[Task failed, taskId=task_1432809678493_0891_4_06_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running
> task:java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0)
> {"key":{"reducesinkkey0":"305"},"value":{"_col1":-1600.0}}
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:186)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row (tag=0)
> {"key":{"reducesinkkey0":"305"},"value":{"_col1":-1600.0}}
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:337)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:218)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:168)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:163)
> ... 13 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 6
> at
>
[jira] [Updated] (HIVE-10867) ArrayIndexOutOfBoundsException LazyBinaryUtils.byteArrayToLong with Hive on Tez
[
https://issues.apache.org/jira/browse/HIVE-10867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alina Abramova updated HIVE-10867:
--
Affects Version/s: 1.0.0
Status: Patch Available (was: In Progress)
> ArrayIndexOutOfBoundsException LazyBinaryUtils.byteArrayToLong with Hive on
> Tez
> ---
>
> Key: HIVE-10867
> URL: https://issues.apache.org/jira/browse/HIVE-10867
> Project: Hive
> Issue Type: Bug
> Components: Hive, Tez
>Affects Versions: 1.0.0, 0.14.0
> Environment: Hortwonworks distribution 2.2.4-2
> Hive 0.14.0
> Tez 0.5.2.2.2.4.2-2 on cluster
> Tez 0.7.0 in local setup
>Reporter: Per Ullberg
>Assignee: Alina Abramova
> Attachments: HIVE-10867.patch
>
>
> Hi,
> The following query runs fine on map reduce engine but when setting the
> hive.exection.engine to tez it produces an ArrayIndexOutOfBoundsException.
> Query
> {code}
> create external table table_1 (id string, date string, amount bigint);
> insert into table table_1 values (305,'2013-03-02',3790);
> create external table table_2 (id string);
> insert into table table_2 VALUES (305);
> create external table table_3 (id string, date_3 string, amount_3 bigint);
> insert into table table_3 values (305,'2013-03-01',-1600);
> create external table table_4 (id bigint, str_4 string, amount_4 bigint);
> create table table_5
> as
> SELECT
> c.diff
> FROM (
> SELECT
> id AS id,
> date AS create_date,
> -amount AS diff
> FROM table_1
> UNION ALL
> SELECT
> p.id AS id,
> p.str_4 AS create_date,
> -p.amount_4 AS diff
> FROM table_4 p
> UNION ALL
> SELECT
> id,
> create_date,
> diff
> FROM (
> SELECT
> i.id AS id,
> tp.date_3 AS create_date,
> cast(amount_3 as double) AS diff
> FROM table_3 tp
> INNER JOIN table_2 i ON cast(tp.id as string) = cast(i.id as string)
> ) fees
> ) c
> INNER JOIN table_2 i ON cast(c.id as string) = cast(i.id as string);
> {code}
> Results with map reduce engine:
> {code}
> hive> select * from table_5;
> OK
> -1600.0
> -3790.0
> Time taken: 0.061 seconds, Fetched: 2 row(s)
> {code}
> Exception with tez engine:
> {code}
> Status: Failed
> Vertex failed, vertexName=Reducer 4, vertexId=vertex_1432809678493_0891_4_06,
> diagnostics=[Task failed, taskId=task_1432809678493_0891_4_06_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running
> task:java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0)
> {"key":{"reducesinkkey0":"305"},"value":{"_col1":-1600.0}}
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:186)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row (tag=0)
> {"key":{"reducesinkkey0":"305"},"value":{"_col1":-1600.0}}
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:337)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:218)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:168)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:163)
> ... 13 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 6
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.byteArrayToLong(LazyBinaryUtils.java:84)
> at
>
[jira] [Assigned] (HIVE-10867) ArrayIndexOutOfBoundsException LazyBinaryUtils.byteArrayToLong with Hive on Tez
[
https://issues.apache.org/jira/browse/HIVE-10867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alina Abramova reassigned HIVE-10867:
-
Assignee: Alina Abramova
> ArrayIndexOutOfBoundsException LazyBinaryUtils.byteArrayToLong with Hive on
> Tez
> ---
>
> Key: HIVE-10867
> URL: https://issues.apache.org/jira/browse/HIVE-10867
> Project: Hive
> Issue Type: Bug
> Components: Hive, Tez
>Affects Versions: 0.14.0
> Environment: Hortwonworks distribution 2.2.4-2
> Hive 0.14.0
> Tez 0.5.2.2.2.4.2-2 on cluster
> Tez 0.7.0 in local setup
>Reporter: Per Ullberg
>Assignee: Alina Abramova
>
> Hi,
> The following query runs fine on map reduce engine but when setting the
> hive.exection.engine to tez it produces an ArrayIndexOutOfBoundsException.
> Query
> {code}
> create external table table_1 (id string, date string, amount bigint);
> insert into table table_1 values (305,'2013-03-02',3790);
> create external table table_2 (id string);
> insert into table table_2 VALUES (305);
> create external table table_3 (id string, date_3 string, amount_3 bigint);
> insert into table table_3 values (305,'2013-03-01',-1600);
> create external table table_4 (id bigint, str_4 string, amount_4 bigint);
> create table table_5
> as
> SELECT
> c.diff
> FROM (
> SELECT
> id AS id,
> date AS create_date,
> -amount AS diff
> FROM table_1
> UNION ALL
> SELECT
> p.id AS id,
> p.str_4 AS create_date,
> -p.amount_4 AS diff
> FROM table_4 p
> UNION ALL
> SELECT
> id,
> create_date,
> diff
> FROM (
> SELECT
> i.id AS id,
> tp.date_3 AS create_date,
> cast(amount_3 as double) AS diff
> FROM table_3 tp
> INNER JOIN table_2 i ON cast(tp.id as string) = cast(i.id as string)
> ) fees
> ) c
> INNER JOIN table_2 i ON cast(c.id as string) = cast(i.id as string);
> {code}
> Results with map reduce engine:
> {code}
> hive> select * from table_5;
> OK
> -1600.0
> -3790.0
> Time taken: 0.061 seconds, Fetched: 2 row(s)
> {code}
> Exception with tez engine:
> {code}
> Status: Failed
> Vertex failed, vertexName=Reducer 4, vertexId=vertex_1432809678493_0891_4_06,
> diagnostics=[Task failed, taskId=task_1432809678493_0891_4_06_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running
> task:java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0)
> {"key":{"reducesinkkey0":"305"},"value":{"_col1":-1600.0}}
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:186)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row (tag=0)
> {"key":{"reducesinkkey0":"305"},"value":{"_col1":-1600.0}}
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:337)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:218)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:168)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:163)
> ... 13 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 6
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.byteArrayToLong(LazyBinaryUtils.java:84)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryDouble.init(LazyBinaryDouble.java:43)
> at
>
[jira] [Work started] (HIVE-10867) ArrayIndexOutOfBoundsException LazyBinaryUtils.byteArrayToLong with Hive on Tez
[
https://issues.apache.org/jira/browse/HIVE-10867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-10867 started by Alina Abramova.
-
> ArrayIndexOutOfBoundsException LazyBinaryUtils.byteArrayToLong with Hive on
> Tez
> ---
>
> Key: HIVE-10867
> URL: https://issues.apache.org/jira/browse/HIVE-10867
> Project: Hive
> Issue Type: Bug
> Components: Hive, Tez
>Affects Versions: 0.14.0
> Environment: Hortwonworks distribution 2.2.4-2
> Hive 0.14.0
> Tez 0.5.2.2.2.4.2-2 on cluster
> Tez 0.7.0 in local setup
>Reporter: Per Ullberg
>Assignee: Alina Abramova
>
> Hi,
> The following query runs fine on map reduce engine but when setting the
> hive.exection.engine to tez it produces an ArrayIndexOutOfBoundsException.
> Query
> {code}
> create external table table_1 (id string, date string, amount bigint);
> insert into table table_1 values (305,'2013-03-02',3790);
> create external table table_2 (id string);
> insert into table table_2 VALUES (305);
> create external table table_3 (id string, date_3 string, amount_3 bigint);
> insert into table table_3 values (305,'2013-03-01',-1600);
> create external table table_4 (id bigint, str_4 string, amount_4 bigint);
> create table table_5
> as
> SELECT
> c.diff
> FROM (
> SELECT
> id AS id,
> date AS create_date,
> -amount AS diff
> FROM table_1
> UNION ALL
> SELECT
> p.id AS id,
> p.str_4 AS create_date,
> -p.amount_4 AS diff
> FROM table_4 p
> UNION ALL
> SELECT
> id,
> create_date,
> diff
> FROM (
> SELECT
> i.id AS id,
> tp.date_3 AS create_date,
> cast(amount_3 as double) AS diff
> FROM table_3 tp
> INNER JOIN table_2 i ON cast(tp.id as string) = cast(i.id as string)
> ) fees
> ) c
> INNER JOIN table_2 i ON cast(c.id as string) = cast(i.id as string);
> {code}
> Results with map reduce engine:
> {code}
> hive> select * from table_5;
> OK
> -1600.0
> -3790.0
> Time taken: 0.061 seconds, Fetched: 2 row(s)
> {code}
> Exception with tez engine:
> {code}
> Status: Failed
> Vertex failed, vertexName=Reducer 4, vertexId=vertex_1432809678493_0891_4_06,
> diagnostics=[Task failed, taskId=task_1432809678493_0891_4_06_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running
> task:java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0)
> {"key":{"reducesinkkey0":"305"},"value":{"_col1":-1600.0}}
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:186)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row (tag=0)
> {"key":{"reducesinkkey0":"305"},"value":{"_col1":-1600.0}}
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:337)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:218)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:168)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:163)
> ... 13 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 6
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.byteArrayToLong(LazyBinaryUtils.java:84)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryDouble.init(LazyBinaryDouble.java:43)
> at
>
[jira] [Commented] (HIVE-11351) Column Found in more than One Tables/Subqueries
[ https://issues.apache.org/jira/browse/HIVE-11351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15237217#comment-15237217 ] Alina Abramova commented on HIVE-11351: --- After local testing I see that my fix was the cause of regression and I create the new patch > Column Found in more than One Tables/Subqueries > --- > > Key: HIVE-11351 > URL: https://issues.apache.org/jira/browse/HIVE-11351 > Project: Hive > Issue Type: Bug > Environment: HIVE 1.1.0 >Reporter: MK >Assignee: Alina Abramova > Attachments: HIVE-11351-branch-1.0.patch, > HIVE-11351.2-branch-1.0.patch > > > when execute a script: > INSERT overwrite TABLE tmp.tmp_dim_cpttr_categ1 >SELECT DISTINCT cur.categ_id AS categ_id, >cur.categ_code AS categ_code, >cur.categ_name AS categ_name, >cur.categ_parnt_id AS categ_parnt_id, >par.categ_name AS categ_parnt_name, >cur.mc_site_id AS mc_site_id >FROM tmp.tmp_dim_cpttr_categ cur >LEFT OUTER JOIN tmp.tmp_dim_cpttr_categ par >ON cur.categ_parnt_id = par.categ_id; > error occur : SemanticException Column categ_name Found in more than One > Tables/Subqueries > when modify the alias categ_name to categ_name_cur, it will be execute > successfully. > INSERT overwrite TABLE tmp.tmp_dim_cpttr_categ1 >SELECT DISTINCT cur.categ_id AS categ_id, >cur.categ_code AS categ_code, >cur.categ_name AS categ_name_cur, >cur.categ_parnt_id AS categ_parnt_id, >par.categ_name AS categ_parnt_name, >cur.mc_site_id AS mc_site_id >FROM tmp.tmp_dim_cpttr_categ cur >LEFT OUTER JOIN tmp.tmp_dim_cpttr_categ par >ON cur.categ_parnt_id = par.categ_id; > this happen when we upgrade hive from 0.10 to 1.1.0 . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11351) Column Found in more than One Tables/Subqueries
[ https://issues.apache.org/jira/browse/HIVE-11351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-11351: -- Attachment: HIVE-11351.2-branch-1.0.patch > Column Found in more than One Tables/Subqueries > --- > > Key: HIVE-11351 > URL: https://issues.apache.org/jira/browse/HIVE-11351 > Project: Hive > Issue Type: Bug > Environment: HIVE 1.1.0 >Reporter: MK >Assignee: Alina Abramova > Attachments: HIVE-11351-branch-1.0.patch, > HIVE-11351.2-branch-1.0.patch > > > when execute a script: > INSERT overwrite TABLE tmp.tmp_dim_cpttr_categ1 >SELECT DISTINCT cur.categ_id AS categ_id, >cur.categ_code AS categ_code, >cur.categ_name AS categ_name, >cur.categ_parnt_id AS categ_parnt_id, >par.categ_name AS categ_parnt_name, >cur.mc_site_id AS mc_site_id >FROM tmp.tmp_dim_cpttr_categ cur >LEFT OUTER JOIN tmp.tmp_dim_cpttr_categ par >ON cur.categ_parnt_id = par.categ_id; > error occur : SemanticException Column categ_name Found in more than One > Tables/Subqueries > when modify the alias categ_name to categ_name_cur, it will be execute > successfully. > INSERT overwrite TABLE tmp.tmp_dim_cpttr_categ1 >SELECT DISTINCT cur.categ_id AS categ_id, >cur.categ_code AS categ_code, >cur.categ_name AS categ_name_cur, >cur.categ_parnt_id AS categ_parnt_id, >par.categ_name AS categ_parnt_name, >cur.mc_site_id AS mc_site_id >FROM tmp.tmp_dim_cpttr_categ cur >LEFT OUTER JOIN tmp.tmp_dim_cpttr_categ par >ON cur.categ_parnt_id = par.categ_id; > this happen when we upgrade hive from 0.10 to 1.1.0 . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11351) Column Found in more than One Tables/Subqueries
[ https://issues.apache.org/jira/browse/HIVE-11351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-11351: -- Status: Patch Available (was: In Progress) > Column Found in more than One Tables/Subqueries > --- > > Key: HIVE-11351 > URL: https://issues.apache.org/jira/browse/HIVE-11351 > Project: Hive > Issue Type: Bug > Environment: HIVE 1.1.0 >Reporter: MK >Assignee: Alina Abramova > Attachments: HIVE-11351-branch-1.0.patch > > > when execute a script: > INSERT overwrite TABLE tmp.tmp_dim_cpttr_categ1 >SELECT DISTINCT cur.categ_id AS categ_id, >cur.categ_code AS categ_code, >cur.categ_name AS categ_name, >cur.categ_parnt_id AS categ_parnt_id, >par.categ_name AS categ_parnt_name, >cur.mc_site_id AS mc_site_id >FROM tmp.tmp_dim_cpttr_categ cur >LEFT OUTER JOIN tmp.tmp_dim_cpttr_categ par >ON cur.categ_parnt_id = par.categ_id; > error occur : SemanticException Column categ_name Found in more than One > Tables/Subqueries > when modify the alias categ_name to categ_name_cur, it will be execute > successfully. > INSERT overwrite TABLE tmp.tmp_dim_cpttr_categ1 >SELECT DISTINCT cur.categ_id AS categ_id, >cur.categ_code AS categ_code, >cur.categ_name AS categ_name_cur, >cur.categ_parnt_id AS categ_parnt_id, >par.categ_name AS categ_parnt_name, >cur.mc_site_id AS mc_site_id >FROM tmp.tmp_dim_cpttr_categ cur >LEFT OUTER JOIN tmp.tmp_dim_cpttr_categ par >ON cur.categ_parnt_id = par.categ_id; > this happen when we upgrade hive from 0.10 to 1.1.0 . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11351) Column Found in more than One Tables/Subqueries
[ https://issues.apache.org/jira/browse/HIVE-11351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-11351: -- Attachment: HIVE-11351-branch-1.0.patch > Column Found in more than One Tables/Subqueries > --- > > Key: HIVE-11351 > URL: https://issues.apache.org/jira/browse/HIVE-11351 > Project: Hive > Issue Type: Bug > Environment: HIVE 1.1.0 >Reporter: MK >Assignee: Alina Abramova > Attachments: HIVE-11351-branch-1.0.patch > > > when execute a script: > INSERT overwrite TABLE tmp.tmp_dim_cpttr_categ1 >SELECT DISTINCT cur.categ_id AS categ_id, >cur.categ_code AS categ_code, >cur.categ_name AS categ_name, >cur.categ_parnt_id AS categ_parnt_id, >par.categ_name AS categ_parnt_name, >cur.mc_site_id AS mc_site_id >FROM tmp.tmp_dim_cpttr_categ cur >LEFT OUTER JOIN tmp.tmp_dim_cpttr_categ par >ON cur.categ_parnt_id = par.categ_id; > error occur : SemanticException Column categ_name Found in more than One > Tables/Subqueries > when modify the alias categ_name to categ_name_cur, it will be execute > successfully. > INSERT overwrite TABLE tmp.tmp_dim_cpttr_categ1 >SELECT DISTINCT cur.categ_id AS categ_id, >cur.categ_code AS categ_code, >cur.categ_name AS categ_name_cur, >cur.categ_parnt_id AS categ_parnt_id, >par.categ_name AS categ_parnt_name, >cur.mc_site_id AS mc_site_id >FROM tmp.tmp_dim_cpttr_categ cur >LEFT OUTER JOIN tmp.tmp_dim_cpttr_categ par >ON cur.categ_parnt_id = par.categ_id; > this happen when we upgrade hive from 0.10 to 1.1.0 . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-11351) Column Found in more than One Tables/Subqueries
[ https://issues.apache.org/jira/browse/HIVE-11351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-11351 started by Alina Abramova. - > Column Found in more than One Tables/Subqueries > --- > > Key: HIVE-11351 > URL: https://issues.apache.org/jira/browse/HIVE-11351 > Project: Hive > Issue Type: Bug > Environment: HIVE 1.1.0 >Reporter: MK >Assignee: Alina Abramova > > when execute a script: > INSERT overwrite TABLE tmp.tmp_dim_cpttr_categ1 >SELECT DISTINCT cur.categ_id AS categ_id, >cur.categ_code AS categ_code, >cur.categ_name AS categ_name, >cur.categ_parnt_id AS categ_parnt_id, >par.categ_name AS categ_parnt_name, >cur.mc_site_id AS mc_site_id >FROM tmp.tmp_dim_cpttr_categ cur >LEFT OUTER JOIN tmp.tmp_dim_cpttr_categ par >ON cur.categ_parnt_id = par.categ_id; > error occur : SemanticException Column categ_name Found in more than One > Tables/Subqueries > when modify the alias categ_name to categ_name_cur, it will be execute > successfully. > INSERT overwrite TABLE tmp.tmp_dim_cpttr_categ1 >SELECT DISTINCT cur.categ_id AS categ_id, >cur.categ_code AS categ_code, >cur.categ_name AS categ_name_cur, >cur.categ_parnt_id AS categ_parnt_id, >par.categ_name AS categ_parnt_name, >cur.mc_site_id AS mc_site_id >FROM tmp.tmp_dim_cpttr_categ cur >LEFT OUTER JOIN tmp.tmp_dim_cpttr_categ par >ON cur.categ_parnt_id = par.categ_id; > this happen when we upgrade hive from 0.10 to 1.1.0 . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-11351) Column Found in more than One Tables/Subqueries
[ https://issues.apache.org/jira/browse/HIVE-11351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova reassigned HIVE-11351: - Assignee: Alina Abramova > Column Found in more than One Tables/Subqueries > --- > > Key: HIVE-11351 > URL: https://issues.apache.org/jira/browse/HIVE-11351 > Project: Hive > Issue Type: Bug > Environment: HIVE 1.1.0 >Reporter: MK >Assignee: Alina Abramova > > when execute a script: > INSERT overwrite TABLE tmp.tmp_dim_cpttr_categ1 >SELECT DISTINCT cur.categ_id AS categ_id, >cur.categ_code AS categ_code, >cur.categ_name AS categ_name, >cur.categ_parnt_id AS categ_parnt_id, >par.categ_name AS categ_parnt_name, >cur.mc_site_id AS mc_site_id >FROM tmp.tmp_dim_cpttr_categ cur >LEFT OUTER JOIN tmp.tmp_dim_cpttr_categ par >ON cur.categ_parnt_id = par.categ_id; > error occur : SemanticException Column categ_name Found in more than One > Tables/Subqueries > when modify the alias categ_name to categ_name_cur, it will be execute > successfully. > INSERT overwrite TABLE tmp.tmp_dim_cpttr_categ1 >SELECT DISTINCT cur.categ_id AS categ_id, >cur.categ_code AS categ_code, >cur.categ_name AS categ_name_cur, >cur.categ_parnt_id AS categ_parnt_id, >par.categ_name AS categ_parnt_name, >cur.mc_site_id AS mc_site_id >FROM tmp.tmp_dim_cpttr_categ cur >LEFT OUTER JOIN tmp.tmp_dim_cpttr_categ par >ON cur.categ_parnt_id = par.categ_id; > this happen when we upgrade hive from 0.10 to 1.1.0 . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12963) LIMIT statement with SORT BY creates additional MR job with hardcoded only one reducer
[ https://issues.apache.org/jira/browse/HIVE-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15204073#comment-15204073 ] Alina Abramova commented on HIVE-12963: --- Hi! What about this test? Does it not work as it should? > LIMIT statement with SORT BY creates additional MR job with hardcoded only > one reducer > -- > > Key: HIVE-12963 > URL: https://issues.apache.org/jira/browse/HIVE-12963 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.0.0, 1.2.1, 0.13 >Reporter: Alina Abramova >Assignee: Alina Abramova > Attachments: HIVE-12963.1.patch, HIVE-12963.2.patch, > HIVE-12963.3.patch, HIVE-12963.4.patch, HIVE-12963.6.patch > > > I execute query: > hive> select age from test1 sort by age.age limit 10; > Total jobs = 2 > Launching Job 1 out of 2 > Number of reduce tasks not specified. Estimated from input data size: 1 > Launching Job 2 out of 2 > Number of reduce tasks determined at compile time: 1 > When I have a large number of rows then the last stage of the job takes a > long time. I think we could allow to user choose number of reducers of last > job or refuse extra MR job. > The same behavior I observed with querie: > hive> create table new_test as select age from test1 group by age.age limit > 10; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12779) Buffer underflow when inserting data to table
[ https://issues.apache.org/jira/browse/HIVE-12779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15204052#comment-15204052 ] Alina Abramova commented on HIVE-12779: --- I investigated this issue and see that this bug appears randomly and for me only in case concurrently running jobs in Hive which address to the same table. I ran two queries in two beelines and about one of thirty attempts finished with this exception. After investigating issues with kryo's serializer I found that this exception appears in few cases: 1) Kryo is used in multithreading env. But I see that in Hive kryo created as thread local variable and this case is excluded, I think 2) Two Input uses the same buffer. As example Input inp1 = new Input(buff); Input inp2 = new Input(buff); For me suspected causes cases when Hive receives the stream to read from FileSystem, maybe two threads use the one path for reading a plan, but, in theory, it is excluded, because jobs can not have the one plan with one path. 3) When output stream was not closed after serialization, but I did not found such incidents in Hive's code after HIVE-8688. Maybe reporter could give more information about the query for finding 100% reproducable case. > Buffer underflow when inserting data to table > - > > Key: HIVE-12779 > URL: https://issues.apache.org/jira/browse/HIVE-12779 > Project: Hive > Issue Type: Bug > Components: Database/Schema, SQL > Environment: CDH 5.4.9 >Reporter: Ming Hsuan Tu >Assignee: Alan Gates > > I face a buffer underflow problem when inserting data to table from hive > 1.1.0. > the block size is 128 MB and the data size is only 10MB, but it gives me 891 > mappers. > Task with the most failures(4): > - > Task ID: > task_1451989578563_0001_m_08 > URL: > > http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1451989578563_0001=task_1451989578563_0001_m_08 > - > Diagnostic Messages for this Task: > Error: java.lang.RuntimeException: Failed to load plan: > hdfs://tpe-nn-3-1:8020/tmp/hive/alec.tu/af798488-dbf5-45da-8adb-e4f2ddde1242/hive_2016-01-05_18-34-26_864_3947114301988950007-1/-mr-10004/bb86c923-0dca-43cd-aa5d-ef575d764e06/map.xml: > org.apache.hive.com.esotericsoftware.kryo.KryoException: Buffer underflow. > at > org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:450) > at > org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:296) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:268) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:234) > at > org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:701) > at > org.apache.hadoop.mapred.MapTask$TrackedRecordReader.(MapTask.java:169) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: Buffer > underflow. > at > org.apache.hive.com.esotericsoftware.kryo.io.Input.require(Input.java:181) > at > org.apache.hive.com.esotericsoftware.kryo.io.Input.readBoolean(Input.java:783) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.UnsafeCacheFields$UnsafeBooleanField.read(UnsafeCacheFields.java:120) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) > at > org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672) > at > org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:1069) > at > org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:960) > at > org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:974) > at > org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:416) > ... 12 more > Container killed by the ApplicationMaster. > Container killed on request. Exit code is 143 > Container exited with a non-zero exit code 143 > Thank you. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15195422#comment-15195422 ] Alina Abramova commented on HIVE-12244: --- Could somebody apply my last patch locally and run this part of tests? > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Labels: patch > Fix For: 1.2.1 > > Attachments: HIVE-12244.1.patch, HIVE-12244.2.patch, > HIVE-12244.3.patch, HIVE-12244.4.patch, HIVE-12244.5.patch, > HIVE-12244.6.patch, HIVE-12244.7.patch, HIVE-12244.8.patch, > HIVE-12244.8.patch, HIVE-12244.9.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we use Path from org.apache.hadoop.fs we will avoid new problems > in future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15195283#comment-15195283 ] Alina Abramova commented on HIVE-12244: --- I see that most test that passed locally are failed by Jenkins. For example: I ran tests with -Dqfile_regex=smb_mapjoin.* Tests run: 33, Failures: 3, Errors: 0, Skipped: 0, Failed tests : org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_mapjoin11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_mapjoin13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_mapjoin15 >From Jenkins message failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_mapjoin1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_mapjoin10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_mapjoin11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_mapjoin2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_mapjoin3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_mapjoin4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_mapjoin5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_mapjoin6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_mapjoin7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_mapjoin8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_mapjoin9 I do not understand what happens and why these tests results are different. Could somebody run this part of tests ( -Dqfile_regex=smb_mapjoin.* ) locally and show me results please? Maybe I do something wrong. > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Labels: patch > Fix For: 1.2.1 > > Attachments: HIVE-12244.1.patch, HIVE-12244.2.patch, > HIVE-12244.3.patch, HIVE-12244.4.patch, HIVE-12244.5.patch, > HIVE-12244.6.patch, HIVE-12244.7.patch, HIVE-12244.8.patch, > HIVE-12244.8.patch, HIVE-12244.9.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we use Path from org.apache.hadoop.fs we will avoid new problems > in future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12963) LIMIT statement with SORT BY creates additional MR job with hardcoded only one reducer
[ https://issues.apache.org/jira/browse/HIVE-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15190963#comment-15190963 ] Alina Abramova commented on HIVE-12963: --- Anybody has comments for this issue? > LIMIT statement with SORT BY creates additional MR job with hardcoded only > one reducer > -- > > Key: HIVE-12963 > URL: https://issues.apache.org/jira/browse/HIVE-12963 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.0.0, 1.2.1, 0.13 >Reporter: Alina Abramova >Assignee: Alina Abramova > Attachments: HIVE-12963.1.patch, HIVE-12963.2.patch, > HIVE-12963.3.patch, HIVE-12963.4.patch, HIVE-12963.6.patch > > > I execute query: > hive> select age from test1 sort by age.age limit 10; > Total jobs = 2 > Launching Job 1 out of 2 > Number of reduce tasks not specified. Estimated from input data size: 1 > Launching Job 2 out of 2 > Number of reduce tasks determined at compile time: 1 > When I have a large number of rows then the last stage of the job takes a > long time. I think we could allow to user choose number of reducers of last > job or refuse extra MR job. > The same behavior I observed with querie: > hive> create table new_test as select age from test1 group by age.age limit > 10; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12244: -- Attachment: HIVE-12244.9.patch > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Labels: patch > Fix For: 1.2.1 > > Attachments: HIVE-12244.1.patch, HIVE-12244.2.patch, > HIVE-12244.3.patch, HIVE-12244.4.patch, HIVE-12244.5.patch, > HIVE-12244.6.patch, HIVE-12244.7.patch, HIVE-12244.8.patch, > HIVE-12244.8.patch, HIVE-12244.9.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we use Path from org.apache.hadoop.fs we will avoid new problems > in future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12244: -- Attachment: HIVE-12244.8.patch > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Labels: patch > Fix For: 1.2.1 > > Attachments: HIVE-12244.1.patch, HIVE-12244.2.patch, > HIVE-12244.3.patch, HIVE-12244.4.patch, HIVE-12244.5.patch, > HIVE-12244.6.patch, HIVE-12244.7.patch, HIVE-12244.8.patch, HIVE-12244.8.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we use Path from org.apache.hadoop.fs we will avoid new problems > in future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12963) LIMIT statement with SORT BY creates additional MR job with hardcoded only one reducer
[ https://issues.apache.org/jira/browse/HIVE-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12963: -- Attachment: HIVE-12963.6.patch > LIMIT statement with SORT BY creates additional MR job with hardcoded only > one reducer > -- > > Key: HIVE-12963 > URL: https://issues.apache.org/jira/browse/HIVE-12963 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.0.0, 1.2.1, 0.13 >Reporter: Alina Abramova >Assignee: Alina Abramova > Attachments: HIVE-12963.1.patch, HIVE-12963.2.patch, > HIVE-12963.3.patch, HIVE-12963.4.patch, HIVE-12963.6.patch > > > I execute query: > hive> select age from test1 sort by age.age limit 10; > Total jobs = 2 > Launching Job 1 out of 2 > Number of reduce tasks not specified. Estimated from input data size: 1 > Launching Job 2 out of 2 > Number of reduce tasks determined at compile time: 1 > When I have a large number of rows then the last stage of the job takes a > long time. I think we could allow to user choose number of reducers of last > job or refuse extra MR job. > The same behavior I observed with querie: > hive> create table new_test as select age from test1 group by age.age limit > 10; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12244: -- Attachment: HIVE-12244.8.patch > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Labels: patch > Fix For: 1.2.1 > > Attachments: HIVE-12244.1.patch, HIVE-12244.2.patch, > HIVE-12244.3.patch, HIVE-12244.4.patch, HIVE-12244.5.patch, > HIVE-12244.6.patch, HIVE-12244.7.patch, HIVE-12244.8.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we use Path from org.apache.hadoop.fs we will avoid new problems > in future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13000) Hive returns useless parsing error
[ https://issues.apache.org/jira/browse/HIVE-13000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147328#comment-15147328 ] Alina Abramova commented on HIVE-13000: --- Have we any mention about this ticket? > Hive returns useless parsing error > --- > > Key: HIVE-13000 > URL: https://issues.apache.org/jira/browse/HIVE-13000 > Project: Hive > Issue Type: Bug >Affects Versions: 0.13.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Attachments: HIVE-13000.1.patch, HIVE-13000.2.patch, > HIVE-13000.3.patch > > > When I run query like these I receive unclear exception > hive> SELECT record FROM ctest GROUP BY record.instance_id; > FAILED: SemanticException Error in parsing > It will be clearer if it would be like: > hive> SELECT record FROM ctest GROUP BY record.instance_id; > FAILED: SemanticException Expression not in GROUP BY key record -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12244: -- Attachment: HIVE-12244.7.patch > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Labels: patch > Fix For: 1.2.1 > > Attachments: HIVE-12244.1.patch, HIVE-12244.2.patch, > HIVE-12244.3.patch, HIVE-12244.4.patch, HIVE-12244.5.patch, > HIVE-12244.6.patch, HIVE-12244.7.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we use Path from org.apache.hadoop.fs we will avoid new problems > in future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13000) Hive returns useless parsing error
[ https://issues.apache.org/jira/browse/HIVE-13000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-13000: -- Attachment: HIVE-13000.3.patch > Hive returns useless parsing error > --- > > Key: HIVE-13000 > URL: https://issues.apache.org/jira/browse/HIVE-13000 > Project: Hive > Issue Type: Bug >Affects Versions: 0.13.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Attachments: HIVE-13000.1.patch, HIVE-13000.2.patch, > HIVE-13000.3.patch > > > When I run query like these I receive unclear exception > hive> SELECT record FROM ctest GROUP BY record.instance_id; > FAILED: SemanticException Error in parsing > It will be clearer if it would be like: > hive> SELECT record FROM ctest GROUP BY record.instance_id; > FAILED: SemanticException Expression not in GROUP BY key record -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12963) LIMIT statement with SORT BY creates additional MR job with hardcoded only one reducer
[ https://issues.apache.org/jira/browse/HIVE-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12963: -- Attachment: HIVE-12963.4.patch > LIMIT statement with SORT BY creates additional MR job with hardcoded only > one reducer > -- > > Key: HIVE-12963 > URL: https://issues.apache.org/jira/browse/HIVE-12963 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.0.0, 1.2.1, 0.13 >Reporter: Alina Abramova >Assignee: Alina Abramova > Attachments: HIVE-12963.1.patch, HIVE-12963.2.patch, > HIVE-12963.3.patch, HIVE-12963.4.patch > > > I execute query: > hive> select age from test1 sort by age.age limit 10; > Total jobs = 2 > Launching Job 1 out of 2 > Number of reduce tasks not specified. Estimated from input data size: 1 > Launching Job 2 out of 2 > Number of reduce tasks determined at compile time: 1 > When I have a large number of rows then the last stage of the job takes a > long time. I think we could allow to user choose number of reducers of last > job or refuse extra MR job. > The same behavior I observed with querie: > hive> create table new_test as select age from test1 group by age.age limit > 10; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13000) Hive returns useless parsing error
[ https://issues.apache.org/jira/browse/HIVE-13000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-13000: -- Attachment: HIVE-13000.2.patch > Hive returns useless parsing error > --- > > Key: HIVE-13000 > URL: https://issues.apache.org/jira/browse/HIVE-13000 > Project: Hive > Issue Type: Bug >Affects Versions: 0.13.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Attachments: HIVE-13000.1.patch, HIVE-13000.2.patch > > > When I run query like these I receive unclear exception > hive> SELECT record FROM ctest GROUP BY record.instance_id; > FAILED: SemanticException Error in parsing > It will be clearer if it would be like: > hive> SELECT record FROM ctest GROUP BY record.instance_id; > FAILED: SemanticException Expression not in GROUP BY key record -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12244: -- Attachment: HIVE-12244.6.patch > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Labels: patch > Fix For: 1.2.1 > > Attachments: HIVE-12244.1.patch, HIVE-12244.2.patch, > HIVE-12244.3.patch, HIVE-12244.4.patch, HIVE-12244.5.patch, HIVE-12244.6.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we use Path from org.apache.hadoop.fs we will avoid new problems > in future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13000) Hive returns useless parsing error
[ https://issues.apache.org/jira/browse/HIVE-13000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-13000: -- Attachment: HIVE-13000.1.patch > Hive returns useless parsing error > --- > > Key: HIVE-13000 > URL: https://issues.apache.org/jira/browse/HIVE-13000 > Project: Hive > Issue Type: Bug >Affects Versions: 0.13.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Attachments: HIVE-13000.1.patch > > > When I run query like these I receive unclear exception > hive> SELECT record FROM ctest GROUP BY record.instance_id; > FAILED: SemanticException Error in parsing > It will be clearer if it would be like: > hive> SELECT record FROM ctest GROUP BY record.instance_id; > FAILED: SemanticException Expression not in GROUP BY key record -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12244: -- Attachment: HIVE-12244.5.patch > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Labels: patch > Fix For: 1.2.1 > > Attachments: HIVE-12244.1.patch, HIVE-12244.2.patch, > HIVE-12244.3.patch, HIVE-12244.4.patch, HIVE-12244.5.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we use Path from org.apache.hadoop.fs we will avoid new problems > in future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12963) LIMIT statement with SORT BY creates additional MR job with hardcoded only one reducer
[ https://issues.apache.org/jira/browse/HIVE-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12963: -- Attachment: HIVE-12963.3.patch > LIMIT statement with SORT BY creates additional MR job with hardcoded only > one reducer > -- > > Key: HIVE-12963 > URL: https://issues.apache.org/jira/browse/HIVE-12963 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.0.0, 1.2.1, 0.13 >Reporter: Alina Abramova >Assignee: Alina Abramova > Attachments: HIVE-12963.1.patch, HIVE-12963.2.patch, > HIVE-12963.3.patch > > > I execute query: > hive> select age from test1 sort by age.age limit 10; > Total jobs = 2 > Launching Job 1 out of 2 > Number of reduce tasks not specified. Estimated from input data size: 1 > Launching Job 2 out of 2 > Number of reduce tasks determined at compile time: 1 > When I have a large number of rows then the last stage of the job takes a > long time. I think we could allow to user choose number of reducers of last > job or refuse extra MR job. > The same behavior I observed with querie: > hive> create table new_test as select age from test1 group by age.age limit > 10; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12244: -- Attachment: HIVE-12244.4.patch > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Labels: patch > Fix For: 1.2.1 > > Attachments: HIVE-12244.1.patch, HIVE-12244.2.patch, > HIVE-12244.3.patch, HIVE-12244.4.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we use Path from org.apache.hadoop.fs we will avoid new problems > in future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12963) LIMIT statement with SORT BY creates additional MR job with hardcoded only one reducer
[ https://issues.apache.org/jira/browse/HIVE-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12963: -- Attachment: (was: HIVE-12963.2.patch) > LIMIT statement with SORT BY creates additional MR job with hardcoded only > one reducer > -- > > Key: HIVE-12963 > URL: https://issues.apache.org/jira/browse/HIVE-12963 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.0.0, 1.2.1, 0.13 >Reporter: Alina Abramova >Assignee: Alina Abramova > Attachments: HIVE-12963.1.patch > > > I execute query: > hive> select age from test1 sort by age.age limit 10; > Total jobs = 2 > Launching Job 1 out of 2 > Number of reduce tasks not specified. Estimated from input data size: 1 > Launching Job 2 out of 2 > Number of reduce tasks determined at compile time: 1 > When I have a large number of rows then the last stage of the job takes a > long time. I think we could allow to user choose number of reducers of last > job or refuse extra MR job. > The same behavior I observed with queries: > hive> create table new_test as select age from test1 group by age.age limit > 10; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12963) LIMIT statement with SORT BY creates additional MR job with hardcoded only one reducer
[ https://issues.apache.org/jira/browse/HIVE-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12963: -- Attachment: HIVE-12963.2.patch > LIMIT statement with SORT BY creates additional MR job with hardcoded only > one reducer > -- > > Key: HIVE-12963 > URL: https://issues.apache.org/jira/browse/HIVE-12963 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.0.0, 1.2.1, 0.13 >Reporter: Alina Abramova >Assignee: Alina Abramova > Attachments: HIVE-12963.1.patch, HIVE-12963.2.patch > > > I execute query: > hive> select age from test1 sort by age.age limit 10; > Total jobs = 2 > Launching Job 1 out of 2 > Number of reduce tasks not specified. Estimated from input data size: 1 > Launching Job 2 out of 2 > Number of reduce tasks determined at compile time: 1 > When I have a large number of rows then the last stage of the job takes a > long time. I think we could allow to user choose number of reducers of last > job or refuse extra MR job. > The same behavior I observed with queries: > hive> create table new_test as select age from test1 group by age.age limit > 10; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12963) LIMIT statement with SORT BY creates additional MR job with hardcoded only one reducer
[ https://issues.apache.org/jira/browse/HIVE-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12963: -- Description: I execute query: hive> select age from test1 sort by age.age limit 10; Total jobs = 2 Launching Job 1 out of 2 Number of reduce tasks not specified. Estimated from input data size: 1 Launching Job 2 out of 2 Number of reduce tasks determined at compile time: 1 When I have a large number of rows then the last stage of the job takes a long time. I think we could allow to user choose number of reducers of last job or refuse extra MR job. The same behavior I observed with querie: hive> create table new_test as select age from test1 group by age.age limit 10; was: I execute query: hive> select age from test1 sort by age.age limit 10; Total jobs = 2 Launching Job 1 out of 2 Number of reduce tasks not specified. Estimated from input data size: 1 Launching Job 2 out of 2 Number of reduce tasks determined at compile time: 1 When I have a large number of rows then the last stage of the job takes a long time. I think we could allow to user choose number of reducers of last job or refuse extra MR job. The same behavior I observed with queries: hive> create table new_test as select age from test1 group by age.age limit 10; > LIMIT statement with SORT BY creates additional MR job with hardcoded only > one reducer > -- > > Key: HIVE-12963 > URL: https://issues.apache.org/jira/browse/HIVE-12963 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.0.0, 1.2.1, 0.13 >Reporter: Alina Abramova >Assignee: Alina Abramova > Attachments: HIVE-12963.1.patch, HIVE-12963.2.patch > > > I execute query: > hive> select age from test1 sort by age.age limit 10; > Total jobs = 2 > Launching Job 1 out of 2 > Number of reduce tasks not specified. Estimated from input data size: 1 > Launching Job 2 out of 2 > Number of reduce tasks determined at compile time: 1 > When I have a large number of rows then the last stage of the job takes a > long time. I think we could allow to user choose number of reducers of last > job or refuse extra MR job. > The same behavior I observed with querie: > hive> create table new_test as select age from test1 group by age.age limit > 10; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12963) LIMIT statement with SORT BY creates additional MR job with hardcoded only one reducer
[ https://issues.apache.org/jira/browse/HIVE-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12963: -- Attachment: (was: HIVE-12244.2.patch) > LIMIT statement with SORT BY creates additional MR job with hardcoded only > one reducer > -- > > Key: HIVE-12963 > URL: https://issues.apache.org/jira/browse/HIVE-12963 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.0.0, 1.2.1, 0.13 >Reporter: Alina Abramova >Assignee: Alina Abramova > Attachments: HIVE-12963.1.patch > > > I execute query: > hive> select age from test1 sort by age.age limit 10; > Total jobs = 2 > Launching Job 1 out of 2 > Number of reduce tasks not specified. Estimated from input data size: 1 > Launching Job 2 out of 2 > Number of reduce tasks determined at compile time: 1 > When I have a large number of rows then the last stage of the job takes a > long time. I think we could allow to user choose number of reducers of last > job or refuse extra MR job. > The same behavior I observed with queries: > hive> create table new_test as select age from test1 group by age.age limit > 10; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12963) LIMIT statement with SORT BY creates additional MR job with hardcoded only one reducer
[ https://issues.apache.org/jira/browse/HIVE-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12963: -- Attachment: HIVE-12244.2.patch > LIMIT statement with SORT BY creates additional MR job with hardcoded only > one reducer > -- > > Key: HIVE-12963 > URL: https://issues.apache.org/jira/browse/HIVE-12963 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.0.0, 1.2.1, 0.13 >Reporter: Alina Abramova >Assignee: Alina Abramova > Attachments: HIVE-12963.1.patch > > > I execute query: > hive> select age from test1 sort by age.age limit 10; > Total jobs = 2 > Launching Job 1 out of 2 > Number of reduce tasks not specified. Estimated from input data size: 1 > Launching Job 2 out of 2 > Number of reduce tasks determined at compile time: 1 > When I have a large number of rows then the last stage of the job takes a > long time. I think we could allow to user choose number of reducers of last > job or refuse extra MR job. > The same behavior I observed with queries: > hive> create table new_test as select age from test1 group by age.age limit > 10; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12963) LIMIT statement with SORT BY creates additional MR job with hardcoded only one reducer
[ https://issues.apache.org/jira/browse/HIVE-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12963: -- Attachment: HIVE-12963.2.patch > LIMIT statement with SORT BY creates additional MR job with hardcoded only > one reducer > -- > > Key: HIVE-12963 > URL: https://issues.apache.org/jira/browse/HIVE-12963 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.0.0, 1.2.1, 0.13 >Reporter: Alina Abramova >Assignee: Alina Abramova > Attachments: HIVE-12963.1.patch, HIVE-12963.2.patch > > > I execute query: > hive> select age from test1 sort by age.age limit 10; > Total jobs = 2 > Launching Job 1 out of 2 > Number of reduce tasks not specified. Estimated from input data size: 1 > Launching Job 2 out of 2 > Number of reduce tasks determined at compile time: 1 > When I have a large number of rows then the last stage of the job takes a > long time. I think we could allow to user choose number of reducers of last > job or refuse extra MR job. > The same behavior I observed with queries: > hive> create table new_test as select age from test1 group by age.age limit > 10; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12244: -- Attachment: HIVE-12244.3.patch > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Labels: patch > Fix For: 1.2.1 > > Attachments: HIVE-12244.1.patch, HIVE-12244.2.patch, > HIVE-12244.3.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we use Path from org.apache.hadoop.fs we will avoid new problems > in future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12963) LIMIT statement with SORT BY creates additional MR job with hardcoded only one reducer
[ https://issues.apache.org/jira/browse/HIVE-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15124823#comment-15124823 ] Alina Abramova commented on HIVE-12963: --- But I see that if line with creating of genReduceSinkPlan in method genLimitMapRedPlan is commented then finish set is sorted too. It means that we could refuse the creating of extra job, and do sorting in the same MR job, doesn't it? > LIMIT statement with SORT BY creates additional MR job with hardcoded only > one reducer > -- > > Key: HIVE-12963 > URL: https://issues.apache.org/jira/browse/HIVE-12963 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.0.0, 1.2.1, 0.13 >Reporter: Alina Abramova >Assignee: Alina Abramova > Attachments: HIVE-12963.1.patch > > > I execute query: > hive> select age from test1 sort by age.age limit 10; > Total jobs = 2 > Launching Job 1 out of 2 > Number of reduce tasks not specified. Estimated from input data size: 1 > Launching Job 2 out of 2 > Number of reduce tasks determined at compile time: 1 > When I have a large number of rows then the last stage of the job takes a > long time. I think we could allow to user choose number of reducers of last > job or refuse extra MR job. > The same behavior I observed with queries: > hive> create table new_test as select age from test1 group by age.age limit > 10; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12963) LIMIT statement with SORT BY creates additional MR job with hardcoded only one reducer
[ https://issues.apache.org/jira/browse/HIVE-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12963: -- Attachment: HIVE-12963.1.patch > LIMIT statement with SORT BY creates additional MR job with hardcoded only > one reducer > -- > > Key: HIVE-12963 > URL: https://issues.apache.org/jira/browse/HIVE-12963 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.0.0, 1.2.1, 0.13 >Reporter: Alina Abramova >Assignee: Alina Abramova > Attachments: HIVE-12963.1.patch > > > I execute query: > hive> select age from test1 sort by age.age limit 10; > Total jobs = 2 > Launching Job 1 out of 2 > Number of reduce tasks not specified. Estimated from input data size: 1 > Launching Job 2 out of 2 > Number of reduce tasks determined at compile time: 1 > When I have a large number of rows then the last stage of the job takes a > long time. I think we could allow to user choose number of reducers of last > job or refuse extra MR job. > The same behavior I observed with queries: > hive> create table new_test as select age from test1 group by age.age limit > 10; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12963) LIMIT statement with SORT BY creates additional MR job with hardcoded only one reducer
[ https://issues.apache.org/jira/browse/HIVE-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12963: -- Attachment: (was: HIVE-12963.patch.1) > LIMIT statement with SORT BY creates additional MR job with hardcoded only > one reducer > -- > > Key: HIVE-12963 > URL: https://issues.apache.org/jira/browse/HIVE-12963 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.0.0, 1.2.1, 0.13 >Reporter: Alina Abramova >Assignee: Alina Abramova > Attachments: HIVE-12963.1.patch > > > I execute query: > hive> select age from test1 sort by age.age limit 10; > Total jobs = 2 > Launching Job 1 out of 2 > Number of reduce tasks not specified. Estimated from input data size: 1 > Launching Job 2 out of 2 > Number of reduce tasks determined at compile time: 1 > When I have a large number of rows then the last stage of the job takes a > long time. I think we could allow to user choose number of reducers of last > job or refuse extra MR job. > The same behavior I observed with queries: > hive> create table new_test as select age from test1 group by age.age limit > 10; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12963) LIMIT statement with SORT BY creates additional MR job with hardcoded only one reducer
[ https://issues.apache.org/jira/browse/HIVE-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12963: -- Attachment: HIVE-12963.patch.1 > LIMIT statement with SORT BY creates additional MR job with hardcoded only > one reducer > -- > > Key: HIVE-12963 > URL: https://issues.apache.org/jira/browse/HIVE-12963 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.0.0, 1.2.1, 0.13 >Reporter: Alina Abramova >Assignee: Alina Abramova > Attachments: HIVE-12963.patch.1 > > > I execute query: > hive> select age from test1 sort by age.age limit 10; > Total jobs = 2 > Launching Job 1 out of 2 > Number of reduce tasks not specified. Estimated from input data size: 1 > Launching Job 2 out of 2 > Number of reduce tasks determined at compile time: 1 > When I have a large number of rows then the last stage of the job takes a > long time. I think we could allow to user choose number of reducers of last > job or refuse extra MR job. > The same behavior I observed with queries: > hive> create table new_test as select age from test1 group by age.age limit > 10; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12244: -- Attachment: HIVE-12244.2.patch > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Labels: patch > Fix For: 1.2.1 > > Attachments: HIVE-12244.1.patch, HIVE-12244.2.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we use Path from org.apache.hadoop.fs we will avoid new problems > in future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12244: -- Attachment: (was: HIVE-12244.3.patch) > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Labels: patch > Fix For: 1.2.1 > > Attachments: HIVE-12244.1.patch, HIVE-12244.2.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we use Path from org.apache.hadoop.fs we will avoid new problems > in future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12244: -- Attachment: (was: HIVE-12244.2.patch) > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Labels: patch > Fix For: 1.2.1 > > Attachments: HIVE-12244.1.patch, HIVE-12244.2.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we use Path from org.apache.hadoop.fs we will avoid new problems > in future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12244: -- Attachment: HIVE-12244.3.patch Fixed for tests > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Labels: patch > Fix For: 1.2.1 > > Attachments: HIVE-12244.1.patch, HIVE-12244.2.patch, > HIVE-12244.3.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we use Path from org.apache.hadoop.fs we will avoid new problems > in future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12244: -- Attachment: HIVE-12244.2.patch Rebased patch > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Labels: patch > Fix For: 1.2.1 > > Attachments: HIVE-12244.1.patch, HIVE-12244.2.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we use Path from org.apache.hadoop.fs we will avoid new problems > in future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106909#comment-15106909 ] Alina Abramova commented on HIVE-12244: --- Rebased patch was attached to the issue. > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Labels: patch > Fix For: 1.2.1 > > Attachments: HIVE-12244.1.patch, HIVE-12244.2.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we use Path from org.apache.hadoop.fs we will avoid new problems > in future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12244: -- Attachment: HIVE-12244.1.patch > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Priority: Minor > Attachments: HIVE-12244.1.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we used Path from org.apache.hadoop.fs we avoid new problems in > future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova reassigned HIVE-12244: - Assignee: Alina Abramova > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Attachments: HIVE-12244.1.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we used Path from org.apache.hadoop.fs we avoid new problems in > future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12244: -- Labels: patch (was: ) > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Labels: patch > Attachments: HIVE-12244.1.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we used Path from org.apache.hadoop.fs we avoid new problems in > future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971206#comment-14971206 ] Alina Abramova commented on HIVE-12244: --- https://reviews.apache.org/r/39601/ > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Labels: patch > Attachments: HIVE-12244.1.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we used Path from org.apache.hadoop.fs we avoid new problems in > future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12244: -- Component/s: Hive > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Labels: patch > Attachments: HIVE-12244.1.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we used Path from org.apache.hadoop.fs we avoid new problems in > future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12244: -- Description: In Hive often String is used for representation path and it causes new issues. We need to compare it with equals() but comparing Strings often is not right in terms comparing paths . I think if we used Path from org.apache.hadoop.fs we will avoid new problems in future. was: In Hive often String is used for representation path and it causes new issues. We need to compare it with equals() but comparing Strings often is not right in terms comparing paths . I think if we used Path from org.apache.hadoop.fs we avoid new problems in future. > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Labels: patch > Attachments: HIVE-12244.1.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we used Path from org.apache.hadoop.fs we will avoid new problems > in future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12244: -- Description: In Hive often String is used for representation path and it causes new issues. We need to compare it with equals() but comparing Strings often is not right in terms comparing paths . I think if we use Path from org.apache.hadoop.fs we will avoid new problems in future. was: In Hive often String is used for representation path and it causes new issues. We need to compare it with equals() but comparing Strings often is not right in terms comparing paths . I think if we used Path from org.apache.hadoop.fs we will avoid new problems in future. > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Labels: patch > Attachments: HIVE-12244.1.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we use Path from org.apache.hadoop.fs we will avoid new problems > in future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12244: -- Flags: (was: Patch) > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Labels: patch > Attachments: HIVE-12244.1.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we used Path from org.apache.hadoop.fs we avoid new problems in > future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12244: -- Affects Version/s: 0.13.0 0.14.0 1.0.0 1.2.1 > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Priority: Minor > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we used Path from org.apache.hadoop.fs we avoid new problems in > future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
