[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
[
https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201523#comment-16201523
]
Wang Haihua commented on HIVE-12727:
thanks for your reply [~sershe].
Our version is 2.1.1 (i suppose what happened is same with 2.0.0)
{{hive.mapred.mode}}(A) default value is "nonstrict" (i just noticed default
value become "strict" and return to "nonstrict" again in 2.0.0), so means
cartesian product is {{enabled}}.
This week i want to use three new configuration to control more accurately (In
my case, i just wanna forbidden query without specifying partitions. Query with
cartesian product should perform as usual), so just apply this path HIVE-14346
which set default value of {{hive.mapred.mode}} to "null", and set
{{hive.strict.checks.large.query}} to true in hive-site.xml.
But Incident occurred, because many queries with cartesian product failed
also. The reason is i had not noticed default value of
{{hive.strict.checks.cartesian.product}} is true...
So i suppose for smooth upgrade, should we consider default value of
{{hive.strict.checks.cartesian.product}}, {{hive.strict.checks.large.query}}
and other related configuration item have same semantic with legacy item
{{hive.mapred.mode}}
> refactor Hive strict checks to be more granular, allow order by no limit and
> no partition filter by default for now
> ---
>
> Key: HIVE-12727
> URL: https://issues.apache.org/jira/browse/HIVE-12727
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Blocker
> Labels: TODOC2.0
> Fix For: 2.0.0
>
> Attachments: HIVE-12727.01.patch, HIVE-12727.02.patch,
> HIVE-12727.03.patch, HIVE-12727.04.patch, HIVE-12727.05.patch,
> HIVE-12727.06.patch, HIVE-12727.07.patch, HIVE-12727.patch
>
>
> Making strict mode the default recently appears to have broken many normal
> queries, such as some TPCDS benchmark queries, e.g. Q85:
> Response message: org.apache.hive.service.cli.HiveSQLException: Error while
> compiling statement: FAILED: SemanticException [Error 10041]: No partition
> predicate found for Alias "web_sales" Table "web_returns"
> We should remove this restriction from strict mode, or change the default
> back to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict,
> and strict, for backward compat for people who are relying on strict already.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
[ https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201086#comment-16201086 ] Sergey Shelukhin commented on HIVE-12727: - The legacy mode is actually more restrictive than the new mode. The cartesian product check used to be true in the strict mode (the default), and false, with all the other checks, in nonstrict. The new configs only apply in new versions; if you set hive.mapred.mode to null on older versions, it assumes the default value of "strict", applying all the checks. Otherwise please describe the scenario (the old value, old version of Hive, new version of Hive, and what changes were made to this setting) that you are encountering, afaik it should have no upgrade ases where it changes the behavior in a manner that breaks the previous user-specific setting, or makes the checks more restrictive if the setting was not set. > refactor Hive strict checks to be more granular, allow order by no limit and > no partition filter by default for now > --- > > Key: HIVE-12727 > URL: https://issues.apache.org/jira/browse/HIVE-12727 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Blocker > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-12727.01.patch, HIVE-12727.02.patch, > HIVE-12727.03.patch, HIVE-12727.04.patch, HIVE-12727.05.patch, > HIVE-12727.06.patch, HIVE-12727.07.patch, HIVE-12727.patch > > > Making strict mode the default recently appears to have broken many normal > queries, such as some TPCDS benchmark queries, e.g. Q85: > Response message: org.apache.hive.service.cli.HiveSQLException: Error while > compiling statement: FAILED: SemanticException [Error 10041]: No partition > predicate found for Alias "web_sales" Table "web_returns" > We should remove this restriction from strict mode, or change the default > back to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict, > and strict, for backward compat for people who are relying on strict already. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
[
https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200401#comment-16200401
]
Wang Haihua commented on HIVE-12727:
Hi, should we follow the meaning of legacy default nonstrict value of
{{hive.mapred.mode}}, which means that {{hive.strict.checks.cartesian.product
}} should has default value of false?
I suppose this will be welcome to the legacy user for compatibility.
In my case, change {{hive.mapred.mode}} to null, will result in many SQL
including cartesian product executed error.
cc [~xuefuz] [~sershe]]
> refactor Hive strict checks to be more granular, allow order by no limit and
> no partition filter by default for now
> ---
>
> Key: HIVE-12727
> URL: https://issues.apache.org/jira/browse/HIVE-12727
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Blocker
> Labels: TODOC2.0
> Fix For: 2.0.0
>
> Attachments: HIVE-12727.01.patch, HIVE-12727.02.patch,
> HIVE-12727.03.patch, HIVE-12727.04.patch, HIVE-12727.05.patch,
> HIVE-12727.06.patch, HIVE-12727.07.patch, HIVE-12727.patch
>
>
> Making strict mode the default recently appears to have broken many normal
> queries, such as some TPCDS benchmark queries, e.g. Q85:
> Response message: org.apache.hive.service.cli.HiveSQLException: Error while
> compiling statement: FAILED: SemanticException [Error 10041]: No partition
> predicate found for Alias "web_sales" Table "web_returns"
> We should remove this restriction from strict mode, or change the default
> back to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict,
> and strict, for backward compat for people who are relying on strict already.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
[ https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15686092#comment-15686092 ] Lefty Leverenz commented on HIVE-12727: --- HIVE-15148 changes the description of *hive.strict.checks.cartesian.product* in release 2.2.0. > refactor Hive strict checks to be more granular, allow order by no limit and > no partition filter by default for now > --- > > Key: HIVE-12727 > URL: https://issues.apache.org/jira/browse/HIVE-12727 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Blocker > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-12727.01.patch, HIVE-12727.02.patch, > HIVE-12727.03.patch, HIVE-12727.04.patch, HIVE-12727.05.patch, > HIVE-12727.06.patch, HIVE-12727.07.patch, HIVE-12727.patch > > > Making strict mode the default recently appears to have broken many normal > queries, such as some TPCDS benchmark queries, e.g. Q85: > Response message: org.apache.hive.service.cli.HiveSQLException: Error while > compiling statement: FAILED: SemanticException [Error 10041]: No partition > predicate found for Alias "web_sales" Table "web_returns" > We should remove this restriction from strict mode, or change the default > back to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict, > and strict, for backward compat for people who are relying on strict already. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
[ https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394548#comment-15394548 ] Chao Sun commented on HIVE-12727: - Cool. Do you know the JIRA number for that? I'll create a follow up JIRA for this to change it to null. Thanks. > refactor Hive strict checks to be more granular, allow order by no limit and > no partition filter by default for now > --- > > Key: HIVE-12727 > URL: https://issues.apache.org/jira/browse/HIVE-12727 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Blocker > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-12727.01.patch, HIVE-12727.02.patch, > HIVE-12727.03.patch, HIVE-12727.04.patch, HIVE-12727.05.patch, > HIVE-12727.06.patch, HIVE-12727.07.patch, HIVE-12727.patch > > > Making strict mode the default recently appears to have broken many normal > queries, such as some TPCDS benchmark queries, e.g. Q85: > Response message: org.apache.hive.service.cli.HiveSQLException: Error while > compiling statement: FAILED: SemanticException [Error 10041]: No partition > predicate found for Alias "web_sales" Table "web_returns" > We should remove this restriction from strict mode, or change the default > back to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict, > and strict, for backward compat for people who are relying on strict already. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
[ https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394544#comment-15394544 ] Sergey Shelukhin commented on HIVE-12727: - Yeah, we found it recently, I think there's a JIRA to remove that. Meanwhile it makes sense to change to null > refactor Hive strict checks to be more granular, allow order by no limit and > no partition filter by default for now > --- > > Key: HIVE-12727 > URL: https://issues.apache.org/jira/browse/HIVE-12727 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Blocker > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-12727.01.patch, HIVE-12727.02.patch, > HIVE-12727.03.patch, HIVE-12727.04.patch, HIVE-12727.05.patch, > HIVE-12727.06.patch, HIVE-12727.07.patch, HIVE-12727.patch > > > Making strict mode the default recently appears to have broken many normal > queries, such as some TPCDS benchmark queries, e.g. Q85: > Response message: org.apache.hive.service.cli.HiveSQLException: Error while > compiling statement: FAILED: SemanticException [Error 10041]: No partition > predicate found for Alias "web_sales" Table "web_returns" > We should remove this restriction from strict mode, or change the default > back to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict, > and strict, for backward compat for people who are relying on strict already. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
[
https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394533#comment-15394533
]
Chao Sun commented on HIVE-12727:
-
Do you mean:
{code}
String mode = HiveConf.getVar(conf, ConfVars.HIVEMAPREDMODE, null);
{code}?
Yeah, I think that's a little strange. Inside {{HiveConf#initialize()}}, it
calls this method:
{code}
private static void applyDefaultNonNullConfVars(Configuration conf) {
for (ConfVars var : ConfVars.values()) {
String defaultValue = var.getDefaultValue();
if (defaultValue == null) {
// Don't override ConfVars with null values
continue;
}
conf.set(var.varname, defaultValue);
}
}
{code}
which initialize all conf vars whose default value are not null. So that code
doesn't work. I think we may have to change the default value to null.
> refactor Hive strict checks to be more granular, allow order by no limit and
> no partition filter by default for now
> ---
>
> Key: HIVE-12727
> URL: https://issues.apache.org/jira/browse/HIVE-12727
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Blocker
> Labels: TODOC2.0
> Fix For: 2.0.0
>
> Attachments: HIVE-12727.01.patch, HIVE-12727.02.patch,
> HIVE-12727.03.patch, HIVE-12727.04.patch, HIVE-12727.05.patch,
> HIVE-12727.06.patch, HIVE-12727.07.patch, HIVE-12727.patch
>
>
> Making strict mode the default recently appears to have broken many normal
> queries, such as some TPCDS benchmark queries, e.g. Q85:
> Response message: org.apache.hive.service.cli.HiveSQLException: Error while
> compiling statement: FAILED: SemanticException [Error 10041]: No partition
> predicate found for Alias "web_sales" Table "web_returns"
> We should remove this restriction from strict mode, or change the default
> back to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict,
> and strict, for backward compat for people who are relying on strict already.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
[ https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394322#comment-15394322 ] Sergey Shelukhin commented on HIVE-12727: - That makes sense. Although doesn't the above code explicitly use default == null as an argument? > refactor Hive strict checks to be more granular, allow order by no limit and > no partition filter by default for now > --- > > Key: HIVE-12727 > URL: https://issues.apache.org/jira/browse/HIVE-12727 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Blocker > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-12727.01.patch, HIVE-12727.02.patch, > HIVE-12727.03.patch, HIVE-12727.04.patch, HIVE-12727.05.patch, > HIVE-12727.06.patch, HIVE-12727.07.patch, HIVE-12727.patch > > > Making strict mode the default recently appears to have broken many normal > queries, such as some TPCDS benchmark queries, e.g. Q85: > Response message: org.apache.hive.service.cli.HiveSQLException: Error while > compiling statement: FAILED: SemanticException [Error 10041]: No partition > predicate found for Alias "web_sales" Table "web_returns" > We should remove this restriction from strict mode, or change the default > back to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict, > and strict, for backward compat for people who are relying on strict already. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
[ https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394138#comment-15394138 ] Xuefu Zhang commented on HIVE-12727: It seems to be that the default value for hive.mapred.mode should be null then. Right now the default value is "nonstrict", which makes the three new configs useless. Thoughts, [~sershe]? > refactor Hive strict checks to be more granular, allow order by no limit and > no partition filter by default for now > --- > > Key: HIVE-12727 > URL: https://issues.apache.org/jira/browse/HIVE-12727 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Blocker > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-12727.01.patch, HIVE-12727.02.patch, > HIVE-12727.03.patch, HIVE-12727.04.patch, HIVE-12727.05.patch, > HIVE-12727.06.patch, HIVE-12727.07.patch, HIVE-12727.patch > > > Making strict mode the default recently appears to have broken many normal > queries, such as some TPCDS benchmark queries, e.g. Q85: > Response message: org.apache.hive.service.cli.HiveSQLException: Error while > compiling statement: FAILED: SemanticException [Error 10041]: No partition > predicate found for Alias "web_sales" Table "web_returns" > We should remove this restriction from strict mode, or change the default > back to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict, > and strict, for backward compat for people who are relying on strict already. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
[
https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393004#comment-15393004
]
Chao Sun commented on HIVE-12727:
-
OK, got it. So there are 3 cases:
# hive.mapred.mode is not set: use the values set for the 3 configurations
# hive.mapred.mode is set to 'strict': disable checks from the 3
configurations, ignoring their values
# hive.mapred.mode is set to something other than 'strict': enable checks from
the 3 configurations, ignoring their values.
Perhaps the documentation could be more explicit about this. Also, in
[here|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.mapred.mode]
the default value is still strict - should be changed to nonstrict.
[~sershe] do you know how can we unset the {{hive.mapred.mode}} so to use this
feature?
> refactor Hive strict checks to be more granular, allow order by no limit and
> no partition filter by default for now
> ---
>
> Key: HIVE-12727
> URL: https://issues.apache.org/jira/browse/HIVE-12727
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Blocker
> Labels: TODOC2.0
> Fix For: 2.0.0
>
> Attachments: HIVE-12727.01.patch, HIVE-12727.02.patch,
> HIVE-12727.03.patch, HIVE-12727.04.patch, HIVE-12727.05.patch,
> HIVE-12727.06.patch, HIVE-12727.07.patch, HIVE-12727.patch
>
>
> Making strict mode the default recently appears to have broken many normal
> queries, such as some TPCDS benchmark queries, e.g. Q85:
> Response message: org.apache.hive.service.cli.HiveSQLException: Error while
> compiling statement: FAILED: SemanticException [Error 10041]: No partition
> predicate found for Alias "web_sales" Table "web_returns"
> We should remove this restriction from strict mode, or change the default
> back to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict,
> and strict, for backward compat for people who are relying on strict already.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
[ https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392966#comment-15392966 ] Sergey Shelukhin commented on HIVE-12727: - Yeah, that was the intent... maintaining backward compat for strict mode. > refactor Hive strict checks to be more granular, allow order by no limit and > no partition filter by default for now > --- > > Key: HIVE-12727 > URL: https://issues.apache.org/jira/browse/HIVE-12727 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Blocker > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-12727.01.patch, HIVE-12727.02.patch, > HIVE-12727.03.patch, HIVE-12727.04.patch, HIVE-12727.05.patch, > HIVE-12727.06.patch, HIVE-12727.07.patch, HIVE-12727.patch > > > Making strict mode the default recently appears to have broken many normal > queries, such as some TPCDS benchmark queries, e.g. Q85: > Response message: org.apache.hive.service.cli.HiveSQLException: Error while > compiling statement: FAILED: SemanticException [Error 10041]: No partition > predicate found for Alias "web_sales" Table "web_returns" > We should remove this restriction from strict mode, or change the default > back to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict, > and strict, for backward compat for people who are relying on strict already. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
[ https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392935#comment-15392935 ] Xuefu Zhang commented on HIVE-12727: My understanding is that hive.mapred.mode=strict is deprecated, being replaced with the three new configurations. To use the new configurations, hive.mapred.mode cannot be strict. If it's nonstrict, then the three new configuration are checked. As to null value, I'm not sure if it can be ever null. Nevertheless, the above logic should stand. > refactor Hive strict checks to be more granular, allow order by no limit and > no partition filter by default for now > --- > > Key: HIVE-12727 > URL: https://issues.apache.org/jira/browse/HIVE-12727 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Blocker > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-12727.01.patch, HIVE-12727.02.patch, > HIVE-12727.03.patch, HIVE-12727.04.patch, HIVE-12727.05.patch, > HIVE-12727.06.patch, HIVE-12727.07.patch, HIVE-12727.patch > > > Making strict mode the default recently appears to have broken many normal > queries, such as some TPCDS benchmark queries, e.g. Q85: > Response message: org.apache.hive.service.cli.HiveSQLException: Error while > compiling statement: FAILED: SemanticException [Error 10041]: No partition > predicate found for Alias "web_sales" Table "web_returns" > We should remove this restriction from strict mode, or change the default > back to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict, > and strict, for backward compat for people who are relying on strict already. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
[
https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392921#comment-15392921
]
Chao Sun commented on HIVE-12727:
-
I'm actually a little confused about this. From the code:
{code}
private static String makeMessage(String what, ConfVars setting) {
return what + " are disabled for safety reasons. If you know what you are
doing, please make"
+ " sure that " + setting.varname + " is set to false and that "
+ ConfVars.HIVEMAPREDMODE.varname + " is not set to 'strict' to
enable them.";
}
{code}
it seems like if {{hive.mapred.mode}} is NOT set to 'strict', then these 3 new
configurations are activated.
However, in another piece of code:
{code}
private static boolean isAllowed(Configuration conf, ConfVars setting) {
String mode = HiveConf.getVar(conf, ConfVars.HIVEMAPREDMODE, null);
return (mode != null) ? !"strict".equals(mode) :
!HiveConf.getBoolVar(conf, setting);
}
{code}
it seems like as long as {{hive.mapred.mode}} is not null AND not 'strict',
then the above 3 configurations are disabled, i.e., their values are ignored.
Is this intentional?
(BTW, how can I set the {{hive.mapred.mode}} to null? it seems Hive just load
the default value which is 'nonstrict')
> refactor Hive strict checks to be more granular, allow order by no limit and
> no partition filter by default for now
> ---
>
> Key: HIVE-12727
> URL: https://issues.apache.org/jira/browse/HIVE-12727
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Blocker
> Labels: TODOC2.0
> Fix For: 2.0.0
>
> Attachments: HIVE-12727.01.patch, HIVE-12727.02.patch,
> HIVE-12727.03.patch, HIVE-12727.04.patch, HIVE-12727.05.patch,
> HIVE-12727.06.patch, HIVE-12727.07.patch, HIVE-12727.patch
>
>
> Making strict mode the default recently appears to have broken many normal
> queries, such as some TPCDS benchmark queries, e.g. Q85:
> Response message: org.apache.hive.service.cli.HiveSQLException: Error while
> compiling statement: FAILED: SemanticException [Error 10041]: No partition
> predicate found for Alias "web_sales" Table "web_returns"
> We should remove this restriction from strict mode, or change the default
> back to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict,
> and strict, for backward compat for people who are relying on strict already.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
[ https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15124535#comment-15124535 ] Lefty Leverenz commented on HIVE-12727: --- Doc note: This deprecates *hive.mapred.mode* in 2.0.0, changing its default value back to nonstrict after HIVE-12413 changed it to strict in the same release, and adds three new configuration parameters to replace *hive.mapred.mode* (*hive.strict.checks.large.query* with default false, *hive.strict.checks.type.safety* with default true, and *hive.strict.checks.cartesian.product* with default true), so I added a TODOC2.0 label. The parameter changes should be documented in the wiki here: * [Configuration Properties -- hive.mapred.mode | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.mapred.mode] > refactor Hive strict checks to be more granular, allow order by no limit and > no partition filter by default for now > --- > > Key: HIVE-12727 > URL: https://issues.apache.org/jira/browse/HIVE-12727 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Blocker > Labels: TODOC2.0 > Fix For: 2.0.0, 2.1.0 > > Attachments: HIVE-12727.01.patch, HIVE-12727.02.patch, > HIVE-12727.03.patch, HIVE-12727.04.patch, HIVE-12727.05.patch, > HIVE-12727.06.patch, HIVE-12727.07.patch, HIVE-12727.patch > > > Making strict mode the default recently appears to have broken many normal > queries, such as some TPCDS benchmark queries, e.g. Q85: > Response message: org.apache.hive.service.cli.HiveSQLException: Error while > compiling statement: FAILED: SemanticException [Error 10041]: No partition > predicate found for Alias "web_sales" Table "web_returns" > We should remove this restriction from strict mode, or change the default > back to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict, > and strict, for backward compat for people who are relying on strict already. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
[ https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122339#comment-15122339 ] Sergey Shelukhin commented on HIVE-12727: - Given that only 3 tests failed due to test-specific issues, and HiveQA queue is very long, I am going to run tests here and commit if that comes back earlier than ptest. > refactor Hive strict checks to be more granular, allow order by no limit and > no partition filter by default for now > --- > > Key: HIVE-12727 > URL: https://issues.apache.org/jira/browse/HIVE-12727 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Blocker > Attachments: HIVE-12727.01.patch, HIVE-12727.02.patch, > HIVE-12727.03.patch, HIVE-12727.04.patch, HIVE-12727.05.patch, > HIVE-12727.06.patch, HIVE-12727.07.patch, HIVE-12727.patch > > > Making strict mode the default recently appears to have broken many normal > queries, such as some TPCDS benchmark queries, e.g. Q85: > Response message: org.apache.hive.service.cli.HiveSQLException: Error while > compiling statement: FAILED: SemanticException [Error 10041]: No partition > predicate found for Alias "web_sales" Table "web_returns" > We should remove this restriction from strict mode, or change the default > back to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict, > and strict, for backward compat for people who are relying on strict already. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
[
https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121294#comment-15121294
]
Hive QA commented on HIVE-12727:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12784723/HIVE-12727.06.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10036 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_invalid_distinct1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_invalid_distinct3
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query12
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarDataNucleusUnCaching
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6777/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6777/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6777/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12784723 - PreCommit-HIVE-TRUNK-Build
> refactor Hive strict checks to be more granular, allow order by no limit and
> no partition filter by default for now
> ---
>
> Key: HIVE-12727
> URL: https://issues.apache.org/jira/browse/HIVE-12727
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Blocker
> Attachments: HIVE-12727.01.patch, HIVE-12727.02.patch,
> HIVE-12727.03.patch, HIVE-12727.04.patch, HIVE-12727.05.patch,
> HIVE-12727.06.patch, HIVE-12727.patch
>
>
> Making strict mode the default recently appears to have broken many normal
> queries, such as some TPCDS benchmark queries, e.g. Q85:
> Response message: org.apache.hive.service.cli.HiveSQLException: Error while
> compiling statement: FAILED: SemanticException [Error 10041]: No partition
> predicate found for Alias "web_sales" Table "web_returns"
> We should remove this restriction from strict mode, or change the default
> back to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict,
> and strict, for backward compat for people who are relying on strict already.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
[ https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120200#comment-15120200 ] Sergey Shelukhin commented on HIVE-12727: - Also attached the patch to HIVE-12855 to get the queue spot sooner for this patch rather than for other, less important patch of mine... > refactor Hive strict checks to be more granular, allow order by no limit and > no partition filter by default for now > --- > > Key: HIVE-12727 > URL: https://issues.apache.org/jira/browse/HIVE-12727 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Blocker > Attachments: HIVE-12727.01.patch, HIVE-12727.02.patch, > HIVE-12727.03.patch, HIVE-12727.04.patch, HIVE-12727.05.patch, > HIVE-12727.06.patch, HIVE-12727.patch > > > Making strict mode the default recently appears to have broken many normal > queries, such as some TPCDS benchmark queries, e.g. Q85: > Response message: org.apache.hive.service.cli.HiveSQLException: Error while > compiling statement: FAILED: SemanticException [Error 10041]: No partition > predicate found for Alias "web_sales" Table "web_returns" > We should remove this restriction from strict mode, or change the default > back to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict, > and strict, for backward compat for people who are relying on strict already. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
[ https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120207#comment-15120207 ] Sergey Shelukhin commented on HIVE-12727: - [~xuefuz] does the change make sense, other than the test mismatches? > refactor Hive strict checks to be more granular, allow order by no limit and > no partition filter by default for now > --- > > Key: HIVE-12727 > URL: https://issues.apache.org/jira/browse/HIVE-12727 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Blocker > Attachments: HIVE-12727.01.patch, HIVE-12727.02.patch, > HIVE-12727.03.patch, HIVE-12727.04.patch, HIVE-12727.05.patch, > HIVE-12727.06.patch, HIVE-12727.patch > > > Making strict mode the default recently appears to have broken many normal > queries, such as some TPCDS benchmark queries, e.g. Q85: > Response message: org.apache.hive.service.cli.HiveSQLException: Error while > compiling statement: FAILED: SemanticException [Error 10041]: No partition > predicate found for Alias "web_sales" Table "web_returns" > We should remove this restriction from strict mode, or change the default > back to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict, > and strict, for backward compat for people who are relying on strict already. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
[ https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120252#comment-15120252 ] Xuefu Zhang commented on HIVE-12727: Yeah. looks good to me. > refactor Hive strict checks to be more granular, allow order by no limit and > no partition filter by default for now > --- > > Key: HIVE-12727 > URL: https://issues.apache.org/jira/browse/HIVE-12727 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Blocker > Attachments: HIVE-12727.01.patch, HIVE-12727.02.patch, > HIVE-12727.03.patch, HIVE-12727.04.patch, HIVE-12727.05.patch, > HIVE-12727.06.patch, HIVE-12727.patch > > > Making strict mode the default recently appears to have broken many normal > queries, such as some TPCDS benchmark queries, e.g. Q85: > Response message: org.apache.hive.service.cli.HiveSQLException: Error while > compiling statement: FAILED: SemanticException [Error 10041]: No partition > predicate found for Alias "web_sales" Table "web_returns" > We should remove this restriction from strict mode, or change the default > back to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict, > and strict, for backward compat for people who are relying on strict already. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
[
https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119124#comment-15119124
]
Hive QA commented on HIVE-12727:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12784492/HIVE-12727.05.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 30 failed/errored test(s), 10018 tests
executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-vector_left_outer_join2.q-vector_outer_join5.q-custom_input_output_format.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_genericudf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cte_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_empty_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_duplicate_key
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_unqual5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_literal_decimal
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_offset_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_offset_limit_ppd_optimizer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rand_partitionpruner2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_18
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_empty_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_concat_insert2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_date_trim
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_26
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_when_case_null
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_empty_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_when_case_null
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_invalid_distinct1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_invalid_distinct3
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query12
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_insert1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part14
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_18
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_8
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_date_trim
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6757/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6757/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6757/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 30 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12784492 - PreCommit-HIVE-TRUNK-Build
> refactor Hive strict checks to be more granular, allow order by no limit and
> no partition filter by default for now
> ---
>
> Key: HIVE-12727
> URL: https://issues.apache.org/jira/browse/HIVE-12727
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Blocker
> Attachments: HIVE-12727.01.patch, HIVE-12727.02.patch,
> HIVE-12727.03.patch, HIVE-12727.04.patch, HIVE-12727.05.patch,
> HIVE-12727.patch
>
>
> Making strict mode the default recently appears to have broken many normal
> queries, such as some TPCDS benchmark queries, e.g. Q85:
> Response message: org.apache.hive.service.cli.HiveSQLException: Error while
> compiling statement: FAILED: SemanticException [Error 10041]: No partition
> predicate found for Alias "web_sales" Table "web_returns"
> We should remove this restriction from strict mode, or change the default
> back to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict,
> and strict, for backward compat for people who are relying on strict already.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
[ https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15117755#comment-15117755 ] Xuefu Zhang commented on HIVE-12727: That sounds more reasonable. Quick observation: though private, the method, isOk() seems too generic to be meaningful. > refactor Hive strict checks to be more granular, allow order by no limit and > no partition filter by default for now > --- > > Key: HIVE-12727 > URL: https://issues.apache.org/jira/browse/HIVE-12727 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Blocker > Attachments: HIVE-12727.01.patch, HIVE-12727.02.patch, > HIVE-12727.03.patch, HIVE-12727.04.patch, HIVE-12727.patch > > > Making strict mode the default recently appears to have broken many normal > queries, such as some TPCDS benchmark queries, e.g. Q85: > Response message: org.apache.hive.service.cli.HiveSQLException: Error while > compiling statement: FAILED: SemanticException [Error 10041]: No partition > predicate found for Alias "web_sales" Table "web_returns" > We should remove this restriction from strict mode, or change the default > back to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict, > and strict, for backward compat for people who are relying on strict already. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
[
https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15116714#comment-15116714
]
Hive QA commented on HIVE-12727:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12784228/HIVE-12727.03.patch
{color:green}SUCCESS:{color} +1 due to 996 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 67 failed/errored test(s), 10029 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_genericudf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cte_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_duplicate_key
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_const_type
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_unqual5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_literals
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_literal_decimal
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge_join_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_offset_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_offset_limit_ppd_optimizer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rand_partitionpruner2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_18
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_special_character_in_tabnames_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_empty_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_concat_insert2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_date_trim
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_26
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_when_case_null
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_0
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_not
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_offset_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_pushdown
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_short_regress
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_filter_join_breaktask2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_when_case_null
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_0
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_limit
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_not
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_pushdown
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_short_regress
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_alter_view_failure6
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_compare_double_bigint
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_compare_string_bigint
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_input4
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_input_part0_neg
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_invalid_distinct1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_invalid_distinct3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_strict_join
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_strict_orderby
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_strict_pruning
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_subquery_notexists_implicit_gby
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query12
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_filter_join_breaktask2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_literals
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part14
[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
[ https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15114354#comment-15114354 ] Xuefu Zhang commented on HIVE-12727: Other than granular control over checks, is "hive.mapred.mode" still effective and does it mean that all those strict checks are automatically enforced? I'm not clear from the patch/proposal. Image an admin who has enabled strict mode and now upgrades to 2.0. > refactor Hive strict checks to be more granular, allow order by no limit and > no partition filter by default for now > --- > > Key: HIVE-12727 > URL: https://issues.apache.org/jira/browse/HIVE-12727 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Blocker > Attachments: HIVE-12727.01.patch, HIVE-12727.02.patch, > HIVE-12727.patch > > > Making strict mode the default recently appears to have broken many normal > queries, such as some TPCDS benchmark queries, e.g. Q85: > Response message: org.apache.hive.service.cli.HiveSQLException: Error while > compiling statement: FAILED: SemanticException [Error 10041]: No partition > predicate found for Alias "web_sales" Table "web_returns" > We should remove this restriction from strict mode, or change the default > back to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict, > and strict, for backward compat for people who are relying on strict already. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
[
https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15114369#comment-15114369
]
Hive QA commented on HIVE-12727:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12783926/HIVE-12727.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 170 failed/errored test(s), 10026 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_explain
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join0
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_filters
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_nulls
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_auto_join0
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_cross_product_check_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_join1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_subq_not_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_subq_not_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_complex_alias
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_genericudf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_product_check_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_product_check_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cte_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_duplicate_key
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_const_type
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join0
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_alt_syntax
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_unqual1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_unqual3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_unqual5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_literals
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_nulls
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lineage3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_literal_decimal
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge_join_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mergejoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_no_hooks
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonblock_op_deduplicate
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_offset_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_offset_limit_ppd_optimizer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_llap
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parallel_join0
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pointlookup2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pointlookup3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_repeated_alias
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_print_header
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rand_partitionpruner2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_18
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_special_character_in_tabnames_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_empty_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin_having
[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
[
https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15114482#comment-15114482
]
Sergey Shelukhin commented on HIVE-12727:
-
Yes, strict mode is enforced.
{noformat}
+private static String check(Configuration conf, ConfVars setting, String
message) {
+ if (!HiveConf.getBoolVar(conf, setting) &&
!"strict".equals(HiveConf.getVar(
+ conf, ConfVars.HIVEMAPREDMODE, null))) return null;
+ return message;
+}
{noformat}
I should actually probably change it to return boolean, the message is just a
pass-thru object. null means the operation is allowed; it is allowed iff the
granular setting is false AND if mapred mode is not strict.
> refactor Hive strict checks to be more granular, allow order by no limit and
> no partition filter by default for now
> ---
>
> Key: HIVE-12727
> URL: https://issues.apache.org/jira/browse/HIVE-12727
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Blocker
> Attachments: HIVE-12727.01.patch, HIVE-12727.02.patch,
> HIVE-12727.patch
>
>
> Making strict mode the default recently appears to have broken many normal
> queries, such as some TPCDS benchmark queries, e.g. Q85:
> Response message: org.apache.hive.service.cli.HiveSQLException: Error while
> compiling statement: FAILED: SemanticException [Error 10041]: No partition
> predicate found for Alias "web_sales" Table "web_returns"
> We should remove this restriction from strict mode, or change the default
> back to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict,
> and strict, for backward compat for people who are relying on strict already.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
[ https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113107#comment-15113107 ] Sergey Shelukhin commented on HIVE-12727: - [~xuefuz] are you ok with this approach? > refactor Hive strict checks to be more granular, allow order by no limit and > no partition filter by default for now > --- > > Key: HIVE-12727 > URL: https://issues.apache.org/jira/browse/HIVE-12727 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Blocker > Attachments: HIVE-12727.01.patch, HIVE-12727.02.patch, > HIVE-12727.patch > > > Making strict mode the default recently appears to have broken many normal > queries, such as some TPCDS benchmark queries, e.g. Q85: > Response message: org.apache.hive.service.cli.HiveSQLException: Error while > compiling statement: FAILED: SemanticException [Error 10041]: No partition > predicate found for Alias "web_sales" Table "web_returns" > We should remove this restriction from strict mode, or change the default > back to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict, > and strict, for backward compat for people who are relying on strict already. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
