[jira] [Commented] (HIVE-9503) Update 'hive.auto.convert.join.noconditionaltask.*' descriptions
[ https://issues.apache.org/jira/browse/HIVE-9503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14296187#comment-14296187 ] Xuefu Zhang commented on HIVE-9503: --- I see. I guess the overhead is bearable. It gives much better a user experience than if we auto convert the task and the query fails, leaving the user in the blue. Update 'hive.auto.convert.join.noconditionaltask.*' descriptions Key: HIVE-9503 URL: https://issues.apache.org/jira/browse/HIVE-9503 Project: Hive Issue Type: Bug Components: Configuration Reporter: Szehon Ho Priority: Minor 'hive.auto.convert.join.noconditionaltask' flag does not apply to Spark or Tez, and only to MR (which has the legacy conditional mapjoin) However, 'hive.auto.convert.join.noconditionaltask.size' flag does apply to Spark, Tez, and MR, even though the description indicates it only applies if the above flag is on, which is true only for MR. These configs should be updated to reflect this case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9503) Update 'hive.auto.convert.join.noconditionaltask.*' descriptions
[ https://issues.apache.org/jira/browse/HIVE-9503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14296112#comment-14296112 ] Szehon Ho commented on HIVE-9503: - Well its not overriding the meaning, the value means the same thing (size of small-tables), except for the clause that it depends on the first property. The name also makes sense as we don't use a conditional task in Spark. So I think having a Spark-only property for size of small-tables in mapjoin might be more confusing, as users will need to set both properties to get the same behavior in different execution engines. Update 'hive.auto.convert.join.noconditionaltask.*' descriptions Key: HIVE-9503 URL: https://issues.apache.org/jira/browse/HIVE-9503 Project: Hive Issue Type: Bug Components: Configuration Reporter: Szehon Ho Priority: Minor 'hive.auto.convert.join.noconditionaltask' flag does not apply to Spark or Tez, and only to MR (which has the legacy conditional mapjoin) However, 'hive.auto.convert.join.noconditionaltask.size' flag does apply to Spark, Tez, and MR, even though the description indicates it only applies if the above flag is on, which is true only for MR. These configs should be updated to reflect this case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9503) Update 'hive.auto.convert.join.noconditionaltask.*' descriptions
[ https://issues.apache.org/jira/browse/HIVE-9503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14296154#comment-14296154 ] Xuefu Zhang commented on HIVE-9503: --- I'm not sure of the difference between backup task and the conditional task that these two properties are referring to, but I don't feel we need a property to control whether to have a backup task. As long as we auto converted a join, we should have a backup task. Update 'hive.auto.convert.join.noconditionaltask.*' descriptions Key: HIVE-9503 URL: https://issues.apache.org/jira/browse/HIVE-9503 Project: Hive Issue Type: Bug Components: Configuration Reporter: Szehon Ho Priority: Minor 'hive.auto.convert.join.noconditionaltask' flag does not apply to Spark or Tez, and only to MR (which has the legacy conditional mapjoin) However, 'hive.auto.convert.join.noconditionaltask.size' flag does apply to Spark, Tez, and MR, even though the description indicates it only applies if the above flag is on, which is true only for MR. These configs should be updated to reflect this case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9503) Update 'hive.auto.convert.join.noconditionaltask.*' descriptions
[ https://issues.apache.org/jira/browse/HIVE-9503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14296175#comment-14296175 ] Chao commented on HIVE-9503: In MR, both conditional task AND backup task are used, but for us, only backup task is needed, since no decision needs to be made (only one mapjoin task). If we always use backup task for auto converted join, it will add overhead to plan compilation, because to generate a backup task we need to clone the whole operator tree. Update 'hive.auto.convert.join.noconditionaltask.*' descriptions Key: HIVE-9503 URL: https://issues.apache.org/jira/browse/HIVE-9503 Project: Hive Issue Type: Bug Components: Configuration Reporter: Szehon Ho Priority: Minor 'hive.auto.convert.join.noconditionaltask' flag does not apply to Spark or Tez, and only to MR (which has the legacy conditional mapjoin) However, 'hive.auto.convert.join.noconditionaltask.size' flag does apply to Spark, Tez, and MR, even though the description indicates it only applies if the above flag is on, which is true only for MR. These configs should be updated to reflect this case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9503) Update 'hive.auto.convert.join.noconditionaltask.*' descriptions
[ https://issues.apache.org/jira/browse/HIVE-9503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14296098#comment-14296098 ] Xuefu Zhang commented on HIVE-9503: --- If hive.auto.convert.join.noconditionaltask.size is used by spark regardless of hive.auto.convert.join.noconditionaltask, we should probably have a different property. Reusing the same property while overwriting its meaning could cause confusion for either existing users or new users. Update 'hive.auto.convert.join.noconditionaltask.*' descriptions Key: HIVE-9503 URL: https://issues.apache.org/jira/browse/HIVE-9503 Project: Hive Issue Type: Bug Components: Configuration Reporter: Szehon Ho Priority: Minor 'hive.auto.convert.join.noconditionaltask' flag does not apply to Spark or Tez, and only to MR (which has the legacy conditional mapjoin) However, 'hive.auto.convert.join.noconditionaltask.size' flag does apply to Spark, Tez, and MR, even though the description indicates it only applies if the above flag is on, which is true only for MR. These configs should be updated to reflect this case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9503) Update 'hive.auto.convert.join.noconditionaltask.*' descriptions
[ https://issues.apache.org/jira/browse/HIVE-9503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14296141#comment-14296141 ] Chao commented on HIVE-9503: For backup task (HIVE-9103), I'm thinking about reusing hive.auto.convert.join.noconditionaltask to specify whether backup task is needed. This is slightly misleading, but we can add some description to the property. Thoughts? Update 'hive.auto.convert.join.noconditionaltask.*' descriptions Key: HIVE-9503 URL: https://issues.apache.org/jira/browse/HIVE-9503 Project: Hive Issue Type: Bug Components: Configuration Reporter: Szehon Ho Priority: Minor 'hive.auto.convert.join.noconditionaltask' flag does not apply to Spark or Tez, and only to MR (which has the legacy conditional mapjoin) However, 'hive.auto.convert.join.noconditionaltask.size' flag does apply to Spark, Tez, and MR, even though the description indicates it only applies if the above flag is on, which is true only for MR. These configs should be updated to reflect this case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9503) Update 'hive.auto.convert.join.noconditionaltask.*' descriptions
[ https://issues.apache.org/jira/browse/HIVE-9503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14296178#comment-14296178 ] Xuefu Zhang commented on HIVE-9503: --- Yeah, Hive has long reached to a point where the properties are confusing and sometimes contradicting and duplicating. These two properties plus hive.auto.convert.join are an example. The two properties are meant to be used together. Ignoring one while taking the other doesn't seem to be a clean solution. While it's already a legacy for MR and Tez, I'd like to have a cleaner solution for Spark since we still have the chance. Update 'hive.auto.convert.join.noconditionaltask.*' descriptions Key: HIVE-9503 URL: https://issues.apache.org/jira/browse/HIVE-9503 Project: Hive Issue Type: Bug Components: Configuration Reporter: Szehon Ho Priority: Minor 'hive.auto.convert.join.noconditionaltask' flag does not apply to Spark or Tez, and only to MR (which has the legacy conditional mapjoin) However, 'hive.auto.convert.join.noconditionaltask.size' flag does apply to Spark, Tez, and MR, even though the description indicates it only applies if the above flag is on, which is true only for MR. These configs should be updated to reflect this case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
