[jira] [Updated] (HIVE-8111) CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO
[ https://issues.apache.org/jira/browse/HIVE-8111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-8111: --- Fix Version/s: (was: 0.15.0) 0.14.0 CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO Key: HIVE-8111 URL: https://issues.apache.org/jira/browse/HIVE-8111 Project: Hive Issue Type: Sub-task Components: CBO Affects Versions: 0.14.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.14.0 Attachments: HIVE-8111.01.patch, HIVE-8111.02.patch, HIVE-8111.03.patch, HIVE-8111.patch Original test failure: looks like column type changes to different decimals in most cases. In one case it causes the integer part to be too big to fit, so the result becomes null it seems. What happens is that CBO adds casts to arithmetic expressions to make them type compatible; these casts become part of new AST, and then Hive adds casts on top of these casts. This (the first part) also causes lots of out file changes. It's not clear how to best fix it so far, in addition to incorrect decimal width and sometimes nulls when width is larger than allowed in Hive. Option one - don't add those for numeric ops - cannot be done if numeric op is a part of compare, for which CBO needs correct types. Option two - unwrap casts when determining type in Hive - hard or impossible to tell apart CBO-added casts and user casts. Option three - don't change types in Hive if CBO has run - seems hacky and hard to ensure it's applied everywhere. Option four - map all expressions precisely between two trees and remove casts again after optimization, will be pretty difficult. Option five - somehow mark those casts. Not sure about how yet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8111) CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO
[ https://issues.apache.org/jira/browse/HIVE-8111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-8111: --- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk. Thanks, Sergey. [~vikram.dixit] It will be good to have this bug fix in 0.14 branch as well. CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO Key: HIVE-8111 URL: https://issues.apache.org/jira/browse/HIVE-8111 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-8111.01.patch, HIVE-8111.02.patch, HIVE-8111.03.patch, HIVE-8111.patch Original test failure: looks like column type changes to different decimals in most cases. In one case it causes the integer part to be too big to fit, so the result becomes null it seems. What happens is that CBO adds casts to arithmetic expressions to make them type compatible; these casts become part of new AST, and then Hive adds casts on top of these casts. This (the first part) also causes lots of out file changes. It's not clear how to best fix it so far, in addition to incorrect decimal width and sometimes nulls when width is larger than allowed in Hive. Option one - don't add those for numeric ops - cannot be done if numeric op is a part of compare, for which CBO needs correct types. Option two - unwrap casts when determining type in Hive - hard or impossible to tell apart CBO-added casts and user casts. Option three - don't change types in Hive if CBO has run - seems hacky and hard to ensure it's applied everywhere. Option four - map all expressions precisely between two trees and remove casts again after optimization, will be pretty difficult. Option five - somehow mark those casts. Not sure about how yet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8111) CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO
[ https://issues.apache.org/jira/browse/HIVE-8111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-8111: --- Fix Version/s: 0.15.0 CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO Key: HIVE-8111 URL: https://issues.apache.org/jira/browse/HIVE-8111 Project: Hive Issue Type: Sub-task Components: CBO Affects Versions: 0.14.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.15.0 Attachments: HIVE-8111.01.patch, HIVE-8111.02.patch, HIVE-8111.03.patch, HIVE-8111.patch Original test failure: looks like column type changes to different decimals in most cases. In one case it causes the integer part to be too big to fit, so the result becomes null it seems. What happens is that CBO adds casts to arithmetic expressions to make them type compatible; these casts become part of new AST, and then Hive adds casts on top of these casts. This (the first part) also causes lots of out file changes. It's not clear how to best fix it so far, in addition to incorrect decimal width and sometimes nulls when width is larger than allowed in Hive. Option one - don't add those for numeric ops - cannot be done if numeric op is a part of compare, for which CBO needs correct types. Option two - unwrap casts when determining type in Hive - hard or impossible to tell apart CBO-added casts and user casts. Option three - don't change types in Hive if CBO has run - seems hacky and hard to ensure it's applied everywhere. Option four - map all expressions precisely between two trees and remove casts again after optimization, will be pretty difficult. Option five - somehow mark those casts. Not sure about how yet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8111) CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO
[ https://issues.apache.org/jira/browse/HIVE-8111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-8111: --- Affects Version/s: 0.14.0 CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO Key: HIVE-8111 URL: https://issues.apache.org/jira/browse/HIVE-8111 Project: Hive Issue Type: Sub-task Components: CBO Affects Versions: 0.14.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.15.0 Attachments: HIVE-8111.01.patch, HIVE-8111.02.patch, HIVE-8111.03.patch, HIVE-8111.patch Original test failure: looks like column type changes to different decimals in most cases. In one case it causes the integer part to be too big to fit, so the result becomes null it seems. What happens is that CBO adds casts to arithmetic expressions to make them type compatible; these casts become part of new AST, and then Hive adds casts on top of these casts. This (the first part) also causes lots of out file changes. It's not clear how to best fix it so far, in addition to incorrect decimal width and sometimes nulls when width is larger than allowed in Hive. Option one - don't add those for numeric ops - cannot be done if numeric op is a part of compare, for which CBO needs correct types. Option two - unwrap casts when determining type in Hive - hard or impossible to tell apart CBO-added casts and user casts. Option three - don't change types in Hive if CBO has run - seems hacky and hard to ensure it's applied everywhere. Option four - map all expressions precisely between two trees and remove casts again after optimization, will be pretty difficult. Option five - somehow mark those casts. Not sure about how yet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8111) CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO
[ https://issues.apache.org/jira/browse/HIVE-8111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-8111: --- Attachment: HIVE-8111.03.patch Test failed due to plan changes. Meanwhile remove cbo flag from the test. CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO Key: HIVE-8111 URL: https://issues.apache.org/jira/browse/HIVE-8111 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-8111.01.patch, HIVE-8111.02.patch, HIVE-8111.03.patch, HIVE-8111.patch Original test failure: looks like column type changes to different decimals in most cases. In one case it causes the integer part to be too big to fit, so the result becomes null it seems. What happens is that CBO adds casts to arithmetic expressions to make them type compatible; these casts become part of new AST, and then Hive adds casts on top of these casts. This (the first part) also causes lots of out file changes. It's not clear how to best fix it so far, in addition to incorrect decimal width and sometimes nulls when width is larger than allowed in Hive. Option one - don't add those for numeric ops - cannot be done if numeric op is a part of compare, for which CBO needs correct types. Option two - unwrap casts when determining type in Hive - hard or impossible to tell apart CBO-added casts and user casts. Option three - don't change types in Hive if CBO has run - seems hacky and hard to ensure it's applied everywhere. Option four - map all expressions precisely between two trees and remove casts again after optimization, will be pretty difficult. Option five - somehow mark those casts. Not sure about how yet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8111) CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO
[ https://issues.apache.org/jira/browse/HIVE-8111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-8111: - Parent Issue: HIVE-5775 (was: HIVE-7946) CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO Key: HIVE-8111 URL: https://issues.apache.org/jira/browse/HIVE-8111 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-8111.01.patch, HIVE-8111.patch Original test failure: looks like column type changes to different decimals in most cases. In one case it causes the integer part to be too big to fit, so the result becomes null it seems. What happens is that CBO adds casts to arithmetic expressions to make them type compatible; these casts become part of new AST, and then Hive adds casts on top of these casts. This (the first part) also causes lots of out file changes. It's not clear how to best fix it so far, in addition to incorrect decimal width and sometimes nulls when width is larger than allowed in Hive. Option one - don't add those for numeric ops - cannot be done if numeric op is a part of compare, for which CBO needs correct types. Option two - unwrap casts when determining type in Hive - hard or impossible to tell apart CBO-added casts and user casts. Option three - don't change types in Hive if CBO has run - seems hacky and hard to ensure it's applied everywhere. Option four - map all expressions precisely between two trees and remove casts again after optimization, will be pretty difficult. Option five - somehow mark those casts. Not sure about how yet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8111) CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO
[ https://issues.apache.org/jira/browse/HIVE-8111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-8111: --- Status: Patch Available (was: Open) CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO Key: HIVE-8111 URL: https://issues.apache.org/jira/browse/HIVE-8111 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-8111.01.patch, HIVE-8111.02.patch, HIVE-8111.patch Original test failure: looks like column type changes to different decimals in most cases. In one case it causes the integer part to be too big to fit, so the result becomes null it seems. What happens is that CBO adds casts to arithmetic expressions to make them type compatible; these casts become part of new AST, and then Hive adds casts on top of these casts. This (the first part) also causes lots of out file changes. It's not clear how to best fix it so far, in addition to incorrect decimal width and sometimes nulls when width is larger than allowed in Hive. Option one - don't add those for numeric ops - cannot be done if numeric op is a part of compare, for which CBO needs correct types. Option two - unwrap casts when determining type in Hive - hard or impossible to tell apart CBO-added casts and user casts. Option three - don't change types in Hive if CBO has run - seems hacky and hard to ensure it's applied everywhere. Option four - map all expressions precisely between two trees and remove casts again after optimization, will be pretty difficult. Option five - somehow mark those casts. Not sure about how yet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8111) CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO
[ https://issues.apache.org/jira/browse/HIVE-8111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-8111: --- Attachment: HIVE-8111.02.patch update patch for trunk CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO Key: HIVE-8111 URL: https://issues.apache.org/jira/browse/HIVE-8111 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-8111.01.patch, HIVE-8111.02.patch, HIVE-8111.patch Original test failure: looks like column type changes to different decimals in most cases. In one case it causes the integer part to be too big to fit, so the result becomes null it seems. What happens is that CBO adds casts to arithmetic expressions to make them type compatible; these casts become part of new AST, and then Hive adds casts on top of these casts. This (the first part) also causes lots of out file changes. It's not clear how to best fix it so far, in addition to incorrect decimal width and sometimes nulls when width is larger than allowed in Hive. Option one - don't add those for numeric ops - cannot be done if numeric op is a part of compare, for which CBO needs correct types. Option two - unwrap casts when determining type in Hive - hard or impossible to tell apart CBO-added casts and user casts. Option three - don't change types in Hive if CBO has run - seems hacky and hard to ensure it's applied everywhere. Option four - map all expressions precisely between two trees and remove casts again after optimization, will be pretty difficult. Option five - somehow mark those casts. Not sure about how yet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8111) CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO
[ https://issues.apache.org/jira/browse/HIVE-8111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-8111: --- Attachment: (was: HIVE-8111.02.patch) CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO Key: HIVE-8111 URL: https://issues.apache.org/jira/browse/HIVE-8111 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-8111.01.patch, HIVE-8111.patch Original test failure: looks like column type changes to different decimals in most cases. In one case it causes the integer part to be too big to fit, so the result becomes null it seems. What happens is that CBO adds casts to arithmetic expressions to make them type compatible; these casts become part of new AST, and then Hive adds casts on top of these casts. This (the first part) also causes lots of out file changes. It's not clear how to best fix it so far, in addition to incorrect decimal width and sometimes nulls when width is larger than allowed in Hive. Option one - don't add those for numeric ops - cannot be done if numeric op is a part of compare, for which CBO needs correct types. Option two - unwrap casts when determining type in Hive - hard or impossible to tell apart CBO-added casts and user casts. Option three - don't change types in Hive if CBO has run - seems hacky and hard to ensure it's applied everywhere. Option four - map all expressions precisely between two trees and remove casts again after optimization, will be pretty difficult. Option five - somehow mark those casts. Not sure about how yet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8111) CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO
[ https://issues.apache.org/jira/browse/HIVE-8111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-8111: --- Attachment: HIVE-8111.02.patch Also enable cbo for this test CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO Key: HIVE-8111 URL: https://issues.apache.org/jira/browse/HIVE-8111 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-8111.01.patch, HIVE-8111.02.patch, HIVE-8111.patch Original test failure: looks like column type changes to different decimals in most cases. In one case it causes the integer part to be too big to fit, so the result becomes null it seems. What happens is that CBO adds casts to arithmetic expressions to make them type compatible; these casts become part of new AST, and then Hive adds casts on top of these casts. This (the first part) also causes lots of out file changes. It's not clear how to best fix it so far, in addition to incorrect decimal width and sometimes nulls when width is larger than allowed in Hive. Option one - don't add those for numeric ops - cannot be done if numeric op is a part of compare, for which CBO needs correct types. Option two - unwrap casts when determining type in Hive - hard or impossible to tell apart CBO-added casts and user casts. Option three - don't change types in Hive if CBO has run - seems hacky and hard to ensure it's applied everywhere. Option four - map all expressions precisely between two trees and remove casts again after optimization, will be pretty difficult. Option five - somehow mark those casts. Not sure about how yet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8111) CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO
[ https://issues.apache.org/jira/browse/HIVE-8111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-8111: --- Attachment: HIVE-8111.01.patch updated patch. CBO build appears to be broken so not tested for now CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO Key: HIVE-8111 URL: https://issues.apache.org/jira/browse/HIVE-8111 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-8111.01.patch, HIVE-8111.patch Original test failure: looks like column type changes to different decimals in most cases. In one case it causes the integer part to be too big to fit, so the result becomes null it seems. What happens is that CBO adds casts to arithmetic expressions to make them type compatible; these casts become part of new AST, and then Hive adds casts on top of these casts. This (the first part) also causes lots of out file changes. It's not clear how to best fix it so far, in addition to incorrect decimal width and sometimes nulls when width is larger than allowed in Hive. Option one - don't add those for numeric ops - cannot be done if numeric op is a part of compare, for which CBO needs correct types. Option two - unwrap casts when determining type in Hive - hard or impossible to tell apart CBO-added casts and user casts. Option three - don't change types in Hive if CBO has run - seems hacky and hard to ensure it's applied everywhere. Option four - map all expressions precisely between two trees and remove casts again after optimization, will be pretty difficult. Option five - somehow mark those casts. Not sure about how yet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8111) CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO
[ https://issues.apache.org/jira/browse/HIVE-8111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-8111: --- Attachment: HIVE-8111.patch [~ashutoshc] [~jpullokkaran] Attaching the patch. While the spurious casts are retained, result types are now all correct and the query results are correct for decimal_udf. Notes: - there's a bug on CBO branch right now, so I reverted my recent merge and obtained the out file before that; - the out file includes changes for the remaining casts (which are ok) - the out file also includes changes where extra stage is added to each query (same on CBO branch). I am not sure if this is ok. Will post rb shortly CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO Key: HIVE-8111 URL: https://issues.apache.org/jira/browse/HIVE-8111 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-8111.patch Original test failure: looks like column type changes to different decimals in most cases. In one case it causes the integer part to be too big to fit, so the result becomes null it seems. What happens is that CBO adds casts to arithmetic expressions to make them type compatible; these casts become part of new AST, and then Hive adds casts on top of these casts. This (the first part) also causes lots of out file changes. It's not clear how to best fix it so far, in addition to incorrect decimal width and sometimes nulls when width is larger than allowed in Hive. Option one - don't add those for numeric ops - cannot be done if numeric op is a part of compare, for which CBO needs correct types. Option two - unwrap casts when determining type in Hive - hard or impossible to tell apart CBO-added casts and user casts. Option three - don't change types in Hive if CBO has run - seems hacky and hard to ensure it's applied everywhere. Option four - map all expressions precisely between two trees and remove casts again after optimization, will be pretty difficult. Option five - somehow mark those casts. Not sure about how yet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8111) CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO
[ https://issues.apache.org/jira/browse/HIVE-8111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-8111: --- Summary: CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO (was: CBO trunk merge: decimal_udf test output changes) CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO Key: HIVE-8111 URL: https://issues.apache.org/jira/browse/HIVE-8111 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Looks like column type changes to different decimals in most cases. In one case it causes the integer part to be too big to fit, so the result becomes null it seems. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8111) CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO
[ https://issues.apache.org/jira/browse/HIVE-8111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-8111: --- Description: Original test failure: looks like column type changes to different decimals in most cases. In one case it causes the integer part to be too big to fit, so the result becomes null it seems. (was: Looks like column type changes to different decimals in most cases. In one case it causes the integer part to be too big to fit, so the result becomes null it seems.) CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO Key: HIVE-8111 URL: https://issues.apache.org/jira/browse/HIVE-8111 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Original test failure: looks like column type changes to different decimals in most cases. In one case it causes the integer part to be too big to fit, so the result becomes null it seems. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8111) CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO
[ https://issues.apache.org/jira/browse/HIVE-8111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-8111: --- Description: Original test failure: looks like column type changes to different decimals in most cases. In one case it causes the integer part to be too big to fit, so the result becomes null it seems. What happens is that CBO adds casts to arithmetic expressions to make them type compatible; these casts become part of new AST, and then Hive adds casts on top of these casts. This (the first part) also causes lots of out file changes. It's not clear how to best fix it so far, in addition to incorrect decimal width and sometimes nulls when width is larger than allowed in Hive. Option one - don't add those for numeric ops - cannot be done if numeric op is a part of compare, for which CBO needs correct types. Option two - unwrap casts when determining type in Hive - hard or impossible to tell apart CBO-added casts and user casts. Option three - don't change types in Hive if CBO has run - seems hacky and hard to ensure it's applied everywhere. Option four - map all expressions precisely between two trees and remove casts again after optimization, will be pretty difficult. Option five - somehow mark those casts. Not sure about how yet. was:Original test failure: looks like column type changes to different decimals in most cases. In one case it causes the integer part to be too big to fit, so the result becomes null it seems. CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO Key: HIVE-8111 URL: https://issues.apache.org/jira/browse/HIVE-8111 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Original test failure: looks like column type changes to different decimals in most cases. In one case it causes the integer part to be too big to fit, so the result becomes null it seems. What happens is that CBO adds casts to arithmetic expressions to make them type compatible; these casts become part of new AST, and then Hive adds casts on top of these casts. This (the first part) also causes lots of out file changes. It's not clear how to best fix it so far, in addition to incorrect decimal width and sometimes nulls when width is larger than allowed in Hive. Option one - don't add those for numeric ops - cannot be done if numeric op is a part of compare, for which CBO needs correct types. Option two - unwrap casts when determining type in Hive - hard or impossible to tell apart CBO-added casts and user casts. Option three - don't change types in Hive if CBO has run - seems hacky and hard to ensure it's applied everywhere. Option four - map all expressions precisely between two trees and remove casts again after optimization, will be pretty difficult. Option five - somehow mark those casts. Not sure about how yet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
