[jira] [Commented] (KYLIN-1789) Couldn't use View as Lookup when join type is "inner"
[ https://issues.apache.org/jira/browse/KYLIN-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15361047#comment-15361047 ] Shaofeng SHI commented on KYLIN-1789: - Right; the "pk" issue under "inner" scenario is found by Bhanu Mohanty; > Couldn't use View as Lookup when join type is "inner" > - > > Key: KYLIN-1789 > URL: https://issues.apache.org/jira/browse/KYLIN-1789 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.5.2, v1.5.2.1 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI > Fix For: v1.5.3 > > > Reported by Bhanu Mohanty in user mailing list: > I am using kylin-1.5.2.1 > Added hive view as a look up table > Getting error at Build Dimension Dictionary > DEFAULT.kylin_intermediate_DEFAULT_* > If the join is "inner" > It works when I changed the join to "left" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1789) Couldn't use View as Lookup when join type is "inner"
[ https://issues.apache.org/jira/browse/KYLIN-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15361036#comment-15361036 ] wangxianbin commented on KYLIN-1789: hi shaofeng! so the problem for Bhanu Mohanty is that he have duplicate keys in lookup, however you found another issue which is we miss materializing lookup views which PK appeared to be a dimension on fact table? > Couldn't use View as Lookup when join type is "inner" > - > > Key: KYLIN-1789 > URL: https://issues.apache.org/jira/browse/KYLIN-1789 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.5.2, v1.5.2.1 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI > Fix For: v1.5.3 > > > Reported by Bhanu Mohanty in user mailing list: > I am using kylin-1.5.2.1 > Added hive view as a look up table > Getting error at Build Dimension Dictionary > DEFAULT.kylin_intermediate_DEFAULT_* > If the join is "inner" > It works when I changed the join to "left" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1789) Couldn't use View as Lookup when join type is "inner"
[ https://issues.apache.org/jira/browse/KYLIN-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15333186#comment-15333186 ] Shaofeng SHI commented on KYLIN-1789: - option 1) may impact legacy cubes; option 3) will add the code complexity; I choose 2) now, assuming the number of lookup views is minor. > Couldn't use View as Lookup when join type is "inner" > - > > Key: KYLIN-1789 > URL: https://issues.apache.org/jira/browse/KYLIN-1789 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.5.2, v1.5.2.1 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI > Fix For: v1.5.3 > > > Reported by Bhanu Mohanty in user mailing list: > I am using kylin-1.5.2.1 > Added hive view as a look up table > Getting error at Build Dimension Dictionary > DEFAULT.kylin_intermediate_DEFAULT_* > If the join is "inner" > It works when I changed the join to "left" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1789) Couldn't use View as Lookup when join type is "inner"
[ https://issues.apache.org/jira/browse/KYLIN-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15333176#comment-15333176 ] Shaofeng SHI commented on KYLIN-1789: - Pushed fix to master branch, and cover the inner join case in the integration test. > Couldn't use View as Lookup when join type is "inner" > - > > Key: KYLIN-1789 > URL: https://issues.apache.org/jira/browse/KYLIN-1789 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.5.2, v1.5.2.1 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI > Fix For: v1.5.3 > > > Reported by Bhanu Mohanty in user mailing list: > I am using kylin-1.5.2.1 > Added hive view as a look up table > Getting error at Build Dimension Dictionary > DEFAULT.kylin_intermediate_DEFAULT_* > If the join is "inner" > It works when I changed the join to "left" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1789) Couldn't use View as Lookup when join type is "inner"
[ https://issues.apache.org/jira/browse/KYLIN-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15333175#comment-15333175 ] Shaofeng SHI commented on KYLIN-1789: - Hi Richard, Kylin need the lookup table's PK be unique, regardless the join type; The phenomenon you described is just because that that table wasn't taken snapshot in "left outer" so kylin didn't check that. If the cube has a dimension column from that table, it will report the same error. > Couldn't use View as Lookup when join type is "inner" > - > > Key: KYLIN-1789 > URL: https://issues.apache.org/jira/browse/KYLIN-1789 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.5.2, v1.5.2.1 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI > Fix For: v1.5.3 > > > Reported by Bhanu Mohanty in user mailing list: > I am using kylin-1.5.2.1 > Added hive view as a look up table > Getting error at Build Dimension Dictionary > DEFAULT.kylin_intermediate_DEFAULT_* > If the join is "inner" > It works when I changed the join to "left" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1789) Couldn't use View as Lookup when join type is "inner"
[ https://issues.apache.org/jira/browse/KYLIN-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332920#comment-15332920 ] Richard Calaba commented on KYLIN-1789: --- Additional info - also obtained from Bhanu Mohanty - the reporter of the bug: While looking into logs he saw a message: Dup key found, key=[-289271615434074838,-7076210457049756771], value1=[2016-03-24,-289271615434074838,-7076210457049756771,Medium,6,4,,null,null,null,null,null], value2=[2016-03-24,-289271615434074838,-7076210457049756771,Medium,6,4,,null,null,null,null,null] at org.apache.kylin.dict.lookup.LookupTable.initRow(LookupTable.java:83) That would indicate that the Inner Join expected in the lookup table to find only one record ... if this is true and the Inner Join on Lookup req. one record that it is clear inconsistency (1st the model is defined as FACT INNER JOIN LOOKUP ON key1, key2, ... so this has to allow multiple candidates in lookup (even this is not typical) ; 2nd there is no reason why Left Outer join would accept duplicate key entries in the Right operand and Inner join won't allow it) > Couldn't use View as Lookup when join type is "inner" > - > > Key: KYLIN-1789 > URL: https://issues.apache.org/jira/browse/KYLIN-1789 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.5.2, v1.5.2.1 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI > Fix For: v1.5.3 > > > Reported by Bhanu Mohanty in user mailing list: > I am using kylin-1.5.2.1 > Added hive view as a look up table > Getting error at Build Dimension Dictionary > DEFAULT.kylin_intermediate_DEFAULT_* > If the join is "inner" > It works when I changed the join to "left" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1789) Couldn't use View as Lookup when join type is "inner"
[ https://issues.apache.org/jira/browse/KYLIN-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332914#comment-15332914 ] Shaofeng SHI commented on KYLIN-1789: - Kylin does have a speical optimization for "inner": if a dimension column is the FK on fact table, and the join type is "inner", it will defer the dictionary build to lookup table's PK; While when materializing the views, as that dimension appeared to be on fact table, the view is skipped for materializing. possible solutions: 1) remove that optimization on "inner" or 2) materializing all lookup views of the model (regradless column source and join types) or 3) check whether a dimension column is FK, if yes check whether the join type is "inner"; if all true, materializing the view. > Couldn't use View as Lookup when join type is "inner" > - > > Key: KYLIN-1789 > URL: https://issues.apache.org/jira/browse/KYLIN-1789 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.5.2, v1.5.2.1 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI > Fix For: v1.5.3 > > > Reported by Bhanu Mohanty in user mailing list: > I am using kylin-1.5.2.1 > Added hive view as a look up table > Getting error at Build Dimension Dictionary > DEFAULT.kylin_intermediate_DEFAULT_* > If the join is "inner" > It works when I changed the join to "left" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
