[jira] [Commented] (KYLIN-3084) File not found Exception when processing union-all in TEZ mode
[ https://issues.apache.org/jira/browse/KYLIN-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407391#comment-16407391 ] ASF subversion and git services commented on KYLIN-3084: Commit aa2f8e796f387c1227c8fa216b9af69d0ee9de42 in kylin's branch refs/heads/2.3.x from shaofengshi [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=aa2f8e7 ] KYLIN-3278 Revert "KYLIN-3084, fix issue by using distribute by" This reverts commit 849faec7d63b59af58eadc331f00c96343f18cc8. Remove unnecessary UT > File not found Exception when processing union-all in TEZ mode > -- > > Key: KYLIN-3084 > URL: https://issues.apache.org/jira/browse/KYLIN-3084 > Project: Kylin > Issue Type: Bug >Reporter: Wang Cheng >Assignee: Wang Cheng >Priority: Minor > Fix For: v2.3.0 > > > If hive.execution.engine=TEZ and hql contains union all, it causes exception > like: file not found when materializing the view or redistributing flat hive > table. > Here is the reason: > http://grokbase.com/t/hive/user/162r80a2g9/anyway-to-avoid-creating-subdirectories-by-insert-with-union > i.e. "The Tez execution of UNION is entirely parallel & > the task-ids overlaps - so the files created have to have unique names. > But the total counts for "Map 1" and "Map 2" are only available as the job > runs, so they write to different dirs." > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3084) File not found Exception when processing union-all in TEZ mode
[ https://issues.apache.org/jira/browse/KYLIN-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407372#comment-16407372 ] Shaofeng SHI commented on KYLIN-3084: - Hi Le Anh Vu, this change will be reverted, and a fixpack 2.3.1 will be released soon. > File not found Exception when processing union-all in TEZ mode > -- > > Key: KYLIN-3084 > URL: https://issues.apache.org/jira/browse/KYLIN-3084 > Project: Kylin > Issue Type: Bug >Reporter: Wang Cheng >Assignee: Wang Cheng >Priority: Minor > Fix For: v2.3.0 > > > If hive.execution.engine=TEZ and hql contains union all, it causes exception > like: file not found when materializing the view or redistributing flat hive > table. > Here is the reason: > http://grokbase.com/t/hive/user/162r80a2g9/anyway-to-avoid-creating-subdirectories-by-insert-with-union > i.e. "The Tez execution of UNION is entirely parallel & > the task-ids overlaps - so the files created have to have unique names. > But the total counts for "Map 1" and "Map 2" are only available as the job > runs, so they write to different dirs." > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3084) File not found Exception when processing union-all in TEZ mode
[ https://issues.apache.org/jira/browse/KYLIN-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396384#comment-16396384 ] Le Anh Vu commented on KYLIN-3084: -- Using distribute by makes Create flat table step run slower in kylin 2.3.0 compare to previous versions. Can you guys have another solution for this issue? I've already made a comment here on github: [https://github.com/apache/kylin/commit/849faec]. Thanks > File not found Exception when processing union-all in TEZ mode > -- > > Key: KYLIN-3084 > URL: https://issues.apache.org/jira/browse/KYLIN-3084 > Project: Kylin > Issue Type: Bug >Reporter: Wang Cheng >Assignee: Wang Cheng >Priority: Minor > Fix For: v2.3.0 > > > If hive.execution.engine=TEZ and hql contains union all, it causes exception > like: file not found when materializing the view or redistributing flat hive > table. > Here is the reason: > http://grokbase.com/t/hive/user/162r80a2g9/anyway-to-avoid-creating-subdirectories-by-insert-with-union > i.e. "The Tez execution of UNION is entirely parallel & > the task-ids overlaps - so the files created have to have unique names. > But the total counts for "Map 1" and "Map 2" are only available as the job > runs, so they write to different dirs." > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3084) File not found Exception when processing union-all in TEZ mode
[ https://issues.apache.org/jira/browse/KYLIN-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16337181#comment-16337181 ] Billy Liu commented on KYLIN-3084: -- Test case: https://github.com/apache/kylin/commit/d173c7e > File not found Exception when processing union-all in TEZ mode > -- > > Key: KYLIN-3084 > URL: https://issues.apache.org/jira/browse/KYLIN-3084 > Project: Kylin > Issue Type: Bug >Reporter: Wang Cheng >Assignee: Wang Cheng >Priority: Minor > Fix For: v2.3.0 > > > If hive.execution.engine=TEZ and hql contains union all, it causes exception > like: file not found when materializing the view or redistributing flat hive > table. > Here is the reason: > http://grokbase.com/t/hive/user/162r80a2g9/anyway-to-avoid-creating-subdirectories-by-insert-with-union > i.e. "The Tez execution of UNION is entirely parallel & > the task-ids overlaps - so the files created have to have unique names. > But the total counts for "Map 1" and "Map 2" are only available as the job > runs, so they write to different dirs." > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3084) File not found Exception when processing union-all in TEZ mode
[ https://issues.apache.org/jira/browse/KYLIN-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16288612#comment-16288612 ] Wang Cheng commented on KYLIN-3084: --- [~seva_ostapenko] Verified it works in Kylin, thanks. > File not found Exception when processing union-all in TEZ mode > -- > > Key: KYLIN-3084 > URL: https://issues.apache.org/jira/browse/KYLIN-3084 > Project: Kylin > Issue Type: Bug >Reporter: Wang Cheng >Assignee: Wang Cheng >Priority: Minor > > If hive.execution.engine=TEZ and hql contains union all, it causes exception > like: file not found when materializing the view or redistributing flat hive > table. > Here is the reason: > http://grokbase.com/t/hive/user/162r80a2g9/anyway-to-avoid-creating-subdirectories-by-insert-with-union > i.e. "The Tez execution of UNION is entirely parallel & > the task-ids overlaps - so the files created have to have unique names. > But the total counts for "Map 1" and "Map 2" are only available as the job > runs, so they write to different dirs." > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
