[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15594989#comment-15594989
]
Ashish Shrowty commented on SPARK-17709:
[~sowen], [~smilegator] - I confirmed that this is not a
[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578200#comment-15578200
]
Ashish Shrowty commented on SPARK-17709:
Oh .. sorry .. I misread. Will try with 2.0.1 later
>
[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576763#comment-15576763
]
Xiao Li commented on SPARK-17709:
-
That is what I said above. The deduplication is not triggered. It
[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576143#comment-15576143
]
Ashish Shrowty commented on SPARK-17709:
There is a slight difference .. in my case its
[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576141#comment-15576141
]
Ashish Shrowty commented on SPARK-17709:
There is a slight difference, in my case the IDs
[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15575999#comment-15575999
]
Xiao Li commented on SPARK-17709:
-
Below is the statements I used to recreate the problem
{noformat}
[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15575995#comment-15575995
]
Xiao Li commented on SPARK-17709:
-
Still works well in 2.0.1
> spark 2.0 join - column resolution error
[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15575938#comment-15575938
]
Xiao Li commented on SPARK-17709:
-
I can get an exactly same plan in the master branch, but my job can
[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573512#comment-15573512
]
Ashish Shrowty commented on SPARK-17709:
[~smilegator] I compiled with the added debug
[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566160#comment-15566160
]
Ashish Shrowty commented on SPARK-17709:
Cool.. thanks. Will do this in next day or two.
> spark
[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566126#comment-15566126
]
Xiao Li commented on SPARK-17709:
-
Below is the link:
[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566124#comment-15566124
]
Xiao Li commented on SPARK-17709:
-
Below is the link:
[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566114#comment-15566114
]
Ashish Shrowty commented on SPARK-17709:
I assume I would need to modify the Spark code and build
[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562826#comment-15562826
]
Xiao Li commented on SPARK-17709:
-
It is pretty hard to reproduce it in our environment. Are you able to
[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562368#comment-15562368
]
Ashish Shrowty commented on SPARK-17709:
[~dkbiswal], [~smilegator] - Hi guys .. any thoughts? I
[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15543921#comment-15543921
]
Ashish Shrowty commented on SPARK-17709:
[~dkbiswal] Sorry Dilip .. I keep making typos .. the
[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15543141#comment-15543141
]
Dilip Biswal commented on SPARK-17709:
--
@ashrowty Hi Ashish, in your example, the column
[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15543046#comment-15543046
]
Dilip Biswal commented on SPARK-17709:
--
Hi Ashish, Thanks a lot.. will try and get back.
> spark
[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15538927#comment-15538927
]
Ashish Shrowty commented on SPARK-17709:
[~dkbiswal] I just went through manual steps of creating
[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537740#comment-15537740
]
Ashish Shrowty commented on SPARK-17709:
Join keys are both companyid and loyaltycardnumber.
[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537336#comment-15537336
]
Dilip Biswal commented on SPARK-17709:
--
[~ashrowty] Hmmn.. and your join keys are companyid or
[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537264#comment-15537264
]
Ashish Shrowty commented on SPARK-17709:
[~dkbiswal] Attached are the explain() outputs -
[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537205#comment-15537205
]
Dilip Biswal commented on SPARK-17709:
--
@ashrowty Hi Ashish, is it possible for you to post explain
[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536694#comment-15536694
]
Ashish Shrowty commented on SPARK-17709:
Sorry ... its not really col1, its another column ..
[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536464#comment-15536464
]
Dilip Biswal commented on SPARK-17709:
--
[~ashrowty] Ashish, you have the same column name as regular
[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15535957#comment-15535957
]
Ashish Shrowty commented on SPARK-17709:
Sure .. the data is brought over into the EMR (5.0.0)
[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534910#comment-15534910
]
Xiao Li commented on SPARK-17709:
-
[~ashrowty]Can you share the exact way how you load the external
[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534518#comment-15534518
]
Ashish Shrowty commented on SPARK-17709:
Dilip,
I tried your code and it works on my end too.
[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534434#comment-15534434
]
Dilip Biswal commented on SPARK-17709:
--
[~smilegator] Sure.
> spark 2.0 join - column resolution
[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534429#comment-15534429
]
Xiao Li commented on SPARK-17709:
-
Can you try it in the latest 2.0?
> spark 2.0 join - column
[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534417#comment-15534417
]
Dilip Biswal commented on SPARK-17709:
--
[~smilegator] Hi Sean, I tried it on my master branch and
[
https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534338#comment-15534338
]
Xiao Li commented on SPARK-17709:
-
Let me try to reproduce it. Thanks!
> spark 2.0 join - column
32 matches
Mail list logo
