[
https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15670679#comment-15670679
]
Herman van Hovell commented on SPARK-12179:
---
[~litao1990] is this still a problem?
> Spark SQL
[
https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15060213#comment-15060213
]
Davies Liu commented on SPARK-12179:
I think this UDF is not thread safe, rowNum and comparedColumn
[
https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15060217#comment-15060217
]
Davies Liu commented on SPARK-12179:
Which version of Spark are you using?
> Spark SQL get
[
https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048496#comment-15048496
]
Tao Li commented on SPARK-12179:
I try to use spark internal Row_Number() udf, but still has the problem.
[
https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15047186#comment-15047186
]
Davies Liu commented on SPARK-12179:
Could you also test 1.6-RC1?
I'm just wondering that the window
[
https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15047193#comment-15047193
]
Davies Liu commented on SPARK-12179:
There are two direction to narrow down the problem:
1) simplify
[
https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15047962#comment-15047962
]
Tao Li commented on SPARK-12179:
Sorry, row_number is udf written by myself, not spark internal udf. Do I
[
https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15047938#comment-15047938
]
Tao Li commented on SPARK-12179:
ok, i will try on it
> Spark SQL get different result with the same
[
https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15047965#comment-15047965
]
Tao Li commented on SPARK-12179:
The row_number implementation is as follows:
package UDF;
import
[
https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045028#comment-15045028
]
Sean Owen commented on SPARK-12179:
---
Yes, that's the critical information. I think it's hard to parse
[
https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045014#comment-15045014
]
Tao Li commented on SPARK-12179:
[~sowen] My sql and command line param is like this:
DATE=$1
[
https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045053#comment-15045053
]
Tao Li commented on SPARK-12179:
The query is on a hive table and the hive data is not changing.
I think
[
https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045012#comment-15045012
]
Tao Li commented on SPARK-12179:
In the same stage with the same shuffle read, I think it should get the
[
https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045015#comment-15045015
]
Sean Owen commented on SPARK-12179:
---
I don't see that you've shown results at all. What do you expect
[
https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045010#comment-15045010
]
Tao Li commented on SPARK-12179:
[~srowen] In this case, there is no task failures and I set
[
https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045024#comment-15045024
]
Tao Li commented on SPARK-12179:
For example:
1. First Time Run, I got:
20151204maingoogle
[
https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046240#comment-15046240
]
Tao Li commented on SPARK-12179:
yes, my spark version is 1.5.3-SNAPSHOT,and the issue SPARK-11009 is
[
https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046238#comment-15046238
]
Tao Li commented on SPARK-12179:
yes, my spark version is 1.5.3-SNAPSHOT,and the issue SPARK-11009 is
[
https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046239#comment-15046239
]
Tao Li commented on SPARK-12179:
yes, my spark version is 1.5.3-SNAPSHOT,and the issue SPARK-11009 is
[
https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046243#comment-15046243
]
Tao Li commented on SPARK-12179:
I see there was some exceptions in my executors stderr log:
```
[
https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046014#comment-15046014
]
Davies Liu commented on SPARK-12179:
This may be related to
21 matches
Mail list logo
