Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/9247#issuecomment-151439866
Thanks, merged to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/9247
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/9247#issuecomment-151035107
I think we should follow hive, thus we also need to update the
`checkInputDataTypes` to deny struct type.
cc @yhuai @marmbrus , what do you think?
---
If
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/9247#issuecomment-151066487
We support sorting on structs due to an `ORDER BY`, I don't see why we
shouldn't support it in arrays. Its useful for computing things like argmax.
---
If your
Github user jliwork commented on the pull request:
https://github.com/apache/spark/pull/9247#issuecomment-151274333
@marmbrus @cloud-fan Thank you for your comment. My current fix takes care
of array of NullType. I should be able to give error to block sort_array on
struct type as
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9247#issuecomment-151314576
**[Test build #44385 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44385/consoleFull)**
for PR 9247 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9247#issuecomment-151329960
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9247#issuecomment-151329981
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9247#issuecomment-151313883
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9247#issuecomment-151313869
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user jliwork commented on a diff in the pull request:
https://github.com/apache/spark/pull/9247#discussion_r43074419
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionFunctionsSuite.scala
---
@@ -64,6 +65,21 @@ class
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/9247#discussion_r43073001
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionFunctionsSuite.scala
---
@@ -64,6 +65,21 @@ class
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9247#issuecomment-151330425
**[Test build #44388 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44388/consoleFull)**
for PR 9247 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9247#issuecomment-151340581
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9247#issuecomment-151340580
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user jliwork commented on the pull request:
https://github.com/apache/spark/pull/9247#issuecomment-151341426
Thank you all. I have updated my fix based on your comments. It should now
be able to:
1) skip sorting an array of NULLs
2) sort an array of StructType
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/9247#issuecomment-151370701
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9247#issuecomment-151356075
**[Test build #44388 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44388/consoleFull)**
for PR 9247 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9247#issuecomment-151356391
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9247#issuecomment-151356393
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9247#issuecomment-151340487
**[Test build #44385 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44385/consoleFull)**
for PR 9247 at commit
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/9247#issuecomment-150918873
Hi @jliwork , thanks for working on it!
But sorting on array of null type doesn't make sense to me, can you check
the behaviour of other SQL system like Hive? And
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/9247#issuecomment-151010243
Ah, I don't mean something like sort_struct, but sort_array with struct
type elements. Does hive support it?
---
If your project is set up for it, you can reply to
Github user jliwork commented on the pull request:
https://github.com/apache/spark/pull/9247#issuecomment-150992372
@cloud-fan Thank you very much for your comment! Sorting on array of NULL
type does not make sense to me, either. That's why my fix is to block this kind
of sorting for
Github user jliwork commented on the pull request:
https://github.com/apache/spark/pull/9247#issuecomment-151030490
@cloud-fan Thanks for the clarification. You have a very good point. This
bug's exception occourred because SortArray's Comparators expect AtomicType
only. According to
GitHub user jliwork opened a pull request:
https://github.com/apache/spark/pull/9247
[SPARK-11277] [SQL] sort_array throws exception scala.MatchError
I'm new to spark. I was trying out the sort_array function then hit this
exception. I looked into the spark source code. I found the
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9247#issuecomment-150526992
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9247#issuecomment-150527270
**[Test build #44218 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44218/consoleFull)**
for PR 9247 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9247#issuecomment-150526958
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9247#issuecomment-150546662
**[Test build #44218 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44218/consoleFull)**
for PR 9247 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9247#issuecomment-150546789
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9247#issuecomment-150525067
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9247#issuecomment-150546790
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/9247#issuecomment-150526270
ok to test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
34 matches
Mail list logo
