Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/3378#issuecomment-64971765
I don't understand the architecture here as well as @rxin but this change
seems like a strict improvement in its current form, so I'm gonna pull it in.
LGTM.
---
If
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/3378
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/3378#issuecomment-64303243
@rxin Is it OK to merge?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/3378#issuecomment-63774979
We should definitely add a ClassTag since this can be used for primitive
types. However, there might be places where we create a lot of CompactBuffers.
I haven't had a
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/3378#issuecomment-63775682
Cogroup uses `CompactBuffer`. However, it cannot add ClassTag due to its
signature:
```Scala
class CoGroupedRDD[K](@transient var rdds: Seq[RDD[_ :
GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/3378
[SPARK-4505][Core] Add a ClassTag parameter to CompactBuffer[T]
Added a ClassTag parameter to CompactBuffer. So CompactBuffer[T] can create
primitive arrays for primitive types. It will reduce the
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3378#issuecomment-63765080
[Test build #23661 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23661/consoleFull)
for PR 3378 at commit
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/3378#issuecomment-63765566
This seems like probably a great idea. Do you know what the overhead of
including a classtag is? Does it mean an extra pointer per object?
---
If your project is set up
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/3378#issuecomment-63766519
Does it mean an extra pointer per object?
No. E.g., ClassTag.Int will be shared by all CompactBuffer[Int]. Same
approach has already bean used in RDD.
---
If
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/3378#issuecomment-63769414
It's weird. I just found both the sizes of old and new `CompactBuffer(1)`
are 56. I cannot explain why.
Then I added a field to the old CompactBuffer like this:
Github user aarondav commented on the pull request:
https://github.com/apache/spark/pull/3378#issuecomment-63769839
This does seem like a good change, though I'll note that I think groupBy is
the only current user of this API that is able to have a primitive ClassTag.
Still
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3378#issuecomment-63770652
[Test build #23661 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23661/consoleFull)
for PR 3378 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3378#issuecomment-63770658
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/3378#issuecomment-63770746
It's weird. I just found both the sizes of old and new CompactBuffer(1)
are 56.
Found the cause. My JVM enables `UseCompressedOops`. So in such case, due
to
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/3378#issuecomment-63772021
My motivation is that we encountered a skew data set that a special hot key
has too many values and could not fit into memory. Spilling helps nothing in
this case since
15 matches
Mail list logo
