Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/17300
Hi, @jerryshao @mridulm Thanks for your review, I have updated the code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/17300#discussion_r107401186
--- Diff:
core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
@@ -497,7 +497,30 @@ class BlockManagerSuite extends SparkFunSuite with
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/17300
Ok, thanks a lot.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/17088
I agree with @mridulm, file fetch failure does not imply the executor down
or all the executor of the host down.
---
If your project is set up for it, you can reply to this email and have your
GitHub user ConeyLiu opened a pull request:
https://github.com/apache/spark/pull/17243
[SPARK-19901][Core]Clean up the clunky method signature of acquireMemory
## What changes were proposed in this pull request?
Clean up the clunky method signature of acquireMemory
Github user ConeyLiu closed the pull request at:
https://github.com/apache/spark/pull/17243
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/17243
Sorry for the mistake, reopen it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
GitHub user ConeyLiu reopened a pull request:
https://github.com/apache/spark/pull/17243
[SPARK-19901][Core]Clean up the clunky method signature of acquireMemory
## What changes were proposed in this pull request?
Clean up the clunky method signature of acquireMemory
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/17243
Hi, @srowen @HyukjinKwon, thanks for the review. This patch stems from the
`TODO` list, which is used to streamline the method signature, make it more
readable. If you think there is no essential
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/17243
ok, I will close it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes
Github user ConeyLiu closed the pull request at:
https://github.com/apache/spark/pull/17243
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
GitHub user ConeyLiu opened a pull request:
https://github.com/apache/spark/pull/17300
[SPARK-19956][Core]Optimize a location order of blocks with topology
information
## What changes were proposed in this pull request?
When call the method getLocations of BlockManager, we
GitHub user ConeyLiu opened a pull request:
https://github.com/apache/spark/pull/15864
[SPARK-18420]Fix the compile errors caused by checkstyle
## What changes were proposed in this pull request?
Small fix, fix the compile errors caused by checkstyle
## How was
Github user ConeyLiu closed the pull request at:
https://github.com/apache/spark/pull/15864
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
GitHub user ConeyLiu opened a pull request:
https://github.com/apache/spark/pull/15865
[SPARK-18420]Fix the compile errors caused by checkstyle
## What changes were proposed in this pull request?
Small fix, fix the compile errors caused by checkstyle
## How was
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/15865
@HyukjinKwon I'm sorry for that, I've fixed it, please take a look. Thanks
a lot.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitH
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/15865
@srowen Thanks for the review, I've updated it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/15865
@HyukjinKwon Thanks for the review and suggestion, I've updated it. Clear
the unused object `hasher` and add suppression rules for the method `finalize`
of `NioBufferedFileInputStream`. P
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/15865#discussion_r87799361
--- Diff: dev/checkstyle-suppressions.xml ---
@@ -30,6 +30,8 @@
+
--- End diff --
@HyukjinKwon Also we could try
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/15865
@HyukjinKwon I'm sorry for so late reply. I update the code and style
followed by your suggestion, that's so useful. And also I change the method of
disabling the check around `finalize
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/15865#discussion_r87996878
--- Diff: dev/checkstyle.xml ---
@@ -52,6 +52,20
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/15865
@srowen hi owen, Can you give me some advice, and if so, what a great thing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/15865#discussion_r88036045
--- Diff: dev/checkstyle.xml ---
@@ -168,5 +182,6 @@
+
--- End diff --
@srowen
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/15865
@HyukjinKwon @srowen Thank you very much.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
GitHub user ConeyLiu opened a pull request:
https://github.com/apache/spark/pull/20026
[SPARK-22838][Core] Avoid unnecessary copying of data
## What changes were proposed in this pull request?
If we read data from FileChannel to HeapByteBuffer, there is a need to copy
the
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/20026
@cloud-fan Please take a look, thanks a lot.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/20026#discussion_r157938250
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -208,7 +209,7 @@ private class EncryptedBlockData(
conf: SparkConf
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/20026#discussion_r158181658
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -208,7 +209,7 @@ private class EncryptedBlockData(
conf: SparkConf
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/20026#discussion_r158220052
--- Diff:
core/src/main/java/org/apache/spark/io/NioBufferedFileInputStream.java ---
@@ -61,6 +61,7 @@ private boolean refill() throws IOException
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/20026#discussion_r158220107
--- Diff:
core/src/main/java/org/apache/spark/io/NioBufferedFileInputStream.java ---
@@ -91,7 +92,12 @@ public synchronized int read(byte[] b, int offset
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/20026#discussion_r158243377
--- Diff:
core/src/main/java/org/apache/spark/io/NioBufferedFileInputStream.java ---
@@ -61,6 +61,7 @@ private boolean refill() throws IOException
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/20026
It seems the error not related. And can you add me to the whitelist?
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/20026#discussion_r158279099
--- Diff:
core/src/main/java/org/apache/spark/io/NioBufferedFileInputStream.java ---
@@ -61,6 +61,7 @@ private boolean refill() throws IOException
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/20026
I'll update it tomorrow.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/20026
cc @jiangxb1987 any comments on this?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/20844#discussion_r175634287
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala
---
@@ -396,9 +396,11 @@ case class RangeExec(range
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/20844#discussion_r175658889
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala
---
@@ -396,9 +396,11 @@ case class RangeExec(range
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/20844#discussion_r175966108
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala
---
@@ -396,9 +396,11 @@ case class RangeExec(range
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/20844
@cloud-fan, pls take a look, thanks a lot.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/20844
This change is very simple, and just make it consistent with other
`LeafNode`.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/20844#discussion_r176327636
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala
---
@@ -348,6 +348,13 @@ case class RangeExec(range
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/19285#discussion_r162802949
--- Diff:
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---
@@ -261,37 +263,93 @@ private[spark] class MemoryStore
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/19285
Thanks for reviewing. The code has updated, pls help to review. Thanks
again.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/20026#discussion_r162803175
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -152,7 +153,7 @@ private class DiskBlockData(
file: File
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/20026#discussion_r162810684
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -152,7 +153,7 @@ private class DiskBlockData(
file: File
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/20026
close it, thanks for everyone.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user ConeyLiu closed the pull request at:
https://github.com/apache/spark/pull/20026
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/19285#discussion_r162840776
--- Diff:
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---
@@ -162,26 +162,29 @@ private[spark] class MemoryStore
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/19285#discussion_r162840896
--- Diff:
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---
@@ -162,26 +162,29 @@ private[spark] class MemoryStore
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/19285#discussion_r163131383
--- Diff:
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---
@@ -233,17 +235,13 @@ private[spark] class MemoryStore
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/19285#discussion_r163131519
--- Diff:
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---
@@ -162,26 +162,29 @@ private[spark] class MemoryStore
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/19285#discussion_r163131741
--- Diff:
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---
@@ -162,26 +162,29 @@ private[spark] class MemoryStore
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/19285
Thanks for your valuable suggestion, the code has been updated.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/19285#discussion_r163462053
--- Diff:
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---
@@ -233,17 +235,13 @@ private[spark] class MemoryStore
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/19285#discussion_r163551817
--- Diff:
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---
@@ -702,6 +645,83 @@ private[spark] class MemoryStore
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/19285#discussion_r163551992
--- Diff:
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---
@@ -346,85 +350,24 @@ private[spark] class MemoryStore
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/19285#discussion_r163743072
--- Diff:
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---
@@ -346,85 +350,24 @@ private[spark] class MemoryStore
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/19285#discussion_r163768689
--- Diff:
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---
@@ -232,78 +236,93 @@ private[spark] class MemoryStore
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/19285
thanks all.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/20461#discussion_r165246022
--- Diff:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
---
@@ -171,7 +171,9 @@ private void
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/20676
This is for compatibility reasons. The memory management also support
legacy memory management (`StaticMemoryManager`). In `StaticMemoryManager`, the
storage memory and unroll memory is managed
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/20676#discussion_r171115071
--- Diff:
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---
@@ -246,18 +246,18 @@ private[spark] class MemoryStore
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/20676
Yeah, I see that. I'm not sure it's OK to change. But I think we should
follow the interface design, not the underlying impl
GitHub user ConeyLiu opened a pull request:
https://github.com/apache/spark/pull/20844
[SPARK-23707][SQL] Fresh 'initRange' name to avoid method name conflicts
## What changes were proposed in this pull request?
We should call `ctx.freshName` to get the `initRange` to
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/20844
@cloud-fan pls take a look, this is a small change. Thanks a lot.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/20844#discussion_r175315224
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala
---
@@ -396,9 +396,11 @@ case class RangeExec(range
GitHub user ConeyLiu opened a pull request:
https://github.com/apache/spark/pull/18670
[SPARK-21455]RpcFailure should be call on RpcResponseCallback.onFailure
## What changes were proposed in this pull request?
Currently, when there is a `RpcFailure` need be sent back to
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/18670
Hi, @cloud-fan, @zsxwing, would you mind take a look? Thanks a lot.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/18670#discussion_r127994614
--- Diff: core/src/test/scala/org/apache/spark/rpc/RpcEnvSuite.scala ---
@@ -624,7 +624,9 @@ abstract class RpcEnvSuite extends SparkFunSuite with
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/18670#discussion_r128132310
--- Diff: core/src/test/scala/org/apache/spark/rpc/RpcEnvSuite.scala ---
@@ -624,7 +624,9 @@ abstract class RpcEnvSuite extends SparkFunSuite with
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/18670
Thanks for your reviewing. @vanzin @zsxwing I can understand what you mean.
But if we really need a `Throwable` message in client, such as
[NettyRpcEnv.scala#L205](https://github.com/apache/spark
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/18670
Ok, thanks for the answer. I think we can use `ByteBuffer` to solve the
wire compatibility problem, maybe I'm wrong.
I think we should change this, because we also using `onFaliure` to
Github user ConeyLiu closed the pull request at:
https://github.com/apache/spark/pull/18670
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/18670
Ok, thanks a lot for all. close it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
GitHub user ConeyLiu opened a pull request:
https://github.com/apache/spark/pull/19511
[SPARK-22293][SQL] Avoid unnecessary traversal in ResolveReferences
## What changes were proposed in this pull request?
We don't need traverse the children expression to determine wh
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/19511
Hi, @cloud-fan @gatorsmile. Would you mind take a look?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/19511#discussion_r145041400
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -890,32 +890,39 @@ class Analyzer
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/19511
Hi @gatorsmile, if we can combine the two traverse, this should be simplify
the code not complicate. However, this can't get big performance improvement.
And I can close it if this c
Github user ConeyLiu closed the pull request at:
https://github.com/apache/spark/pull/19317
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/19317#discussion_r145294297
--- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala
---
@@ -180,6 +180,56 @@ class PairRDDFunctions[K, V](self: RDD[(K, V
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/19511
OK, close it.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user ConeyLiu closed the pull request at:
https://github.com/apache/spark/pull/19511
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
GitHub user ConeyLiu opened a pull request:
https://github.com/apache/spark/pull/19586
[SPARK-22367][CORE] Separate the serialization of class and object for
iteraor
## What changes were proposed in this pull request?
Becuase they are all the same class for an iterator
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/19586
Hi, @cloud-fan @jiangxb1987 @chenghao-intel. Would you mind take a look?
Thanks a lot.
---
-
To unsubscribe, e-mail: reviews
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/19586#discussion_r147346131
--- Diff: pom.xml ---
@@ -133,7 +133,7 @@
1.6.0
9.3.20.v20170531
3.1.0
-0.8.4
+0.9.2
--- End diff
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/19586
One executor, the configuration as follows:
the script:
```shell
${SPARK_HOME}/bin/spark-submit \
--class com.intel.KryoTest \
--master yarn
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/19586#discussion_r147367241
--- Diff: pom.xml ---
@@ -133,7 +133,7 @@
1.6.0
9.3.20.v20170531
3.1.0
-0.8.4
+0.9.2
--- End diff
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/19586#discussion_r147368002
--- Diff:
core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala ---
@@ -205,11 +205,45 @@ class KryoSerializationStream
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/19586#discussion_r147368368
--- Diff:
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---
@@ -860,9 +876,26 @@ private[storage] class PartiallySerializedBlock[T
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/19586
@srowen Thanks for the reviewing.
What do you meaning here?
> I'm trying to think if there's any case where we intend to support
kryo/java serialized objects fr
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/19586#discussion_r147371400
--- Diff:
core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala ---
@@ -376,7 +382,17 @@ private[spark] class MemoryStore
Github user ConeyLiu commented on a diff in the pull request:
https://github.com/apache/spark/pull/19586#discussion_r147709649
--- Diff:
core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala ---
@@ -205,11 +205,45 @@ class KryoSerializationStream
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/19586
Hi @cloud-fan, thanks for reviewing. There are some errors about
`UnsafeShuffleWrite` need further fixed. I am not familiar with this code, so I
need some time
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/19586
Hi @jerryshao, Thanks for the reminder, it doesn't support it. I'm sorry I
did not take that into account. How about using configuration to determine
whether we should use `Serializ
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/19586
Currently, I use it directly. Maybe this is suitable for some special case
which has same type data, such as ml or else
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/19586
Hi @cloud-fan, for most case the data type should be same. So I think this
optimization is valuable, because it can save the space and cpu resource
considerable. What about setting a flag for the
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/19586
OK, I can understand your concern. There is huge gc problem for K-means
workload, it occupied about 10-20% percent. The source data is cached in
memory, there is even worse performance when the
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/19586
Hi @cloud-fan, @jerryshao. The problem of `writeClass` and `readClass` can
be solved by register the class: Vector, DenseVector, SparseVector. The follow
is the test results:
```scala
val
GitHub user ConeyLiu opened a pull request:
https://github.com/apache/spark/pull/19661
[SPARK-22450][Core][Mllib]safely register class for mllib
## What changes were proposed in this pull request?
There are still some algorithms based on mllib, such as KMeans. For now
Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/19661
#19586
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
1 - 100 of 237 matches
Mail list logo
