[GitHub] spark issue #17088: [SPARK-19753][CORE] All shuffle files on a host should b...

2017-02-27 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17088 I agree with @mridulm, file fetch failure does not imply the executor down or all the executor of the host down. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request #15864: [SPARK-18420]Fix the compile errors caused by che...

2016-11-12 Thread ConeyLiu
GitHub user ConeyLiu opened a pull request: https://github.com/apache/spark/pull/15864 [SPARK-18420]Fix the compile errors caused by checkstyle ## What changes were proposed in this pull request? Small fix, fix the compile errors caused by checkstyle ## How

[GitHub] spark pull request #15864: [SPARK-18420]Fix the compile errors caused by che...

2016-11-12 Thread ConeyLiu
Github user ConeyLiu closed the pull request at: https://github.com/apache/spark/pull/15864 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request #15865: [SPARK-18420]Fix the compile errors caused by che...

2016-11-12 Thread ConeyLiu
GitHub user ConeyLiu opened a pull request: https://github.com/apache/spark/pull/15865 [SPARK-18420]Fix the compile errors caused by checkstyle ## What changes were proposed in this pull request? Small fix, fix the compile errors caused by checkstyle ## How

[GitHub] spark issue #15865: [SPARK-18420][BUILD] Fix the errors caused by lint check...

2016-11-14 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/15865 @HyukjinKwon Thanks for the review and suggestion, I've updated it. Clear the unused object `hasher` and add suppression rules for the method `finalize` of `NioBufferedFileInputStream`. Please

[GitHub] spark pull request #15865: [SPARK-18420][BUILD] Fix the errors caused by lin...

2016-11-14 Thread ConeyLiu
Github user ConeyLiu commented on a diff in the pull request: https://github.com/apache/spark/pull/15865#discussion_r87799361 --- Diff: dev/checkstyle-suppressions.xml --- @@ -30,6 +30,8 @@ + --- End diff -- @HyukjinKwon Also we could try

[GitHub] spark issue #15865: [SPARK-18420][BUILD] Fix the errors caused by lint check...

2016-11-15 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/15865 @HyukjinKwon I'm sorry for so late reply. I update the code and style followed by your suggestion, that's so useful. And also I change the method of disabling the check around `finalize`. Please

[GitHub] spark pull request #15865: [SPARK-18420][BUILD] Fix the errors caused by lin...

2016-11-15 Thread ConeyLiu
Github user ConeyLiu commented on a diff in the pull request: https://github.com/apache/spark/pull/15865#discussion_r88036045 --- Diff: dev/checkstyle.xml --- @@ -168,5 +182,6 @@ + --- End diff -- @srowen

[GitHub] spark issue #15865: [SPARK-18420]Fix the compile errors caused by checkstyle

2016-11-12 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/15865 @HyukjinKwon I'm sorry for that, I've fixed it, please take a look. Thanks a lot. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark issue #15865: [SPARK-18420][SPARK][BUILD]Fix the compile errors caused...

2016-11-13 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/15865 @srowen Thanks for the review, I've updated it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #15865: [SPARK-18420][BUILD] Fix the errors caused by lint check...

2016-11-15 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/15865 @HyukjinKwon @srowen Thank you very much. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #15865: [SPARK-18420][BUILD] Fix the errors caused by lin...

2016-11-15 Thread ConeyLiu
Github user ConeyLiu commented on a diff in the pull request: https://github.com/apache/spark/pull/15865#discussion_r87996878 --- Diff: dev/checkstyle.xml --- @@ -52,6 +52,20

[GitHub] spark issue #15865: [SPARK-18420][BUILD] Fix the errors caused by lint check...

2016-11-15 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/15865 @srowen hi owen, Can you give me some advice, and if so, what a great thing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #17300: [SPARK-19956][Core]Optimize a location order of blocks w...

2017-03-23 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17300 Ok, thanks a lot. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #17300: [SPARK-19956][Core]Optimize a location order of blocks w...

2017-03-22 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17300 Hi, @jerryshao @mridulm Thanks for your review, I have updated the code. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request #17300: [SPARK-19956][Core]Optimize a location order of b...

2017-03-22 Thread ConeyLiu
Github user ConeyLiu commented on a diff in the pull request: https://github.com/apache/spark/pull/17300#discussion_r107401186 --- Diff: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala --- @@ -497,7 +497,30 @@ class BlockManagerSuite extends SparkFunSuite

[GitHub] spark pull request #17300: [SPARK-19956][Core]Optimize a location order of b...

2017-03-14 Thread ConeyLiu
GitHub user ConeyLiu opened a pull request: https://github.com/apache/spark/pull/17300 [SPARK-19956][Core]Optimize a location order of blocks with topology information ## What changes were proposed in this pull request? When call the method getLocations of BlockManager, we

[GitHub] spark pull request #17243: [SPARK-19901][Core]Clean up the clunky method sig...

2017-03-10 Thread ConeyLiu
Github user ConeyLiu closed the pull request at: https://github.com/apache/spark/pull/17243 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request #17243: [SPARK-19901][Core]Clean up the clunky method sig...

2017-03-10 Thread ConeyLiu
GitHub user ConeyLiu reopened a pull request: https://github.com/apache/spark/pull/17243 [SPARK-19901][Core]Clean up the clunky method signature of acquireMemory ## What changes were proposed in this pull request? Clean up the clunky method signature of acquireMemory

[GitHub] spark issue #17243: [SPARK-19901][Core]Clean up the clunky method signature ...

2017-03-10 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17243 Sorry for the mistake, reopen it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #17243: [SPARK-19901][Core]Clean up the clunky method sig...

2017-03-10 Thread ConeyLiu
GitHub user ConeyLiu opened a pull request: https://github.com/apache/spark/pull/17243 [SPARK-19901][Core]Clean up the clunky method signature of acquireMemory ## What changes were proposed in this pull request? Clean up the clunky method signature of acquireMemory

[GitHub] spark issue #17243: [SPARK-19901][Core]Clean up the clunky method signature ...

2017-03-10 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17243 Hi, @srowen @HyukjinKwon, thanks for the review. This patch stems from the `TODO` list, which is used to streamline the method signature, make it more readable. If you think there is no essential

[GitHub] spark pull request #17243: [SPARK-19901][Core]Clean up the clunky method sig...

2017-03-10 Thread ConeyLiu
Github user ConeyLiu closed the pull request at: https://github.com/apache/spark/pull/17243 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #17243: [SPARK-19901][Core]Clean up the clunky method signature ...

2017-03-10 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17243 ok, I will close it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark pull request #18830: [SPARK-21621][Core] Reset numRecordsWritten after...

2017-08-03 Thread ConeyLiu
GitHub user ConeyLiu opened a pull request: https://github.com/apache/spark/pull/18830 [SPARK-21621][Core] Reset numRecordsWritten after DiskBlockObjectWriter.commitAndGet called ## What changes were proposed in this pull request? We should reset numRecordsWritten to zero

[GitHub] spark issue #18830: [SPARK-21621][Core] Reset numRecordsWritten after DiskBl...

2017-08-03 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/18830 @cloud-fan @vanzin Would you mind take a look? Thanks a lot. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #18830: [SPARK-21621][Core] Reset numRecordsWritten after DiskBl...

2017-08-03 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/18830 You can see here [L208](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/DiskBlockObjectWriter.scala#L208), when we called 'revertPartialWritesAndClose

[GitHub] spark pull request #18670: [SPARK-21455]RpcFailure should be call on RpcResp...

2017-07-18 Thread ConeyLiu
GitHub user ConeyLiu opened a pull request: https://github.com/apache/spark/pull/18670 [SPARK-21455]RpcFailure should be call on RpcResponseCallback.onFailure ## What changes were proposed in this pull request? Currently, when there is a `RpcFailure` need be sent back

[GitHub] spark issue #18670: [SPARK-21455]RpcFailure should be call on RpcResponseCal...

2017-07-18 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/18670 Hi, @cloud-fan, @zsxwing, would you mind take a look? Thanks a lot. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request #18670: [SPARK-21455][CORE]RpcFailure should be call on R...

2017-07-18 Thread ConeyLiu
Github user ConeyLiu commented on a diff in the pull request: https://github.com/apache/spark/pull/18670#discussion_r127994614 --- Diff: core/src/test/scala/org/apache/spark/rpc/RpcEnvSuite.scala --- @@ -624,7 +624,9 @@ abstract class RpcEnvSuite extends SparkFunSuite

[GitHub] spark issue #18670: [SPARK-21455][CORE]RpcFailure should be call on RpcRespo...

2017-07-18 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/18670 Thanks for your reviewing. @vanzin @zsxwing I can understand what you mean. But if we really need a `Throwable` message in client, such as [NettyRpcEnv.scala#L205](https://github.com/apache/spark

[GitHub] spark pull request #18670: [SPARK-21455][CORE]RpcFailure should be call on R...

2017-07-18 Thread ConeyLiu
Github user ConeyLiu commented on a diff in the pull request: https://github.com/apache/spark/pull/18670#discussion_r128132310 --- Diff: core/src/test/scala/org/apache/spark/rpc/RpcEnvSuite.scala --- @@ -624,7 +624,9 @@ abstract class RpcEnvSuite extends SparkFunSuite

[GitHub] spark issue #18670: [SPARK-21455][CORE]RpcFailure should be call on RpcRespo...

2017-07-18 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/18670 Ok, thanks for the answer. I think we can use `ByteBuffer` to solve the wire compatibility problem, maybe I'm wrong. I think we should change this, because we also using `onFaliure` to send

[GitHub] spark pull request #18670: [SPARK-21455][CORE]RpcFailure should be call on R...

2017-07-19 Thread ConeyLiu
Github user ConeyLiu closed the pull request at: https://github.com/apache/spark/pull/18670 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #18670: [SPARK-21455][CORE]RpcFailure should be call on RpcRespo...

2017-07-19 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/18670 Ok, thanks a lot for all. close it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18830: [SPARK-21621][Core] Reset numRecordsWritten after DiskBl...

2017-08-04 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/18830 Thanks for reviewing. Hi @jiangxb1987, seems the test didn't triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #18830: [SPARK-21621][Core] Reset numRecordsWritten after DiskBl...

2017-08-07 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/18830 Thank you all. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #17936: [SPARK-20638][Core][WIP]Optimize the CartesianRDD to red...

2017-05-11 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17936 The cluster test result. The `RDD.cartesian` is used in Spark mllib ALS algorithm, and compared with the latest spark master branch. Environments: Spark on Yarn with 9 executors(10 cores

[GitHub] spark issue #17936: [SPARK-20638][Core]Optimize the CartesianRDD to reduce r...

2017-05-14 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17936 Yeah, I can test it. You see, the `ALS` is an pratical use case. So, choose it as a test case more convincing. And I also want to see the improvement of this `pr` even after merged #17742

[GitHub] spark issue #17936: [SPARK-20638][Core]Optimize the CartesianRDD to reduce r...

2017-05-15 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17936 Hi, @jtengyp the test Results as follow: | Improved Branch | Master Branch | Percentage of ascension | | --| -- | -- | | 15.877s | 2827.373s | 178x | | 16.781s

[GitHub] spark issue #17890: [MINOR][BUILD] Fix lint-java breaks.

2017-05-10 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17890 Thansk @dongjoon-hyun. Hi @srowen, code is updated, because the `Tigger` location is changed after your pr. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark issue #17936: [SPARK-20638][Core][WIP]Optimize the CartesianRDD to red...

2017-05-10 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17936 hi @jerryshao,thanks for your review. In #17898,there is a potential buffer to cache the data,so we should control the groupsize very careful. Because for small size,it need fetch more

[GitHub] spark issue #17936: [SPARK-20638][Core][WIP]Optimize the CartesianRDD to red...

2017-05-10 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17936 Cool, you see the `iterator` operation can be divided in two cases: 1. get the block from local, this case is very good. 2. get the block from remote. - The block is cached

[GitHub] spark issue #17936: [SPARK-20638][Core][WIP]Optimize the CartesianRDD to red...

2017-05-10 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17936 A cluster version of the comparison results, I will be given later. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #17936: [SPARK-20638][Core][WIP]Optimize the CartesianRDD to red...

2017-05-10 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17936 Hi @viirya, can you help to review this? I thinks you are familiar with this, because you dad tried to solve it before. And also ping @srowen , @mridulm, @jerryshao. --- If your project

[GitHub] spark pull request #17936: [SPARK-20638][Core][WIP]Optimize the CartesianRDD...

2017-05-10 Thread ConeyLiu
GitHub user ConeyLiu opened a pull request: https://github.com/apache/spark/pull/17936 [SPARK-20638][Core][WIP]Optimize the CartesianRDD to reduce repeatedly data fetching ## What changes were proposed in this pull request? This path aims to solve the poor performance

[GitHub] spark issue #17890: [MINOR][BUILD] Fix lint-java breaks.

2017-05-10 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17890 thanks a lot. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #17890: [MINOR][BUILD] Fix lint-java breaks.

2017-05-09 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17890 @srowen Thanks a lot. Thanks OK, It is really unreasonable. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request #18213: [SPARK-20996][YARN] Better handling AM reattempt ...

2017-06-08 Thread ConeyLiu
Github user ConeyLiu commented on a diff in the pull request: https://github.com/apache/spark/pull/18213#discussion_r121028779 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala --- @@ -744,9 +746,23 @@ object ApplicationMaster

[GitHub] spark issue #18350: [MINOR] Fix some typo of the document

2017-06-19 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/18350 thanks @srowen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #17936: [SPARK-20638][Core]Optimize the CartesianRDD to reduce r...

2017-05-19 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17936 `Broadcast` should first fetch the all block to driver, and cached in the local, then the executor fetch it from the driver. I think it's really time consuming. --- If your project is set up

[GitHub] spark issue #17936: [SPARK-20638][Core]Optimize the CartesianRDD to reduce r...

2017-05-19 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17936 Sorry for the mistake, this test result should be the cached situation: | --| -- | -- | | 15.877s | 2827.373s | 178x | | 16.781s | 2809.502s | 167x | | 16.320s

[GitHub] spark issue #17936: [SPARK-20638][Core]Optimize the CartesianRDD to reduce r...

2017-05-19 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17936 OK, I'll add it. From the test data, performance is still very obvious. Mainly from the network and disk overhead. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request #17936: [SPARK-20638][Core]Optimize the CartesianRDD to r...

2017-05-19 Thread ConeyLiu
Github user ConeyLiu commented on a diff in the pull request: https://github.com/apache/spark/pull/17936#discussion_r117432268 --- Diff: core/src/main/scala/org/apache/spark/rdd/CartesianRDD.scala --- @@ -71,9 +72,92 @@ class CartesianRDD[T: ClassTag, U: ClassTag

[GitHub] spark pull request #17936: [SPARK-20638][Core]Optimize the CartesianRDD to r...

2017-05-19 Thread ConeyLiu
Github user ConeyLiu commented on a diff in the pull request: https://github.com/apache/spark/pull/17936#discussion_r117427240 --- Diff: core/src/main/scala/org/apache/spark/rdd/CartesianRDD.scala --- @@ -71,9 +72,92 @@ class CartesianRDD[T: ClassTag, U: ClassTag

[GitHub] spark issue #17936: [SPARK-20638][Core]Optimize the CartesianRDD to reduce r...

2017-05-19 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17936 Yeah, I think I can do the performance comparison. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #17936: [SPARK-20638][Core]Optimize the CartesianRDD to reduce r...

2017-05-19 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17936 I did not directly test this situation. But I have test the this pr compared with latest `ALS`(after merge #17742 ). In `ALS`, the both RDDs are cached, and also grouped the iterator

[GitHub] spark pull request #18090: [SPARK-20250][Core]Improper OOM error when a task...

2017-05-24 Thread ConeyLiu
GitHub user ConeyLiu opened a pull request: https://github.com/apache/spark/pull/18090 [SPARK-20250][Core]Improper OOM error when a task been killed while spilling data ## What changes were proposed in this pull request? Currently, when a task is calling spill

[GitHub] spark issue #18090: [SPARK-20250][Core]Improper OOM error when a task been k...

2017-05-24 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/18090 Hi @cloud-fan, @srowen, Can you help take a look ? Thanks a lot. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request #18099: [SPARK-18406][CORE][Backport-2.1] Race between en...

2017-05-24 Thread ConeyLiu
Github user ConeyLiu commented on a diff in the pull request: https://github.com/apache/spark/pull/18099#discussion_r118396798 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -454,14 +454,20 @@ private[spark] class BlockManager( case

[GitHub] spark pull request #18099: [SPARK-18406][CORE][Backport-2.1] Race between en...

2017-05-24 Thread ConeyLiu
Github user ConeyLiu commented on a diff in the pull request: https://github.com/apache/spark/pull/18099#discussion_r118394510 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -454,14 +454,20 @@ private[spark] class BlockManager( case

[GitHub] spark issue #18090: [SPARK-20250][Core]Improper OOM error when a task been k...

2017-05-25 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/18090 thanks @cloud-fan @viirya for review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #17936: [SPARK-20638][Core]Optimize the CartesianRDD to r...

2017-05-18 Thread ConeyLiu
Github user ConeyLiu commented on a diff in the pull request: https://github.com/apache/spark/pull/17936#discussion_r117393634 --- Diff: core/src/test/scala/org/apache/spark/metrics/InputOutputMetricsSuite.scala --- @@ -198,8 +198,12 @@ class InputOutputMetricsSuite extends

[GitHub] spark issue #17936: [SPARK-20638][Core]Optimize the CartesianRDD to reduce r...

2017-05-18 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17936 @srowen Sorry for the late reply. I updated the code. Because we should reduce times of the remotely fetch, the second partition should be cached in locally. There are two ways, first cached

[GitHub] spark pull request #17936: [SPARK-20638][Core]Optimize the CartesianRDD to r...

2017-05-18 Thread ConeyLiu
Github user ConeyLiu commented on a diff in the pull request: https://github.com/apache/spark/pull/17936#discussion_r117393923 --- Diff: core/src/test/scala/org/apache/spark/metrics/InputOutputMetricsSuite.scala --- @@ -198,8 +198,12 @@ class InputOutputMetricsSuite extends

[GitHub] spark issue #17936: [SPARK-20638][Core]Optimize the CartesianRDD to reduce r...

2017-05-30 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17936 Hi @rxin, would you mind take a look? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18284: [SPARK-21072][SQL] `TreeNode.mapChildren` should only ap...

2017-06-14 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/18284 @cloud-fan thanks for reviewing, code has updated. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #18284: [SPARK-21072][SQL] `TreeNode.mapChildren` should only ap...

2017-06-15 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/18284 I can't find the test failed message, seems like unrelated errors. Can you trigger jenkins test again, because I'm not in the whitelist. Thanks a lot. --- If your project is set up for it, you

[GitHub] spark pull request #18284: [SPARK-21072][SQL] `TreeNode.mapChildren` should ...

2017-06-15 Thread ConeyLiu
Github user ConeyLiu commented on a diff in the pull request: https://github.com/apache/spark/pull/18284#discussion_r122139485 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/trees/TreeNodeSuite.scala --- @@ -146,6 +154,17 @@ class TreeNodeSuite extends

[GitHub] spark pull request #18284: [SPARK-21072][SQL] TreeNode.mapChildren should on...

2017-06-15 Thread ConeyLiu
Github user ConeyLiu commented on a diff in the pull request: https://github.com/apache/spark/pull/18284#discussion_r122205853 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/trees/TreeNodeSuite.scala --- @@ -61,6 +61,14 @@ case class ExpressionInMap(map: Map

[GitHub] spark issue #18284: [SPARK-21072][SQL] `TreeNode.mapChildren` should only ap...

2017-06-14 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/18284 Thanks for reviewing, I'll add it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18284: [SPARK-21072][SQL] TreeNode.mapChildren should only appl...

2017-06-15 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/18284 thanks everyone for reviewing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #18350: [MINOR] Fix some typo of the document

2017-06-18 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/18350 Hi @srowen, would you mind take a look? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #18350: [MINOR] Fix some typo of the document

2017-06-18 Thread ConeyLiu
GitHub user ConeyLiu opened a pull request: https://github.com/apache/spark/pull/18350 [MINOR] Fix some typo of the document ## What changes were proposed in this pull request? Fix some typo of the document. ## How was this patch tested? Existing tests

[GitHub] spark pull request #18284: [SPARK-21072] `TreeNode.mapChildren` should only ...

2017-06-12 Thread ConeyLiu
GitHub user ConeyLiu opened a pull request: https://github.com/apache/spark/pull/18284 [SPARK-21072] `TreeNode.mapChildren` should only apply to the children node. ## What changes were proposed in this pull request? Just as the function name and comments

[GitHub] spark issue #18284: [SPARK-21072] `TreeNode.mapChildren` should only apply t...

2017-06-12 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/18284 @cloud-fan Would you mind take a look? Thanks a lot. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #17300: [SPARK-19956][Core]Optimize a location order of blocks w...

2017-05-04 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17300 Hi, @cloud-fan @zsxwing Can you take a look? Thanks a lot. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request #17859: [SPARK-20595][Deploy]Parse the 'SPARK_EXECUTOR_IN...

2017-05-04 Thread ConeyLiu
GitHub user ConeyLiu opened a pull request: https://github.com/apache/spark/pull/17859 [SPARK-20595][Deploy]Parse the 'SPARK_EXECUTOR_INSTANCES' into the parsed arguments ## What changes were proposed in this pull request? Currently, when we set the parameter

[GitHub] spark issue #17859: [SPARK-20595][Deploy]Parse the 'SPARK_EXECUTOR_INSTANCES...

2017-05-04 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17859 @srowen @HyukjinKwon Can you take a look, thanks a lot. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request #17890: [MINOR][BUILD] Fix lint-java breaks.

2017-05-07 Thread ConeyLiu
GitHub user ConeyLiu opened a pull request: https://github.com/apache/spark/pull/17890 [MINOR][BUILD] Fix lint-java breaks. ## What changes were proposed in this pull request? This PR proposes to fix the lint-breaks as below: ``` [ERROR] src/main/java/org/apache

[GitHub] spark pull request #17890: [MINOR][BUILD] Fix lint-java breaks.

2017-05-07 Thread ConeyLiu
Github user ConeyLiu commented on a diff in the pull request: https://github.com/apache/spark/pull/17890#discussion_r115146202 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala --- @@ -379,7 +379,7 @@ final class DataStreamWriter[T] private

[GitHub] spark pull request #17889: [MINOR][BUILD] Fix lint-java breaks.

2017-05-07 Thread ConeyLiu
Github user ConeyLiu closed the pull request at: https://github.com/apache/spark/pull/17889 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request #17889: [MINOR][BUILD] Fix lint-java breaks.

2017-05-07 Thread ConeyLiu
GitHub user ConeyLiu opened a pull request: https://github.com/apache/spark/pull/17889 [MINOR][BUILD] Fix lint-java breaks. ## What changes were proposed in this pull request? This PR proposes to fix the lint-breaks as below: ``` [ERROR] src/main/java/org/apache

[GitHub] spark pull request #17890: [MINOR][BUILD] Fix lint-java breaks.

2017-05-08 Thread ConeyLiu
Github user ConeyLiu commented on a diff in the pull request: https://github.com/apache/spark/pull/17890#discussion_r115229257 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala --- @@ -379,7 +379,7 @@ final class DataStreamWriter[T] private

[GitHub] spark issue #17300: [SPARK-19956][Core]Optimize a location order of blocks w...

2017-05-08 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17300 Thanks @cloud-fan @mridulm @gatorsmile --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #17898: Optimize the CartesianRDD to reduce repeatedly da...

2017-05-08 Thread ConeyLiu
Github user ConeyLiu commented on a diff in the pull request: https://github.com/apache/spark/pull/17898#discussion_r115232250 --- Diff: core/src/main/scala/org/apache/spark/rdd/CartesianRDD.scala --- @@ -72,8 +72,10 @@ class CartesianRDD[T: ClassTag, U: ClassTag

[GitHub] spark issue #17859: [SPARK-20595][Deploy]Parse the 'SPARK_EXECUTOR_INSTANCES...

2017-05-04 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17859 Hi @srowen, the follow is the test result, the submit script: `./spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode client --verbose ../examples/target/scala-2.11

[GitHub] spark pull request #17300: [SPARK-19956][Core]Optimize a location order of b...

2017-05-04 Thread ConeyLiu
Github user ConeyLiu commented on a diff in the pull request: https://github.com/apache/spark/pull/17300#discussion_r114935050 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -555,12 +555,15 @@ private[spark] class BlockManager

[GitHub] spark issue #17300: [SPARK-19956][Core]Optimize a location order of blocks w...

2017-05-04 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17300 Thanks both of you for review, I have addressed the comments and modified the test case. Please help calling jenkins for test, because I can't trigger that. Thanks again. --- If your project

[GitHub] spark pull request #17881: [SPARK-20621][deploy]Delete deprecated config par...

2017-05-06 Thread ConeyLiu
GitHub user ConeyLiu opened a pull request: https://github.com/apache/spark/pull/17881 [SPARK-20621][deploy]Delete deprecated config parameter in 'spark-env.sh' ## What changes were proposed in this pull request? Currently, `spark.executor.instances` is deprecated in `spark

[GitHub] spark issue #17881: [SPARK-20621][deploy]Delete deprecated config parameter ...

2017-05-06 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17881 Thanks @srowen. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #17881: [SPARK-20621][deploy]Delete deprecated config parameter ...

2017-05-06 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17881 Hi @vanzin, just as talked, we should remote the `spark.executor.instances` from `spark-env.sh`. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark issue #17300: [SPARK-19956][Core]Optimize a location order of blocks w...

2017-05-06 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17300 Thanks for your review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #17859: [SPARK-20595][Deploy]Parse the 'SPARK_EXECUTOR_INSTANCES...

2017-05-04 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17859 Hi @vanzin, can you help to take a look? The affect of `spark.executor.instances` changed after [SPARK-9092](https://github.com/apache/spark/pull/7657), but the comments in the config template

[GitHub] spark issue #17859: [SPARK-20595][Deploy]Parse the 'SPARK_EXECUTOR_INSTANCES...

2017-05-04 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17859 It only affect the YARN mode, just see the follow code: ``` OptionAssigner(args.numExecutors, YARN, ALL_DEPLOY_MODES, sysProp = "spark.executor.instances"), ``` Bu

[GitHub] spark issue #17859: [SPARK-20595][Deploy]Parse the 'SPARK_EXECUTOR_INSTANCES...

2017-05-04 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17859 Ok, I will open another pr to remove it. Thanks a lot both of you. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request #17859: [SPARK-20595][Deploy]Parse the 'SPARK_EXECUTOR_IN...

2017-05-04 Thread ConeyLiu
Github user ConeyLiu closed the pull request at: https://github.com/apache/spark/pull/17859 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #17859: [SPARK-20595][Deploy]Parse the 'SPARK_EXECUTOR_INSTANCES...

2017-05-04 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17859 @vanzin Thanks a lot for you review. Do we need remove the comments from template config? It doesn't work anymore in current version. --- If your project is set up for it, you can reply

[GitHub] spark issue #17300: [SPARK-19956][Core]Optimize a location order of blocks w...

2017-05-04 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17300 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #17890: [MINOR][BUILD] Fix lint-java breaks.

2017-05-08 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17890 Hi @srowen, the code have updated, and this is the test result of [Travis-CI](https://travis-ci.org/ConeyLiu/spark/builds/229970944). The above error seems not relately, could you take a look

  1   2   >