GitHub user kellyzly opened a pull request:
https://github.com/apache/spark/pull/19976
[SPARK-22660][BUILD] Use position() and limit() to fix ambiguity issue in
scala-2.12
## What changes were proposed in this pull request?
Missing some changes about limit
Github user kellyzly commented on the issue:
https://github.com/apache/spark/pull/19854
thanks @HyukjinKwon ,@srowen, @viirya 's review
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user kellyzly commented on a diff in the pull request:
https://github.com/apache/spark/pull/19854#discussion_r155168946
--- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java
---
@@ -169,8 +169,10 @@ public static ByteBuffer allocateDirectBuffer(int
Github user kellyzly closed the pull request at:
https://github.com/apache/spark/pull/19853
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user kellyzly commented on a diff in the pull request:
https://github.com/apache/spark/pull/19854#discussion_r155083419
--- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java
---
@@ -169,8 +169,10 @@ public static ByteBuffer allocateDirectBuffer(int
Github user kellyzly commented on a diff in the pull request:
https://github.com/apache/spark/pull/19854#discussion_r155080359
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala
---
@@ -296,7 +296,8 @@ class KafkaTestUtils
GitHub user kellyzly opened a pull request:
https://github.com/apache/spark/pull/19854
SPARK-22660:Use position() and limit() to fix ambiguity issue in scalâ¦
â¦a-2.12 and JDK9
## What changes were proposed in this pull request?
(Please fill in changes proposed
GitHub user kellyzly opened a pull request:
https://github.com/apache/spark/pull/19853
SPARK-22659:remove sun.misc.Cleaner references
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How was this patch tested
Github user kellyzly commented on the pull request:
https://github.com/apache/spark/pull/5307#issuecomment-91146737
@vanzin :
Left some initial comments. I like the idea of using a library for this,
my main concern is about the stability of that library (and who'll
Github user kellyzly commented on a diff in the pull request:
https://github.com/apache/spark/pull/5307#discussion_r28030065
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala ---
@@ -123,12 +130,34 @@ private[spark] class DiskBlockObjectWriter
Github user kellyzly commented on a diff in the pull request:
https://github.com/apache/spark/pull/5307#discussion_r28029243
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala ---
@@ -123,12 +130,34 @@ private[spark] class DiskBlockObjectWriter
GitHub user kellyzly opened a pull request:
https://github.com/apache/spark/pull/5307
[Spark-5682] Add spark encrypted shuffle by using chimera lib
[Chimera](https://github.com/intel-hadoop/chimera) is a project which
strips code related to CryptoInputStream/CryptoOutputStream from
Github user kellyzly commented on the pull request:
https://github.com/apache/spark/pull/4491#issuecomment-85348209
@steveloughran: I have updated code according to your comments:[make
CryptoOutputStream.scala#close
safe](https://github.com/kellyzly/spark/blob
Github user kellyzly commented on the pull request:
https://github.com/apache/spark/pull/4491#issuecomment-84855188
@steveloughran: i don't understand why need make
CryptoOutputStream.scala#close safe. Is there situation when multiple threads
call this function at the same time
Github user kellyzly commented on the pull request:
https://github.com/apache/spark/pull/4491#issuecomment-84893111
@steveloughran : in hadoop, if we need add a native lib path to hadoop
execution path, we need export LD_LIBRARY_PATH
export LD_LIBRARY_PATH=x
in hadoop
Github user kellyzly commented on the pull request:
https://github.com/apache/spark/pull/4491#issuecomment-83426561
@tgravescs,@vanzin,@srowen,@JoshRosen,@CodingCat: I have updated code for
your previous comments. Big change are made in this submission:
1. Delete hadoop2.6
Github user kellyzly commented on a diff in the pull request:
https://github.com/apache/spark/pull/4491#discussion_r26818763
--- Diff: core/src/main/scala/org/apache/spark/crypto/CipherSuite.scala ---
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF
Github user kellyzly commented on the pull request:
https://github.com/apache/spark/pull/4491#issuecomment-83903992
@steveloughran: Thanks for your valuables suggestions.
* I will update latest crypto code in hadoop latest trunk and rewrite it
to scala. I looked into HADOOP
Github user kellyzly commented on a diff in the pull request:
https://github.com/apache/spark/pull/4491#discussion_r26818889
--- Diff:
core/src/main/scala/org/apache/spark/crypto/CryptoInputStream.scala ---
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software
Github user kellyzly commented on a diff in the pull request:
https://github.com/apache/spark/pull/4491#discussion_r26463740
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala ---
@@ -123,12 +133,30 @@ private[spark] class DiskBlockObjectWriter
Github user kellyzly commented on a diff in the pull request:
https://github.com/apache/spark/pull/4491#discussion_r26455600
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala ---
@@ -123,12 +133,30 @@ private[spark] class DiskBlockObjectWriter
Github user kellyzly commented on a diff in the pull request:
https://github.com/apache/spark/pull/4491#discussion_r26455837
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala ---
@@ -123,12 +133,30 @@ private[spark] class DiskBlockObjectWriter
Github user kellyzly commented on the pull request:
https://github.com/apache/spark/pull/4491#issuecomment-77972708
@vanzin: I have renamed the jira name from Reuse hadoop encrypted shuffle
algorithm to enable spark encrypted shuffle to Add encrypted shuffle in
spark. I will update
Github user kellyzly commented on the pull request:
https://github.com/apache/spark/pull/4491#issuecomment-77786363
@srowen,@tgravescs,@vanzin: Encrypted shuffle can make the process of
shuffle more safer. I think it is necessary in spark. Previous design is
reusing hadoop encrypted
Github user kellyzly commented on the pull request:
https://github.com/apache/spark/pull/4491#issuecomment-77786367
@srowen,@tgravescs,@vanzin: Encrypted shuffle can make the process of
shuffle more safer. I think it is necessary in spark. Previous design is
reusing hadoop encrypted
Github user kellyzly commented on the pull request:
https://github.com/apache/spark/pull/4491#issuecomment-77786385
@srowen,@tgravescs,@vanzin: Encrypted shuffle can make the process of
shuffle more safer. I think it is necessary in spark. Previous design is
reusing hadoop encrypted
Github user kellyzly commented on the pull request:
https://github.com/apache/spark/pull/4491#issuecomment-77786348
@srowen,@tgravescs,@vanzin: Encrypted shuffle can make the process of
shuffle more safer. I think it is necessary in spark. Previous design is
reusing hadoop encrypted
Github user kellyzly commented on the pull request:
https://github.com/apache/spark/pull/4491#issuecomment-77786280
@srowen,@tgravescs,@vanzin: Encrypted shuffle can make the process of
shuffle more safer. I think it is necessary in spark. Previous design is
reusing hadoop encrypted
Github user kellyzly commented on the pull request:
https://github.com/apache/spark/pull/4491#issuecomment-77786424
@srowen,@tgravescs,@vanzin: Encrypted shuffle can make the process of
shuffle more safer. I think it is necessary in spark. Previous design is
reusing hadoop encrypted
Github user kellyzly commented on the pull request:
https://github.com/apache/spark/pull/4491#issuecomment-77786413
@srowen,@tgravescs,@vanzin: Encrypted shuffle can make the process of
shuffle more safer. I think it is necessary in spark. Previous design is
reusing hadoop encrypted
Github user kellyzly commented on the pull request:
https://github.com/apache/spark/pull/4491#issuecomment-77786426
@srowen,@tgravescs,@vanzin: Encrypted shuffle can make the process of
shuffle more safer. I think it is necessary in spark. Previous design is
reusing hadoop encrypted
Github user kellyzly commented on the pull request:
https://github.com/apache/spark/pull/4491#issuecomment-77521213
@tgravescs: My teammates and i are trying to stripping classes like
CryptoInputStream.java and CryptoOutputStream.java from package
org.apache.hadoop.crypto in hadoop
Github user kellyzly commented on the pull request:
https://github.com/apache/spark/pull/4491#issuecomment-74218847
@srowen : the unit test fails. The error message like
/home/jenkins/workspace/SparkPullRequestBuilder/core/src/main/scala/org/apache/spark/storage
Github user kellyzly commented on a diff in the pull request:
https://github.com/apache/spark/pull/4491#discussion_r24636611
--- Diff: pom.xml ---
@@ -1600,6 +1600,19 @@
/profile
profile
+ idhadoop-2.6/id
+ properties
Github user kellyzly commented on the pull request:
https://github.com/apache/spark/pull/4491#issuecomment-74207662
@tgravescs @vanzin @srowen @JoshRosen
I was glad to receive all the suggestions from you.
I summarized all of your suggestions:
* CryptoInputStream
Github user kellyzly commented on the pull request:
https://github.com/apache/spark/pull/4491#issuecomment-74207814
@tgravescs
Did you test it with the YARN external shuffle service?
Sorry, i have not tested on this situation and will test it later.
---
If your
Github user kellyzly commented on the pull request:
https://github.com/apache/spark/pull/4491#issuecomment-74217212
@tgravescs: I want ask a questions why
org.apache.hadoop.mapreduce.MRJobConfig which marked
InterfaceAudience.Private can be imported
Github user kellyzly commented on the pull request:
https://github.com/apache/spark/pull/4491#issuecomment-74014097
@JoshRosen: I have updated the patch, added the profile of hadoop 2.4 I
deleted before and add a profile of hadoop2.6.
---
If your project is set up for it, you can
GitHub user kellyzly opened a pull request:
https://github.com/apache/spark/pull/4491
[SPARK-5682] Reuse hadoop encrypted shuffle algorithm to enable spark en...
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kellyzly/spark
39 matches
Mail list logo
