[GitHub] spark pull request: [SPARK-10180] [SQL] JDBC datasource are not pr...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/8743


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10180] [SQL] JDBC datasource are not pr...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/8743#issuecomment-168372683
  
Thanks - I've merged this.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: size RollingPolicy shouldRollover loginfo shou...

[ouyangshourui]

Github user ouyangshourui commented on the pull request:

https://github.com/apache/spark/pull/10549#issuecomment-168376381
  
thank you ,srowen .you can  close this PR. I will argue this change is 
necessary in the JIRA in SPARK-12601.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: size RollingPolicy shouldRollover loginfo shou...

[ouyangshourui]

Github user ouyangshourui closed the pull request at:

https://github.com/apache/spark/pull/10549


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Fixed bug in setMinPartitions

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/10546#issuecomment-168381491
  
@datafarmer go ahead and update the title here and consider updating the PR 
itself per above.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12421][SQL] Prevent Internal/External r...

[hvanhovell]

GitHub user hvanhovell opened a pull request:

https://github.com/apache/spark/pull/10553

[SPARK-12421][SQL] Prevent Internal/External row from exposing state.

It is currently possible to change the values of the supposedly immutable 
```GenericRow``` and ```GenericInternalRow``` classes. This is caused by the 
fact that scala's ArrayOps ```toArray``` will return the backing array instead 
of a copy. This PR fixes this problem.

This PR was inspired by https://github.com/apache/spark/pull/10374 by @apo1.

cc @apo1 @sarutak @marmbrus @cloud-fan @nongli (everyone in the previous 
conversation).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hvanhovell/spark SPARK-12421

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10553.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10553


commit 632b5dc202cce7ec7e2826018e06d120a4cd33d1
Author: Herman van Hovell 
Date:   2016-01-02T11:34:05Z

Prevent leaking state from internal/external row.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: size RollingPolicy shouldRollover loginfo shou...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/10549#issuecomment-168375947
  
Please read 
https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark  Do you 
mind closing this PR or fixing it? though I think you first need to argue this 
change is necessary in the JIRA, and that means fixing the JIRA.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12603] [MLlib] PySpark MLlib GaussianMi...

[yanboliang]

GitHub user yanboliang opened a pull request:

https://github.com/apache/spark/pull/10552

[SPARK-12603] [MLlib] PySpark MLlib GaussianMixtureModel should support 
single instance predict/predictSoft

PySpark MLlib GaussianMixtureModel should support single instance 
predict/predictSoft just like Scala one.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yanboliang/spark spark-12603

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10552.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10552


commit 8371f342fed8cc688e2e97b521890c46de19c68f
Author: Yanbo Liang 
Date:   2016-01-02T08:30:14Z

MLlib GaussianMixtureModel should support single instance 
predict/predictSoft

commit cbda57e9cd3c0a9c1900a81f24b716a2d335c476
Author: Yanbo Liang 
Date:   2016-01-02T09:47:35Z

Fix python3 compatibility issue




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12603] [MLlib] PySpark MLlib GaussianMi...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10552#issuecomment-168378126
  
**[Test build #48580 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48580/consoleFull)**
 for PR 10552 at commit 
[`cbda57e`](https://github.com/apache/spark/commit/cbda57e9cd3c0a9c1900a81f24b716a2d335c476).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12481] [CORE] [STREAMING] [SQL] Remove ...

[srowen]

Github user srowen closed the pull request at:

https://github.com/apache/spark/pull/10446


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12598] bug in setMinPartitions

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/10546#issuecomment-168390342
  
@datafarmer I've just seconds ago merged a change that replaces these 
deprecated calls, since we can assume Hadoop 2.2+ now. Yes, isDirectory is 
correct now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12481] [CORE] [STREAMING] [SQL] Remove ...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/10446#issuecomment-168390287
  
Merged to master


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12599][MLlib][SQL] Remove the use of ca...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10547#issuecomment-168372795
  
**[Test build #2296 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2296/consoleFull)**
 for PR 10547 at commit 
[`f6eb35a`](https://github.com/apache/spark/commit/f6eb35a74c5a51951a6b34dcef5793b58e7d00bf).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12603] [MLlib] PySpark MLlib GaussianMi...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10552#issuecomment-168379412
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12513] [Streaming] SocketReceiver hang ...

[guoxu1231]

Github user guoxu1231 commented on a diff in the pull request:

https://github.com/apache/spark/pull/10464#discussion_r48684054
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/dstream/SocketInputDStream.scala
 ---
@@ -51,29 +51,44 @@ class SocketReceiver[T: ClassTag](
 storageLevel: StorageLevel
   ) extends Receiver[T](storageLevel) with Logging {
 
+  private var socket: Socket = _
+
   def onStart() {
-// Start the thread that receives data over a connection
-new Thread("Socket Receiver") {
-  setDaemon(true)
-  override def run() { receive() }
-}.start()
+try {
+  logInfo(s"Connecting to $host:$port")
+  socket = new Socket(host, port)
+  logInfo(s"Connected to $host:$port")
+} catch {
+  case NonFatal(e) =>
+restart(s"Error connecting to $host:$port", e)
+}
+
+if (socket != null && socket.isConnected) {
+  // Start the thread that receives data over a connection
+  new Thread("Socket Receiver") {
+setDaemon(true)
+override def run() { receive() }
+  }.start()
+}
   }
 
   def onStop() {
-// There is nothing much to do as the thread calling receive()
-// is designed to stop by itself isStopped() returns false
+//in case restart thread close it twice
+synchronized {
+  if (socket != null) {
+socket.close()
+socket = null
+logInfo(s"Closed socket to $host:$port")
+  }
+}
   }
 
   /** Create a socket connection and receive data until receiver is 
stopped */
   def receive() {
-var socket: Socket = null
 try {
-  logInfo("Connecting to " + host + ":" + port)
-  socket = new Socket(host, port)
-  logInfo("Connected to " + host + ":" + port)
-  val iterator = bytesToObjects(socket.getInputStream())
+  val iterator = bytesToObjects(socket.getInputStream)
--- End diff --

Done.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9026] Refactor SimpleFutureAction.onCom...

[jaceklaskowski]

Github user jaceklaskowski commented on the pull request:

https://github.com/apache/spark/pull/7385#issuecomment-168384426
  
@zsxwing @JoshRosen Does the comment need attention since the pr is closed, 
 
https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceiverTracker.scala#L438?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12421][SQL] Prevent Internal/External r...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10553#issuecomment-168390480
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48581/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12421][SQL] Prevent Internal/External r...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10553#issuecomment-168390446
  
**[Test build #48581 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48581/consoleFull)**
 for PR 10553 at commit 
[`632b5dc`](https://github.com/apache/spark/commit/632b5dc202cce7ec7e2826018e06d120a4cd33d1).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12604] [CORE] Java count(AprroxDistinct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10554#issuecomment-168391002
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12604] [CORE] Java count(AprroxDistinct...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10554#issuecomment-168391003
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48583/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Minor][SQL] Replace logger name

[viirya]

GitHub user viirya opened a pull request:

https://github.com/apache/spark/pull/10555

[Minor][SQL] Replace logger name

This is very minor update. @rxin Please see if it is proper for you. If 
not, I will close it. Thanks.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/viirya/spark-1 minor-parser-logger

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10555.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10555


commit 2132171c7e0137bab78082857c40a431de4ab039
Author: Liang-Chi Hsieh 
Date:   2016-01-02T14:44:05Z

Minor: replace logger name.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Minor][SQL] Replace logger name

[hvanhovell]

Github user hvanhovell commented on the pull request:

https://github.com/apache/spark/pull/10555#issuecomment-168396939
  
LGTM - missed that one.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12602] [SQL] Join Reordering: Pushing I...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10551#issuecomment-168372904
  
**[Test build #48577 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48577/consoleFull)**
 for PR 10551 at commit 
[`5f67a74`](https://github.com/apache/spark/commit/5f67a7476365828a5eb920b3af3ddad312799509).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12602] [SQL] Join Reordering: Pushing I...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10551#issuecomment-168372938
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48577/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12602] [SQL] Join Reordering: Pushing I...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10551#issuecomment-168372937
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12599][MLlib][SQL] Remove the use of ca...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10547#issuecomment-168375242
  
**[Test build #2296 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2296/consoleFull)**
 for PR 10547 at commit 
[`f6eb35a`](https://github.com/apache/spark/commit/f6eb35a74c5a51951a6b34dcef5793b58e7d00bf).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12598] bug in setMinPartitions

[datafarmer]

Github user datafarmer commented on the pull request:

https://github.com/apache/spark/pull/10546#issuecomment-168389034
  
@srowen I'll update the PR per your changes.  BTW, the FileStatus method 
isDir is deprecated.  Should I change it to isDirectory, or is that something 
for another PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12421][SQL] Prevent Internal/External r...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10553#issuecomment-168392086
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12421][SQL] Prevent Internal/External r...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10553#issuecomment-168391748
  
**[Test build #48582 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48582/consoleFull)**
 for PR 10553 at commit 
[`daa123b`](https://github.com/apache/spark/commit/daa123bc76c16bc3331c140c6c4656f02af89a53).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12600][SQL] Remove deprecated methods i...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/10548#issuecomment-168375899
  
I generally favor removing deprecated methods in a 2.x release. I wanted to 
ask what your (and others') opinions are on it, since there's some argument for 
possibly keeping some very-commonly-used deprecated methods across major 
releases, and I know you had questioned some other changes on grounds of 
compatibility even for 2.x. I'm in favor of a change like this, just wondering 
out loud whether anyone has objection to taking it further and removing more, 
or all, deprecated methods?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12513] [Streaming] SocketReceiver hang ...

[guoxu1231]

Github user guoxu1231 commented on a diff in the pull request:

https://github.com/apache/spark/pull/10464#discussion_r48684056
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/dstream/SocketInputDStream.scala
 ---
@@ -51,29 +51,44 @@ class SocketReceiver[T: ClassTag](
 storageLevel: StorageLevel
   ) extends Receiver[T](storageLevel) with Logging {
 
+  private var socket: Socket = _
+
   def onStart() {
-// Start the thread that receives data over a connection
-new Thread("Socket Receiver") {
-  setDaemon(true)
-  override def run() { receive() }
-}.start()
+try {
+  logInfo(s"Connecting to $host:$port")
+  socket = new Socket(host, port)
+  logInfo(s"Connected to $host:$port")
+} catch {
+  case NonFatal(e) =>
--- End diff --

Revert to catch only ConnectionException.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12592][SQL][WIP] Converts resolved logi...

[hvanhovell]

Github user hvanhovell commented on the pull request:

https://github.com/apache/spark/pull/10541#issuecomment-168384367
  
@liancheng this looks cool!

I was wondering why we are bound to SQL? Is this because of Hive? I was 
thinking of the following, we could also store the logical plan's json 
representation. This should alot easier to (de)serialize. Could we store that 
in the Hive metadata store?

Another idea I was having. If a view is defined in HQL, we could also store 
that in some way with the query execution. This saves us a 
serialization/deserialization trip, and allows the user to recognize his own 
query.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12421][SQL] Prevent Internal/External r...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10553#issuecomment-168384349
  
**[Test build #48582 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48582/consoleFull)**
 for PR 10553 at commit 
[`daa123b`](https://github.com/apache/spark/commit/daa123bc76c16bc3331c140c6c4656f02af89a53).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12537] [SQL] Add option to accept quoti...

[Cazen]

Github user Cazen commented on a diff in the pull request:

https://github.com/apache/spark/pull/10497#discussion_r48684498
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JSONOptions.scala
 ---
@@ -59,6 +62,8 @@ object JSONOptions {
 allowNumericLeadingZeros =
   
parameters.get("allowNumericLeadingZeros").map(_.toBoolean).getOrElse(false),
 allowNonNumericNumbers =
-  
parameters.get("allowNonNumericNumbers").map(_.toBoolean).getOrElse(true)
+  
parameters.get("allowNonNumericNumbers").map(_.toBoolean).getOrElse(true),
+allowBackslashEscapingAnyCharacter =
+  
parameters.get("allowBackslashEscapingAnyCharacter").map(_.toBoolean).getOrElse(true)
--- End diff --

Happy New Year Owen!

At first, I've created PR with false default.

But Xin advised to me that "I'd actually change the default value to true" 
in previous comment so I've changed.

If you think it doesn't make sense, please call me again. I will change 
that immediately

Thank you


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12604] [CORE] Java count(AprroxDistinct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10554#issuecomment-168393617
  
**[Test build #2297 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2297/consoleFull)**
 for PR 10554 at commit 
[`1a27421`](https://github.com/apache/spark/commit/1a27421aea247420b4aed597f8cc19767d11b6f0).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12597] [ML] Use udf to replace callUDF ...

[yanboliang]

Github user yanboliang commented on the pull request:

https://github.com/apache/spark/pull/10544#issuecomment-168376268
  
@rxin ```transformFunc``` is not a public API, but I think your PR is more 
concise and I will close my PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12453][Streaming] Remove explicit depen...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/10492#issuecomment-168381598
  
@JoshRosen are you OK with the reasoning above?
@BrianLondon sorry but the style checker doesn't like the white space on 
the blank line you added to trigger tests :(. If that's removed I can merge 
this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12421][SQL] Prevent Internal/External r...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10553#issuecomment-168383990
  
**[Test build #48581 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48581/consoleFull)**
 for PR 10553 at commit 
[`632b5dc`](https://github.com/apache/spark/commit/632b5dc202cce7ec7e2826018e06d120a4cd33d1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12604] [CORE] Java count(AprroxDistinct...

[srowen]

GitHub user srowen opened a pull request:

https://github.com/apache/spark/pull/10554

[SPARK-12604] [CORE] Java count(AprroxDistinct)ByKey methods return Scala 
Long not Java

Change Java countByKey, countApproxDistinctByKey return types to use Java 
Long, not Scala; update similar methods for consistency on 
java.long.Long.valueOf with no API change

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/srowen/spark SPARK-12604

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10554.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10554


commit 1a27421aea247420b4aed597f8cc19767d11b6f0
Author: Sean Owen 
Date:   2016-01-02T13:09:40Z

Change Java countByKey, countApproxDistinctByKey return types to use Java 
Long, not Scala; update similar methods for consistency on 
java.long.Long.valueOf with no API change




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12602] [SQL] Join Reordering: Pushing I...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10551#issuecomment-168373130
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12602] [SQL] Join Reordering: Pushing I...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10551#issuecomment-168373133
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48578/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12602] [SQL] Join Reordering: Pushing I...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10551#issuecomment-168373107
  
**[Test build #48578 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48578/consoleFull)**
 for PR 10551 at commit 
[`5f67a74`](https://github.com/apache/spark/commit/5f67a7476365828a5eb920b3af3ddad312799509).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12597] [ML] Use udf to replace callUDF ...

[yanboliang]

Github user yanboliang closed the pull request at:

https://github.com/apache/spark/pull/10544


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12603] [MLlib] PySpark MLlib GaussianMi...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10552#issuecomment-168379413
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48580/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12603] [MLlib] PySpark MLlib GaussianMi...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10552#issuecomment-168379387
  
**[Test build #48580 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48580/consoleFull)**
 for PR 10552 at commit 
[`cbda57e`](https://github.com/apache/spark/commit/cbda57e9cd3c0a9c1900a81f24b716a2d335c476).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12421][SQL] Prevent Internal/External r...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10553#issuecomment-168390479
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12421][SQL] Prevent Internal/External r...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10553#issuecomment-168392090
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48582/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12582][Test] IndexShuffleBlockResolverS...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/10526#issuecomment-168395739
  
@yucai can you update accordingly?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12421][SQL] Prevent Internal/External r...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10553#issuecomment-168395737
  
**[Test build #48584 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48584/consoleFull)**
 for PR 10553 at commit 
[`4d5d162`](https://github.com/apache/spark/commit/4d5d1621cc94bbf8d90ad99b13fba25e816058c0).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Minor][SQL] Replace logger name

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10555#issuecomment-168405571
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48585/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Minor][SQL] Replace logger name

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10555#issuecomment-168405569
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12605] [SQL] [WIP] Pushing Join Predica...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10556#issuecomment-168409759
  
**[Test build #48586 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48586/consoleFull)**
 for PR 10556 at commit 
[`b5fa574`](https://github.com/apache/spark/commit/b5fa5746934b44bd996e6a1961956c5f676d159d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12578][SQL] Distinct should not be sile...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10557#issuecomment-168426136
  
**[Test build #48587 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48587/consoleFull)**
 for PR 10557 at commit 
[`79db0eb`](https://github.com/apache/spark/commit/79db0eb149c025d214b0213d9cc671faa2b10ac6).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12578][SQL] Distinct should not be sile...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10557#issuecomment-168426184
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48587/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12578][SQL] Distinct should not be sile...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10557#issuecomment-168426183
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Minor][SQL] Replace logger name

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10555#issuecomment-168397196
  
**[Test build #48585 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48585/consoleFull)**
 for PR 10555 at commit 
[`2132171`](https://github.com/apache/spark/commit/2132171c7e0137bab78082857c40a431de4ab039).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Minor][SQL] Replace logger name

[hvanhovell]

Github user hvanhovell commented on the pull request:

https://github.com/apache/spark/pull/10555#issuecomment-168399954
  
@viirya I am working on a PR to remove all Hive dependencies from the 
parser. In order to do this, I am also touching the ```ParseDriver```. That 
might supercede this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Minor][SQL] Replace logger name

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10555#issuecomment-168405533
  
**[Test build #48585 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48585/consoleFull)**
 for PR 10555 at commit 
[`2132171`](https://github.com/apache/spark/commit/2132171c7e0137bab78082857c40a431de4ab039).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12605] [SQL] [WIP] Pushing Join Predica...

[gatorsmile]

GitHub user gatorsmile opened a pull request:

https://github.com/apache/spark/pull/10556

[SPARK-12605] [SQL] [WIP] Pushing Join Predicates Through Union All

When selectivity of Join predicates is high, we can push join through union 
all for performance improvement. 

**Question**: How to decide the selectivity of Join predicates in Spark 
SQL? 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gatorsmile/spark unionAllJoin

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10556.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10556


commit 90dd5f15a554560f25e812da3796ef711fd630f6
Author: gatorsmile 
Date:   2016-01-02T08:15:00Z

join push through unionall

commit b5fa5746934b44bd996e6a1961956c5f676d159d
Author: gatorsmile 
Date:   2016-01-02T17:07:52Z

Merge remote-tracking branch 'upstream/master' into unionAllJoin




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12421][SQL] Prevent Internal/External r...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10553#issuecomment-168401275
  
**[Test build #48584 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48584/consoleFull)**
 for PR 10553 at commit 
[`4d5d162`](https://github.com/apache/spark/commit/4d5d1621cc94bbf8d90ad99b13fba25e816058c0).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12421][SQL] Prevent Internal/External r...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10553#issuecomment-168401330
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48584/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12421][SQL] Prevent Internal/External r...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10553#issuecomment-168401328
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12604] [CORE] Java count(AprroxDistinct...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10554#issuecomment-168400458
  
**[Test build #2297 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2297/consoleFull)**
 for PR 10554 at commit 
[`1a27421`](https://github.com/apache/spark/commit/1a27421aea247420b4aed597f8cc19767d11b6f0).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12578][SQL] Distinct should not be sile...

[viirya]

GitHub user viirya opened a pull request:

https://github.com/apache/spark/pull/10557

[SPARK-12578][SQL] Distinct should not be silently ignored when used in an 
aggregate function with OVER clause

JIRA: https://issues.apache.org/jira/browse/SPARK-12578

Slightly update to Hive parser. We should keep the distinct keyword when 
used in an aggregate function with OVER clause. So the CheckAnalysis will 
detect it and throw exception later.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/viirya/spark-1 keep-distinct-hivesql

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10557.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10557


commit 79db0eb149c025d214b0213d9cc671faa2b10ac6
Author: Liang-Chi Hsieh 
Date:   2016-01-02T18:34:01Z

window function: distinct should not be silently ignored.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12578][SQL] Distinct should not be sile...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10557#issuecomment-168416561
  
**[Test build #48587 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48587/consoleFull)**
 for PR 10557 at commit 
[`79db0eb`](https://github.com/apache/spark/commit/79db0eb149c025d214b0213d9cc671faa2b10ac6).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Minor][SQL] Replace logger name

[viirya]

Github user viirya closed the pull request at:

https://github.com/apache/spark/pull/10555


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Minor][SQL] Replace logger name

[viirya]

Github user viirya commented on the pull request:

https://github.com/apache/spark/pull/10555#issuecomment-168401446
  
Ok. No problem. Close this now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12533][SQL] hiveContext.table() throws ...

[thomastechs]

Github user thomastechs commented on the pull request:

https://github.com/apache/spark/pull/10529#issuecomment-168407320
  
@SparkQA , @rxin  The changes are done to fix the unit test failure. Could 
you please initiate a test build?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12605] [SQL] [WIP] Pushing Join Predica...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10556#issuecomment-168416991
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48586/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12605] [SQL] [WIP] Pushing Join Predica...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10556#issuecomment-168416990
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12605] [SQL] [WIP] Pushing Join Predica...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10556#issuecomment-168416841
  
**[Test build #48586 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48586/consoleFull)**
 for PR 10556 at commit 
[`b5fa574`](https://github.com/apache/spark/commit/b5fa5746934b44bd996e6a1961956c5f676d159d).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7689][WIP] Remove TTL-based metadata cl...

[JoshRosen]

Github user JoshRosen commented on a diff in the pull request:

https://github.com/apache/spark/pull/10534#discussion_r48687981
  
--- Diff: 
streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala 
---
@@ -81,6 +81,7 @@ class StreamingContextSuite extends SparkFunSuite with 
BeforeAndAfter with Timeo
 
   test("from conf with settings") {
 val myConf = SparkContext.updatedConf(new SparkConf(false), master, 
appName)
+// TODO(josh): Update these exmaples to use a different configuration.
--- End diff --

TODO.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12579][SQL] Force user-specified JDBC d...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/10519#issuecomment-168435471
  
I'd appreciate any feedback on how we can/should test this change and 
prevent this behavior from regressing in the future.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12602] [SQL] Join Reordering: Pushing I...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10551#issuecomment-168435937
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12602] [SQL] Join Reordering: Pushing I...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10551#issuecomment-168435938
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48588/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7689][WIP] Remove TTL-based metadata cl...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10534#issuecomment-168439939
  
**[Test build #48592 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48592/consoleFull)**
 for PR 10534 at commit 
[`e6482fa`](https://github.com/apache/spark/commit/e6482fad7da812fd3fe775f064e19893717f7a88).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7689][WIP] Remove TTL-based metadata cl...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10534#issuecomment-168439980
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48592/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7689][WIP] Remove TTL-based metadata cl...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10534#issuecomment-168439979
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10359][PROJECT-INFRA] Use a more random...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10558#issuecomment-168439965
  
**[Test build #48594 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48594/consoleFull)**
 for PR 10558 at commit 
[`0a6b120`](https://github.com/apache/spark/commit/0a6b120b13cca8b4c4264bbda6ceb7c3ec5b7135).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12602] [SQL] Join Reordering: Pushing I...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10551#issuecomment-168440612
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12602] [SQL] Join Reordering: Pushing I...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10551#issuecomment-168440613
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48593/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12602] [SQL] Join Reordering: Pushing I...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10551#issuecomment-168440559
  
**[Test build #48593 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48593/consoleFull)**
 for PR 10551 at commit 
[`7909d4c`](https://github.com/apache/spark/commit/7909d4ca68dda7bfa428ef2d4e7d02b2641be8c5).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10359][PROJECT-INFRA] Use a more random...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10558#issuecomment-168441964
  
**[Test build #2298 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2298/consoleFull)**
 for PR 10558 at commit 
[`0a6b120`](https://github.com/apache/spark/commit/0a6b120b13cca8b4c4264bbda6ceb7c3ec5b7135).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10359][PROJECT-INFRA] Use a more random...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10558#issuecomment-168449633
  
**[Test build #2298 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2298/consoleFull)**
 for PR 10558 at commit 
[`0a6b120`](https://github.com/apache/spark/commit/0a6b120b13cca8b4c4264bbda6ceb7c3ec5b7135).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12602] [SQL] Join Reordering: Pushing I...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10551#issuecomment-168428246
  
**[Test build #48588 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48588/consoleFull)**
 for PR 10551 at commit 
[`70336a4`](https://github.com/apache/spark/commit/70336a4fe841c1e6581b0c9cbefa6296bb9818cf).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12602] [SQL] Join Reordering: Pushing I...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10551#issuecomment-168435788
  
**[Test build #48593 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48593/consoleFull)**
 for PR 10551 at commit 
[`7909d4c`](https://github.com/apache/spark/commit/7909d4ca68dda7bfa428ef2d4e7d02b2641be8c5).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10359][PROJECT-INFRA] Use a more random...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10558#issuecomment-168434910
  
**[Test build #48589 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48589/consoleFull)**
 for PR 10558 at commit 
[`8e86e9c`](https://github.com/apache/spark/commit/8e86e9c41f9cf813bf92791cb85567c4a5b843cb).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10359][PROJECT-INFRA] Use a more random...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/10558#issuecomment-168435305
  
Test failure was due to Python 3.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7689][WIP] Remove TTL-based metadata cl...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10534#issuecomment-168435869
  
**[Test build #48592 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48592/consoleFull)**
 for PR 10534 at commit 
[`e6482fa`](https://github.com/apache/spark/commit/e6482fad7da812fd3fe775f064e19893717f7a88).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10359][PROJECT-INFRA] Use a more random...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10558#issuecomment-168435906
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10359][PROJECT-INFRA] Use a more random...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10558#issuecomment-168435904
  
**[Test build #48591 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48591/consoleFull)**
 for PR 10558 at commit 
[`77a23bf`](https://github.com/apache/spark/commit/77a23bf44ac320baba0a34088aa02e2df402d797).
 * This patch **fails build dependency tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10359][PROJECT-INFRA] Use a more random...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10558#issuecomment-168435907
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48591/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10359][PROJECT-INFRA] Use a more random...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10558#issuecomment-168435836
  
**[Test build #48591 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48591/consoleFull)**
 for PR 10558 at commit 
[`77a23bf`](https://github.com/apache/spark/commit/77a23bf44ac320baba0a34088aa02e2df402d797).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12602] [SQL] Join Reordering: Pushing I...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10551#issuecomment-168435871
  
**[Test build #48588 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48588/consoleFull)**
 for PR 10551 at commit 
[`70336a4`](https://github.com/apache/spark/commit/70336a4fe841c1e6581b0c9cbefa6296bb9818cf).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7689][WIP] Remove TTL-based metadata cl...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10534#issuecomment-168440019
  
**[Test build #48590 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48590/consoleFull)**
 for PR 10534 at commit 
[`5ffe30f`](https://github.com/apache/spark/commit/5ffe30f99f4213457cc6c84032932319905f4420).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7689][WIP] Remove TTL-based metadata cl...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10534#issuecomment-168440046
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48590/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7689][WIP] Remove TTL-based metadata cl...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10534#issuecomment-168440044
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10359][PROJECT-INFRA] Use a more random...

[JoshRosen]

GitHub user JoshRosen opened a pull request:

https://github.com/apache/spark/pull/10558

[SPARK-10359][PROJECT-INFRA] Use a more random suffix in 
dev/test-dependencies.sh

This patch aims to fix another potential source of flakiness in the 
`dev/test-dependencies.sh` script.

@pwendell's original patch and my version used `$(date +%s | tail -c6)` to 
generate a suffix to use when installing temporary Spark versions into the 
local Maven cache, but this value only changes once per second and thus is 
highly collision-prone when concurrent builds launch on AMPLab Jenkins. In 
order to reduce the potential for conflicts, this patch updates the script to 
call Python's random number generator instead.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JoshRosen/spark build-dep-tests-round-3

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10558.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10558


commit 8e86e9c41f9cf813bf92791cb85567c4a5b843cb
Author: Josh Rosen 
Date:   2016-01-02T21:59:10Z

Use a more random suffix in dev/test-dependencies.sh




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10359][PROJECT-INFRA] Use a more random...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/10558#issuecomment-168432407
  
/cc @rxin 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7689][WIP] Remove TTL-based metadata cl...

[JoshRosen]

Github user JoshRosen commented on a diff in the pull request:

https://github.com/apache/spark/pull/10534#discussion_r48688012
  
--- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ---
@@ -291,17 +289,10 @@ private[spark] class MapOutputTrackerMaster(conf: 
SparkConf)
   // can be read locally, but may lead to more delay in scheduling if 
those locations are busy.
   private val REDUCER_PREF_LOCS_FRACTION = 0.2
 
-  /**
-   * Timestamp based HashMap for storing mapStatuses and cached serialized 
statuses in the driver,
-   * so that statuses are dropped only by explicit de-registering or by 
TTL-based cleaning (if set).
-   * Other than these two scenarios, nothing should be dropped from this 
HashMap.
-   */
-  protected val mapStatuses = new TimeStampedHashMap[Int, 
Array[MapStatus]]()
-  private val cachedSerializedStatuses = new TimeStampedHashMap[Int, 
Array[Byte]]()
-
-  // For cleaning up TimeStampedHashMaps
-  private val metadataCleaner =
-new MetadataCleaner(MetadataCleanerType.MAP_OUTPUT_TRACKER, 
this.cleanup, conf)
+  // HashMaps for storing mapStatuses and cached serialized statuses in 
the driver.
+  // Statuses are dropped only by explicit de-registering.
+  protected val mapStatuses = new HashMap[Int, Array[MapStatus]]()
--- End diff --

I think that these need to be ConcurrentHashMaps in order to preserve the 
old code's thread-safety guarantees.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org