[GitHub] spark issue #18605: [SparkR][SPARK-21381]:SparkR: pass on setHandleInvalid f...

[wangmiao1981]

Github user wangmiao1981 commented on the issue:

https://github.com/apache/spark/pull/18605
  
@felixcheung This is a follow-up PR of JIRA-20307.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18554: [SPARK-21306][ML] OneVsRest should cache weightCo...

[yanboliang]

Github user yanboliang commented on a diff in the pull request:

https://github.com/apache/spark/pull/18554#discussion_r126868713
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/ml/classification/OneVsRest.scala ---
@@ -317,7 +318,12 @@ final class OneVsRest @Since("1.4.0") (
 val numClasses = 
MetadataUtils.getNumClasses(labelSchema).fold(computeNumClasses())(identity)
 instr.logNumClasses(numClasses)
 
-val multiclassLabeled = dataset.select($(labelCol), $(featuresCol))
+val multiclassLabeled = getClassifier match {
+  // SPARK-21306: cache weightCol if necessary
+  case c: HasWeightCol if c.isDefined(c.weightCol) && 
c.getWeightCol.nonEmpty =>
+dataset.select($(labelCol), $(featuresCol), c.getWeightCol)
+  case _ => dataset.select($(labelCol), $(featuresCol))
+}
--- End diff --

@facaiy It doesn't matter. If the classifier doesn't inherit from 
```HasWeightCol```, we don't run ```setWeightCol``` for that classifier instead 
of printing out warning log to say ```weightCol``` doesn't take effect. You can 
refer [these lines of 
code](https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/classification/OneVsRest.scala#L338)
 to learn how ```featuresCol``` be handled. Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18600: [SPARK-17701][SQL] Refactor RowDataSourceScanExec...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18600#discussion_r126868678
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala 
---
@@ -73,34 +72,24 @@ trait DataSourceScanExec extends LeafExecNode with 
CodegenSupport {
 
 /** Physical plan node for scanning data from a relation. */
 case class RowDataSourceScanExec(
-output: Seq[Attribute],
+fullOutput: Seq[Attribute],
+requiredColumnsIndex: Seq[Int],
+filters: Set[Filter],
 rdd: RDD[InternalRow],
 @transient relation: BaseRelation,
-override val outputPartitioning: Partitioning,
-override val metadata: Map[String, String],
--- End diff --

uh... This is not being used after our previous refactoring.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18228: [SPARK-21007][SQL]Add SQL function - RIGHT && LEFT

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18228
  
**[Test build #79550 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79550/testReport)**
 for PR 18228 at commit 
[`991bf99`](https://github.com/apache/spark/commit/991bf9980010ec85c2325109d3afaceecd7c4c23).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18228: [SPARK-21007][SQL]Add SQL function - RIGHT && LEFT

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18228
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18228: [SPARK-21007][SQL]Add SQL function - RIGHT && LEFT

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18228
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79548/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18228: [SPARK-21007][SQL]Add SQL function - RIGHT && LEFT

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18228
  
**[Test build #79548 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79548/testReport)**
 for PR 18228 at commit 
[`b75b6ac`](https://github.com/apache/spark/commit/b75b6ac7d4664a8c86a7e4e47ee848921fb5610d).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class Right(str: Expression, len: Expression, child: Expression) 
extends RuntimeReplaceable `
  * `case class Left(str: Expression, len: Expression, child: Expression) 
extends RuntimeReplaceable `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18600: [SPARK-17701][SQL] Refactor RowDataSourceScanExec...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18600#discussion_r126866057
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala 
---
@@ -73,34 +72,24 @@ trait DataSourceScanExec extends LeafExecNode with 
CodegenSupport {
 
 /** Physical plan node for scanning data from a relation. */
 case class RowDataSourceScanExec(
-output: Seq[Attribute],
+fullOutput: Seq[Attribute],
+requiredColumnsIndex: Seq[Int],
+filters: Set[Filter],
--- End diff --

Start it in this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18023: [SPARK-12139] [SQL] REGEX Column Specification

[janewangfb]

Github user janewangfb commented on the issue:

https://github.com/apache/spark/pull/18023
  
@gatorsmile 

Sure, I could have a follow-up PR to resolve DataFrameNaFunctions.fill. 

thanks for reviewing this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18023: [SPARK-12139] [SQL] REGEX Column Specification

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/18023


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18023: [SPARK-12139] [SQL] REGEX Column Specification

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/18023
  
Thanks! Merging to master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18023: [SPARK-12139] [SQL] REGEX Column Specification

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/18023
  
LGTM 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18023: [SPARK-12139] [SQL] REGEX Column Specification

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/18023
  
The last comment is about `DataFrameNaFunctions.fill`. It does not work 
when `spark.sql.parser.quotedRegexColumnNames` is on. Could you resolve that in 
the follow-up PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18444: [SPARK-16542][SQL][PYSPARK] Fix bugs about types that re...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18444
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18444: [SPARK-16542][SQL][PYSPARK] Fix bugs about types that re...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18444
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79547/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18444: [SPARK-16542][SQL][PYSPARK] Fix bugs about types that re...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18444
  
**[Test build #79547 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79547/testReport)**
 for PR 18444 at commit 
[`9ee439f`](https://github.com/apache/spark/commit/9ee439f74b88faa1e79cf55ac50b35f650fedca6).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18554: [SPARK-21306][ML] OneVsRest should cache weightCo...

[facaiy]

Github user facaiy commented on a diff in the pull request:

https://github.com/apache/spark/pull/18554#discussion_r126863072
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/ml/classification/OneVsRest.scala ---
@@ -317,7 +318,12 @@ final class OneVsRest @Since("1.4.0") (
 val numClasses = 
MetadataUtils.getNumClasses(labelSchema).fold(computeNumClasses())(identity)
 instr.logNumClasses(numClasses)
 
-val multiclassLabeled = dataset.select($(labelCol), $(featuresCol))
+val multiclassLabeled = getClassifier match {
+  // SPARK-21306: cache weightCol if necessary
+  case c: HasWeightCol if c.isDefined(c.weightCol) && 
c.getWeightCol.nonEmpty =>
+dataset.select($(labelCol), $(featuresCol), c.getWeightCol)
+  case _ => dataset.select($(labelCol), $(featuresCol))
+}
--- End diff --

Hi, @yanboliang . As @MLnick said, no all classifiers inherits 
HasWeightCol, so it might cause confusion.

In my opinion, `setWeightCol` is an attribute owned by one specific 
classifier, like `setProbabilityCol` and `setRawPredictionCol` for Logistic 
Regreesion. So I'd suggest that user should configure the classifier itself, 
rather than OneVsRest. Is it OK?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18594: [SPARK-20904][core] Don't report task failures to driver...

[jiangxb1987]

Github user jiangxb1987 commented on the issue:

https://github.com/apache/spark/pull/18594
  
I'm hesitant to support the change. If we don't notify the failure to 
driver, the status of the failed task would not be updated, thus not 
rescheduled, perhaps it's not the behavior we expect to see?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18604: [SPARK-21219][CORE] Task retry occurs on same executor d...

[jsoltren]

Github user jsoltren commented on the issue:

https://github.com/apache/spark/pull/18604
  
My preference is to backport this and other blacklisting related fixes as 
far back as possible on Spark2 - meaning 2.1 and 2.0 as well, unless convinced 
otherwise. So, yes, @cloud-fan, I hope we do backport this!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18228: [SPARK-21007][SQL]Add SQL function - RIGHT && LEF...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/18228#discussion_r126857958
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
 ---
@@ -1199,6 +1199,49 @@ case class Substring(str: Expression, pos: 
Expression, len: Expression)
 }
 
 /**
+ * Returns the rightmost n characters from the string.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+  usage = "_FUNC_(str, len) - Returns the rightmost `len`(`len` can be 
string type) characters from the string `str`,if `len` is less or equal than 0 
the result is an empty string.",
+  extended = """
+Examples:
+  > SELECT _FUNC_('Spark SQL', 3);
+   SQL
+  """)
+// scalastyle:on line.size.limit
+case class Right(str: Expression, len: Expression, child: Expression) 
extends RuntimeReplaceable {
+  def this(str: Expression, len: Expression) = {
+this(str, len, If(LessThanOrEqual(len, Literal(0)), If(IsNull(str), 
Literal(null, StringType),
--- End diff --

we can do the null check first, e.g.
```
If(
  IsNull(str),
  Literal(null, StringType),
  If(
LessThanOrEqual(len, Literal(0)),
Literal(UTF8String.EMPTY_UTF8, StringType),
new Substring(str, UnaryMinus(len))
  )
)
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18604: [SPARK-21219][CORE] Task retry occurs on same executor d...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18604
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18604: [SPARK-21219][CORE] Task retry occurs on same executor d...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18604
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79545/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18604: [SPARK-21219][CORE] Task retry occurs on same executor d...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18604
  
**[Test build #79545 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79545/testReport)**
 for PR 18604 at commit 
[`2ea00a5`](https://github.com/apache/spark/commit/2ea00a58a18359f8916b7a9f5e56ae7bea9d1208).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

[heary-cao]

Github user heary-cao commented on the issue:

https://github.com/apache/spark/pull/18555
  
@gatorsmile @cloud-fan 
please review it again.
thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18555
  
**[Test build #79549 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79549/testReport)**
 for PR 18555 at commit 
[`dd066b6`](https://github.com/apache/spark/commit/dd066b60bcb4257d3825a47840b68b9b55f4131a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18601: [SPARK-21373][core] Update Jetty to 9.3.20.v20170531

[kiszk]

Github user kiszk commented on the issue:

https://github.com/apache/spark/pull/18601
  
@srowen thank you for your suggestion. I executed `mvn dependency:tree` in 
the current master and the version with this PR. I confirmed that only the 
difference is related to `org.eclipse.jetty:jetty-...`.
Thus, I think that this PR does not change internal dependency structure 
beyond `jetty`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18228: [SPARK-21007][SQL]Add SQL function - RIGHT && LEFT

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18228
  
**[Test build #79548 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79548/testReport)**
 for PR 18228 at commit 
[`b75b6ac`](https://github.com/apache/spark/commit/b75b6ac7d4664a8c86a7e4e47ee848921fb5610d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18606: [SPARK-21382] The note about Scala 2.10 in building-spar...

[liu-zhaokun]

Github user liu-zhaokun commented on the issue:

https://github.com/apache/spark/pull/18606
  
@srowen 
But now,spark 2.2.0 released.we  know exactly that Scala 2.10 isn't removed 
in Spark 2.2.0,so we shouldn't give the user an inaccurate message.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18596: [SPARK-21371] dev/make-distribution.sh scripts use of $@...

[liu-zhaokun]

Github user liu-zhaokun commented on the issue:

https://github.com/apache/spark/pull/18596
  
@srowen  @jiangxb1987 
Hello,I have modified the PR according to your opinion.Could you help me 
review it again.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18606: [SPARK-21382] The note about Scala 2.10 in building-spar...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18606
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18606: [SPARK-21382] The note about Scala 2.10 in building-spar...

[srowen]

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/18606
  
No, the statement was correct. It said may be removed. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18606: [SPARK-21382] The note about Scala 2.10 in buildi...

[liu-zhaokun]

GitHub user liu-zhaokun opened a pull request:

https://github.com/apache/spark/pull/18606

[SPARK-21382] The note about Scala 2.10 in building-spark.md is wrong.


[https://issues.apache.org/jira/browse/SPARK-21382](https://issues.apache.org/jira/browse/SPARK-21382)
There should be "Note that support for Scala 2.10 is deprecated as of Spark 
2.1.0 and may be removed in Spark 2.3.0",right?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/liu-zhaokun/spark new07120923

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/18606.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #18606


commit 92dd0880d71ba9b3adb9812ab94989a4d62e1195
Author: liuzhaokun 
Date:   2017-07-12T02:19:46Z

[SPARK-21382] The note about Scala 2.10 in building-spark.md is wrong.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18444: [SPARK-16542][SQL][PYSPARK] Fix bugs about types that re...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18444
  
**[Test build #79547 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79547/testReport)**
 for PR 18444 at commit 
[`9ee439f`](https://github.com/apache/spark/commit/9ee439f74b88faa1e79cf55ac50b35f650fedca6).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18468: [SPARK-20873][SQL] Enhance ColumnVector to support compr...

[kiszk]

Github user kiszk commented on the issue:

https://github.com/apache/spark/pull/18468
  
@cloud-fan could you please review this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18281: [SPARK-21027][ML][PYTHON] Added tunable parallelism to o...

[WeichenXu123]

Github user WeichenXu123 commented on the issue:

https://github.com/apache/spark/pull/18281
  
@ajaysaini725 @jkbradley  Can we avoid python-side to re-implement the 
logic of OneVsRest? It can simply python-side code I think. Just let the 
wrapper inherit `JavaEstimator`, and when we setting`setClassifer()` in 
python-side we can get the backend java object through `classifer._java_obj` 
and pass it to the scala side...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18605: [SparkR][SPARK-21381]:SparkR: pass on setHandleInvalid f...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18605
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79546/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18605: [SparkR][SPARK-21381]:SparkR: pass on setHandleInvalid f...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18605
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18605: [SparkR][SPARK-21381]:SparkR: pass on setHandleInvalid f...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18605
  
**[Test build #79546 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79546/testReport)**
 for PR 18605 at commit 
[`77b04a3`](https://github.com/apache/spark/commit/77b04a37e93d6967def24c0a8265ed784875f5b0).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18428: [Spark-21221][ML] CrossValidator and TrainValidationSpli...

[jkbradley]

Github user jkbradley commented on the issue:

https://github.com/apache/spark/pull/18428
  
Also, can you please add "OneVsRest" to the PR and JIRA titles since this 
touches that class?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18428: [Spark-21221][ML] CrossValidator and TrainValidat...

[jkbradley]

Github user jkbradley commented on a diff in the pull request:

https://github.com/apache/spark/pull/18428#discussion_r126849117
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/ml/tuning/ValidatorParams.scala ---
@@ -183,8 +198,14 @@ private[ml] object ValidatorParams {
   val paramPairs = pMap.map { case pInfo: Map[String, String] =>
 val est = uidToParams(pInfo("parent"))
 val param = est.getParam(pInfo("name"))
-val value = param.jsonDecode(pInfo("value"))
-param -> value
+if (pInfo("isJson").toBoolean.booleanValue()) {
--- End diff --

I *think* fixing backwards compatibility will just mean testing for whether 
the field "isJson" is present here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18428: [Spark-21221][ML] CrossValidator and TrainValidationSpli...

[jkbradley]

Github user jkbradley commented on the issue:

https://github.com/apache/spark/pull/18428
  
LGTM
I couldn't think of a great way to reduce code duplication between 
JavaWrapper and OneVsRest.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18599: [SPARK-21372] spark writes one log file even I set the n...

[liu-zhaokun]

Github user liu-zhaokun commented on the issue:

https://github.com/apache/spark/pull/18599
  
@srowen 
I think  spark provides the param,so I can pass it in the script.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18599: [SPARK-21372] spark writes one log file even I se...

[liu-zhaokun]

Github user liu-zhaokun commented on a diff in the pull request:

https://github.com/apache/spark/pull/18599#discussion_r126847486
  
--- Diff: sbin/spark-daemon.sh ---
@@ -78,6 +78,12 @@ spark_rotate_log ()
 if [ -n "$2" ]; then
num=$2
--- End diff --

@srowen 
There provide a param,num, to set the number of logfile,but it wasn't used 
in line 179.Since it doesn't work,can I remove the param "num" and use it just 
as a variable？


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18444: [SPARK-16542][SQL][PYSPARK] Fix bugs about types that re...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18444
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18444: [SPARK-16542][SQL][PYSPARK] Fix bugs about types that re...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18444
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79544/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18444: [SPARK-16542][SQL][PYSPARK] Fix bugs about types that re...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18444
  
**[Test build #79544 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79544/testReport)**
 for PR 18444 at commit 
[`38a8bef`](https://github.com/apache/spark/commit/38a8bef13cce1fef4c427292786229c92c52fcfc).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18143: [SPARK-20919][SS] Simplificaiton of CachedKafkaConsumer ...

[zsxwing]

Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/18143
  
@ScrapCodes I think it should be bounded by 
`spark.sql.kafkaConsumerCache.capacity`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18603: [SPARK-21370][SS] Add test for state reliability when on...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18603
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18603: [SPARK-21370][SS] Add test for state reliability when on...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18603
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79543/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18228: [SPARK-21007][SQL]Add SQL function - RIGHT && LEF...

[10110346]

Github user 10110346 commented on a diff in the pull request:

https://github.com/apache/spark/pull/18228#discussion_r126842908
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
 ---
@@ -1199,6 +1199,49 @@ case class Substring(str: Expression, pos: 
Expression, len: Expression)
 }
 
 /**
+ * Returns the rightmost n characters from the string.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+  usage = "_FUNC_(str, len) - Returns the rightmost `len`(`len` can be 
string type) characters from the string `str`,if `len` is less or equal than 0 
the result is ``.",
--- End diff --

ok,  thanks


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18228: [SPARK-21007][SQL]Add SQL function - RIGHT && LEF...

[10110346]

Github user 10110346 commented on a diff in the pull request:

https://github.com/apache/spark/pull/18228#discussion_r126842852
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
 ---
@@ -1199,6 +1199,49 @@ case class Substring(str: Expression, pos: 
Expression, len: Expression)
 }
 
 /**
+ * Returns the rightmost n characters from the string.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+  usage = "_FUNC_(str, len) - Returns the rightmost `len`(`len` can be 
string type) characters from the string `str`,if `len` is less or equal than 0 
the result is ``.",
+  extended = """
+Examples:
+  > SELECT _FUNC_('Spark SQL', 3);
+   SQL
+  """)
+// scalastyle:on line.size.limit
+case class Right(str: Expression, len: Expression, child: Expression) 
extends RuntimeReplaceable {
+  def this(str: Expression, len: Expression) = {
+this(str, len, Substring(str, If(LessThanOrEqual(len, Literal(0)),
+  Literal(Integer.MAX_VALUE), UnaryMinus(len)), len))
--- End diff --

`right(null, -10)`
I agree with you, but , for this test case, there is a problem:
Which we expected is `null`,but it is an empty string


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18603: [SPARK-21370][SS] Add test for state reliability when on...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18603
  
**[Test build #79543 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79543/testReport)**
 for PR 18603 at commit 
[`4783b12`](https://github.com/apache/spark/commit/4783b124c365cde6a9398fc9278f8611aa4d7598).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18605: [SparkR][SPARK-21381]:SparkR: pass on setHandleInvalid f...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18605
  
**[Test build #79546 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79546/testReport)**
 for PR 18605 at commit 
[`77b04a3`](https://github.com/apache/spark/commit/77b04a37e93d6967def24c0a8265ed784875f5b0).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18605: [SparkR][SPARK-21381]:SparkR: pass on setHandleIn...

[wangmiao1981]

GitHub user wangmiao1981 opened a pull request:

https://github.com/apache/spark/pull/18605

[SparkR][SPARK-21381]:SparkR: pass on setHandleInvalid for classification 
algorithms

## What changes were proposed in this pull request?

SPARK-20307 Added handleInvalid option to RFormula for tree-based 
classification algorithms. We should add this parameter for other 
classification algorithms in SparkR.

This is a followup PR for SPARK-20307.

## How was this patch tested?

New Unit tests are added.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wangmiao1981/spark class

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/18605.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #18605


commit 77b04a37e93d6967def24c0a8265ed784875f5b0
Author: wangmiao1981 
Date:   2017-07-12T00:40:58Z

add handleInvalid for classifications




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18604: [SPARK-21219][CORE] Task retry occurs on same executor d...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18604
  
**[Test build #79545 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79545/testReport)**
 for PR 18604 at commit 
[`2ea00a5`](https://github.com/apache/spark/commit/2ea00a58a18359f8916b7a9f5e56ae7bea9d1208).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18604: [SPARK-21219][CORE] Task retry occurs on same executor d...

[vanzin]

Github user vanzin commented on the issue:

https://github.com/apache/spark/pull/18604
  
ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18604: [SPARK-21219][CORE] Task retry occurs on same executor d...

[vanzin]

Github user vanzin commented on the issue:

https://github.com/apache/spark/pull/18604
  
I don't see why we shouldn't backport a fix to 2.2.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18357: [SPARK-21146] [CORE] Master/Worker should handle and shu...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18357
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18357: [SPARK-21146] [CORE] Master/Worker should handle and shu...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18357
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79542/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18357: [SPARK-21146] [CORE] Master/Worker should handle and shu...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18357
  
**[Test build #79542 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79542/testReport)**
 for PR 18357 at commit 
[`3d5106b`](https://github.com/apache/spark/commit/3d5106b4b0ec67350e89b1b7579cd2c164bc1b4b).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18604: [SPARK-21219][CORE] Task retry occurs on same executor d...

[jiangxb1987]

Github user jiangxb1987 commented on the issue:

https://github.com/apache/spark/pull/18604
  
Do we want to backport this to 2.2? @cloud-fan 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18253: [SPARK-18838][CORE] Introduce multiple queues in LiveLis...

[bOOm-X]

Github user bOOm-X commented on the issue:

https://github.com/apache/spark/pull/18253
  
@vanzin I put an ArrayBlockingQueue as you wanted


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18603: [SPARK-21370][SS] Add test for state reliability when on...

[tdas]

Github user tdas commented on the issue:

https://github.com/apache/spark/pull/18603
  
LGTM!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18444: [SPARK-16542][SQL][PYSPARK] Fix bugs about types that re...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18444
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79540/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18444: [SPARK-16542][SQL][PYSPARK] Fix bugs about types that re...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18444
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18444: [SPARK-16542][SQL][PYSPARK] Fix bugs about types that re...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18444
  
**[Test build #79540 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79540/testReport)**
 for PR 18444 at commit 
[`9ee439f`](https://github.com/apache/spark/commit/9ee439f74b88faa1e79cf55ac50b35f650fedca6).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18444: [SPARK-16542][SQL][PYSPARK] Fix bugs about types that re...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18444
  
**[Test build #79544 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79544/testReport)**
 for PR 18444 at commit 
[`38a8bef`](https://github.com/apache/spark/commit/38a8bef13cce1fef4c427292786229c92c52fcfc).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18604: [SPARK-21219][CORE] Task retry occurs on same executor d...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18604
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18444: [SPARK-16542][SQL][PYSPARK] Fix bugs about types that re...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18444
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18444: [SPARK-16542][SQL][PYSPARK] Fix bugs about types that re...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18444
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79539/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18444: [SPARK-16542][SQL][PYSPARK] Fix bugs about types that re...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18444
  
**[Test build #79539 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79539/testReport)**
 for PR 18444 at commit 
[`5e3128c`](https://github.com/apache/spark/commit/5e3128ce8bd5f9529099c1cc974adfeb24d1a261).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class ArrowSerializer(FramedSerializer):`
  * `class AstBuilder(conf: SQLConf) extends SqlBaseBaseVisitor[AnyRef] 
with Logging `
  * `class CatalystSqlParser(conf: SQLConf) extends AbstractSqlParser `
  * `class SparkSqlParser(conf: SQLConf) extends AbstractSqlParser `
  * `class SparkSqlAstBuilder(conf: SQLConf) extends AstBuilder(conf) `
  * `class VariableSubstitution(conf: SQLConf) `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18604: [SPARK-21219][CORE] Task retry occurs on same exe...

[jsoltren]

GitHub user jsoltren opened a pull request:

https://github.com/apache/spark/pull/18604

[SPARK-21219][CORE] Task retry occurs on same executor due to race co…

…ndition with blacklisting

There's a race condition in the current TaskSetManager where a failed task 
is added for retry (addPendingTask), and can asynchronously be assigned to an 
executor *prior* to the blacklist state (updateBlacklistForFailedTask), the 
result is the task might re-execute on the same executor.  This is particularly 
problematic if the executor is shutting down since the retry task immediately 
becomes a lost task (ExecutorLostFailure).  Another side effect is that the 
actual failure reason gets obscured by the retry task which never actually 
executed.  There are sample logs showing the issue in the 
https://issues.apache.org/jira/browse/SPARK-21219

The fix is to change the ordering of the addPendingTask and 
updatingBlackListForFailedTask calls in TaskSetManager.handleFailedTask

Implemented a unit test that verifies the task is black listed before it is 
added to the pending task.  Ran the unit test without the fix and it fails.  
Ran the unit test with the fix and it passes.

Please review http://spark.apache.org/contributing.html before opening a 
pull request.

Author: Eric Vandenberg 

Closes #18427 from ericvandenbergfb/blacklistFix.

## What changes were proposed in this pull request?

This is a backport of the fix to SPARK-21219, already checked in as 96d58f2.

## How was this patch tested?

Ran TaskSetManagerSuite tests locally.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jsoltren/spark branch-2.2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/18604.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #18604


commit 2ea00a58a18359f8916b7a9f5e56ae7bea9d1208
Author: Eric Vandenberg 
Date:   2017-07-10T06:40:20Z

[SPARK-21219][CORE] Task retry occurs on same executor due to race 
condition with blacklisting

There's a race condition in the current TaskSetManager where a failed task 
is added for retry (addPendingTask), and can asynchronously be assigned to an 
executor *prior* to the blacklist state (updateBlacklistForFailedTask), the 
result is the task might re-execute on the same executor.  This is particularly 
problematic if the executor is shutting down since the retry task immediately 
becomes a lost task (ExecutorLostFailure).  Another side effect is that the 
actual failure reason gets obscured by the retry task which never actually 
executed.  There are sample logs showing the issue in the 
https://issues.apache.org/jira/browse/SPARK-21219

The fix is to change the ordering of the addPendingTask and 
updatingBlackListForFailedTask calls in TaskSetManager.handleFailedTask

Implemented a unit test that verifies the task is black listed before it is 
added to the pending task.  Ran the unit test without the fix and it fails.  
Ran the unit test with the fix and it passes.

Please review http://spark.apache.org/contributing.html before opening a 
pull request.

Author: Eric Vandenberg 

Closes #18427 from ericvandenbergfb/blacklistFix.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15496: [SPARK-17950] [Python] Match SparseVector behavior with ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15496
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18598: [SPARK-19285] [SQL] Implement UDF0

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/18598


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18598: [SPARK-19285] [SQL] Implement UDF0

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/18598
  
Thanks! Merging to master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18602: [SPARK-21377][YARN] Add a new configuration to extend AM...

[jerryshao]

Github user jerryshao commented on the issue:

https://github.com/apache/spark/pull/18602
  
Thanks @vanzin , based on the comment of JIRA, I will try another approach, 
so closing it for now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18602: [SPARK-21377][YARN] Add a new configuration to ex...

[jerryshao]

Github user jerryshao closed the pull request at:

https://github.com/apache/spark/pull/18602


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18603: [SPARK-21370][SS] Add test for state reliability when on...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18603
  
**[Test build #79543 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79543/testReport)**
 for PR 18603 at commit 
[`4783b12`](https://github.com/apache/spark/commit/4783b124c365cde6a9398fc9278f8611aa4d7598).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18601: [SPARK-21373][core] Update Jetty to 9.3.20.v20170531

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18601
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18601: [SPARK-21373][core] Update Jetty to 9.3.20.v20170531

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18601
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79538/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18601: [SPARK-21373][core] Update Jetty to 9.3.20.v20170531

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18601
  
**[Test build #79538 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79538/testReport)**
 for PR 18601 at commit 
[`3b36793`](https://github.com/apache/spark/commit/3b367932a75a42ae982fbebb73e246a81da14a6e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18603: [SPARK-21370][SS] Add test for state reliability ...

[brkyvz]

GitHub user brkyvz opened a pull request:

https://github.com/apache/spark/pull/18603

[SPARK-21370][SS] Add test for state reliability when one read-only state 
store aborts after read-write state store commits

## What changes were proposed in this pull request?

During Streaming Aggregation, we have two StateStores per task, one used as 
read-only in
`StateStoreRestoreExec`, and one read-write used in `StateStoreSaveExec`. 
`StateStore.abort`
will be called for these StateStores if they haven't committed their 
results. We need to
make sure that `abort` in read-only store after a `commit` in the 
read-write store doesn't
accidentally lead to the deletion of state.

This PR adds a test for this condition.

## How was this patch tested?

This PR adds a test.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/brkyvz/spark ss-test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/18603.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #18603


commit 4783b124c365cde6a9398fc9278f8611aa4d7598
Author: Burak Yavuz 
Date:   2017-07-11T22:27:41Z

Added test for two concurrent state stores




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15009: [SPARK-17443][SPARK-11035] Stop Spark Application if lau...

[vanzin]

Github user vanzin commented on the issue:

https://github.com/apache/spark/pull/15009
  
@kishorvpatil are you planning to address the last bit of feedback 
remaining here? It shouldn't be that hard to make that test better.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18602: [SPARK-21377][YARN] Add a new configuration to extend AM...

[vanzin]

Github user vanzin commented on the issue:

https://github.com/apache/spark/pull/18602
  
I commented on the bug.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18602: [SPARK-21377][YARN] Add a new configuration to extend AM...

[jerryshao]

Github user jerryshao commented on the issue:

https://github.com/apache/spark/pull/18602
  
CC @vanzin @tgravescs would you please help to review? Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18357: [SPARK-21146] [CORE] Master/Worker should handle and shu...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18357
  
**[Test build #79542 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79542/testReport)**
 for PR 18357 at commit 
[`3d5106b`](https://github.com/apache/spark/commit/3d5106b4b0ec67350e89b1b7579cd2c164bc1b4b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18357: [SPARK-21146] [CORE] Master/Worker should handle and shu...

[zsxwing]

Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/18357
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18602: [SPARK-21377][YARN] Add a new configuration to extend AM...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18602
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18602: [SPARK-21377][YARN] Add a new configuration to extend AM...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18602
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79541/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18602: [SPARK-21377][YARN] Add a new configuration to extend AM...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18602
  
**[Test build #79541 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79541/testReport)**
 for PR 18602 at commit 
[`6f91356`](https://github.com/apache/spark/commit/6f9135645cd767e9d69d98157189c2e7ba08a5cc).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18357: [SPARK-21146] [CORE] Master/Worker should handle and shu...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18357
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79537/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18357: [SPARK-21146] [CORE] Master/Worker should handle and shu...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18357
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18444: [SPARK-16542][SQL][PYSPARK] Fix bugs about types ...

[zasdfgbnm]

Github user zasdfgbnm commented on a diff in the pull request:

https://github.com/apache/spark/pull/18444#discussion_r126809517
  
--- Diff: core/src/main/scala/org/apache/spark/api/python/SerDeUtil.scala 
---
@@ -72,7 +72,11 @@ private[spark] object SerDeUtil extends Logging {
 val typecode = args(0).asInstanceOf[String].charAt(0)
 // This must be ISO 8859-1 / Latin 1, not UTF-8, to interoperate 
correctly
 val data = 
args(1).asInstanceOf[String].getBytes(StandardCharsets.ISO_8859_1)
--- End diff --

Can anyone explain why `ISO_8859_1` is used here instead of UTF16 or UTF32?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18357: [SPARK-21146] [CORE] Master/Worker should handle and shu...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18357
  
**[Test build #79537 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79537/testReport)**
 for PR 18357 at commit 
[`3d5106b`](https://github.com/apache/spark/commit/3d5106b4b0ec67350e89b1b7579cd2c164bc1b4b).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18587: [SPARK-12559][Mesos] fix --packages for mesos

[vanzin]

Github user vanzin commented on the issue:

https://github.com/apache/spark/pull/18587
  
I'm 99% sure there's nothing to do for YARN. This line takes care of it:

args.jars = mergeFileLists(args.jars, resolvedMavenCoordinates)

YARN cluster mode will distribute all jars in `args.jars` to the app.

As for the change, it seems to work because the Mesos backend starts the 
driver using `spark-submit`, right? (It would probably have to change the 
deploy mode from "cluster" to "client" when doing that but I didn't dig that 
much into the code...)

If that's the case it seems fine, although it kinda loses the ability to 
use the ivy cache on the machine launching the job...

Also, I'd be more comfortable if someone more familiar with the Mesos 
backend could take a look. Not sure who that person is.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18602: [SPARK-21377][YARN] Add a new configuration to extend AM...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18602
  
**[Test build #79541 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79541/testReport)**
 for PR 18602 at commit 
[`6f91356`](https://github.com/apache/spark/commit/6f9135645cd767e9d69d98157189c2e7ba08a5cc).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18602: [SPARK-21377][YARN] Add a new configuration to ex...

[jerryshao]

GitHub user jerryshao opened a pull request:

https://github.com/apache/spark/pull/18602

[SPARK-21377][YARN] Add a new configuration to extend AM classpath in yarn 
client mode

## What changes were proposed in this pull request?

This PR propose a new configuration "spark.yarn.am.extraClassPath" to 
extend AM classpath in yarn client mode. The specific scenario is that we have 
custom `ServiceCredentialProvider` which will be loaded in AM, and this 
provider requires its additional dependencies to be added in AM classpath. 

Using "spark.driver.extraClassPath" (the current code) is not so proper in 
yarn client mode and if dependency paths are different for driver and AM node, 
then it is impossible to use this configuration. So instead we add a new 
configuration to extend AM classpath in yarn client mode.

## How was this patch tested?

UT added and manual verification on local cluster.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jerryshao/apache-spark SPARK-21377

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/18602.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #18602


commit 6f9135645cd767e9d69d98157189c2e7ba08a5cc
Author: jerryshao 
Date:   2017-07-11T20:52:22Z

Add a new configuration to extend AM Classpath when running in yarn client 
mode

Change-Id: I2d9e1c3ab65b648bc1ad321268394e0d15b1eb3f




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18444: [SPARK-16542][SQL][PYSPARK] Fix bugs about types ...

[zasdfgbnm]

Github user zasdfgbnm commented on a diff in the pull request:

https://github.com/apache/spark/pull/18444#discussion_r126806942
  
--- Diff: core/src/main/scala/org/apache/spark/api/python/SerDeUtil.scala 
---
@@ -72,7 +72,11 @@ private[spark] object SerDeUtil extends Logging {
 val typecode = args(0).asInstanceOf[String].charAt(0)
 // This must be ISO 8859-1 / Latin 1, not UTF-8, to interoperate 
correctly
 val data = 
args(1).asInstanceOf[String].getBytes(StandardCharsets.ISO_8859_1)
-construct(typecode, machineCodes(typecode), data)
+val machine_code = machineCodes(typecode)
+// fix data alignment
+val unit_length = if (machine_code==18 || machine_code==19) 2 else 
4
+val aligned_data = data ++ Array.fill[Byte](unit_length - 
data.length % unit_length)(0)
--- End diff --

Not done yet. I think this will only works on little endian


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18444: [SPARK-16542][SQL][PYSPARK] Fix bugs about types that re...

[zasdfgbnm]

Github user zasdfgbnm commented on the issue:

https://github.com/apache/spark/pull/18444
  
For some reason, I can not reproduce the error on my machine. I run the 
test using the following command:
```bash
PYSPARK_PYTHON=$(which python2) ./bin/spark-submit 
python/pyspark/sql/tests.py SQLTests.test_array_types 2>/dev/null
```
and I always get a pass...

So I have to commit and push to let Jenkins run the test to see if it will 
pass...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18444: [SPARK-16542][SQL][PYSPARK] Fix bugs about types that re...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18444
  
**[Test build #79540 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79540/testReport)**
 for PR 18444 at commit 
[`9ee439f`](https://github.com/apache/spark/commit/9ee439f74b88faa1e79cf55ac50b35f650fedca6).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org