[GitHub] spark issue #20024: [SPARK-22825][SQL] Fix incorrect results of Casting Arra...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/20024
  
I feel it's not the most effcient way to cast array to string by 
deserializing the catalyst array to java array. Instead, I think we should have 
a schema-aware string casting function, i.e. using `ArrayData.get` to get each 
element, and recursively cast each element to string, and concat them with `,`


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19813: [SPARK-22600][SQL] Fix 64kb limit for deeply nested expr...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/19813
  
To me whole stage codegen compilation fix is less important as we can 
fallback to non whole stage codegen, so we don't need to rush.

> As we don't use such statement as codegen output, I think the easiest 
approach is adding assertions

What about future? Will we need to output statement for some reason? like 
reducing the usage of local variables?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r158441506
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
 ---
@@ -56,7 +56,36 @@ import org.apache.spark.util.{ParentClassLoader, Utils}
  * @param value A term for a (possibly primitive) value of the result of 
the evaluation. Not
  *  valid if `isNull` is set to `true`.
  */
-case class ExprCode(var code: String, var isNull: String, var value: 
String)
+case class ExprCode(var code: String, var isNull: ExprValue, var value: 
ExprValue)
+
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue
--- End diff --

If no strong preference for combining them, I'd keep it as two concepts for 
now, if we foresee the need to distinguish them. 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20043: [SPARK-22856][SQL] Add wrappers for codegen output and n...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/20043
  
@gczsjdy By this, we can easily know what the output is. Is it a global 
variable, a local variable or a literal? For now, we just get a string. It 
makes us hard to act according to the type of codegen output. It's also hard to 
search the codes to find out where we output global variable, local variable or 
literal, BTW.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20024: [SPARK-22825][SQL] Fix incorrect results of Casting Arra...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/20024
  
cc @cloud-fan 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20052: [SPARK-20694][EXAMPLES]Update SQLDataSourceExample.scala

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/20052
  
Can you printed out the generated doc after the changes?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20025: [SPARK-22837][SQL]Session timeout checker does not work ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20025
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20025: [SPARK-22837][SQL]Session timeout checker does not work ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20025
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85299/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20025: [SPARK-22837][SQL]Session timeout checker does not work ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20025
  
**[Test build #85299 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85299/testReport)**
 for PR 20025 at commit 
[`34b7680`](https://github.com/apache/spark/commit/34b7680dfd915449087b8f5541c682b02df5dfbc).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20036: [SPARK-18016][SQL][FOLLOW-UP] Code Generation: Constant ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20036
  
**[Test build #85300 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85300/testReport)**
 for PR 20036 at commit 
[`8d8b669`](https://github.com/apache/spark/commit/8d8b669d4eca9181420d71b5c69f6ced8924b6e3).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20010: [SPARK-22826][SQL] findWiderTypeForTwo Fails over...

[gczsjdy]

Github user gczsjdy commented on a diff in the pull request:

https://github.com/apache/spark/pull/20010#discussion_r158440114
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
 ---
@@ -158,11 +169,6 @@ object TypeCoercion {
 findTightestCommonType(t1, t2)
   .orElse(findWiderTypeForDecimal(t1, t2))
   .orElse(stringPromotion(t1, t2))
-  .orElse((t1, t2) match {
-case (ArrayType(et1, containsNull1), ArrayType(et2, 
containsNull2)) =>
-  findWiderTypeForTwo(et1, et2).map(ArrayType(_, containsNull1 || 
containsNull2))
-case _ => None
-  })
--- End diff --

I like your implementation, except for the concerns I raised in another 
thread. Have you tested locally?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20010: [SPARK-22826][SQL] findWiderTypeForTwo Fails over...

[gczsjdy]

Github user gczsjdy commented on a diff in the pull request:

https://github.com/apache/spark/pull/20010#discussion_r158440005
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
 ---
@@ -158,11 +213,8 @@ object TypeCoercion {
 findTightestCommonType(t1, t2)
   .orElse(findWiderTypeForDecimal(t1, t2))
   .orElse(stringPromotion(t1, t2))
-  .orElse((t1, t2) match {
-case (ArrayType(et1, containsNull1), ArrayType(et2, 
containsNull2)) =>
-  findWiderTypeForTwo(et1, et2).map(ArrayType(_, containsNull1 || 
containsNull2))
-case _ => None
-  })
+  .orElse(findWiderTypeForTwoComplex(t1, t2, findWiderTypeForTwo))
--- End diff --

Will this cause infinite loop? Calling `findWiderTypeForTwo` in 
`findWiderTypeForTwo`?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20036: [SPARK-18016][SQL][FOLLOW-UP] Code Generation: Co...

[kiszk]

Github user kiszk commented on a diff in the pull request:

https://github.com/apache/spark/pull/20036#discussion_r158439773
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala
 ---
@@ -118,9 +118,8 @@ case class Like(left: Expression, right: Expression) 
extends StringRegexExpressi
   if (rVal != null) {
 val regexStr =
   
StringEscapeUtils.escapeJava(escape(rVal.asInstanceOf[UTF8String].toString()))
-// inline mutable state since not many Like operations in a task
 val pattern = ctx.addMutableState(patternClass, "patternLike",
-  v => s"""$v = ${patternClass}.compile("$regexStr");""", 
forceInline = true)
+  v => s"""$v = ${patternClass}.compile("$regexStr");""")
--- End diff --

Sure, done for other places, too.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20025: [SPARK-22837][SQL]Session timeout checker does not work ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20025
  
**[Test build #85299 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85299/testReport)**
 for PR 20025 at commit 
[`34b7680`](https://github.com/apache/spark/commit/34b7680dfd915449087b8f5541c682b02df5dfbc).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20025: [SPARK-22837][SQL]Session timeout checker does not work ...

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/20025
  
Jenkins, ok to test


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20025: [SPARK-22837][SQL]Session timeout checker does no...

[felixcheung]

Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/20025#discussion_r158438561
  
--- Diff: 
sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/SessionManager.java
 ---
@@ -148,6 +148,15 @@ public synchronized void start() {
 }
   }
 
+  private void initSessionTimeoutCheckerConfig() {
--- End diff --

should we refactor createBackgroundOperationPool to call this?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20050: [SPARK-22864][core] Disable allocation schedule i...

[felixcheung]

Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/20050#discussion_r158438115
  
--- Diff: 
core/src/test/scala/org/apache/spark/ExecutorAllocationManagerSuite.scala ---
@@ -1063,6 +1067,9 @@ class ExecutorAllocationManagerSuite
 s"${sustainedSchedulerBacklogTimeout.toString}s")
   .set("spark.dynamicAllocation.executorIdleTimeout", 
s"${executorIdleTimeout.toString}s")
   .set("spark.dynamicAllocation.testing", "true")
+  // SPARK-22864: effectively disable the allocation schedule by 
setting the period to a
+  // really long value.
+  .set(TESTING_SCHEDULE_INTERVAL_KEY, "1")
--- End diff --

this is 10sec? it's not really that long...


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19683: [SPARK-21657][SQL] optimize explode quadratic memory con...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/19683
  
the idea LGTM, it's a special column pruning inside Generate operator.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19683: [SPARK-21657][SQL] optimize explode quadratic mem...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/19683#discussion_r158436989
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala ---
@@ -474,10 +474,10 @@ abstract class SparkStrategies extends 
QueryPlanner[SparkPlan] {
 execution.GlobalLimitExec(limit, planLater(child)) :: Nil
   case logical.Union(unionChildren) =>
 execution.UnionExec(unionChildren.map(planLater)) :: Nil
-  case g @ logical.Generate(generator, join, outer, _, _, child) =>
+  case g @ logical.Generate(generator, join, outer, 
omitGeneratorChild, _, _, child) =>
 execution.GenerateExec(
-  generator, join = join, outer = outer, 
g.qualifiedGeneratorOutput,
-  planLater(child)) :: Nil
+  generator, join = join, outer = outer, omitGeneratorChild = 
omitGeneratorChild,
--- End diff --

`omitGeneratorChild = omitGeneratorChild` looks redundant, we can just pass 
`omitGeneratorChild`


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19683: [SPARK-21657][SQL] optimize explode quadratic mem...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/19683#discussion_r158436905
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/MiscBenchmark.scala
 ---
@@ -227,4 +227,31 @@ class MiscBenchmark extends BenchmarkBase {
 generate stack wholestage on   836 /  847 20.1 
 49.8  15.5X
  */
   }
+
+  ignore("generate explode big struct array") {
+val BASE = 1234567890
+val N = 4
+
+val spark = sparkSession
+import spark.implicits._
+import org.apache.spark.sql.functions._
+
+val df = sparkSession.sparkContext.parallelize(
+  List(("1234567890", (BASE to (BASE + N)).map(
+x => (x.toString, (x + 1).toString, (x + 2).toString, (x + 
3).toString)).toArray)))
+  .toDF("c1", "c_arr")
+  .select(col("c1"), explode(col("c_arr")))
+  .selectExpr("c1", "col.*")
+
+df.cache.count
--- End diff --

can you follow the existing benchmark style in this file?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19683: [SPARK-21657][SQL] optimize explode quadratic mem...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/19683#discussion_r158436824
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/MiscBenchmark.scala
 ---
@@ -227,4 +227,31 @@ class MiscBenchmark extends BenchmarkBase {
 generate stack wholestage on   836 /  847 20.1 
 49.8  15.5X
  */
   }
+
+  ignore("generate explode big struct array") {
+val BASE = 1234567890
--- End diff --

why need this?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19683: [SPARK-21657][SQL] optimize explode quadratic mem...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/19683#discussion_r158436801
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala ---
@@ -86,10 +96,24 @@ case class GenerateExec(
 child.execute().mapPartitionsWithIndexInternal { (index, iter) =>
   val generatorNullRow = new 
GenericInternalRow(generator.elementSchema.length)
   val rows = if (join) {
+
+lazy val project = UnsafeProjection.create(
+  projectedChildOutput,
+  child.output,
+  subexpressionEliminationEnabled)
+
 val joinedRow = new JoinedRow
 iter.flatMap { row =>
+
+  val projectedRow = if (omitGeneratorChild) {
+project.initialize(index)
+project(row)
+  } else {
+row
+  }
+
   // we should always set the left (child output)
-  joinedRow.withLeft(row)
+  joinedRow.withLeft(projectedRow)
--- End diff --

here just `jonedRow.withLeft(pruneChildForResult(row))`


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19683: [SPARK-21657][SQL] optimize explode quadratic mem...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/19683#discussion_r158436751
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala ---
@@ -86,10 +96,24 @@ case class GenerateExec(
 child.execute().mapPartitionsWithIndexInternal { (index, iter) =>
   val generatorNullRow = new 
GenericInternalRow(generator.elementSchema.length)
   val rows = if (join) {
+
+lazy val project = UnsafeProjection.create(
+  projectedChildOutput,
+  child.output,
+  subexpressionEliminationEnabled)
--- End diff --

we can simplify the code and centralize all the logic here
```
val pruneChildForResult: InternalRow => InternalRow = if 
(omitGeneratorReferences) {
  val generatorReferences = generator.references
  val requiredChildOutput = 
child.output.filterNot(generatorReferences.contains)
  UnsafeProjection.create(requiredChildOutput, child.output)
} else {
  identity
}
```


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20051: [SPARK-22866] [K8S] Fix path issue in Kubernetes dockerf...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20051
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20051: [SPARK-22866] [K8S] Fix path issue in Kubernetes dockerf...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20051
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85294/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20051: [SPARK-22866] [K8S] Fix path issue in Kubernetes dockerf...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20051
  
**[Test build #85294 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85294/testReport)**
 for PR 20051 at commit 
[`f631331`](https://github.com/apache/spark/commit/f6313311b3ba1a589abde6bb534daad2c6860005).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19683: [SPARK-21657][SQL] optimize explode quadratic mem...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/19683#discussion_r158436519
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala ---
@@ -86,10 +96,24 @@ case class GenerateExec(
 child.execute().mapPartitionsWithIndexInternal { (index, iter) =>
   val generatorNullRow = new 
GenericInternalRow(generator.elementSchema.length)
   val rows = if (join) {
+
+lazy val project = UnsafeProjection.create(
+  projectedChildOutput,
+  child.output,
+  subexpressionEliminationEnabled)
--- End diff --

no need to set this. It's a very simple projection that only deal with 
attributes


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20053: [SPARK-22873] [CORE] Init lastReportTimestamp with syste...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20053
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19683: [SPARK-21657][SQL] optimize explode quadratic mem...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/19683#discussion_r158436309
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala ---
@@ -59,13 +61,21 @@ case class GenerateExec(
 generator: Generator,
 join: Boolean,
 outer: Boolean,
+omitGeneratorChild: Boolean,
 generatorOutput: Seq[Attribute],
 child: SparkPlan)
   extends UnaryExecNode with CodegenSupport {
 
+  private def projectedChildOutput = generator match {
+case g: UnaryExpression if omitGeneratorChild =>
--- End diff --

why limit to `UnaryExpression`? Think about if we have an array concat 
function in the future, and when we do `explode(array_concat(col1, col2))`, we 
should be able to omit both `col1` and `col2`.

I'd like to add a `omitGeneratorReferences` parameter, and here can be 
simplified to
```
private def requiredChildOutput = if (omitGeneratorReferences) {
  val generatorReferences = generator.references
  child.output.filterNot(generatorReferences.contains)
} else {
  child.output
}
```


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20053: Init lastReportTimestamp with system current time...

[Ngone51]

GitHub user Ngone51 opened a pull request:

https://github.com/apache/spark/pull/20053

Init lastReportTimestamp with system current time when start() called…

… in AsyncEventQueue

## What changes were proposed in this pull request?
```
 if (droppedEventsCounter.compareAndSet(droppedCount, 0)) {
  val prevLastReportTimestamp = lastReportTimestamp
  lastReportTimestamp = System.currentTimeMillis()
  val previous = new java.util.Date(prevLastReportTimestamp)
  logWarning(s"Dropped $droppedEvents events from $name since 
$previous.")
   }
```
First time we log previous date, it would be "Thu Jan 01 08:00:00 CST 1970" 
because of lastReportTimestamp was inited by 0L. Although there is no mistake 
in theory, AsyncEventQueue's starting time seems better:
```
private[scheduler] def start(sc: SparkContext): Unit = {
if (started.compareAndSet(false, true)) {
  this.sc = sc
  lastReportTimestamp = System.currentTimeMillis()
  dispatchThread.start()
} else {
  throw new IllegalStateException(s"$name already started!")
}
  }

```
## How was this patch tested?

manual test:
Debug unit test '  test("metrics for dropped listener events") ' in 
SparkListenerSuit to check the previous date first time we log for drop events.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Ngone51/spark SPARK-22873

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/20053.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #20053


commit 64facd95611ec1a5f3a51b456ff8e48ecefa9a93
Author: wuyi 
Date:   2017-12-22T06:25:36Z

Init lastReportTimestamp with system current time when start() called in 
AsyncEventQueue




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to...

[holdenk]

Github user holdenk commented on a diff in the pull request:

https://github.com/apache/spark/pull/19876#discussion_r158434850
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala ---
@@ -554,7 +555,49 @@ class LinearRegressionModel private[ml] (
* This also does not save the [[parent]] currently.
*/
   @Since("1.6.0")
-  override def write: MLWriter = new 
LinearRegressionModel.LinearRegressionModelWriter(this)
+  override def write: GeneralMLWriter = new GeneralMLWriter(this)
--- End diff --

fixed


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20052: [SPARK-20694][EXAMPLES]Update SQLDataSourceExampl...

[CNRui]

GitHub user CNRui opened a pull request:

https://github.com/apache/spark/pull/20052

[SPARK-20694][EXAMPLES]Update SQLDataSourceExample.scala

## What changes were proposed in this pull request?

Create table using the right DataFrame. peopleDF->usersDF

peopleDF:
++---+
| age|   name|
++---+
usersDF:
+--+--++
|  name|favorite_color|favorite_numbers|
+--+--++

## How was this patch tested?

Manually tested.




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/CNRui/spark patch-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/20052.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #20052


commit 770d6146e0dc18e49289efce64bf80d44fd21e74
Author: CNRui <13266776177@...>
Date:   2017-12-22T06:03:56Z

Update SQLDataSourceExample.scala

Create table using the right DataFrame. peopleDF->usersDF




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20052: [SPARK-20694][EXAMPLES]Update SQLDataSourceExample.scala

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20052
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20043: [SPARK-22856][SQL] Add wrappers for codegen output and n...

[gczsjdy]

Github user gczsjdy commented on the issue:

https://github.com/apache/spark/pull/20043
  
@viirya Sorry I didn't quite understand, how do we easily know the value by 
adding wrappers? Could you explain a little bit?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18754: [WIP][SPARK-21552][SQL] Add DecimalType support to Arrow...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18754
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85293/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18754: [WIP][SPARK-21552][SQL] Add DecimalType support to Arrow...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18754
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20024: [SPARK-22825][SQL] Fix incorrect results of Casting Arra...

[maropu]

Github user maropu commented on the issue:

https://github.com/apache/spark/pull/20024
  
ping


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18754: [WIP][SPARK-21552][SQL] Add DecimalType support to Arrow...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18754
  
**[Test build #85293 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85293/testReport)**
 for PR 18754 at commit 
[`e29f833`](https://github.com/apache/spark/commit/e29f8330e455e6bf66d2c69f2875a70ae2c71cdb).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20048: [SPARK-22862] Docs on lazy elimination of columns...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/20048


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20048: [SPARK-22862] Docs on lazy elimination of columns missin...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/20048
  
Thanks! Merged to master/2.2./2.1


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20036: [SPARK-18016][SQL][FOLLOW-UP] Code Generation: Co...

[ueshin]

Github user ueshin commented on a diff in the pull request:

https://github.com/apache/spark/pull/20036#discussion_r158430491
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala
 ---
@@ -194,9 +193,8 @@ case class RLike(left: Expression, right: Expression) 
extends StringRegexExpress
   if (rVal != null) {
 val regexStr =
   
StringEscapeUtils.escapeJava(rVal.asInstanceOf[UTF8String].toString())
-// inline mutable state since not many RLike operations in a task
 val pattern = ctx.addMutableState(patternClass, "patternRLike",
-  v => s"""$v = ${patternClass}.compile("$regexStr");""", 
forceInline = true)
+  v => s"""$v = ${patternClass}.compile("$regexStr");""")
--- End diff --

ditto.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20036: [SPARK-18016][SQL][FOLLOW-UP] Code Generation: Co...

[ueshin]

Github user ueshin commented on a diff in the pull request:

https://github.com/apache/spark/pull/20036#discussion_r158430481
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala
 ---
@@ -118,9 +118,8 @@ case class Like(left: Expression, right: Expression) 
extends StringRegexExpressi
   if (rVal != null) {
 val regexStr =
   
StringEscapeUtils.escapeJava(escape(rVal.asInstanceOf[UTF8String].toString()))
-// inline mutable state since not many Like operations in a task
 val pattern = ctx.addMutableState(patternClass, "patternLike",
-  v => s"""$v = ${patternClass}.compile("$regexStr");""", 
forceInline = true)
+  v => s"""$v = ${patternClass}.compile("$regexStr");""")
--- End diff --

nit: we can remove `{` and `}` around `patternClass`.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20026: [SPARK-22838][Core] Avoid unnecessary copying of data

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20026
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20026: [SPARK-22838][Core] Avoid unnecessary copying of data

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20026
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85292/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20026: [SPARK-22838][Core] Avoid unnecessary copying of data

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20026
  
**[Test build #85292 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85292/testReport)**
 for PR 20026 at commit 
[`b412b54`](https://github.com/apache/spark/commit/b412b54f56097a09d8b1683c9db4afa741206613).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19977: [SPARK-22771][SQL] Concatenate binary inputs into a bina...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19977
  
**[Test build #85298 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85298/testReport)**
 for PR 19977 at commit 
[`9f5157c`](https://github.com/apache/spark/commit/9f5157c70fdbb9362c7ee8691c86ab239fe450ae).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #11994: [SPARK-14151] Expose metrics Source and Sink interface

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/11994
  
What's our thought on this, @jerryshao ?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19813: [SPARK-22600][SQL] Fix 64kb limit for deeply nested expr...

[kiszk]

Github user kiszk commented on the issue:

https://github.com/apache/spark/pull/19813
  
I think that this PR is necessary to fix SPARK-22868 and SPARK-22869


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20051: [SPARK-22866] [K8S] Fix path issue in Kubernetes ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/20051


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20051: [SPARK-22866] [K8S] Fix path issue in Kubernetes dockerf...

[mridulm]

Github user mridulm commented on the issue:

https://github.com/apache/spark/pull/20051
  
Merged, thanks @foxish 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20043: [SPARK-22856][SQL] Add wrappers for codegen output and n...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/20043
  
Thanks.

If we have forseen this need in near future, do we bother to combine them
here and separate it again later?

On Dec 22, 2017 12:02 PM, "Kazuaki Ishizaki" 
wrote:

*@kiszk* commented on this pull request.
--

In sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/
expressions/codegen/CodeGenerator.scala
:

> @@ -56,7 +56,36 @@ import org.apache.spark.util.{ParentClassLoader, Utils}
  * @param value A term for a (possibly primitive) value of the result
of the evaluation. Not
  *  valid if `isNull` is set to `true`.
  */
-case class ExprCode(var code: String, var isNull: String, var value: 
String)
+case class ExprCode(var code: String, var isNull: ExprValue, var
value: ExprValue)
+
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue

In summary, I have no strong preference.

In the future, we will want to distinguish Literal and Global for some
optimizations. This


is already one of optimizations for Literal.

If this PR just focuses on classifying types between arguments and
non-arguments, it is fine to combine Literal and Global. Then, another PR
will separate one type into Literal and Global.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
, or mute
the thread


.



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19977: [SPARK-22771][SQL] Concatenate binary inputs into...

[maropu]

Github user maropu commented on a diff in the pull request:

https://github.com/apache/spark/pull/19977#discussion_r158429095
  
--- Diff: 
sql/core/src/test/resources/sql-tests/results/string-functions.sql.out ---
@@ -42,7 +42,7 @@ struct
 
 == Analyzed Logical Plan ==
 col: string
-Project [concat(concat(concat(cast(col1#xL as string), cast(col2#xL as 
string)), cast(col3#xL as string)), cast(col4#xL as string)) AS col#x]
+Project [concat(cast(concat(cast(concat(cast(col1#xL as string), 
cast(col2#xL as string)) as string), cast(col3#xL as string)) as string), 
cast(col4#xL as string)) AS col#x]
--- End diff --

aha, it's bug.. I'll fix soon.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19977: [SPARK-22771][SQL] Concatenate binary inputs into...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/19977#discussion_r158428849
  
--- Diff: 
sql/core/src/test/resources/sql-tests/results/string-functions.sql.out ---
@@ -42,7 +42,7 @@ struct
 
 == Analyzed Logical Plan ==
 col: string
-Project [concat(concat(concat(cast(col1#xL as string), cast(col2#xL as 
string)), cast(col3#xL as string)), cast(col4#xL as string)) AS col#x]
+Project [concat(cast(concat(cast(concat(cast(col1#xL as string), 
cast(col2#xL as string)) as string), cast(col3#xL as string)) as string), 
cast(col4#xL as string)) AS col#x]
--- End diff --

why this change?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20051: [SPARK-22866] [K8S] Fix path issue in Kubernetes dockerf...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20051
  
**[Test build #85297 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85297/testReport)**
 for PR 20051 at commit 
[`657b48a`](https://github.com/apache/spark/commit/657b48a11b8d0f62763f2fa863c0e265280ba63d).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20051: [SPARK-22866] [K8S] Fix path issue in Kubernetes dockerf...

[foxish]

Github user foxish commented on the issue:

https://github.com/apache/spark/pull/20051
  
Done, comments updated.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19977: [SPARK-22771][SQL] Concatenate binary inputs into a bina...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19977
  
**[Test build #85296 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85296/testReport)**
 for PR 19977 at commit 
[`7c93dbc`](https://github.com/apache/spark/commit/7c93dbc47725426f04039a0dce3371dcb1e31a30).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19977: [SPARK-22771][SQL] Concatenate binary inputs into a bina...

[maropu]

Github user maropu commented on the issue:

https://github.com/apache/spark/pull/19977
  
ok, all comments addressed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19977: [SPARK-22771][SQL] Concatenate binary inputs into a bina...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19977
  
**[Test build #85295 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85295/testReport)**
 for PR 19977 at commit 
[`7c055b0`](https://github.com/apache/spark/commit/7c055b0f94f01e83171875d19ebdf98968bebc3d).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20051: [SPARK-22866] [K8S] Fix path issue in Kubernetes dockerf...

[foxish]

Github user foxish commented on the issue:

https://github.com/apache/spark/pull/20051
  
@mridulm thanks for looking into it! Looks like I missed updating the 
dockerfile comments. On it.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20043: [SPARK-22856][SQL] Add wrappers for codegen output and n...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20043
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20043: [SPARK-22856][SQL] Add wrappers for codegen output and n...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20043
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85291/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20051: [SPARK-22866] [K8S] Fix path issue in Kubernetes dockerf...

[mridulm]

Github user mridulm commented on the issue:

https://github.com/apache/spark/pull/20051
  
A grep indicates this is the full list:

* 
./resource-managers/kubernetes/docker/src/main/dockerfiles/driver/Dockerfile:# 
docker build -t spark-driver:latest -f dockerfiles/spark-base/Dockerfile .
* 
./resource-managers/kubernetes/docker/src/main/dockerfiles/spark-base/Dockerfile:#
 docker build -t spark-base:latest -f dockerfiles/spark-base/Dockerfile .
* 
./resource-managers/kubernetes/docker/src/main/dockerfiles/spark-base/Dockerfile:COPY
 dockerfiles/spark-base/entrypoint.sh /opt/
* 
./resource-managers/kubernetes/docker/src/main/dockerfiles/executor/Dockerfile:#
 docker build -t spark-executor:latest -f dockerfiles/spark-base/Dockerfile .
* ./dev/make-distribution.sh:# Only create and copy the dockerfiles 
directory if the kubernetes artifacts were built.
* ./dev/make-distribution.sh:  cp -a 
"$SPARK_HOME"/resource-managers/kubernetes/docker/src/main/dockerfiles 
"$DISTDIR/kubernetes/"





---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20043: [SPARK-22856][SQL] Add wrappers for codegen output and n...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20043
  
**[Test build #85291 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85291/testReport)**
 for PR 20043 at commit 
[`4384c84`](https://github.com/apache/spark/commit/4384c84bd039addab34ba126447462ea87e13734).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20051: [SPARK-22866] [K8S] Fix path issue in Kubernetes ...

[foxish]

Github user foxish commented on a diff in the pull request:

https://github.com/apache/spark/pull/20051#discussion_r158427619
  
--- Diff: 
resource-managers/kubernetes/docker/src/main/dockerfiles/spark-base/Dockerfile 
---
@@ -38,7 +38,7 @@ COPY jars /opt/spark/jars
 COPY bin /opt/spark/bin
 COPY sbin /opt/spark/sbin
 COPY conf /opt/spark/conf
-COPY dockerfiles/spark-base/entrypoint.sh /opt/
+COPY kubernetes/dockerfiles/spark-base/entrypoint.sh /opt/
--- End diff --

Yes, to elaborate, on our fork, we used to have `dockerfiles/` in the root 
of the distribution. But now, it's copied to `kubernetes/dockerfiles` which is 
a more logical place, since there is a `yarn/` in the distribution as well.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20051: [SPARK-22866] [K8S] Fix path issue in Kubernetes ...

[mridulm]

Github user mridulm commented on a diff in the pull request:

https://github.com/apache/spark/pull/20051#discussion_r158427437
  
--- Diff: 
resource-managers/kubernetes/docker/src/main/dockerfiles/spark-base/Dockerfile 
---
@@ -38,7 +38,7 @@ COPY jars /opt/spark/jars
 COPY bin /opt/spark/bin
 COPY sbin /opt/spark/sbin
 COPY conf /opt/spark/conf
-COPY dockerfiles/spark-base/entrypoint.sh /opt/
+COPY kubernetes/dockerfiles/spark-base/entrypoint.sh /opt/
--- End diff --

Do you mean this ? 
https://github.com/apache/spark/pull/19946/files#diff-cb1784535ec45c5b890bb16c5971be65R26


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19893
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19893
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85290/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19893
  
**[Test build #85290 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85290/testReport)**
 for PR 19893 at commit 
[`ef00796`](https://github.com/apache/spark/commit/ef007969a576dd5f84440ce78a325d6617dc5278).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...

[kiszk]

Github user kiszk commented on a diff in the pull request:

https://github.com/apache/spark/pull/20043#discussion_r158426351
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
 ---
@@ -56,7 +56,36 @@ import org.apache.spark.util.{ParentClassLoader, Utils}
  * @param value A term for a (possibly primitive) value of the result of 
the evaluation. Not
  *  valid if `isNull` is set to `true`.
  */
-case class ExprCode(var code: String, var isNull: String, var value: 
String)
+case class ExprCode(var code: String, var isNull: ExprValue, var value: 
ExprValue)
+
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue
--- End diff --

In summary, I have no strong preference.

In the future, we will want to distinguish `Literal` and `Global` for some 
optimizations. 
[This](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala#L121)
 is already one of optimizations for `Literal`.

If this PR just focuses on classifying types between arguments and 
non-arguments, it is fine to combine `Literal` and `Global`. Then, another PR 
will separate one type into `Literal` and `Global`. 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20039: [SPARK-22850][core] Ensure queued events are deli...

[Ngone51]

Github user Ngone51 commented on a diff in the pull request:

https://github.com/apache/spark/pull/20039#discussion_r158424923
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala ---
@@ -149,7 +158,11 @@ private[spark] class LiveListenerBus(conf: SparkConf) {
 }
 
 this.sparkContext = sc
-queues.asScala.foreach(_.start(sc))
+queues.asScala.foreach { q =>
+  q.start(sc)
+  queuedEvents.foreach(q.post)
--- End diff --

```
q.start(sc)
queuedEvents.foreach(q.post)
```
Ummm... In my opinion, exchange these two lines sequence would be better 
for following the original logic of events buffered before a queue calls 
start(). So, queuedEvents post to queues first before queues start would be 
unified logically.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20039: [SPARK-22850][core] Ensure queued events are deli...

[gczsjdy]

Github user gczsjdy commented on a diff in the pull request:

https://github.com/apache/spark/pull/20039#discussion_r158424211
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala ---
@@ -124,13 +127,19 @@ private[spark] class LiveListenerBus(conf: SparkConf) 
{
   }
 
   /** Post an event to all queues. */
-  def post(event: SparkListenerEvent): Unit = {
-if (!stopped.get()) {
-  metrics.numEventsPosted.inc()
+  def post(event: SparkListenerEvent): Unit = synchronized {
+if (stopped.get()) {
+  return
+}
--- End diff --

Is `synchronized` needed only due to the modification of `queuedEvents`?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19984: [SPARK-22789] Map-only continuous processing execution

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19984
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19984: [SPARK-22789] Map-only continuous processing execution

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19984
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85287/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20013: [SPARK-20657][core] Speed up rendering of the stages pag...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20013
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85289/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19984: [SPARK-22789] Map-only continuous processing execution

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19984
  
**[Test build #85287 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85287/testReport)**
 for PR 19984 at commit 
[`07a9e06`](https://github.com/apache/spark/commit/07a9e0654df61ad52f7db28ee663a380dee3c2a8).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20013: [SPARK-20657][core] Speed up rendering of the stages pag...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20013
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19872: WIP: [SPARK-22274][PySpark] User-defined aggregat...

[ueshin]

Github user ueshin commented on a diff in the pull request:

https://github.com/apache/spark/pull/19872#discussion_r158423362
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/python/AggregateInPandasExec.scala
 ---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.python
+
+import java.io.File
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.{SparkEnv, TaskContext}
+import org.apache.spark.api.python.{ChainedPythonFunctions, PythonEvalType}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Ascending, Attribute, 
AttributeSet, Expression, JoinedRow, NamedExpression, PythonUDF, SortOrder, 
UnsafeProjection, UnsafeRow}
+import org.apache.spark.sql.catalyst.plans.physical.{AllTuples, 
ClusteredDistribution, Distribution, Partitioning}
+import org.apache.spark.sql.execution.{GroupedIterator, SparkPlan, 
UnaryExecNode}
+import org.apache.spark.sql.types.{DataType, StructField, StructType}
+import org.apache.spark.util.Utils
+
+case class AggregateInPandasExec(
+groupingExpressions: Seq[Expression],
+udfExpressions: Seq[PythonUDF],
+resultExpressions: Seq[NamedExpression],
+child: SparkPlan)
+  extends UnaryExecNode {
+
+  override def output: Seq[Attribute] = 
resultExpressions.map(_.toAttribute)
+
+  override def outputPartitioning: Partitioning = child.outputPartitioning
+
+  override def producedAttributes: AttributeSet = AttributeSet(output)
+
+  override def requiredChildDistribution: Seq[Distribution] = {
+if (groupingExpressions.isEmpty) {
+  AllTuples :: Nil
+} else {
+  ClusteredDistribution(groupingExpressions) :: Nil
+}
+  }
+
+  private def collectFunctions(udf: PythonUDF): (ChainedPythonFunctions, 
Seq[Expression]) = {
+udf.children match {
+  case Seq(u: PythonUDF) =>
+val (chained, children) = collectFunctions(u)
+(ChainedPythonFunctions(chained.funcs ++ Seq(udf.func)), children)
+  case children =>
+// There should not be any other UDFs, or the children can't be 
evaluated directly.
+assert(children.forall(_.find(_.isInstanceOf[PythonUDF]).isEmpty))
+(ChainedPythonFunctions(Seq(udf.func)), udf.children)
+}
+  }
+
+  override def requiredChildOrdering: Seq[Seq[SortOrder]] =
+Seq(groupingExpressions.map(SortOrder(_, Ascending)))
+
+  override protected def doExecute(): RDD[InternalRow] = {
+val inputRDD = child.execute()
+
+val bufferSize = inputRDD.conf.getInt("spark.buffer.size", 65536)
+val reuseWorker = 
inputRDD.conf.getBoolean("spark.python.worker.reuse", defaultValue = true)
+val sessionLocalTimeZone = conf.sessionLocalTimeZone
+val pandasRespectSessionTimeZone = conf.pandasRespectSessionTimeZone
+
+val (pyFuncs, inputs) = udfExpressions.map(collectFunctions).unzip
+
+val allInputs = new ArrayBuffer[Expression]
+val dataTypes = new ArrayBuffer[DataType]
+
+allInputs.appendAll(groupingExpressions)
+
+val argOffsets = inputs.map { input =>
+  input.map { e =>
+allInputs += e
+dataTypes += e.dataType
+allInputs.length - 1 - groupingExpressions.length
+  }.toArray
+}.toArray
+
+val schema = StructType(dataTypes.zipWithIndex.map { case (dt, i) =>
+  StructField(s"_$i", dt)
+})
+
+inputRDD.mapPartitionsInternal { iter =>
+  val grouped = if (groupingExpressions.isEmpty) {
+Iterator((null, iter))
+  } else {
+val groupedIter = GroupedIterator(iter, groupingExpressions, 
child.output)
+
+val dropGrouping =
+  
UnsafeProjection.create(allInputs.drop(groupingExpressions.length), 
child.output)
+
+

[GitHub] spark issue #20013: [SPARK-20657][core] Speed up rendering of the stages pag...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20013
  
**[Test build #85289 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85289/testReport)**
 for PR 20013 at commit 
[`429d12e`](https://github.com/apache/spark/commit/429d12e121841b7f1d051a89632de7449603b665).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20051: [SPARK-22866] [K8S] Fix path issue in Kubernetes dockerf...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20051
  
**[Test build #85294 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85294/testReport)**
 for PR 20051 at commit 
[`f631331`](https://github.com/apache/spark/commit/f6313311b3ba1a589abde6bb534daad2c6860005).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20051: [SPARK-22866] [K8S] Fix path issue in Kubernetes ...

[foxish]

GitHub user foxish opened a pull request:

https://github.com/apache/spark/pull/20051

[SPARK-22866] [K8S] Fix path issue in Kubernetes dockerfile

## What changes were proposed in this pull request?

The path was recently changed in 
https://github.com/apache/spark/pull/19946, but the dockerfile was not updated.

## How was this patch tested?

`./sbin/build-push-docker-images.sh -r spark-repo -t latest build`

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/foxish/spark patch-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/20051.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #20051


commit f6313311b3ba1a589abde6bb534daad2c6860005
Author: Anirudh Ramanathan 
Date:   2017-12-22T03:05:33Z

[SPARK-22866] [K8S] Fix path issue in Kubernetes dockerfile

The path was recently changed in 
https://github.com/apache/spark/pull/19946, but the dockerfile was not updated.




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18754: [WIP][SPARK-21552][SQL] Add DecimalType support t...

[ueshin]

Github user ueshin commented on a diff in the pull request:

https://github.com/apache/spark/pull/18754#discussion_r158422348
  
--- Diff: python/pyspark/sql/types.py ---
@@ -1617,7 +1617,7 @@ def to_arrow_type(dt):
 elif type(dt) == DoubleType:
 arrow_type = pa.float64()
 elif type(dt) == DecimalType:
-arrow_type = pa.decimal(dt.precision, dt.scale)
+arrow_type = pa.decimal128(dt.precision, dt.scale)
--- End diff --

@wesm @BryanCutler Is this a right way to define decimal type for Arrow?
I also wonder if there is a limit for precision and scale?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18754: [WIP][SPARK-21552][SQL] Add DecimalType support to Arrow...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18754
  
**[Test build #85293 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85293/testReport)**
 for PR 18754 at commit 
[`e29f833`](https://github.com/apache/spark/commit/e29f8330e455e6bf66d2c69f2875a70ae2c71cdb).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19950: [SPARK-22450][Core][MLLib][FollowUp] safely regis...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/19950


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19950: [SPARK-22450][Core][MLLib][FollowUp] safely register cla...

[srowen]

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/19950
  
Merged to master


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

[gengliangwang]

Github user gengliangwang commented on the issue:

https://github.com/apache/spark/pull/20020
  
@gatorsmile This PR focus on the commands that using `FileFormatWriter`. I 
will investigate how to resolve the other Insertion commands and create a 
followup PR, is that OK?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20050: [SPARK-22864][core] Disable allocation schedule in Execu...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20050
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85286/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20050: [SPARK-22864][core] Disable allocation schedule in Execu...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20050
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20050: [SPARK-22864][core] Disable allocation schedule in Execu...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20050
  
**[Test build #85286 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85286/testReport)**
 for PR 20050 at commit 
[`0a72977`](https://github.com/apache/spark/commit/0a7297730b10f924fc511b2e47f8ae50ae9fdb1a).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19940: [SPARK-22750][SQL] Reuse mutable states when poss...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/19940


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19940: [SPARK-22750][SQL] Reuse mutable states when possible

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/19940
  
thanks, merging to master!


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20020: [SPARK-22834][SQL] Make insertion commands have r...

[gengliangwang]

Github user gengliangwang commented on a diff in the pull request:

https://github.com/apache/spark/pull/20020#discussion_r158416541
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
 ---
@@ -479,14 +468,15 @@ case class DataSource(
   outputPath = outputPath,
   staticPartitions = Map.empty,
   ifPartitionNotExists = false,
-  partitionColumns = partitionAttributes,
+  partitionColumns = partitionColumns.map(UnresolvedAttribute.quoted),
   bucketSpec = bucketSpec,
   fileFormat = format,
   options = options,
   query = data,
--- End diff --

I think @cloud-fan wants `AnalysisBarrier` to be in trait 
`DataWritingCommand` to make sure the `query` is analyzed.
Adding `AnalysisBarrier` will not be helpful, the query should be analyzed 
here.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20026: [SPARK-22838][Core] Avoid unnecessary copying of data

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20026
  
**[Test build #85292 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85292/testReport)**
 for PR 20026 at commit 
[`b412b54`](https://github.com/apache/spark/commit/b412b54f56097a09d8b1683c9db4afa741206613).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19884: [SPARK-22324][SQL][PYTHON] Upgrade Arrow to 0.8.0

[zsxwing]

Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/19884
  
@HyukjinKwon yeah, I closed the ticket.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20049: [SPARK-22854][ui] Read Spark version from event l...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/20049


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20043: [SPARK-22856][SQL] Add wrappers for codegen output and n...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20043
  
**[Test build #85291 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85291/testReport)**
 for PR 20043 at commit 
[`4384c84`](https://github.com/apache/spark/commit/4384c84bd039addab34ba126447462ea87e13734).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20049: [SPARK-22854][ui] Read Spark version from event logs.

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/20049
  
merging to master!


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20049: [SPARK-22854][ui] Read Spark version from event logs.

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/20049
  
LGTM


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19946: [SPARK-22648] [K8S] Spark on Kubernetes - Documen...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/19946


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19946: [SPARK-22648] [K8S] Spark on Kubernetes - Documentation

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/19946
  
Merging in master.



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20039: [SPARK-22850][core] Ensure queued events are delivered t...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20039
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org