[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24221: [SPARK-27248][SQL] `refreshTable` should recreate cache with same cache name and storage level
dongjoon-hyun commented on a change in pull request #24221: [SPARK-27248][SQL]
`refreshTable` should recreate cache with same cache name and storage level
URL: https://github.com/apache/spark/pull/24221#discussion_r285328795
##
File path: sql/core/src/test/scala/org/apache/spark/sql/test/SQLTestUtils.scala
##
@@ -248,57 +244,91 @@ private[sql] trait SQLTestUtilsBase
!spark.sessionState.catalog.functionExists(FunctionIdentifier(functionName)),
s"Function $functionName should have been dropped. But, it still
exists.")
}
-}
+)
}
/**
* Drops temporary view `viewNames` after calling `f`.
*/
protected def withTempView(viewNames: String*)(f: => Unit): Unit = {
-try f finally {
- // If the test failed part way, we don't want to mask the failure by
failing to remove
- // temp views that never got created.
- try viewNames.foreach(spark.catalog.dropTempView) catch {
-case _: NoSuchTableException =>
- }
-}
+tryWithFinallyBlock(f)(viewNames.foreach(spark.catalog.dropTempView))
}
/**
* Drops global temporary view `viewNames` after calling `f`.
*/
protected def withGlobalTempView(viewNames: String*)(f: => Unit): Unit = {
-try f finally {
- // If the test failed part way, we don't want to mask the failure by
failing to remove
- // global temp views that never got created.
- try viewNames.foreach(spark.catalog.dropGlobalTempView) catch {
-case _: NoSuchTableException =>
- }
-}
+tryWithFinallyBlock(f)(viewNames.foreach(spark.catalog.dropGlobalTempView))
}
/**
* Drops table `tableName` after calling `f`.
*/
protected def withTable(tableNames: String*)(f: => Unit): Unit = {
-try f finally {
+tryWithFinallyBlock(f)(
tableNames.foreach { name =>
spark.sql(s"DROP TABLE IF EXISTS $name")
}
-}
+)
}
/**
* Drops view `viewName` after calling `f`.
*/
protected def withView(viewNames: String*)(f: => Unit): Unit = {
-try f finally {
+tryWithFinallyBlock(f)(
viewNames.foreach { name =>
spark.sql(s"DROP VIEW IF EXISTS $name")
}
+)
+ }
+
+ /**
+ * Drops cache `cacheName` after calling `f`.
+ */
+ protected def withCache(cacheNames: String*)(f: => Unit): Unit = {
+tryWithFinallyBlock(f)(cacheNames.foreach(uncacheTable))
+ }
+
+ /**
+ * Executes the given tryBlock and then the given finallyBlock no matter
whether tryBlock throws
+ * an exception. If both tryBlock and finallyBlock throw exceptions, the
exception thrown
+ * from the finallyBlock with be added to the exception thrown from tryBlock
as a
+ * suppress exception. It helps to avoid masking the exception from tryBlock
with exception
+ * from finallyBlock
+ */
+ private def tryWithFinallyBlock(tryBlock: => Unit)(finallyBlock: => Unit):
Unit = {
Review comment:
You can do that kind of refactoring later in another PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24221: [SPARK-27248][SQL] `refreshTable` should recreate cache with same cache name and storage level
dongjoon-hyun commented on a change in pull request #24221: [SPARK-27248][SQL]
`refreshTable` should recreate cache with same cache name and storage level
URL: https://github.com/apache/spark/pull/24221#discussion_r285328776
##
File path: sql/core/src/test/scala/org/apache/spark/sql/test/SQLTestUtils.scala
##
@@ -248,57 +244,91 @@ private[sql] trait SQLTestUtilsBase
!spark.sessionState.catalog.functionExists(FunctionIdentifier(functionName)),
s"Function $functionName should have been dropped. But, it still
exists.")
}
-}
+)
}
/**
* Drops temporary view `viewNames` after calling `f`.
*/
protected def withTempView(viewNames: String*)(f: => Unit): Unit = {
-try f finally {
- // If the test failed part way, we don't want to mask the failure by
failing to remove
- // temp views that never got created.
- try viewNames.foreach(spark.catalog.dropTempView) catch {
-case _: NoSuchTableException =>
- }
-}
+tryWithFinallyBlock(f)(viewNames.foreach(spark.catalog.dropTempView))
}
/**
* Drops global temporary view `viewNames` after calling `f`.
*/
protected def withGlobalTempView(viewNames: String*)(f: => Unit): Unit = {
-try f finally {
- // If the test failed part way, we don't want to mask the failure by
failing to remove
- // global temp views that never got created.
- try viewNames.foreach(spark.catalog.dropGlobalTempView) catch {
-case _: NoSuchTableException =>
- }
-}
+tryWithFinallyBlock(f)(viewNames.foreach(spark.catalog.dropGlobalTempView))
}
/**
* Drops table `tableName` after calling `f`.
*/
protected def withTable(tableNames: String*)(f: => Unit): Unit = {
-try f finally {
+tryWithFinallyBlock(f)(
tableNames.foreach { name =>
spark.sql(s"DROP TABLE IF EXISTS $name")
}
-}
+)
}
/**
* Drops view `viewName` after calling `f`.
*/
protected def withView(viewNames: String*)(f: => Unit): Unit = {
-try f finally {
+tryWithFinallyBlock(f)(
viewNames.foreach { name =>
spark.sql(s"DROP VIEW IF EXISTS $name")
}
+)
+ }
+
+ /**
+ * Drops cache `cacheName` after calling `f`.
+ */
+ protected def withCache(cacheNames: String*)(f: => Unit): Unit = {
+tryWithFinallyBlock(f)(cacheNames.foreach(uncacheTable))
+ }
+
+ /**
+ * Executes the given tryBlock and then the given finallyBlock no matter
whether tryBlock throws
+ * an exception. If both tryBlock and finallyBlock throw exceptions, the
exception thrown
+ * from the finallyBlock with be added to the exception thrown from tryBlock
as a
+ * suppress exception. It helps to avoid masking the exception from tryBlock
with exception
+ * from finallyBlock
+ */
+ private def tryWithFinallyBlock(tryBlock: => Unit)(finallyBlock: => Unit):
Unit = {
Review comment:
Could you implement `withCache` simply like the other `withXXX` functions
instead of introducing `tryWithFinallyBlock` in this PR?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24221: [SPARK-27248][SQL] `refreshTable` should recreate cache with same cache name and storage level
dongjoon-hyun commented on a change in pull request #24221: [SPARK-27248][SQL]
`refreshTable` should recreate cache with same cache name and storage level
URL: https://github.com/apache/spark/pull/24221#discussion_r285328702
##
File path: sql/core/src/test/scala/org/apache/spark/sql/test/SQLTestUtils.scala
##
@@ -248,57 +244,91 @@ private[sql] trait SQLTestUtilsBase
!spark.sessionState.catalog.functionExists(FunctionIdentifier(functionName)),
s"Function $functionName should have been dropped. But, it still
exists.")
}
-}
+)
}
/**
* Drops temporary view `viewNames` after calling `f`.
*/
protected def withTempView(viewNames: String*)(f: => Unit): Unit = {
-try f finally {
- // If the test failed part way, we don't want to mask the failure by
failing to remove
- // temp views that never got created.
- try viewNames.foreach(spark.catalog.dropTempView) catch {
-case _: NoSuchTableException =>
- }
-}
+tryWithFinallyBlock(f)(viewNames.foreach(spark.catalog.dropTempView))
Review comment:
Is this equivalent? Previously, we swallow only `NoSuchTableException`.
Could you revert the last commit?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24631: [MINOR][CORE] Avoid hardcoded configs
dongjoon-hyun commented on a change in pull request #24631: [MINOR][CORE] Avoid
hardcoded configs
URL: https://github.com/apache/spark/pull/24631#discussion_r285328545
##
File path: core/src/main/scala/org/apache/spark/SparkConf.scala
##
@@ -667,12 +667,12 @@ private[spark] object SparkConf extends Logging {
translation = s => s"${s.toLong * 10}s")),
REDUCER_MAX_SIZE_IN_FLIGHT.key -> Seq(
AlternateConfig("spark.reducer.maxMbInFlight", "1.4")),
-"spark.kryoserializer.buffer" -> Seq(
Review comment:
You can use `git grep` to search these instances, @wenxuanguan .
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24631: [MINOR][CORE] Avoid hardcoded configs
dongjoon-hyun commented on a change in pull request #24631: [MINOR][CORE] Avoid
hardcoded configs
URL: https://github.com/apache/spark/pull/24631#discussion_r285328525
##
File path: core/src/main/scala/org/apache/spark/SparkConf.scala
##
@@ -667,12 +667,12 @@ private[spark] object SparkConf extends Logging {
translation = s => s"${s.toLong * 10}s")),
REDUCER_MAX_SIZE_IN_FLIGHT.key -> Seq(
AlternateConfig("spark.reducer.maxMbInFlight", "1.4")),
-"spark.kryoserializer.buffer" -> Seq(
Review comment:
For these `spark.kryoserializer.buffer` and
`spark.kryoserializer.buffer.max`, it seems there are too many instances.
```
core/src/test/scala/org/apache/spark/SparkConfSuite.scala:
assert(conf.getSizeAsKb("spark.kryoserializer.buffer") === 1100)
core/src/test/scala/org/apache/spark/storage/BlockManagerReplicationSuite.scala:
conf.set("spark.kryoserializer.buffer", "1m")
core/src/test/scala/org/apache/spark/storage/BlockManagerReplicationSuite.scala:
conf.set("spark.kryoserializer.buffer", "1m")
core/src/test/scala/org/apache/spark/storage/BlockManagerReplicationSuite.scala:
conf.set("spark.kryoserializer.buffer", "1m")
core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala: val
serializer = new KryoSerializer(new
SparkConf(false).set("spark.kryoserializer.buffer", "1m"))
core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala:
.set("spark.kryoserializer.buffer", "1m")
core/src/test/scala/org/apache/spark/storage/MemoryStoreSuite.scala: val
serializer = new KryoSerializer(new
SparkConf(false).set("spark.kryoserializer.buffer", "1m"))
examples/src/main/scala/org/apache/spark/examples/mllib/MovieLensALS.scala:
.set("spark.kryoserializer.buffer", "8m")
```
```
mllib/src/main/scala/org/apache/spark/ml/feature/Word2Vec.scala:
sc.conf.get("spark.kryoserializer.buffer.max", "64m"))
mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala:
spark.conf.get("spark.kryoserializer.buffer.max", "64m"))
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24631: [MINOR][CORE] Avoid hardcoded configs
dongjoon-hyun commented on a change in pull request #24631: [MINOR][CORE] Avoid
hardcoded configs
URL: https://github.com/apache/spark/pull/24631#discussion_r285328421
##
File path: core/src/main/scala/org/apache/spark/SparkConf.scala
##
@@ -595,7 +595,7 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable
with Logging with Seria
// it will almost always cause ExecutorLostFailure. See SPARK-22754.
require(executorTimeoutThresholdMs > executorHeartbeatIntervalMs, "The
value of " +
s"${networkTimeout}=${executorTimeoutThresholdMs}ms must be no less than
the value of " +
- s"spark.executor.heartbeatInterval=${executorHeartbeatIntervalMs}ms.")
+ s"${EXECUTOR_HEARTBEAT_INTERVAL.key}=${executorHeartbeatIntervalMs}ms.")
Review comment:
Could you update `SparkContextSuite.scala`, too?
```
- .set("spark.executor.heartbeatInterval", "1s")
+ .set(EXECUTOR_HEARTBEAT_INTERVAL.key, "1s")
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24631: [MINOR][CORE] Avoid hardcoded configs
dongjoon-hyun commented on a change in pull request #24631: [MINOR][CORE] Avoid
hardcoded configs
URL: https://github.com/apache/spark/pull/24631#discussion_r285328421
##
File path: core/src/main/scala/org/apache/spark/SparkConf.scala
##
@@ -595,7 +595,7 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable
with Logging with Seria
// it will almost always cause ExecutorLostFailure. See SPARK-22754.
require(executorTimeoutThresholdMs > executorHeartbeatIntervalMs, "The
value of " +
s"${networkTimeout}=${executorTimeoutThresholdMs}ms must be no less than
the value of " +
- s"spark.executor.heartbeatInterval=${executorHeartbeatIntervalMs}ms.")
+ s"${EXECUTOR_HEARTBEAT_INTERVAL.key}=${executorHeartbeatIntervalMs}ms.")
Review comment:
Could you update `SparkContextSuite.scala`, too?
```scala
- .set("spark.executor.heartbeatInterval", "1s")
+ .set(EXECUTOR_HEARTBEAT_INTERVAL.key, "1s")
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24633: [SPARK-27757][BUILD] Bump Jackson to 2.9.9
dongjoon-hyun commented on a change in pull request #24633: [SPARK-27757][BUILD] Bump Jackson to 2.9.9 URL: https://github.com/apache/spark/pull/24633#discussion_r285328249 ## File path: pom.xml ## @@ -166,7 +166,7 @@ true 1.9.13 -2.9.8 +2.9.9 Review comment: When this is uploaded to the Maven Central, please update the dependency with @srowen 's comment. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
AmplabJenkins removed a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493645644 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105510/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
SparkQA removed a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493645123 **[Test build #105510 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105510/testReport)** for PR 24499 at commit [`32cdfe3`](https://github.com/apache/spark/commit/32cdfe3d6b15cec3dc669257548fae67c2d31404). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
AmplabJenkins removed a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493645643 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
SparkQA commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493645636 **[Test build #105510 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105510/testReport)** for PR 24499 at commit [`32cdfe3`](https://github.com/apache/spark/commit/32cdfe3d6b15cec3dc669257548fae67c2d31404). * This patch **fails Java style tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
AmplabJenkins commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493645643 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
AmplabJenkins commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493645644 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105510/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
SparkQA commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493645123 **[Test build #105510 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105510/testReport)** for PR 24499 at commit [`32cdfe3`](https://github.com/apache/spark/commit/32cdfe3d6b15cec3dc669257548fae67c2d31404). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
AmplabJenkins commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493645022 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10767/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
AmplabJenkins commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493645020 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
AmplabJenkins removed a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493645020 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
AmplabJenkins removed a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493645022 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10767/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24630: [SPARK-27754][K8S] Introduce additional config (spark.kubernetes.driver.request.cores) for driver request cores for spark on k8s
AmplabJenkins removed a comment on issue #24630: [SPARK-27754][K8S] Introduce additional config (spark.kubernetes.driver.request.cores) for driver request cores for spark on k8s URL: https://github.com/apache/spark/pull/24630#issuecomment-493644612 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24630: [SPARK-27754][K8S] Introduce additional config (spark.kubernetes.driver.request.cores) for driver request cores for spark on k8s
AmplabJenkins commented on issue #24630: [SPARK-27754][K8S] Introduce additional config (spark.kubernetes.driver.request.cores) for driver request cores for spark on k8s URL: https://github.com/apache/spark/pull/24630#issuecomment-493644616 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105509/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24630: [SPARK-27754][K8S] Introduce additional config (spark.kubernetes.driver.request.cores) for driver request cores for spark on k8s
SparkQA removed a comment on issue #24630: [SPARK-27754][K8S] Introduce additional config (spark.kubernetes.driver.request.cores) for driver request cores for spark on k8s URL: https://github.com/apache/spark/pull/24630#issuecomment-493644031 **[Test build #105509 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105509/testReport)** for PR 24630 at commit [`ff688c2`](https://github.com/apache/spark/commit/ff688c23de7c716cd0d9484846cbd39cafc17a21). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24630: [SPARK-27754][K8S] Introduce additional config (spark.kubernetes.driver.request.cores) for driver request cores for spark on k8s
AmplabJenkins commented on issue #24630: [SPARK-27754][K8S] Introduce additional config (spark.kubernetes.driver.request.cores) for driver request cores for spark on k8s URL: https://github.com/apache/spark/pull/24630#issuecomment-493644612 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24630: [SPARK-27754][K8S] Introduce additional config (spark.kubernetes.driver.request.cores) for driver request cores for spark on k8s
AmplabJenkins removed a comment on issue #24630: [SPARK-27754][K8S] Introduce additional config (spark.kubernetes.driver.request.cores) for driver request cores for spark on k8s URL: https://github.com/apache/spark/pull/24630#issuecomment-493644616 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105509/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24630: [SPARK-27754][K8S] Introduce additional config (spark.kubernetes.driver.request.cores) for driver request cores for spark on k8s
SparkQA commented on issue #24630: [SPARK-27754][K8S] Introduce additional config (spark.kubernetes.driver.request.cores) for driver request cores for spark on k8s URL: https://github.com/apache/spark/pull/24630#issuecomment-493644593 **[Test build #105509 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105509/testReport)** for PR 24630 at commit [`ff688c2`](https://github.com/apache/spark/commit/ff688c23de7c716cd0d9484846cbd39cafc17a21). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #24635: [SPARK-27762][SQL] Support user provided avro schema for writing fields with different ordering
viirya commented on a change in pull request #24635: [SPARK-27762][SQL] Support
user provided avro schema for writing fields with different ordering
URL: https://github.com/apache/spark/pull/24635#discussion_r285327314
##
File path:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala
##
@@ -205,18 +205,28 @@ class AvroSerializer(rootCatalystType: DataType,
rootAvroType: Schema, nullable:
throw new IncompatibleSchemaException(s"Cannot convert Catalyst type
$catalystStruct to " +
s"Avro type $avroStruct.")
}
-val fieldConverters = catalystStruct.zip(avroStruct.getFields.asScala).map
{
- case (f1, f2) => newConverter(f1.dataType,
resolveNullableType(f2.schema(), f1.nullable))
-}
+
+val (avroIndices: Array[Int], fieldConverters: Array[Converter]) =
+ catalystStruct.map { catalystField =>
+val avroField = avroStruct.getField(catalystField.name)
+if (avroField == null) {
Review comment:
When converting catalyst struct "a int, b int" -> avro type "b int, c int",
we will update only the first avro field.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24630: [SPARK-27754][K8S] Introduce additional config (spark.kubernetes.driver.request.cores) for driver request cores for spark on k8s
SparkQA commented on issue #24630: [SPARK-27754][K8S] Introduce additional config (spark.kubernetes.driver.request.cores) for driver request cores for spark on k8s URL: https://github.com/apache/spark/pull/24630#issuecomment-493644031 **[Test build #105509 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105509/testReport)** for PR 24630 at commit [`ff688c2`](https://github.com/apache/spark/commit/ff688c23de7c716cd0d9484846cbd39cafc17a21). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] arunmahadevan commented on issue #24630: [SPARK-27754][K8S] Introduce additional config (spark.kubernetes.driver.request.cores) for driver request cores for spark on k8s
arunmahadevan commented on issue #24630: [SPARK-27754][K8S] Introduce additional config (spark.kubernetes.driver.request.cores) for driver request cores for spark on k8s URL: https://github.com/apache/spark/pull/24630#issuecomment-493643924 > I didn't suggest removing, @arunmahadevan . That kind of PR will be considered negatively too in the same way. You know, we usually are conservative for both sides (adding and removing). In this case I think we will get more consistent by introducing a config specific to k8s driver for which the executor config already exists. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] arunmahadevan commented on a change in pull request #24630: [SPARK-27754][K8S] Introduce additional config (spark.kubernetes.driver.request.cores) for driver request cores for spark o
arunmahadevan commented on a change in pull request #24630: [SPARK-27754][K8S]
Introduce additional config (spark.kubernetes.driver.request.cores) for driver
request cores for spark on k8s
URL: https://github.com/apache/spark/pull/24630#discussion_r285327085
##
File path:
resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStepSuite.scala
##
@@ -117,6 +117,33 @@ class BasicDriverFeatureStepSuite extends SparkFunSuite {
assert(featureStep.getAdditionalPodSystemProperties() ===
expectedSparkConf)
}
+ test("Check driver pod respects kubernetes driver request cores") {
+val sparkConf = new SparkConf()
+ .set(KUBERNETES_DRIVER_POD_NAME, "spark-driver-pod")
+ .set(CONTAINER_IMAGE, "spark-driver:latest")
+
+val basePod = SparkPod.initialPod()
+val requests1 = new
BasicDriverFeatureStep(KubernetesTestConf.createDriverConf(sparkConf))
+ .configurePod(basePod)
+ .container.getResources
+ .getRequests.asScala
+assert(requests1("cpu").getAmount === "1")
Review comment:
updated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] attilapiros edited a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
attilapiros edited a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493643283 The new message `RemoveBlocks` used only when the fetch RDD via shuffle service feature is enabled and done asynchronously: called within a `Future` block. There is one change I am thinking about right now where I could been overeager. Before my change the old `RemoveRDD` was sent to every executor: https://github.com/apache/spark/blob/dfeeda24c0f5d60bf6d2e1868c5290a1f62dc558/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala#L171-L178 Now I am only sending `RemoveRDD` to those executors where its blocks are registered. And I am wondering whether we miss to delete those blocks which not reported to the master. I can easily change that small logic back to be on the safe side. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #24635: [SPARK-27762][SQL] Support user provided avro schema for writing fields with different ordering
gengliangwang commented on a change in pull request #24635: [SPARK-27762][SQL]
Support user provided avro schema for writing fields with different ordering
URL: https://github.com/apache/spark/pull/24635#discussion_r285326928
##
File path:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala
##
@@ -205,18 +205,28 @@ class AvroSerializer(rootCatalystType: DataType,
rootAvroType: Schema, nullable:
throw new IncompatibleSchemaException(s"Cannot convert Catalyst type
$catalystStruct to " +
s"Avro type $avroStruct.")
}
-val fieldConverters = catalystStruct.zip(avroStruct.getFields.asScala).map
{
- case (f1, f2) => newConverter(f1.dataType,
resolveNullableType(f2.schema(), f1.nullable))
-}
+
+val (avroIndices: Array[Int], fieldConverters: Array[Converter]) =
+ catalystStruct.map { catalystField =>
+val avroField = avroStruct.getField(catalystField.name)
+if (avroField == null) {
Review comment:
I think we should try matching the Avro field name under `union`s with
`null`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] attilapiros edited a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
attilapiros edited a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493643283 The new message `RemoveBlocks` used only when the fetch RDD via shuffle service feature is enabled and done asynchronously: called within a `Future` block. There is one change I am thinking right now where I could be overeager. Before my change the old `RemoveRDD` was sent to every executor: https://github.com/apache/spark/blob/dfeeda24c0f5d60bf6d2e1868c5290a1f62dc558/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala#L171-L178 Now I am only sending `RemoveRDD` to those executors where its blocks are registered. And I am wondering whether we miss to delete those blocks which not reported to the master. I can easily change that small logic back to be on the safe side. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #24635: [SPARK-27762][SQL] Support user provided avro schema for writing fields with different ordering
gengliangwang commented on a change in pull request #24635: [SPARK-27762][SQL]
Support user provided avro schema for writing fields with different ordering
URL: https://github.com/apache/spark/pull/24635#discussion_r285326824
##
File path:
external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
##
@@ -692,7 +692,7 @@ class AvroSuite extends QueryTest with SharedSQLContext
with SQLTestUtils {
| "type" : "record",
| "name" : "test_schema",
| "fields" : [{
- |"name": "enum",
+ |"name": "Suit",
Review comment:
In the original test case, the avro schema is `enum: union`, while the
catalyst schema is `StructType(Seq(StructField("Suit", StringType, true)))`.
Thus the field names can't match with each other.
We should fix such case.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] attilapiros commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
attilapiros commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493643283 The new message `RemoveBlocks` used only when the fetch RDD via shuffle service feature is enabled and done asynchronously: called within a `Future` block. There is one change I am thinking right now where I could be overeager. Before my change the old `RemoveRDD` was sent to every executor: https://github.com/apache/spark/blob/dfeeda24c0f5d60bf6d2e1868c5290a1f62dc558/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala#L171-L178 Now I am only sending `RemoveRDD` to those executors where its blocks are registered. And I am wondering whether we miss to delete those blocks which not reported to the master. I can easily change that small logic back, to be on the safe side. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24635: [SPARK-27762][SQL] Support user provided avro schema for writing fields with different ordering
dongjoon-hyun commented on a change in pull request #24635: [SPARK-27762][SQL]
Support user provided avro schema for writing fields with different ordering
URL: https://github.com/apache/spark/pull/24635#discussion_r285326200
##
File path:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala
##
@@ -205,18 +205,28 @@ class AvroSerializer(rootCatalystType: DataType,
rootAvroType: Schema, nullable:
throw new IncompatibleSchemaException(s"Cannot convert Catalyst type
$catalystStruct to " +
s"Avro type $avroStruct.")
}
-val fieldConverters = catalystStruct.zip(avroStruct.getFields.asScala).map
{
- case (f1, f2) => newConverter(f1.dataType,
resolveNullableType(f2.schema(), f1.nullable))
-}
+
+val (avroIndices: Array[Int], fieldConverters: Array[Converter]) =
+ catalystStruct.map { catalystField =>
+val avroField = avroStruct.getField(catalystField.name)
+if (avroField == null) {
Review comment:
To fix `enum` issue at
https://github.com/apache/spark/pull/24635#discussion_r285326100, can we try to
use the corresponding field at the same position when `avroField == null`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24630: [SPARK-27754][K8S] Introduce additional config (spark.kubernetes.driver.request.cores) for driver request cores for spark on k8s
dongjoon-hyun commented on issue #24630: [SPARK-27754][K8S] Introduce additional config (spark.kubernetes.driver.request.cores) for driver request cores for spark on k8s URL: https://github.com/apache/spark/pull/24630#issuecomment-493642791 I didn't suggest removing, @arunmahadevan . That kind of PR will be considered negatively too in the same way. You know, we usually are conservative for both sides (adding and removing). > So the point is then should we consider removing all the "spark.kubernetes" configs that can be specified via pod templates? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24630: [SPARK-27754][K8S] Introduce additional config (spark.kubernetes.driver.request.cores) for driver request cores for spark o
dongjoon-hyun commented on a change in pull request #24630: [SPARK-27754][K8S]
Introduce additional config (spark.kubernetes.driver.request.cores) for driver
request cores for spark on k8s
URL: https://github.com/apache/spark/pull/24630#discussion_r285326341
##
File path:
resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStepSuite.scala
##
@@ -117,6 +117,33 @@ class BasicDriverFeatureStepSuite extends SparkFunSuite {
assert(featureStep.getAdditionalPodSystemProperties() ===
expectedSparkConf)
}
+ test("Check driver pod respects kubernetes driver request cores") {
+val sparkConf = new SparkConf()
+ .set(KUBERNETES_DRIVER_POD_NAME, "spark-driver-pod")
+ .set(CONTAINER_IMAGE, "spark-driver:latest")
+
+val basePod = SparkPod.initialPod()
+val requests1 = new
BasicDriverFeatureStep(KubernetesTestConf.createDriverConf(sparkConf))
+ .configurePod(basePod)
+ .container.getResources
+ .getRequests.asScala
+assert(requests1("cpu").getAmount === "1")
Review comment:
That should be the following because we should not have a magic number in
the code.
```scala
- private val driverCpuCores = conf.get(DRIVER_CORES.key, "1")
+ private val driverCpuCores = conf.get(DRIVER_CORES)
```
```scala
val driverCpuQuantity = new QuantityBuilder(false)
- .withAmount(driverCpuCores)
+ .withAmount(driverCpuCores.toString)
```
Please fix like the above. And don't use magic numbers.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24635: [SPARK-27762][SQL] Support user provided avro schema for writing fields with different ordering
dongjoon-hyun commented on a change in pull request #24635: [SPARK-27762][SQL]
Support user provided avro schema for writing fields with different ordering
URL: https://github.com/apache/spark/pull/24635#discussion_r285326200
##
File path:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala
##
@@ -205,18 +205,28 @@ class AvroSerializer(rootCatalystType: DataType,
rootAvroType: Schema, nullable:
throw new IncompatibleSchemaException(s"Cannot convert Catalyst type
$catalystStruct to " +
s"Avro type $avroStruct.")
}
-val fieldConverters = catalystStruct.zip(avroStruct.getFields.asScala).map
{
- case (f1, f2) => newConverter(f1.dataType,
resolveNullableType(f2.schema(), f1.nullable))
-}
+
+val (avroIndices: Array[Int], fieldConverters: Array[Converter]) =
+ catalystStruct.map { catalystField =>
+val avroField = avroStruct.getField(catalystField.name)
+if (avroField == null) {
Review comment:
To fix `enum` issue at
https://github.com/apache/spark/pull/24635#discussion_r285326100, can we use
the corresponding field at the same position when `avroField == null`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24635: [SPARK-27762][SQL] Support user provided avro schema for writing fields with different ordering
dongjoon-hyun commented on a change in pull request #24635: [SPARK-27762][SQL]
Support user provided avro schema for writing fields with different ordering
URL: https://github.com/apache/spark/pull/24635#discussion_r285326200
##
File path:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala
##
@@ -205,18 +205,28 @@ class AvroSerializer(rootCatalystType: DataType,
rootAvroType: Schema, nullable:
throw new IncompatibleSchemaException(s"Cannot convert Catalyst type
$catalystStruct to " +
s"Avro type $avroStruct.")
}
-val fieldConverters = catalystStruct.zip(avroStruct.getFields.asScala).map
{
- case (f1, f2) => newConverter(f1.dataType,
resolveNullableType(f2.schema(), f1.nullable))
-}
+
+val (avroIndices: Array[Int], fieldConverters: Array[Converter]) =
+ catalystStruct.map { catalystField =>
+val avroField = avroStruct.getField(catalystField.name)
+if (avroField == null) {
Review comment:
To fix `enum` issue, can we use the corresponding field at the same position
when `avroField == null`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24635: [SPARK-27762][SQL] Support user provided avro schema for writing fields with different ordering
dongjoon-hyun commented on a change in pull request #24635: [SPARK-27762][SQL]
Support user provided avro schema for writing fields with different ordering
URL: https://github.com/apache/spark/pull/24635#discussion_r285326100
##
File path:
external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
##
@@ -692,7 +692,7 @@ class AvroSuite extends QueryTest with SharedSQLContext
with SQLTestUtils {
| "type" : "record",
| "name" : "test_schema",
| "fields" : [{
- |"name": "enum",
+ |"name": "Suit",
Review comment:
`enum` type name and the field name (of that `enum` type) can be different.
We should not assume that this line is equal to the line 697.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] vanzin commented on a change in pull request #24616: [SPARK-27726] [Core] Fix performance of ElementTrackingStore deletes when using InMemoryStore under high loads
vanzin commented on a change in pull request #24616: [SPARK-27726] [Core] Fix
performance of ElementTrackingStore deletes when using InMemoryStore under high
loads
URL: https://github.com/apache/spark/pull/24616#discussion_r285313680
##
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##
@@ -1142,20 +1144,10 @@ private[spark] class AppStatusListener(
s.info.status != v1.StageStatus.ACTIVE && s.info.status !=
v1.StageStatus.PENDING
}
-stages.foreach { s =>
+val stageIndexValues = stages.map { s =>
Review comment:
`s/stageIndexValues/stageIds` (or `stageKeys`, since here they're actually
the primary keys of the stages being removed...)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] vanzin commented on a change in pull request #24616: [SPARK-27726] [Core] Fix performance of ElementTrackingStore deletes when using InMemoryStore under high loads
vanzin commented on a change in pull request #24616: [SPARK-27726] [Core] Fix
performance of ElementTrackingStore deletes when using InMemoryStore under high
loads
URL: https://github.com/apache/spark/pull/24616#discussion_r285313162
##
File path:
common/kvstore/src/main/java/org/apache/spark/util/kvstore/InMemoryStore.java
##
@@ -126,64 +134,149 @@ public void close() {
return (Comparable) in;
}
- private static class InstanceList {
+ @SuppressWarnings("unchecked")
+ private static KVStoreView emptyView() {
+return (InMemoryView) InMemoryView.EMPTY_VIEW;
+ }
+
+ /**
+ * Encapsulates ConcurrentHashMap so that the typing in and out of the map
strictly maps a
+ * class of type T to an InstanceList of type T.
+ */
+ private static class InMemoryLists {
+private ConcurrentMap, InstanceList> data = new
ConcurrentHashMap<>();
+
+@SuppressWarnings("unchecked")
+public InstanceList get(Class type) {
+ return (InstanceList) data.get(type);
+}
+
+@SuppressWarnings("unchecked")
+public void write(T value) throws Exception {
+ InstanceList list =
+(InstanceList) data.computeIfAbsent(value.getClass(),
InstanceList::new);
+ list.put(value);
+}
+
+public void clear() {
+ data.clear();
+}
+ }
+
+ private static class InstanceList {
+
+/**
+ * A BiConsumer to control multi-entity removal. We use this in a forEach
rather than an
+ * iterator because there is a bug in jdk8 which affects remove() on all
concurrent map
+ * iterators. https://bugs.openjdk.java.net/browse/JDK-8078645
+ */
+private static class CountingRemoveIfForEach implements
BiConsumer, T> {
+ private final ConcurrentMap, T> data;
+ private final Predicate filter;
+
+ /**
+ * Keeps a count of the number of elements removed. This count is not
currently surfaced
+ * to clients of KVStore as Java's generic removeAll() construct returns
only a boolean,
+ * but I found it handy to have the count of elements removed while
debugging; a count being
+ * no more complicated than a boolean, I've retained that behavior here,
even though there
+ * is no current requirement.
+ */
+ private int count = 0;
+
+ CountingRemoveIfForEach(
+ ConcurrentMap, T> data,
+ Predicate filter) {
+this.data = data;
+this.filter = filter;
+ }
+
+ public void accept(Comparable key, T value) {
+if (filter.test(value)) {
+ if (data.remove(key, value)) {
+count++;
+ }
+}
+ }
+
+ public int count() { return count; }
+}
private final KVTypeInfo ti;
private final KVTypeInfo.Accessor naturalKey;
-private final ConcurrentMap, Object> data;
+private final ConcurrentMap, T> data;
-private int size;
-
-private InstanceList(Class type) throws Exception {
- this.ti = new KVTypeInfo(type);
+private InstanceList(Class klass) {
+ this.ti = new KVTypeInfo(klass);
this.naturalKey = ti.getAccessor(KVIndex.NATURAL_INDEX_NAME);
this.data = new ConcurrentHashMap<>();
- this.size = 0;
}
KVTypeInfo.Accessor getIndexAccessor(String indexName) {
return ti.getAccessor(indexName);
}
-public Object get(Object key) {
+int countingRemoveAllByIndexValues(String index, Collection
indexValues) {
+ Predicate filter = getPredicate(ti.getAccessor(index),
indexValues);
+ CountingRemoveIfForEach callback = new
CountingRemoveIfForEach<>(data, filter);
+
+ data.forEach(callback);
+ return callback.count();
+}
+
+public T get(Object key) {
return data.get(asKey(key));
}
-public void put(Object value) throws Exception {
- Preconditions.checkArgument(ti.type().equals(value.getClass()),
-"Unexpected type: %s", value.getClass());
- if (data.put(asKey(naturalKey.get(value)), value) == null) {
-size++;
- }
+public void put(T value) throws Exception {
+ data.put(asKey(naturalKey.get(value)), value);
}
public void delete(Object key) {
- if (data.remove(asKey(key)) != null) {
-size--;
- }
+ data.remove(asKey(key));
}
public int size() {
- return size;
+ return data.size();
}
-@SuppressWarnings("unchecked")
-public InMemoryView view(Class type) {
- Preconditions.checkArgument(ti.type().equals(type), "Unexpected type:
%s", type);
- Collection all = (Collection) data.values();
- return new InMemoryView<>(type, all, ti);
+public InMemoryView view() {
+ return new InMemoryView<>(data.values(), ti);
+}
+
+private static Predicate getPredicate(
+KVTypeInfo.Accessor getter,
+Collection keys) {
Review comment:
s/keys/values
This is an automated message from
[GitHub] [spark] vanzin commented on a change in pull request #24616: [SPARK-27726] [Core] Fix performance of ElementTrackingStore deletes when using InMemoryStore under high loads
vanzin commented on a change in pull request #24616: [SPARK-27726] [Core] Fix performance of ElementTrackingStore deletes when using InMemoryStore under high loads URL: https://github.com/apache/spark/pull/24616#discussion_r285313350 ## File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStore.java ## @@ -126,4 +127,9 @@ */ long count(Class type, String index, Object indexedValue) throws Exception; + /** + * A cheaper way to remove multiple items from the KVStore + */ + boolean removeAllByIndexValues(Class klass, String index, Collection indexValues) + throws Exception; Review comment: nit: indented too much (2 or 4 spaces is fine here) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] vanzin commented on a change in pull request #24616: [SPARK-27726] [Core] Fix performance of ElementTrackingStore deletes when using InMemoryStore under high loads
vanzin commented on a change in pull request #24616: [SPARK-27726] [Core] Fix
performance of ElementTrackingStore deletes when using InMemoryStore under high
loads
URL: https://github.com/apache/spark/pull/24616#discussion_r285313778
##
File path:
core/src/main/scala/org/apache/spark/status/ElementTrackingStore.scala
##
@@ -46,7 +50,28 @@ import org.apache.spark.util.kvstore._
*/
private[spark] class ElementTrackingStore(store: KVStore, conf: SparkConf)
extends KVStore {
- private val triggers = new HashMap[Class[_], Seq[Trigger[_]]]()
+ private class LatchedTriggers(val triggers: Seq[Trigger[_]]) {
+private val pending = new AtomicBoolean(false)
+
+def fireOnce(f: Seq[Trigger[_]] => Unit): WriteQueueResult = {
+ val shouldEnqueue = pending.compareAndSet(false, true)
Review comment:
Actually don't need this variable now.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] vanzin commented on a change in pull request #24616: [SPARK-27726] [Core] Fix performance of ElementTrackingStore deletes when using InMemoryStore under high loads
vanzin commented on a change in pull request #24616: [SPARK-27726] [Core] Fix
performance of ElementTrackingStore deletes when using InMemoryStore under high
loads
URL: https://github.com/apache/spark/pull/24616#discussion_r285313219
##
File path:
common/kvstore/src/main/java/org/apache/spark/util/kvstore/InMemoryStore.java
##
@@ -126,64 +134,149 @@ public void close() {
return (Comparable) in;
}
- private static class InstanceList {
+ @SuppressWarnings("unchecked")
+ private static KVStoreView emptyView() {
+return (InMemoryView) InMemoryView.EMPTY_VIEW;
+ }
+
+ /**
+ * Encapsulates ConcurrentHashMap so that the typing in and out of the map
strictly maps a
+ * class of type T to an InstanceList of type T.
+ */
+ private static class InMemoryLists {
+private ConcurrentMap, InstanceList> data = new
ConcurrentHashMap<>();
+
+@SuppressWarnings("unchecked")
+public InstanceList get(Class type) {
+ return (InstanceList) data.get(type);
+}
+
+@SuppressWarnings("unchecked")
+public void write(T value) throws Exception {
+ InstanceList list =
+(InstanceList) data.computeIfAbsent(value.getClass(),
InstanceList::new);
+ list.put(value);
+}
+
+public void clear() {
+ data.clear();
+}
+ }
+
+ private static class InstanceList {
+
+/**
+ * A BiConsumer to control multi-entity removal. We use this in a forEach
rather than an
+ * iterator because there is a bug in jdk8 which affects remove() on all
concurrent map
+ * iterators. https://bugs.openjdk.java.net/browse/JDK-8078645
+ */
+private static class CountingRemoveIfForEach implements
BiConsumer, T> {
+ private final ConcurrentMap, T> data;
+ private final Predicate filter;
+
+ /**
+ * Keeps a count of the number of elements removed. This count is not
currently surfaced
+ * to clients of KVStore as Java's generic removeAll() construct returns
only a boolean,
+ * but I found it handy to have the count of elements removed while
debugging; a count being
+ * no more complicated than a boolean, I've retained that behavior here,
even though there
+ * is no current requirement.
+ */
+ private int count = 0;
+
+ CountingRemoveIfForEach(
+ ConcurrentMap, T> data,
+ Predicate filter) {
+this.data = data;
+this.filter = filter;
+ }
+
+ public void accept(Comparable key, T value) {
+if (filter.test(value)) {
+ if (data.remove(key, value)) {
+count++;
+ }
+}
+ }
+
+ public int count() { return count; }
+}
private final KVTypeInfo ti;
private final KVTypeInfo.Accessor naturalKey;
-private final ConcurrentMap, Object> data;
+private final ConcurrentMap, T> data;
-private int size;
-
-private InstanceList(Class type) throws Exception {
- this.ti = new KVTypeInfo(type);
+private InstanceList(Class klass) {
+ this.ti = new KVTypeInfo(klass);
this.naturalKey = ti.getAccessor(KVIndex.NATURAL_INDEX_NAME);
this.data = new ConcurrentHashMap<>();
- this.size = 0;
}
KVTypeInfo.Accessor getIndexAccessor(String indexName) {
return ti.getAccessor(indexName);
}
-public Object get(Object key) {
+int countingRemoveAllByIndexValues(String index, Collection
indexValues) {
+ Predicate filter = getPredicate(ti.getAccessor(index),
indexValues);
+ CountingRemoveIfForEach callback = new
CountingRemoveIfForEach<>(data, filter);
+
+ data.forEach(callback);
+ return callback.count();
+}
+
+public T get(Object key) {
return data.get(asKey(key));
}
-public void put(Object value) throws Exception {
- Preconditions.checkArgument(ti.type().equals(value.getClass()),
-"Unexpected type: %s", value.getClass());
- if (data.put(asKey(naturalKey.get(value)), value) == null) {
-size++;
- }
+public void put(T value) throws Exception {
+ data.put(asKey(naturalKey.get(value)), value);
}
public void delete(Object key) {
- if (data.remove(asKey(key)) != null) {
-size--;
- }
+ data.remove(asKey(key));
}
public int size() {
- return size;
+ return data.size();
}
-@SuppressWarnings("unchecked")
-public InMemoryView view(Class type) {
- Preconditions.checkArgument(ti.type().equals(type), "Unexpected type:
%s", type);
- Collection all = (Collection) data.values();
- return new InMemoryView<>(type, all, ti);
+public InMemoryView view() {
+ return new InMemoryView<>(data.values(), ti);
+}
+
+private static Predicate getPredicate(
+KVTypeInfo.Accessor getter,
+Collection keys) {
+ if (Comparable.class.isAssignableFrom(getter.getType())) {
+HashSet set = new HashSet<>(keys);
+
+return (value)
[GitHub] [spark] vanzin commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
vanzin commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493621811 I was kinda hoping you'd make the remove RPC a separate change... I need more time to look at that part, whereas the rest of the change was already ok for me. Two things I'll be looking at: whether the new RPC is only sent if the new feature it enabled (otherwise things will almost certainly break with older shuffle services), and whether it really needs to be an RPC (instead of just a one-way message with not reply, which is cheaper). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
AmplabJenkins removed a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493618696 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105508/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
AmplabJenkins commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493618689 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
AmplabJenkins commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493618696 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105508/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
AmplabJenkins removed a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493618689 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
SparkQA removed a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493601036 **[Test build #105508 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105508/testReport)** for PR 24499 at commit [`b30122f`](https://github.com/apache/spark/commit/b30122f2a90c4ef904809f9c5bf2c4c55f2c2291). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
SparkQA commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493618577 **[Test build #105508 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105508/testReport)** for PR 24499 at commit [`b30122f`](https://github.com/apache/spark/commit/b30122f2a90c4ef904809f9c5bf2c4c55f2c2291). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #17862: [SPARK-20602] [ML]Adding LBFGS optimizer and Squared_hinge loss for LinearSVC
AmplabJenkins removed a comment on issue #17862: [SPARK-20602] [ML]Adding LBFGS optimizer and Squared_hinge loss for LinearSVC URL: https://github.com/apache/spark/pull/17862#issuecomment-493616289 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105501/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #17862: [SPARK-20602] [ML]Adding LBFGS optimizer and Squared_hinge loss for LinearSVC
AmplabJenkins removed a comment on issue #17862: [SPARK-20602] [ML]Adding LBFGS optimizer and Squared_hinge loss for LinearSVC URL: https://github.com/apache/spark/pull/17862#issuecomment-493616283 Build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #17862: [SPARK-20602] [ML]Adding LBFGS optimizer and Squared_hinge loss for LinearSVC
AmplabJenkins commented on issue #17862: [SPARK-20602] [ML]Adding LBFGS optimizer and Squared_hinge loss for LinearSVC URL: https://github.com/apache/spark/pull/17862#issuecomment-493616289 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105501/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #17862: [SPARK-20602] [ML]Adding LBFGS optimizer and Squared_hinge loss for LinearSVC
AmplabJenkins commented on issue #17862: [SPARK-20602] [ML]Adding LBFGS optimizer and Squared_hinge loss for LinearSVC URL: https://github.com/apache/spark/pull/17862#issuecomment-493616283 Build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #17862: [SPARK-20602] [ML]Adding LBFGS optimizer and Squared_hinge loss for LinearSVC
SparkQA removed a comment on issue #17862: [SPARK-20602] [ML]Adding LBFGS optimizer and Squared_hinge loss for LinearSVC URL: https://github.com/apache/spark/pull/17862#issuecomment-493566849 **[Test build #105501 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105501/testReport)** for PR 17862 at commit [`0bb5afe`](https://github.com/apache/spark/commit/0bb5afe54a9a53054d2076ac28b09234a7380bbf). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] vanzin commented on issue #24565: [SPARK-27665][Core] Split fetch shuffle blocks protocol from OpenBlocks
vanzin commented on issue #24565: [SPARK-27665][Core] Split fetch shuffle blocks protocol from OpenBlocks URL: https://github.com/apache/spark/pull/24565#issuecomment-493616187 > I was originally thinking that we'd extend the RegisterExecutor msg which the client sends to the shuffle server with a version Is that always sent? Haven't traced the code but IIRC that's only the message when you register with your local shuffle service, not when you're getting data from a remote one... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #17862: [SPARK-20602] [ML]Adding LBFGS optimizer and Squared_hinge loss for LinearSVC
SparkQA commented on issue #17862: [SPARK-20602] [ML]Adding LBFGS optimizer and Squared_hinge loss for LinearSVC URL: https://github.com/apache/spark/pull/17862#issuecomment-493616145 **[Test build #105501 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105501/testReport)** for PR 17862 at commit [`0bb5afe`](https://github.com/apache/spark/commit/0bb5afe54a9a53054d2076ac28b09234a7380bbf). * This patch **fails Spark unit tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24615: [SPARK-27488][CORE] Driver interface to support GPU resources
AmplabJenkins commented on issue #24615: [SPARK-27488][CORE] Driver interface to support GPU resources URL: https://github.com/apache/spark/pull/24615#issuecomment-493614994 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24615: [SPARK-27488][CORE] Driver interface to support GPU resources
AmplabJenkins commented on issue #24615: [SPARK-27488][CORE] Driver interface to support GPU resources URL: https://github.com/apache/spark/pull/24615#issuecomment-493615000 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105504/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24615: [SPARK-27488][CORE] Driver interface to support GPU resources
AmplabJenkins removed a comment on issue #24615: [SPARK-27488][CORE] Driver interface to support GPU resources URL: https://github.com/apache/spark/pull/24615#issuecomment-493615000 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105504/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24615: [SPARK-27488][CORE] Driver interface to support GPU resources
AmplabJenkins removed a comment on issue #24615: [SPARK-27488][CORE] Driver interface to support GPU resources URL: https://github.com/apache/spark/pull/24615#issuecomment-493614994 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24615: [SPARK-27488][CORE] Driver interface to support GPU resources
SparkQA removed a comment on issue #24615: [SPARK-27488][CORE] Driver interface to support GPU resources URL: https://github.com/apache/spark/pull/24615#issuecomment-493578181 **[Test build #105504 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105504/testReport)** for PR 24615 at commit [`a06967d`](https://github.com/apache/spark/commit/a06967d60fac6b5710a514b31f8fb4ac18249f77). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24615: [SPARK-27488][CORE] Driver interface to support GPU resources
SparkQA commented on issue #24615: [SPARK-27488][CORE] Driver interface to support GPU resources URL: https://github.com/apache/spark/pull/24615#issuecomment-493614609 **[Test build #105504 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105504/testReport)** for PR 24615 at commit [`a06967d`](https://github.com/apache/spark/commit/a06967d60fac6b5710a514b31f8fb4ac18249f77). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] BryanCutler commented on issue #24614: [SPARK-27712][PySpark][SQL] Returns correct schema even under different column order when creating dataframe
BryanCutler commented on issue #24614: [SPARK-27712][PySpark][SQL] Returns
correct schema even under different column order when creating dataframe
URL: https://github.com/apache/spark/pull/24614#issuecomment-493608601
The problem I when using a positional schema when constructing `Row`s using
a dict is that the `Row` constructor sorts the fields, so any sense of position
is lost. For example:
This works
```python
data = [Row(k=i, v=str(i)) for i in range(100)]
rdd = spark.sparkContext.parallelize(data, 5)
# field names can differ.
df = rdd.toDF(" a: int, b: string ")
```
This fails
```python
data = [Row(z=i, y=str(i)) for i in range(100)]
rdd = spark.sparkContext.parallelize(data, 5)
# field names can differ.
df = rdd.toDF(" a: int, b: string ")
```
where the only difference is the field name from `Row`, so it really isn't
positional.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
AmplabJenkins removed a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-493607685 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105503/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
AmplabJenkins removed a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-493607681 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
AmplabJenkins commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-493607685 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105503/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
AmplabJenkins commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-493607681 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
SparkQA removed a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-493569303 **[Test build #105503 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105503/testReport)** for PR 23546 at commit [`3c58f7b`](https://github.com/apache/spark/commit/3c58f7bea99001c9fe5fda8b8ca73160d1f3bf6d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
SparkQA commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-493607264 **[Test build #105503 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105503/testReport)** for PR 23546 at commit [`3c58f7b`](https://github.com/apache/spark/commit/3c58f7bea99001c9fe5fda8b8ca73160d1f3bf6d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] BryanCutler commented on issue #24614: [SPARK-27712][PySpark][SQL] Returns correct schema even under different column order when creating dataframe
BryanCutler commented on issue #24614: [SPARK-27712][PySpark][SQL] Returns correct schema even under different column order when creating dataframe URL: https://github.com/apache/spark/pull/24614#issuecomment-493606770 Ah yes, there are all kinds of inconsistencies in the PySpark Row class. I think this is a duplicate of SPARK-22232 and we discussed in the PR here https://github.com/apache/spark/pull/20280. The fix there was to also pickle the `__from_dict__` flag, but the problem becomes that the input schema in `createDataFrame` must match field names and no longer goes off of position. At the time, I thought this was the most consistent fix, but I'll have to look at it again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] squito commented on issue #24565: [SPARK-27665][Core] Split fetch shuffle blocks protocol from OpenBlocks
squito commented on issue #24565: [SPARK-27665][Core] Split fetch shuffle blocks protocol from OpenBlocks URL: https://github.com/apache/spark/pull/24565#issuecomment-493605920 I think its very useful to have spark 3.0 be able to talk to a spark 2.4 shuffle service. The shuffle service can be harder to upgrade, as it effects all applications, while you could have a different spark version per application. Especially across a version breaking compatibility, this makes it easier for users to upgrade incrementally. I was originally thinking that we'd extend the `RegisterExecutor` msg which the client sends to the shuffle server with a version: https://github.com/apache/spark/blob/master/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleClient.java#L143 And I was hoping that only if that msg was not properly handled, then you'd auto-detect that you were talking to an old shuffle server. So it would be the same number of rpcs when all the versions match, but an extra one if you're using new spark with old shuffle server, just across this initial boundary where we update this message type. But that could actually be pretty slow I guess, so a config seems like a good option as well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
AmplabJenkins removed a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493602424 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10766/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
AmplabJenkins removed a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493602418 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
AmplabJenkins commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493602424 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10766/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
AmplabJenkins commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493602418 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
SparkQA commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493601036 **[Test build #105508 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105508/testReport)** for PR 24499 at commit [`b30122f`](https://github.com/apache/spark/commit/b30122f2a90c4ef904809f9c5bf2c4c55f2c2291). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24635: [SPARK-27762] [SQL] Support user provided avro schema for writing fields with different ordering
AmplabJenkins removed a comment on issue #24635: [SPARK-27762] [SQL] Support user provided avro schema for writing fields with different ordering URL: https://github.com/apache/spark/pull/24635#issuecomment-493600477 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105506/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24635: [SPARK-27762] [SQL] Support user provided avro schema for writing fields with different ordering
AmplabJenkins removed a comment on issue #24635: [SPARK-27762] [SQL] Support user provided avro schema for writing fields with different ordering URL: https://github.com/apache/spark/pull/24635#issuecomment-493600473 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24635: [SPARK-27762] [SQL] Support user provided avro schema for writing fields with different ordering
AmplabJenkins commented on issue #24635: [SPARK-27762] [SQL] Support user provided avro schema for writing fields with different ordering URL: https://github.com/apache/spark/pull/24635#issuecomment-493600477 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105506/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24635: [SPARK-27762] [SQL] Support user provided avro schema for writing fields with different ordering
AmplabJenkins commented on issue #24635: [SPARK-27762] [SQL] Support user provided avro schema for writing fields with different ordering URL: https://github.com/apache/spark/pull/24635#issuecomment-493600473 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24635: [SPARK-27762] [SQL] Support user provided avro schema for writing fields with different ordering
SparkQA removed a comment on issue #24635: [SPARK-27762] [SQL] Support user provided avro schema for writing fields with different ordering URL: https://github.com/apache/spark/pull/24635#issuecomment-493591283 **[Test build #105506 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105506/testReport)** for PR 24635 at commit [`968ee01`](https://github.com/apache/spark/commit/968ee01c2177a1051f41e4409b87432810905ca9). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24635: [SPARK-27762] [SQL] Support user provided avro schema for writing fields with different ordering
SparkQA commented on issue #24635: [SPARK-27762] [SQL] Support user provided avro schema for writing fields with different ordering URL: https://github.com/apache/spark/pull/24635#issuecomment-493600121 **[Test build #105506 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105506/testReport)** for PR 24635 at commit [`968ee01`](https://github.com/apache/spark/commit/968ee01c2177a1051f41e4409b87432810905ca9). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
AmplabJenkins removed a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493598220 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105507/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
AmplabJenkins removed a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493598213 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
AmplabJenkins commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493598220 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105507/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
SparkQA removed a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493595344 **[Test build #105507 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105507/testReport)** for PR 24499 at commit [`ef85815`](https://github.com/apache/spark/commit/ef85815263bc6de05eb4af138741e4dca285a849). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
AmplabJenkins commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493598213 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
SparkQA commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493598195 **[Test build #105507 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105507/testReport)** for PR 24499 at commit [`ef85815`](https://github.com/apache/spark/commit/ef85815263bc6de05eb4af138741e4dca285a849). * This patch **fails Java style tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
AmplabJenkins removed a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493596770 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
AmplabJenkins removed a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493596775 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10765/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
AmplabJenkins commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493596770 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
AmplabJenkins commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493596775 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10765/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
SparkQA commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493595344 **[Test build #105507 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105507/testReport)** for PR 24499 at commit [`ef85815`](https://github.com/apache/spark/commit/ef85815263bc6de05eb4af138741e4dca285a849). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] attilapiros commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
attilapiros commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493594925 Test failure is unrelated: `org.apache.spark.sql.hive.client.HiveClientSuites.(It is not a test it is a sbt.testing.SuiteSelector)` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] attilapiros commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation
attilapiros commented on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493595037 Jenkins retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24635: [SPARK-27762] [SQL] Support user provided avro schema for writing fields with different ordering
AmplabJenkins removed a comment on issue #24635: [SPARK-27762] [SQL] Support user provided avro schema for writing fields with different ordering URL: https://github.com/apache/spark/pull/24635#issuecomment-493592922 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10764/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24635: [SPARK-27762] [SQL] Support user provided avro schema for writing fields with different ordering
AmplabJenkins removed a comment on issue #24635: [SPARK-27762] [SQL] Support user provided avro schema for writing fields with different ordering URL: https://github.com/apache/spark/pull/24635#issuecomment-493592920 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
