svn commit: r25303 - in /dev/spark/2.4.0-SNAPSHOT-2018_02_27_00_01-8077bb0-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s
Author: pwendell Date: Tue Feb 27 08:16:51 2018 New Revision: 25303 Log: Apache Spark 2.4.0-SNAPSHOT-2018_02_27_00_01-8077bb0 docs [This commit notification would consist of 1444 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-23509][BUILD] Upgrade commons-net from 2.2 to 3.1
Repository: spark Updated Branches: refs/heads/master 8077bb04f -> 649ed9c57 [SPARK-23509][BUILD] Upgrade commons-net from 2.2 to 3.1 ## What changes were proposed in this pull request? This PR avoids version conflicts of `commons-net` by upgrading commons-net from 2.2 to 3.1. We are seeing the following message during the build using sbt. ``` [warn] Found version conflict(s) in library dependencies; some are suspected to be binary incompatible: ... [warn] * commons-net:commons-net:3.1 is selected over 2.2 [warn] +- org.apache.hadoop:hadoop-common:2.6.5 (depends on 3.1) [warn] +- org.apache.spark:spark-core_2.11:2.4.0-SNAPSHOT(depends on 2.2) [warn] ``` [Here](https://commons.apache.org/proper/commons-net/changes-report.html) is a release history. [Here](https://commons.apache.org/proper/commons-net/migration.html) is a migration guide from 2.x to 3.0. ## How was this patch tested? Existing tests Author: Kazuaki Ishizaki Closes #20672 from kiszk/SPARK-23509. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/649ed9c5 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/649ed9c5 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/649ed9c5 Branch: refs/heads/master Commit: 649ed9c5732f85ef1306576fdd3a9278a2a6410c Parents: 8077bb0 Author: Kazuaki Ishizaki Authored: Tue Feb 27 08:18:41 2018 -0600 Committer: Sean Owen Committed: Tue Feb 27 08:18:41 2018 -0600 -- dev/deps/spark-deps-hadoop-2.6 | 2 +- dev/deps/spark-deps-hadoop-2.7 | 2 +- pom.xml| 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/649ed9c5/dev/deps/spark-deps-hadoop-2.6 -- diff --git a/dev/deps/spark-deps-hadoop-2.6 b/dev/deps/spark-deps-hadoop-2.6 index ed31050..c3d1dd4 100644 --- a/dev/deps/spark-deps-hadoop-2.6 +++ b/dev/deps/spark-deps-hadoop-2.6 @@ -48,7 +48,7 @@ commons-lang-2.6.jar commons-lang3-3.5.jar commons-logging-1.1.3.jar commons-math3-3.4.1.jar -commons-net-2.2.jar +commons-net-3.1.jar commons-pool-1.5.4.jar compress-lzf-1.0.3.jar core-1.1.2.jar http://git-wip-us.apache.org/repos/asf/spark/blob/649ed9c5/dev/deps/spark-deps-hadoop-2.7 -- diff --git a/dev/deps/spark-deps-hadoop-2.7 b/dev/deps/spark-deps-hadoop-2.7 index 04dec04..2908670 100644 --- a/dev/deps/spark-deps-hadoop-2.7 +++ b/dev/deps/spark-deps-hadoop-2.7 @@ -48,7 +48,7 @@ commons-lang-2.6.jar commons-lang3-3.5.jar commons-logging-1.1.3.jar commons-math3-3.4.1.jar -commons-net-2.2.jar +commons-net-3.1.jar commons-pool-1.5.4.jar compress-lzf-1.0.3.jar core-1.1.2.jar http://git-wip-us.apache.org/repos/asf/spark/blob/649ed9c5/pom.xml -- diff --git a/pom.xml b/pom.xml index ac30107..b839616 100644 --- a/pom.xml +++ b/pom.xml @@ -579,7 +579,7 @@ commons-net commons-net -2.2 +3.1 io.netty - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-17147][STREAMING][KAFKA] Allow non-consecutive offsets
Repository: spark Updated Branches: refs/heads/master 649ed9c57 -> eac0b0672 [SPARK-17147][STREAMING][KAFKA] Allow non-consecutive offsets ## What changes were proposed in this pull request? Add a configuration spark.streaming.kafka.allowNonConsecutiveOffsets to allow streaming jobs to proceed on compacted topics (or other situations involving gaps between offsets in the log). ## How was this patch tested? Added new unit test justinrmiller has been testing this branch in production for a few weeks Author: cody koeninger Closes #20572 from koeninger/SPARK-17147. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/eac0b067 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/eac0b067 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/eac0b067 Branch: refs/heads/master Commit: eac0b067222a3dfa52be20360a453cb7bd420bf2 Parents: 649ed9c Author: cody koeninger Authored: Tue Feb 27 08:21:11 2018 -0600 Committer: Sean Owen Committed: Tue Feb 27 08:21:11 2018 -0600 -- .../kafka010/CachedKafkaConsumer.scala | 55 - .../spark/streaming/kafka010/KafkaRDD.scala | 236 +-- .../streaming/kafka010/KafkaRDDSuite.scala | 106 + .../streaming/kafka010/KafkaTestUtils.scala | 25 +- .../kafka010/mocks/MockScheduler.scala | 96 .../streaming/kafka010/mocks/MockTime.scala | 51 6 files changed, 487 insertions(+), 82 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/eac0b067/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/CachedKafkaConsumer.scala -- diff --git a/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/CachedKafkaConsumer.scala b/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/CachedKafkaConsumer.scala index fa3ea61..aeb8c1d 100644 --- a/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/CachedKafkaConsumer.scala +++ b/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/CachedKafkaConsumer.scala @@ -22,10 +22,8 @@ import java.{ util => ju } import org.apache.kafka.clients.consumer.{ ConsumerConfig, ConsumerRecord, KafkaConsumer } import org.apache.kafka.common.{ KafkaException, TopicPartition } -import org.apache.spark.SparkConf import org.apache.spark.internal.Logging - /** * Consumer of single topicpartition, intended for cached reuse. * Underlying consumer is not threadsafe, so neither is this, @@ -38,7 +36,7 @@ class CachedKafkaConsumer[K, V] private( val partition: Int, val kafkaParams: ju.Map[String, Object]) extends Logging { - assert(groupId == kafkaParams.get(ConsumerConfig.GROUP_ID_CONFIG), + require(groupId == kafkaParams.get(ConsumerConfig.GROUP_ID_CONFIG), "groupId used for cache key must match the groupId in kafkaParams") val topicPartition = new TopicPartition(topic, partition) @@ -53,7 +51,7 @@ class CachedKafkaConsumer[K, V] private( // TODO if the buffer was kept around as a random-access structure, // could possibly optimize re-calculating of an RDD in the same batch - protected var buffer = ju.Collections.emptyList[ConsumerRecord[K, V]]().iterator + protected var buffer = ju.Collections.emptyListIterator[ConsumerRecord[K, V]]() protected var nextOffset = -2L def close(): Unit = consumer.close() @@ -71,7 +69,7 @@ class CachedKafkaConsumer[K, V] private( } if (!buffer.hasNext()) { poll(timeout) } -assert(buffer.hasNext(), +require(buffer.hasNext(), s"Failed to get records for $groupId $topic $partition $offset after polling for $timeout") var record = buffer.next() @@ -79,17 +77,56 @@ class CachedKafkaConsumer[K, V] private( logInfo(s"Buffer miss for $groupId $topic $partition $offset") seek(offset) poll(timeout) - assert(buffer.hasNext(), + require(buffer.hasNext(), s"Failed to get records for $groupId $topic $partition $offset after polling for $timeout") record = buffer.next() - assert(record.offset == offset, -s"Got wrong record for $groupId $topic $partition even after seeking to offset $offset") + require(record.offset == offset, +s"Got wrong record for $groupId $topic $partition even after seeking to offset $offset " + + s"got offset ${record.offset} instead. If this is a compacted topic, consider enabling " + + "spark.streaming.kafka.allowNonConsecutiveOffsets" + ) } nextOffset = offset + 1 record } + /** + * Start a batch on a compacted topic + */ + def compactedStart(offset: Long, timeout: Long): Unit = { +logDebug(s"compacted start $groupId $topic $partition starting $offset")
svn commit: r25316 - in /dev/spark/2.4.0-SNAPSHOT-2018_02_27_08_01-eac0b06-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s
Author: pwendell Date: Tue Feb 27 16:18:08 2018 New Revision: 25316 Log: Apache Spark 2.4.0-SNAPSHOT-2018_02_27_08_01-eac0b06 docs [This commit notification would consist of 1444 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-23523][SQL] Fix the incorrect result caused by the rule OptimizeMetadataOnlyQuery
Repository: spark Updated Branches: refs/heads/master eac0b0672 -> 414ee867b [SPARK-23523][SQL] Fix the incorrect result caused by the rule OptimizeMetadataOnlyQuery ## What changes were proposed in this pull request? ```Scala val tablePath = new File(s"${path.getCanonicalPath}/cOl3=c/cOl1=a/cOl5=e") Seq(("a", "b", "c", "d", "e")).toDF("cOl1", "cOl2", "cOl3", "cOl4", "cOl5") .write.json(tablePath.getCanonicalPath) val df = spark.read.json(path.getCanonicalPath).select("CoL1", "CoL5", "CoL3").distinct() df.show() ``` It generates a wrong result. ``` [c,e,a] ``` We have a bug in the rule `OptimizeMetadataOnlyQuery `. We should respect the attribute order in the original leaf node. This PR is to fix it. ## How was this patch tested? Added a test case Author: gatorsmile Closes #20684 from gatorsmile/optimizeMetadataOnly. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/414ee867 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/414ee867 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/414ee867 Branch: refs/heads/master Commit: 414ee867ba0835b97aae2e8d4e489e1879c251dd Parents: eac0b06 Author: gatorsmile Authored: Tue Feb 27 08:44:25 2018 -0800 Committer: gatorsmile Committed: Tue Feb 27 08:44:25 2018 -0800 -- .../catalyst/plans/logical/LocalRelation.scala | 9 .../execution/OptimizeMetadataOnlyQuery.scala | 12 +-- .../datasources/HadoopFsRelation.scala | 3 +++ .../OptimizeMetadataOnlyQuerySuite.scala| 22 4 files changed, 40 insertions(+), 6 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/414ee867/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LocalRelation.scala -- diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LocalRelation.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LocalRelation.scala index d73d7e7..b05508d 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LocalRelation.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LocalRelation.scala @@ -43,10 +43,11 @@ object LocalRelation { } } -case class LocalRelation(output: Seq[Attribute], - data: Seq[InternalRow] = Nil, - // Indicates whether this relation has data from a streaming source. - override val isStreaming: Boolean = false) +case class LocalRelation( +output: Seq[Attribute], +data: Seq[InternalRow] = Nil, +// Indicates whether this relation has data from a streaming source. +override val isStreaming: Boolean = false) extends LeafNode with analysis.MultiInstanceRelation { // A local relation must have resolved output. http://git-wip-us.apache.org/repos/asf/spark/blob/414ee867/sql/core/src/main/scala/org/apache/spark/sql/execution/OptimizeMetadataOnlyQuery.scala -- diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/OptimizeMetadataOnlyQuery.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/OptimizeMetadataOnlyQuery.scala index 18f6f69..0613d90 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/OptimizeMetadataOnlyQuery.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/OptimizeMetadataOnlyQuery.scala @@ -17,6 +17,9 @@ package org.apache.spark.sql.execution +import java.util.Locale + +import org.apache.spark.sql.AnalysisException import org.apache.spark.sql.catalyst.InternalRow import org.apache.spark.sql.catalyst.catalog.{HiveTableRelation, SessionCatalog} import org.apache.spark.sql.catalyst.expressions._ @@ -80,8 +83,13 @@ case class OptimizeMetadataOnlyQuery(catalog: SessionCatalog) extends Rule[Logic private def getPartitionAttrs( partitionColumnNames: Seq[String], relation: LogicalPlan): Seq[Attribute] = { -val partColumns = partitionColumnNames.map(_.toLowerCase).toSet -relation.output.filter(a => partColumns.contains(a.name.toLowerCase)) +val attrMap = relation.output.map(_.name.toLowerCase(Locale.ROOT)).zip(relation.output).toMap +partitionColumnNames.map { colName => + attrMap.getOrElse(colName.toLowerCase(Locale.ROOT), +throw new AnalysisException(s"Unable to find the column `$colName` " + + s"given [${relation.output.map(_.name).mkString(", ")}]") + ) +} } /** http://git-wip-us.apache.org/repos/asf/spark/blob/414ee867/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/HadoopFsRelation.scala -
spark git commit: [SPARK-23365][CORE] Do not adjust num executors when killing idle executors.
Repository: spark Updated Branches: refs/heads/master 414ee867b -> ecb8b383a [SPARK-23365][CORE] Do not adjust num executors when killing idle executors. The ExecutorAllocationManager should not adjust the target number of executors when killing idle executors, as it has already adjusted the target number down based on the task backlog. The name `replace` was misleading with DynamicAllocation on, as the target number of executors is changed outside of the call to `killExecutors`, so I adjusted that name. Also separated out the logic of `countFailures` as you don't always want that tied to `replace`. While I was there I made two changes that weren't directly related to this: 1) Fixed `countFailures` in a couple cases where it was getting an incorrect value since it used to be tied to `replace`, eg. when killing executors on a blacklisted node. 2) hard error if you call `sc.killExecutors` with dynamic allocation on, since that's another way the ExecutorAllocationManager and the CoarseGrainedSchedulerBackend would get out of sync. Added a unit test case which verifies that the calls to ExecutorAllocationClient do not adjust the number of executors. Author: Imran Rashid Closes #20604 from squito/SPARK-23365. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/ecb8b383 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/ecb8b383 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/ecb8b383 Branch: refs/heads/master Commit: ecb8b383af1cf1b67f3111c148229e00c9c17c40 Parents: 414ee86 Author: Imran Rashid Authored: Tue Feb 27 11:12:32 2018 -0800 Committer: Marcelo Vanzin Committed: Tue Feb 27 11:12:32 2018 -0800 -- .../apache/spark/ExecutorAllocationClient.scala | 15 ++--- .../spark/ExecutorAllocationManager.scala | 20 -- .../scala/org/apache/spark/SparkContext.scala | 13 +++- .../spark/scheduler/BlacklistTracker.scala | 3 +- .../cluster/CoarseGrainedSchedulerBackend.scala | 22 --- .../spark/ExecutorAllocationManagerSuite.scala | 66 +++- .../StandaloneDynamicAllocationSuite.scala | 3 +- .../spark/scheduler/BlacklistTrackerSuite.scala | 14 ++--- 8 files changed, 121 insertions(+), 35 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/ecb8b383/core/src/main/scala/org/apache/spark/ExecutorAllocationClient.scala -- diff --git a/core/src/main/scala/org/apache/spark/ExecutorAllocationClient.scala b/core/src/main/scala/org/apache/spark/ExecutorAllocationClient.scala index 9112d93..63d87b4 100644 --- a/core/src/main/scala/org/apache/spark/ExecutorAllocationClient.scala +++ b/core/src/main/scala/org/apache/spark/ExecutorAllocationClient.scala @@ -55,18 +55,18 @@ private[spark] trait ExecutorAllocationClient { /** * Request that the cluster manager kill the specified executors. * - * When asking the executor to be replaced, the executor loss is considered a failure, and - * killed tasks that are running on the executor will count towards the failure limits. If no - * replacement is being requested, then the tasks will not count towards the limit. - * * @param executorIds identifiers of executors to kill - * @param replace whether to replace the killed executors with new ones, default false + * @param adjustTargetNumExecutors whether the target number of executors will be adjusted down + * after these executors have been killed + * @param countFailures if there are tasks running on the executors when they are killed, whether +* to count those failures toward task failure limits * @param force whether to force kill busy executors, default false * @return the ids of the executors acknowledged by the cluster manager to be removed. */ def killExecutors( executorIds: Seq[String], -replace: Boolean = false, +adjustTargetNumExecutors: Boolean, +countFailures: Boolean, force: Boolean = false): Seq[String] /** @@ -81,7 +81,8 @@ private[spark] trait ExecutorAllocationClient { * @return whether the request is acknowledged by the cluster manager. */ def killExecutor(executorId: String): Boolean = { -val killedExecutors = killExecutors(Seq(executorId)) +val killedExecutors = killExecutors(Seq(executorId), adjustTargetNumExecutors = true, + countFailures = false) killedExecutors.nonEmpty && killedExecutors(0).equals(executorId) } } http://git-wip-us.apache.org/repos/asf/spark/blob/ecb8b383/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala -- diff --git a/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.
spark git commit: [SPARK-23365][CORE] Do not adjust num executors when killing idle executors.
Repository: spark Updated Branches: refs/heads/branch-2.3 6eee545f9 -> 30242b664 [SPARK-23365][CORE] Do not adjust num executors when killing idle executors. The ExecutorAllocationManager should not adjust the target number of executors when killing idle executors, as it has already adjusted the target number down based on the task backlog. The name `replace` was misleading with DynamicAllocation on, as the target number of executors is changed outside of the call to `killExecutors`, so I adjusted that name. Also separated out the logic of `countFailures` as you don't always want that tied to `replace`. While I was there I made two changes that weren't directly related to this: 1) Fixed `countFailures` in a couple cases where it was getting an incorrect value since it used to be tied to `replace`, eg. when killing executors on a blacklisted node. 2) hard error if you call `sc.killExecutors` with dynamic allocation on, since that's another way the ExecutorAllocationManager and the CoarseGrainedSchedulerBackend would get out of sync. Added a unit test case which verifies that the calls to ExecutorAllocationClient do not adjust the number of executors. Author: Imran Rashid Closes #20604 from squito/SPARK-23365. (cherry picked from commit ecb8b383af1cf1b67f3111c148229e00c9c17c40) Signed-off-by: Marcelo Vanzin Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/30242b66 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/30242b66 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/30242b66 Branch: refs/heads/branch-2.3 Commit: 30242b664a05120c01b54ba9ac0ebed114f0d54e Parents: 6eee545 Author: Imran Rashid Authored: Tue Feb 27 11:12:32 2018 -0800 Committer: Marcelo Vanzin Committed: Tue Feb 27 11:12:54 2018 -0800 -- .../apache/spark/ExecutorAllocationClient.scala | 15 ++--- .../spark/ExecutorAllocationManager.scala | 20 -- .../scala/org/apache/spark/SparkContext.scala | 13 +++- .../spark/scheduler/BlacklistTracker.scala | 3 +- .../cluster/CoarseGrainedSchedulerBackend.scala | 22 --- .../spark/ExecutorAllocationManagerSuite.scala | 66 +++- .../StandaloneDynamicAllocationSuite.scala | 3 +- .../spark/scheduler/BlacklistTrackerSuite.scala | 14 ++--- 8 files changed, 121 insertions(+), 35 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/30242b66/core/src/main/scala/org/apache/spark/ExecutorAllocationClient.scala -- diff --git a/core/src/main/scala/org/apache/spark/ExecutorAllocationClient.scala b/core/src/main/scala/org/apache/spark/ExecutorAllocationClient.scala index 9112d93..63d87b4 100644 --- a/core/src/main/scala/org/apache/spark/ExecutorAllocationClient.scala +++ b/core/src/main/scala/org/apache/spark/ExecutorAllocationClient.scala @@ -55,18 +55,18 @@ private[spark] trait ExecutorAllocationClient { /** * Request that the cluster manager kill the specified executors. * - * When asking the executor to be replaced, the executor loss is considered a failure, and - * killed tasks that are running on the executor will count towards the failure limits. If no - * replacement is being requested, then the tasks will not count towards the limit. - * * @param executorIds identifiers of executors to kill - * @param replace whether to replace the killed executors with new ones, default false + * @param adjustTargetNumExecutors whether the target number of executors will be adjusted down + * after these executors have been killed + * @param countFailures if there are tasks running on the executors when they are killed, whether +* to count those failures toward task failure limits * @param force whether to force kill busy executors, default false * @return the ids of the executors acknowledged by the cluster manager to be removed. */ def killExecutors( executorIds: Seq[String], -replace: Boolean = false, +adjustTargetNumExecutors: Boolean, +countFailures: Boolean, force: Boolean = false): Seq[String] /** @@ -81,7 +81,8 @@ private[spark] trait ExecutorAllocationClient { * @return whether the request is acknowledged by the cluster manager. */ def killExecutor(executorId: String): Boolean = { -val killedExecutors = killExecutors(Seq(executorId)) +val killedExecutors = killExecutors(Seq(executorId), adjustTargetNumExecutors = true, + countFailures = false) killedExecutors.nonEmpty && killedExecutors(0).equals(executorId) } } http://git-wip-us.apache.org/repos/asf/spark/blob/30242b66/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala ---
spark git commit: [SPARK-23501][UI] Refactor AllStagesPage in order to avoid redundant code
Repository: spark Updated Branches: refs/heads/master ecb8b383a -> 598446b74 [SPARK-23501][UI] Refactor AllStagesPage in order to avoid redundant code As suggested in #20651, the code is very redundant in `AllStagesPage` and modifying it is a copy-and-paste work. We should avoid such a pattern, which is error prone, and have a cleaner solution which avoids code redundancy. existing UTs Author: Marco Gaido Closes #20663 from mgaido91/SPARK-23475_followup. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/598446b7 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/598446b7 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/598446b7 Branch: refs/heads/master Commit: 598446b74b61fee272d3aee3a2e9a3fc90a70d6a Parents: ecb8b38 Author: Marco Gaido Authored: Tue Feb 27 11:33:10 2018 -0800 Committer: Marcelo Vanzin Committed: Tue Feb 27 11:35:36 2018 -0800 -- .../apache/spark/ui/jobs/AllStagesPage.scala| 261 --- 1 file changed, 102 insertions(+), 159 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/598446b7/core/src/main/scala/org/apache/spark/ui/jobs/AllStagesPage.scala -- diff --git a/core/src/main/scala/org/apache/spark/ui/jobs/AllStagesPage.scala b/core/src/main/scala/org/apache/spark/ui/jobs/AllStagesPage.scala index 38450b9..4658aa1 100644 --- a/core/src/main/scala/org/apache/spark/ui/jobs/AllStagesPage.scala +++ b/core/src/main/scala/org/apache/spark/ui/jobs/AllStagesPage.scala @@ -19,46 +19,20 @@ package org.apache.spark.ui.jobs import javax.servlet.http.HttpServletRequest -import scala.xml.{Node, NodeSeq} +import scala.xml.{Attribute, Elem, Node, NodeSeq, Null, Text} import org.apache.spark.scheduler.Schedulable -import org.apache.spark.status.PoolData -import org.apache.spark.status.api.v1._ +import org.apache.spark.status.{AppSummary, PoolData} +import org.apache.spark.status.api.v1.{StageData, StageStatus} import org.apache.spark.ui.{UIUtils, WebUIPage} /** Page showing list of all ongoing and recently finished stages and pools */ private[ui] class AllStagesPage(parent: StagesTab) extends WebUIPage("") { private val sc = parent.sc + private val subPath = "stages" private def isFairScheduler = parent.isFairScheduler def render(request: HttpServletRequest): Seq[Node] = { -val allStages = parent.store.stageList(null) - -val activeStages = allStages.filter(_.status == StageStatus.ACTIVE) -val pendingStages = allStages.filter(_.status == StageStatus.PENDING) -val skippedStages = allStages.filter(_.status == StageStatus.SKIPPED) -val completedStages = allStages.filter(_.status == StageStatus.COMPLETE) -val failedStages = allStages.filter(_.status == StageStatus.FAILED).reverse - -val numFailedStages = failedStages.size -val subPath = "stages" - -val activeStagesTable = - new StageTableBase(parent.store, request, activeStages, "active", "activeStage", -parent.basePath, subPath, parent.isFairScheduler, parent.killEnabled, false) -val pendingStagesTable = - new StageTableBase(parent.store, request, pendingStages, "pending", "pendingStage", -parent.basePath, subPath, parent.isFairScheduler, false, false) -val completedStagesTable = - new StageTableBase(parent.store, request, completedStages, "completed", "completedStage", -parent.basePath, subPath, parent.isFairScheduler, false, false) -val skippedStagesTable = - new StageTableBase(parent.store, request, skippedStages, "skipped", "skippedStage", -parent.basePath, subPath, parent.isFairScheduler, false, false) -val failedStagesTable = - new StageTableBase(parent.store, request, failedStages, "failed", "failedStage", -parent.basePath, subPath, parent.isFairScheduler, false, true) - // For now, pool information is only accessible in live UIs val pools = sc.map(_.getAllPools).getOrElse(Seq.empty[Schedulable]).map { pool => val uiPool = parent.store.asOption(parent.store.pool(pool.name)).getOrElse( @@ -67,152 +41,121 @@ private[ui] class AllStagesPage(parent: StagesTab) extends WebUIPage("") { }.toMap val poolTable = new PoolTable(pools, parent) -val shouldShowActiveStages = activeStages.nonEmpty -val shouldShowPendingStages = pendingStages.nonEmpty -val shouldShowCompletedStages = completedStages.nonEmpty -val shouldShowSkippedStages = skippedStages.nonEmpty -val shouldShowFailedStages = failedStages.nonEmpty +val allStatuses = Seq(StageStatus.ACTIVE, StageStatus.PENDING, StageStatus.COMPLETE, + StageStatus.SKIPPED, StageStatus.FAILED) +val allStages = parent.store.stageList(null) val appSummary = parent.store.appSu
svn commit: r25318 - in /dev/spark/2.4.0-SNAPSHOT-2018_02_27_12_01-598446b-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s
Author: pwendell Date: Tue Feb 27 20:15:52 2018 New Revision: 25318 Log: Apache Spark 2.4.0-SNAPSHOT-2018_02_27_12_01-598446b docs [This commit notification would consist of 1444 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
svn commit: r25320 - in /dev/spark/2.3.1-SNAPSHOT-2018_02_27_14_01-30242b6-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s
Author: pwendell Date: Tue Feb 27 22:16:01 2018 New Revision: 25320 Log: Apache Spark 2.3.1-SNAPSHOT-2018_02_27_14_01-30242b6 docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-23417][PYTHON] Fix the build instructions supplied by exception messages in python streaming tests
Repository: spark Updated Branches: refs/heads/master 598446b74 -> 23ac3aaba [SPARK-23417][PYTHON] Fix the build instructions supplied by exception messages in python streaming tests ## What changes were proposed in this pull request? Fix the build instructions supplied by exception messages in python streaming tests. I also added -DskipTests to the maven instructions to avoid the 170 minutes of scala tests that occurs each time one wants to add a jar to the assembly directory. ## How was this patch tested? - clone branch - run build/sbt package - run python/run-tests --modules "pyspark-streaming" , expect error message - follow instructions in error message. i.e., run build/sbt assembly/package streaming-kafka-0-8-assembly/assembly - rerun python tests, expect error message - follow instructions in error message. i.e run build/sbt -Pflume assembly/package streaming-flume-assembly/assembly - rerun python tests, see success. - repeated all of the above for mvn version of the process. Author: Bruce Robbins Closes #20638 from bersprockets/SPARK-23417_propa. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/23ac3aab Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/23ac3aab Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/23ac3aab Branch: refs/heads/master Commit: 23ac3aaba4a33bc3d31d01f21e93c4681ef6de03 Parents: 598446b Author: Bruce Robbins Authored: Wed Feb 28 09:25:02 2018 +0900 Committer: hyukjinkwon Committed: Wed Feb 28 09:25:02 2018 +0900 -- python/pyspark/streaming/tests.py | 8 1 file changed, 4 insertions(+), 4 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/23ac3aab/python/pyspark/streaming/tests.py -- diff --git a/python/pyspark/streaming/tests.py b/python/pyspark/streaming/tests.py index 5b86c1c..71f8101 100644 --- a/python/pyspark/streaming/tests.py +++ b/python/pyspark/streaming/tests.py @@ -1477,8 +1477,8 @@ def search_kafka_assembly_jar(): raise Exception( ("Failed to find Spark Streaming kafka assembly jar in %s. " % kafka_assembly_dir) + "You need to build Spark with " -"'build/sbt assembly/package streaming-kafka-0-8-assembly/assembly' or " -"'build/mvn -Pkafka-0-8 package' before running this test.") +"'build/sbt -Pkafka-0-8 assembly/package streaming-kafka-0-8-assembly/assembly' or " +"'build/mvn -DskipTests -Pkafka-0-8 package' before running this test.") elif len(jars) > 1: raise Exception(("Found multiple Spark Streaming Kafka assembly JARs: %s; please " "remove all but one") % (", ".join(jars))) @@ -1494,8 +1494,8 @@ def search_flume_assembly_jar(): raise Exception( ("Failed to find Spark Streaming Flume assembly jar in %s. " % flume_assembly_dir) + "You need to build Spark with " -"'build/sbt assembly/assembly streaming-flume-assembly/assembly' or " -"'build/mvn -Pflume package' before running this test.") +"'build/sbt -Pflume assembly/package streaming-flume-assembly/assembly' or " +"'build/mvn -DskipTests -Pflume package' before running this test.") elif len(jars) > 1: raise Exception(("Found multiple Spark Streaming Flume assembly JARs: %s; please " "remove all but one") % (", ".join(jars))) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-23448][SQL] Clarify JSON and CSV parser behavior in document
Repository: spark Updated Branches: refs/heads/master 23ac3aaba -> b14993e1f [SPARK-23448][SQL] Clarify JSON and CSV parser behavior in document ## What changes were proposed in this pull request? Clarify JSON and CSV reader behavior in document. JSON doesn't support partial results for corrupted records. CSV only supports partial results for the records with more or less tokens. ## How was this patch tested? Pass existing tests. Author: Liang-Chi Hsieh Closes #20666 from viirya/SPARK-23448-2. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b14993e1 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b14993e1 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b14993e1 Branch: refs/heads/master Commit: b14993e1fcb68e1c946a671c6048605ab4afdf58 Parents: 23ac3aa Author: Liang-Chi Hsieh Authored: Wed Feb 28 11:00:54 2018 +0900 Committer: hyukjinkwon Committed: Wed Feb 28 11:00:54 2018 +0900 -- python/pyspark/sql/readwriter.py| 30 +++- python/pyspark/sql/streaming.py | 30 +++- .../spark/sql/catalyst/json/JacksonParser.scala | 3 ++ .../org/apache/spark/sql/DataFrameReader.scala | 22 +++--- .../datasources/csv/UnivocityParser.scala | 5 .../spark/sql/streaming/DataStreamReader.scala | 22 +++--- 6 files changed, 64 insertions(+), 48 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/b14993e1/python/pyspark/sql/readwriter.py -- diff --git a/python/pyspark/sql/readwriter.py b/python/pyspark/sql/readwriter.py index 49af1bc..9d05ac7 100644 --- a/python/pyspark/sql/readwriter.py +++ b/python/pyspark/sql/readwriter.py @@ -209,13 +209,13 @@ class DataFrameReader(OptionUtils): :param mode: allows a mode for dealing with corrupt records during parsing. If None is set, it uses the default value, ``PERMISSIVE``. -* ``PERMISSIVE`` : sets other fields to ``null`` when it meets a corrupted \ - record, and puts the malformed string into a field configured by \ - ``columnNameOfCorruptRecord``. To keep corrupt records, an user can set \ - a string type field named ``columnNameOfCorruptRecord`` in an user-defined \ - schema. If a schema does not have the field, it drops corrupt records during \ - parsing. When inferring a schema, it implicitly adds a \ - ``columnNameOfCorruptRecord`` field in an output schema. +* ``PERMISSIVE`` : when it meets a corrupted record, puts the malformed string \ + into a field configured by ``columnNameOfCorruptRecord``, and sets other \ + fields to ``null``. To keep corrupt records, an user can set a string type \ + field named ``columnNameOfCorruptRecord`` in an user-defined schema. If a \ + schema does not have the field, it drops corrupt records during parsing. \ + When inferring a schema, it implicitly adds a ``columnNameOfCorruptRecord`` \ + field in an output schema. * ``DROPMALFORMED`` : ignores the whole corrupted records. * ``FAILFAST`` : throws an exception when it meets corrupted records. @@ -393,13 +393,15 @@ class DataFrameReader(OptionUtils): :param mode: allows a mode for dealing with corrupt records during parsing. If None is set, it uses the default value, ``PERMISSIVE``. -* ``PERMISSIVE`` : sets other fields to ``null`` when it meets a corrupted \ - record, and puts the malformed string into a field configured by \ - ``columnNameOfCorruptRecord``. To keep corrupt records, an user can set \ - a string type field named ``columnNameOfCorruptRecord`` in an \ - user-defined schema. If a schema does not have the field, it drops corrupt \ - records during parsing. When a length of parsed CSV tokens is shorter than \ - an expected length of a schema, it sets `null` for extra fields. +* ``PERMISSIVE`` : when it meets a corrupted record, puts the malformed string \ + into a field configured by ``columnNameOfCorruptRecord``, and sets other \ + fields to ``null``. To keep corrupt records, an user can set a string type \ + field named ``columnNameOfCorruptRecord`` in an user-defined schema. If a \ + schema does not have the field, it drops corrupt records during parsing. \ + A record with less/more tokens than schema i
spark git commit: [SPARK-23448][SQL] Clarify JSON and CSV parser behavior in document
Repository: spark Updated Branches: refs/heads/branch-2.3 30242b664 -> fe9cb4afe [SPARK-23448][SQL] Clarify JSON and CSV parser behavior in document ## What changes were proposed in this pull request? Clarify JSON and CSV reader behavior in document. JSON doesn't support partial results for corrupted records. CSV only supports partial results for the records with more or less tokens. ## How was this patch tested? Pass existing tests. Author: Liang-Chi Hsieh Closes #20666 from viirya/SPARK-23448-2. (cherry picked from commit b14993e1fcb68e1c946a671c6048605ab4afdf58) Signed-off-by: hyukjinkwon Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/fe9cb4af Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/fe9cb4af Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/fe9cb4af Branch: refs/heads/branch-2.3 Commit: fe9cb4afe39944b394b39cc25622e311554ae187 Parents: 30242b6 Author: Liang-Chi Hsieh Authored: Wed Feb 28 11:00:54 2018 +0900 Committer: hyukjinkwon Committed: Wed Feb 28 11:01:12 2018 +0900 -- python/pyspark/sql/readwriter.py| 30 +++- python/pyspark/sql/streaming.py | 30 +++- .../spark/sql/catalyst/json/JacksonParser.scala | 3 ++ .../org/apache/spark/sql/DataFrameReader.scala | 22 +++--- .../datasources/csv/UnivocityParser.scala | 5 .../spark/sql/streaming/DataStreamReader.scala | 22 +++--- 6 files changed, 64 insertions(+), 48 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/fe9cb4af/python/pyspark/sql/readwriter.py -- diff --git a/python/pyspark/sql/readwriter.py b/python/pyspark/sql/readwriter.py index 49af1bc..9d05ac7 100644 --- a/python/pyspark/sql/readwriter.py +++ b/python/pyspark/sql/readwriter.py @@ -209,13 +209,13 @@ class DataFrameReader(OptionUtils): :param mode: allows a mode for dealing with corrupt records during parsing. If None is set, it uses the default value, ``PERMISSIVE``. -* ``PERMISSIVE`` : sets other fields to ``null`` when it meets a corrupted \ - record, and puts the malformed string into a field configured by \ - ``columnNameOfCorruptRecord``. To keep corrupt records, an user can set \ - a string type field named ``columnNameOfCorruptRecord`` in an user-defined \ - schema. If a schema does not have the field, it drops corrupt records during \ - parsing. When inferring a schema, it implicitly adds a \ - ``columnNameOfCorruptRecord`` field in an output schema. +* ``PERMISSIVE`` : when it meets a corrupted record, puts the malformed string \ + into a field configured by ``columnNameOfCorruptRecord``, and sets other \ + fields to ``null``. To keep corrupt records, an user can set a string type \ + field named ``columnNameOfCorruptRecord`` in an user-defined schema. If a \ + schema does not have the field, it drops corrupt records during parsing. \ + When inferring a schema, it implicitly adds a ``columnNameOfCorruptRecord`` \ + field in an output schema. * ``DROPMALFORMED`` : ignores the whole corrupted records. * ``FAILFAST`` : throws an exception when it meets corrupted records. @@ -393,13 +393,15 @@ class DataFrameReader(OptionUtils): :param mode: allows a mode for dealing with corrupt records during parsing. If None is set, it uses the default value, ``PERMISSIVE``. -* ``PERMISSIVE`` : sets other fields to ``null`` when it meets a corrupted \ - record, and puts the malformed string into a field configured by \ - ``columnNameOfCorruptRecord``. To keep corrupt records, an user can set \ - a string type field named ``columnNameOfCorruptRecord`` in an \ - user-defined schema. If a schema does not have the field, it drops corrupt \ - records during parsing. When a length of parsed CSV tokens is shorter than \ - an expected length of a schema, it sets `null` for extra fields. +* ``PERMISSIVE`` : when it meets a corrupted record, puts the malformed string \ + into a field configured by ``columnNameOfCorruptRecord``, and sets other \ + fields to ``null``. To keep corrupt records, an user can set a string type \ + field named ``columnNameOfCorruptRecord`` in an user-defined schema. If a \ + schema does not have the field, i
svn commit: r25321 - in /dev/spark/2.4.0-SNAPSHOT-2018_02_27_20_01-b14993e-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s
Author: pwendell Date: Wed Feb 28 04:17:40 2018 New Revision: 25321 Log: Apache Spark 2.4.0-SNAPSHOT-2018_02_27_20_01-b14993e docs [This commit notification would consist of 1444 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
svn commit: r25323 - in /dev/spark/2.3.1-SNAPSHOT-2018_02_27_22_01-fe9cb4a-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s
Author: pwendell Date: Wed Feb 28 06:16:02 2018 New Revision: 25323 Log: Apache Spark 2.3.1-SNAPSHOT-2018_02_27_22_01-fe9cb4a docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
svn commit: r25324 - /dev/spark/v2.3.0-rc5-bin/ /release/spark/spark-2.3.0/
Author: yhuai Date: Wed Feb 28 07:25:53 2018 New Revision: 25324 Log: Releasing Apache Spark 2.3.0 Added: release/spark/spark-2.3.0/ - copied from r25323, dev/spark/v2.3.0-rc5-bin/ Removed: dev/spark/v2.3.0-rc5-bin/ - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
svn commit: r25325 - in /dev/spark: 2.3.0-SNAPSHOT-2017_12_09_08_01-ab1b6ee-docs/ 2.3.0-SNAPSHOT-2017_12_11_00_01-4289ac9-docs/ 2.3.0-SNAPSHOT-2017_12_11_08_01-6cc7021-docs/ 2.3.0-SNAPSHOT-2017_12_11_
Author: sameerag Date: Wed Feb 28 07:28:47 2018 New Revision: 25325 Log: Removing extraneous doc snapshot uploads Removed: dev/spark/2.3.0-SNAPSHOT-2017_12_09_08_01-ab1b6ee-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_11_00_01-4289ac9-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_11_08_01-6cc7021-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_11_12_01-bf20abb-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_11_16_01-3d82f6e-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_11_20_01-a400265-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_12_00_01-ecc179e-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_12_04_01-bc8933f-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_12_12_01-7a51e71-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_12_16_01-17cdabb-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_12_20_01-c7d0148-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_13_00_01-682eb4f-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_13_04_01-7453ab0-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_13_08_02-58f7c82-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_13_12_01-1abcbed-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_13_16_01-ef92999-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_13_20_01-2a29a60-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_14_00_01-c3dd2a2-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_14_04_01-7d8e2ca-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_14_12_01-6d99940-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_14_16_01-0ea2d8c-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_15_00_01-3775dd3-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_15_12_01-4677623-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_15_20_01-0c8fca4-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_17_00_01-c2aeddf-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_17_12_01-7f6d10a-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_18_16_01-0609dcc-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_18_20_01-d4e6959-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_19_08_01-b779c93-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_19_12_01-6129ffa-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_20_00_01-9962390-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_20_12_02-c89b431-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_20_16_01-b176014-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_20_20_01-d3ae3e1-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_21_00_01-cb9fc8d-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_21_04_01-59d5263-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_21_08_01-0abaf31-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_21_12_01-fe65361-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_21_16_01-7beb375-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_21_20_01-a36b78b-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_22_00_01-8df1da3-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_22_04_01-13190a4-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_22_16_01-d23dc5b-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_23_00_01-8941a4a-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_23_12_01-1219d7a-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_24_12_01-0bf1a74-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_25_00_01-fba0313-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_25_04_39-12d20dd-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_25_20_01-be03d3a-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_26_00_01-0e68330-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_26_08_01-ff48b1b-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_26_12_01-91d1b30-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_26_20_01-6674acd-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_27_04_01-b8bfce5-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_27_08_01-774715d-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_27_20_01-753793b-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_28_00_01-ded6d27-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_28_04_01-1eebfbe-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_28_08_01-5536f31-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_28_12_01-8f6d573-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_28_16_01-ffe6fd7-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_28_20_01-c745730-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_29_00_01-224375c-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_29_04_01-cc30ef8-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_29_08_01-11a849b-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_29_12_01-66a7d6b-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_29_16_01-ccda75b-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_29_20_01-8169630-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_30_00_01-14c4a62-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_30_08_01-234d943-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_30_12_01-ea0a5ee-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_31_00_01-cfbe11e-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_31_08_01-028ee40-docs/ dev/spark/2.3.0-SNAPSHOT-2017_12_31_16_01-994065d-docs/ dev/spark/2.3.0-SNAPSHOT-2018_01_01_08_01-f5b7714-docs/ dev/spark/2.3.0-SNAPSHOT-2018_01_01_20_01-e0c090f-docs/ dev/spark/2.3.0-SNAPSHOT-2018_01_02_08_01-a6fc300-docs/ dev/spark/2.3.0-SNAPSHOT-2018_01_03_08_01-a66fe36-docs/ dev/spark/2.3.0-SNAPSHOT-2018_01_03_12_01-9a2b65a-docs/ dev/spark/2.3.0-SNAPSHOT-2018_01_03_13_34-79f7263-docs/ dev/spark/2.3.0-SNAPSHOT-2018_01_03_16_01-b297029-docs/ dev/spark/
[spark] Git Push Summary
Repository: spark Updated Tags: refs/tags/v2.3.0 [created] 992447fb3 - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] Git Push Summary
Repository: spark Updated Tags: refs/tags/v2.3.0-rc5 [deleted] 992447fb3 - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org