[spark] branch master updated (857f109 -> a988aaf)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 857f109 [SPARK-10614][CORE] Add monotonic time to Clock interface add a988aaf [SPARK-29454][SQL] Reduce unsafeProjection times when read Parquet file use non-vectorized mode No new revisions were added by this update. Summary of changes: .../datasources/parquet/ParquetFileFormat.scala | 20 .../datasources/parquet/ParquetReadSupport.scala | 10 +- .../parquet/ParquetRecordMaterializer.scala | 6 +++--- .../datasources/parquet/ParquetRowConverter.scala| 6 ++ .../v2/parquet/ParquetPartitionReaderFactory.scala | 11 +-- 5 files changed, 23 insertions(+), 30 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (9407fba -> 857f109)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 9407fba [SPARK-29412][SQL] refine the document of v2 session catalog config add 857f109 [SPARK-10614][CORE] Add monotonic time to Clock interface No new revisions were added by this update. Summary of changes: .../apache/spark/ExecutorAllocationManager.scala | 6 ++-- .../spark/scheduler/dynalloc/ExecutorMonitor.scala | 17 - .../main/scala/org/apache/spark/util/Clock.scala | 42 +++--- .../scala/org/apache/spark/util/ManualClock.scala | 16 ++--- .../spark/ExecutorAllocationManagerSuite.scala | 10 +++--- .../scheduler/dynalloc/ExecutorMonitorSuite.scala | 24 ++--- 6 files changed, 79 insertions(+), 36 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (1f1443e -> 9407fba)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 1f1443e [SPARK-29347][SQL] Add JSON serialization for external Rows add 9407fba [SPARK-29412][SQL] refine the document of v2 session catalog config No new revisions were added by this update. Summary of changes: .../spark/sql/connector/catalog/CatalogManager.scala | 8 .../org/apache/spark/sql/internal/SQLConf.scala | 14 +- .../DataSourceV2DataFrameSessionCatalogSuite.scala | 9 + .../spark/sql/connector/DataSourceV2SQLSuite.scala | 20 +++- .../sql/execution/command/PlanResolutionSuite.scala | 6 +++--- 5 files changed, 32 insertions(+), 25 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (ff9fcd5 -> 1f1443e)
This is an automated email from the ASF dual-hosted git repository. hvanhovell pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ff9fcd5 Revert "[SPARK-29107][SQL][TESTS] Port window.sql (Part 1)" add 1f1443e [SPARK-29347][SQL] Add JSON serialization for external Rows No new revisions were added by this update. Summary of changes: .../src/main/scala/org/apache/spark/sql/Row.scala | 108 +++- .../scala/org/apache/spark/sql/RowJsonSuite.scala | 140 + 2 files changed, 246 insertions(+), 2 deletions(-) create mode 100644 sql/catalyst/src/test/scala/org/apache/spark/sql/RowJsonSuite.scala - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (bfa09cf -> ff9fcd5)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from bfa09cf [SPARK-29463][SQL] move v2 commands to a new file add ff9fcd5 Revert "[SPARK-29107][SQL][TESTS] Port window.sql (Part 1)" No new revisions were added by this update. Summary of changes: .../sql-tests/inputs/pgSQL/window_part1.sql| 348 .../sql-tests/results/pgSQL/window_part1.sql.out | 902 - 2 files changed, 1250 deletions(-) delete mode 100644 sql/core/src/test/resources/sql-tests/inputs/pgSQL/window_part1.sql delete mode 100644 sql/core/src/test/resources/sql-tests/results/pgSQL/window_part1.sql.out - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (a42d894 -> bfa09cf)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from a42d894 [SPARK-29417][CORE] Resource Scheduling - add TaskContext.resource java api add bfa09cf [SPARK-29463][SQL] move v2 commands to a new file No new revisions were added by this update. Summary of changes: .../plans/logical/basicLogicalOperators.scala | 305 -- .../sql/catalyst/plans/logical/v2Commands.scala| 342 + 2 files changed, 342 insertions(+), 305 deletions(-) create mode 100644 sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-29417][CORE] Resource Scheduling - add TaskContext.resource java api
This is an automated email from the ASF dual-hosted git repository.
meng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new a42d894 [SPARK-29417][CORE] Resource Scheduling - add
TaskContext.resource java api
a42d894 is described below
commit a42d894a4090c97a90ce23b0989163909ebf548d
Author: Thomas Graves
AuthorDate: Mon Oct 14 13:27:34 2019 -0700
[SPARK-29417][CORE] Resource Scheduling - add TaskContext.resource java api
### What changes were proposed in this pull request?
We added a TaskContext.resources() api, but I realized this is returning a
scala Map which is not ideal for access from Java. Here I add a resourcesJMap
function which returns a java.util.Map to make it easily accessible from Java.
### Why are the changes needed?
Java API access
### Does this PR introduce any user-facing change?
Yes, new TaskContext function to access from Java
### How was this patch tested?
new unit test
Closes #26083 from tgravescs/SPARK-29417.
Lead-authored-by: Thomas Graves
Co-authored-by: Thomas Graves
Co-authored-by: Thomas Graves
Signed-off-by: Xiangrui Meng
---
core/src/main/scala/org/apache/spark/BarrierTaskContext.scala | 5 +
core/src/main/scala/org/apache/spark/TaskContext.scala| 8
core/src/main/scala/org/apache/spark/TaskContextImpl.scala| 5 +
.../java/test/org/apache/spark/JavaTaskContextCompileCheck.java | 5 +
project/MimaExcludes.scala| 3 +++
5 files changed, 26 insertions(+)
diff --git a/core/src/main/scala/org/apache/spark/BarrierTaskContext.scala
b/core/src/main/scala/org/apache/spark/BarrierTaskContext.scala
index 5afd8a5..3d36980 100644
--- a/core/src/main/scala/org/apache/spark/BarrierTaskContext.scala
+++ b/core/src/main/scala/org/apache/spark/BarrierTaskContext.scala
@@ -19,6 +19,7 @@ package org.apache.spark
import java.util.{Properties, Timer, TimerTask}
+import scala.collection.JavaConverters._
import scala.concurrent.TimeoutException
import scala.concurrent.duration._
@@ -211,6 +212,10 @@ class BarrierTaskContext private[spark] (
override def resources(): Map[String, ResourceInformation] =
taskContext.resources()
+ override def resourcesJMap(): java.util.Map[String, ResourceInformation] = {
+resources().asJava
+ }
+
override private[spark] def killTaskIfInterrupted(): Unit =
taskContext.killTaskIfInterrupted()
override private[spark] def getKillReason(): Option[String] =
taskContext.getKillReason()
diff --git a/core/src/main/scala/org/apache/spark/TaskContext.scala
b/core/src/main/scala/org/apache/spark/TaskContext.scala
index 2299c54..fd41fac 100644
--- a/core/src/main/scala/org/apache/spark/TaskContext.scala
+++ b/core/src/main/scala/org/apache/spark/TaskContext.scala
@@ -185,6 +185,14 @@ abstract class TaskContext extends Serializable {
@Evolving
def resources(): Map[String, ResourceInformation]
+ /**
+ * (java-specific) Resources allocated to the task. The key is the resource
name and the value
+ * is information about the resource. Please refer to
+ * [[org.apache.spark.resource.ResourceInformation]] for specifics.
+ */
+ @Evolving
+ def resourcesJMap(): java.util.Map[String, ResourceInformation]
+
@DeveloperApi
def taskMetrics(): TaskMetrics
diff --git a/core/src/main/scala/org/apache/spark/TaskContextImpl.scala
b/core/src/main/scala/org/apache/spark/TaskContextImpl.scala
index 516fb95..08a58a0 100644
--- a/core/src/main/scala/org/apache/spark/TaskContextImpl.scala
+++ b/core/src/main/scala/org/apache/spark/TaskContextImpl.scala
@@ -20,6 +20,7 @@ package org.apache.spark
import java.util.Properties
import javax.annotation.concurrent.GuardedBy
+import scala.collection.JavaConverters._
import scala.collection.mutable.ArrayBuffer
import org.apache.spark.executor.TaskMetrics
@@ -101,6 +102,10 @@ private[spark] class TaskContextImpl(
this
}
+ override def resourcesJMap(): java.util.Map[String, ResourceInformation] = {
+resources.asJava
+ }
+
@GuardedBy("this")
private[spark] override def markTaskFailed(error: Throwable): Unit =
synchronized {
if (failed) return
diff --git
a/core/src/test/java/test/org/apache/spark/JavaTaskContextCompileCheck.java
b/core/src/test/java/test/org/apache/spark/JavaTaskContextCompileCheck.java
index 62a0b85..5ce7937 100644
--- a/core/src/test/java/test/org/apache/spark/JavaTaskContextCompileCheck.java
+++ b/core/src/test/java/test/org/apache/spark/JavaTaskContextCompileCheck.java
@@ -17,7 +17,10 @@
package test.org.apache.spark;
+import java.util.Map;
+
import org.apache.spark.TaskContext;
+import org.apache.spark.resource.ResourceInformation;
import org.apache.spark.util.TaskCompletionListener;
import
[spark] branch master updated (e696c36 -> 52186af)
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from e696c36 [SPARK-29442][SQL] Set `default` mode should override the existing mode add 52186af [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes No new revisions were added by this update. Summary of changes: .../integration-tests/scripts/setup-integration-test-env.sh | 4 ++-- .../org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (81915da -> e696c36)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 81915da [SPARK-29107][SQL][TESTS] Port window.sql (Part 1) add e696c36 [SPARK-29442][SQL] Set `default` mode should override the existing mode No new revisions were added by this update. Summary of changes: .../src/main/scala/org/apache/spark/sql/DataFrameWriter.scala | 7 +++ sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala | 8 2 files changed, 11 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (ba04562 -> 81915da)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ba04562 [SPARK-29435][CORE] MapId in Shuffle Block is inconsistent at the writer and reader part when spark.shuffle.useOldFetchProtocol=true add 81915da [SPARK-29107][SQL][TESTS] Port window.sql (Part 1) No new revisions were added by this update. Summary of changes: .../sql-tests/inputs/pgSQL/window_part1.sql| 348 .../sql-tests/results/pgSQL/window_part1.sql.out | 902 + 2 files changed, 1250 insertions(+) create mode 100644 sql/core/src/test/resources/sql-tests/inputs/pgSQL/window_part1.sql create mode 100644 sql/core/src/test/resources/sql-tests/results/pgSQL/window_part1.sql.out - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (ba04562 -> 81915da)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ba04562 [SPARK-29435][CORE] MapId in Shuffle Block is inconsistent at the writer and reader part when spark.shuffle.useOldFetchProtocol=true add 81915da [SPARK-29107][SQL][TESTS] Port window.sql (Part 1) No new revisions were added by this update. Summary of changes: .../sql-tests/inputs/pgSQL/window_part1.sql| 348 .../sql-tests/results/pgSQL/window_part1.sql.out | 902 + 2 files changed, 1250 insertions(+) create mode 100644 sql/core/src/test/resources/sql-tests/inputs/pgSQL/window_part1.sql create mode 100644 sql/core/src/test/resources/sql-tests/results/pgSQL/window_part1.sql.out - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (cfcaf52 -> ba04562)
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from cfcaf52 [SPARK-29381][PYTHON][ML] Add _ before the XXXParams classes
add ba04562 [SPARK-29435][CORE] MapId in Shuffle Block is inconsistent at
the writer and reader part when spark.shuffle.useOldFetchProtocol=true
No new revisions were added by this update.
Summary of changes:
.../scala/org/apache/spark/MapOutputTracker.scala | 30 +++---
.../org/apache/spark/scheduler/MapStatus.scala | 9 ---
.../apache/spark/scheduler/ShuffleMapTask.scala| 9 +--
.../spark/shuffle/BlockStoreShuffleReader.scala| 3 +--
.../spark/shuffle/ShuffleWriteProcessor.scala | 3 ++-
.../org/apache/spark/MapOutputTrackerSuite.scala | 2 +-
...te.scala => ShuffleOldFetchProtocolSuite.scala} | 7 +++--
.../apache/spark/scheduler/DAGSchedulerSuite.scala | 4 +--
.../apache/spark/scheduler/MapStatusSuite.scala| 2 +-
.../shuffle/BlockStoreShuffleReaderSuite.scala | 2 +-
10 files changed, 32 insertions(+), 39 deletions(-)
copy core/src/test/scala/org/apache/spark/{ShuffleNettySuite.scala =>
ShuffleOldFetchProtocolSuite.scala} (80%)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (da576a7 -> cfcaf52)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from da576a7 [SPARK-29369][SQL] Support string intervals without the `interval` prefix add cfcaf52 [SPARK-29381][PYTHON][ML] Add _ before the XXXParams classes No new revisions were added by this update. Summary of changes: python/pyspark/ml/classification.py | 39 + python/pyspark/ml/wrapper.py| 10 ++ 2 files changed, 28 insertions(+), 21 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (ef6dce2 -> da576a7)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ef6dce2 [SPARK-29279][SQL] Merge SHOW NAMESPACES and SHOW DATABASES code path add da576a7 [SPARK-29369][SQL] Support string intervals without the `interval` prefix No new revisions were added by this update. Summary of changes: .../spark/unsafe/types/CalendarInterval.java | 59 ++ .../spark/unsafe/types/CalendarIntervalSuite.java | 69 +++--- .../spark/sql/catalyst/expressions/CastSuite.scala | 2 + .../catalyst/parser/ExpressionParserSuite.scala| 5 +- .../resources/sql-tests/results/literals.sql.out | 4 +- 5 files changed, 75 insertions(+), 64 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (67e1360 -> ef6dce2)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 67e1360 [SPARK-29377][PYTHON][ML] Parity between Scala ML tuning and Python ML tuning add ef6dce2 [SPARK-29279][SQL] Merge SHOW NAMESPACES and SHOW DATABASES code path No new revisions were added by this update. Summary of changes: .../apache/spark/sql/catalyst/parser/SqlBase.g4| 3 +- .../sql/catalyst/analysis/ResolveCatalogs.scala| 13 ++--- .../spark/sql/catalyst/parser/AstBuilder.scala | 4 ++ .../sql/connector/catalog/LookupCatalog.scala | 29 ++ .../spark/sql/catalyst/parser/DDLParserSuite.scala | 19 +++ .../sql/catalyst/parser/ParserUtilsSuite.scala | 2 +- .../catalyst/analysis/ResolveSessionCatalog.scala | 9 .../spark/sql/execution/SparkSqlParser.scala | 11 .../spark/sql/execution/command/databases.scala| 62 -- .../spark/sql/connector/DataSourceV2SQLSuite.scala | 29 ++ .../sql/execution/command/DDLParserSuite.scala | 11 11 files changed, 55 insertions(+), 137 deletions(-) delete mode 100644 sql/core/src/main/scala/org/apache/spark/sql/execution/command/databases.scala - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (67e1360 -> ef6dce2)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 67e1360 [SPARK-29377][PYTHON][ML] Parity between Scala ML tuning and Python ML tuning add ef6dce2 [SPARK-29279][SQL] Merge SHOW NAMESPACES and SHOW DATABASES code path No new revisions were added by this update. Summary of changes: .../apache/spark/sql/catalyst/parser/SqlBase.g4| 3 +- .../sql/catalyst/analysis/ResolveCatalogs.scala| 13 ++--- .../spark/sql/catalyst/parser/AstBuilder.scala | 4 ++ .../sql/connector/catalog/LookupCatalog.scala | 29 ++ .../spark/sql/catalyst/parser/DDLParserSuite.scala | 19 +++ .../sql/catalyst/parser/ParserUtilsSuite.scala | 2 +- .../catalyst/analysis/ResolveSessionCatalog.scala | 9 .../spark/sql/execution/SparkSqlParser.scala | 11 .../spark/sql/execution/command/databases.scala| 62 -- .../spark/sql/connector/DataSourceV2SQLSuite.scala | 29 ++ .../sql/execution/command/DDLParserSuite.scala | 11 11 files changed, 55 insertions(+), 137 deletions(-) delete mode 100644 sql/core/src/main/scala/org/apache/spark/sql/execution/command/databases.scala - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (ef81525 -> 67e1360)
This is an automated email from the ASF dual-hosted git repository. ruifengz pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ef81525 [SPARK-29308][BUILD] Update deps in dev/deps/spark-deps-hadoop-3.2 for hadoop-3.2 add 67e1360 [SPARK-29377][PYTHON][ML] Parity between Scala ML tuning and Python ML tuning No new revisions were added by this update. Summary of changes: python/pyspark/ml/tuning.py | 175 +--- 1 file changed, 132 insertions(+), 43 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
