[spark] branch master updated (abc8ccc -> 6fe3bf6)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from abc8ccc [SPARK-31926][SQL][TESTS][FOLLOWUP][TEST-HIVE1.2][TEST-MAVEN] Fix concurrency issue for ThriftCLIService to getPortNumber add 6fe3bf6 [SPARK-31993][SQL] Build arrays for passing variables generated from children for 'concat_ws' with columns having at least one of array type No new revisions were added by this update. Summary of changes: .../catalyst/expressions/stringExpressions.scala | 63 -- .../apache/spark/sql/StringFunctionsSuite.scala| 60 + 2 files changed, 106 insertions(+), 17 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (abc8ccc -> 6fe3bf6)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from abc8ccc [SPARK-31926][SQL][TESTS][FOLLOWUP][TEST-HIVE1.2][TEST-MAVEN] Fix concurrency issue for ThriftCLIService to getPortNumber add 6fe3bf6 [SPARK-31993][SQL] Build arrays for passing variables generated from children for 'concat_ws' with columns having at least one of array type No new revisions were added by this update. Summary of changes: .../catalyst/expressions/stringExpressions.scala | 63 -- .../apache/spark/sql/StringFunctionsSuite.scala| 60 + 2 files changed, 106 insertions(+), 17 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (86b54f3 -> abc8ccc)
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 86b54f3 [SPARK-31894][SS] Introduce UnsafeRow format validation for
streaming state store
add abc8ccc [SPARK-31926][SQL][TESTS][FOLLOWUP][TEST-HIVE1.2][TEST-MAVEN]
Fix concurrency issue for ThriftCLIService to getPortNumber
No new revisions were added by this update.
Summary of changes:
project/SparkBuild.scala | 1 -
.../sql/hive/thriftserver/SparkSQLCLIService.scala | 50 --
.../src/test/resources/log4j.properties| 2 +-
.../sql/hive/thriftserver/SharedThriftServer.scala | 60 --
.../thriftserver/ThriftServerQueryTestSuite.scala | 3 ++
.../ThriftServerWithSparkContextSuite.scala| 11 +++-
.../service/cli/thrift/ThriftBinaryCLIService.java | 11 +++-
.../hive/service/cli/thrift/ThriftCLIService.java | 3 ++
.../service/cli/thrift/ThriftHttpCLIService.java | 21 +---
.../service/cli/thrift/ThriftBinaryCLIService.java | 11 +++-
.../hive/service/cli/thrift/ThriftCLIService.java | 3 ++
.../service/cli/thrift/ThriftHttpCLIService.java | 21 +---
12 files changed, 160 insertions(+), 37 deletions(-)
copy sql/{hive => hive-thriftserver}/src/test/resources/log4j.properties (96%)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (86b54f3 -> abc8ccc)
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 86b54f3 [SPARK-31894][SS] Introduce UnsafeRow format validation for
streaming state store
add abc8ccc [SPARK-31926][SQL][TESTS][FOLLOWUP][TEST-HIVE1.2][TEST-MAVEN]
Fix concurrency issue for ThriftCLIService to getPortNumber
No new revisions were added by this update.
Summary of changes:
project/SparkBuild.scala | 1 -
.../sql/hive/thriftserver/SparkSQLCLIService.scala | 50 --
.../src/test/resources/log4j.properties| 2 +-
.../sql/hive/thriftserver/SharedThriftServer.scala | 60 --
.../thriftserver/ThriftServerQueryTestSuite.scala | 3 ++
.../ThriftServerWithSparkContextSuite.scala| 11 +++-
.../service/cli/thrift/ThriftBinaryCLIService.java | 11 +++-
.../hive/service/cli/thrift/ThriftCLIService.java | 3 ++
.../service/cli/thrift/ThriftHttpCLIService.java | 21 +---
.../service/cli/thrift/ThriftBinaryCLIService.java | 11 +++-
.../hive/service/cli/thrift/ThriftCLIService.java | 3 ++
.../service/cli/thrift/ThriftHttpCLIService.java | 21 +---
12 files changed, 160 insertions(+), 37 deletions(-)
copy sql/{hive => hive-thriftserver}/src/test/resources/log4j.properties (96%)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (86b54f3 -> abc8ccc)
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 86b54f3 [SPARK-31894][SS] Introduce UnsafeRow format validation for
streaming state store
add abc8ccc [SPARK-31926][SQL][TESTS][FOLLOWUP][TEST-HIVE1.2][TEST-MAVEN]
Fix concurrency issue for ThriftCLIService to getPortNumber
No new revisions were added by this update.
Summary of changes:
project/SparkBuild.scala | 1 -
.../sql/hive/thriftserver/SparkSQLCLIService.scala | 50 --
.../src/test/resources/log4j.properties| 2 +-
.../sql/hive/thriftserver/SharedThriftServer.scala | 60 --
.../thriftserver/ThriftServerQueryTestSuite.scala | 3 ++
.../ThriftServerWithSparkContextSuite.scala| 11 +++-
.../service/cli/thrift/ThriftBinaryCLIService.java | 11 +++-
.../hive/service/cli/thrift/ThriftCLIService.java | 3 ++
.../service/cli/thrift/ThriftHttpCLIService.java | 21 +---
.../service/cli/thrift/ThriftBinaryCLIService.java | 11 +++-
.../hive/service/cli/thrift/ThriftCLIService.java | 3 ++
.../service/cli/thrift/ThriftHttpCLIService.java | 21 +---
12 files changed, 160 insertions(+), 37 deletions(-)
copy sql/{hive => hive-thriftserver}/src/test/resources/log4j.properties (96%)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (17a5007 -> 86b54f3)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 17a5007 [SPARK-30865][SQL][SS] Refactor DateTimeUtils add 86b54f3 [SPARK-31894][SS] Introduce UnsafeRow format validation for streaming state store No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/util/UnsafeRowUtils.scala | 86 ++ .../org/apache/spark/sql/internal/SQLConf.scala| 16 +++- .../sql/catalyst/util/UnsafeRowUtilsSuite.scala| 55 ++ .../state/HDFSBackedStateStoreProvider.scala | 13 .../sql/execution/streaming/state/StateStore.scala | 33 - .../execution/streaming/state/StateStoreConf.scala | 19 - .../execution/streaming/state/StateStoreRDD.scala | 5 +- .../sql/execution/streaming/state/package.scala| 6 +- .../execution/streaming/statefulOperators.scala| 6 +- 9 files changed, 228 insertions(+), 11 deletions(-) create mode 100644 sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/UnsafeRowUtils.scala create mode 100644 sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/UnsafeRowUtilsSuite.scala - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (17a5007 -> 86b54f3)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 17a5007 [SPARK-30865][SQL][SS] Refactor DateTimeUtils add 86b54f3 [SPARK-31894][SS] Introduce UnsafeRow format validation for streaming state store No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/util/UnsafeRowUtils.scala | 86 ++ .../org/apache/spark/sql/internal/SQLConf.scala| 16 +++- .../sql/catalyst/util/UnsafeRowUtilsSuite.scala| 55 ++ .../state/HDFSBackedStateStoreProvider.scala | 13 .../sql/execution/streaming/state/StateStore.scala | 33 - .../execution/streaming/state/StateStoreConf.scala | 19 - .../execution/streaming/state/StateStoreRDD.scala | 5 +- .../sql/execution/streaming/state/package.scala| 6 +- .../execution/streaming/statefulOperators.scala| 6 +- 9 files changed, 228 insertions(+), 11 deletions(-) create mode 100644 sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/UnsafeRowUtils.scala create mode 100644 sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/UnsafeRowUtilsSuite.scala - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (8750363 -> 17a5007)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8750363 [MINOR][DOCS] Emphasize the Streaming tab is for DStream API add 17a5007 [SPARK-30865][SQL][SS] Refactor DateTimeUtils No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/expressions/Cast.scala | 13 +- .../catalyst/expressions/datetimeExpressions.scala | 12 +- .../spark/sql/catalyst/util/DateTimeUtils.scala| 393 ++--- .../sql/catalyst/util/TimestampFormatter.scala | 6 +- .../spark/sql/catalyst/expressions/CastSuite.scala | 13 +- .../catalyst/expressions/CsvExpressionsSuite.scala | 4 +- .../expressions/DateExpressionsSuite.scala | 5 +- .../expressions/ObjectExpressionsSuite.scala | 4 +- .../sql/catalyst/json/JacksonGeneratorSuite.scala | 8 +- .../sql/catalyst/util/DateTimeUtilsSuite.scala | 10 +- .../sql/catalyst/util/DatetimeFormatterSuite.scala | 2 +- .../sql/catalyst/util/RebaseDateTimeSuite.scala| 2 +- .../apache/spark/sql/util/ArrowUtilsSuite.scala| 5 +- .../execution/datasources/jdbc/JDBCRelation.scala | 2 +- .../datasources/parquet/ParquetFilters.scala | 3 +- .../datasources/parquet/ParquetRowConverter.scala | 3 +- .../apache/spark/sql/DataFrameFunctionsSuite.scala | 6 +- .../spark/sql/StatisticsCollectionSuite.scala | 7 +- .../sql/execution/datasources/json/JsonSuite.scala | 4 +- .../parquet/ParquetPartitionDiscoverySuite.scala | 5 +- .../sql/sources/ResolvedDataSourceSuite.scala | 4 +- .../apache/spark/sql/streaming/StreamSuite.scala | 4 +- 22 files changed, 241 insertions(+), 274 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (8750363 -> 17a5007)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8750363 [MINOR][DOCS] Emphasize the Streaming tab is for DStream API add 17a5007 [SPARK-30865][SQL][SS] Refactor DateTimeUtils No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/expressions/Cast.scala | 13 +- .../catalyst/expressions/datetimeExpressions.scala | 12 +- .../spark/sql/catalyst/util/DateTimeUtils.scala| 393 ++--- .../sql/catalyst/util/TimestampFormatter.scala | 6 +- .../spark/sql/catalyst/expressions/CastSuite.scala | 13 +- .../catalyst/expressions/CsvExpressionsSuite.scala | 4 +- .../expressions/DateExpressionsSuite.scala | 5 +- .../expressions/ObjectExpressionsSuite.scala | 4 +- .../sql/catalyst/json/JacksonGeneratorSuite.scala | 8 +- .../sql/catalyst/util/DateTimeUtilsSuite.scala | 10 +- .../sql/catalyst/util/DatetimeFormatterSuite.scala | 2 +- .../sql/catalyst/util/RebaseDateTimeSuite.scala| 2 +- .../apache/spark/sql/util/ArrowUtilsSuite.scala| 5 +- .../execution/datasources/jdbc/JDBCRelation.scala | 2 +- .../datasources/parquet/ParquetFilters.scala | 3 +- .../datasources/parquet/ParquetRowConverter.scala | 3 +- .../apache/spark/sql/DataFrameFunctionsSuite.scala | 6 +- .../spark/sql/StatisticsCollectionSuite.scala | 7 +- .../sql/execution/datasources/json/JsonSuite.scala | 4 +- .../parquet/ParquetPartitionDiscoverySuite.scala | 5 +- .../sql/sources/ResolvedDataSourceSuite.scala | 4 +- .../apache/spark/sql/streaming/StreamSuite.scala | 4 +- 22 files changed, 241 insertions(+), 274 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (8750363 -> 17a5007)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8750363 [MINOR][DOCS] Emphasize the Streaming tab is for DStream API add 17a5007 [SPARK-30865][SQL][SS] Refactor DateTimeUtils No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/expressions/Cast.scala | 13 +- .../catalyst/expressions/datetimeExpressions.scala | 12 +- .../spark/sql/catalyst/util/DateTimeUtils.scala| 393 ++--- .../sql/catalyst/util/TimestampFormatter.scala | 6 +- .../spark/sql/catalyst/expressions/CastSuite.scala | 13 +- .../catalyst/expressions/CsvExpressionsSuite.scala | 4 +- .../expressions/DateExpressionsSuite.scala | 5 +- .../expressions/ObjectExpressionsSuite.scala | 4 +- .../sql/catalyst/json/JacksonGeneratorSuite.scala | 8 +- .../sql/catalyst/util/DateTimeUtilsSuite.scala | 10 +- .../sql/catalyst/util/DatetimeFormatterSuite.scala | 2 +- .../sql/catalyst/util/RebaseDateTimeSuite.scala| 2 +- .../apache/spark/sql/util/ArrowUtilsSuite.scala| 5 +- .../execution/datasources/jdbc/JDBCRelation.scala | 2 +- .../datasources/parquet/ParquetFilters.scala | 3 +- .../datasources/parquet/ParquetRowConverter.scala | 3 +- .../apache/spark/sql/DataFrameFunctionsSuite.scala | 6 +- .../spark/sql/StatisticsCollectionSuite.scala | 7 +- .../sql/execution/datasources/json/JsonSuite.scala | 4 +- .../parquet/ParquetPartitionDiscoverySuite.scala | 5 +- .../sql/sources/ResolvedDataSourceSuite.scala | 4 +- .../apache/spark/sql/streaming/StreamSuite.scala | 4 +- 22 files changed, 241 insertions(+), 274 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (ac98a9a -> 8750363)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ac98a9a [MINOR][DOCS] Update running-on-kubernetes.md add 8750363 [MINOR][DOCS] Emphasize the Streaming tab is for DStream API No new revisions were added by this update. Summary of changes: docs/web-ui.md | 8 1 file changed, 4 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [MINOR][DOCS] Emphasize the Streaming tab is for DStream API
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 73af23c [MINOR][DOCS] Emphasize the Streaming tab is for DStream API 73af23c is described below commit 73af23cabee6aac471fbdc9f4a35d118517fad65 Author: Yuanjian Li AuthorDate: Fri Jun 19 12:17:40 2020 +0900 [MINOR][DOCS] Emphasize the Streaming tab is for DStream API ### What changes were proposed in this pull request? Emphasize the Streaming tab is for DStream API. ### Why are the changes needed? Some users reported that it's a little confusing of the streaming tab and structured streaming tab. ### Does this PR introduce _any_ user-facing change? Document change. ### How was this patch tested? N/A Closes #28854 from xuanyuanking/minor-doc. Authored-by: Yuanjian Li Signed-off-by: HyukjinKwon (cherry picked from commit 8750363c8ddaa649d52e6d9001b9871d379fc4e1) Signed-off-by: HyukjinKwon --- docs/web-ui.md | 8 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/web-ui.md b/docs/web-ui.md index 6653b00..134a8c8 100644 --- a/docs/web-ui.md +++ b/docs/web-ui.md @@ -434,10 +434,10 @@ The tracked operations are listed as follows. As an early-release version, the statistics page is still under development and will be improved in future releases. -## Streaming Tab -The web UI includes a Streaming tab if the application uses Spark streaming. This tab displays -scheduling delay and processing time for each micro-batch in the data stream, which can be useful -for troubleshooting the streaming application. +## Streaming (DStreams) Tab +The web UI includes a Streaming tab if the application uses Spark Streaming with DStream API. +This tab displays scheduling delay and processing time for each micro-batch in the data stream, +which can be useful for troubleshooting the streaming application. ## JDBC/ODBC Server Tab We can see this tab when Spark is running as a [distributed SQL engine](sql-distributed-sql-engine.html). It shows information about sessions and submitted SQL operations. - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (ac98a9a -> 8750363)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ac98a9a [MINOR][DOCS] Update running-on-kubernetes.md add 8750363 [MINOR][DOCS] Emphasize the Streaming tab is for DStream API No new revisions were added by this update. Summary of changes: docs/web-ui.md | 8 1 file changed, 4 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [MINOR][DOCS] Emphasize the Streaming tab is for DStream API
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 73af23c [MINOR][DOCS] Emphasize the Streaming tab is for DStream API 73af23c is described below commit 73af23cabee6aac471fbdc9f4a35d118517fad65 Author: Yuanjian Li AuthorDate: Fri Jun 19 12:17:40 2020 +0900 [MINOR][DOCS] Emphasize the Streaming tab is for DStream API ### What changes were proposed in this pull request? Emphasize the Streaming tab is for DStream API. ### Why are the changes needed? Some users reported that it's a little confusing of the streaming tab and structured streaming tab. ### Does this PR introduce _any_ user-facing change? Document change. ### How was this patch tested? N/A Closes #28854 from xuanyuanking/minor-doc. Authored-by: Yuanjian Li Signed-off-by: HyukjinKwon (cherry picked from commit 8750363c8ddaa649d52e6d9001b9871d379fc4e1) Signed-off-by: HyukjinKwon --- docs/web-ui.md | 8 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/web-ui.md b/docs/web-ui.md index 6653b00..134a8c8 100644 --- a/docs/web-ui.md +++ b/docs/web-ui.md @@ -434,10 +434,10 @@ The tracked operations are listed as follows. As an early-release version, the statistics page is still under development and will be improved in future releases. -## Streaming Tab -The web UI includes a Streaming tab if the application uses Spark streaming. This tab displays -scheduling delay and processing time for each micro-batch in the data stream, which can be useful -for troubleshooting the streaming application. +## Streaming (DStreams) Tab +The web UI includes a Streaming tab if the application uses Spark Streaming with DStream API. +This tab displays scheduling delay and processing time for each micro-batch in the data stream, +which can be useful for troubleshooting the streaming application. ## JDBC/ODBC Server Tab We can see this tab when Spark is running as a [distributed SQL engine](sql-distributed-sql-engine.html). It shows information about sessions and submitted SQL operations. - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (ac98a9a -> 8750363)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ac98a9a [MINOR][DOCS] Update running-on-kubernetes.md add 8750363 [MINOR][DOCS] Emphasize the Streaming tab is for DStream API No new revisions were added by this update. Summary of changes: docs/web-ui.md | 8 1 file changed, 4 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [MINOR][DOCS] Emphasize the Streaming tab is for DStream API
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 73af23c [MINOR][DOCS] Emphasize the Streaming tab is for DStream API 73af23c is described below commit 73af23cabee6aac471fbdc9f4a35d118517fad65 Author: Yuanjian Li AuthorDate: Fri Jun 19 12:17:40 2020 +0900 [MINOR][DOCS] Emphasize the Streaming tab is for DStream API ### What changes were proposed in this pull request? Emphasize the Streaming tab is for DStream API. ### Why are the changes needed? Some users reported that it's a little confusing of the streaming tab and structured streaming tab. ### Does this PR introduce _any_ user-facing change? Document change. ### How was this patch tested? N/A Closes #28854 from xuanyuanking/minor-doc. Authored-by: Yuanjian Li Signed-off-by: HyukjinKwon (cherry picked from commit 8750363c8ddaa649d52e6d9001b9871d379fc4e1) Signed-off-by: HyukjinKwon --- docs/web-ui.md | 8 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/web-ui.md b/docs/web-ui.md index 6653b00..134a8c8 100644 --- a/docs/web-ui.md +++ b/docs/web-ui.md @@ -434,10 +434,10 @@ The tracked operations are listed as follows. As an early-release version, the statistics page is still under development and will be improved in future releases. -## Streaming Tab -The web UI includes a Streaming tab if the application uses Spark streaming. This tab displays -scheduling delay and processing time for each micro-batch in the data stream, which can be useful -for troubleshooting the streaming application. +## Streaming (DStreams) Tab +The web UI includes a Streaming tab if the application uses Spark Streaming with DStream API. +This tab displays scheduling delay and processing time for each micro-batch in the data stream, +which can be useful for troubleshooting the streaming application. ## JDBC/ODBC Server Tab We can see this tab when Spark is running as a [distributed SQL engine](sql-distributed-sql-engine.html). It shows information about sessions and submitted SQL operations. - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (ac98a9a -> 8750363)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ac98a9a [MINOR][DOCS] Update running-on-kubernetes.md add 8750363 [MINOR][DOCS] Emphasize the Streaming tab is for DStream API No new revisions were added by this update. Summary of changes: docs/web-ui.md | 8 1 file changed, 4 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [MINOR][DOCS] Emphasize the Streaming tab is for DStream API
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 73af23c [MINOR][DOCS] Emphasize the Streaming tab is for DStream API 73af23c is described below commit 73af23cabee6aac471fbdc9f4a35d118517fad65 Author: Yuanjian Li AuthorDate: Fri Jun 19 12:17:40 2020 +0900 [MINOR][DOCS] Emphasize the Streaming tab is for DStream API ### What changes were proposed in this pull request? Emphasize the Streaming tab is for DStream API. ### Why are the changes needed? Some users reported that it's a little confusing of the streaming tab and structured streaming tab. ### Does this PR introduce _any_ user-facing change? Document change. ### How was this patch tested? N/A Closes #28854 from xuanyuanking/minor-doc. Authored-by: Yuanjian Li Signed-off-by: HyukjinKwon (cherry picked from commit 8750363c8ddaa649d52e6d9001b9871d379fc4e1) Signed-off-by: HyukjinKwon --- docs/web-ui.md | 8 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/web-ui.md b/docs/web-ui.md index 6653b00..134a8c8 100644 --- a/docs/web-ui.md +++ b/docs/web-ui.md @@ -434,10 +434,10 @@ The tracked operations are listed as follows. As an early-release version, the statistics page is still under development and will be improved in future releases. -## Streaming Tab -The web UI includes a Streaming tab if the application uses Spark streaming. This tab displays -scheduling delay and processing time for each micro-batch in the data stream, which can be useful -for troubleshooting the streaming application. +## Streaming (DStreams) Tab +The web UI includes a Streaming tab if the application uses Spark Streaming with DStream API. +This tab displays scheduling delay and processing time for each micro-batch in the data stream, +which can be useful for troubleshooting the streaming application. ## JDBC/ODBC Server Tab We can see this tab when Spark is running as a [distributed SQL engine](sql-distributed-sql-engine.html). It shows information about sessions and submitted SQL operations. - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (ac98a9a -> 8750363)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ac98a9a [MINOR][DOCS] Update running-on-kubernetes.md add 8750363 [MINOR][DOCS] Emphasize the Streaming tab is for DStream API No new revisions were added by this update. Summary of changes: docs/web-ui.md | 8 1 file changed, 4 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [MINOR][DOCS] Emphasize the Streaming tab is for DStream API
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 73af23c [MINOR][DOCS] Emphasize the Streaming tab is for DStream API 73af23c is described below commit 73af23cabee6aac471fbdc9f4a35d118517fad65 Author: Yuanjian Li AuthorDate: Fri Jun 19 12:17:40 2020 +0900 [MINOR][DOCS] Emphasize the Streaming tab is for DStream API ### What changes were proposed in this pull request? Emphasize the Streaming tab is for DStream API. ### Why are the changes needed? Some users reported that it's a little confusing of the streaming tab and structured streaming tab. ### Does this PR introduce _any_ user-facing change? Document change. ### How was this patch tested? N/A Closes #28854 from xuanyuanking/minor-doc. Authored-by: Yuanjian Li Signed-off-by: HyukjinKwon (cherry picked from commit 8750363c8ddaa649d52e6d9001b9871d379fc4e1) Signed-off-by: HyukjinKwon --- docs/web-ui.md | 8 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/web-ui.md b/docs/web-ui.md index 6653b00..134a8c8 100644 --- a/docs/web-ui.md +++ b/docs/web-ui.md @@ -434,10 +434,10 @@ The tracked operations are listed as follows. As an early-release version, the statistics page is still under development and will be improved in future releases. -## Streaming Tab -The web UI includes a Streaming tab if the application uses Spark streaming. This tab displays -scheduling delay and processing time for each micro-batch in the data stream, which can be useful -for troubleshooting the streaming application. +## Streaming (DStreams) Tab +The web UI includes a Streaming tab if the application uses Spark Streaming with DStream API. +This tab displays scheduling delay and processing time for each micro-batch in the data stream, +which can be useful for troubleshooting the streaming application. ## JDBC/ODBC Server Tab We can see this tab when Spark is running as a [distributed SQL engine](sql-distributed-sql-engine.html). It shows information about sessions and submitted SQL operations. - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] dongjoon-hyun edited a comment on pull request #273: Update Release Window from 3.0 to 3.1
dongjoon-hyun edited a comment on pull request #273: URL: https://github.com/apache/spark-website/pull/273#issuecomment-646372759 cc @dbtsai , @holdenk , @gatorsmile , @rxin , @cloud-fan , @srowen , @HyukjinKwon , @maropu , @rdblue This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] dongjoon-hyun commented on pull request #273: Update Release Window from 3.0 to 3.1
dongjoon-hyun commented on pull request #273: URL: https://github.com/apache/spark-website/pull/273#issuecomment-646372759 cc @dbtsai , @holdenk , @gatorsmile , @rxin , @cloud-fan , @srowen , @HyukjinKwon , @maropu This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] dongjoon-hyun opened a new pull request #273: Update Release Window from 3.0 to 3.1
dongjoon-hyun opened a new pull request #273: URL: https://github.com/apache/spark-website/pull/273 According to our release cadence, this PR aims to update release window from 3.0 to 3.1. > In general, feature minor releases occur about every 6 months. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [MINOR][DOCS] Update running-on-kubernetes.md
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new cfeb7fa [MINOR][DOCS] Update running-on-kubernetes.md
cfeb7fa is described below
commit cfeb7faf215b006463c5e676ebf7a3317f7730f1
Author: James Yu
AuthorDate: Thu Jun 18 14:36:20 2020 -0700
[MINOR][DOCS] Update running-on-kubernetes.md
### What changes were proposed in this pull request?
Fix executor container name typo. `executor` should be
`spark-kubernetes-executor`.
### Why are the changes needed?
The Executor pod container name the users actually get from their
Kubernetes clusters is different from that described in the documentation.
For example, below is what a user get from an executor pod.
```
Containers:
spark-kubernetes-executor:
Container ID: docker://
Image:
Image ID:
docker-pullable://.dkr.ecr.us-east-0.amazonaws.com/spark
Port: 7079/TCP
Host Port: 0/TCP
Args:
executor
State: Running
Started: Thu, 28 May 2020 05:54:04 -0700
Ready: True
Restart Count: 0
Limits:
memory: 16Gi
```
### Does this PR introduce _any_ user-facing change?
Document change.
### How was this patch tested?
N/A
Closes #28862 from yuj/patch-1.
Authored-by: James Yu
Signed-off-by: Dongjoon Hyun
(cherry picked from commit ac98a9a07fe3e76c16bd5bb03aca553e80629e01)
Signed-off-by: Dongjoon Hyun
---
docs/running-on-kubernetes.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md
index 3abb891..e6a000d 100644
--- a/docs/running-on-kubernetes.md
+++ b/docs/running-on-kubernetes.md
@@ -1333,7 +1333,7 @@ The following affect the driver and executor containers.
All other containers in
See description
The container name will be assigned by spark ("spark-kubernetes-driver"
for the driver container, and
-"executor" for each executor container) if not defined by the pod
template. If the container is defined by the
+"spark-kubernetes-executor" for each executor container) if not defined by
the pod template. If the container is defined by the
template, the template's name will be used.
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [MINOR][DOCS] Update running-on-kubernetes.md
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new cfeb7fa [MINOR][DOCS] Update running-on-kubernetes.md
cfeb7fa is described below
commit cfeb7faf215b006463c5e676ebf7a3317f7730f1
Author: James Yu
AuthorDate: Thu Jun 18 14:36:20 2020 -0700
[MINOR][DOCS] Update running-on-kubernetes.md
### What changes were proposed in this pull request?
Fix executor container name typo. `executor` should be
`spark-kubernetes-executor`.
### Why are the changes needed?
The Executor pod container name the users actually get from their
Kubernetes clusters is different from that described in the documentation.
For example, below is what a user get from an executor pod.
```
Containers:
spark-kubernetes-executor:
Container ID: docker://
Image:
Image ID:
docker-pullable://.dkr.ecr.us-east-0.amazonaws.com/spark
Port: 7079/TCP
Host Port: 0/TCP
Args:
executor
State: Running
Started: Thu, 28 May 2020 05:54:04 -0700
Ready: True
Restart Count: 0
Limits:
memory: 16Gi
```
### Does this PR introduce _any_ user-facing change?
Document change.
### How was this patch tested?
N/A
Closes #28862 from yuj/patch-1.
Authored-by: James Yu
Signed-off-by: Dongjoon Hyun
(cherry picked from commit ac98a9a07fe3e76c16bd5bb03aca553e80629e01)
Signed-off-by: Dongjoon Hyun
---
docs/running-on-kubernetes.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md
index 3abb891..e6a000d 100644
--- a/docs/running-on-kubernetes.md
+++ b/docs/running-on-kubernetes.md
@@ -1333,7 +1333,7 @@ The following affect the driver and executor containers.
All other containers in
See description
The container name will be assigned by spark ("spark-kubernetes-driver"
for the driver container, and
-"executor" for each executor container) if not defined by the pod
template. If the container is defined by the
+"spark-kubernetes-executor" for each executor container) if not defined by
the pod template. If the container is defined by the
template, the template's name will be used.
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (8a9ae01 -> ac98a9a)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8a9ae01 [MINOR] update dev/create-release/known_translations add ac98a9a [MINOR][DOCS] Update running-on-kubernetes.md No new revisions were added by this update. Summary of changes: docs/running-on-kubernetes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [MINOR][DOCS] Update running-on-kubernetes.md
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new cfeb7fa [MINOR][DOCS] Update running-on-kubernetes.md
cfeb7fa is described below
commit cfeb7faf215b006463c5e676ebf7a3317f7730f1
Author: James Yu
AuthorDate: Thu Jun 18 14:36:20 2020 -0700
[MINOR][DOCS] Update running-on-kubernetes.md
### What changes were proposed in this pull request?
Fix executor container name typo. `executor` should be
`spark-kubernetes-executor`.
### Why are the changes needed?
The Executor pod container name the users actually get from their
Kubernetes clusters is different from that described in the documentation.
For example, below is what a user get from an executor pod.
```
Containers:
spark-kubernetes-executor:
Container ID: docker://
Image:
Image ID:
docker-pullable://.dkr.ecr.us-east-0.amazonaws.com/spark
Port: 7079/TCP
Host Port: 0/TCP
Args:
executor
State: Running
Started: Thu, 28 May 2020 05:54:04 -0700
Ready: True
Restart Count: 0
Limits:
memory: 16Gi
```
### Does this PR introduce _any_ user-facing change?
Document change.
### How was this patch tested?
N/A
Closes #28862 from yuj/patch-1.
Authored-by: James Yu
Signed-off-by: Dongjoon Hyun
(cherry picked from commit ac98a9a07fe3e76c16bd5bb03aca553e80629e01)
Signed-off-by: Dongjoon Hyun
---
docs/running-on-kubernetes.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md
index 3abb891..e6a000d 100644
--- a/docs/running-on-kubernetes.md
+++ b/docs/running-on-kubernetes.md
@@ -1333,7 +1333,7 @@ The following affect the driver and executor containers.
All other containers in
See description
The container name will be assigned by spark ("spark-kubernetes-driver"
for the driver container, and
-"executor" for each executor container) if not defined by the pod
template. If the container is defined by the
+"spark-kubernetes-executor" for each executor container) if not defined by
the pod template. If the container is defined by the
template, the template's name will be used.
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (8a9ae01 -> ac98a9a)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8a9ae01 [MINOR] update dev/create-release/known_translations add ac98a9a [MINOR][DOCS] Update running-on-kubernetes.md No new revisions were added by this update. Summary of changes: docs/running-on-kubernetes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [MINOR][DOCS] Update running-on-kubernetes.md
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new cfeb7fa [MINOR][DOCS] Update running-on-kubernetes.md
cfeb7fa is described below
commit cfeb7faf215b006463c5e676ebf7a3317f7730f1
Author: James Yu
AuthorDate: Thu Jun 18 14:36:20 2020 -0700
[MINOR][DOCS] Update running-on-kubernetes.md
### What changes were proposed in this pull request?
Fix executor container name typo. `executor` should be
`spark-kubernetes-executor`.
### Why are the changes needed?
The Executor pod container name the users actually get from their
Kubernetes clusters is different from that described in the documentation.
For example, below is what a user get from an executor pod.
```
Containers:
spark-kubernetes-executor:
Container ID: docker://
Image:
Image ID:
docker-pullable://.dkr.ecr.us-east-0.amazonaws.com/spark
Port: 7079/TCP
Host Port: 0/TCP
Args:
executor
State: Running
Started: Thu, 28 May 2020 05:54:04 -0700
Ready: True
Restart Count: 0
Limits:
memory: 16Gi
```
### Does this PR introduce _any_ user-facing change?
Document change.
### How was this patch tested?
N/A
Closes #28862 from yuj/patch-1.
Authored-by: James Yu
Signed-off-by: Dongjoon Hyun
(cherry picked from commit ac98a9a07fe3e76c16bd5bb03aca553e80629e01)
Signed-off-by: Dongjoon Hyun
---
docs/running-on-kubernetes.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md
index 3abb891..e6a000d 100644
--- a/docs/running-on-kubernetes.md
+++ b/docs/running-on-kubernetes.md
@@ -1333,7 +1333,7 @@ The following affect the driver and executor containers.
All other containers in
See description
The container name will be assigned by spark ("spark-kubernetes-driver"
for the driver container, and
-"executor" for each executor container) if not defined by the pod
template. If the container is defined by the
+"spark-kubernetes-executor" for each executor container) if not defined by
the pod template. If the container is defined by the
template, the template's name will be used.
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (8a9ae01 -> ac98a9a)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8a9ae01 [MINOR] update dev/create-release/known_translations add ac98a9a [MINOR][DOCS] Update running-on-kubernetes.md No new revisions were added by this update. Summary of changes: docs/running-on-kubernetes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (8a9ae01 -> ac98a9a)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8a9ae01 [MINOR] update dev/create-release/known_translations add ac98a9a [MINOR][DOCS] Update running-on-kubernetes.md No new revisions were added by this update. Summary of changes: docs/running-on-kubernetes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] dongjoon-hyun commented on pull request #272: Remove preview download direction.
dongjoon-hyun commented on pull request #272: URL: https://github.com/apache/spark-website/pull/272#issuecomment-646240304 Thank you, @srowen . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] dongjoon-hyun merged pull request #272: Remove preview download direction.
dongjoon-hyun merged pull request #272: URL: https://github.com/apache/spark-website/pull/272 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark-website] branch asf-site updated: Remove preview download direction.
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/spark-website.git The following commit(s) were added to refs/heads/asf-site by this push: new 38eaa66 Remove preview download direction. 38eaa66 is described below commit 38eaa66253bce92bbe9a46fd45d83dc46978fabf Author: Dongjoon Hyun AuthorDate: Thu Jun 18 11:42:30 2020 -0700 Remove preview download direction. --- downloads.md| 1 - site/downloads.html | 3 +-- 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/downloads.md b/downloads.md index f385cfc..2ed9870 100644 --- a/downloads.md +++ b/downloads.md @@ -35,7 +35,6 @@ to satisfy the legal requirements of Apache Software Foundation's release policy Preview releases are not meant to be functional, i.e. they can and highly likely will contain critical bugs or documentation errors. The latest preview release is Spark 3.0.0-preview2, published on Dec 23, 2019. -You can select and download it above. ### Link with Spark Spark artifacts are [hosted in Maven Central](https://search.maven.org/search?q=g:org.apache.spark). You can add a Maven dependency with the following coordinates: diff --git a/site/downloads.html b/site/downloads.html index e8f7da6..e3b060f 100644 --- a/site/downloads.html +++ b/site/downloads.html @@ -234,8 +234,7 @@ Unlike nightly packages, preview releases have been audited by the project’ to satisfy the legal requirements of Apache Software Foundation’s release policy. Preview releases are not meant to be functional, i.e. they can and highly likely will contain critical bugs or documentation errors. -The latest preview release is Spark 3.0.0-preview2, published on Dec 23, 2019. -You can select and download it above. +The latest preview release is Spark 3.0.0-preview2, published on Dec 23, 2019. Link with Spark Spark artifacts are https://search.maven.org/search?q=g:org.apache.spark";>hosted in Maven Central. You can add a Maven dependency with the following coordinates: - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] dongjoon-hyun commented on pull request #272: Remove preview download direction.
dongjoon-hyun commented on pull request #272: URL: https://github.com/apache/spark-website/pull/272#issuecomment-646212348 We may want to delete the whole section, `Latest Preview Release`, but for the the definition and warning looks worthy to have if we have a next preview release someday. So, I only remove the invalid part. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] dongjoon-hyun commented on pull request #272: Remove preview download direction.
dongjoon-hyun commented on pull request #272: URL: https://github.com/apache/spark-website/pull/272#issuecomment-646211844 cc @cloud-fan , @gatorsmile , @srowen . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] dongjoon-hyun opened a new pull request #272: Remove preview download direction.
dongjoon-hyun opened a new pull request #272: URL: https://github.com/apache/spark-website/pull/272 This update the download direction because preview is removed from the dropbox. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] gatorsmile merged pull request #271: the date of the post should be the date you create it
gatorsmile merged pull request #271: URL: https://github.com/apache/spark-website/pull/271 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] cloud-fan commented on a change in pull request #271: the date of the post should be the date you create it
cloud-fan commented on a change in pull request #271: URL: https://github.com/apache/spark-website/pull/271#discussion_r442391584 ## File path: releases/_posts/2020-06-18-spark-release-3-0-0.md ## @@ -11,7 +11,7 @@ meta: _wpas_done_all: '1' --- -Apache Spark 3.0.0 is the first release of the 3.x line. Apache Spark 3.0 builds on many of the innovations from Spark 2.x, bringing new ideas as well as continuing long-term projects that have been in development. With the help of tremendous contributions from the open-source community, this release resolved more than 3400 tickets as the result of contributions from over 440 contributors. +Apache Spark 3.0.0 is the first release of the 3.x line. The vote passed on the 10th of June, 2020. This release is based on git tag `v3.0.0` which includes all commits up to June 10. Apache Spark 3.0 builds on many of the innovations from Spark 2.x, bringing new ideas as well as continuing long-term projects that have been in development. With the help of tremendous contributions from the open-source community, this release resolved more than 3400 tickets as the result of contributions from over 440 contributors. Review comment: This change is necessary as we need to mention the vote pass date somewhere. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] dongjoon-hyun commented on pull request #271: the date of the post should be the date you create it
dongjoon-hyun commented on pull request #271: URL: https://github.com/apache/spark-website/pull/271#issuecomment-646203798 If you think so, okay. @srowen . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] srowen commented on pull request #271: the date of the post should be the date you create it
srowen commented on pull request #271: URL: https://github.com/apache/spark-website/pull/271#issuecomment-646202686 Meh, I'm sure we have roughly the same review practices here. I'd also generally separate concerns. In practice, it's also OK to have a 'misc minor edits' PR. I know we want to make sure it's fixed up ASAP now that 3.0 is rolling out the door, announced. So probably no big deal to just make this one omnibus last minute tweak change. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] dongjoon-hyun edited a comment on pull request #271: the date of the post should be the date you create it
dongjoon-hyun edited a comment on pull request #271: URL: https://github.com/apache/spark-website/pull/271#issuecomment-646194278 Hi, @cloud-fan , @rxin , @gatorsmile . This PR should change news date only~ - 2.4.4 (https://github.com/apache/spark-website/pull/217/files#diff-2e7c596ac11621f7b55e3299dd80f1b3R25) - 2.4.5 (https://github.com/apache/spark-website/pull/261#discussion_r376745966) - 2.3.4 (https://github.com/apache/spark-website/pull/221/files#diff-2e7c596ac11621f7b55e3299dd80f1b3R26) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] dongjoon-hyun commented on pull request #271: the date of the post should the date you create it
dongjoon-hyun commented on pull request #271: URL: https://github.com/apache/spark-website/pull/271#issuecomment-646197761 Please see the 2.3.4/2.4.5 and old practices. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] dongjoon-hyun edited a comment on pull request #271: the date of the post should the date you create it
dongjoon-hyun edited a comment on pull request #271: URL: https://github.com/apache/spark-website/pull/271#issuecomment-646194278 Hi, @cloud-fan , @rxin , @gatorsmile . This PR should change news date only~ - 2.4.5 (https://github.com/apache/spark-website/pull/261#discussion_r376745966) - 2.3.4 (https://github.com/apache/spark-website/pull/221/files#diff-2e7c596ac11621f7b55e3299dd80f1b3R26) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] cloud-fan commented on pull request #271: the date of the post should the date you create it
cloud-fan commented on pull request #271: URL: https://github.com/apache/spark-website/pull/271#issuecomment-646197363 It's not a Spark PR, and I don't think the Spark PR policies apply here... We just do what's the most convenient. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] dongjoon-hyun commented on pull request #271: the date of the post should the date you create it
dongjoon-hyun commented on pull request #271: URL: https://github.com/apache/spark-website/pull/271#issuecomment-646194278 Hi, @cloud-fan , @rxin , @gatorsmile . This PR should change news date only~ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] dongjoon-hyun commented on pull request #270: Release Spark 3.0.0
dongjoon-hyun commented on pull request #270: URL: https://github.com/apache/spark-website/pull/270#issuecomment-646192994 Yes. News date should be today as I mentioned here (https://github.com/apache/spark-website/pull/270#discussion_r442350319) > The date of this part is different from news release date. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] gatorsmile commented on a change in pull request #271: the date of the post should the date you create it
gatorsmile commented on a change in pull request #271: URL: https://github.com/apache/spark-website/pull/271#discussion_r442380758 ## File path: releases/_posts/2020-06-18-spark-release-3-0-0.md ## @@ -315,18 +315,18 @@ Please read the [migration guide](https://spark.apache.org/docs/3.0.0/sparkr-mig ### Known Issues - - Streaming queries with `dropDuplicates` operator may not be able to restart with the checkpoint written by Spark 2.x. ([SPARK-31990](https://issues.apache.org/jira/browse/SPARK-31990)) - - In Web UI, the job list page may hang for more than 40 seconds. ([SPARK-31967](https://issues.apache.org/jira/browse/SPARK-31967)) + - Streaming queries with `dropDuplicates` operator may not be able to restart with the checkpoint written by Spark 2.x. This is fixed in the next release 3.0.1. ([SPARK-31990](https://issues.apache.org/jira/browse/SPARK-31990)) + - In Web UI, the job list page may hang for more than 40 seconds. This is fixed in the next release 3.0.1. ([SPARK-31967](https://issues.apache.org/jira/browse/SPARK-31967)) - Set `io.netty.tryReflectionSetAccessible` for Arrow on JDK9+ ([SPARK-29923](https://issues.apache.org/jira/browse/SPARK-29923)) - With AWS SDK upgrade to 1.11.655, we strongly encourage the users that use S3N file system (open-source NativeS3FileSystem that is based on jets3t library) on Hadoop 2.7.3 to upgrade to use AWS Signature V4 and set the bucket endpoint or migrate to S3A (“s3a://” prefix) - jets3t library uses AWS v2 by default and s3.amazonaws.com as an endpoint. Otherwise, the 403 Forbidden error may be thrown in the following cases: - If a user accesses an S3 path that contains “+” characters and uses the legacy S3N file system, e.g. s3n://bucket/path/+file. - If a user has configured AWS V2 signature to sign requests to S3 with S3N file system. Note that if you use S3AFileSystem, e.g. (“s3a://bucket/path”) to access S3 in S3Select or SQS connectors, then everything will work as expected. ([SPARK-30968](https://issues.apache.org/jira/browse/SPARK-30968)) - - Parsing day of year using pattern letter 'D' returns the wrong result if the year field is missing. This can happen in SQL functions like `to_timestamp` which parses datetime string to datetime values using a pattern string. ([SPARK-31939](https://issues.apache.org/jira/browse/SPARK-31939)) - - Join/Window/Aggregate inside subqueries may lead to wrong results if the keys have values -0.0 and 0.0. ([SPARK-31958](https://issues.apache.org/jira/browse/SPARK-31958)) - - A window query may fail with ambiguous self-join error unexpectedly. ([SPARK-31956](https://issues.apache.org/jira/browse/SPARK-31956)) + - Parsing day of year using pattern letter 'D' returns the wrong result if the year field is missing. This can happen in SQL functions like `to_timestamp` which parses datetime string to datetime values using a pattern string. This is fixed in the next release 3.0.1. ([SPARK-31939](https://issues.apache.org/jira/browse/SPARK-31939)) Review comment: => This will be fixed in the next release 3.0.1. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] srowen commented on pull request #271: the date of the post should the date you create it
srowen commented on pull request #271: URL: https://github.com/apache/spark-website/pull/271#issuecomment-646186207 I agree with that take. The VOTE passing is just an internal procedural step. The release date is when it's fully available to end users, which isn't the same thing. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] cloud-fan commented on a change in pull request #271: the date of the post should the date you create it
cloud-fan commented on a change in pull request #271: URL: https://github.com/apache/spark-website/pull/271#discussion_r442373302 ## File path: release-process.md ## @@ -259,7 +259,8 @@ Then, create the release notes. Go to the https://issues.apache.org/jira/projects/SPARK?selectedItem=com.atlassian.jira.jira-projects-plugin:release-page";>release page in JIRA, pick the release version from the list, then click on "Release Notes". Copy this URL and then make a short URL on https://s.apache.org/";>s.apache.org, sign in to your Apache account, and pick the ID as something like -`spark-2.1.2`. Create a new release post under `releases/_posts` to include this short URL. +`spark-2.1.2`. Create a new release post under `releases/_posts` to include this short URL. The date of the post should +be the date you create it. Review comment: I put it here so future releases can just follow it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] cloud-fan commented on pull request #271: the date of the post should the date you create it
cloud-fan commented on pull request #271: URL: https://github.com/apache/spark-website/pull/271#issuecomment-646175051 cc @dongjoon-hyun @rxin @srowen This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] cloud-fan opened a new pull request #271: the date of the post should the date you create it
cloud-fan opened a new pull request #271: URL: https://github.com/apache/spark-website/pull/271 After talking with @rxin offline, we think it's confusing to use the RC vote pass date as the date of the post. People may be wondering why they didn't see the post before, as the date is usually many days before. This PR tries to make it the policy clear: The date of the post should be the date you create it. This is the most intuitive IMO. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] gatorsmile commented on pull request #270: Release Spark 3.0.0
gatorsmile commented on pull request #270: URL: https://github.com/apache/spark-website/pull/270#issuecomment-646163523  Yes. Just checked the website, it shows a strange order. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] rxin commented on pull request #270: Release Spark 3.0.0
rxin commented on pull request #270: URL: https://github.com/apache/spark-website/pull/270#issuecomment-646162161 I find it quite strange that the news date is not today. I think we should use today’s date and then say the vote passed on June 6. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] rxin edited a comment on pull request #270: Release Spark 3.0.0
rxin edited a comment on pull request #270: URL: https://github.com/apache/spark-website/pull/270#issuecomment-646162161 I find it quite strange that the news date is not today. I think we should use today’s date and then say the vote passed on June 10 and this is based on tag v3.0.0 including all commits up to June 10. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] cloud-fan merged pull request #270: Release Spark 3.0.0
cloud-fan merged pull request #270: URL: https://github.com/apache/spark-website/pull/270 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
svn commit: r40088 - in /dev/spark: v3.0.0-rc1-bin/ v3.0.0-rc1-docs/ v3.0.0-rc2-bin/ v3.0.0-rc2-docs/ v3.0.0-rc3-docs/
Author: rxin Date: Thu Jun 18 16:41:27 2020 New Revision: 40088 Log: Removing RC artifacts. Removed: dev/spark/v3.0.0-rc1-bin/ dev/spark/v3.0.0-rc1-docs/ dev/spark/v3.0.0-rc2-bin/ dev/spark/v3.0.0-rc2-docs/ dev/spark/v3.0.0-rc3-docs/ - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] cloud-fan commented on a change in pull request #270: Release Spark 3.0.0
cloud-fan commented on a change in pull request #270:
URL: https://github.com/apache/spark-website/pull/270#discussion_r442358456
##
File path: js/downloads.js
##
@@ -23,7 +23,7 @@ var packagesV9 = [hadoop2p7, hadoop2p6, hadoopFree,
scala2p12_hadoopFree, source
// 3.0.0+
var packagesV10 = [hadoop2p7, hadoop3p2, hadoopFree, sources];
-addRelease("3.0.0-preview2", new Date("12/23/2019"), packagesV10, true);
+addRelease("3.0.0", new Date("06/18/2020"), packagesV10, true);
Review comment:
I recalled that we had a similar discussion at the time of 2.4.0 and we
picked the RC vote pass date.
I've updated to use June 10.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] srowen commented on a change in pull request #270: Release Spark 3.0.0
srowen commented on a change in pull request #270:
URL: https://github.com/apache/spark-website/pull/270#discussion_r442355349
##
File path: js/downloads.js
##
@@ -23,7 +23,7 @@ var packagesV9 = [hadoop2p7, hadoop2p6, hadoopFree,
scala2p12_hadoopFree, source
// 3.0.0+
var packagesV10 = [hadoop2p7, hadoop3p2, hadoopFree, sources];
-addRelease("3.0.0-preview2", new Date("12/23/2019"), packagesV10, true);
+addRelease("3.0.0", new Date("06/18/2020"), packagesV10, true);
Review comment:
I don't think it should be the vote date - it wasn't available to users
at that time. What you want to call the release date is a little subjective
(when artifacts were uploaded? announced?) but anything like June 18 seems
reasonable. It doesn't matter much.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] dongjoon-hyun commented on a change in pull request #270: Release Spark 3.0.0
dongjoon-hyun commented on a change in pull request #270:
URL: https://github.com/apache/spark-website/pull/270#discussion_r442350319
##
File path: js/downloads.js
##
@@ -23,7 +23,7 @@ var packagesV9 = [hadoop2p7, hadoop2p6, hadoopFree,
scala2p12_hadoopFree, source
// 3.0.0+
var packagesV10 = [hadoop2p7, hadoop3p2, hadoopFree, sources];
-addRelease("3.0.0-preview2", new Date("12/23/2019"), packagesV10, true);
+addRelease("3.0.0", new Date("06/18/2020"), packagesV10, true);
Review comment:
The vote passed on `06/10/2020`, @cloud-fan . The date of this part is
different from news release date. (cc @gatorsmile )
-
https://lists.apache.org/thread.html/r1f6109df463166b826b7b2d180f74885cd95ecfaa524d722b1e665d8%40%3Cdev.spark.apache.org%3E
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] dongjoon-hyun commented on a change in pull request #270: Release Spark 3.0.0
dongjoon-hyun commented on a change in pull request #270:
URL: https://github.com/apache/spark-website/pull/270#discussion_r442350319
##
File path: js/downloads.js
##
@@ -23,7 +23,7 @@ var packagesV9 = [hadoop2p7, hadoop2p6, hadoopFree,
scala2p12_hadoopFree, source
// 3.0.0+
var packagesV10 = [hadoop2p7, hadoop3p2, hadoopFree, sources];
-addRelease("3.0.0-preview2", new Date("12/23/2019"), packagesV10, true);
+addRelease("3.0.0", new Date("06/18/2020"), packagesV10, true);
Review comment:
The vote passed on `06/10/2020`, @cloud-fan . This date is different
from news release date. (cc @gatorsmile )
-
https://lists.apache.org/thread.html/r1f6109df463166b826b7b2d180f74885cd95ecfaa524d722b1e665d8%40%3Cdev.spark.apache.org%3E
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (e4f5036 -> 8a9ae01)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from e4f5036 [SPARK-32020][SQL] Better error message when SPARK_HOME or spark.test.home is not set add 8a9ae01 [MINOR] update dev/create-release/known_translations No new revisions were added by this update. Summary of changes: dev/create-release/known_translations | 148 ++ 1 file changed, 148 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (e4f5036 -> 8a9ae01)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from e4f5036 [SPARK-32020][SQL] Better error message when SPARK_HOME or spark.test.home is not set add 8a9ae01 [MINOR] update dev/create-release/known_translations No new revisions were added by this update. Summary of changes: dev/create-release/known_translations | 148 ++ 1 file changed, 148 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (e4f5036 -> 8a9ae01)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from e4f5036 [SPARK-32020][SQL] Better error message when SPARK_HOME or spark.test.home is not set add 8a9ae01 [MINOR] update dev/create-release/known_translations No new revisions were added by this update. Summary of changes: dev/create-release/known_translations | 148 ++ 1 file changed, 148 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] cloud-fan commented on pull request #270: Release Spark 3.0.0
cloud-fan commented on pull request #270: URL: https://github.com/apache/spark-website/pull/270#issuecomment-646067724 cc @rxin @gatorsmile @dongjoon-hyun @srowen This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] cloud-fan opened a new pull request #270: Release Spark 3.0.0
cloud-fan opened a new pull request #270: URL: https://github.com/apache/spark-website/pull/270 Add the 3.0.0 [release notes](https://docs.google.com/document/d/1NrTqxf2f39AXDF8VTIch6kwD8VKPaIlLW1QvuqEcwR4/edit) and update the website. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-32020][SQL] Better error message when SPARK_HOME or spark.test.home is not set
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 70f3ce6 [SPARK-32020][SQL] Better error message when SPARK_HOME or
spark.test.home is not set
70f3ce6 is described below
commit 70f3ce69ee6ae01393e853ccea16a259804b5d85
Author: Dilip Biswal
AuthorDate: Thu Jun 18 22:45:55 2020 +0900
[SPARK-32020][SQL] Better error message when SPARK_HOME or spark.test.home
is not set
### What changes were proposed in this pull request?
Better error message when SPARK_HOME or spark,test.home is not set.
### Why are the changes needed?
Currently the error message is not easily consumable as it prints (see
below) the real error after printing the current environment which is rather
long.
**Old output**
`
time.name" -> "Java(TM) SE Runtime Environment", "sun.boot.library.path"
-> "/Library/Java/JavaVirtualMachines/jdk1.8.0_221.jdk/Contents/Home/jre/lib",
"java.vm.version" -> "25.221-b11",
. . .
. . .
. . .
) did not contain key "SPARK_HOME" spark.test.home or SPARK_HOME is not set.
at org.scalatest.Assertions.newAssertionFailedExceptio
`
**New output**
An exception or error caused a run to abort: spark.test.home or SPARK_HOME
is not set.
org.scalatest.exceptions.TestFailedException: spark.test.home or SPARK_HOME
is not set
### Does this PR introduce any user-facing change?
`
No.
### How was this patch tested?
Ran the tests in intellej manually to see the new error.
Closes #28825 from dilipbiswal/minor-spark-31950-followup.
Authored-by: Dilip Biswal
Signed-off-by: HyukjinKwon
(cherry picked from commit e4f503614625f81aa86455ad5f6bec3b61d525f1)
Signed-off-by: HyukjinKwon
---
.../sql/catalyst/parser/TableIdentifierParserSuite.scala | 6 --
.../org/apache/spark/sql/catalyst/plans/SQLHelper.scala | 9 +
.../scala/org/apache/spark/sql/ExpressionsSchemaSuite.scala | 6 --
.../scala/org/apache/spark/sql/IntegratedUDFTestUtils.scala | 9 +
.../test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala | 12 +++-
5 files changed, 13 insertions(+), 29 deletions(-)
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/TableIdentifierParserSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/TableIdentifierParserSuite.scala
index 04c427d..a721e17 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/TableIdentifierParserSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/TableIdentifierParserSuite.scala
@@ -18,7 +18,6 @@ package org.apache.spark.sql.catalyst.parser
import java.io.File
import java.nio.file.Files
-import java.util.Locale
import scala.collection.JavaConverters._
import scala.collection.mutable
@@ -295,11 +294,6 @@ class TableIdentifierParserSuite extends SparkFunSuite
with SQLHelper {
private val sqlSyntaxDefs = {
val sqlBasePath = {
- val sparkHome = {
-assert(sys.props.contains("spark.test.home") ||
- sys.env.contains("SPARK_HOME"), "spark.test.home or SPARK_HOME is
not set.")
-sys.props.getOrElse("spark.test.home", sys.env("SPARK_HOME"))
- }
java.nio.file.Paths.get(sparkHome, "sql", "catalyst", "src", "main",
"antlr4", "org",
"apache", "spark", "sql", "catalyst", "parser", "SqlBase.g4").toFile
}
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/SQLHelper.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/SQLHelper.scala
index d213743..5deab79 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/SQLHelper.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/SQLHelper.scala
@@ -21,6 +21,8 @@ import java.time.ZoneId
import scala.util.control.NonFatal
+import org.scalatest.Assertions.fail
+
import org.apache.spark.sql.AnalysisException
import org.apache.spark.sql.catalyst.util.DateTimeTestUtils
import org.apache.spark.sql.catalyst.util.DateTimeUtils.getZoneId
@@ -83,4 +85,11 @@ trait SQLHelper {
}
}
}
+
+ protected lazy val sparkHome: String = {
+if (!(sys.props.contains("spark.test.home") ||
sys.env.contains("SPARK_HOME"))) {
+ fail("spark.test.home or SPARK_HOME is not set.")
+}
+sys.props.getOrElse("spark.test.home", sys.env("SPARK_HOME"))
+ }
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/ExpressionsSchemaSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/ExpressionsSchemaSuite.scala
index d69ecd7..81c09d1 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/ExpressionsSchemaSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/ExpressionsSc
[spark] branch branch-3.0 updated: [SPARK-32020][SQL] Better error message when SPARK_HOME or spark.test.home is not set
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 70f3ce6 [SPARK-32020][SQL] Better error message when SPARK_HOME or
spark.test.home is not set
70f3ce6 is described below
commit 70f3ce69ee6ae01393e853ccea16a259804b5d85
Author: Dilip Biswal
AuthorDate: Thu Jun 18 22:45:55 2020 +0900
[SPARK-32020][SQL] Better error message when SPARK_HOME or spark.test.home
is not set
### What changes were proposed in this pull request?
Better error message when SPARK_HOME or spark,test.home is not set.
### Why are the changes needed?
Currently the error message is not easily consumable as it prints (see
below) the real error after printing the current environment which is rather
long.
**Old output**
`
time.name" -> "Java(TM) SE Runtime Environment", "sun.boot.library.path"
-> "/Library/Java/JavaVirtualMachines/jdk1.8.0_221.jdk/Contents/Home/jre/lib",
"java.vm.version" -> "25.221-b11",
. . .
. . .
. . .
) did not contain key "SPARK_HOME" spark.test.home or SPARK_HOME is not set.
at org.scalatest.Assertions.newAssertionFailedExceptio
`
**New output**
An exception or error caused a run to abort: spark.test.home or SPARK_HOME
is not set.
org.scalatest.exceptions.TestFailedException: spark.test.home or SPARK_HOME
is not set
### Does this PR introduce any user-facing change?
`
No.
### How was this patch tested?
Ran the tests in intellej manually to see the new error.
Closes #28825 from dilipbiswal/minor-spark-31950-followup.
Authored-by: Dilip Biswal
Signed-off-by: HyukjinKwon
(cherry picked from commit e4f503614625f81aa86455ad5f6bec3b61d525f1)
Signed-off-by: HyukjinKwon
---
.../sql/catalyst/parser/TableIdentifierParserSuite.scala | 6 --
.../org/apache/spark/sql/catalyst/plans/SQLHelper.scala | 9 +
.../scala/org/apache/spark/sql/ExpressionsSchemaSuite.scala | 6 --
.../scala/org/apache/spark/sql/IntegratedUDFTestUtils.scala | 9 +
.../test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala | 12 +++-
5 files changed, 13 insertions(+), 29 deletions(-)
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/TableIdentifierParserSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/TableIdentifierParserSuite.scala
index 04c427d..a721e17 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/TableIdentifierParserSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/TableIdentifierParserSuite.scala
@@ -18,7 +18,6 @@ package org.apache.spark.sql.catalyst.parser
import java.io.File
import java.nio.file.Files
-import java.util.Locale
import scala.collection.JavaConverters._
import scala.collection.mutable
@@ -295,11 +294,6 @@ class TableIdentifierParserSuite extends SparkFunSuite
with SQLHelper {
private val sqlSyntaxDefs = {
val sqlBasePath = {
- val sparkHome = {
-assert(sys.props.contains("spark.test.home") ||
- sys.env.contains("SPARK_HOME"), "spark.test.home or SPARK_HOME is
not set.")
-sys.props.getOrElse("spark.test.home", sys.env("SPARK_HOME"))
- }
java.nio.file.Paths.get(sparkHome, "sql", "catalyst", "src", "main",
"antlr4", "org",
"apache", "spark", "sql", "catalyst", "parser", "SqlBase.g4").toFile
}
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/SQLHelper.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/SQLHelper.scala
index d213743..5deab79 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/SQLHelper.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/SQLHelper.scala
@@ -21,6 +21,8 @@ import java.time.ZoneId
import scala.util.control.NonFatal
+import org.scalatest.Assertions.fail
+
import org.apache.spark.sql.AnalysisException
import org.apache.spark.sql.catalyst.util.DateTimeTestUtils
import org.apache.spark.sql.catalyst.util.DateTimeUtils.getZoneId
@@ -83,4 +85,11 @@ trait SQLHelper {
}
}
}
+
+ protected lazy val sparkHome: String = {
+if (!(sys.props.contains("spark.test.home") ||
sys.env.contains("SPARK_HOME"))) {
+ fail("spark.test.home or SPARK_HOME is not set.")
+}
+sys.props.getOrElse("spark.test.home", sys.env("SPARK_HOME"))
+ }
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/ExpressionsSchemaSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/ExpressionsSchemaSuite.scala
index d69ecd7..81c09d1 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/ExpressionsSchemaSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/ExpressionsSc
[spark] branch master updated (9b79251 -> e4f5036)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 9b79251 [SPARK-31960][YARN][BUILD] Only populate Hadoop classpath for no-hadoop build add e4f5036 [SPARK-32020][SQL] Better error message when SPARK_HOME or spark.test.home is not set No new revisions were added by this update. Summary of changes: .../sql/catalyst/parser/TableIdentifierParserSuite.scala | 6 -- .../org/apache/spark/sql/catalyst/plans/SQLHelper.scala | 9 + .../scala/org/apache/spark/sql/ExpressionsSchemaSuite.scala | 6 -- .../scala/org/apache/spark/sql/IntegratedUDFTestUtils.scala | 9 + .../test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala | 12 +++- 5 files changed, 13 insertions(+), 29 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (9b79251 -> e4f5036)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 9b79251 [SPARK-31960][YARN][BUILD] Only populate Hadoop classpath for no-hadoop build add e4f5036 [SPARK-32020][SQL] Better error message when SPARK_HOME or spark.test.home is not set No new revisions were added by this update. Summary of changes: .../sql/catalyst/parser/TableIdentifierParserSuite.scala | 6 -- .../org/apache/spark/sql/catalyst/plans/SQLHelper.scala | 9 + .../scala/org/apache/spark/sql/ExpressionsSchemaSuite.scala | 6 -- .../scala/org/apache/spark/sql/IntegratedUDFTestUtils.scala | 9 + .../test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala | 12 +++- 5 files changed, 13 insertions(+), 29 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
