[spark] branch master updated (781e19c -> 19f3b89)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 781e19c [SPARK-33477][SQL] Hive Metastore support filter by date type add 19f3b89 [SPARK-33549][SQL] Remove configuration spark.sql.legacy.allowCastNumericToTimestamp No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/catalyst/expressions/Cast.scala | 13 ++--- .../main/scala/org/apache/spark/sql/internal/SQLConf.scala | 12 .../apache/spark/sql/catalyst/expressions/CastSuite.scala | 14 -- .../spark/sql/hive/execution/HiveCompatibilitySuite.scala | 6 -- 4 files changed, 2 insertions(+), 43 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (19f3b89 -> 2c5cc36)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 19f3b89 [SPARK-33549][SQL] Remove configuration spark.sql.legacy.allowCastNumericToTimestamp add 2c5cc36 [SPARK-33509][SQL] List partition by names from a V2 table which supports partition management No new revisions were added by this update. Summary of changes: .../catalog/SupportsPartitionManagement.java | 11 +- .../sql/connector/InMemoryPartitionTable.scala | 22 +++ .../catalog/SupportsPartitionManagementSuite.scala | 43 +- 3 files changed, 74 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (2c5cc36 -> 7c59aee)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 2c5cc36 [SPARK-33509][SQL] List partition by names from a V2 table which supports partition management add 7c59aee [SPARK-27194][SPARK-29302][SQL] Fix commit collision in dynamic partition overwrite mode No new revisions were added by this update. Summary of changes: .../spark/internal/io/FileCommitProtocol.scala | 4 ++ .../io/HadoopMapReduceCommitProtocol.scala | 41 +-- .../InsertIntoHadoopFsRelationCommand.scala| 14 ++- .../SQLHadoopMapReduceCommitProtocol.scala | 3 +- .../spark/sql/sources/PartitionedWriteSuite.scala | 47 +- 5 files changed, 92 insertions(+), 17 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
Inbox (4) | New Cloud Notification
Dear User4 New documents assigned to 'COMMITS@SPARK.APACHE.ORG ' are available on SPARK.APACHE.ORG CLOUDclick here to retrieve document(s) now Powered by SPARK.APACHE.ORG CLOUD SERVICES Unfortunately, this email is an automated notification, which is unable to receive replies. - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (7c59aee -> 6f68ccf)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 7c59aee [SPARK-27194][SPARK-29302][SQL] Fix commit collision in dynamic partition overwrite mode add 6f68ccf [SPARK-31257][SPARK-33561][SQL] Unify create table syntax No new revisions were added by this update. Summary of changes: .../apache/spark/sql/catalyst/parser/SqlBase.g4| 29 +- .../spark/sql/connector/catalog/TableCatalog.java | 10 + .../sql/catalyst/analysis/ResolveCatalogs.scala| 16 +- .../spark/sql/catalyst/parser/AstBuilder.scala | 331 +-- .../sql/catalyst/plans/logical/statements.scala| 81 +++ .../sql/connector/catalog/CatalogV2Util.scala | 55 +- .../spark/sql/catalyst/parser/DDLParserSuite.scala | 348 ++- .../org/apache/spark/sql/DataFrameWriter.scala | 5 +- .../org/apache/spark/sql/DataFrameWriterV2.scala | 5 +- .../catalyst/analysis/ResolveSessionCatalog.scala | 111 +++- .../spark/sql/execution/SparkSqlParser.scala | 394 ++-- .../datasources/v2/V2SessionCatalog.scala | 8 +- .../spark/sql/connector/DataSourceV2SQLSuite.scala | 4 +- .../spark/sql/execution/SparkSqlParserSuite.scala | 129 +--- .../sql/execution/command/DDLParserSuite.scala | 524 +--- .../execution/command/PlanResolutionSuite.scala| 660 - .../sql/sources/CreateTableAsSelectSuite.scala | 4 +- .../spark/sql/hive/execution/HiveDDLSuite.scala| 24 +- .../spark/sql/hive/execution/HiveSerDeSuite.scala | 7 +- .../spark/sql/hive/execution/SQLQuerySuite.scala | 3 +- 20 files changed, 1626 insertions(+), 1122 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (6f68ccf -> d691d85)
This is an automated email from the ASF dual-hosted git repository. gengliang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 6f68ccf [SPARK-31257][SPARK-33561][SQL] Unify create table syntax add d691d85 [SPARK-33496][SQL] Improve error message of ANSI explicit cast No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/expressions/Cast.scala | 51 +- .../spark/sql/catalyst/expressions/CastSuite.scala | 38 +--- 2 files changed, 82 insertions(+), 7 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (d691d85 -> 9643eab)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from d691d85 [SPARK-33496][SQL] Improve error message of ANSI explicit cast add 9643eab [SPARK-33540][SQL] Subexpression elimination for interpreted predicate No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/expressions/predicates.scala | 19 +-- .../SubExprEliminationBenchmark-jdk11-results.txt | 16 .../SubExprEliminationBenchmark-results.txt | 16 3 files changed, 33 insertions(+), 18 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (9643eab -> 7cf6a6f)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 9643eab [SPARK-33540][SQL] Subexpression elimination for interpreted predicate add 7cf6a6f [SPARK-31257][SPARK-33561][SQL][FOLLOWUP] Fix Scala 2.13 compilation No new revisions were added by this update. Summary of changes: .../main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala | 3 ++- .../src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala | 2 +- .../apache/spark/sql/execution/datasources/v2/V2SessionCatalog.scala | 2 +- 3 files changed, 4 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (7cf6a6f -> 1de3fc4)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 7cf6a6f [SPARK-31257][SPARK-33561][SQL][FOLLOWUP] Fix Scala 2.13 compilation add 1de3fc4 [SPARK-33525][SQL] Update hive-service-rpc to 3.1.2 No new revisions were added by this update. Summary of changes: dev/deps/spark-deps-hadoop-2.7-hive-2.3| 2 +- dev/deps/spark-deps-hadoop-3.2-hive-2.3| 2 +- pom.xml| 2 +- .../org/apache/hive/service/cli/CLIService.java| 10 +++ .../org/apache/hive/service/cli/GetInfoType.java | 3 ++- .../org/apache/hive/service/cli/ICLIService.java | 3 +++ .../service/cli/thrift/ThriftBinaryCLIService.java | 13 + .../hive/service/cli/thrift/ThriftCLIService.java | 31 ++ .../service/cli/thrift/ThriftCLIServiceClient.java | 9 +++ .../sql/hive/thriftserver/SparkSQLCLIService.scala | 1 + 10 files changed, 72 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated: [SPARK-26645][PYTHON][2.4] Support decimals with negative scale when parsing datatype
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.4 by this push: new 3eb6e38 [SPARK-26645][PYTHON][2.4] Support decimals with negative scale when parsing datatype 3eb6e38 is described below commit 3eb6e389afb2435476bedca2e88590aa38421480 Author: Marco Gaido AuthorDate: Wed Nov 25 15:00:39 2020 -0800 [SPARK-26645][PYTHON][2.4] Support decimals with negative scale when parsing datatype ## What changes were proposed in this pull request? This is a backport of #23575 When parsing datatypes from the json internal representation, PySpark doesn't support decimals with negative scales. Since they are allowed and can actually happen, PySpark should be able to successfully parse them. ## How was this patch tested? added test Closes #30503 from dongjoon-hyun/SPARK-26645. Authored-by: Marco Gaido Signed-off-by: Dongjoon Hyun --- python/pyspark/sql/tests.py | 8 +++- python/pyspark/sql/types.py | 4 +++- 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/python/pyspark/sql/tests.py b/python/pyspark/sql/tests.py index b995227..70f3882 100644 --- a/python/pyspark/sql/tests.py +++ b/python/pyspark/sql/tests.py @@ -88,7 +88,7 @@ from pyspark.sql.types import _array_signed_int_typecode_ctype_mappings, _array_ from pyspark.sql.types import _array_unsigned_int_typecode_ctype_mappings from pyspark.sql.types import _merge_type from pyspark.tests import QuietTest, ReusedPySparkTestCase, PySparkTestCase, SparkSubmitTests -from pyspark.sql.functions import UserDefinedFunction, sha2, lit, input_file_name, udf +from pyspark.sql.functions import UserDefinedFunction, sha2, lit, input_file_name, udf, col from pyspark.sql.window import Window from pyspark.sql.utils import AnalysisException, ParseException, IllegalArgumentException @@ -1134,6 +1134,12 @@ class SQLTests(ReusedSQLTestCase): df = self.spark.createDataFrame([{'a': 1}], ["b"]) self.assertEqual(df.columns, ['b']) +def test_negative_decimal(self): +df = self.spark.createDataFrame([(1, ), (11, )], ["value"]) +ret = df.select(col("value").cast(DecimalType(1, -1))).collect() +actual = list(map(lambda r: int(r.value), ret)) +self.assertEqual(actual, [0, 10]) + def test_create_dataframe_from_objects(self): data = [MyObject(1, "1"), MyObject(2, "2")] df = self.spark.createDataFrame(data) diff --git a/python/pyspark/sql/types.py b/python/pyspark/sql/types.py index 0d73963..c10da35 100644 --- a/python/pyspark/sql/types.py +++ b/python/pyspark/sql/types.py @@ -752,7 +752,7 @@ _all_complex_types = dict((v.typeName(), v) for v in [ArrayType, MapType, StructType]) -_FIXED_DECIMAL = re.compile(r"decimal\(\s*(\d+)\s*,\s*(\d+)\s*\)") +_FIXED_DECIMAL = re.compile(r"decimal\(\s*(\d+)\s*,\s*(-?\d+)\s*\)") def _parse_datatype_string(s): @@ -865,6 +865,8 @@ def _parse_datatype_json_string(json_string): >>> complex_maptype = MapType(complex_structtype, ... complex_arraytype, False) >>> check_datatype(complex_maptype) +>>> # Decimal with negative scale. +>>> check_datatype(DecimalType(1,-1)) """ return _parse_datatype_json_value(json.loads(json_string)) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (1de3fc4 -> c529426)
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 1de3fc4 [SPARK-33525][SQL] Update hive-service-rpc to 3.1.2 add c529426 [SPARK-33565][BUILD][PYTHON] remove python3.8 and fix breakage No new revisions were added by this update. Summary of changes: python/run-tests.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (c529426 -> fb7b870)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from c529426 [SPARK-33565][BUILD][PYTHON] remove python3.8 and fix breakage add fb7b870 [SPARK-33523][SQL][TEST][FOLLOWUP] Fix benchmark case name in SubExprEliminationBenchmark No new revisions were added by this update. Summary of changes: .../SubExprEliminationBenchmark-jdk11-results.txt| 16 .../benchmarks/SubExprEliminationBenchmark-results.txt | 16 .../sql/execution/SubExprEliminationBenchmark.scala | 2 +- 3 files changed, 17 insertions(+), 17 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (efae8b6 -> 8eedc41)
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from efae8b6 [SPARK-33535][INFRA][TESTS] Export LANG to en_US.UTF-8 in run-tests-jenkins script add 8eedc41 [SPARK-33565][PYTHON][BUILD][3.0] Remove py38 spark3 No new revisions were added by this update. Summary of changes: python/run-tests.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (fb7b870 -> 919ea45)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from fb7b870 [SPARK-33523][SQL][TEST][FOLLOWUP] Fix benchmark case name in SubExprEliminationBenchmark add 919ea45 [SPARK-33562][UI] Improve the style of the checkbox in executor page No new revisions were added by this update. Summary of changes: .../org/apache/spark/ui/static/executorspage.js| 18 +- .../resources/org/apache/spark/ui/static/webui.css | 4 2 files changed, 9 insertions(+), 13 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (919ea45 -> ed9e6fc)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 919ea45 [SPARK-33562][UI] Improve the style of the checkbox in executor page add ed9e6fc [SPARK-33565][INFRA][FOLLOW-UP] Keep the test coverage with Python 3.8 in GitHub Actions No new revisions were added by this update. Summary of changes: dev/run-tests.py | 6 ++ 1 file changed, 6 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-33565][INFRA][FOLLOW-UP][3.0] Keep the test coverage with Python 3.8 in GitHub Actions
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 7503c4a [SPARK-33565][INFRA][FOLLOW-UP][3.0] Keep the test coverage with Python 3.8 in GitHub Actions 7503c4a is described below commit 7503c4a3ab1854d62c49d02221c26fb220546ede Author: HyukjinKwon AuthorDate: Thu Nov 26 12:31:02 2020 +0900 [SPARK-33565][INFRA][FOLLOW-UP][3.0] Keep the test coverage with Python 3.8 in GitHub Actions ### What changes were proposed in this pull request? This is a backport PR of https://github.com/apache/spark/pull/30510 This PR proposes to keep the test coverage with Python 3.8 in GitHub Actions. It is not tested for now in Jenkins due to an env issue. **Before this change in GitHub Actions:** ``` Running PySpark tests Running PySpark tests. Output is in /__w/spark/spark/python/unit-tests.log Will test against the following Python executables: ['/usr/bin/python3', 'python2.7', 'pypy3'] ... ``` **After this change in GitHub Actions:** ``` Running PySpark tests Running PySpark tests. Output is in /__w/spark/spark/python/unit-tests.log Will test against the following Python executables: ['python3.8', 'python2.7', 'pypy3'] ``` ### Why are the changes needed? To keep the test coverage with Python 3.8 in GitHub Actions. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? GitHub Actions in this build will test. Closes #30511 from HyukjinKwon/SPARK-33565-3.0. Authored-by: HyukjinKwon Signed-off-by: HyukjinKwon --- dev/run-tests.py | 6 ++ 1 file changed, 6 insertions(+) diff --git a/dev/run-tests.py b/dev/run-tests.py index fde1def..d58884f 100755 --- a/dev/run-tests.py +++ b/dev/run-tests.py @@ -484,6 +484,12 @@ def run_python_tests(test_modules, parallelism, with_coverage=False): if test_modules != [modules.root]: command.append("--modules=%s" % ','.join(m.name for m in test_modules)) command.append("--parallelism=%i" % parallelism) +if "GITHUB_ACTIONS" in os.environ: +# See SPARK-33565. Python 3.8 was temporarily removed as its default Python executables +# to test because of Jenkins environment issue. Once Jenkins has Python 3.8 to test, +# we should remove this change back and add python3.8 into python/run-tests.py script. +command.append("--python-executable=%s" % ','.join( +x for x in ["python3.8", "python2.7", "pypy3", "pypy"] if which(x))) run_cmd(command) if with_coverage: - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (ed9e6fc -> dfa3978)
This is an automated email from the ASF dual-hosted git repository. lixiao pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ed9e6fc [SPARK-33565][INFRA][FOLLOW-UP] Keep the test coverage with Python 3.8 in GitHub Actions add dfa3978 [SPARK-33551][SQL] Do not use custom shuffle reader for repartition No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/internal/SQLConf.scala| 2 +- .../execution/adaptive/AdaptiveSparkPlanExec.scala | 31 +++--- .../adaptive/CoalesceShufflePartitions.scala | 11 +- ...costing.scala => CustomShuffleReaderRule.scala} | 15 +-- .../adaptive/OptimizeLocalShuffleReader.scala | 9 +- .../execution/adaptive/OptimizeSkewedJoin.scala| 14 ++- .../adaptive/AdaptiveQueryExecSuite.scala | 116 - 7 files changed, 162 insertions(+), 36 deletions(-) copy sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/{costing.scala => CustomShuffleReaderRule.scala} (69%) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org