(spark) branch master updated (081809667611 -> c4e4497ff7e7)
This is an automated email from the ASF dual-hosted git repository. weichenxu123 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 081809667611 [MINOR][SQL] Remove `unsupportedOperationMsg` from `CaseInsensitiveStringMap` add c4e4497ff7e7 [SPARK-46812][SQL][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile No new revisions were added by this update. Summary of changes: .../sql/connect/planner/SparkConnectPlanner.scala | 6 +- dev/sparktestsupport/modules.py| 1 + python/pyspark/sql/pandas/map_ops.py | 61 +++- python/pyspark/sql/tests/test_resources.py | 104 + .../catalyst/analysis/DeduplicateRelations.scala | 4 +- .../plans/logical/pythonLogicalOperators.scala | 7 +- .../sql/catalyst/analysis/AnalysisSuite.scala | 9 +- .../main/scala/org/apache/spark/sql/Dataset.scala | 17 +++- .../spark/sql/execution/SparkStrategies.scala | 8 +- .../sql/execution/python/MapInArrowExec.scala | 4 +- .../sql/execution/python/MapInBatchExec.scala | 6 +- .../sql/execution/python/MapInPandasExec.scala | 4 +- 12 files changed, 206 insertions(+), 25 deletions(-) create mode 100644 python/pyspark/sql/tests/test_resources.py - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
(spark) branch branch-3.5 updated: [SPARK-47072][SQL][3.5] Fix supported interval formats in error messages
This is an automated email from the ASF dual-hosted git repository.
maxgekk pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push:
new 93a09ea279e6 [SPARK-47072][SQL][3.5] Fix supported interval formats in
error messages
93a09ea279e6 is described below
commit 93a09ea279e6bd2515ced66d8f38053e4a5514ce
Author: Max Gekk
AuthorDate: Mon Feb 19 10:29:08 2024 +0300
[SPARK-47072][SQL][3.5] Fix supported interval formats in error messages
### What changes were proposed in this pull request?
In the PR, I propose to add one more field to keys of `supportedFormat` in
`IntervalUtils` because current implementation has duplicate keys that
overwrites each other. For instance, the following keys are the same:
```
(YM.YEAR, YM.MONTH)
...
(DT.DAY, DT.HOUR)
```
because `YM.YEAR = DT.DAY = 0` and `YM.MONTH = DT.HOUR = 1`.
This is a backport of https://github.com/apache/spark/pull/45127.
### Why are the changes needed?
To fix the incorrect error message when Spark cannot parse ANSI interval
string. For example, the expected format should be some year-month format but
Spark outputs day-time one:
```sql
spark-sql (default)> select interval '-\t2-2\t' year to month;
Interval string does not match year-month format of `[+|-]d h`, `INTERVAL
[+|-]'[+|-]d h' DAY TO HOUR` when cast to interval year to month: - 2-2 .
(line 1, pos 16)
== SQL ==
select interval '-\t2-2\t' year to month
^^^
```
### Does this PR introduce _any_ user-facing change?
Yes.
### How was this patch tested?
By running the existing test suite:
```
$ build/sbt "test:testOnly *IntervalUtilsSuite"
```
and regenerating the golden files:
```
$ SPARK_GENERATE_GOLDEN_FILES=1 PYSPARK_PYTHON=python3 build/sbt
"sql/testOnly org.apache.spark.sql.SQLQueryTestSuite"
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Authored-by: Max Gekk
(cherry picked from commit 074fcf2807000d342831379de0fafc1e49a6bf19)
Closes #45139 from MaxGekk/fix-supportedFormat-3.5.
Authored-by: Max Gekk
Signed-off-by: Max Gekk
---
.../spark/sql/catalyst/util/IntervalUtils.scala| 33 +-
.../sql/catalyst/expressions/CastSuiteBase.scala | 8 +++---
.../analyzer-results/ansi/interval.sql.out | 2 +-
.../sql-tests/analyzer-results/interval.sql.out| 2 +-
.../sql-tests/results/ansi/interval.sql.out| 2 +-
.../resources/sql-tests/results/interval.sql.out | 2 +-
6 files changed, 27 insertions(+), 22 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
index e051cfc37f12..4d90007400ea 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
@@ -107,25 +107,30 @@ object IntervalUtils extends SparkIntervalUtils {
fallBackNotice: Option[String] = None) = {
throw new IllegalArgumentException(
s"Interval string does not match $intervalStr format of " +
-s"${supportedFormat((startFiled, endField)).map(format =>
s"`$format`").mkString(", ")} " +
+s"${supportedFormat((intervalStr, startFiled, endField))
+ .map(format => s"`$format`").mkString(", ")} " +
s"when cast to $typeName: ${input.toString}" +
s"${fallBackNotice.map(s => s", $s").getOrElse("")}")
}
val supportedFormat = Map(
-(YM.YEAR, YM.MONTH) -> Seq("[+|-]y-m", "INTERVAL [+|-]'[+|-]y-m' YEAR TO
MONTH"),
-(YM.YEAR, YM.YEAR) -> Seq("[+|-]y", "INTERVAL [+|-]'[+|-]y' YEAR"),
-(YM.MONTH, YM.MONTH) -> Seq("[+|-]m", "INTERVAL [+|-]'[+|-]m' MONTH"),
-(DT.DAY, DT.DAY) -> Seq("[+|-]d", "INTERVAL [+|-]'[+|-]d' DAY"),
-(DT.DAY, DT.HOUR) -> Seq("[+|-]d h", "INTERVAL [+|-]'[+|-]d h' DAY TO
HOUR"),
-(DT.DAY, DT.MINUTE) -> Seq("[+|-]d h:m", "INTERVAL [+|-]'[+|-]d h:m' DAY
TO MINUTE"),
-(DT.DAY, DT.SECOND) -> Seq("[+|-]d h:m:s.n", "INTERVAL [+|-]'[+|-]d
h:m:s.n' DAY TO SECOND"),
-(DT.HOUR, DT.HOUR) -> Seq("[+|-]h", "INTERVAL [+|-]'[+|-]h' HOUR"),
-(DT.HOUR, DT.MINUTE) -> Seq("[+|-]h:m", "INTERVAL [+|-]'[+|-]h:m' HOUR TO
MINUTE"),
-(DT.HOUR, DT.SECOND) -> Seq("[+|-]h:m:s.n", "INTERVAL [+|-]'[+|-]h:m:s.n'
HOUR TO SECOND"),
-(DT.MINUTE, DT.MINUTE) -> Seq("[+|-]m", "INTERVAL [+|-]'[+|-]m' MINUTE"),
-(DT.MINUTE, DT.SECOND) -> Seq("[+|-]m:s.n", "INTERVAL [+|-]'[+|-]m:s.n'
MINUTE TO SECOND"),
-(DT.SECOND, DT.SECOND) -> Seq("[+|-]s.n", "INTERVAL [+|-]'[+|-]s.n'
SECOND")
+("year-month", YM.YEAR, YM.MONTH) -> Seq("[+|-]y-m", "INTERVAL
(spark) branch branch-3.4 updated: [SPARK-47072][SQL][3.4] Fix supported interval formats in error messages
This is an automated email from the ASF dual-hosted git repository.
maxgekk pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new 081c7a7947a4 [SPARK-47072][SQL][3.4] Fix supported interval formats in
error messages
081c7a7947a4 is described below
commit 081c7a7947a47bf0b2bfd478abdd4b78a1db3ddb
Author: Max Gekk
AuthorDate: Mon Feb 19 10:26:47 2024 +0300
[SPARK-47072][SQL][3.4] Fix supported interval formats in error messages
### What changes were proposed in this pull request?
In the PR, I propose to add one more field to keys of `supportedFormat` in
`IntervalUtils` because current implementation has duplicate keys that
overwrites each other. For instance, the following keys are the same:
```
(YM.YEAR, YM.MONTH)
...
(DT.DAY, DT.HOUR)
```
because `YM.YEAR = DT.DAY = 0` and `YM.MONTH = DT.HOUR = 1`.
This is a backport of https://github.com/apache/spark/pull/45127.
### Why are the changes needed?
To fix the incorrect error message when Spark cannot parse ANSI interval
string. For example, the expected format should be some year-month format but
Spark outputs day-time one:
```sql
spark-sql (default)> select interval '-\t2-2\t' year to month;
Interval string does not match year-month format of `[+|-]d h`, `INTERVAL
[+|-]'[+|-]d h' DAY TO HOUR` when cast to interval year to month: - 2-2 .
(line 1, pos 16)
== SQL ==
select interval '-\t2-2\t' year to month
^^^
```
### Does this PR introduce _any_ user-facing change?
Yes.
### How was this patch tested?
By running the existing test suite:
```
$ build/sbt "test:testOnly *IntervalUtilsSuite"
```
and regenerating the golden files:
```
$ SPARK_GENERATE_GOLDEN_FILES=1 PYSPARK_PYTHON=python3 build/sbt
"sql/testOnly org.apache.spark.sql.SQLQueryTestSuite"
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Authored-by: Max Gekk
(cherry picked from commit 074fcf2807000d342831379de0fafc1e49a6bf19)
Closes #45140 from MaxGekk/fix-supportedFormat-3.4.
Authored-by: Max Gekk
Signed-off-by: Max Gekk
---
.../spark/sql/catalyst/util/IntervalUtils.scala| 33 +-
.../sql/catalyst/expressions/CastSuiteBase.scala | 8 +++---
.../sql-tests/results/ansi/interval.sql.out| 2 +-
.../resources/sql-tests/results/interval.sql.out | 2 +-
4 files changed, 25 insertions(+), 20 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
index 7b574e987d91..93b8f2f512ea 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
@@ -124,25 +124,30 @@ object IntervalUtils {
fallBackNotice: Option[String] = None) = {
throw new IllegalArgumentException(
s"Interval string does not match $intervalStr format of " +
-s"${supportedFormat((startFiled, endField)).map(format =>
s"`$format`").mkString(", ")} " +
+s"${supportedFormat((intervalStr, startFiled, endField))
+ .map(format => s"`$format`").mkString(", ")} " +
s"when cast to $typeName: ${input.toString}" +
s"${fallBackNotice.map(s => s", $s").getOrElse("")}")
}
val supportedFormat = Map(
-(YM.YEAR, YM.MONTH) -> Seq("[+|-]y-m", "INTERVAL [+|-]'[+|-]y-m' YEAR TO
MONTH"),
-(YM.YEAR, YM.YEAR) -> Seq("[+|-]y", "INTERVAL [+|-]'[+|-]y' YEAR"),
-(YM.MONTH, YM.MONTH) -> Seq("[+|-]m", "INTERVAL [+|-]'[+|-]m' MONTH"),
-(DT.DAY, DT.DAY) -> Seq("[+|-]d", "INTERVAL [+|-]'[+|-]d' DAY"),
-(DT.DAY, DT.HOUR) -> Seq("[+|-]d h", "INTERVAL [+|-]'[+|-]d h' DAY TO
HOUR"),
-(DT.DAY, DT.MINUTE) -> Seq("[+|-]d h:m", "INTERVAL [+|-]'[+|-]d h:m' DAY
TO MINUTE"),
-(DT.DAY, DT.SECOND) -> Seq("[+|-]d h:m:s.n", "INTERVAL [+|-]'[+|-]d
h:m:s.n' DAY TO SECOND"),
-(DT.HOUR, DT.HOUR) -> Seq("[+|-]h", "INTERVAL [+|-]'[+|-]h' HOUR"),
-(DT.HOUR, DT.MINUTE) -> Seq("[+|-]h:m", "INTERVAL [+|-]'[+|-]h:m' HOUR TO
MINUTE"),
-(DT.HOUR, DT.SECOND) -> Seq("[+|-]h:m:s.n", "INTERVAL [+|-]'[+|-]h:m:s.n'
HOUR TO SECOND"),
-(DT.MINUTE, DT.MINUTE) -> Seq("[+|-]m", "INTERVAL [+|-]'[+|-]m' MINUTE"),
-(DT.MINUTE, DT.SECOND) -> Seq("[+|-]m:s.n", "INTERVAL [+|-]'[+|-]m:s.n'
MINUTE TO SECOND"),
-(DT.SECOND, DT.SECOND) -> Seq("[+|-]s.n", "INTERVAL [+|-]'[+|-]s.n'
SECOND")
+("year-month", YM.YEAR, YM.MONTH) -> Seq("[+|-]y-m", "INTERVAL
[+|-]'[+|-]y-m' YEAR TO MONTH"),
+("year-month", YM.YEAR, YM.YEAR) -> Seq("[+|-]y", "INTERVAL [+|-]'[+|-]y'
YEAR"),
+("year-month", YM.MONTH,
(spark) branch master updated: [MINOR][SQL] Remove `unsupportedOperationMsg` from `CaseInsensitiveStringMap`
This is an automated email from the ASF dual-hosted git repository.
maxgekk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 081809667611 [MINOR][SQL] Remove `unsupportedOperationMsg` from
`CaseInsensitiveStringMap`
081809667611 is described below
commit 081809667611ceca926e9ef66f0bc3a786039314
Author: yangjie01
AuthorDate: Mon Feb 19 10:25:01 2024 +0300
[MINOR][SQL] Remove `unsupportedOperationMsg` from
`CaseInsensitiveStringMap`
### What changes were proposed in this pull request?
This pr remove unused private field `unsupportedOperationMsg` from
`CaseInsensitiveStringMap`, it is replaced by `_LEGACY_ERROR_TEMP_3206` after
https://github.com/apache/spark/pull/44549 is merged,
### Why are the changes needed?
Remove unused private field.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Pass GitHub Actions
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #45161 from LuciferYang/minior-CaseInsensitiveStringMap.
Authored-by: yangjie01
Signed-off-by: Max Gekk
---
.../main/java/org/apache/spark/sql/util/CaseInsensitiveStringMap.java | 2 --
1 file changed, 2 deletions(-)
diff --git
a/sql/catalyst/src/main/java/org/apache/spark/sql/util/CaseInsensitiveStringMap.java
b/sql/catalyst/src/main/java/org/apache/spark/sql/util/CaseInsensitiveStringMap.java
index 1c5c38ba705f..00a3de692fbf 100644
---
a/sql/catalyst/src/main/java/org/apache/spark/sql/util/CaseInsensitiveStringMap.java
+++
b/sql/catalyst/src/main/java/org/apache/spark/sql/util/CaseInsensitiveStringMap.java
@@ -45,8 +45,6 @@ import java.util.Set;
public class CaseInsensitiveStringMap implements Map {
private static final Logger logger =
LoggerFactory.getLogger(CaseInsensitiveStringMap.class);
- private String unsupportedOperationMsg = "CaseInsensitiveStringMap is
read-only.";
-
public static CaseInsensitiveStringMap empty() {
return new CaseInsensitiveStringMap(new HashMap<>(0));
}
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
(spark) branch master updated (174a19c1c039 -> 91dfc31ff89c)
This is an automated email from the ASF dual-hosted git repository. yao pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 174a19c1c039 [MINOR][INFRA] Rename build_maven.yml and build_maven_java21.yml add 91dfc31ff89c [SPARK-47085][SQL] reduce the complexity of toTRowSet from n^2 to n No new revisions were added by this update. Summary of changes: .../spark/sql/hive/thriftserver/RowSetUtils.scala | 23 +++--- .../SparkExecuteStatementOperation.scala | 2 +- 2 files changed, 8 insertions(+), 17 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
(spark) branch master updated: [MINOR][INFRA] Rename build_maven.yml and build_maven_java21.yml
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 174a19c1c039 [MINOR][INFRA] Rename build_maven.yml and build_maven_java21.yml 174a19c1c039 is described below commit 174a19c1c0391b1084e6a0b8fdbc5c019a662b01 Author: Hyukjin Kwon AuthorDate: Mon Feb 19 12:45:32 2024 +0900 [MINOR][INFRA] Rename build_maven.yml and build_maven_java21.yml ### What changes were proposed in this pull request? This PR proposes to change the name of `build_maven.yml` and `build_maven_java21.yml` from - Build using Maven (master, Scala 2.13, Hadoop 3, JDK 17) -> Build \ Maven (master, Scala 2.13, Hadoop 3, JDK 17) - Build using Maven (master, Scala 2.13, Hadoop 3, JDK 21) -> Build \ Maven (master, Scala 2.13, Hadoop 3, JDK 21) ### Why are the changes needed? To make it easier to navigate, and to be consistent. By doing this, you don't have to click "Show more workloads" at https://github.com/apache/spark/actions  ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? Will be tested in this PR's GitHub Actions. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #45160 from HyukjinKwon/maven-builds. Authored-by: Hyukjin Kwon Signed-off-by: Hyukjin Kwon --- .github/workflows/build_maven.yml| 2 +- .github/workflows/build_maven_java21.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/build_maven.yml b/.github/workflows/build_maven.yml index d43366fa86e7..863c5e16ba4b 100644 --- a/.github/workflows/build_maven.yml +++ b/.github/workflows/build_maven.yml @@ -17,7 +17,7 @@ # under the License. # -name: "Build using Maven (master, Scala 2.13, Hadoop 3, JDK 17)" +name: "Build / Maven (master, Scala 2.13, Hadoop 3, JDK 17)" on: schedule: diff --git a/.github/workflows/build_maven_java21.yml b/.github/workflows/build_maven_java21.yml index e50f996a50ba..127904145464 100644 --- a/.github/workflows/build_maven_java21.yml +++ b/.github/workflows/build_maven_java21.yml @@ -17,7 +17,7 @@ # under the License. # -name: "Build using Maven (master, Scala 2.13, Hadoop 3, JDK 21)" +name: "Build / Maven (master, Scala 2.13, Hadoop 3, JDK 21)" on: schedule: - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
(spark) branch master updated: [SPARK-47066][INFRA][FOLLOW-UP] Deduplicate Apple Silicon Maven build definition
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 8a9a1be9b02a [SPARK-47066][INFRA][FOLLOW-UP] Deduplicate Apple Silicon
Maven build definition
8a9a1be9b02a is described below
commit 8a9a1be9b02ad772aa5bdb61429b92a70e894cce
Author: Hyukjin Kwon
AuthorDate: Mon Feb 19 12:10:46 2024 +0900
[SPARK-47066][INFRA][FOLLOW-UP] Deduplicate Apple Silicon Maven build
definition
### What changes were proposed in this pull request?
This PR is a followup of https://github.com/apache/spark/pull/45126 that
deduplicates the build code.
### Why are the changes needed?
For better maintenance of the code
### Does this PR introduce _any_ user-facing change?
No, dev-only.
### How was this patch tested?
This PR should test it out.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #45148 from HyukjinKwon/SPARK-47066-followup.
Lead-authored-by: Hyukjin Kwon
Co-authored-by: Hyukjin Kwon
Signed-off-by: Hyukjin Kwon
---
.github/workflows/build_and_test.yml | 78 ++--
1 file changed, 13 insertions(+), 65 deletions(-)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index bad34fd746ba..eda86d1066fb 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -82,7 +82,7 @@ jobs:
sparkr=`./dev/is-changed.py -m sparkr`
tpcds=`./dev/is-changed.py -m sql`
docker=`./dev/is-changed.py -m docker-integration-tests`
- # 'build' and 'java-other-versions' are always true for now.
+ # 'build' and 'maven-build' are always true for now.
# It does not save significant time and most of PRs trigger the
build.
precondition="
{
@@ -91,7 +91,7 @@ jobs:
\"sparkr\": \"$sparkr\",
\"tpcds-1g\": \"$tpcds\",
\"docker-integration-tests\": \"$docker\",
- \"java-other-versions\": \"true\",
+ \"maven-build\": \"true\",
\"lint\" : \"true\",
\"k8s-integration-tests\" : \"true\",
\"buf\" : \"true\",
@@ -788,17 +788,21 @@ jobs:
path: site.tar.bz2
retention-days: 1
- java-other-versions:
+ maven-build:
needs: precondition
-if: fromJson(needs.precondition.outputs.required).java-other-versions ==
'true'
-name: Java ${{ matrix.java }} build with Maven
+if: fromJson(needs.precondition.outputs.required).maven-build == 'true'
+name: Java ${{ matrix.java }} build with Maven (${{ matrix.os }})
strategy:
fail-fast: false
matrix:
-java:
- - 17
- - 21
-runs-on: ubuntu-22.04
+include:
+ - java: 17
+os: ubuntu-22.04
+ - java: 21
+os: ubuntu-22.04
+ - java: 21
+os: macos-14
+runs-on: ${{ matrix.os }}
timeout-minutes: 300
steps:
- name: Checkout Spark repository
@@ -845,62 +849,6 @@ jobs:
./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pkubernetes -Pvolcano
-Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=${JAVA_VERSION/-ea}
install
rm -rf ~/.m2/repository/org/apache/spark
- apple-silicon:
-needs: precondition
-if: fromJson(needs.precondition.outputs.required).build == 'true'
-name: Apple Silicon build with Maven
-strategy:
- fail-fast: false
- matrix:
-java:
- - 21
-runs-on: macos-14
-timeout-minutes: 300
-steps:
-- name: Checkout Spark repository
- uses: actions/checkout@v4
- with:
-fetch-depth: 0
-repository: apache/spark
-ref: ${{ inputs.branch }}
-- name: Sync the current branch with the latest in Apache Spark
- if: github.repository != 'apache/spark'
- run: |
-git fetch https://github.com/$GITHUB_REPOSITORY.git
${GITHUB_REF#refs/heads/}
-git -c user.name='Apache Spark Test Account' -c
user.email='sparktest...@gmail.com' merge --no-commit --progress --squash
FETCH_HEAD
-git -c user.name='Apache Spark Test Account' -c
user.email='sparktest...@gmail.com' commit -m "Merged commit" --allow-empty
-- name: Cache Scala, SBT and Maven
- uses: actions/cache@v4
- with:
-path: |
- build/apache-maven-*
- build/scala-*
- build/*.jar
- ~/.sbt
-key: build-${{ hashFiles('**/pom.xml', 'project/build.properties',
'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash',
'build/spark-build-info') }}
-restore-keys: |
- apple-silicon-build-
-- name: Cache Maven local repository
- uses:
(spark) branch master updated: [SPARK-47083][BUILD] Upgrade `commons-codec` to 1.16.1
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new b61a00c26f0b [SPARK-47083][BUILD] Upgrade `commons-codec` to 1.16.1 b61a00c26f0b is described below commit b61a00c26f0b738ea0f5603dd04fe605e0578ea3 Author: panbingkun AuthorDate: Sun Feb 18 18:56:46 2024 -0800 [SPARK-47083][BUILD] Upgrade `commons-codec` to 1.16.1 ### What changes were proposed in this pull request? The pr aims to upgrade `commons-codec` from `1.16.0` to `1.16.1`. ### Why are the changes needed? 1.The new version brings some bug fixed, eg: - Fix possible IndexOutOfBoundException in PhoneticEngine.encode method #223. Fixes [CODEC-315](https://issues.apache.org/jira/browse/CODEC-315) - Fix possible IndexOutOfBoundsException in PercentCodec.insertAlwaysEncodeChars() method #222. Fixes [CODEC-314](https://issues.apache.org/jira/browse/CODEC-314). 2.The full release notes: https://commons.apache.org/proper/commons-codec/changes-report.html#a1.16.1 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #45152 from panbingkun/SPARK-47083. Authored-by: panbingkun Signed-off-by: Dongjoon Hyun --- dev/deps/spark-deps-hadoop-3-hive-2.3 | 2 +- pom.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3 b/dev/deps/spark-deps-hadoop-3-hive-2.3 index 5aabe0e4aef1..dbbddbc54c11 100644 --- a/dev/deps/spark-deps-hadoop-3-hive-2.3 +++ b/dev/deps/spark-deps-hadoop-3-hive-2.3 @@ -36,7 +36,7 @@ cats-kernel_2.13/2.8.0//cats-kernel_2.13-2.8.0.jar chill-java/0.10.0//chill-java-0.10.0.jar chill_2.13/0.10.0//chill_2.13-0.10.0.jar commons-cli/1.6.0//commons-cli-1.6.0.jar -commons-codec/1.16.0//commons-codec-1.16.0.jar +commons-codec/1.16.1//commons-codec-1.16.1.jar commons-collections/3.2.2//commons-collections-3.2.2.jar commons-collections4/4.4//commons-collections4-4.4.jar commons-compiler/3.1.9//commons-compiler-3.1.9.jar diff --git a/pom.xml b/pom.xml index 64931dd73282..410cefb0e8d9 100644 --- a/pom.xml +++ b/pom.xml @@ -191,7 +191,7 @@ 3.0.2 1.1.10.5 3.0.3 -1.16.0 +1.16.1 1.25.0 2.15.1 - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
(spark) branch master updated: [SPARK-47084][BUILD] Upgrade joda-time to 2.12.7
This is an automated email from the ASF dual-hosted git repository. yangjie01 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 85108b0cb605 [SPARK-47084][BUILD] Upgrade joda-time to 2.12.7 85108b0cb605 is described below commit 85108b0cb6059e9a5301b63ab266084defd0ddf2 Author: panbingkun AuthorDate: Mon Feb 19 10:15:37 2024 +0800 [SPARK-47084][BUILD] Upgrade joda-time to 2.12.7 ### What changes were proposed in this pull request? The pr aims to upgrade `joda-time` from `2.12.6` to `2.12.7`. ### Why are the changes needed? The version `DateTimeZone` data updated to version `2024agtz`. The full release notes: https://www.joda.org/joda-time/changes-report.html#a2.12.7 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #45153 from panbingkun/SPARK-47084. Authored-by: panbingkun Signed-off-by: yangjie01 --- dev/deps/spark-deps-hadoop-3-hive-2.3 | 2 +- pom.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3 b/dev/deps/spark-deps-hadoop-3-hive-2.3 index 0b619a249e96..5aabe0e4aef1 100644 --- a/dev/deps/spark-deps-hadoop-3-hive-2.3 +++ b/dev/deps/spark-deps-hadoop-3-hive-2.3 @@ -139,7 +139,7 @@ jetty-util/10.0.19//jetty-util-10.0.19.jar jline/2.14.6//jline-2.14.6.jar jline/3.22.0//jline-3.22.0.jar jna/5.13.0//jna-5.13.0.jar -joda-time/2.12.6//joda-time-2.12.6.jar +joda-time/2.12.7//joda-time-2.12.7.jar jodd-core/3.5.2//jodd-core-3.5.2.jar jpam/1.1//jpam-1.1.jar json/1.8//json-1.8.jar diff --git a/pom.xml b/pom.xml index a14f2d255a90..64931dd73282 100644 --- a/pom.xml +++ b/pom.xml @@ -208,7 +208,7 @@ Because it transitions Jakarta REST API from javax to jakarta package. --> 2.41 -2.12.6 +2.12.7 3.5.2 3.0.0 0.12.0 - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
(spark) branch master updated: [MINOR][INFRA][DOCS] Remove undated comment in build_and_test.yml
This is an automated email from the ASF dual-hosted git repository. yangjie01 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 7c60fe21a29d [MINOR][INFRA][DOCS] Remove undated comment in build_and_test.yml 7c60fe21a29d is described below commit 7c60fe21a29dd852de01da214c84e6a3deb38e31 Author: Hyukjin Kwon AuthorDate: Mon Feb 19 10:13:52 2024 +0800 [MINOR][INFRA][DOCS] Remove undated comment in build_and_test.yml ### What changes were proposed in this pull request? This PR removes outdated comment. We don't use branch-3.3 anymore ### Why are the changes needed? To remove obsolete information ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? N/A ### Was this patch authored or co-authored using generative AI tooling? No. Closes #45149 from HyukjinKwon/minor-ga. Authored-by: Hyukjin Kwon Signed-off-by: yangjie01 --- .github/workflows/build_and_test.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/.github/workflows/build_and_test.yml b/.github/workflows/build_and_test.yml index c578d5079be8..bad34fd746ba 100644 --- a/.github/workflows/build_and_test.yml +++ b/.github/workflows/build_and_test.yml @@ -285,7 +285,6 @@ jobs: infra-image: name: "Base image build" needs: precondition -# Currently, enable docker build from cache for `master` and branch (since 3.4) jobs if: >- fromJson(needs.precondition.outputs.required).pyspark == 'true' || fromJson(needs.precondition.outputs.required).lint == 'true' || - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
