[GitHub] [spark] LuciferYang commented on pull request #42673: [SPARK-44959][BUILD] Upgrade sbt to 1.9.5

LuciferYang commented on PR #42673: URL: https://github.com/apache/spark/pull/42673#issuecomment-1733038803 > > could you rebase this one ? @panbingkun > > I have submitted a test on the latest master, but unfortunately it still has issues. I have identified the issue, but I need some tim

[GitHub] [spark] yaooqinn commented on a diff in pull request #43092: [SPARK-45305][TESTS] Remove JDK 8 workaround added TreeNodeSuite

yaooqinn commented on code in PR #43092: URL: https://github.com/apache/spark/pull/43092#discussion_r1335451453 ## sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/trees/TreeNodeSuite.scala: ## @@ -871,28 +871,6 @@ class TreeNodeSuite extends SparkFunSuite with SQLHelpe

[GitHub] [spark] panbingkun commented on pull request #42673: [SPARK-44959][BUILD] Upgrade sbt to 1.9.5

panbingkun commented on PR #42673: URL: https://github.com/apache/spark/pull/42673#issuecomment-1733011847 > could you rebase this one ? @panbingkun I have submitted a test on the latest master, but unfortunately it still has issues. I have identified the issue, but I need some time to se

[GitHub] [spark] itholic opened a new pull request, #43096: [SPARK-45308][PS][TESTS] Enable `GroupbySplitApplyTests.test_split_apply_combine_on_series` for pandas 2.0.0.

itholic opened a new pull request, #43096: URL: https://github.com/apache/spark/pull/43096 ### What changes were proposed in this pull request? This PR proposes to enable `GroupbySplitApplyTests.test_split_apply_combine_on_series`. ### Why are the changes needed? Sim

[GitHub] [spark] pan3793 commented on pull request #42599: [SPARK-45292][SQL][HIVE] Remove Guava from shared classes from IsolatedClientLoader

pan3793 commented on PR #42599: URL: https://github.com/apache/spark/pull/42599#issuecomment-1733003556 cc @wangyum @LuciferYang too -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

[GitHub] [spark] HyukjinKwon commented on pull request #43089: [SPARK-37508][SQL] Add Unicode[encode/decode]() function

HyukjinKwon commented on PR #43089: URL: https://github.com/apache/spark/pull/43089#issuecomment-1733003061 Do you happen to have any reference for these functions in other DBMSes? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

[GitHub] [spark] HyukjinKwon commented on pull request #43064: [SPARK-45265][SQL][WIP] Supporting Hive 4.0 metastore

HyukjinKwon commented on PR #43064: URL: https://github.com/apache/spark/pull/43064#issuecomment-1733000616 cc @wangyum too -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [spark] yaooqinn commented on pull request #43073: [SPARK-45291][SQL][REST] Use unknown query execution id instead of no such app when id is invalid

yaooqinn commented on PR #43073: URL: https://github.com/apache/spark/pull/43073#issuecomment-1732997365 Thanks, merged to master and 3.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

[GitHub] [spark] yaooqinn closed pull request #43073: [SPARK-45291][SQL][REST] Use unknown query execution id instead of no such app when id is invalid

yaooqinn closed pull request #43073: [SPARK-45291][SQL][REST] Use unknown query execution id instead of no such app when id is invalid URL: https://github.com/apache/spark/pull/43073 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] HyukjinKwon closed pull request #40420: [SPARK-42617][PS] Support `isocalendar` from the pandas 2.0.0

HyukjinKwon closed pull request #40420: [SPARK-42617][PS] Support `isocalendar` from the pandas 2.0.0 URL: https://github.com/apache/spark/pull/40420 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] HyukjinKwon commented on pull request #40420: [SPARK-42617][PS] Support `isocalendar` from the pandas 2.0.0

HyukjinKwon commented on PR #40420: URL: https://github.com/apache/spark/pull/40420#issuecomment-1732993368 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [spark] Ngone51 commented on a diff in pull request #42950: [SPARK-45182][CORE] Ignore task completion from old stage after retrying indeterminate stages

Ngone51 commented on code in PR #42950: URL: https://github.com/apache/spark/pull/42950#discussion_r1335422697 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -1903,13 +1903,20 @@ private[spark] class DAGScheduler( case smt: ShuffleMapTas

[GitHub] [spark] yaooqinn opened a new pull request, #43095: [SPARK-44895][CORE][UI] Add 'daemon', 'priority' for ThreadStackTrace

yaooqinn opened a new pull request, #43095: URL: https://github.com/apache/spark/pull/43095 ### What changes were proposed in this pull request? Since version 9, Java has supported the 'daemon' and 'priority' fields in ThreadInfo. In this PR, we extract them from ThreadInf

[GitHub] [spark] panbingkun commented on pull request #43094: [SPARK-45307][INFRA] Use Zulu JDK in `benchmark` GitHub Action and Java 21

panbingkun commented on PR #43094: URL: https://github.com/apache/spark/pull/43094#issuecomment-1732983311 cc @LuciferYang @dongjoon-hyun @HyukjinKwon -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

[GitHub] [spark] panbingkun opened a new pull request, #43094: [SPARK-45307][INFRA] Use Zulu JDK in `benchmark` GitHub Action and Java 21

panbingkun opened a new pull request, #43094: URL: https://github.com/apache/spark/pull/43094 … ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change?

[GitHub] [spark] LuciferYang commented on a diff in pull request #43058: Test new ammonite and Scala 2.13.12

LuciferYang commented on code in PR #43058: URL: https://github.com/apache/spark/pull/43058#discussion_r1335406826 ## pom.xml: ## @@ -228,7 +228,7 @@ ./python/pyspark/sql/pandas/utils.py, and ./python/setup.py too. --> 13.0.0 -2.5.9 +2.5.9-2-7633e640 Revi

[GitHub] [spark] LuciferYang commented on pull request #42673: [SPARK-44959][BUILD] Upgrade sbt to 1.9.5

LuciferYang commented on PR #42673: URL: https://github.com/apache/spark/pull/42673#issuecomment-1732962993 could you rebase this one ? @panbingkun -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[GitHub] [spark] LuciferYang commented on pull request #43093: [SPARK-45306][SQL][TESTS] Make `InMemoryColumnarBenchmark` use AQE-aware utils to collect plans

LuciferYang commented on PR #43093: URL: https://github.com/apache/spark/pull/43093#issuecomment-1732961961 cc @dongjoon-hyun @cloud-fan FYI I contacted @ulysses-you offline, he is on vacation, so I give this pr. I will update the Java 17/21 test results of `InMemoryColumnarBen

[GitHub] [spark] Ngone51 commented on pull request #42426: [SPARK-44756][CORE] Executor hangs when RetryingBlockTransferor fails to initiate retry

Ngone51 commented on PR #42426: URL: https://github.com/apache/spark/pull/42426#issuecomment-1732956928 > ...the OOM was thrown within submit() itself even before a thread was spawned. @hdaikoku You're right. The issue is different here. -- This is an automated message from the Apa

[GitHub] [spark] zhengruifeng commented on pull request #43011: [SPARK-45232][SQL][DOCS] Add missing function groups to SQL references

zhengruifeng commented on PR #43011: URL: https://github.com/apache/spark/pull/43011#issuecomment-1732950643  ![image](https://github.com/apache/spark/assets/7322292/8503b5ab-794a-418

[GitHub] [spark] LuciferYang opened a new pull request, #43093: Fix InMemoryColumnarBenchmark

LuciferYang opened a new pull request, #43093: URL: https://github.com/apache/spark/pull/43093 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How

[GitHub] [spark] LuciferYang commented on pull request #43074: [SPARK-44170][BUILD][TESTS] Migrate Junit 4 to Junit 5

LuciferYang commented on PR #43074: URL: https://github.com/apache/spark/pull/43074#issuecomment-1732927710 cc @srowen FYI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

[GitHub] [spark] itholic commented on pull request #40420: [SPARK-42617][PS] Support `isocalendar` from the pandas 2.0.0

itholic commented on PR #40420: URL: https://github.com/apache/spark/pull/40420#issuecomment-1732912031 LGTM now. Thanks for your consistent work on this! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] itholic commented on a diff in pull request #42985: [SPARK-44838][SQL][WIP] raise_error improvement

itholic commented on code in PR #42985: URL: https://github.com/apache/spark/pull/42985#discussion_r1335371923 ## python/pyspark/sql/tests/test_functions.py: ## @@ -1032,9 +1032,11 @@ def check_assert_true(self, tpe): ) with self.assertRaisesRegex(tpe, "too b

[GitHub] [spark] yaooqinn commented on a diff in pull request #43084: [SPARK-45299][TESTS] Remove JDK 8 workaround in UtilsSuite

yaooqinn commented on code in PR #43084: URL: https://github.com/apache/spark/pull/43084#discussion_r1335366773 ## core/src/test/scala/org/apache/spark/util/UtilsSuite.scala: ## @@ -1018,37 +1012,34 @@ class UtilsSuite extends SparkFunSuite with ResetSystemProperties {

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #43085: [SPARK-45300][SQL][TESTS] Remove JDK 8 workaround in TimestampFormatterSuite

HyukjinKwon commented on code in PR #43085: URL: https://github.com/apache/spark/pull/43085#discussion_r1335359916 ## sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/TimestampFormatterSuite.scala: ## @@ -333,14 +333,8 @@ class TimestampFormatterSuite extends Date

[GitHub] [spark] HyukjinKwon commented on pull request #43092: [SPARK-45305][TESTS] Remove JDK 8 workaround added TreeNodeSuite

HyukjinKwon commented on PR #43092: URL: https://github.com/apache/spark/pull/43092#issuecomment-1732881962 cc @rednaxelafx -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [spark] HyukjinKwon opened a new pull request, #43092: [SPARK-45305][TESTS] Remove JDK 8 workaround added TreeNodeSuite

HyukjinKwon opened a new pull request, #43092: URL: https://github.com/apache/spark/pull/43092 ### What changes were proposed in this pull request? In theory, we don't need https://github.com/apache/spark/pull/29875 anymore because we dropped JDK 8 (according to the PR description) bu

[GitHub] [spark] HyukjinKwon commented on pull request #43088: [SPARK-45303][CORE] Remove JDK 8/11 workaround in KryoSerializerBenchmark

HyukjinKwon commented on PR #43088: URL: https://github.com/apache/spark/pull/43088#issuecomment-1732878129 cc @dongjoon-hyun -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[GitHub] [spark] HyukjinKwon commented on pull request #43091: [MINOR][SQL][DOCS] Remove JDK 8 related information in the comemnts for aes_encrypt and aes_decrypt

HyukjinKwon commented on PR #43091: URL: https://github.com/apache/spark/pull/43091#issuecomment-1732876723 cc @MaxGekk -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] HyukjinKwon opened a new pull request, #43091: [MINOR][SQL][DOCS] Remove JDK 8 related information in the comemnts for aes_encrypt and aes_decrypt

HyukjinKwon opened a new pull request, #43091: URL: https://github.com/apache/spark/pull/43091 ### What changes were proposed in this pull request? This PR proposes to fix the comments in both `aes_encrypt` and `aes_decrypt`. Did a quick check for Scala/Python/R API, and seems like th

[GitHub] [spark] HyukjinKwon commented on pull request #43090: [SPARK-45304][BUILD] Remove test classloader workaround for SBT build

HyukjinKwon commented on PR #43090: URL: https://github.com/apache/spark/pull/43090#issuecomment-1732870726 This one I am pretty sure as it reverts my own PR :-). I am going to merge this when the tests pass - it's a clean revert. -- This is an automated message from the Apache Git Servic

[GitHub] [spark] HyukjinKwon opened a new pull request, #43090: [SPARK-45304][BUILD] Remove test classloader workaround for SBT build

HyukjinKwon opened a new pull request, #43090: URL: https://github.com/apache/spark/pull/43090 ### What changes were proposed in this pull request? This PR proposes to remove the workaround for JDK 8 in SBT build, see https://github.com/apache/spark/pull/30198 ### Why are the c

[GitHub] [spark] feiyx opened a new pull request, #43089: [SPARK-37508][SQL] Add Unicode[encode/decode]() function

feiyx opened a new pull request, #43089: URL: https://github.com/apache/spark/pull/43089 ### What changes were proposed in this pull request? Currently, Spark don't support unicode encode/decode as built-in functions, the usermight use reflect instead , It's a bit of poor efficiency

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #43088: [SPARK-45303][CORE] Remove JDK 8/11 workaround in KryoSerializerBenchmark

HyukjinKwon commented on code in PR #43088: URL: https://github.com/apache/spark/pull/43088#discussion_r1335350938 ## core/src/test/scala/org/apache/spark/benchmark/BenchmarkBase.scala: ## @@ -23,7 +23,7 @@ import org.apache.spark.internal.config.Tests.IS_TESTING /** * A ba

[GitHub] [spark] HyukjinKwon opened a new pull request, #43088: [SPARK-45303][CORE] Remove JDK 8/11 workaround in KryoSerializerBenchmark

HyukjinKwon opened a new pull request, #43088: URL: https://github.com/apache/spark/pull/43088 ### What changes were proposed in this pull request? This PR removes the legacy workaround for JDK 8/11 in SPARK-29282. They were already removed in SPARK-37293. This is the leftover.

[GitHub] [spark] cloud-fan commented on pull request #40390: [SPARK-42768][SQL] Enable cached plan apply AQE by default

cloud-fan commented on PR #40390: URL: https://github.com/apache/spark/pull/40390#issuecomment-1732857566 The benchmark should either disable the conf, or use AQE-aware utils to collect plans (See `AdaptiveSparkPlanHelper`). -- This is an automated message from the Apache Git Service. To

[GitHub] [spark] HyukjinKwon commented on pull request #43087: [SPARK-45302][PYTHON] Remove PID communication between Python workers when no demon is used

HyukjinKwon commented on PR #43087: URL: https://github.com/apache/spark/pull/43087#issuecomment-1732856384 cc @ueshin -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

[GitHub] [spark] HyukjinKwon opened a new pull request, #43087: [SPARK-45302][PYTHON] Remove PID communication between Python workers when no demon is used

HyukjinKwon opened a new pull request, #43087: URL: https://github.com/apache/spark/pull/43087 ### What changes were proposed in this pull request? This PR removes the legacy workaround for JDK 8 in `PythonWorkerFactory`. ### Why are the changes needed? No need to manuall

[GitHub] [spark] cloud-fan commented on pull request #42950: [SPARK-45182][CORE] Ignore task completion from old stage after retrying indeterminate stages

cloud-fan commented on PR #42950: URL: https://github.com/apache/spark/pull/42950#issuecomment-1732852240 LGTM if all tests pass -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

[GitHub] [spark] HyukjinKwon commented on pull request #43086: [SPARK-28932][BUILD] Remove org.scala-lang scala-library added for JDK 11 workaround

HyukjinKwon commented on PR #43086: URL: https://github.com/apache/spark/pull/43086#issuecomment-1732851718 cc @dongjoon-hyun -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[GitHub] [spark] HyukjinKwon opened a new pull request, #43086: [SPARK-28932][BUILD] Remove org.scala-lang scala-library added for JDK 11 workaround

HyukjinKwon opened a new pull request, #43086: URL: https://github.com/apache/spark/pull/43086 ### What changes were proposed in this pull request? This PR removes the legacy workaround for JDK 11 added at SPARK-28932 ### Why are the changes needed? To remove legacy worka

[GitHub] [spark] yaooqinn commented on a diff in pull request #43084: [SPARK-45299][TESTS] Remove JDK 8 workaround in UtilsSuite

yaooqinn commented on code in PR #43084: URL: https://github.com/apache/spark/pull/43084#discussion_r1335338130 ## core/src/test/scala/org/apache/spark/util/UtilsSuite.scala: ## @@ -1018,37 +1018,35 @@ class UtilsSuite extends SparkFunSuite with ResetSystemProperties {

[GitHub] [spark] panbingkun commented on pull request #42143: [SPARK-44539][BUILD] Upgrade RoaringBitmap to 1.0.0

panbingkun commented on PR #42143: URL: https://github.com/apache/spark/pull/42143#issuecomment-1732844716 From version 1.0.0, the `ArraysShim` class has been moved from `shims-x.x.x.jar` jar to `RoaringBitmap-x.x.x.jar` jar, so we no longer need to rely on it. https://github.com/apa

[GitHub] [spark] yaooqinn commented on a diff in pull request #43084: [SPARK-45299][TESTS] Remove JDK 8 workaround in UtilsSuite

yaooqinn commented on code in PR #43084: URL: https://github.com/apache/spark/pull/43084#discussion_r1335338130 ## core/src/test/scala/org/apache/spark/util/UtilsSuite.scala: ## @@ -1018,37 +1018,35 @@ class UtilsSuite extends SparkFunSuite with ResetSystemProperties {

[GitHub] [spark] HyukjinKwon commented on pull request #43085: [SPARK-45300][SQL][TESTS] Remove JDK 8 workaround in TimestampFormatterSuite

HyukjinKwon commented on PR #43085: URL: https://github.com/apache/spark/pull/43085#issuecomment-1732843175 cc @yaooqinn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] HyukjinKwon opened a new pull request, #43085: [SPARK-45300][SQL][TESTS] Remove JDK 8 workaround in TimestampFormatterSuite

HyukjinKwon opened a new pull request, #43085: URL: https://github.com/apache/spark/pull/43085 ### What changes were proposed in this pull request? This PR removes the legacy workaround for JDK 8 in https://github.com/apache/spark/pull/28736. ### Why are the changes needed?

[GitHub] [spark] HyukjinKwon commented on pull request #43084: [SPARK-45299][TESTS] Remove JDK 8 workaround in UtilsSuite

HyukjinKwon commented on PR #43084: URL: https://github.com/apache/spark/pull/43084#issuecomment-1732836567 cc @srowen -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

[GitHub] [spark] HyukjinKwon opened a new pull request, #43084: [SPARK-45299][TESTS] Remove JDK 8 workaround in UtilsSuite

HyukjinKwon opened a new pull request, #43084: URL: https://github.com/apache/spark/pull/43084 ### What changes were proposed in this pull request? This PR removes the legacy workaround for JDK 7 and below at SPARK-12486. The main code was cleaned up at SPARK-16182 but the test code w

[GitHub] [spark] HyukjinKwon commented on pull request #43083: [SPARK-45298][SQL][TESTS] Remove the workaround for JDK-8228469 in SPARK-31959 test

HyukjinKwon commented on PR #43083: URL: https://github.com/apache/spark/pull/43083#issuecomment-1732830208 cc @MaxGekk -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] HyukjinKwon opened a new pull request, #43083: [SPARK-45298][SQL][TESTS] Remove the workaround for JDK-8228469 in SPARK-31959 test

HyukjinKwon opened a new pull request, #43083: URL: https://github.com/apache/spark/pull/43083 ### What changes were proposed in this pull request? This PR removes the legacy workaround for old JDK added at SPARK-31959 ### Why are the changes needed? To remove legacy work

[GitHub] [spark] HyukjinKwon commented on pull request #43082: [SPARK-45297][SQL] Remove workaround for dateformatter added in SPARK-31827

HyukjinKwon commented on PR #43082: URL: https://github.com/apache/spark/pull/43082#issuecomment-1732827642 cc @cloud-fan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

[GitHub] [spark] HyukjinKwon opened a new pull request, #43082: [SPARK-45297][SQL] Remove workaround for dateformatter added in SPARK-31827

HyukjinKwon opened a new pull request, #43082: URL: https://github.com/apache/spark/pull/43082 ### What changes were proposed in this pull request? This PR removes the legacy workaround for JDK 8 added at SPARK-31827 ### Why are the changes needed? To remove legacy workar

[GitHub] [spark] HyukjinKwon opened a new pull request, #43081: [SPARK-45296][INFRA][BUILD] Comment out unused JDK 11 related in dev/run-tests.py

HyukjinKwon opened a new pull request, #43081: URL: https://github.com/apache/spark/pull/43081 ### What changes were proposed in this pull request? This PR proposes to comment unused JDK 11 related in `dev/run-tests.py`. ### Why are the changes needed? For readability, an

[GitHub] [spark] HyukjinKwon opened a new pull request, #43080: [SPARK-45295][CORE][SQL] Remove Utils.isMemberClass workaround for JDK 8

HyukjinKwon opened a new pull request, #43080: URL: https://github.com/apache/spark/pull/43080 ### What changes were proposed in this pull request? This PR removes the legacy workaround for JDK 8 added at SPARK-34607 ### Why are the changes needed? To remove legacy workar

[GitHub] [spark] yaooqinn commented on pull request #43073: [SPARK-45291][SQL][REST] Use unknown query execution id instead of no such app when id is invalid

yaooqinn commented on PR #43073: URL: https://github.com/apache/spark/pull/43073#issuecomment-1732813479 cc @dongjoon-hyun @HyukjinKwon, thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] HyukjinKwon closed pull request #43077: [SPARK-45294][PYTHON][DOCS] Use JDK 17 in Binder integration for PySpark live notebooks

HyukjinKwon closed pull request #43077: [SPARK-45294][PYTHON][DOCS] Use JDK 17 in Binder integration for PySpark live notebooks URL: https://github.com/apache/spark/pull/43077 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

[GitHub] [spark] HyukjinKwon commented on pull request #43077: [SPARK-45294][PYTHON][DOCS] Use JDK 17 in Binder integration for PySpark live notebooks

HyukjinKwon commented on PR #43077: URL: https://github.com/apache/spark/pull/43077#issuecomment-1732809225 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [spark] panbingkun opened a new pull request, #43079: [Don't merge & review] verify sbt on master

panbingkun opened a new pull request, #43079: URL: https://github.com/apache/spark/pull/43079 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How

[GitHub] [spark] chenyu-opensource opened a new pull request, #43078: [SPARK-45248][CORE]Set the timeout for spark ui server

chenyu-opensource opened a new pull request, #43078: URL: https://github.com/apache/spark/pull/43078 **What changes were proposed in this pull request?** The PR supports to set the timeout for spark ui server. **Why are the changes needed?** It can avoid slow HTTP Denial of Servi

[GitHub] [spark] chenyu-opensource closed pull request #43028: [SPARK-45248][CORE]Set the timeout for spark ui server

chenyu-opensource closed pull request #43028: [SPARK-45248][CORE]Set the timeout for spark ui server URL: https://github.com/apache/spark/pull/43028 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[GitHub] [spark] zhengruifeng commented on pull request #43011: [SPARK-45232][SQL][DOCS] Add missing function groups to SQL references

zhengruifeng commented on PR #43011: URL: https://github.com/apache/spark/pull/43011#issuecomment-1732777865 > > > > > What is the purpose of "lambda function"? All others are type-specific or "functionality"-specific. But lambda is "technology". > > > > > > > > > > > > lambda fun

[GitHub] [spark] chenyu-opensource commented on pull request #43028: [SPARK-45248][CORE]Set the timeout for spark ui server

chenyu-opensource commented on PR #43028: URL: https://github.com/apache/spark/pull/43028#issuecomment-173244 > Can you rebase your branch on master? i think you need to pick up the latest test config for the CI/CD jobs ok，i will checkout a new branch from the master and open a ne

[GitHub] [spark] yaooqinn commented on pull request #43072: [SPARK-45257][CORE][FOLLOWUP] Correct the from version in migration guide

yaooqinn commented on PR #43072: URL: https://github.com/apache/spark/pull/43072#issuecomment-1732773397 thanks, merged to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

[GitHub] [spark] yaooqinn closed pull request #43072: [SPARK-45257][CORE][FOLLOWUP] Correct the from version in migration guide

yaooqinn closed pull request #43072: [SPARK-45257][CORE][FOLLOWUP] Correct the from version in migration guide URL: https://github.com/apache/spark/pull/43072 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [spark] panbingkun commented on a diff in pull request #43076: [SPARK-45276][INFRA] Replace Java 8 and Java 11 installed in the Dockerfile with Java

panbingkun commented on code in PR #43076: URL: https://github.com/apache/spark/pull/43076#discussion_r1335298085 ## dev/create-release/spark-rm/Dockerfile: ## @@ -61,9 +61,9 @@ RUN apt-get clean && apt-get update && $APT_INSTALL gnupg ca-certificates && \ apt-get update &&

[GitHub] [spark] HyukjinKwon opened a new pull request, #43077: [SPARK-45294][PYTHON][DOCS] Use JDK 17 in Binder integration for PySpark live notebooks

HyukjinKwon opened a new pull request, #43077: URL: https://github.com/apache/spark/pull/43077 ### What changes were proposed in this pull request? This PR upgrades the JRE from 8 to 17 in Binder live notebooks. ### Why are the changes needed? To use the properly supporte

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #43076: [SPARK-45276][INFRA] Replace Java 8 and Java 11 installed in the Dockerfile with Java

HyukjinKwon commented on code in PR #43076: URL: https://github.com/apache/spark/pull/43076#discussion_r1335295623 ## dev/create-release/spark-rm/Dockerfile: ## @@ -61,9 +61,9 @@ RUN apt-get clean && apt-get update && $APT_INSTALL gnupg ca-certificates && \ apt-get update &&

[GitHub] [spark] panbingkun commented on pull request #43076: [SPARK-45276][INFRA] Replace Java 8 and Java 11 installed in the Dockerfile with Java

panbingkun commented on PR #43076: URL: https://github.com/apache/spark/pull/43076#issuecomment-1732759125 cc @LuciferYang @dongjoon-hyun -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

[GitHub] [spark] panbingkun opened a new pull request, #43076: [SPARK-45293][INFRA] Install Java 17 for docker

panbingkun opened a new pull request, #43076: URL: https://github.com/apache/spark/pull/43076 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this

[GitHub] [spark] HyukjinKwon closed pull request #43034: [SPARK-45240][SQL][CONNECT] Implement Error Enrichment for Python Client

HyukjinKwon closed pull request #43034: [SPARK-45240][SQL][CONNECT] Implement Error Enrichment for Python Client URL: https://github.com/apache/spark/pull/43034 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

[GitHub] [spark] HyukjinKwon closed pull request #42987: [SPARK-45207][SQL][CONNECT] Implement Error Enrichment for Scala Client

HyukjinKwon closed pull request #42987: [SPARK-45207][SQL][CONNECT] Implement Error Enrichment for Scala Client URL: https://github.com/apache/spark/pull/42987 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [spark] HyukjinKwon commented on pull request #42987: [SPARK-45207][SQL][CONNECT] Implement Error Enrichment for Scala Client

HyukjinKwon commented on PR #42987: URL: https://github.com/apache/spark/pull/42987#issuecomment-1732724709 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [spark] HyukjinKwon commented on pull request #43034: [SPARK-45240][SQL][CONNECT] Implement Error Enrichment for Python Client

HyukjinKwon commented on PR #43034: URL: https://github.com/apache/spark/pull/43034#issuecomment-1732724619 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [spark] github-actions[bot] commented on pull request #40577: [SPARK-42947][SQL] Spark Thriftserver LDAP should not use DN pattern if user contains domain

github-actions[bot] commented on PR #40577: URL: https://github.com/apache/spark/pull/40577#issuecomment-1732714205 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.

[GitHub] [spark] github-actions[bot] commented on pull request #41407: [SPARK-43900][SQL] Support optimize skewed partitions even if introduce extra shuffle

github-actions[bot] commented on PR #41407: URL: https://github.com/apache/spark/pull/41407#issuecomment-1732714193 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.

[GitHub] [spark] github-actions[bot] commented on pull request #41607: [WIP][SPARK-43259][SQL] Rename the error class `_LEGACY_ERROR_TEMP_2024` to `UNSUPPORTED_ENCODER_ERROR`

github-actions[bot] commented on PR #41607: URL: https://github.com/apache/spark/pull/41607#issuecomment-1732714153 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.

[GitHub] [spark] github-actions[bot] commented on pull request #41457: [SPARK-44077] Session Configs were not getting honored in RDDs

github-actions[bot] commented on PR #41457: URL: https://github.com/apache/spark/pull/41457#issuecomment-1732714167 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.

[GitHub] [spark] zhengruifeng commented on pull request #43055: [SPARK-45279][PYTHON][CONNECT] Attach plan_id for all logical plans

zhengruifeng commented on PR #43055: URL: https://github.com/apache/spark/pull/43055#issuecomment-1732714144 thanks, merged to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] zhengruifeng closed pull request #43055: [SPARK-45279][PYTHON][CONNECT] Attach plan_id for all logical plans

zhengruifeng closed pull request #43055: [SPARK-45279][PYTHON][CONNECT] Attach plan_id for all logical plans URL: https://github.com/apache/spark/pull/43055 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] srowen commented on pull request #43063: [SPARK-45286][DOCS] Add back Matomo analytics

srowen commented on PR #43063: URL: https://github.com/apache/spark/pull/43063#issuecomment-1732648591 Merged to master/3.5/3.4/3.3 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

[GitHub] [spark] srowen closed pull request #43063: [SPARK-45286][DOCS] Add back Matomo analytics

srowen closed pull request #43063: [SPARK-45286][DOCS] Add back Matomo analytics URL: https://github.com/apache/spark/pull/43063 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [spark] srielau commented on pull request #43011: [SPARK-45232][SQL][DOCS] Add missing function groups to SQL references

srielau commented on PR #43011: URL: https://github.com/apache/spark/pull/43011#issuecomment-1732636254 > > > > What is the purpose of "lambda function"? All others are type-specific or "functionality"-specific. But lambda is "technology". > > > > > > > > > lambda functions were a

[GitHub] [spark] pan3793 commented on a diff in pull request #43064: [SPARK-45265][SQL][WIP] Supporting Hive 4.0 metastore

pan3793 commented on code in PR #43064: URL: https://github.com/apache/spark/pull/43064#discussion_r1335221474 ## sql/hive/src/main/scala/org/apache/spark/sql/hive/client/package.scala: ## @@ -131,8 +131,16 @@ package object client { "org.pentaho:pentaho-aggdesigner-alg

[GitHub] [spark] LuciferYang commented on pull request #43074: [SPARK-44170][TESTS] Junit 4 to Junit 5

LuciferYang commented on PR #43074: URL: https://github.com/apache/spark/pull/43074#issuecomment-1732619918 Test first -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

[GitHub] [spark] LuciferYang commented on pull request #43075: [SPARK-44366][BUILD] Upgrade antlr4 to 4.13.1

LuciferYang commented on PR #43075: URL: https://github.com/apache/spark/pull/43075#issuecomment-1732619865 Test first -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

[GitHub] [spark] LuciferYang opened a new pull request, #43075: [SPARK-44366][BUILD] Test antlr4 4.13.1

LuciferYang opened a new pull request, #43075: URL: https://github.com/apache/spark/pull/43075 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How

[GitHub] [spark] LuciferYang opened a new pull request, #43074: [SPARK-44170][TESTS] Junit 4 to Junit 5

LuciferYang opened a new pull request, #43074: URL: https://github.com/apache/spark/pull/43074 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How

[GitHub] [spark] yaooqinn opened a new pull request, #43073: [SPARK-45291][SQL][REST] Use unknown query execution id instead of no such app when id is invalid

yaooqinn opened a new pull request, #43073: URL: https://github.com/apache/spark/pull/43073 ### What changes were proposed in this pull request? This PR fixes `/api/v1/applications/{appId}/sql/{executionId}` API when the executionId is invalid. Before this, we g

[GitHub] [spark] pan3793 opened a new pull request, #43072: [SPARK-45257][CORE][FOLLOWUP] Correct the from version in migration guide

pan3793 opened a new pull request, #43072: URL: https://github.com/apache/spark/pull/43072 ### What changes were proposed in this pull request? Correct the from version in migration guide ### Why are the changes needed? Address comments https://github.com/apache

[GitHub] [spark] srowen commented on pull request #43028: [SPARK-45248][CORE]Set the timeout for spark ui server

srowen commented on PR #43028: URL: https://github.com/apache/spark/pull/43028#issuecomment-1732576148 Can you rebase your branch on master? i think you need to pick up the latest test config for the CI/CD jobs -- This is an automated message from the Apache Git Service. To respond to the