HyukjinKwon closed pull request #42693: [SPARK-44980][PYTHON][CONNECT] Fix
inherited namedtuples to work in createDataFrame
URL: https://github.com/apache/spark/pull/42693
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use t
HyukjinKwon commented on PR #42693:
URL: https://github.com/apache/spark/pull/42693#issuecomment-1695124698
Merged to master and branch-3.5 - this is a bug fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL a
hgs19921112 closed pull request #42572: [SPARK-44881[COMMON]Executor stucked on
retrying to fetch shuffle data when `java.lang.OutOfMemoryError. unable to
create native thread` exception occurred.
URL: https://github.com/apache/spark/pull/42572
--
This is an automated message from the Apache
cloud-fan commented on code in PR #42661:
URL: https://github.com/apache/spark/pull/42661#discussion_r1306954376
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TryEval.scala:
##
@@ -236,3 +236,35 @@ case class TryToBinary(
override protected def with
wForget opened a new pull request, #42697:
URL: https://github.com/apache/spark/pull/42697
### What changes were proposed in this pull request?
Fix incorrect link in sql menu and typo.
### Why are the changes needed?
### Does this PR introduce _any_ user-
yaooqinn commented on PR #42674:
URL: https://github.com/apache/spark/pull/42674#issuecomment-1695053749
thanks @srowen @gengliangwang @sarutak @HyukjinKwon @dongjoon-hyun, merged
to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log
yaooqinn closed pull request #42674: [SPARK-44960][UI] Unescape and consist
error summary across UI pages
URL: https://github.com/apache/spark/pull/42674
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
maheshk114 commented on code in PR #41860:
URL: https://github.com/apache/spark/pull/41860#discussion_r1306937895
##
sql/core/src/test/scala/org/apache/spark/sql/InjectRuntimeFilterSuite.scala:
##
@@ -644,4 +644,76 @@ class InjectRuntimeFilterSuite extends QueryTest with
SQLTes
maheshk114 commented on code in PR #41860:
URL: https://github.com/apache/spark/pull/41860#discussion_r1306937806
##
sql/core/src/test/scala/org/apache/spark/sql/InjectRuntimeFilterSuite.scala:
##
@@ -644,4 +644,76 @@ class InjectRuntimeFilterSuite extends QueryTest with
SQLTes
Hisoka-X commented on code in PR #42661:
URL: https://github.com/apache/spark/pull/42661#discussion_r1306927209
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/CallMethodViaReflection.scala:
##
@@ -133,8 +138,13 @@ case class CallMethodViaReflection(chil
Hisoka-X commented on code in PR #42661:
URL: https://github.com/apache/spark/pull/42661#discussion_r1306925328
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TryEval.scala:
##
@@ -236,3 +236,35 @@ case class TryToBinary(
override protected def withN
cloud-fan commented on code in PR #42661:
URL: https://github.com/apache/spark/pull/42661#discussion_r1306915491
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TryEval.scala:
##
@@ -236,3 +236,35 @@ case class TryToBinary(
override protected def with
cloud-fan commented on code in PR #42661:
URL: https://github.com/apache/spark/pull/42661#discussion_r1306914923
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/CallMethodViaReflection.scala:
##
@@ -133,8 +138,13 @@ case class CallMethodViaReflection(chi
LuciferYang commented on PR #42518:
URL: https://github.com/apache/spark/pull/42518#issuecomment-1695020180
The test failure is not related to the current pr, but the scala format need
to be fixed.
```
Run ./dev/lint-scala
[info] [launcher] getting org.scala-sbt sbt 1.9.3 (this may
LuciferYang closed pull request #42696: Test Yarn module with
`-Dtest.exclude.tags=org.apache.spark.tags.ExtendedLevelDBTest`
URL: https://github.com/apache/spark/pull/42696
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use
dongjoon-hyun commented on PR #41673:
URL: https://github.com/apache/spark/pull/41673#issuecomment-1695005013
Thank you, @LuciferYang !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specifi
LuciferYang commented on PR #41673:
URL: https://github.com/apache/spark/pull/41673#issuecomment-1695004548
Thanks @dongjoon-hyun ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific co
dongjoon-hyun commented on PR #41673:
URL: https://github.com/apache/spark/pull/41673#issuecomment-1694995407
BTW, I updated the JIRA title because
https://github.com/apache/spark/pull/42696 proves that this is not MacOS or
Apple-Silicon-related issues, @LuciferYang .
--
This is an autom
dongjoon-hyun closed pull request #41673: [SPARK-44091][YARN][TESTS] Introduce
`withResourceTypes` to `ResourceRequestTestHelper` to restore `resourceTypes`
as default value after testing
URL: https://github.com/apache/spark/pull/41673
--
This is an automated message from the Apache Git Serv
HyukjinKwon closed pull request #42692: [SPARK-42944][PYTHON][FOLLOW-UP][3.5]
Rename tests from foreachBatch to foreach_batch
URL: https://github.com/apache/spark/pull/42692
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use
HyukjinKwon commented on PR #42692:
URL: https://github.com/apache/spark/pull/42692#issuecomment-1694990647
Merged to branch-3.5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comm
Hisoka-X commented on code in PR #42661:
URL: https://github.com/apache/spark/pull/42661#discussion_r1306888014
##
sql/core/src/test/resources/sql-tests/inputs/try_reflect.sql:
##
@@ -0,0 +1,19 @@
+-- positive
+SELECT try_reflect("java.util.UUID", "fromString",
"a5cf6c42-0c85-4
HyukjinKwon closed pull request #42658: [SPARK-44945][DOCS][PYTHON] Automate
PySpark error class documentation
URL: https://github.com/apache/spark/pull/42658
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above
HyukjinKwon commented on PR #42658:
URL: https://github.com/apache/spark/pull/42658#issuecomment-1694976628
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
HyukjinKwon commented on PR #42694:
URL: https://github.com/apache/spark/pull/42694#issuecomment-1694971316
But they weren't passed to the remote session before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL
HyukjinKwon commented on PR #42694:
URL: https://github.com/apache/spark/pull/42694#issuecomment-1694971159
Yeah, because the configurations are actually being passed above
(https://github.com/apache/spark/pull/42694/files#diff-68f27a777d79599c1821c967a6c1e7e9ba436296e6800f7361c6f9554cf6b05c
zhengruifeng commented on PR #42694:
URL: https://github.com/apache/spark/pull/42694#issuecomment-1694957561
In local mode, does setting all configurations works before
https://github.com/apache/spark/pull/42548 ?
--
This is an automated message from the Apache Git Service.
To respond to
LuciferYang commented on PR #41673:
URL: https://github.com/apache/spark/pull/41673#issuecomment-1694951106
@dongjoon-hyun Do you have time to look at this pr? I found that if we test
the Yarn module with
`-Dtest.exclude.tags=org.apache.spark.tags.ExtendedLevelDBTest`(like adding
some con
cloud-fan commented on code in PR #42661:
URL: https://github.com/apache/spark/pull/42661#discussion_r1306869761
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/CallMethodViaReflection.scala:
##
@@ -55,64 +55,73 @@ import org.apache.spark.util.Utils
"
LuciferYang opened a new pull request, #42696:
URL: https://github.com/apache/spark/pull/42696
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How
LuciferYang commented on code in PR #42518:
URL: https://github.com/apache/spark/pull/42518#discussion_r1306868324
##
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/client/CheckConnectJvmClientCompatibility.scala:
##
@@ -46,18 +47,30 @@ object CheckConn
yaooqinn commented on PR #42674:
URL: https://github.com/apache/spark/pull/42674#issuecomment-1694942009
> The second example just shows <> which can't the 'right' version of
whatever this is
It's the source code version, the rendered HTML.
> What happened to the garbled text
Hisoka-X commented on code in PR #42661:
URL: https://github.com/apache/spark/pull/42661#discussion_r1306867550
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/CallMethodViaReflection.scala:
##
@@ -55,64 +55,73 @@ import org.apache.spark.util.Utils
""
cloud-fan commented on code in PR #42667:
URL: https://github.com/apache/spark/pull/42667#discussion_r1306867310
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:
##
@@ -420,17 +420,17 @@ class JacksonParser(
case VALUE_STRING if parser.
HyukjinKwon commented on PR #42694:
URL: https://github.com/apache/spark/pull/42694#issuecomment-1694940489
cc @zhengruifeng @michaelzhan-db
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the s
HyukjinKwon commented on PR #42693:
URL: https://github.com/apache/spark/pull/42693#issuecomment-1694940370
cc @zhengruifeng @ueshin @grundprinzip
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to
LuciferYang commented on PR #42236:
URL: https://github.com/apache/spark/pull/42236#issuecomment-1694939498
I will merge this pr today If there are no more comments @rangadi
@HyukjinKwon @hvanhovell @zhengruifeng @rangadi @pan3793 @grundprinzip
@zhenlineo
--
This is an automated messag
HyukjinKwon opened a new pull request, #42695:
URL: https://github.com/apache/spark/pull/42695
### What changes were proposed in this pull request?
This PR proposes to mark all Spark Connect server configurations as static
configurations.
### Why are the changes needed?
cloud-fan commented on code in PR #42661:
URL: https://github.com/apache/spark/pull/42661#discussion_r1306864355
##
sql/core/src/test/resources/sql-tests/inputs/try_reflect.sql:
##
@@ -0,0 +1,19 @@
+-- positive
+SELECT try_reflect("java.util.UUID", "fromString",
"a5cf6c42-0c85-
cloud-fan commented on code in PR #42661:
URL: https://github.com/apache/spark/pull/42661#discussion_r1306863750
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/CallMethodViaReflection.scala:
##
@@ -55,64 +55,73 @@ import org.apache.spark.util.Utils
"
cloud-fan closed pull request #42587: [SPARK-44897][SQL] Propagating local
properties to subquery broadcast exec
URL: https://github.com/apache/spark/pull/42587
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL abov
cloud-fan commented on PR #42587:
URL: https://github.com/apache/spark/pull/42587#issuecomment-1694929266
thanks, merging to master/3.5!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specif
HyukjinKwon closed pull request #42678: [SPARK-44963][PYTHON][ML][TESTS] Make
PySpark (pyspark-ml module) tests passing without any optional dependency
URL: https://github.com/apache/spark/pull/42678
--
This is an automated message from the Apache Git Service.
To respond to the message, pleas
HyukjinKwon commented on PR #42678:
URL: https://github.com/apache/spark/pull/42678#issuecomment-1694928402
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
HyukjinKwon opened a new pull request, #42694:
URL: https://github.com/apache/spark/pull/42694
### What changes were proposed in this pull request?
This PR is a kind of a followup of
https://github.com/apache/spark/pull/42548. This PR proposes to filter static
configurations out in r
srowen commented on PR #42674:
URL: https://github.com/apache/spark/pull/42674#issuecomment-1694922638
I don't quite follow - the error message actually looks as if it's messed up
to begin with and is some bytes being interpreted as Latin-1. The second
example just shows `<>` which can't th
HyukjinKwon opened a new pull request, #42693:
URL: https://github.com/apache/spark/pull/42693
### What changes were proposed in this pull request?
This PR fixes the bug in createDataFrame with Python Spark Connect client.
Now it respects inherited namedtuples as below:
```
beliefer commented on PR #42683:
URL: https://github.com/apache/spark/pull/42683#issuecomment-1694900622
ping @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To
zhengruifeng commented on PR #42680:
URL: https://github.com/apache/spark/pull/42680#issuecomment-1694875018
also cc @ueshin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
srowen commented on PR #42674:
URL: https://github.com/apache/spark/pull/42674#issuecomment-1694870735
OK I believe this, but it's strange, the garbled text you show does not seem
to be the result of double-escaping HTML? You'd find output that reads like
`<` or something
--
This is an a
yaooqinn commented on PR #42674:
URL: https://github.com/apache/spark/pull/42674#issuecomment-1694869333
Hi @srowen, it actually removes double-escaping
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
Hisoka-X commented on PR #42691:
URL: https://github.com/apache/spark/pull/42691#issuecomment-1694856745
Thanks @HyukjinKwon !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment
HyukjinKwon commented on PR #42691:
URL: https://github.com/apache/spark/pull/42691#issuecomment-1694855831
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
HyukjinKwon closed pull request #42691: [SPARK-44978][SQL][TEST] Fix
SQLQueryTestSuite unable create table normally
URL: https://github.com/apache/spark/pull/42691
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL a
Hisoka-X commented on code in PR #42691:
URL: https://github.com/apache/spark/pull/42691#discussion_r1306796272
##
sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala:
##
@@ -689,12 +689,10 @@ class SQLQueryTestSuite extends QueryTest with
SharedSparkSession wi
Hisoka-X commented on PR #41808:
URL: https://github.com/apache/spark/pull/41808#issuecomment-1694846857
@mridulm Thanks for advise! I moved new metrics to the end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
U
HyukjinKwon commented on code in PR #42686:
URL: https://github.com/apache/spark/pull/42686#discussion_r1306790110
##
python/pyspark/sql/streaming/listener.py:
##
@@ -477,7 +477,7 @@ def fromJson(cls, j: Dict[str, Any]) ->
"StreamingQueryProgress":
name=j["name"],
HyukjinKwon commented on code in PR #42691:
URL: https://github.com/apache/spark/pull/42691#discussion_r1306781982
##
sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala:
##
@@ -689,12 +689,10 @@ class SQLQueryTestSuite extends QueryTest with
SharedSparkSession
HyukjinKwon commented on PR #42675:
URL: https://github.com/apache/spark/pull/42675#issuecomment-1694828581
https://github.com/apache/spark/pull/42692
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
HyukjinKwon opened a new pull request, #42692:
URL: https://github.com/apache/spark/pull/42692
This PR cherry-picks https://github.com/apache/spark/pull/42675 to
branch-3.5.
---
### What changes were proposed in this pull request?
This PR proposes to rename tests from fo
github-actions[bot] closed pull request #39819: [SPARK-42252][CORE] Add
`spark.shuffle.localDisk.file.output.buffer` and deprecate
`spark.shuffle.unsafe.file.output.buffer`
URL: https://github.com/apache/spark/pull/39819
--
This is an automated message from the Apache Git Service.
To respond
github-actions[bot] closed pull request #41139: [SPARK-40887][K8S] Make
`SPARK_DRIVER_LOG_URL_` and `SPARK_DRIVER_ATTRIBUTE_` work for Spark on K8S
URL: https://github.com/apache/spark/pull/41139
--
This is an automated message from the Apache Git Service.
To respond to the message, please lo
github-actions[bot] closed pull request #41189: [DO NOT MERGE] [POC] run
foreachBatch() on client
URL: https://github.com/apache/spark/pull/41189
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
github-actions[bot] closed pull request #41071: [SPARK-43391][CORE] Idle
connection should be kept when closeIdleConnection is disabled
URL: https://github.com/apache/spark/pull/41071
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitH
github-actions[bot] closed pull request #41196: [SPARK-43505][K8S] support env
variables substitution and executor library path
URL: https://github.com/apache/spark/pull/41196
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and u
sadikovi commented on code in PR #42667:
URL: https://github.com/apache/spark/pull/42667#discussion_r1306742597
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/BadRecordException.scala:
##
@@ -65,3 +93,25 @@ case class StringAsDataTypeException(
fieldName:
srowen commented on PR #42674:
URL: https://github.com/apache/spark/pull/42674#issuecomment-1694762651
Sorry for the perhaps silly question, but does this mean that some error
messages or traces are displayed without any escaping? or is this only removing
double-escaping?
--
This is an a
sarutak commented on code in PR #42674:
URL: https://github.com/apache/spark/pull/42674#discussion_r1306710065
##
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:
##
@@ -330,24 +328,10 @@ private[ui] class SqlStatsPagedTable(
mridulm commented on PR #42093:
URL: https://github.com/apache/spark/pull/42093#issuecomment-1694726763
Thanks for merging it @yaooqinn !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the speci
mridulm commented on PR #42529:
URL: https://github.com/apache/spark/pull/42529#issuecomment-1694726401
(Sorry for the delay on responding to this)
Just a minor comment, this does look like the right fix.
+CC @tgravescs as well
--
This is an automated message from the Apache Git
mridulm commented on code in PR #42529:
URL: https://github.com/apache/spark/pull/42529#discussion_r1306701075
##
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:
##
@@ -1618,9 +1618,9 @@ private[spark] object Client extends Logging {
retur
mridulm commented on code in PR #42529:
URL: https://github.com/apache/spark/pull/42529#discussion_r1306701075
##
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:
##
@@ -1618,9 +1618,9 @@ private[spark] object Client extends Logging {
retur
mridulm commented on PR #42426:
URL: https://github.com/apache/spark/pull/42426#issuecomment-1694723882
Thanks for clarifying @hdaikoku , let me take a look at this PR this week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub
ChenMichael commented on code in PR #42587:
URL: https://github.com/apache/spark/pull/42587#discussion_r1306699621
##
sql/core/src/test/scala/org/apache/spark/sql/internal/ExecutorSideSQLConfSuite.scala:
##
@@ -191,6 +191,52 @@ class ExecutorSideSQLConfSuite extends SparkFunSuit
mridulm commented on PR #41808:
URL: https://github.com/apache/spark/pull/41808#issuecomment-1694723367
Looking at this more, this should not be that bad an issue as I initially
expected - it will lead to older SHS not seeing the newer metric, but not
result in failures.
`ExecutorMetrics
cloud-fan commented on PR #42660:
URL: https://github.com/apache/spark/pull/42660#issuecomment-1694595440
late LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsub
LuciferYang commented on code in PR #42673:
URL: https://github.com/apache/spark/pull/42673#discussion_r1306613468
##
.github/workflows/build_and_test.yml:
##
@@ -856,7 +864,7 @@ jobs:
- name: Build with SBT
run: |
./dev/change-scala-version.sh 2.13
-
LuciferYang commented on code in PR #42673:
URL: https://github.com/apache/spark/pull/42673#discussion_r1306613214
##
.github/workflows/build_and_test.yml:
##
@@ -856,7 +864,7 @@ jobs:
- name: Build with SBT
run: |
./dev/change-scala-version.sh 2.13
-
LuciferYang commented on PR #42673:
URL: https://github.com/apache/spark/pull/42673#issuecomment-1694590528
Can we try to delete `~/.cache/coursier/v1/https` as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and
LuciferYang commented on PR #38609:
URL: https://github.com/apache/spark/pull/38609#issuecomment-1694589092
What's your compile command? I think it's not related to this PR. I'm using
M2 Max, and I can build successfully with the default profile.I think M2
doesn't need to use `user-defined-
80 matches
Mail list logo