commits  

Messages by Thread

• [PR] Remove Spark 3.4.4 release from download dropdown list [spark-website] via GitHub
• (spark) branch master updated (443e170397a5 -> 2513f4123539) dongjoon
• (spark) branch master updated (fd5ed10d01dc -> 443e170397a5) dongjoon
• (spark) branch master updated (bd76faf83bff -> fd5ed10d01dc) gurwls223
• (spark) branch master updated: [SPARK-53068][SQL][TESTS] Mark `TransformWith*Suite` as `SlowSQLTest` in `sql/core` dongjoon
• (spark) branch master updated (0225d4cf76cc -> 6e7c67fef7af) dongjoon
• (spark) branch master updated: [SPARK-53001][CORE][SQL][FOLLOW-UP] Disable `spark.memory.unmanagedMemoryPollingInterval` by default dongjoon
• (spark) branch master updated: [SPARK-53035][TESTS][FOLLOWUP] Use `String.repeat` in `tests` too dongjoon
• (spark) branch master updated: [SPARK-52968][SS] Emit additional state store metrics ashrigondekar
• (spark) branch master updated: [SPARK-53061][CORE][SQL] Support `copyFileToDirectory` in `SparkFileUtils` dongjoon
• (spark) branch branch-3.5 updated: [SPARK-53054][CONNECT][3.5] Fix the connect.DataFrameReader default format behavior ruifengz
• (spark) branch master updated: [SPARK-53059][PYTHON] Arrow UDF no need to depend on pandas ruifengz
• (spark) branch master updated (032677cb329d -> 6b0d2ca00121) dongjoon
• (spark) branch master updated: [SPARK-52928][PYTHON][FOLLOW-UP] Remove unreachable code after upgrading pyarrow minimum version to 15.0.0 dongjoon
• (spark) branch master updated (0dd0e022b411 -> 3839fed74c20) wenchen
• (spark) branch branch-4.0 updated: [SPARK-53054][CONNECT] Fix the connect.DataFrameReader default format behavior gurwls223
• (spark) branch master updated (92df232addc5 -> 0dd0e022b411) gurwls223
• [PR] Fix broken Pandas On PySpark link doc link [spark-website] via GitHub
• (spark) branch master updated (fbebc20e61d1 -> 92df232addc5) dongjoon
• (spark) branch master updated (d0a8b18c1adb -> fbebc20e61d1) dongjoon
• (spark) branch master updated (57dfc0a21bf2 -> d0a8b18c1adb) ruifengz
• (spark) branch master updated: [SPARK-53039][PYTHON][TESTS] Add unit test for complex arrow UDF used in window dongjoon
• (spark) branch master updated (8c90e6fa77c4 -> 133d9f60e72f) dongjoon
• (spark) branch master updated (a75e1aa1cac2 -> 8c90e6fa77c4) dongjoon
• (spark) branch master updated: [SPARK-53001] Integrate RocksDB Memory Usage with the Unified Memory Manager ashrigondekar
• (spark) branch master updated (fd17a8d06e23 -> 3024c7c21db8) dongjoon
• (spark) branch master updated (707d9eef6bfa -> fd17a8d06e23) gurwls223
• (spark) branch master updated (c5789579de20 -> 707d9eef6bfa) dongjoon
• (spark) branch master updated: [SPARK-53036][PYTHON][DOCS][TESTS] Enable doctest `pyspark.sql.pandas.functions` gurwls223
• (spark) branch master updated: [SPARK-53043][CORE][SQL][K8S] Use Java `transferTo` instead of `IOUtils.copy` dongjoon
• (spark) branch master updated (e9a285eaf273 -> 9c35c43eaf3b) xinrong
• (spark) branch master updated: Revert "[SPARK-52610][BUILD] Upgrade rocksdbjni to 10.2.1" dongjoon
• (spark) branch master updated: [MINOR][CORE][TESTS] Rename `TestMemoryManager.(markconsequentOOM -> markConsequentOOM)` dongjoon
• (spark) branch master updated (6a6317d6fd2f -> 1ce28ccfb7de) yangjie01
• (spark) branch branch-4.0 updated: [SPARK-53020][DEPLOY][4.0] JPMS args should also apply to non-SparkSubmit process dongjoon
• (spark) branch master updated (293342da1185 -> 6a6317d6fd2f) dongjoon
• (spark) branch master updated: [SPARK-53017][SQL][FOLLOWUP] Add logs in DSv2 Join pushdown rule wenchen
• (spark) branch master updated: [SPARK-52926][SQL] Added SQLMetric for remote schema fetching time duration wenchen
• (spark) branch master updated: [SPARK-53035][CORE][SQL][K8S][MLLIB] Use `String.repeat` instead of Scala string multiplication dongjoon
• (spark) branch master updated (b03a86923c0d -> d892ca40fa66) dongjoon
• (spark) branch master updated: [SPARK-51555][SQL] Add the time_diff() function wenchen
• (spark) branch master updated: [SPARK-52940][PYTHON][TESTS][FOLLOW-UP] Correct the return type in test UDFs gurwls223
• (spark) branch master updated: [MINOR][PYTHON][DOCS] Fix a pandas UDF example gurwls223
• (spark) branch master updated: [SPARK-53031][CORE] Support `getFile` in `SparkFileUtils` dongjoon
• (spark) branch master updated (3738cb3dd701 -> 1f425c7433b4) dongjoon
• (spark) branch master updated (dd6525acd927 -> 3738cb3dd701) dongjoon
• (spark) branch master updated: [SPARK-53023][SQL] Remove `commons-io` dependency from `sql/api` module dongjoon
• (spark) branch master updated: [SPARK-53014][PYTHON][DOCS] Make Arrow UDF public gurwls223
• (spark) branch master updated: [SPARK-53013][PYTHON] Fix Arrow-optimized Python UDTF returning no rows on lateral join gurwls223
• (spark) branch master updated: [SPARK-53024][BUILD] Upgrade `commons-io` to 2.20.0 dongjoon
• (spark) branch master updated: [SPARK-52622][PS] Avoid CAST_INVALID_INPUT of `DataFrame.melt` in ANSI mode xinrong
• (spark) branch master updated (9a1c1ab4710a -> dc8fba647ac1) dongjoon
• (spark) branch master updated: [SPARK-52008] Throwing an error if State Stores do not commit at the end of a batch when ForeachBatch is used ashrigondekar
• (spark) branch master updated: [SPARK-53020][DEPLOY] JPMS args should also apply to non-SparkSubmit process dongjoon
• (spark) branch master updated: [SPARK-52967][BUILD] Upgrade ORC to 2.2.0 dongjoon
• (spark) branch master updated: [SPARK-52954][PYTHON][TESTS][FOLLOW-UP] Alway set safe_check=True in Arrow UDFs gurwls223
• (spark) branch master updated: [SPARK-53018][PYTHON] ArrowStreamArrowUDFSerializer should respect argument arrow_cast gurwls223
• (spark) branch master updated: [SPARK-51554][SQL] Add the time_trunc() function wenchen
• (spark) branch master updated: [SPARK-53003][CORE][FOLLOWUP] Handle null input values dongjoon
• (spark) branch master updated: [SPARK-53004][CORE] Support `abbreviate` in `SparkStringUtils` dongjoon
• (spark) branch master updated: [SPARK-53010][MLLIB][YARN] Ban `com.google.common.base.Strings` dongjoon
• (spark) branch master updated (161f447d6432 -> 783beb648d0e) dongjoon
• (spark) branch master updated (42553b94e368 -> 161f447d6432) dongjoon
• (spark) branch master updated: [SPARK-52992][PYTHON][DOCS] Restore API reference of `pandas_udf` gurwls223
• (spark) branch master updated: [SPARK-52985][PS] Raise TypeError for pandas numpy operand in comparison operators xinrong
• (spark) branch master updated (c3fa01032fcc -> 2d1f77f9f744) xinrong
• (spark) branch master updated (475d72f61c92 -> c3fa01032fcc) dongjoon
• (spark) branch master updated: [SPARK-52999][K8S][TESTS] Clean up the deprecated APIs usage in `kubernetes-integration-tests` module dongjoon
• (spark) branch master updated: [SPARK-52995][YARN] Use Buffered I/O for creating spark jar archive dongjoon
• (spark) branch master updated: [SPARK-52959][PYTHON] Support UDT in Arrow-optimized Python UDTF dongjoon
• (spark) branch master updated: [SPARK-52990][CORE] Support `StringSubstitutor` dongjoon
• (spark) branch master updated: [SPARK-52952][PYTHON] Add PySpark UDF Type Coercion Dev Script gurwls223
• (spark) branch master updated: [SPARK-52889][PYTHON] Implement the current_time function in PySpark ruifengz
• (spark) branch master updated (8274b7c78455 -> 7a6a6e66df20) wenchen
• (spark) branch master updated: [SPARK-52993][BUILD] Bump Snappy 1.1.10.8 dongjoon
• (spark) branch master updated: [SPARK-52977][TESTS] Fix npm vulnerabilities by `npm audit fix` yangjie01
• (spark) branch master updated (8348f2a845f1 -> 633ffe4f3744) ruifengz
• (spark) branch master updated: [SPARK-52983][BUILD] Upgrade Netty to 4.1.123.Final dongjoon
• (spark) branch master updated (c7d780b0bb3f -> 9eee6bf6be85) yao
• (spark) branch master updated: [SPARK-52987][SQL][K8S] Use Java `String.(equals|replace)` method instead of `commons-lang3` dongjoon
• (spark) branch master updated (13c43bc4fd2c -> e65341397b2e) wenchen
• (spark) branch master updated: [SPARK-52890][SPARK-52891][PYTHON] Implement the to_time and try_to_time functions in PySpark gurwls223
• (spark) branch branch-3.5 updated: [SPARK-52945][SQL][TESTS] Split `CastSuiteBase#checkInvalidCastFromNumericType` into three methods and guarantee assertions are valid yangjie01
• (spark) branch branch-4.0 updated: [SPARK-52945][SQL][TESTS] Split `CastSuiteBase#checkInvalidCastFromNumericType` into three methods and guarantee assertions are valid yangjie01
• (spark) branch master updated (2817654d439b -> 9a452f81dbdd) yangjie01
• (spark) branch master updated (cdb4f713402c -> 2817654d439b) gurwls223
• (spark) branch master updated: [SPARK-52888][PYTHON] Implement the make_time function in PySpark gurwls223
• (spark) branch master updated: [SPARK-52973][TESTS] Fix the execution failure of StateStoreBasicOperationsBenchmark yao
• (spark) branch master updated: [SPARK-52689][SQL] Send DML Metrics to V2Write wenchen
• (spark) branch branch-4.0 updated: [SPARK-52146][SQL] Detect cyclic function references in SQL UDFs wenchen
• (spark) branch master updated: [SPARK-52146][SQL] Detect cyclic function references in SQL UDFs wenchen
• (spark) branch master updated: [SPARK-52853][TESTS][FOLLOW-UP] Import SDP module when connect dependencies are available gurwls223
• (spark) branch master updated (0c4a36f392b0 -> dd36f61decd4) gurwls223
• (spark) branch master updated: [SPARK-52954][PYTHON] Arrow UDF support return type coercion ruifengz
• (spark) branch master updated: [SPARK-51415][SQL] Support the time type by make_timestamp() wenchen
• (spark) branch branch-3.5 updated: [SPARK-52944][CORE][SQL][YARN][TESTS][3.5] Fix invalid assertions in tests yangjie01
• (spark) branch master updated (cdc89aea9ac6 -> 4dc3f0fcf987) dongjoon
• (spark) branch master updated: [SPARK-52961][PYTHON] Fix Arrow-optimized Python UDTF with 0-arg eval on lateral join gurwls223
• (spark) branch master updated: [SPARK-52904][PYTHON] Enable convertToArrowArraySafely by default dongjoon
• (spark) branch branch-4.0 updated: [SPARK-52944][CORE][TESTS][FOLLOWUP] Avoid hard-coding the checksum algorithm name yangjie01
• (spark) branch branch-4.0 updated: [SPARK-52944][CORE][SQL][YARN] Fix invalid assertions in tests yangjie01
• (spark) branch branch-4.0 updated (097a26742e87 -> e21749dd36fd) dongjoon
• (spark) branch master updated (a823f95c5220 -> afd595a57f1d) dongjoon
• (spark) branch master updated: [SPARK-52962][SQL] BroadcastExchangeExec should not reset metrics viirya
• (spark) branch branch-3.5 updated: [SPARK-52737][CORE] Pushdown predicate and number of apps to FsHistoryProvider when listing applications yangjie01
• (spark) branch branch-4.0 updated: [SPARK-52737][CORE] Pushdown predicate and number of apps to FsHistoryProvider when listing applications yangjie01
• (spark) branch master updated: [SPARK-52737][CORE] Pushdown predicate and number of apps to FsHistoryProvider when listing applications yangjie01
• (spark-connect-rust) branch master updated: [SPARK-52941] Make GitHub Actions work for spark-connect-rust (#2) liyuanjian
• (spark) branch master updated: [SPARK-52949][PYTHON] Avoid roundtrip between RecordBatch and Table in Arrow-optimized Python UDTF ueshin
• (spark) branch master updated: [SPARK-52141][SQL] Display constraints in DESC commands gengliang
• (spark) branch master updated (d35399acb3d3 -> 90fd991d992b) yangjie01
• (spark) branch master updated: [SPARK-52955] Change return types of WindowResolution.resolveOrder and WindowResolution.resolveFrame to WindowExpression wenchen
• (spark) branch master updated: [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit wenchen
• (spark) branch master updated: [SPARK-50614][FOLLOW-UP] Add assert(false) to test in catch block wenchen
• (spark) branch master updated: [SPARK-52877][PYTHON][FOLLOW-UP] Use columns instead of itercolumns in RecordBatch gurwls223
• (spark) branch master updated (d148e9be24f4 -> 3ae3e344da07) gurwls223
• (spark) branch master updated: [SPARK-52840][PYTHON][DOCS][FOLLOW-UP] Increase Pandas minimum version to 2.2.0 ruifengz
• (spark) branch master updated (d57dc7de62a1 -> e8015e9a89f9) ruifengz
• (spark) branch master updated: [SPARK-52948][PS] Enable test_np_spark_compat_frame under ANSI xinrong
• (spark) branch master updated (eb63949298b3 -> f9347bc18ddf) ruifengz
• (spark) branch branch-4.0 updated: [SPARK-52908][CORE] Prevent for iterator variable name clashing with names of labels in the path to the root of AST wenchen
• (spark) branch master updated: [SPARK-52908][CORE] Prevent for iterator variable name clashing with names of labels in the path to the root of AST wenchen
• (spark) branch master updated: [SPARK-52947][SDP] Fix image path in declarative pipelines programming guide gurwls223
• (spark) branch master updated: [SPARK-50889][CONNECT][TESTS] Fix Flaky Test: `SparkSessionE2ESuite.interrupt operation` (Hang) gurwls223
• (spark) branch master updated (a82b4158d448 -> ff980cc4aefa) gurwls223
• (spark) branch master updated: [SPARK-52946][PYTHON] Fix Arrow-optimized Python UDTF to support large var types ueshin
• (spark) branch master updated: [SPARK-52934][PYTHON] Allow yielding scalar values with Arrow-optimized Python UDTF ueshin
• (spark) branch master updated (dc687d4c83b8 -> acdec9bafb8a) ueshin
• (spark) branch master updated: [SPARK-52853][SDP] Prevent imperative PySpark methods in declarative pipelines sandy
• (spark) branch master updated: [SPARK-52918][SQL][TESTS] Batch JDBC database statements in JDBC suites wenchen
• (spark) branch master updated (0802097c4767 -> 4dc426085d20) yao
• (spark) branch master updated (3bba8c892e66 -> 0802097c4767) ruifengz
• (spark) branch master updated (f003453a6117 -> 3bba8c892e66) wenchen
• (spark) branch master updated: [SPARK-52882][SQL] Implement the current_time function in Scala maxgekk
• (spark) branch master updated: [SPARK-52897][PYTHON] Update `pandas` to 2.3.1 yao
• (spark) branch master updated: [SPARK-7008][INFRA][PS] Upgrade pyarrow to 15.0 in image python-ps-minimum ruifengz
• (spark) branch master updated: [SPARK-52686][SQL][FOLLOWUP] Don't push `Project` through `Union` if there are duplicates in the project list wenchen
• (spark) branch master updated: [SPARK-51505][SQL] Always show empty partition number metrics in AQEShuffleReadExec wenchen
• (spark) branch master updated: [SPARK-52925][SQL] Return correct error message for anchor self references in rCTEs wenchen
• (spark) branch master updated: [SPARK-52709][SQL] Fix parsing of STRUCT<> wenchen
• (spark) branch master updated (40f3ea7c6258 -> 479410594fb5) ruifengz
• (spark) branch master updated (79ba12afdbfe -> 40f3ea7c6258) ruifengz
• (spark) branch master updated (e824f88c40a9 -> 79ba12afdbfe) wenchen
• (spark) branch master updated (634362cbe2d5 -> e824f88c40a9) gurwls223
• (spark) branch branch-4.0 updated: [SPARK-52147][SQL][TESTS] Block temporary object references in persistent SQL UDFs allisonwang
• (spark) branch master updated: [SPARK-52147][SQL][TESTS] Block temporary object references in persistent SQL UDFs allisonwang
• (spark-connect-rust) branch master updated (84db605 -> 257df1c) liyuanjian
  • (spark-connect-rust) 01/01: Merge pull request #1 from sjrusso8/source liyuanjian
• (spark) branch master updated: [SPARK-52914][CORE] Support `On-Demand Log Loading` for rolling logs in `History Server` dongjoon
• (spark) branch master updated (03cb4d9d6874 -> a81d79256027) dongjoon
• (spark) branch master updated: [SPARK-52883][SPARK-52884][SQL] Implement the to_time and try_to_time functions in Scala maxgekk
• (spark) branch master updated (a08d8b093c0e -> e888e37ee2eb) yao
• (spark) branch master updated: [SPARK-47547][CORE] Add `BloomFilter` V2 and use it as default ptoth
• (spark) branch master updated (f34563442a7c -> 23a19e6b5b03) ruifengz
• (spark) branch master updated: [SPARK-52919][SQL] Fix DSv2 Join pushdown to use previously aliased column wenchen
• (spark) branch master updated: [SPARK-52751][PYTHON][CONNECT] Don't eagerly validate column name in `dataframe['col_name']` ruifengz
• (spark) branch branch-4.0 updated (7d112bcecd93 -> 75b081b1703f) maxgekk
• (spark) branch branch-3.5 updated: [SPARK-52791][PS] Fix error when inferring a UDT with a null first element gurwls223
• (spark) branch branch-4.0 updated: [SPARK-52791][PS] Fix error when inferring a UDT with a null first element gurwls223
• (spark) branch branch-4.0 updated: [SPARK-52300][SQL][TEST] Fix invalid AnalysisConfOverrideSuite yangjie01
• (spark) branch master updated: [SPARK-52300][SQL][TEST] Fix invalid AnalysisConfOverrideSuite yangjie01
• (spark) branch master updated (5182eb4c6a51 -> e31ea9f1645f) gurwls223
• (spark) branch master updated (125c79aec851 -> 5182eb4c6a51) gurwls223
• (spark) branch master updated (4de866146228 -> 125c79aec851) gurwls223
• (spark) branch master updated (a8111b222340 -> 4de866146228) gurwls223
• (spark) branch master updated: [SPARK-52875][SQL] Simplify V2 expression translation if the input is context-independent-foldable gengliang
• (spark) branch master updated: [SPARK-52917][SQL] Read support to enable round-trip for binary in xml format dongjoon
• (spark) branch master updated (c2ff983145a4 -> 47b08a0e9588) dongjoon
• (spark) branch master updated (628422027be0 -> c2ff983145a4) dongjoon
• (spark) branch master updated (77dc7f3deb15 -> 628422027be0) wenchen
• (spark) branch master updated: [SPARK-52903][SQL] Trim non-top-level aliases before LCA resolution wenchen
• (spark) branch master updated (1c5908e84639 -> 75721ad9629e) maxgekk
• (spark) branch branch-4.0 updated: [SPARK-50614][FOLLOW-UP] Fix bug where shredded timestamp values did not conform to the Parquet Variant Shredding spec wenchen
• (spark) branch master updated: [SPARK-50614][FOLLOW-UP] Fix bug where shredded timestamp values did not conform to the Parquet Variant Shredding spec wenchen
• (spark) branch master updated: [SPARK-52916][BUILD] Exclude slf4j-simple from SBT yangjie01
• (spark) branch master updated (1a2977e289ac -> 3e0d2ebb8d7d) ptoth
• (spark) branch master updated: [SPARK-52881][SQL] Implement the make_time function in Scala maxgekk
• (spark) branch master updated: [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector wenchen
• (spark) branch master updated: [SPARK-52852][SDP] Remove unused spark_conf in create_streaming_table gurwls223
• (spark) branch master updated (1a8c26c3f67e -> a50fbf76ba56) wenchen
• (spark) branch branch-4.0 updated: [SPARK-52788][SQL][4.0] Fix error of converting binary value in BinaryType to XML yao
• (spark) branch master updated: [SPARK-52829][PYTHON][FOLLOWUP] Remove unnecessary special handling gurwls223
• (spark) branch master updated: [SPARK-52804][BUILD][FOLLOWUP] Revert Java minimum version check for Maven yangjie01
• (spark) branch master updated: [SPARK-52912][CORE] Improve `SparkStringUtils` to support `is(Not)?(Blank|Empty)` dongjoon
• (spark) branch dependabot/npm_and_yarn/ui-test/form-data-4.0.4 deleted (was 62eab002db32) github-bot
• (spark) branch dependabot/npm_and_yarn/ui-test/form-data-4.0.4 created (now 62eab002db32) github-bot
• (spark) branch master updated (39fbf594fa73 -> 7255e2cc8395) gurwls223
• (spark) branch master updated: [SPARK-52448][CONNECT] Add simplified Struct Expression.Literal gurwls223
• (spark) branch master updated (9a60a5408f1d -> 4a4505457544) sandy
• (spark) branch master updated (5d0556bae2c3 -> 9a60a5408f1d) dongjoon
• (spark) branch master updated: [SPARK-52902][K8S] Support `SPARK_VERSION` placeholder in container image names dongjoon
• (spark) branch master updated (0177265b6cb9 -> 27dcbcd4b075) dongjoon
• (spark) branch master updated (689e4580e143 -> 0177265b6cb9) sandy
• (spark) branch master updated: [SPARK-52846][SQL] Add a metric in JDBCRDD for how long it takes to fetch the resultset wenchen
• (spark) branch branch-4.0 updated: [SPARK-52870][SQL] Properly quote variable names in `FOR` statement wenchen
• (spark) branch master updated (f8c2671ada36 -> 386e4646cff4) wenchen
• (spark) branch master updated: [SPARK-52872][SQL][TESTS] Improve test coverage for `HigherOrderFunctions` wenchen
• (spark) branch master updated: [SPARK-52895][SQL] Don't add duplicate elements in `resolveExprsWithAggregate` wenchen
• (spark) branch branch-4.0 updated: [SPARK-52899][SQL] Fix QueryExecutionErrorsSuite test to register H2Dialect back maxgekk
• (spark) branch master updated: [SPARK-52899][SQL] Fix QueryExecutionErrorsSuite test to register H2Dialect back maxgekk

• Earlier messages