Messages by Date
-
2026/11/18
[jira] [Updated] (SPARK-62060) Fix CSV reader re-analyzing every existence default value for each missing column in each row
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-61558) Fix broadcasting the large side of a join under OFFSET when the child has no row count
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62060) Fix CSV reader re-analyzing every existence default value for each missing column in each row
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62059) Bound transpiler early-return normalization and emit the NULL-guard advisory once
Holden Karau (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62058) Log the events of a pipeline run at the exact times of their timestamps, as PySpark's Timestamp.ToDatetime() gives them
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62058) Log the events of a pipeline run at the exact times of their timestamps, as PySpark's Timestamp.ToDatetime() gives them
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62057) Stream-stream left semi join over a streaming union fails with a missing state file when a union input gets its first data after the first micro-batch
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62057) Stream-stream left semi join over a streaming union fails with a missing state file when a union input gets its first data after the first micro-batch
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62057) Stream-stream left semi join over a streaming union fails with a missing state file when a union input gets its first data after the first micro-batch
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62056) Take the arguments of the methods of the maps and repeated containers of the Rust protobuf runtime as upb's, with upb's errors
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62056) Take the arguments of the methods of the maps and repeated containers of the Rust protobuf runtime as upb's, with upb's errors
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62055) Fix vectorized Parquet and ORC readers re-analyzing every existence default value for each column
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62055) Fix vectorized Parquet and ORC readers re-analyzing every existence default value for each column
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-60301) Spark doesn't respect session timezone when casting nested variant to string
Wenchen Fan (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-60301) Spark doesn't respect session timezone when casting nested variant to string
Wenchen Fan (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-60755) Fix codegen compilation failure of `element_at` and array index when the result is not nullable
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-61669) Fix CALL failing with INTERNAL_ERROR when an argument has a Python UDF or a nested IN subquery
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Commented] (SPARK-59718) Add opt-in in-process Python UDF execution via Arrow C Data Interface
Yicong Huang (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-61377) Compute each input column only before the first condition reading it in the subexpression elimination path of FilterExec
Kousuke Saruta (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-61377) Compute each input column only before the first condition reading it in the subexpression elimination path of FilterExec
Kousuke Saruta (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-61685) Add `DataSourceV2Test` test tag
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-61685) Add `DataSourceV2Test` test tag
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62022) NullPointerException for a NULL window watermark column in stream-stream join state format V4
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62038) Stream-stream join state format V4 never evicts rows that an older version's cutoff left in state after a restart
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62054) Fix wrong results and query failures when cost-based join reorder drops or moves join conditions
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62054) Fix wrong results and query failures when cost-based join reorder drops or moves join conditions
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62053) Remove outdated unsupported `ArrayType of TimestampType` note from `spark.sql.execution.arrow.pyspark.enabled` documentation
Zhou Yifan (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62041) Warn of a window in turn with the picklings of UDFs, send the clauses of a merge as they were built, keep the execution info of a failed toPandas, and pickle subquery Columns and Column._expr, as PySpark
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62041) Warn of a window in turn with the picklings of UDFs, send the clauses of a merge as they were built, keep the execution info of a failed toPandas, and pickle subquery Columns and Column._expr, as PySpark
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62047) Pickle the Any and Timestamp messages of the drop-in's protobuf runtime, and fail each request of a user context extension that is not an Any, as PySpark
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62051) Fix wrong results by Dataset.unpivot when an outer join makes a value column nullable
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62051) Fix wrong results by Dataset.unpivot when an outer join makes a value column nullable
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62052) Fix DPP evaluating a pruning key's Python or external UDF twice without pruning any partition
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62051) Fix wrong results by Dataset.unpivot when an outer join makes a value column nullable
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62052) Fix DPP evaluating a pruning key's Python or external UDF twice without pruning any partition
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62051) Fix wrong results by Dataset.unpivot when an outer join makes a value column nullable
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62051) Fix wrong results by Dataset.unpivot when an outer join makes a value column nullable
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-61935) Fix DSv2 writes failing or writing NULL into NOT NULL columns for a Scala UDF with a primitive input
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-61935) Fix DSv2 writes failing or writing NULL into NOT NULL columns for a Scala UDF with a primitive input
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62042) Exclude non-deterministic nested field accesses from nested column aliasing
Xi Liang (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-61301) Improve the FILE type error messages and their builders
Daniel Tenedorio (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62037) Share the options and hooks of a builder's shallow copies, hold PySpark's session and builder locks, let SparkSession, Catalog and DataSourceRegistration be subclassed, and load PySpark's pickles of readers
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62037) Share the options and hooks of a builder's shallow copies, hold PySpark's session and builder locks, let SparkSession, Catalog and DataSourceRegistration be subclassed, and load PySpark's pickles of readers
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62050) Tag more DSv2-only test suites with `DataSourceV2Test`
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62050) Tag more DSv2-only test suites with `DataSourceV2Test`
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62050) Tag more DSv2-only test suites with `DataSourceV2Test`
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-61930) Fix query failures when `ReorderJoin` moves a throwable condition, or one that can fail after a non-deterministic or throwable one, into a lower join
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62049) Use `splitMovable` in `CombineFilters`
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62048) MemoryStore.putIteratorAsBytes leaks the compression stream when unrolling a block fails
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62049) Use `splitMovable` in `CombineFilters`
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62043) Remove test tags that ScalaTest never reads
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62043) Remove test tags that ScalaTest never reads
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-59816) Transpile str upper, lower, startswith, and endswith
Holden Karau (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62032) Operators that pair data with an initial state by key do not check that the two sides' grouping keys have one type
Anil K (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62043) Remove test tags that ScalaTest never reads
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62047) Pickle the Any and Timestamp messages of the drop-in's protobuf runtime, and fail each request of a user context extension that is not an Any, as PySpark
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62013) Make the result objects of a DataFrame PySpark's: the bytes WKB of spatial values, the selfDestruct copy of toPandas, one executionInfo object and the Arrow metadata of _to_table
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62048) MemoryStore.putIteratorAsBytes leaks the compression stream when unrolling a block fails
Ivan Sadikov (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-61656) Fix CALL failing with INTERNAL_ERROR when an argument has an IN subquery
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62040) Improve hypothesis testing by batching examples when it makes sense through df creation
Holden Karau (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62046) Stream-stream outer join never outputs a row whose watermarked equality key is NULL, on the input that has that watermark
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Commented] (SPARK-61382) Stream-stream full outer join joins rows whose equality keys are both NULL
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62046) Stream-stream outer join never outputs a row whose watermarked equality key is NULL, on the input that has that watermark
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62046) Stream-stream outer join never outputs a row whose watermarked equality key is NULL, on the input that has that watermark
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-60533) Make spark.sql.streaming.checkpoint.renamedFileCheck.enabled public and document it for object stores
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-60533) Make spark.sql.streaming.checkpoint.renamedFileCheck.enabled public and document it for object stores
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62045) Driver-side Python workers of a streaming query started through StreamingPythonRunner can't see the session's artifacts
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62042) Exclude non-deterministic nested field accesses from nested column aliasing
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-61930) Fix query failures when `ReorderJoin` moves a throwable condition, or one that can fail after a non-deterministic or throwable one, into a lower join
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62012) Send local rows and pandas string columns of PySpark's large Arrow types under spark.sql.execution.arrow.useLargeVarTypes, and compare the cache threshold with pyarrow's nbytes
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62043) Remove test tags that ScalaTest never reads
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62044) Stream-stream join evicts state bounded through a CAST to TIMESTAMP by the event time, losing matches
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62043) Remove test tags that ScalaTest never reads
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62044) Stream-stream join evicts state bounded through a CAST to TIMESTAMP by the event time, losing matches
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62044) Stream-stream join evicts state bounded through a CAST to TIMESTAMP by the event time, losing matches
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62042) Exclude non-deterministic nested field accesses from nested column aliasing
Xi Liang (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62042) Exclude non-deterministic nested field accesses from nested column aliasing
Xi Liang (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62042) Exclude non-deterministic nested field accesses from nested column aliasing
Xi Liang (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-60979) Fix wrong results by nested schema pruning when column names differ only by case
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-60650) pandas-on-Spark named size aggregation is mapped to SQL size and fails on an integer column
Holden Karau (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-59816) Transpile str upper, lower, startswith, and endswith
Holden Karau (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62023) Make SparkSession.client PySpark's SparkConnectClient: its state per session, a threading.local of live tags and extensions, PySpark's argument conversions and error order in its registrations and executions, and the members PySpark's tests read
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62023) Make SparkSession.client PySpark's SparkConnectClient: its state per session, a threading.local of live tags and extensions, PySpark's argument conversions and error order in its registrations and executions, and the members PySpark's tests read
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-60650) pandas-on-Spark named size aggregation is mapped to SQL size and fails on an integer column
Holden Karau (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-59816) Transpile str upper, lower, startswith, and endswith
Holden Karau (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62039) Split the Rust migration notes into the 4.3 line and the unreleased 4.4 line, and qualify the claim that the Rust API mirrors PySpark's surface
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62039) Split the Rust migration notes into the 4.3 line and the unreleased 4.4 line, and qualify the claim that the Rust API mirrors PySpark's surface
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-61557) Fix CALL failing with IllegalArgumentException when an argument is not foldable
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-59828) Support `skipFixedLenByteArray` in `VectorizedDeltaByteArrayReader`
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62041) Warn of a window in turn with the picklings of UDFs, send the clauses of a merge as they were built, keep the execution info of a failed toPandas, and pickle subquery Columns and Column._expr, as PySpark
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62041) Warn of a window in turn with the picklings of UDFs, send the clauses of a merge as they were built, keep the execution info of a failed toPandas, and pickle subquery Columns and Column._expr, as PySpark
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62017) Report the types, flags, metadata and fields that the array, map, field and struct types hold to Python's garbage collector
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Commented] (SPARK-61833) Set up CI for branch-4.4
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-61833) Set up CI for branch-4.4
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62017) Report the types, flags, metadata and fields that the array, map, field and struct types hold to Python's garbage collector
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62039) Split the Rust migration notes into the 4.3 line and the unreleased 4.4 line, and qualify the claim that the Rust API mirrors PySpark's surface
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62039) Split the Rust migration notes into the 4.3 line and the unreleased 4.4 line, and qualify the claim that the Rust API mirrors PySpark's surface
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62014) Keep the state of Python subclasses of the native ML classes in their pickles and copies, give pyspark.ml.stat PySpark's docs, and make the ML objects' pickles, uids, metadata and attributes PySpark's
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62034) Fix vectorized Parquet/ORC reads of existence defaults with wide decimals or nested NULLs
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-59791) Add optional pre-push checks to the merge script's backports
Holden Karau (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62038) Stream-stream join state format V4 never evicts rows that an older version's cutoff left in state after a restart
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62038) Stream-stream join state format V4 never evicts rows that an older version's cutoff left in state after a restart
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-59791) Add optional pre-push checks to the merge script's backports
Holden Karau (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62037) Share the options and hooks of a builder's shallow copies, hold PySpark's session and builder locks, let SparkSession, Catalog and DataSourceRegistration be subclassed, and load PySpark's pickles of readers
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-59791) Add optional pre-push checks to the merge script's backports
Holden Karau (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62037) Share the options and hooks of a builder's shallow copies, hold PySpark's session and builder locks, let SparkSession, Catalog and DataSourceRegistration be subclassed, and load PySpark's pickles of readers
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62036) Support pinning a SparkApplication attempt to a single topology zone
Alex Zhilko (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-60465) Late-materialization storage-filter pushdown for data source V2 scans
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62021) Make the relation of df._plan.plan(client) hold the relation of the DataFrame, built at the call, so that it is sent after the DataFrame is gone, as PySpark's
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-60541) Per-partition runtime bloom filters for storage-partitioned joins
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-61826) Projects generated by spark-pipelines init lose their tables after the run and fail the second run with LOCATION_ALREADY_EXISTS
Sudhanva Huruli (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62036) Support pinning a SparkApplication attempt to a single topology zone
Alex Zhilko (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62021) Make the relation of df._plan.plan(client) hold the relation of the DataFrame, built at the call, so that it is sent after the DataFrame is gone, as PySpark's
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-61534) Estimate the size of an existence join from its left side
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62022) NullPointerException for a NULL window watermark column in stream-stream join state format V4
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62035) Add spark.sql.execution.maxScanBytes to cancel queries that read too much data
Gengliang Wang (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62035) Add spark.sql.execution.maxScanBytes to cancel queries that read too much data
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62033) COMMENT ON TABLE clears the default collation, and ALTER TABLE ... DEFAULT COLLATION clears the comment, on the session catalog
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62034) Fix vectorized Parquet/ORC reads of existence defaults with wide decimals or nested NULLs
Daniel Tenedorio (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62017) Report the types, flags, metadata and fields that the array, map, field and struct types hold to Python's garbage collector
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Commented] (SPARK-61654) Place a cogroup key stored like the layout's column by the column's own transform in the one-side shuffle
Anil K (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-60978) Fix wrong results by DSv2 DELETE, UPDATE and MERGE when column names differ only by case
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62007) Make patches of pyspark.sql.DataFrame reach native frames as PySpark's, take PySpark's Arrow path for pandas data of struct subclasses and NUL names, and fail names that UTF-8 cannot encode at the request
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62019) Compare the messages of the Rust protobuf runtime as upb's: floats of their bits, unknown fields of their contents, maps as mappings, and take only protobuf's messages of its default pool
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62018) Keep in the type stubs the docstrings that PySpark's objects have at run time, so that editors show PySpark's documentation as with pyspark-client
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62016) Treat user-defined type objects as PySpark does: keep them in the types that hold them until a use needs their JSON, make new ones for the schemas of the server, and render casts by their simpleString()
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62018) Keep in the type stubs the docstrings that PySpark's objects have at run time, so that editors show PySpark's documentation as with pyspark-client
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62019) Compare the messages of the Rust protobuf runtime as upb's: floats of their bits, unknown fields of their contents, maps as mappings, and take only protobuf's messages of its default pool
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62016) Treat user-defined type objects as PySpark does: keep them in the types that hold them until a use needs their JSON, make new ones for the schemas of the server, and render casts by their simpleString()
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62030) Connect directly to the hosts that no_proxy excludes despite an invalid proxy, end a connection whose server sends no HTTP/2 settings in the connect deadline, and write grpcio's errors of the proxy and the TLS roots to stderr
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62030) Connect directly to the hosts that no_proxy excludes despite an invalid proxy, end a connection whose server sends no HTTP/2 settings in the connect deadline, and write grpcio's errors of the proxy and the TLS roots to stderr
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Comment Edited] (SPARK-61654) Place a cogroup key stored like the layout's column by the column's own transform in the one-side shuffle
Anil K (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62033) COMMENT ON TABLE clears the default collation, and ALTER TABLE ... DEFAULT COLLATION clears the comment, on the session catalog
Sudhanva Huruli (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62007) Make patches of pyspark.sql.DataFrame reach native frames as PySpark's, take PySpark's Arrow path for pandas data of struct subclasses and NUL names, and fail names that UTF-8 cannot encode at the request
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Commented] (SPARK-60733) Tell a native data source library to stop a read once Spark closes its reader
Gengliang Wang (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62031) Free the last batch of a native data source scan that stops early without waiting for a GC
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-61301) Improve the FILE type error messages and their builders
Daniel Tenedorio (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-61301) Improve the FILE type error messages and their builders
Daniel Tenedorio (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62016) Treat user-defined type objects as PySpark does: keep them in the types that hold them until a use needs their JSON, make new ones for the schemas of the server, and render casts by their simpleString()
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-60541) Per-partition runtime bloom filters for storage-partitioned joins
Peter Toth (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-60541) Per-partition runtime bloom filters for storage-partitioned joins
Peter Toth (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62030) Connect directly to the hosts that no_proxy excludes despite an invalid proxy, end a connection whose server sends no HTTP/2 settings in the connect deadline, and write grpcio's errors of the proxy and the TLS roots to stderr
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62031) Free the last batch of a native data source scan that stops early without waiting for a GC
Gengliang Wang (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62030) Connect directly to the hosts that no_proxy excludes despite an invalid proxy, end a connection whose server sends no HTTP/2 settings in the connect deadline, and write grpcio's errors of the proxy and the TLS roots to stderr
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62025) Streaming table that reads a materialized view from the same pipeline duplicates rows on every run
Sudhanva Huruli (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62026) A pipeline silently replaces a materialized view created by a different pipeline with the same table name
Sudhanva Huruli (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62028) Re-running a pipeline fails with DELTA_CANNOT_CHANGE_PROVIDER for an existing streaming table declared with format delta
Sudhanva Huruli (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62026) A pipeline silently replaces a materialized view created by a different pipeline with the same table name
Sudhanva Huruli (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-61895) Load the SRS registry PySpark ships in GeometryType and GeographyType
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-61999) Keep the metadata of data types in their copies and pickles, warn of PySpark's modules, and take PySpark's order of the errors of repartition and the schema of a failed plan
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62029) Full refresh of Iceberg tables fails with 'Cannot process delete snapshot' and column type changes are rejected
Sudhanva Huruli (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62027) spark-submit in Spark Connect mode drops settings beyond the first group of ten, such as spark.jars
Sudhanva Huruli (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62011) Make type checkers see the messages of pyspark.sql.connect.proto, the Avro and Protobuf functions of Spark Connect, the classes of pyspark.ml.connect and PySpark's classic SQLContext, as with pyspark-client
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62004) Pickle DataFrames of failing plans and table arguments, keep the schema that a copy requests, rebuild same-session deep copies, raise frame-bound errors at the build, and report Column._expr to Python's GC, as PySpark
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62004) Pickle DataFrames of failing plans and table arguments, keep the schema that a copy requests, rebuild same-session deep copies, raise frame-bound errors at the build, and report Column._expr to Python's GC, as PySpark
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62024) Declarative Pipelines: a second run of a table with table properties fails in a JDBC catalog
Sudhanva Huruli (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-61999) Keep the metadata of data types in their copies and pickles, warn of PySpark's modules, and take PySpark's order of the errors of repartition and the schema of a failed plan
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62021) Make the relation of df._plan.plan(client) hold the relation of the DataFrame, built at the call, so that it is sent after the DataFrame is gone, as PySpark's
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62023) Make SparkSession.client PySpark's SparkConnectClient: its state per session, a threading.local of live tags and extensions, PySpark's argument conversions and error order in its registrations and executions, and the members PySpark's tests read
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62023) Make SparkSession.client PySpark's SparkConnectClient: its state per session, a threading.local of live tags and extensions, PySpark's argument conversions and error order in its registrations and executions, and the members PySpark's tests read
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-61930) Fix query failures when `ReorderJoin` moves a throwable condition, or one that can fail after a non-deterministic or throwable one, into a lower join
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-61930) Fix query failures when `ReorderJoin` moves a throwable condition, or one that can fail after a non-deterministic or throwable one, into a lower join
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-61992) Take the map fields of a protobuf message's constructor as upb's InitMapAttributes, and make the runtime's DecodeError a subclass of protobuf's
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62022) NullPointerException for a NULL window watermark column in stream-stream join state format V4
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62021) Make the relation of df._plan.plan(client) hold the relation of the DataFrame, built at the call, so that it is sent after the DataFrame is gone, as PySpark's
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62011) Make type checkers see the messages of pyspark.sql.connect.proto, the Avro and Protobuf functions of Spark Connect, the classes of pyspark.ml.connect and PySpark's classic SQLContext, as with pyspark-client
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-61975) Emit PySpark's RETRIES_EXCEEDED warning, log stop's release failure through the connect logger, render gRPC failures as grpcio does, and send PySpark's requests for the first command and the registrations
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Reopened] (SPARK-61930) Fix query failures when `ReorderJoin` moves a throwable condition, or one that can fail after a non-deterministic or throwable one, into a lower join
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-61930) Fix query failures when `ReorderJoin` moves a throwable condition, or one that can fail after a non-deterministic or throwable one, into a lower join
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-61930) Fix query failures when `ReorderJoin` moves a throwable condition, or one that can fail after a non-deterministic or throwable one, into a lower join
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62020) spark-pipelines run names a non-existent --full_refresh option in the CONFLICTING_PIPELINE_REFRESH_OPTIONS error
Sudhanva Huruli (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-61930) Fix query failures when `ReorderJoin` moves a throwable condition, or one that can fail after a non-deterministic or throwable one, into a lower join
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-61927) Fix CHECK constraint failures on rows that are not written
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-61934) Fix wrong results by DSv2 MERGE when a Scala UDF with a primitive input gets NULL
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-61934) Fix wrong results by DSv2 MERGE when a Scala UDF with a primitive input gets NULL
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62015) Convert the local rows of createDataFrame with the type objects of the schema, inferred too, and read tuple rows by their len and iter, as PySpark does
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62015) Convert the local rows of createDataFrame with the type objects of the schema, inferred too, and read tuple rows by their len and iter, as PySpark does
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62002) Stream-stream join loses matches when an equality key on event time is shifted backward or truncated on one side
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62014) Keep the state of Python subclasses of the native ML classes in their pickles and copies, give pyspark.ml.stat PySpark's docs, and make the ML objects' pickles, uids, metadata and attributes PySpark's
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62014) Keep the state of Python subclasses of the native ML classes in their pickles and copies, give pyspark.ml.stat PySpark's docs, and make the ML objects' pickles, uids, metadata and attributes PySpark's
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-61927) Fix CHECK constraint failures on rows that are not written
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-61921) Fix query failures when an inferred filter reads a column that can fail through a limit, a sort, a join or an aggregate
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-61927) Fix CHECK constraint failures on rows that are not written
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-61538) Support special datetime values in casts to nanosecond timestamps
Uroš Bojanić (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62018) Keep in the type stubs the docstrings that PySpark's objects have at run time, so that editors show PySpark's documentation as with pyspark-client
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-61927) Fix CHECK constraint failures on rows that are not written
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-61538) Support special datetime values in casts to nanosecond timestamps
Uroš Bojanić (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-61538) Support special datetime values in casts to nanosecond timestamps
Uroš Bojanić (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62019) Compare the messages of the Rust protobuf runtime as upb's: floats of their bits, unknown fields of their contents, maps as mappings, and take only protobuf's messages of its default pool
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62019) Compare the messages of the Rust protobuf runtime as upb's: floats of their bits, unknown fields of their contents, maps as mappings, and take only protobuf's messages of its default pool
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62012) Send local rows and pandas string columns of PySpark's large Arrow types under spark.sql.execution.arrow.useLargeVarTypes, and compare the cache threshold with pyarrow's nbytes
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-61649) Preserve structured JVM linkage error details
Uroš Bojanić (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-61649) Preserve structured JVM linkage error details
Uroš Bojanić (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62013) Make the result objects of a DataFrame PySpark's: the bytes WKB of spatial values, the selfDestruct copy of toPandas, one executionInfo object and the Arrow metadata of _to_table
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-61649) Preserve structured JVM linkage error details
Uroš Bojanić (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-61931) SPIP: Add the OBJECT data type
Serge Rielau (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62018) Keep in the type stubs the docstrings that PySpark's objects have at run time, so that editors show PySpark's documentation as with pyspark-client
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-61571) Patch remaining Python version checks to enforce 3.11 minimum
Nicholas Chammas (Jira) via issues
-
2026/11/18
[jira] [Commented] (SPARK-62006) Preserve the output ordering of full outer sort merge JOIN
Saath (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-61922) Fix DSv2 generated column writes reading the generation expression with the TIME type disabled
Dongjoon Hyun (Jira) via issues