dependabot[bot] opened a new pull request, #3638: URL: https://github.com/apache/iceberg-python/pull/3638
Bumps [ray](https://github.com/ray-project/ray) from 2.55.1 to 2.56.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/ray-project/ray/releases";>ray's releases</a>.</em></p> <blockquote> <h2>Ray-2.56.0</h2> <h1>Highlights</h1> <ul> <li><strong>Ray Data Stability:</strong> In this Ray release, we've added a variety of stability improvements, including running multiple datasets in a cluster, adding automatic batch size selection to CPU-based map-batches, and default logical memory configuration to prevent OOMs. We've also tightened <code>iter_batches</code> stability by reducing hidden buffering and shutting down the executor when consumers exit early (<a href="https://redirect.github.com/ray-project/ray/issues/63660";>#63660</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63682";>#63682</a>, <a href="https://redirect.github.com/ray-project/ray/issues/62949";>#62949</a>). This reduces object-store spilling for common training workloads</li> <li><strong>Ray Serve:</strong> We re-architected Ray Serve LLM by decoupling request handling from token streaming response path (<a href="https://redirect.github.com/ray-project/ray/issues/62667";>#62667</a>, <a href="https://redirect.github.com/ray-project/ray/issues/62680";>#62680</a>, <a href="https://redirect.github.com/ray-project/ray/issues/62668";>#62668</a>, <a href="https://redirect.github.com/ray-project/ray/issues/62669";>#62669</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63167";>#63167</a>), resulting in significant LLM serving performance improvements. We've also introduced new routing policies such as session-sticky routing via consistent hashing with <code>ConsistentHashRouter</code> (<a href="https://redirect.github.com/ray-project/ray/issues/62905";>#62905</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63096";>#63096</a>, <a href="https://redirect.github.com/ray-project/ray/issues/62906";>#62906</a>) and <code>CapacityQueueRouter</ code> (<a href="https://redirect.github.com/ray-project/ray/issues/62323";>#62323</a>) which is beneficial for supply-constrained workloads.</li> <li><strong>Ray Core:</strong> We've added GPU-domain-aware placement groups using label locality (<a href="https://redirect.github.com/ray-project/ray/issues/61442";>#61442</a>, <a href="https://redirect.github.com/ray-project/ray/issues/61614";>#61614</a>, <a href="https://redirect.github.com/ray-project/ray/issues/62487";>#62487</a>, <a href="https://redirect.github.com/ray-project/ray/issues/62533";>#62533</a>). This enables placement groups to pack bundles onto nodes that share a <code>ray.io/gpu-domain</code> label instead of only packing at the single-node level. We've also added initial Kubernetes in-place pod resizing support for Autoscaler v2 (<a href="https://redirect.github.com/ray-project/ray/issues/55961";>#55961</a>, <a href="https://redirect.github.com/ray-project/ray/issues/62369";>#62369</a>, <a href="https://redirect.github.com/ray-project/ray/issues/62215";>#62215</a>), enabling Ray clusters to resize CPU and memory on existing worker pods before scaling out new pods. </li> </ul> <h1>Ray Data</h1> <h3>🎉 New Features</h3> <ul> <li>Support multiple datasets per cluster via subcluster labels and resource partitioning (<a href="https://redirect.github.com/ray-project/ray/issues/63331";>#63331</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63375";>#63375</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63982";>#63982</a>)</li> <li>Add <code>Dataset.mix()</code> public API and <code>MixOperator</code> for weighted dataset mixing (<a href="https://redirect.github.com/ray-project/ray/issues/63168";>#63168</a>, <a href="https://redirect.github.com/ray-project/ray/issues/62450";>#62450</a>)</li> <li>New DataSourceV2 framework: <code>ParquetDatasourceV2</code>, chunked reader, predicate splitting, listing/scanner infra (<a href="https://redirect.github.com/ray-project/ray/issues/63113";>#63113</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63454";>#63454</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63163";>#63163</a>, <a href="https://redirect.github.com/ray-project/ray/issues/62975";>#62975</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63027";>#63027</a>, <a href="https://redirect.github.com/ray-project/ray/issues/62182";>#62182</a>)</li> <li>Add <code>batch_size='auto'</code> to <code>map_batches</code> to derive batch row count from target row batch size (<a href="https://redirect.github.com/ray-project/ray/issues/62648";>#62648</a>)</li> <li>Implement distributed upsert for Iceberg using task-based merge algorithm, preventing performance bottleneck on driver (<a href="https://redirect.github.com/ray-project/ray/issues/63482";>#63482</a>)</li> <li>Add <code>include_row_hash</code> to <code>read_parquet</code> (<a href="https://redirect.github.com/ray-project/ray/issues/61408";>#61408</a>)</li> <li>Add JAX data iterator (<a href="https://redirect.github.com/ray-project/ray/issues/61630";>#61630</a>)</li> <li>Expose flag to run read tasks on isolated worker processes via <code>isolate_read_workers</code> (<a href="https://redirect.github.com/ray-project/ray/issues/63490";>#63490</a>)</li> <li>Expose flag to set default logical memory for map operators via <code>default_map_logical_memory_enabled</code> (<a href="https://redirect.github.com/ray-project/ray/issues/63814";>#63814</a>)</li> <li>Support predicate pushdown for Lance format (<a href="https://redirect.github.com/ray-project/ray/issues/61400";>#61400</a>)</li> <li>Support per-partition <code>start_offset</code> and <code>end_offset</code> for <code>read_kafka</code> (<a href="https://redirect.github.com/ray-project/ray/issues/61620";>#61620</a>)</li> <li>Add obstore async download backend for download operator (<a href="https://redirect.github.com/ray-project/ray/issues/61735";>#61735</a>)</li> <li>Support UDF retries on transient exceptions (<a href="https://redirect.github.com/ray-project/ray/issues/63023";>#63023</a>)</li> </ul> <h3>💫 Enhancements</h3> <ul> <li>Fix <code>iter_batches</code> spilling by replacing <code>make_async_gen</code> with <code>iter_threaded</code> and reducing buffered batches (<a href="https://redirect.github.com/ray-project/ray/issues/63660";>#63660</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63682";>#63682</a>)</li> <li>Gate <code>restore_original_order</code> in <code>iter_batches</code> behind <code>preserve_order</code> (<a href="https://redirect.github.com/ray-project/ray/issues/63792";>#63792</a>)</li> <li>Convert <code>drop_columns</code> to a <code>Project</code> logical operator when input schema is known (<a href="https://redirect.github.com/ray-project/ray/issues/63813";>#63813</a>)</li> <li>Make <code>ConcatAggregation</code> and <code>TurbopufferDatasink</code> use <code>polars</code> for sorting (<a href="https://redirect.github.com/ray-project/ray/issues/61904";>#61904</a>)</li> <li>Boost and vectorize <code>hash_partition</code> with <code>sort_indices</code>, zero-copy slices, and pandas (<a href="https://redirect.github.com/ray-project/ray/issues/63498";>#63498</a>, <a href="https://redirect.github.com/ray-project/ray/issues/62757";>#62757</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63152";>#63152</a>, <a href="https://redirect.github.com/ray-project/ray/issues/62587";>#62587</a>)</li> <li>Enable <code>GPU_SHUFFLE</code> in <code>grouped_data.py</code> (<a href="https://redirect.github.com/ray-project/ray/issues/62410";>#62410</a>)</li> <li>Eager <code>StarExpr</code> expansion, schema inference for non-black-box UDFs, and Expressions struct support (<a href="https://redirect.github.com/ray-project/ray/issues/63776";>#63776</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63387";>#63387</a>, <a href="https://redirect.github.com/ray-project/ray/issues/62560";>#62560</a>)</li> <li>Make logging configurable via <code>RAY_DATA_LOG_LEVEL</code> and log <code>RAY_DATA</code> env vars at execution start (<a href="https://redirect.github.com/ray-project/ray/issues/63487";>#63487</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63380";>#63380</a>)</li> <li>Display and track logical memory in progress bar (<a href="https://redirect.github.com/ray-project/ray/issues/63379";>#63379</a>)</li> <li>Honor <code>compute=</code> in <code>filter(expr=...)</code> and deprecate <code>concurrency=</code> (<a href="https://redirect.github.com/ray-project/ray/issues/63576";>#63576</a>)</li> <li>Enable filter pushdown through <code>StreamingRepartition</code> and read stage column-rename removal (<a href="https://redirect.github.com/ray-project/ray/issues/62347";>#62347</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63384";>#63384</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63582";>#63582</a>)</li> <li>Cache deserialized Arrow schemas in <code>BlockMetadataWithSchema</code> (<a href="https://redirect.github.com/ray-project/ray/issues/63462";>#63462</a>)</li> <li>Track scheduling-loop step duration (p50/p90/max), peak USS/object-store memory, and task block locality (<a href="https://redirect.github.com/ray-project/ray/issues/63586";>#63586</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63345";>#63345</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63489";>#63489</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63418";>#63418</a>, <a href="https://redirect.github.com/ray-project/ray/issues/62249";>#62249</a>)</li> <li>Replace <code>TaskDurationStats</code> and Timer with <code>DistributionTracker</code> (<a href="https://redirect.github.com/ray-project/ray/issues/63488";>#63488</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63530";>#63530</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63825";>#63825</a>)</li> <li>Introduce <code>BlockEntry</code> on <code>RefBundle</code> in place of <code>(ref, metadata)</code> tuples (<a href="https://redirect.github.com/ray-project/ray/issues/63654";>#63654</a>)</li> <li>Pre-resolve filesystem in threaded download to avoid IMDS herd (<a href="https://redirect.github.com/ray-project/ray/issues/62898";>#62898</a>)</li> <li>Convert logical operators to frozen dataclasses and consolidate operator base/repr (<a href="https://redirect.github.com/ray-project/ray/issues/62593";>#62593</a>, <a href="https://redirect.github.com/ray-project/ray/issues/62568";>#62568</a>, <a href="https://redirect.github.com/ray-project/ray/issues/62400";>#62400</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63137";>#63137</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63140";>#63140</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63108";>#63108</a>)</li> <li>Non-blocking default autoscaling coordinator and resource-aware auto-downscaling (<a href="https://redirect.github.com/ray-project/ray/issues/62725";>#62725</a>, <a href="https://redirect.github.com/ray-project/ray/issues/62574";>#62574</a>)</li> <li>Release pinned blocks after dataset execution and shut down executor on early <code>DataIterator</code> exit (<a href="https://redirect.github.com/ray-project/ray/issues/62456";>#62456</a>, <a href="https://redirect.github.com/ray-project/ray/issues/62949";>#62949</a>)</li> <li>Optimize local shuffle with incremental index and configurable compaction threshold (<a href="https://redirect.github.com/ray-project/ray/issues/62539";>#62539</a>)</li> <li>Speed up checkpoint filter and reduce memory usage (<a href="https://redirect.github.com/ray-project/ray/issues/60294";>#60294</a>)</li> <li>Preserve Arrow types through pandas roundtrip and reorder block columns by name before schema ops (<a href="https://redirect.github.com/ray-project/ray/issues/63017";>#63017</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63582";>#63582</a>)</li> <li>Block pickle object columns when reading untrusted Parquet and gate unsafe WebDataset deserialization (<a href="https://redirect.github.com/ray-project/ray/issues/63470";>#63470</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63469";>#63469</a>)</li> <li>Move backpressure escape hatch across all policies (<a href="https://redirect.github.com/ray-project/ray/issues/63539";>#63539</a>)</li> <li>Update <code>pandas</code>, <code>modin</code>, and <code>pyarrow</code> minimum versions (<a href="https://redirect.github.com/ray-project/ray/issues/62899";>#62899</a>)</li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/ray-project/ray/commit/637fd062205393b9e1929996bfe1d49bd3f8469d";><code>637fd06</code></a> Revert "[Data] Remove safe_round from ExecutionResources hot path (<a href="https://redirect.github.com/ray-project/ray/issues/6";>#6</a>… (<a href="https://redirect.github.com/ray-project/ray/issues/64309";>#64309</a>)</li> <li><a href="https://github.com/ray-project/ray/commit/f211d8a84fbb4213052229239f32f76212300834";><code>f211d8a</code></a> [Core] Compute per component memory usage in MiB (<a href="https://redirect.github.com/ray-project/ray/issues/63932";>#63932</a>) (<a href="https://redirect.github.com/ray-project/ray/issues/64042";>#64042</a>)</li> <li><a href="https://github.com/ray-project/ray/commit/f9476b79d32c80ffd137fa6999bd74075d6ed950";><code>f9476b7</code></a> [Dashboard] Implement Frontend UI for Platform Events (<a href="https://redirect.github.com/ray-project/ray/issues/63332";>#63332</a>) (<a href="https://redirect.github.com/ray-project/ray/issues/64026";>#64026</a>)</li> <li><a href="https://github.com/ray-project/ray/commit/6a8600aef2aee7f629d0a9ce548d68ac670d0997";><code>6a8600a</code></a> [dashboard] Fix TPU metrics (<a href="https://redirect.github.com/ray-project/ray/issues/63998";>#63998</a>) (<a href="https://redirect.github.com/ray-project/ray/issues/64025";>#64025</a>)</li> <li><a href="https://github.com/ray-project/ray/commit/7a07ec34153644c9fd556d6e019e91705ca5f82f";><code>7a07ec3</code></a> [data] Rename subcluster label key from <strong>subcluster</strong> to ray-subcluster (<a href="https://redirect.github.com/ray-project/ray/issues/63";>#63</a>...</li> <li><a href="https://github.com/ray-project/ray/commit/5fe0aa1db832b171cbf2eb1e2c07605dc6d542e1";><code>5fe0aa1</code></a> [Data][Cherry-pick] Add default logical memory for map operators (<a href="https://redirect.github.com/ray-project/ray/issues/63814";>#63814</a>) (<a href="https://redirect.github.com/ray-project/ray/issues/6";>#6</a>...</li> <li><a href="https://github.com/ray-project/ray/commit/45194d5f4b2afdfead53dc4e1a7dba2f2205d50f";><code>45194d5</code></a> bumping ray version</li> <li><a href="https://github.com/ray-project/ray/commit/5d2c4e709b2cd30c59d8b50ebd1b8fc69af57292";><code>5d2c4e7</code></a> [data] Support multiple datasets in a cluster (2/2): partition cluster resour...</li> <li><a href="https://github.com/ray-project/ray/commit/7b18437ed00576e162fd6d45dfa50f530759a5d3";><code>7b18437</code></a> [ci][train] adding torchft flaky test job (<a href="https://redirect.github.com/ray-project/ray/issues/63854";>#63854</a>)</li> <li><a href="https://github.com/ray-project/ray/commit/5c3e900a5cd44603bb805696589acef8a067296d";><code>5c3e900</code></a> [Autoscaler] Remove <code>ray.autoscaler</code> docstring ignores from pydoclint (<a href="https://redirect.github.com/ray-project/ray/issues/63571";>#63571</a>)</li> <li>Additional commits viewable in <a href="https://github.com/ray-project/ray/compare/ray-2.55.1...ray-2.56.0";>compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
