dependabot[bot] opened a new pull request, #4254: URL: https://github.com/apache/iceberg-python/pull/4254
Bumps [huggingface-hub](https://github.com/huggingface/huggingface_hub) from 2.0.0 to 2.3.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/huggingface/huggingface_hub/releases";>huggingface-hub's releases</a>.</em></p> <blockquote> <h2>[v2.3.0] Bucket live-follow, faster <code>HfFileSystem</code> listings and more</h2> <h2>🔄 Keep Bucket listings fresh with <code>live_follow=True</code></h2> <p><code>HfFileSystem</code> caches directory listings, so files added to a Bucket by another process did not show up until you called <code>invalidate_cache()</code>. With <code>live_follow=True</code>, <code>HfFileSystem</code> opens one background connection per listed Bucket to receive its file changes from the Hub. When a change arrives, it removes the cached listings it affects.</p> <pre lang="python"><code>>>> from huggingface_hub import HfFileSystem >>> hffs = HfFileSystem(live_follow=True) >>> hffs.ls("buckets/my-username/my-bucket/data", detail=False) # starts following that bucket >>> # ... another process uploads a file to the bucket ... >>> hffs.ls("buckets/my-username/my-bucket/data", detail=False) # refreshed as changes arrive ['buckets/my-username/my-bucket/data/train.csv', 'buckets/my-username/my-bucket/data/new-file.csv'] </code></pre> <p>It also works through fsspec with <code>fsspec.filesystem("hf", live_follow=True)</code>, and through the <code>storage_options</code> argument of libraries such as pandas and pyarrow. The option is off by default because it keeps a connection open for each followed Bucket. It only affects listings and metadata (<code>ls</code>, <code>info</code>, <code>exists</code>, <code>glob</code>, ...), because reading a file always fetches its content from the Hub. It is not available for model, dataset and Space repos: use <code>refresh=True</code> to see a new commit.</p> <ul> <li>[HfFileSystem] Add live-follow for Buckets by <a href="https://github.com/lhoestq";><code>@​lhoestq</code></a> in <a href="https://redirect.github.com/huggingface/huggingface_hub/issues/5096";>#5096</a></li> </ul> <h2>🚀 <code>HfFileSystem</code> lists large repos in parallel</h2> <p>Recursive listings such as <code>find</code> and <code>glob</code> used to be slow on large repos. The Hub returns a repo's file tree 1000 entries at a time, and each page can only be requested after the previous one has arrived. Listing <code>datasets/allenai/objaverse</code> (~800k files) took ~800 requests in a row, about 13 minutes.</p> <p><code>HfFileSystem</code> now splits recursive listings by sub-directory and lists them in parallel, with almost the same number of requests. A sub-tree that fits in a single page is never split. <code>maxdepth</code> is now applied during the listing, so the Hub only returns the levels you ask for.</p> <table> <thead> <tr> <th><code>datasets/allenai/objaverse</code></th> <th>before</th> <th>after</th> </tr> </thead> <tbody> <tr> <td><code>find()</code>, 798,924 files</td> <td>801 requests, 776 s</td> <td>807 requests, 19.7 s</td> </tr> <tr> <td><code>find(maxdepth=2)</code>, 328 entries</td> <td>801 requests, 776 s</td> <td>6 requests, 1.5 s</td> </tr> </tbody> </table> <pre lang="python"><code>>>> from huggingface_hub import hffs >>> files = hffs.glob("datasets/allenai/objaverse/**/*.glb", maxdepth=3) </code></pre> <p>The new <code>max_workers</code> argument of <code>HfFileSystem</code> sets the number of concurrent requests. It defaults to 16. Set it to 1 to list sequentially like before.</p> <ul> <li>[HfFileSystem] List large repos in parallel by <a href="https://github.com/lhoestq";><code>@​lhoestq</code></a> in <a href="https://redirect.github.com/huggingface/huggingface_hub/issues/5153";>#5153</a></li> </ul> <h2>📎 Attach images and videos to discussions and PRs</h2> <p>The new <code>HfApi.upload_attachment(path)</code> uploads an image, video or audio file and returns its URL, which you can embed in a discussion, PR description or comment. Before, this required calling an undocumented Hub endpoint by hand.</p> <pre lang="python"><code>>>> from huggingface_hub import HfApi >>> api = HfApi() >>> url = api.upload_attachment("screenshot.png") >>> api.comment_discussion("username/repo-name", 5, f"Here is the fix:\n\n") </code></pre> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/huggingface/huggingface_hub/commit/587e09a6b6a8ba2e249fad1d4e9d9711989e544f";><code>587e09a</code></a> Release: v2.3.0</li> <li><a href="https://github.com/huggingface/huggingface_hub/commit/ac4a5f0934005600b2e42d4e394a9555ee38956d";><code>ac4a5f0</code></a> Release: v2.3.0.rc1</li> <li><a href="https://github.com/huggingface/huggingface_hub/commit/64e9850f90aed129f2d2221e1c71374314f5aea7";><code>64e9850</code></a> [HfFileSystem] Fix glob crash when the parent listing is outdated (<a href="https://redirect.github.com/huggingface/huggingface_hub/issues/5237";>#5237</a>)</li> <li><a href="https://github.com/huggingface/huggingface_hub/commit/b1066db38c7269ce9c3c3597246623fa0292e3b4";><code>b1066db</code></a> Release: v2.3.0.rc0</li> <li><a href="https://github.com/huggingface/huggingface_hub/commit/0c7fd06f0bfd8b05976f267ed5cd87d8038cecc5";><code>0c7fd06</code></a> [HfApi] Support gated repos hiding requester username/email (<a href="https://redirect.github.com/huggingface/huggingface_hub/issues/5047";>#5047</a>)</li> <li><a href="https://github.com/huggingface/huggingface_hub/commit/c629fa3418a8615ba8667d60ed46aa4d1921e0d7";><code>c629fa3</code></a> [Jobs] Fix hf jobs run losing logs and ignoring Ctrl+C (<a href="https://redirect.github.com/huggingface/huggingface_hub/issues/5162";>#5162</a>)</li> <li><a href="https://github.com/huggingface/huggingface_hub/commit/785868010b6fa75d75feb00fe45f9b1bca72775e";><code>7858680</code></a> [HfFileSystem] List large repos in parallel (<a href="https://redirect.github.com/huggingface/huggingface_hub/issues/5153";>#5153</a>)</li> <li><a href="https://github.com/huggingface/huggingface_hub/commit/75fbd6d83f517101eaf8c6cdb567823041e41a87";><code>75fbd6d</code></a> fix(kernels): allow uploads and commits to kernel repos (<a href="https://redirect.github.com/huggingface/huggingface_hub/issues/5233";>#5233</a>)</li> <li><a href="https://github.com/huggingface/huggingface_hub/commit/e1f595dd7db4c993837e320a6d62ccdd1717dc2d";><code>e1f595d</code></a> [CLI] Show the Job's scheduling status in hf jobs run (<a href="https://redirect.github.com/huggingface/huggingface_hub/issues/5163";>#5163</a>)</li> <li><a href="https://github.com/huggingface/huggingface_hub/commit/d03a05c1ee023fdca35e68c78ad88bc67010ea8e";><code>d03a05c</code></a> Bump the actions group with 3 updates (<a href="https://redirect.github.com/huggingface/huggingface_hub/issues/5228";>#5228</a>)</li> <li>Additional commits viewable in <a href="https://github.com/huggingface/huggingface_hub/compare/v2.0.0...v2.3.0";>compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
