Sean Whitton wrote: > Do you happen to know whether git-annex currently works with youtube-dl > replaced with yt-dlp? Thanks.
Yes, it does, with this git config:
git config annex.youtube-dl-command yt-dlp
If youtube-dl becomes a wrapper script calling yt-dlp, it would also work
without that config.
I've also changed git-annex to fall back to yt-dlp when youtube-dl is
not in path; see attached patch. Also there was a small problem with
parsing yt-dlp's output that prevented git-annex from displaying
progress; see second attached patch.
--
see shy jo
From 5256be61c12fb030fe2eebe2751ee1601a5e7514 Mon Sep 17 00:00:00 2001
From: Joey Hess <jo...@joeyh.name>
Date: Mon, 21 Nov 2022 14:39:26 -0400
Subject: [PATCH] When youtube-dl is not available in PATH, use yt-dlp instead
Debian is going to drop youtube-dl which is not active upstream, and yt-dlp
is the replacement. This will make it be used if youtube-dl gets removed.
If an old version of youtube-dl remains installed, git-annex will still use
it. That might not be desirable, but changing git-annex to use yt-dlp in
preference to youtube-dl when both are installed risks breaking when
the user has annex.youtube-dl-options set to something that is supported
by youtube-dl, but not by yt-dlp.
Sponsored-by: Boyd Stephen Smith Jr. on Patreon
---
Annex/YoutubeDl.hs | 5 +++--
CHANGELOG | 1 +
debian/control | 2 +-
doc/git-annex.mdwn | 7 ++++---
4 files changed, 9 insertions(+), 6 deletions(-)
diff --git a/Annex/YoutubeDl.hs b/Annex/YoutubeDl.hs
index e466f01eba..ba050d4024 100644
--- a/Annex/YoutubeDl.hs
+++ b/Annex/YoutubeDl.hs
@@ -249,8 +249,9 @@ youtubeDlOpts addopts = do
return (opts ++ addopts)
youtubeDlCommand :: Annex String
-youtubeDlCommand = fromMaybe "youtube-dl" . annexYoutubeDlCommand
- <$> Annex.getGitConfig
+youtubeDlCommand = annexYoutubeDlCommand <$> Annex.getGitConfig >>= \case
+ Just c -> pure c
+ Nothing -> fromMaybe "yt-dlp" <$> liftIO (searchPath "youtube-dl")
supportedScheme :: UrlOptions -> URLString -> Bool
supportedScheme uo url = case parseURIRelaxed url of
diff --git a/CHANGELOG b/CHANGELOG
index d193ceca85..78fa628e4f 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -4,6 +4,7 @@ git-annex (10.20221105) UNRELEASED; urgency=medium
* Sped up the initial scan for annexed files by 21%.
* init: Avoid scanning for annexed files, which can be lengthy in a
large repository. Instead that scan is done on demand.
+ * When youtube-dl is not available in PATH, use yt-dlp instead.
-- Joey Hess <i...@joeyh.name> Fri, 18 Nov 2022 12:58:06 -0400
diff --git a/debian/control b/debian/control
index 762052186a..ac28ca029c 100644
--- a/debian/control
+++ b/debian/control
@@ -111,7 +111,7 @@ Recommends:
lsof,
gnupg,
bind9-host,
- youtube-dl,
+ yt-dlp,
git-remote-gcrypt (>= 0.20130908-6),
nocache,
aria2,
diff --git a/doc/git-annex.mdwn b/doc/git-annex.mdwn
index 5eca6600c8..60811dce06 100644
--- a/doc/git-annex.mdwn
+++ b/doc/git-annex.mdwn
@@ -1719,8 +1719,8 @@ Remotes are configured using these settings in `.git/config`.
* `annex.youtube-dl-options`
- Options to pass to youtube-dl when using it to find the url to download
- for a video.
+ Options to pass to youtube-dl (or yt-dlp) when using it to find the url
+ to download for a video.
Some options may break git-annex's integration with youtube-dl. For
example, the --output option could cause it to store files somewhere
@@ -1730,7 +1730,8 @@ Remotes are configured using these settings in `.git/config`.
* `annex.youtube-dl-command`
- Command to run for youtube-dl. Default is "youtube-dl".
+ Command to run for youtube-dl. Default is to use "youtube-dl" or
+ if that is not available in the PATH, to use "yt-dlp".
* `annex.aria-torrent-options`
--
2.38.1
From 43f681d4c15221096975250c0809ded40bf8a5fd Mon Sep 17 00:00:00 2001
From: Joey Hess <jo...@joeyh.name>
Date: Mon, 21 Nov 2022 15:04:36 -0400
Subject: [PATCH 2/2] Support parsing yt-dpl output to display download
progress
Before this fix, no progress was displayed when yt-dpl was used.
Sponsored-by: Graham Spencer on Patreon
---
Annex/YoutubeDl.hs | 11 ++++++++---
CHANGELOG | 1 +
2 files changed, 9 insertions(+), 3 deletions(-)
diff --git a/Annex/YoutubeDl.hs b/Annex/YoutubeDl.hs
index ba050d4024..c6d0f2253e 100644
--- a/Annex/YoutubeDl.hs
+++ b/Annex/YoutubeDl.hs
@@ -265,8 +265,10 @@ supportedScheme uo url = case parseURIRelaxed url of
_ -> allowedScheme uo u
{- Strategy: Look for chunks prefixed with \r, which look approximately
- - like this:
+ - like this for youtube-dl:
- "ESC[K[download] 26.6% of 60.22MiB at 254.69MiB/s ETA 00:00"
+ - or for yt-dlp, like this:
+ - "\r[download] 1.8% of 1.14GiB at 1.04MiB/s ETA 18:23"
- Look at the number before "% of " and the number and unit after,
- to determine the number of bytes.
-}
@@ -292,8 +294,11 @@ parseYoutubeDlProgress = go [] . reverse . progresschunks
calc percent total = round (percent * fromIntegral total / 100)
parsepercent :: String -> Maybe Double
- parsepercent = readMaybe . reverse . takeWhile (not . isSpace) . reverse
+ parsepercent = readMaybe
+ . reverse . takeWhile (not . isSpace) . reverse
+ . dropWhile isSpace
- parsebytes = readSize units . takeWhile (not . isSpace)
+ parsebytes = readSize units . takeWhile (not . isSpace)
+ . dropWhile isSpace
units = committeeUnits ++ storageUnits
diff --git a/CHANGELOG b/CHANGELOG
index 78fa628e4f..78c8abc552 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -5,6 +5,7 @@ git-annex (10.20221105) UNRELEASED; urgency=medium
* init: Avoid scanning for annexed files, which can be lengthy in a
large repository. Instead that scan is done on demand.
* When youtube-dl is not available in PATH, use yt-dlp instead.
+ * Support parsing yt-dpl output to display download progress.
-- Joey Hess <i...@joeyh.name> Fri, 18 Nov 2022 12:58:06 -0400
--
2.38.1
signature.asc
Description: PGP signature
