Hello community, here is the log from the commit of package you-get for openSUSE:Factory checked in at 2019-03-26 15:40:29 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/you-get (Old) and /work/SRC/openSUSE:Factory/.you-get.new.25356 (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "you-get" Tue Mar 26 15:40:29 2019 rev:20 rq:682985 version:0.4.1270 Changes: -------- --- /work/SRC/openSUSE:Factory/you-get/you-get.changes 2019-02-24 17:17:38.980426512 +0100 +++ /work/SRC/openSUSE:Factory/.you-get.new.25356/you-get.changes 2019-03-26 15:41:38.612270662 +0100 @@ -1,0 +2,5 @@ +Sat Mar 9 08:38:05 UTC 2019 - Luigi Baldoni <[email protected]> + +- Update to version 0.4.1270 (no changelog supplied) + +------------------------------------------------------------------- Old: ---- you-get-0.4.1256.tar.gz New: ---- you-get-0.4.1270.tar.gz ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ you-get.spec ++++++ --- /var/tmp/diff_new_pack.eIyTa0/_old 2019-03-26 15:41:40.040269845 +0100 +++ /var/tmp/diff_new_pack.eIyTa0/_new 2019-03-26 15:41:40.044269843 +0100 @@ -17,7 +17,7 @@ Name: you-get -Version: 0.4.1256 +Version: 0.4.1270 Release: 0 Summary: Dumb downloader that scrapes the web License: MIT ++++++ you-get-0.4.1256.tar.gz -> you-get-0.4.1270.tar.gz ++++++ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/you-get-0.4.1256/.github/ISSUE_TEMPLATE.md new/you-get-0.4.1270/.github/ISSUE_TEMPLATE.md --- old/you-get-0.4.1256/.github/ISSUE_TEMPLATE.md 2019-02-20 16:43:07.000000000 +0100 +++ new/you-get-0.4.1270/.github/ISSUE_TEMPLATE.md 1970-01-01 01:00:00.000000000 +0100 @@ -1,39 +0,0 @@ -Please make sure these boxes are checked before submitting your issue – thank you! - -- [ ] You can actually watch the video in your browser or mobile application, but not download them with `you-get`. -- [ ] Your `you-get` is up-to-date. -- [ ] I have read <https://github.com/soimort/you-get/wiki/FAQ> and tried to do so. -- [ ] The issue is not yet reported on <https://github.com/soimort/you-get/issues> or <https://github.com/soimort/you-get/wiki/Known-Bugs>. If so, please add your comments under the existing issue. -- [ ] The issue (or question) is really about `you-get`, not about some other code or project. - -Run the command with the `--debug` option, and paste the full output inside the fences: - -``` -[PASTE IN ME] -``` - -If there's anything else you would like to say (e.g. in case your issue is not about downloading a specific video; it might as well be a general discussion or proposal for a new feature), fill in the box below; otherwise, you may want to post an emoji or meme instead: - -> [WRITE SOMETHING] -> [OR HAVE SOME :icecream:!] - -汉语翻译最终日期：2016年02月26日 - -在提交前，请确保您已经检查了以下内容! - -- [ ] 你可以在浏览器或移动端中观看视频，但不能使用`you-get`下载. -- [ ] 您的`you-get`为最新版. -- [ ] 我已经阅读并按 <https://github.com/soimort/you-get/wiki/FAQ> 中的指引进行了操作. -- [ ] 您的问题没有在<https://github.com/soimort/you-get/issues> , <https://github.com/soimort/you-get/wiki/FAQ> 或 <https://github.com/soimort/you-get/wiki/Known-Bugs> 报告，否则请在原有issue下报告. -- [ ] 本问题确实关于`you-get`, 而不是其他项目. - -请使用`--debug`运行，并将输出粘贴在下面: - -``` -[在这里粘贴完整日志] -``` - -如果您有其他附言，例如问题只在某个视频发生，或者是一般性讨论或者提出新功能，请在下面添加;或者您可以卖个萌: - -> [您的内容] -> [舔 :icecream:!] diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/you-get-0.4.1256/.github/PULL_REQUEST_TEMPLATE.md new/you-get-0.4.1270/.github/PULL_REQUEST_TEMPLATE.md --- old/you-get-0.4.1256/.github/PULL_REQUEST_TEMPLATE.md 2019-02-20 16:43:07.000000000 +0100 +++ new/you-get-0.4.1270/.github/PULL_REQUEST_TEMPLATE.md 1970-01-01 01:00:00.000000000 +0100 @@ -1,48 +0,0 @@ -**(PLEASE DELETE ALL THESE AFTER READING)** - -Thank you for the pull request! `you-get` is a growing open source project, which would not have been possible without contributors like you. - -Here are some simple rules to follow, please recheck them before sending the pull request: - -- [ ] If you want to propose two or more unrelated patches, please open separate pull requests for them, instead of one; -- [ ] All pull requests should be based upon the latest `develop` branch; -- [ ] Name your branch (from which you will send the pull request) properly; use a meaningful name like `add-this-shining-feature` rather than just `develop`; -- [ ] All commit messages, as well as comments in code, should be written in understandable English. - -As a contributor, you must be aware that - -- [ ] You agree to contribute your code to this project, under the terms of the MIT license, so that any person may freely use or redistribute them; of course, you will still reserve the copyright for your own authorship. -- [ ] You may not contribute any code not authored by yourself, unless they are licensed under either public domain or the MIT license, literally. - -Not all pull requests can eventually be merged. I consider merged / unmerged patches as equally important for the community: as long as you think a patch would be helpful, someone else might find it helpful, too, therefore they could take your fork and benefit in some way. In any case, I would like to thank you in advance for taking your time to contribute to this project. - -Cheers, -Mort - -**(PLEASE REPLACE ALL ABOVE WITH A DETAILED DESCRIPTION OF YOUR PULL REQUEST)** - - -汉语翻译最后日期：2016年02月26日 - -**(阅读后请删除所有内容)** - -感谢您的pull request! `you-get`是稳健成长的开源项目，感谢您的贡献. - -以下简单检查项目望您复查: - -- [ ] 如果您预计提出两个或更多不相关补丁，请为每个使用不同的pull requests，而不是单一; -- [ ] 所有的pull requests应基于最新的`develop`分支; -- [ ] 您预计提出pull requests的分支应有有意义名称，例如`add-this-shining-feature`而不是`develop`; -- [ ] 所有的提交信息与代码中注释应使用可理解的英语. - -作为贡献者，您需要知悉 - -- [ ] 您同意在MIT协议下贡献代码，以便任何人自由使用或分发;当然，你仍旧保留代码的著作权 -- [ ] 你不得贡献非自己编写的代码，除非其属于公有领域或使用MIT协议. - -不是所有的pull requests都会被合并,然而我认为合并/不合并的补丁一样重要：如果您认为补丁重要，其他人也有可能这么认为，那么他们可以从你的fork中提取工作并获益。无论如何，感谢您费心对本项目贡献. - -祝好, -Mort - -**(请将本内容完整替换为PULL REQUEST的详细内容)** diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/you-get-0.4.1256/CONTRIBUTING.md new/you-get-0.4.1270/CONTRIBUTING.md --- old/you-get-0.4.1256/CONTRIBUTING.md 2019-02-20 16:43:07.000000000 +0100 +++ new/you-get-0.4.1270/CONTRIBUTING.md 2019-03-08 23:20:06.000000000 +0100 @@ -1,27 +1,27 @@ -# How to Contribute +# How to Report an Issue -`you-get` is currently experimenting with an aggressive approach to handling issues. Namely, a bug report must be addressed with some code via a pull request. +If you would like to report a problem you find when using `you-get`, please open a [Pull Request](https://github.com/soimort/you-get/pulls), which should include: -## Report a broken extractor +1. A detailed description of the encountered problem; +2. At least one commit, addressing the problem through some unit test(s). + * Examples of good commits: [#2675](https://github.com/soimort/you-get/pull/2675/files), [#2680](https://github.com/soimort/you-get/pull/2680/files), [#2685](https://github.com/soimort/you-get/pull/2685/files) -**How-To:** Please open a new pull request with the following changes: +PRs that fail to meet the above criteria may be closed summarily with no further action. -* Add a new test case in [tests/test.py](https://github.com/soimort/you-get/blob/develop/tests/test.py), with the failing URL(s). +A valid PR will remain open until its addressed problem is fixed. -The Travis CI build will (ideally) fail showing a :x:, which means you have successfully reported a broken extractor. -Such a valid PR will be either *closed* if it's fixed by another PR, or *merged* if it's fixed by follow-up commits from the reporter himself/herself. -## Report other issues / Suggest a new feature +# 如何汇报问题 -**How-To:** Please open a pull request with the proposed changes directly. +为了防止对 GitHub Issues 的滥用，本项目不接受一般的 Issue。 -A valid PR need not be complete (i.e., can be WIP), but it should contain at least one sensible, nontrivial commit. +如您在使用 `you-get` 的过程中发现任何问题，请开启一个 [Pull Request](https://github.com/soimort/you-get/pulls)。该 PR 应当包含： -## Hints +1. 详细的问题描述； +2. 至少一个 commit，其内容是**与问题相关的**单元测试。**不要通过随意修改无关文件的方式来提交 PR！** + * 有效的 commit 示例：[#2675](https://github.com/soimort/you-get/pull/2675/files), [#2680](https://github.com/soimort/you-get/pull/2680/files), [#2685](https://github.com/soimort/you-get/pull/2685/files) -* The [`develop`](https://github.com/soimort/you-get/tree/develop) branch is where your pull request goes. -* Remember to rebase. -* Document your PR clearly, and if applicable, provide some sample links for reviewers to test with. -* Write well-formatted, easy-to-understand commit messages. If you don't know how, look at existing ones. -* We will not ask you to sign a CLA, but you must assure that your code can be legally redistributed (under the terms of the MIT license). +不符合以上条件的 PR 可能被直接关闭。 + +有效的 PR 将会被一直保留，直至相应的问题得以修复。 diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/you-get-0.4.1256/README.md new/you-get-0.4.1270/README.md --- old/you-get-0.4.1256/README.md 2019-02-20 16:43:07.000000000 +0100 +++ new/you-get-0.4.1270/README.md 2019-03-08 23:20:06.000000000 +0100 @@ -4,6 +4,10 @@ [](https://travis-ci.org/soimort/you-get) [](https://gitter.im/soimort/you-get?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) +**NOTICE: Read [this](https://github.com/soimort/you-get/blob/develop/CONTRIBUTING.md) if you are looking for the conventional "Issues" tab.** + +--- + [You-Get](https://you-get.org/) is a tiny command-line utility to download media contents (videos, audios, images) from the Web, in case there is no other handy way to do it. Here's how you use `you-get` to download a video from [YouTube](https://www.youtube.com/watch?v=jNQXAC9IVRw): @@ -448,7 +452,7 @@ If something is broken and `you-get` can't get you things you want, don't panic. (Yes, this happens all the time!) -Check if it's already a known problem on <https://github.com/soimort/you-get/wiki/Known-Bugs>. If not, follow the guidelines on [how to report a broken extractor](https://github.com/soimort/you-get/blob/develop/CONTRIBUTING.md#report-a-broken-extractor). +Check if it's already a known problem on <https://github.com/soimort/you-get/wiki/Known-Bugs>. If not, follow the guidelines on [how to report an issue](https://github.com/soimort/you-get/blob/develop/CONTRIBUTING.md). ## Getting Involved diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/you-get-0.4.1256/src/you_get/common.py new/you-get-0.4.1270/src/you_get/common.py --- old/you-get-0.4.1256/src/you_get/common.py 2019-02-20 16:43:07.000000000 +0100 +++ new/you-get-0.4.1270/src/you_get/common.py 2019-03-08 23:20:06.000000000 +0100 @@ -1277,27 +1277,89 @@ def load_cookies(cookiefile): global cookies - try: - cookies = cookiejar.MozillaCookieJar(cookiefile) - cookies.load() - except Exception: - import sqlite3 + if cookiefile.endswith('.txt'): + # MozillaCookieJar treats prefix '#HttpOnly_' as comments incorrectly! + # do not use its load() + # see also: + # - https://docs.python.org/3/library/http.cookiejar.html#http.cookiejar.MozillaCookieJar + # - https://github.com/python/cpython/blob/4b219ce/Lib/http/cookiejar.py#L2014 + # - https://curl.haxx.se/libcurl/c/CURLOPT_COOKIELIST.html#EXAMPLE + #cookies = cookiejar.MozillaCookieJar(cookiefile) + #cookies.load() + from http.cookiejar import Cookie cookies = cookiejar.MozillaCookieJar() - con = sqlite3.connect(cookiefile) - cur = con.cursor() - try: - cur.execute("""SELECT host, path, isSecure, expiry, name, value - FROM moz_cookies""") - for item in cur.fetchall(): - c = cookiejar.Cookie( - 0, item[4], item[5], None, False, item[0], - item[0].startswith('.'), item[0].startswith('.'), - item[1], False, item[2], item[3], item[3] == '', None, - None, {}, - ) + now = time.time() + ignore_discard, ignore_expires = False, False + with open(cookiefile, 'r') as f: + for line in f: + # last field may be absent, so keep any trailing tab + if line.endswith("\n"): line = line[:-1] + + # skip comments and blank lines XXX what is $ for? + if (line.strip().startswith(("#", "$")) or + line.strip() == ""): + if not line.strip().startswith('#HttpOnly_'): # skip for #HttpOnly_ + continue + + domain, domain_specified, path, secure, expires, name, value = \ + line.split("\t") + secure = (secure == "TRUE") + domain_specified = (domain_specified == "TRUE") + if name == "": + # cookies.txt regards 'Set-Cookie: foo' as a cookie + # with no name, whereas http.cookiejar regards it as a + # cookie with no value. + name = value + value = None + + initial_dot = domain.startswith(".") + if not line.strip().startswith('#HttpOnly_'): # skip for #HttpOnly_ + assert domain_specified == initial_dot + + discard = False + if expires == "": + expires = None + discard = True + + # assume path_specified is false + c = Cookie(0, name, value, + None, False, + domain, domain_specified, initial_dot, + path, False, + secure, + expires, + discard, + None, + None, + {}) + if not ignore_discard and c.discard: + continue + if not ignore_expires and c.is_expired(now): + continue cookies.set_cookie(c) - except Exception: - pass + + elif cookiefile.endswith(('.sqlite', '.sqlite3')): + import sqlite3, shutil, tempfile + temp_dir = tempfile.gettempdir() + temp_cookiefile = os.path.join(temp_dir, 'temp_cookiefile.sqlite') + shutil.copy2(cookiefile, temp_cookiefile) + + cookies = cookiejar.MozillaCookieJar() + con = sqlite3.connect(temp_cookiefile) + cur = con.cursor() + cur.execute("""SELECT host, path, isSecure, expiry, name, value + FROM moz_cookies""") + for item in cur.fetchall(): + c = cookiejar.Cookie( + 0, item[4], item[5], None, False, item[0], + item[0].startswith('.'), item[0].startswith('.'), + item[1], False, item[2], item[3], item[3] == '', None, + None, {}, + ) + cookies.set_cookie(c) + + else: + log.e('[error] unsupported cookies format') # TODO: Chromium Cookies # SELECT host_key, path, secure, expires_utc, name, encrypted_value # FROM cookies diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/you-get-0.4.1256/src/you_get/extractors/bilibili.py new/you-get-0.4.1270/src/you_get/extractors/bilibili.py --- old/you-get-0.4.1256/src/you_get/extractors/bilibili.py 2019-02-20 16:43:07.000000000 +0100 +++ new/you-get-0.4.1270/src/you_get/extractors/bilibili.py 2019-03-08 23:20:06.000000000 +0100 @@ -96,16 +96,16 @@ return 'https://api.live.bilibili.com/room/v1/Room/room_init?id=%s' % room_id @staticmethod - def bilibili_space_channel_api(mid, cid, ps=100): - return 'https://api.bilibili.com/x/space/channel/video?mid=%s&cid=%s&pn=1&ps=%s&order=0&jsonp=jsonp' % (mid, cid, ps) + def bilibili_space_channel_api(mid, cid, pn=1, ps=100): + return 'https://api.bilibili.com/x/space/channel/video?mid=%s&cid=%s&pn=%s&ps=%s&order=0&jsonp=jsonp' % (mid, cid, pn, ps) @staticmethod - def bilibili_space_favlist_api(vmid, fid, ps=100): - return 'https://api.bilibili.com/x/space/fav/arc?vmid=%s&fid=%s&pn=1&ps=%s&order=0&jsonp=jsonp' % (vmid, fid, ps) + def bilibili_space_favlist_api(vmid, fid, pn=1, ps=100): + return 'https://api.bilibili.com/x/space/fav/arc?vmid=%s&fid=%s&pn=%s&ps=%s&order=0&jsonp=jsonp' % (vmid, fid, pn, ps) @staticmethod - def bilibili_space_video_api(mid, ps=100): - return 'https://space.bilibili.com/ajax/member/getSubmitVideos?mid=%s&page=1&pagesize=%s&order=0&jsonp=jsonp' % (mid, ps) + def bilibili_space_video_api(mid, pn=1, ps=100): + return 'https://space.bilibili.com/ajax/member/getSubmitVideos?mid=%s&page=%s&pagesize=%s&order=0&jsonp=jsonp' % (mid, pn, ps) @staticmethod def bilibili_vc_api(video_id): @@ -496,6 +496,8 @@ api_url = self.bilibili_space_channel_api(mid, cid) api_content = get_content(api_url, headers=self.bilibili_headers(referer=self.url)) channel_info = json.loads(api_content) + # TBD: channel of more than 100 videos + epn, i = len(channel_info['data']['list']['archives']), 0 for video in channel_info['data']['list']['archives']: i += 1; log.w('Extracting %s of %s videos ...' % (i, epn)) @@ -508,11 +510,18 @@ api_url = self.bilibili_space_favlist_api(vmid, fid) api_content = get_content(api_url, headers=self.bilibili_headers(referer=self.url)) favlist_info = json.loads(api_content) - epn, i = len(favlist_info['data']['archives']), 0 - for video in favlist_info['data']['archives']: - i += 1; log.w('Extracting %s of %s videos ...' % (i, epn)) - url = 'https://www.bilibili.com/video/av%s' % video['aid'] - self.__class__().download_playlist_by_url(url, **kwargs) + pc = favlist_info['data']['pagecount'] + + for pn in range(1, pc + 1): + api_url = self.bilibili_space_favlist_api(vmid, fid, pn=pn) + api_content = get_content(api_url, headers=self.bilibili_headers(referer=self.url)) + favlist_info = json.loads(api_content) + + epn, i = len(favlist_info['data']['archives']), 0 + for video in favlist_info['data']['archives']: + i += 1; log.w('Extracting %s of %s videos ...' % (i, epn)) + url = 'https://www.bilibili.com/video/av%s' % video['aid'] + self.__class__().download_playlist_by_url(url, **kwargs) elif sort == 'space_video': m = re.match(r'https?://space\.?bilibili\.com/(\d+)/video', self.url) @@ -520,11 +529,18 @@ api_url = self.bilibili_space_video_api(mid) api_content = get_content(api_url, headers=self.bilibili_headers()) videos_info = json.loads(api_content) - epn, i = len(videos_info['data']['vlist']), 0 - for video in videos_info['data']['vlist']: - i += 1; log.w('Extracting %s of %s videos ...' % (i, epn)) - url = 'https://www.bilibili.com/video/av%s' % video['aid'] - self.__class__().download_playlist_by_url(url, **kwargs) + pc = videos_info['data']['pages'] + + for pn in range(1, pc + 1): + api_url = self.bilibili_space_video_api(mid, pn=pn) + api_content = get_content(api_url, headers=self.bilibili_headers()) + videos_info = json.loads(api_content) + + epn, i = len(videos_info['data']['vlist']), 0 + for video in videos_info['data']['vlist']: + i += 1; log.w('Extracting %s of %s videos ...' % (i, epn)) + url = 'https://www.bilibili.com/video/av%s' % video['aid'] + self.__class__().download_playlist_by_url(url, **kwargs) elif sort == 'audio_menu': m = re.match(r'https?://(?:www\.)?bilibili\.com/audio/am(\d+)', self.url) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/you-get-0.4.1256/src/you_get/extractors/google.py new/you-get-0.4.1270/src/you_get/extractors/google.py --- old/you-get-0.4.1256/src/you_get/extractors/google.py 2019-02-20 16:43:07.000000000 +0100 +++ new/you-get-0.4.1270/src/you_get/extractors/google.py 2019-03-08 23:20:06.000000000 +0100 @@ -59,7 +59,7 @@ u = '/'.join(t) real_urls.append(u) if not real_urls: - real_urls = [r1(r'<meta property="og:image" content="([^"]+)', html)] + real_urls = re.findall(r'<meta property="og:image" content="([^"]+)', html) real_urls = [re.sub(r'w\d+-h\d+-p', 's0', u) for u in real_urls] post_date = r1(r'"?(20\d\d[-/]?[01]\d[-/]?[0123]\d)"?', html) post_id = r1(r'/posts/([^"]+)', html) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/you-get-0.4.1256/src/you_get/extractors/pptv.py new/you-get-0.4.1270/src/you_get/extractors/pptv.py --- old/you-get-0.4.1256/src/you_get/extractors/pptv.py 2019-02-20 16:43:07.000000000 +0100 +++ new/you-get-0.4.1270/src/you_get/extractors/pptv.py 2019-03-08 23:20:06.000000000 +0100 @@ -190,7 +190,7 @@ def prepare(self, **kwargs): if self.url and not self.vid: - if not re.match(r'http://v.pptv.com/show/(\w+)\.html', self.url): + if not re.match(r'https?://v.pptv.com/show/(\w+)\.html', self.url): raise('Unknown url pattern') page_content = get_content(self.url,{"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36"}) self.vid = match1(page_content, r'webcfg\s*=\s*{"id":\s*(\d+)') diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/you-get-0.4.1256/src/you_get/extractors/tumblr.py new/you-get-0.4.1270/src/you_get/extractors/tumblr.py --- old/you-get-0.4.1256/src/you_get/extractors/tumblr.py 2019-02-20 16:43:07.000000000 +0100 +++ new/you-get-0.4.1270/src/you_get/extractors/tumblr.py 2019-03-08 23:20:06.000000000 +0100 @@ -49,7 +49,9 @@ tuggles = {} for url in urls: - hd_url = r1(r'(.+)_\d+\.jpg$', url) + '_1280.jpg' # FIXME: decide actual quality + hd_url = r1(r'(.+)_\d+\.jpg$', url) # FIXME: .png and .gif + if hd_url is None: continue + hd_url = hd_url + '_1280.jpg' # FIXME: decide actual quality filename = parse.unquote(hd_url.split('/')[-1]) title = '.'.join(filename.split('.')[:-1]) tumblr_id = r1(r'^tumblr_(.+)_\d+$', title) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/you-get-0.4.1256/src/you_get/extractors/universal.py new/you-get-0.4.1270/src/you_get/extractors/universal.py --- old/you-get-0.4.1256/src/you_get/extractors/universal.py 2019-02-20 16:43:07.000000000 +0100 +++ new/you-get-0.4.1270/src/you_get/extractors/universal.py 2019-03-08 23:20:06.000000000 +0100 @@ -80,6 +80,9 @@ urls += re.findall(r'href="(https?://[^"]+\.png)"', page, re.I) urls += re.findall(r'href="(https?://[^"]+\.gif)"', page, re.I) + # <img> with high widths + urls += re.findall(r'<img src="([^"]*)"[^>]*width="\d\d\d+"', page, re.I) + # relative path rel_urls = [] rel_urls += re.findall(r'href="(\.[^"]+\.jpe?g)"', page, re.I) @@ -101,7 +104,7 @@ for url in set(urls): filename = parse.unquote(url.split('/')[-1]) if 5 <= len(filename) <= 80: - title = '.'.join(filename.split('.')[:-1]) + title = '.'.join(filename.split('.')[:-1]) or filename else: title = '%s' % i i += 1 diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/you-get-0.4.1256/src/you_get/extractors/youtube.py new/you-get-0.4.1270/src/you_get/extractors/youtube.py --- old/you-get-0.4.1256/src/you_get/extractors/youtube.py 2019-02-20 16:43:07.000000000 +0100 +++ new/you-get-0.4.1270/src/you_get/extractors/youtube.py 2019-03-08 23:20:06.000000000 +0100 @@ -8,32 +8,64 @@ class YouTube(VideoExtractor): name = "YouTube" - # YouTube media encoding options, in descending quality order. + # Non-DASH YouTube media encoding options, in descending quality order. # http://en.wikipedia.org/wiki/YouTube#Quality_and_codecs. Retrieved July 17, 2014. stream_types = [ - {'itag': '38', 'container': 'MP4', 'video_resolution': '3072p', 'video_encoding': 'H.264', 'video_profile': 'High', 'video_bitrate': '3.5-5', 'audio_encoding': 'AAC', 'audio_bitrate': '192'}, + {'itag': '38', 'container': 'MP4', 'video_resolution': '3072p', + 'video_encoding': 'H.264', 'video_profile': 'High', 'video_bitrate': '3.5-5', + 'audio_encoding': 'AAC', 'audio_bitrate': '192'}, #{'itag': '85', 'container': 'MP4', 'video_resolution': '1080p', 'video_encoding': 'H.264', 'video_profile': '3D', 'video_bitrate': '3-4', 'audio_encoding': 'AAC', 'audio_bitrate': '192'}, - {'itag': '46', 'container': 'WebM', 'video_resolution': '1080p', 'video_encoding': 'VP8', 'video_profile': '', 'video_bitrate': '', 'audio_encoding': 'Vorbis', 'audio_bitrate': '192'}, - {'itag': '37', 'container': 'MP4', 'video_resolution': '1080p', 'video_encoding': 'H.264', 'video_profile': 'High', 'video_bitrate': '3-4.3', 'audio_encoding': 'AAC', 'audio_bitrate': '192'}, + {'itag': '46', 'container': 'WebM', 'video_resolution': '1080p', + 'video_encoding': 'VP8', 'video_profile': '', 'video_bitrate': '', + 'audio_encoding': 'Vorbis', 'audio_bitrate': '192'}, + {'itag': '37', 'container': 'MP4', 'video_resolution': '1080p', + 'video_encoding': 'H.264', 'video_profile': 'High', 'video_bitrate': '3-4.3', + 'audio_encoding': 'AAC', 'audio_bitrate': '192'}, #{'itag': '102', 'container': 'WebM', 'video_resolution': '720p', 'video_encoding': 'VP8', 'video_profile': '3D', 'video_bitrate': '', 'audio_encoding': 'Vorbis', 'audio_bitrate': '192'}, - {'itag': '45', 'container': 'WebM', 'video_resolution': '720p', 'video_encoding': 'VP8', 'video_profile': '', 'video_bitrate': '2', 'audio_encoding': 'Vorbis', 'audio_bitrate': '192'}, + {'itag': '45', 'container': 'WebM', 'video_resolution': '720p', + 'video_encoding': 'VP8', 'video_profile': '', 'video_bitrate': '2', + 'audio_encoding': 'Vorbis', 'audio_bitrate': '192'}, #{'itag': '84', 'container': 'MP4', 'video_resolution': '720p', 'video_encoding': 'H.264', 'video_profile': '3D', 'video_bitrate': '2-3', 'audio_encoding': 'AAC', 'audio_bitrate': '192'}, - {'itag': '22', 'container': 'MP4', 'video_resolution': '720p', 'video_encoding': 'H.264', 'video_profile': 'High', 'video_bitrate': '2-3', 'audio_encoding': 'AAC', 'audio_bitrate': '192'}, - {'itag': '120', 'container': 'FLV', 'video_resolution': '720p', 'video_encoding': 'H.264', 'video_profile': '[email protected]', 'video_bitrate': '2', 'audio_encoding': 'AAC', 'audio_bitrate': '128'}, # Live streaming only - {'itag': '44', 'container': 'WebM', 'video_resolution': '480p', 'video_encoding': 'VP8', 'video_profile': '', 'video_bitrate': '1', 'audio_encoding': 'Vorbis', 'audio_bitrate': '128'}, - {'itag': '35', 'container': 'FLV', 'video_resolution': '480p', 'video_encoding': 'H.264', 'video_profile': 'Main', 'video_bitrate': '0.8-1', 'audio_encoding': 'AAC', 'audio_bitrate': '128'}, + {'itag': '22', 'container': 'MP4', 'video_resolution': '720p', + 'video_encoding': 'H.264', 'video_profile': 'High', 'video_bitrate': '2-3', + 'audio_encoding': 'AAC', 'audio_bitrate': '192'}, + {'itag': '120', 'container': 'FLV', 'video_resolution': '720p', + 'video_encoding': 'H.264', 'video_profile': '[email protected]', 'video_bitrate': '2', + 'audio_encoding': 'AAC', 'audio_bitrate': '128'}, # Live streaming only + {'itag': '44', 'container': 'WebM', 'video_resolution': '480p', + 'video_encoding': 'VP8', 'video_profile': '', 'video_bitrate': '1', + 'audio_encoding': 'Vorbis', 'audio_bitrate': '128'}, + {'itag': '35', 'container': 'FLV', 'video_resolution': '480p', + 'video_encoding': 'H.264', 'video_profile': 'Main', 'video_bitrate': '0.8-1', + 'audio_encoding': 'AAC', 'audio_bitrate': '128'}, #{'itag': '101', 'container': 'WebM', 'video_resolution': '360p', 'video_encoding': 'VP8', 'video_profile': '3D', 'video_bitrate': '', 'audio_encoding': 'Vorbis', 'audio_bitrate': '192'}, #{'itag': '100', 'container': 'WebM', 'video_resolution': '360p', 'video_encoding': 'VP8', 'video_profile': '3D', 'video_bitrate': '', 'audio_encoding': 'Vorbis', 'audio_bitrate': '128'}, - {'itag': '43', 'container': 'WebM', 'video_resolution': '360p', 'video_encoding': 'VP8', 'video_profile': '', 'video_bitrate': '0.5', 'audio_encoding': 'Vorbis', 'audio_bitrate': '128'}, - {'itag': '34', 'container': 'FLV', 'video_resolution': '360p', 'video_encoding': 'H.264', 'video_profile': 'Main', 'video_bitrate': '0.5', 'audio_encoding': 'AAC', 'audio_bitrate': '128'}, + {'itag': '43', 'container': 'WebM', 'video_resolution': '360p', + 'video_encoding': 'VP8', 'video_profile': '', 'video_bitrate': '0.5', + 'audio_encoding': 'Vorbis', 'audio_bitrate': '128'}, + {'itag': '34', 'container': 'FLV', 'video_resolution': '360p', + 'video_encoding': 'H.264', 'video_profile': 'Main', 'video_bitrate': '0.5', + 'audio_encoding': 'AAC', 'audio_bitrate': '128'}, #{'itag': '82', 'container': 'MP4', 'video_resolution': '360p', 'video_encoding': 'H.264', 'video_profile': '3D', 'video_bitrate': '0.5', 'audio_encoding': 'AAC', 'audio_bitrate': '96'}, - {'itag': '18', 'container': 'MP4', 'video_resolution': '270p/360p', 'video_encoding': 'H.264', 'video_profile': 'Baseline', 'video_bitrate': '0.5', 'audio_encoding': 'AAC', 'audio_bitrate': '96'}, - {'itag': '6', 'container': 'FLV', 'video_resolution': '270p', 'video_encoding': 'Sorenson H.263', 'video_profile': '', 'video_bitrate': '0.8', 'audio_encoding': 'MP3', 'audio_bitrate': '64'}, + {'itag': '18', 'container': 'MP4', 'video_resolution': '360p', + 'video_encoding': 'H.264', 'video_profile': 'Baseline', 'video_bitrate': '0.5', + 'audio_encoding': 'AAC', 'audio_bitrate': '96'}, + {'itag': '6', 'container': 'FLV', 'video_resolution': '270p', + 'video_encoding': 'Sorenson H.263', 'video_profile': '', 'video_bitrate': '0.8', + 'audio_encoding': 'MP3', 'audio_bitrate': '64'}, #{'itag': '83', 'container': 'MP4', 'video_resolution': '240p', 'video_encoding': 'H.264', 'video_profile': '3D', 'video_bitrate': '0.5', 'audio_encoding': 'AAC', 'audio_bitrate': '96'}, - {'itag': '13', 'container': '3GP', 'video_resolution': '', 'video_encoding': 'MPEG-4 Visual', 'video_profile': '', 'video_bitrate': '0.5', 'audio_encoding': 'AAC', 'audio_bitrate': ''}, - {'itag': '5', 'container': 'FLV', 'video_resolution': '240p', 'video_encoding': 'Sorenson H.263', 'video_profile': '', 'video_bitrate': '0.25', 'audio_encoding': 'MP3', 'audio_bitrate': '64'}, - {'itag': '36', 'container': '3GP', 'video_resolution': '240p', 'video_encoding': 'MPEG-4 Visual', 'video_profile': 'Simple', 'video_bitrate': '0.175', 'audio_encoding': 'AAC', 'audio_bitrate': '36'}, - {'itag': '17', 'container': '3GP', 'video_resolution': '144p', 'video_encoding': 'MPEG-4 Visual', 'video_profile': 'Simple', 'video_bitrate': '0.05', 'audio_encoding': 'AAC', 'audio_bitrate': '24'}, + {'itag': '13', 'container': '3GP', 'video_resolution': '', + 'video_encoding': 'MPEG-4 Visual', 'video_profile': '', 'video_bitrate': '0.5', + 'audio_encoding': 'AAC', 'audio_bitrate': ''}, + {'itag': '5', 'container': 'FLV', 'video_resolution': '240p', + 'video_encoding': 'Sorenson H.263', 'video_profile': '', 'video_bitrate': '0.25', + 'audio_encoding': 'MP3', 'audio_bitrate': '64'}, + {'itag': '36', 'container': '3GP', 'video_resolution': '240p', + 'video_encoding': 'MPEG-4 Visual', 'video_profile': 'Simple', 'video_bitrate': '0.175', + 'audio_encoding': 'AAC', 'audio_bitrate': '32'}, + {'itag': '17', 'container': '3GP', 'video_resolution': '144p', + 'video_encoding': 'MPEG-4 Visual', 'video_profile': 'Simple', 'video_bitrate': '0.05', + 'audio_encoding': 'AAC', 'audio_bitrate': '24'}, ] def decipher(js, s): @@ -195,13 +227,29 @@ elif video_info['status'] == ['fail']: if video_info['errorcode'] == ['150']: - video_page = get_content('https://www.youtube.com/watch?v=%s' % self.vid) + if cookies: + # Load necessary cookies into headers (for age-restricted videos) + consent, ssid, hsid, sid = 'YES', '', '', '' + for cookie in cookies: + if cookie.domain.endswith('.youtube.com'): + if cookie.name == 'SSID': + ssid = cookie.value + elif cookie.name == 'HSID': + hsid = cookie.value + elif cookie.name == 'SID': + sid = cookie.value + cookie_str = 'CONSENT=%s; SSID=%s; HSID=%s; SID=%s' % (consent, ssid, hsid, sid) + + video_page = get_content('https://www.youtube.com/watch?v=%s' % self.vid, + headers={'Cookie': cookie_str}) + else: + video_page = get_content('https://www.youtube.com/watch?v=%s' % self.vid) + try: ytplayer_config = json.loads(re.search('ytplayer.config\s*=\s*([^\n]+});ytplayer', video_page).group(1)) except: msg = re.search('class="message">([^<]+)<', video_page).group(1) - log.wtf('[Failed] "%s"' % msg.strip(), exit_code=None) - raise + log.wtf('[Failed] Got message "%s". Try to login with --cookies.' % msg.strip()) if 'title' in ytplayer_config['args']: # 150 Restricted from playback on certain sites @@ -389,7 +437,7 @@ dash_urls = self.__class__.chunk_by_range(dash_url, int(dash_size)) dash_mp4_a_urls = self.__class__.chunk_by_range(dash_mp4_a_url, int(dash_mp4_a_size)) self.dash_streams[itag] = { - 'quality': stream['size'], + 'quality': '%s (%s)' % (stream['size'], stream['quality_label']), 'itag': itag, 'type': mimeType, 'mime': mimeType, @@ -416,7 +464,7 @@ dash_urls = self.__class__.chunk_by_range(dash_url, int(dash_size)) audio_urls = self.__class__.chunk_by_range(audio_url, int(audio_size)) self.dash_streams[itag] = { - 'quality': stream['size'], + 'quality': '%s (%s)' % (stream['size'], stream['quality_label']), 'itag': itag, 'type': mimeType, 'mime': mimeType, diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/you-get-0.4.1256/src/you_get/version.py new/you-get-0.4.1270/src/you_get/version.py --- old/you-get-0.4.1256/src/you_get/version.py 2019-02-20 16:43:07.000000000 +0100 +++ new/you-get-0.4.1270/src/you_get/version.py 2019-03-08 23:20:06.000000000 +0100 @@ -1,4 +1,4 @@ #!/usr/bin/env python script_name = 'you-get' -__version__ = '0.4.1256' +__version__ = '0.4.1270'
