Hello community, here is the log from the commit of package python-fanficfare for openSUSE:Factory checked in at 2019-03-26 22:31:31 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/python-fanficfare (Old) and /work/SRC/openSUSE:Factory/.python-fanficfare.new.25356 (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "python-fanficfare" Tue Mar 26 22:31:31 2019 rev:6 rq:686456 version:3.6.0 Changes: -------- --- /work/SRC/openSUSE:Factory/python-fanficfare/python-fanficfare.changes 2019-03-08 11:02:45.356527842 +0100 +++ /work/SRC/openSUSE:Factory/.python-fanficfare.new.25356/python-fanficfare.changes 2019-03-26 22:31:45.241715462 +0100 @@ -1,0 +2,8 @@ +Tue Mar 19 15:20:41 UTC 2019 - Tomáš Chvátal <[email protected]> + +- Update to 3.6.0: + * Install attached plugin zip file, or use Calibre's 'Get plugins' feature. +- Use dos2unix for file conversions +- Add missing runtime dependencies + +------------------------------------------------------------------- Old: ---- FanFicFare-3.5.0.tar.gz New: ---- FanFicFare-3.6.0.tar.gz ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ python-fanficfare.spec ++++++ --- /var/tmp/diff_new_pack.IHLXlg/_old 2019-03-26 22:31:46.297715208 +0100 +++ /var/tmp/diff_new_pack.IHLXlg/_new 2019-03-26 22:31:46.297715208 +0100 @@ -20,7 +20,7 @@ %define modnamedown fanficfare %{?!python_module:%define python_module() python-%{**} python3-%{**}} Name: python-fanficfare -Version: 3.5.0 +Version: 3.6.0 Release: 0 Summary: Tool for making eBooks from stories on fanfiction and other web sites License: GPL-3.0-only @@ -32,8 +32,13 @@ BuildRequires: %{python_module html2text} BuildRequires: %{python_module html5lib} BuildRequires: %{python_module setuptools >= 17.1} +BuildRequires: dos2unix BuildRequires: fdupes BuildRequires: python-rpm-macros +Requires: python-beautifulsoup4 +Requires: python-chardet +Requires: python-html2text +Requires: python-html5lib Requires(post): update-alternatives Requires(postun): update-alternatives BuildArch: noarch @@ -53,11 +58,7 @@ %prep %setup -q -n %{modname}-%{version} sed -i -e '/^#!\/usr\/bin\/python/d' fanficfare/mobi{,html}.py -for f in DESCRIPTION.rst README.md ; do - tr -d '\r' < $f > $f.new - touch --reference=$f $f.new - mv -f $f.new $f -done +dos2unix DESCRIPTION.rst README.md %build %python_build ++++++ FanFicFare-3.5.0.tar.gz -> FanFicFare-3.6.0.tar.gz ++++++ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/FanFicFare-3.5.0/calibre-plugin/__init__.py new/FanFicFare-3.6.0/calibre-plugin/__init__.py --- old/FanFicFare-3.5.0/calibre-plugin/__init__.py 2019-02-12 02:50:50.000000000 +0100 +++ new/FanFicFare-3.6.0/calibre-plugin/__init__.py 2019-03-12 16:28:29.000000000 +0100 @@ -33,7 +33,7 @@ from calibre.customize import InterfaceActionBase # pulled out from FanFicFareBase for saving in prefs.py -__version__ = (3, 5, 0) +__version__ = (3, 6, 0) ## Apparently the name for this class doesn't matter--it was still ## 'demo' for the first few versions. diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/FanFicFare-3.5.0/calibre-plugin/plugin-defaults.ini new/FanFicFare-3.6.0/calibre-plugin/plugin-defaults.ini --- old/FanFicFare-3.5.0/calibre-plugin/plugin-defaults.ini 2019-02-12 02:50:50.000000000 +0100 +++ new/FanFicFare-3.6.0/calibre-plugin/plugin-defaults.ini 2019-03-12 16:28:29.000000000 +0100 @@ -1438,7 +1438,7 @@ ## Site dedicated to these categories/characters/ships extracategories:Harry Potter -extra_valid_entries: origin,originUrl,originHTML,reviews +extra_valid_entries: origin,originUrl,originHTML originHTML_label:Original Story URL ## Assume entryUrl, apply to "<a class='%slink' href='%s'>%s</a>" to diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/FanFicFare-3.5.0/calibre-plugin/translations/ca.po new/FanFicFare-3.6.0/calibre-plugin/translations/ca.po --- old/FanFicFare-3.5.0/calibre-plugin/translations/ca.po 2019-02-12 02:50:50.000000000 +0100 +++ new/FanFicFare-3.6.0/calibre-plugin/translations/ca.po 2019-03-12 16:28:29.000000000 +0100 @@ -2,7 +2,7 @@ # Copyright (C) YEAR ORGANIZATION # # Translators: -# Adolfo Jayme Barrientos, 2014 +# Adolfo Jayme-Barrientos, 2014 # jmontane, 2014 # Queralt Iglesias <[email protected]>, 2016 # Robert Antoni Buj Gelonch <[email protected]>, 2016-2017 diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/FanFicFare-3.5.0/calibre-plugin/translations/es.po new/FanFicFare-3.6.0/calibre-plugin/translations/es.po --- old/FanFicFare-3.5.0/calibre-plugin/translations/es.po 2019-02-12 02:50:50.000000000 +0100 +++ new/FanFicFare-3.6.0/calibre-plugin/translations/es.po 2019-03-12 16:28:29.000000000 +0100 @@ -2,7 +2,7 @@ # Copyright (C) YEAR ORGANIZATION # # Translators: -# Adolfo Jayme Barrientos, 2014 +# Adolfo Jayme-Barrientos, 2014 # Albert, 2016 # Darío Hereñú <[email protected]>, 2015-2016,2018 # Darío Hereñú <[email protected]>, 2018 diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/FanFicFare-3.5.0/calibre-plugin/translations/et.po new/FanFicFare-3.6.0/calibre-plugin/translations/et.po --- old/FanFicFare-3.5.0/calibre-plugin/translations/et.po 2019-02-12 02:50:50.000000000 +0100 +++ new/FanFicFare-3.6.0/calibre-plugin/translations/et.po 2019-03-12 16:28:29.000000000 +0100 @@ -7,7 +7,7 @@ msgstr "" "Project-Id-Version: calibre-plugins\n" "POT-Creation-Date: 2018-12-25 23:36+Central Standard Time\n" -"PO-Revision-Date: 2019-01-17 03:58+0000\n" +"PO-Revision-Date: 2019-03-03 19:13+0000\n" "Last-Translator: Maidur\n" "Language-Team: Estonian (http://www.transifex.com/calibre/calibre-plugins/language/et/)\n" "MIME-Version: 1.0\n" @@ -1077,18 +1077,18 @@ #: config.py:1498 msgid "Fix Title Case?" -msgstr "" +msgstr "Paranda pealkirjade suurtähestus?" #: config.py:1499 msgid "" "If checked, Calibre's routine for correcting the capitalization of title " "will be applied." -msgstr "" +msgstr "Kui märgistatud, rakendatakse pealkirjade suurtähestuse parandamiseks Calibre rutiini." #: config.py:1500 config.py:1511 msgid "" "This effects Calibre metadata only, not FanFicFare metadata in title page." -msgstr "" +msgstr "See mõjutab ainult Calibre metaandmeid, mitte FanFicFare'i metaandmeid tiitellehel." #: config.py:1504 msgid "Force Author into Author Sort?" @@ -1102,19 +1102,19 @@ #: config.py:1508 msgid "Fix Author Case?" -msgstr "" +msgstr "Paranda autorite suurtähestus?" #: config.py:1509 msgid "" "If checked, Calibre's routine for correcting the capitalization of author " "names will be applied." -msgstr "" +msgstr "Kui märgistatud, rakendatakse autorinimede suurtähestuse parandamiseks Calibre rutiini." #: config.py:1510 msgid "" "Calibre remembers all authors in the library; changing the author case on " "one book will effect all books by that author." -msgstr "" +msgstr "Calibre jätab meelde kõik kogus olevad autorid; autorinime suurtähestuse muutmine ühes raamatus mõjutab kõiki selle autori raamatuid." #: config.py:1522 msgid "Other Standard Column Options" diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/FanFicFare-3.5.0/calibre-plugin/translations/fr.po new/FanFicFare-3.6.0/calibre-plugin/translations/fr.po --- old/FanFicFare-3.5.0/calibre-plugin/translations/fr.po 2019-02-12 02:50:50.000000000 +0100 +++ new/FanFicFare-3.6.0/calibre-plugin/translations/fr.po 2019-03-12 16:28:29.000000000 +0100 @@ -3,7 +3,7 @@ # # Translators: # Xotes <[email protected]>, 2015 -# Doryan R <[email protected]>, 2019 +# Doryan R, 2019 # Franck, 2015 # J M <[email protected]>, 2016 # Ptit Prince <[email protected]>, 2014-2016 @@ -18,7 +18,7 @@ "Project-Id-Version: calibre-plugins\n" "POT-Creation-Date: 2018-12-25 23:36+Central Standard Time\n" "PO-Revision-Date: 2019-02-01 17:14+0000\n" -"Last-Translator: Doryan R <[email protected]>\n" +"Last-Translator: Doryan R\n" "Language-Team: French (http://www.transifex.com/calibre/calibre-plugins/language/fr/)\n" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=UTF-8\n" diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/FanFicFare-3.5.0/fanficfare/adapters/adapter_archiveofourownorg.py new/FanFicFare-3.6.0/fanficfare/adapters/adapter_archiveofourownorg.py --- old/FanFicFare-3.5.0/fanficfare/adapters/adapter_archiveofourownorg.py 2019-02-12 02:50:50.000000000 +0100 +++ new/FanFicFare-3.6.0/fanficfare/adapters/adapter_archiveofourownorg.py 2019-03-12 16:28:29.000000000 +0100 @@ -267,9 +267,10 @@ self.story.setMetadata('language',stripHTML(d.text)) a = metasoup.find('dd',{'class':"fandom tags"}) - fandoms = a.findAll('a',{'class':"tag"}) - for fandom in fandoms: - self.story.addToList('fandoms',fandom.string) + if a != None: + fandoms = a.findAll('a',{'class':"tag"}) + for fandom in fandoms: + self.story.addToList('fandoms',fandom.string) a = metasoup.find('dd',{'class':"warning tags"}) if a != None: diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/FanFicFare-3.5.0/fanficfare/adapters/adapter_fictionhuntcom.py new/FanFicFare-3.6.0/fanficfare/adapters/adapter_fictionhuntcom.py --- old/FanFicFare-3.5.0/fanficfare/adapters/adapter_fictionhuntcom.py 2019-02-12 02:50:50.000000000 +0100 +++ new/FanFicFare-3.6.0/fanficfare/adapters/adapter_fictionhuntcom.py 2019-03-12 16:28:29.000000000 +0100 @@ -1,6 +1,6 @@ # -*- coding: utf-8 -*- -# Copyright 2018 FanFicFare team +# Copyright 2019 FanFicFare team # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -34,16 +34,32 @@ BaseSiteAdapter.__init__(self, config, url) self.story.setMetadata('siteabbrev','fichunt') - # get storyId from url--url validation guarantees second part is storyId - self.story.setMetadata('storyId',self.parsedUrl.path.split('/',)[2]) - - # normalized story URL. - self._setURL("http://"+self.getSiteDomain()\ - +"/read/"+self.story.getMetadata('storyId')+"/1") + ## new types: + ## https://fictionhunt.com/stories/7edm248/the-last-of-his-kind/chapters/1 + ## https://fictionhunt.com/stories/89kzg4z/the-last-of-his-kind-new + ## old type: + ## http://fictionhunt.com/read/12411643/1 + # get storyId from url--url validation guarantees query correct + m = re.match(self.getSiteURLPattern(),url) + if m: + # logger.debug(m.groupdict()) + self.story.setMetadata('storyId',m.group('id')) + if m.group('type') == "stories": # newer URL + # normalized story URL. + self._setURL("https://"+self.getSiteDomain()\ + +"/stories/"+self.story.getMetadata('storyId')+"/"+ (m.group('title') or "")) + else: + self._setURL("https://"+self.getSiteDomain()\ + +"/read/"+self.story.getMetadata('storyId')+"/1") + # logger.debug(self.url) + else: + raise exceptions.InvalidStoryURL(url, + self.getSiteDomain(), + self.getSiteExampleURLs()) # The date format will vary from site to site. # http://docs.python.org/library/datetime.html#strftime-strptime-behavior - self.dateformat = "%d-%m-%Y" + self.dateformat = "%d %b %Y" @staticmethod def getSiteDomain(): @@ -51,10 +67,13 @@ @classmethod def getSiteExampleURLs(cls): - return "http://fictionhunt.com/read/1234/1"; + return "https://fictionhunt.com/stories/1a1a1a/story-title http://fictionhunt.com/read/1234/1"; def getSiteURLPattern(self): - return r"http://(www.)?fictionhunt.com/read/\d+(/\d+)?(/|/[^/]+)?/?$" + ## https://fictionhunt.com/stories/7edm248/the-last-of-his-kind/chapters/1 + ## https://fictionhunt.com/stories/89kzg4z/the-last-of-his-kind-new + ## http://fictionhunt.com/read/12411643/1 + return r"https?://(www.)?fictionhunt.com/(?P<type>read|stories)/(?P<id>[0-9a-z]+)(/(?P<title>[^/]+))?(/|/[^/]+)*/?$" def use_pagecache(self): ''' @@ -71,70 +90,98 @@ url = self.url try: data = self._fetchUrl(url) + soup = self.make_soup(data) + + ## detect old storyUrl, switch to new storyUrl: + canonlink = soup.find('link',rel='canonical') + if canonlink: + # logger.debug(canonlink) + canonlink = re.sub(r"/chapters/\d+","",canonlink['href']) + # logger.debug(canonlink) + self._setURL(canonlink) + url = self.url + data = self._fetchUrl(url) + soup = self.make_soup(data) + else: + # in case title changed + self._setURL(soup.select_one("div.Story__details a")['href']) + url = self.url + + except HTTPError as e: if e.code == 404: - raise exceptions.StoryDoesNotExist(self.meta) + raise exceptions.StoryDoesNotExist(self.url) else: raise e - # use BeautifulSoup HTML parser to make everything easier to find. - soup = self.make_soup(data) - - self.story.setMetadata('title',stripHTML(soup.find('div',{'class':'title'})).strip()) + self.story.setMetadata('title',stripHTML(soup.find('h1',{'class':'Story__title'}))) - self.setDescription(url,'<i>(Story descriptions not available on fictionhunt.com)</i>') + summhead = soup.find('h5',text='Summary') + self.setDescription(url,summhead.find_next('div')) - # Find authorid and URL from... author url. - # fictionhunt doesn't have author pages, use ffnet original author link. - a = soup.find('a', href=re.compile(r"fanfiction.net/u/\d+")) - self.story.setMetadata('authorId',a['href'].split('/')[-1]) - self.story.setMetadata('authorUrl','https://www.fanfiction.net/u/'+self.story.getMetadata('authorId')) - self.story.setMetadata('author',a.string) + ## author: + autha = soup.find('div',{'class':'StoryContents__meta'}).find('a') # first a in StoryContents__meta + self.story.setMetadata('authorId',autha['href'].split('/')[4]) + self.story.setMetadata('authorUrl',autha['href']) + self.story.setMetadata('author',autha.string) + + ## need author page for some metadata. + authsoup = None + authpagea = autha + authstorya = None + + ## find story url, might need to spin through author's pages. + while authpagea and not authstorya: + # logger.debug(authpagea) + authsoup = self.make_soup(self._fetchUrl(authpagea['href'])) + authpagea = authsoup.find('a',{'class':'page-link','rel':'next'}) + authstorya = authsoup.select('h4.Story__item-title a[href=%s]'%self.url) + + if not authstorya: + raise exceptions.FailedToDownload("Error finding %s on author page(s)" % self.url) + + meta = authstorya[0].parent.parent.select("div.Story__meta-info")[0] + ## remove delimiters + for span in authstorya[0].parent.parent.select("div.Story__meta-info span.delimiter"): + span.extract() + meta.find('span').extract() # discard author link + + update = stripHTML(meta.find('span').extract()).split(':')[1].strip() + self.story.setMetadata('dateUpdated', makeDate(update, self.dateformat)) + + pubdate = stripHTML(meta.find('span').extract()).split(':')[1].strip() + self.story.setMetadata('datePublished', makeDate(pubdate, self.dateformat)) + + meta=meta.text.split() + self.story.setMetadata('numWords',meta[meta.index('words')-1]) + self.story.setMetadata('rating',meta[meta.index('Rating:')+1]) + # logger.debug(meta) # Find original ffnet URL - a = soup.find('a', href=re.compile(r"fanfiction.net/s/\d+")) + a = soup.find('a', text="Source") self.story.setMetadata('origin',stripHTML(a)) self.story.setMetadata('originUrl',a['href']) - # Fleur D. & Harry P. & Hermione G. & Susan B. - Words: 42,848 - Rated: M - English - None - Chapters: 9 - Reviews: 248 - Updated: 21-09-2016 - Published: 16-05-2015 - by Elven Sorcerer (FFN) - # None - Words: 13,087 - Rated: M - English - Romance & Supernatural - Chapters: 3 - Reviews: 5 - Updated: 21-09-2016 - Published: 20-09-2016 - # Harry P. & OC - Words: 10,910 - Rated: M - English - None - Chapters: 5 - Reviews: 6 - Updated: 21-09-2016 - Published: 11-09-2016 - # Dudley D. & Harry P. & Nagini & Vernon D. - Words: 4,328 - Rated: K+ - English - None - Chapters: 2 - Updated: 21-09-2016 - Published: 20-09-2016 - - details = soup.find('div',{'class':'details'}) - - detail_re = \ - r'(?P<characters>.+) - Words: (?P<numWords>[0-9,]+) - Rated: (?P<rating>[a-zA-Z\\+]+) - (?P<language>.+) - (?P<genre>.+)'+ \ - r' - Chapters: (?P<numChapters>[0-9,]+)( - Reviews: (?P<reviews>[0-9,]+))? - Updated: (?P<dateUpdated>[0-9-]+)'+ \ - r' - Published: (?P<datePublished>[0-9-]+)(?P<completed> - Complete)?' - - details_dict = re.match(detail_re,stripHTML(details)).groupdict() - - # lists - for meta in ('characters','genre'): - if details_dict[meta] != 'None': - self.story.extendList(meta,details_dict[meta].split(' & ')) - - # scalars - for meta in ('numWords','numChapters','rating','language','reviews'): - self.story.setMetadata(meta,details_dict[meta]) - - # dates - for meta in ('datePublished','dateUpdated'): - self.story.setMetadata(meta, makeDate(details_dict[meta], self.dateformat)) - - # status - if details_dict['completed']: + datesdiv = soup.find('div',{'class':'dates'}) + if stripHTML(datesdiv.find('label')) == 'Completed' : # first label is status. self.story.setMetadata('status', 'Completed') else: self.story.setMetadata('status', 'In-Progress') - # It's assumed that the number of chapters is correct. - # There's no complete list of chapters, so the only - # alternative is to get the num of chaps from the last - # indiated chapter list instead. - for i in range(1,1+int(self.story.getMetadata('numChapters'))): - self.add_chapter("Chapter "+unicode(i),"http://"+self.getSiteDomain()\ - +"/read/"+self.story.getMetadata('storyId')+"/%s"%i) + for a in soup.select("div.genres a"): + self.story.addToList('genre',stripHTML(a)) + + for a in soup.select("section.characters li.Tags__item a"): + self.story.addToList('characters',stripHTML(a)) + + for a in soup.select('a[href*="pairings="]'): + self.story.addToList('ships',stripHTML(a).replace("+","/")) + + for chapa in soup.select('ul.StoryContents__chapters a'): + self.add_chapter(stripHTML(chapa.find('span',{'class':'chapter-title'})),chapa['href']) + + if self.num_chapters() == 0: + raise exceptions.FailedToDownload("Story at %s has no chapters." % self.url) def getChapterText(self, url): logger.debug('Getting chapter text from: %s' % url) @@ -142,7 +189,7 @@ soup = self.make_soup(data) - div = soup.find('div', {'class' : 'text'}) + div = soup.find('div', {'class' : 'StoryChapter__text'}) return self.utf8FromSoup(url,div) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/FanFicFare-3.5.0/fanficfare/adapters/adapter_harrypotterfanfictioncom.py new/FanFicFare-3.6.0/fanficfare/adapters/adapter_harrypotterfanfictioncom.py --- old/FanFicFare-3.5.0/fanficfare/adapters/adapter_harrypotterfanfictioncom.py 2019-02-12 02:50:50.000000000 +0100 +++ new/FanFicFare-3.6.0/fanficfare/adapters/adapter_harrypotterfanfictioncom.py 2019-03-12 16:28:29.000000000 +0100 @@ -19,7 +19,7 @@ import logging logger = logging.getLogger(__name__) import re -from ..htmlcleanup import stripHTML +from ..htmlcleanup import stripHTML, removeAllEntities from .. import exceptions as exceptions # py2 vs py3 transition @@ -115,10 +115,23 @@ chapter_words+=int(tdstr) ## used below if total words from site not found - ## Finding the metadata is a bit of a pain. Desc is the only thing this color. - desctable= soup.find('table',{'bgcolor':'#f0e8e8'}) - #self.setDescription(url,desctable) - #self.story.setMetadata('description',stripHTML(desctable)) + # fetch author page to get story description. + authorsoup = self.make_soup(self._fetchUrl(self.story.getMetadata('authorUrl'))) + + # assumes don't need to worry about story URLs in descs. + storya = authorsoup.find('a', href=re.compile(r"^/viewstory.php\?psid="+self.story.getMetadata('storyId'))) + storydiv = storya.find_parent('div') + + # desc is escaped html in attr on iframe. + iframe = storydiv.find('iframe') + iframesrc = removeAllEntities(iframe['srcdoc']) + # logger.debug(iframesrc) + descsoup=self.make_soup(iframesrc) + desc = descsoup.body + desc.name='div' # change body tag to div + del desc['class'] # clear class='iframe' + # logger.debug(desc.body) + self.setDescription(url,desc) # <div class='entry'> # <div class='entry__key'>Rating</div> @@ -129,6 +142,8 @@ 'Rating':'rating', 'Words':'numWords', 'Characters':'characters', + 'Primary Relationship':'ships', + 'Secondary Relationship(s)':'ships', 'Genre(s)':'genre', 'Era':'era', 'Advisory':'warnings', @@ -144,7 +159,7 @@ if meta: if meta.startswith('date'): value = makeDate(value,self.dateformat) - if meta in ('characters','genre'): + if meta in ('characters','genre','ships'): self.story.extendList(meta,value.split(',')) else: self.story.setMetadata(meta,value) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/FanFicFare-3.5.0/fanficfare/adapters/adapter_storiesonlinenet.py new/FanFicFare-3.6.0/fanficfare/adapters/adapter_storiesonlinenet.py --- old/FanFicFare-3.5.0/fanficfare/adapters/adapter_storiesonlinenet.py 2019-02-12 02:50:50.000000000 +0100 +++ new/FanFicFare-3.6.0/fanficfare/adapters/adapter_storiesonlinenet.py 2019-03-12 16:28:29.000000000 +0100 @@ -353,15 +353,20 @@ if universe_soup: logger.debug("Retrieving Universe - looking for name") universe_name = stripHTML(universe_soup.find('h1', {'id' : 'ptitle'})) - universe_name = re.sub(r' . A Universe from the Mind.*$','',universe_name) + universe_name = re.sub(r' .\s+A Universe from the Mind.*$','',universe_name) # logger.debug("Universes name: '{0}'".format(universe_name)) self.story.setMetadata('universeUrl',universeUrl) # logger.debug("Setting universe name: '{0}'".format(universe_name)) self.story.setMetadata('universe',universe_name) - if self.getConfig("universe_as_series"): - self.setSeries(universe_name, 0) - self.story.setMetadata('seriesUrl',universeUrl) + if self.getConfig("universe_as_series") and not self.story.getMetadata('seriesUrl'): + logger.debug("universe_as_series") + # take position in universe page as number in series. + for i, storya in enumerate(universe_soup.find_all('a',href=re.compile(r'^/s/\d+/'))): + if storya['href'].split('/')[2] == self.story.getMetadata('storyId'): + self.setSeries(universe_name, i+1) + self.story.setMetadata('seriesUrl',universeUrl) + break else: logger.debug("Do not have a universe") except: diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/FanFicFare-3.5.0/fanficfare/adapters/adapter_wuxiaworldco.py new/FanFicFare-3.6.0/fanficfare/adapters/adapter_wuxiaworldco.py --- old/FanFicFare-3.5.0/fanficfare/adapters/adapter_wuxiaworldco.py 2019-02-12 02:50:50.000000000 +0100 +++ new/FanFicFare-3.6.0/fanficfare/adapters/adapter_wuxiaworldco.py 2019-03-12 16:28:29.000000000 +0100 @@ -1,6 +1,6 @@ # -*- coding: utf-8 -*- -# Copyright 2018 FanFicFare team +# Copyright 2019 FanFicFare team # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -50,7 +50,7 @@ story_id = match.group('id') self.story.setMetadata('storyId', story_id) - self._setURL('http://%s/%s/' % (self.getSiteDomain(), story_id)) + self._setURL('https://%s/%s/' % (self.getSiteDomain(), story_id)) @staticmethod def getSiteDomain(): @@ -62,7 +62,7 @@ @classmethod def getSiteExampleURLs(cls): - return 'http://%s/story-name' % cls.getSiteDomain() + return 'https://%s/story-name' % cls.getSiteDomain() def getSiteURLPattern(self): return r'https?://(www|m)\.wuxiaworld\.co/(?P<id>[^/]+)(/)?' diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/FanFicFare-3.5.0/fanficfare/cli.py new/FanFicFare-3.6.0/fanficfare/cli.py --- old/FanFicFare-3.5.0/fanficfare/cli.py 2019-02-12 02:50:50.000000000 +0100 +++ new/FanFicFare-3.6.0/fanficfare/cli.py 2019-03-12 16:28:29.000000000 +0100 @@ -39,7 +39,7 @@ def pickle_load(f): return pickle.load(f,encoding="bytes") -version="3.5.0" +version="3.6.0" os.environ['CURRENT_VERSION_ID']=version global_cache = 'global_cache' @@ -92,31 +92,31 @@ if not parser: parser = OptionParser('usage: %prog [options] [STORYURL]...') parser.add_option('-f', '--format', dest='format', default='epub', - help='write story as FORMAT, epub(default), mobi, txt or html', metavar='FORMAT') + help='Write story as FORMAT, epub(default), mobi, txt or html.', metavar='FORMAT') if passed_defaultsini: - config_help = 'read config from specified file(s) in addition to calibre plugin personal.ini, ~/.fanficfare/personal.ini, and ./personal.ini' + config_help = 'Read config from specified file(s) in addition to calibre plugin personal.ini, ~/.fanficfare/personal.ini, and ./personal.ini' else: - config_help = 'read config from specified file(s) in addition to ~/.fanficfare/defaults.ini, ~/.fanficfare/personal.ini, ./defaults.ini, and ./personal.ini' + config_help = 'Read config from specified file(s) in addition to ~/.fanficfare/defaults.ini, ~/.fanficfare/personal.ini, ./defaults.ini, and ./personal.ini' parser.add_option('-c', '--config', action='append', dest='configfile', default=None, help=config_help, metavar='CONFIG') range_help = ' --begin and --end will be overridden by a chapter range on the STORYURL like STORYURL[1-2], STORYURL[-3], STORYURL[3-] or STORYURL[3]' parser.add_option('-b', '--begin', dest='begin', default=None, - help='Begin with Chapter START.'+range_help, metavar='START') + help='Begin story with Chapter START.'+range_help, metavar='START') parser.add_option('-e', '--end', dest='end', default=None, - help='End with Chapter END.'+range_help, metavar='END') + help='End story with Chapter END.'+range_help, metavar='END') parser.add_option('-o', '--option', action='append', dest='options', - help='set an option NAME=VALUE', metavar='NAME=VALUE') + help='Set config option NAME=VALUE Overrides config file setting.', metavar='NAME=VALUE') parser.add_option('-m', '--meta-only', action='store_true', dest='metaonly', - help='Retrieve metadata and stop. Or, if --update-epub, update metadata title page only.', ) + help='Retrieve and write metadata to stdout without downloading or saving chapters; saves story file with titlepage only. (See also --json-meta)', ) parser.add_option('-z', '--no-meta-chapters', action='store_true', dest='nometachapters', - help='Exclude list of chapters("zchapters") from metadata dump. No effect without --meta-only flag', ) - parser.add_option('--json-meta', + help='Exclude list of chapters("zchapters") from metadata stdout output. No effect without --meta-only or --json-meta flags', ) + parser.add_option('-j', '--json-meta', action='store_true', dest='jsonmeta', - help='When used with --meta-only, output metadata as JSON. No effect without --meta-only flag', ) + help='Output metadata as JSON with download, or with --meta-only flag. (Only JSON will be output with --meta-only flag.)', ) parser.add_option('-u', '--update-epub', action='store_true', dest='update', help='Update an existing epub(if present) with new chapters. Give either epub filename or story URL.', ) @@ -460,14 +460,6 @@ write_story(configuration, adapter, 'epub') else: - # regular download - if options.metaonly: - metadata = adapter.getStoryMetadataOnly().getAllMetadata() - if not options.nometachapters: - metadata['zchapters'] = [] - for i, chap in enumerate(adapter.get_chapters()): - metadata['zchapters'].append((i+1,chap)) - if not options.metaonly and adapter.getConfig('pre_process_cmd'): if adapter.getConfig('pre_process_safepattern'): metadata = adapter.story.get_filename_safe_metadata(pattern=adapter.getConfig('pre_process_safepattern')) @@ -477,14 +469,14 @@ output_filename = write_story(configuration, adapter, options.format, options.metaonly) - if options.metaonly: + if options.metaonly and not options.jsonmeta: + metadata = adapter.getStoryMetadataOnly().getAllMetadata() metadata['output_filename'] = output_filename - if options.jsonmeta: - import json - print(json.dumps(metadata, sort_keys=True, - indent=2, separators=(',', ':'))) - else: - pprint.pprint(metadata) + if not options.nometachapters: + metadata['zchapters'] = [] + for i, chap in enumerate(adapter.get_chapters()): + metadata['zchapters'].append((i+1,chap)) + pprint.pprint(metadata) if not options.metaonly and adapter.getConfig('post_process_cmd'): if adapter.getConfig('post_process_safepattern'): @@ -494,6 +486,17 @@ metadata['output_filename'] = output_filename call(string.Template(adapter.getConfig('post_process_cmd')).substitute(metadata), shell=True) + if options.jsonmeta: + metadata = adapter.getStoryMetadataOnly().getAllMetadata() + metadata['output_filename'] = output_filename + if not options.nometachapters: + metadata['zchapters'] = [] + for i, chap in enumerate(adapter.get_chapters()): + metadata['zchapters'].append((i+1,chap)) + import json + print(json.dumps(metadata, sort_keys=True, + indent=2, separators=(',', ':'))) + del adapter except exceptions.InvalidStoryURL as isu: diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/FanFicFare-3.5.0/fanficfare/defaults.ini new/FanFicFare-3.6.0/fanficfare/defaults.ini --- old/FanFicFare-3.5.0/fanficfare/defaults.ini 2019-02-12 02:50:50.000000000 +0100 +++ new/FanFicFare-3.6.0/fanficfare/defaults.ini 2019-03-12 16:28:29.000000000 +0100 @@ -1472,7 +1472,7 @@ ## Site dedicated to these categories/characters/ships extracategories:Harry Potter -extra_valid_entries: origin,originUrl,originHTML,reviews +extra_valid_entries: origin,originUrl,originHTML originHTML_label:Original Story URL ## Assume entryUrl, apply to "<a class='%slink' href='%s'>%s</a>" to diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/FanFicFare-3.5.0/setup.py new/FanFicFare-3.6.0/setup.py --- old/FanFicFare-3.5.0/setup.py 2019-02-12 02:50:50.000000000 +0100 +++ new/FanFicFare-3.6.0/setup.py 2019-03-12 16:28:29.000000000 +0100 @@ -27,7 +27,7 @@ name=package_name, # Versions should comply with PEP440. - version="3.5.0", + version="3.6.0", description='A tool for downloading fanfiction to eBook formats', long_description=long_description, diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/FanFicFare-3.5.0/webservice/app.yaml new/FanFicFare-3.6.0/webservice/app.yaml --- old/FanFicFare-3.5.0/webservice/app.yaml 2019-02-12 02:50:50.000000000 +0100 +++ new/FanFicFare-3.6.0/webservice/app.yaml 2019-03-12 16:28:29.000000000 +0100 @@ -1,6 +1,6 @@ # ffd-retief-hrd fanficfare application: fanficfare -version: 3-5-0 +version: 3-6-0 runtime: python27 api_version: 1 threadsafe: true diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/FanFicFare-3.5.0/webservice/index.html new/FanFicFare-3.6.0/webservice/index.html --- old/FanFicFare-3.5.0/webservice/index.html 2019-02-12 02:50:50.000000000 +0100 +++ new/FanFicFare-3.6.0/webservice/index.html 2019-03-12 16:28:29.000000000 +0100 @@ -84,7 +84,7 @@ If you have any problems with this application, please report them in the <a href="https://groups.google.com/group/fanfic-downloader";>FanFicFare Google Group</a>. The - <a href="https://3-4-0.fanficfare.appspot.com";>previous version</a> + <a href="https://3-5-0.fanficfare.appspot.com";>previous version</a> is also available for you to use if necessary. </p> <div id='error'>
