Re: [Imdbpy-help] Search for movie by title, year and runtime?
Hi, Could it help to use fuzzy string matching on the local tsv files? https://github.com/seatgeek/fuzzywuzzy -- Turgut On 27-09-2018 18:33, Kesselheim, David DK - NOH wrote: > Hi, > > > > I need to look up quite a large number of titles (16k). I have the title > (more or less, sometimes ‘4k’, ‘3D’, etc is appended to it), the year > and the runtime. With that info I can get the correct title-ID from IMDB > but it is very slow because after having searched for the title and > filtered by year I need to run ia.update(title) on each of my initial > matches to compare the runtime. > > > > Is there a way to speed up the process? > > > > I insourced the tsv.gz files from IMDB but since the title isn’t an > exact match I am not sure how well SQL Server is suited for finding the > right title. > > > > Thanks for the help! > > > > *David * > > > > > > > > ___ > Imdbpy-help mailing list > Imdbpy-help@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/imdbpy-help > ___ Imdbpy-help mailing list Imdbpy-help@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/imdbpy-help
[Imdbpy-help] Documentation updates
Hi, A few days ago the documentation was converted to a Sphinx project. This was mostly a straightforward format conversion that didn't touch the content or the structure of the documents. I've been working on a draft for organizing the structure of the documents. The content is still the same, I've just tried to organize it into sections. This draft is in the reorganize-docs branch. It's still not finished but it's enough to give an idea. Help and feedback would be welcome as always. Best, -- Turgut -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Imdbpy-help mailing list Imdbpy-help@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/imdbpy-help
[Imdbpy-help] Python 2 support
Hi, I've created a new branch "py2" which supports both py2 and py3 in the same codebase. As of today, it's up-to-date with the current master and passes almost all the tests. But it only deals with the http access system; there are no changes regarding sql or s3. If you need Py2 support please test it and send us feedback. Have a nice week, -- Turgut -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Imdbpy-help mailing list Imdbpy-help@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/imdbpy-help
Re: [Imdbpy-help] Generating the HTML parsers
On 02/28/2018 12:13 AM, Davide Alberani wrote: > On Tue, Feb 27, 2018 at 9:05 AM, H. Turgut Uyar <u...@tekir.org> wrote: >> So I decided to develop a parser generator that will read a >> specification for a parser and generate the necessary code > > What kind of help you need, mostly? > > Most importantly, I can't really decide if this is worth pursuing. I feel like the approach has some potential but I can't be sure. The code can be manually written after all. The spec format is more generic, so it might be easier to refactor the parsers in the future but I don't know how likely that is to happen. And also, would we gain from transitioning to piculet based parsers? Possible advantages could be: - Making py2 support easier if we want that. - Dropping the hard dependency on lxml, again, if we want that. - Easier maintenance of the parsers (?). - More involvement from developers for writing parsers (?). So my main problem is that I'm undecided whether I should devote more time to this or not. Any insight into that issue would be much appreciated. -- Turgut -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Imdbpy-help mailing list Imdbpy-help@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/imdbpy-help
[Imdbpy-help] Generating the HTML parsers
Hi, Over the last few years, I've refactored the basis for the IMDbPY HTML parsers into a separate package called "piculet" that could be used with -hopefully- any HTML markup. It has no required external dependency, supports py2/py3/pypy and improves on the current IMDbPY parsers with some features and a more consistent interface. The idea was, and still is, that at some point we can reimplement the IMDbPY parsers using piculet. This shouldn't be too hard since the syntax is quite similar. I've attempted this a few times already and managed to make some headway but trying to fit things into the current codebase kept distracting me from the actual job of dealing with the parsers. So I decided to develop a parser generator that will read a specification for a parser and generate the necessary code. I hope this will make the transition easier. My not-so-preliminary work is here: https://github.com/uyar/piculet_imdb Note that this project is not a full package like IMDbPY. It doesn't have the Movie/Person/etc classes. It doesn't even have the code to fetch the IMDb pages (except for the simple retrievers in the tests). If we decide that this approach makes sense, we could create a template suitable for IMDbPY. If anyone's interested I'd be happy to hear thoughts, suggestions, and of course pull requests. Have a nice day, -- Turgut Uyar -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Imdbpy-help mailing list Imdbpy-help@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/imdbpy-help
Re: [Imdbpy-help] Presentation and windows problem
Hi, Is it possible that pyinstaller doesn't handle the setup.py file and/or the MANIFEST.in file like the sdist command does? Those are the places where the cookies.json file is listed. Turgut On November 18, 2017 10:13:07 AM "H. Turgut Uyar" <htur...@uyar.info> wrote: Hi, Is it possible that pyinstaller doesn't handle the setup.py file and/or the MANIFEST.in file like the sdist command does? Those are the places where the cookies.json file is listed. Turgut On November 16, 2017 9:55:34 PM Martín Torre Castro <martin.torre.cas...@gmail.com> wrote: I'll try to build it on Windows, in the next days. Can you tell me how you have set up the environment? Which version of Python? Which command you run to build the .exe? Python 3.6.3 pyInstaller --onefile sample_get_info.py > 2017-11-16 00:44:46,732 WARNING [imdbpy] C:\Program Files > (x86)\Python36-32\lib\site-packages\imdb\__init__.py:165: Unable to read > configuration file; complete error: 'ConfigParserWithCase' object has no > attribute '_boolean_states' Nice; this seems to be a problem related to the parsing of the imdbpy.cfg file: remove it (I'll try to reproduce and fix the bug later), since you don't really need it. > grParser = GatherRefs(useModule=self._useModule) > AttributeError: 'DOMHTMLPlotParser' object has no attribute '_useModule' It seems you're using an old version. IMDbPY just came out of a huge set of changes, and it still have some bugs here and there. The above one should already be fixed in the repository version: https://github.com/alberanid/imdbpy IMDbPY==6.0 I'm checking now. It says everything is 'already-up-to-date'. Finally, I was trying the file from a very little gui in a '.pyw' file. Now I tried only the text-mode script and it gives this error at the command line. It's happening at the IMDb() call. Traceback (most recent call last): File "sample_get_info.py", line 56, in File "sample_get_info.py", line 24, in get_data File "site-packages\imdb\__init__.py", line 186, in IMDb File "", line 971, in _find_and_load File "", line 955, in _find_and_load_unlocked File "", line 665, in _load_unlocked File "c:\program files (x86)\python36-32\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 631, in exec_module exec(bytecode, module.__dict__) File "site-packages\imdb\parser\http\__init__.py", line 99, in FileNotFoundError: [Errno 2] No such file or directory: 'C:\\UsersAppData\\Local\\Temp\\_MEI56602\\imdb\\parser\\http\\cookie s.json' -- -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot -- ___ Imdbpy-help mailing list Imdbpy-help@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/imdbpy-help -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ Imdbpy-help mailing list Imdbpy-help@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/imdbpy-help
[Imdbpy-help] Help with tests
Hi,
Now that the test skeleton is in the repository (the codename-simplify
branch), it would be very helpful if more people could contribute with
tests. The testing code is quite simple, here's an example from the
person main details page tests:
def test_name_should_be_canonical(person_main_details):
page = person_main_details('keanu_reeves')
data = parser.parse(page)['data']
assert data['name'] == 'Reeves, Keanu'
The body of every test function is like that. Get the page, run through
the parser, check the result. I have written the functions for getting
the movie combined details page and the person main details page. I will
add the other types of pages soon or I can add one when anyone needs it.
The current tests are here:
https://github.com/alberanid/imdbpy/tree/codename-simplify/tests
What needs to be done is as follows:
1. Select something that hasn't been tested yet.
2. Select a person (or movie, or company etc) page that can be used to
test it.
3. If not already there, add its IMDb id to the relevant dictionary in
the conftest.py file (MOVIES, PEOPLE).
4. Figure out the name of the key in the result and write an assert to
check for the correct value.
Having more tests would increase our confidence in the parsers,
especially after the port to Python3. Any contributions would be greatly
appreciated.
Cheers
--
Turgut
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Imdbpy-help mailing list
Imdbpy-help@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/imdbpy-help
Re: [Imdbpy-help] imdbpy 4.6 can't fetch person head shot (and who add IMDb to the names ?)
On 09/29/2010 12:45 PM, Davide Alberani wrote: I'll have time to check the patch and commit it to Mercurial only tomorrow. By the way, anyone should feel free to fork the IMDbPY repository on Bitbucket (specifically http://bitbucket.org/alberanid/imdbpy_new_search_parsers/ ), commit his changes and ask for a pull. Is it that fork or the one called imdbpy_parsers2010? I might have forked the wrong one. tried building it myself, imdb page killed lxml, elementtree and BeautifulSoup, Strange: we're based on lxml (falling back to BeautifulSoup if lxml is not installed). At some places we have to fix the HTML before feeding it to lxml or beautifulsoup (the preprocessors in the code). Maybe it was one of those pages. -- H. Turgut Uyar u...@itu.edu.tr [GPG KeyID: 0xEAF45FB8] http://web.itu.edu.tr/uyar/ -- Start uncovering the many advantages of virtual appliances and start using them to simplify application deployment and accelerate your shift to cloud computing. http://p.sf.net/sfu/novell-sfdev2dev ___ Imdbpy-help mailing list Imdbpy-help@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/imdbpy-help
Re: [Imdbpy-help] five years of IMDbPY!
Many happy returns :-) On 04/01/2009 02:17 PM, ori cohen wrote: heh awesome :) beer for everyone.. Ok, sharing the birthday with GMail is not easy, but exactly five years ago IMDbPY 1.0 was released, too. :-) And I can notice, with a bit of pride, that - _30_ releases later - the main API is still the same and the project is alive and kicking. :-) So, thanks to anyone who contributed with code, patches, bug reports and so on! Let's celebrate! ;-) -- Turgut Uyar -- ___ Imdbpy-help mailing list Imdbpy-help@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/imdbpy-help
Re: [Imdbpy-help] BUG: 4.0-dev - Newline Missing after Genres and incorrect plot summary
On 03/25/2009 05:36 PM, Davide Alberani wrote: On Mar 25, H. Turgut Uyar u...@itu.edu.tr wrote: Maybe we should set the dependency info in the setup.py file so that a release with version greater than 2.1 will be selected. For sure it won't hurt. :-) Greater than 2.1 might be overkill but I'm a bit lazy to find out which exact version of lxml satisfies all the features we use :-) PS: in a matter of days, I hope to add top 250/bottom 100 links to imdbpykit. Great. Whata about i18n? I see a file for English and one for Turkish; I can add Italian. English and Turkish are also nearly empty. The problem is that we have to generate a list of XML tags and that can be quite laborious. gettext utils can not help us either because the tags are generated in the code, so we can't mark them anywhere like _(..) for a utility to collect. Is there an easy way to translate the XML generated by the XSL file? It should be easy, an XML book I have says to put an entry for every language as in: titleIMDbPY gateway/title title xml:lang=trIMDbPY ag gecidi/title And the browser should select the correct one. -- Turgut -- ___ Imdbpy-help mailing list Imdbpy-help@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/imdbpy-help
