Are you going to parse the json files in python?
If you do, just write them in native python data structures
in a python file
and skip the json.load part.
Not only you will avoid using pkgutil
but you can use python syntax
that has raw strings
and allows trailing commas.
On Friday, 25 November 2016 16:00:52 UTC+2, Ciprian Lucaci wrote:
>
> I have run into the same problem.
> I included the files in the setup.py as in the example but I am not able
> to load the data with pkgutil.get_data().
>
> Can someone please post a sample code for that?
>
> I am deploying the spiders on scrapyd and I want to read all the .json
> files in conf/
>
> Thanks.
>
> On Thursday, March 14, 2013 at 6:52:46 PM UTC+2, Pablo Hoffman wrote:
>>
>> This is a snippet I've used to make deploy work with static files:
>>
>> from setuptools import setup, find_packages
>>
>> setup(
>> name='project',
>> version='1.0',
>> packages=find_packages(),
>> package_data={
>> 'myproject': ['conf/*.json', 'conf/cities.txt'],
>> },
>> entry_points={
>> 'scrapy': ['settings = myproject.settings']
>> },
>> zip_safe=False,
>> )
>>
>>
>> The conf/ dir is inside the myproject folder.
>>
>> And then you use myproject.__file__ to access the folder, on your code.
>>
>> Pablo.
>>
>>
>>
>> On Sat, Feb 16, 2013 at 10:55 PM, Thimo Brinkmann <
>> thimo.b...@googlemail.com> wrote:
>>
>>> Hey guys,
>>>
>>> I am having a really hard time deploying to scrapyd because of the
>>> static configuration files (lets call them dictionary files) that need to
>>> be used).
>>> Played around with MANIFEST.in, pkgutil.get_data, StringIO and whatever
>>> I could find for now more than 4 hours, but really can't get it work. Has
>>> anyone a working example?
>>>
>>> I want to load two static JSON files from scrapyd, but it never seemed
>>> to find the files, whatever referencing method I used. Normally I had the
>>> open method followed by the filename and I put the filename in the eggs
>>> root as well as the projects egg folder, but in no case the files were
>>> found. If anyone knows how to do this with a full example, I would be very
>>> helpful.
>>>
>>>
>>> Kind regards,
>>> Thimo
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "scrapy-users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to scrapy-users...@googlegroups.com.
>>> To post to this group, send email to scrapy...@googlegroups.com.
>>> Visit this group at http://groups.google.com/group/scrapy-users?hl=en.
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>>
>>>
>>
>>
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to scrapy-users+unsubscr...@googlegroups.com.
To post to this group, send email to scrapy-users@googlegroups.com.
Visit this group at https://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.
