> files come in all kinds, and zipping them may not be the best option
> depending on the use-case (for already compressed ones for example).
> Maybe you are referring to text files? In that case, zipping them would
> save quite a bit of disk space indeed.
>
I am only interested in text, yes.
Hi Paul,
I haven't tried it, but it's possible.
There is two approach to this:
- Use scrapy signals to compress a text file when an event occur like
spider closed
- Or in pipeline, compress data then save it directly to the file.
- Or use a NoSQL database like MongoDB already support compression:
Hi Kasper,
files come in all kinds, and zipping them may not be the best option
depending on the use-case (for already compressed ones for example).
Maybe you are referring to text files? In that case, zipping them would
save quite a bit of disk space indeed.
> FilesPipeline does not allow me t
Hi Lhassan
Okay, thank your for your reply.
Kasper
On Thursday, August 18, 2016 at 2:50:18 PM UTC+2, Lhassan Baazzi wrote:
>
> Hi,
>
> If you are going with the zip option, just create you own pipeline that
> extend the base file pipeline and publish it as a package on Github, if
> someone els
Hi,
If you are going with the zip option, just create you own pipeline that
extend the base file pipeline and publish it as a package on Github, if
someone else needed and use it.
Best Regards.
Lhassan
Le 18 août 2016 13:16, "Kasper Marstal" a écrit :
> Hi all,
>
> I am scraping a couple of mil
