Hi, I still having this issue.
I'm using python 2.7.8 (Linux), scrapy 0.24.4 and i'm getting the following
error through localhost:6080/crawler
exceptions.TypeError: <scrapy.crawler.Crawler object at 0x7f23ccf44e90> is
not JSON serializable
/usr/local/lib/python2.7/dist-packages/twisted/web/server.py:189 in
process
188 self._encoder = encoder
189 self.render(resrc)
190 except:
Also, scrapy command tool shows the following piece of code :
[TelnetTransport,0,127.0.0.1] Unhandled Error
Traceback (most recent call last):
Failure: twisted.internet.error.ConnectionDone: Connection was closed
cleanly.
[...]
File "/usr/lib/python2.7/json/encoder.py", line 184, in default
raise TypeError(repr(o) + " is not JSON serializable")
exceptions.TypeError: <scrapy.crawler.Crawler object at 0x7f23ccf44e90>
is not JSON serializable
Any ideas?
On Thursday, May 3, 2012 6:10:57 PM UTC+3, Pablo Hoffman wrote:
>
> There was an issue in the webservice code which I fixed here (also fixed
> in stable branch for next release):
>
> https://github.com/scrapy/scrapy/commit/9c3b9f29687c78b56825abcc78f4faa326a90e0a
>
> Still, you shouldn't be able to access http://localhost:6080/crawler
> directly, as the object itself is not JSON serializable. Instead you can do
> a JSON-RPC call to invoke a method of it. See the extras/scrapy-ws.py
> <https://github.com/scrapy/scrapy/blob/master/extras/scrapy-ws.py> script
> for some examples.
>
> Pablo.
>
> On 04/29/2012 03:54 PM, Steven Almeroth wrote:
>
> It seems there is a bug in the way Twisted is processing
> the scrapy.crawler.CrawlerProcess object. In the
> method twisted.web.resource.getChildForRequest() the resource.isLeaf is not
> True and thus the request.postpath value of ['crawler'] is being rendered,
> instead of the scrapy.webservice.RootResource instance, which, when
> rendered, gives the JSON object {"resources": ["enginestatus", "stats",
> "crawler"]}.
>
> I will inform the developers of this issue, and thanks for reporting.
>
> --steven
>
> On Friday, May 13, 2011 5:08:48 AM UTC-5, Anne wrote:
>>
>> Hi all,
>>
>> I'm rather new to scrapy, but managed to get a spider up and running
>> without too much effort: great work, thanks!
>>
>> I would love to (at least) monitor my scraper using the webservice (as
>> described on http://doc.scrapy.org/topics/webservice.html). So I've
>> enabled it in my settings.py, by setting WEBSERVICE_ENABLED = True.
>>
>> Now, if I point my browser at http://localhost:6080/crawler (as
>> mentioned in the documentation) I get a traceback with essentially
>> this error:
>> <type 'exceptions.TypeError'>: <scrapy.crawler.CrawlerProcess object
>> at 0x23d9a50> is not JSON serializable
>>
>> The same goes for http://localhost:6080/stats. But
>> http://localhost:6080/enginestatus,
>> on the other hand, seems fine.
>>
>> What am I doing wrong?
>>
>> Best,
>> Anne
>
> --
> You received this message because you are subscribed to the Google Groups
> "scrapy-users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/scrapy-users/-/JFv-2bTCEoIJ.
> To post to this group, send email to [email protected]
> <javascript:>.
> To unsubscribe from this group, send email to
> [email protected] <javascript:>.
> For more options, visit this group at
> http://groups.google.com/group/scrapy-users?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.
