Re: [twitter-dev] Re: Anyone using phirehose?
It has been happening to me as well. It just stops receiving data after
about 25-26 hours. Any timeout or something?
On Fri, Feb 12, 2010 at 12:52 AM, olmy wrote:
> I am using it for the last 7 days, In this period, it has stopped
> collecting data twice. The program is still running, I can see the
> process ids, but no data is coming. This is my code,
>
> class FilterTrackConsumer extends Phirehose
> {
>public function enqueueStatus($status)
>{
>$tweet = json_decode($status, true);
>if (is_array($tweet) &&
> isset($tweet['user']['screen_name'])) {
>$text = $tweet['text'];
>if($text != ""){
>-storing data
>}
>}
>}
> }
>
> $ptrKeyWord = mysql_query("SELECT * FROM keywords");
> $track = "";
> while($arrKeyWord = mysql_fetch_array($ptrKeyWord))
> {
>$track .= stripslashes($arrKeyWord["keyword_word"]) . ",";
> }
> $track = substr($track, 0, -1);
>
> $ptrFollow = mysql_query("SELECT * FROM users_follow WHERE user_id >
> 0");
> $strFollow = "";
> while($arrFollow = mysql_fetch_array($ptrFollow))
> {
>$strFollow .= $arrFollow["user_id"] . ",";
> }
> $strFollow = substr($strFollow, 0, -1);
>
> // Start streaming
> $sc = new FilterTrackConsumer('user', 'pwd',
> Phirehose::METHOD_FILTER);
> $sc->setTrack(array($track));
> $sc->setFollow(array($strFollow));
> $sc->consume();
>
>
> On Jan 18, 7:44 am, Fenn wrote:
> > Hi there,
> >
> > I'm the author of Phirehose and was wondering if you guys have tried
> > it/are having any sorts of problems.
> >
> > I use it myself and haven't had any problems with it randomly stopping
> > (so far) or at least, not that I've noticed.
> >
> > That's not to say that there's not an implementation problem there
> > somewhere :)
> >
> > Phirehose should be a lot more robust than the simple examples in that
> > old blog post there as it does handle reconnects, HTTP response
> > errors, retry backoff, etc, etc.
> >
> > Happy to work with you guys to solve any probs -
> >
> > Cheers!
> >
> > Fenn.
> >
> > On Jan 17, 5:27 am, Jacopo Gio wrote:> Same for
> us, George.
> >
> > > But what are the alternatives ?
> >
> > > On Jan 15, 6:17 pm, GeorgeMedia wrote:
> >
> > > > I'm looking for a solid PHP library to access the gardenhose and just
> > > > wondering if anyone is successfully implementing this using
> phirehose.
> > > > It seems to be the only one out there...
> >
> > > > This fairly dated code seems to work for random periods of time then
> > > > stops.
> >
> > > >
> http://hasin.wordpress.com/2009/06/20/collecting-data-from-streaming-...
>
--
Regards,
Atul Kulkarni
Re: [twitter-dev] Re: Anyone using phirehose?
You need to look into 'nohup'. jeffrey greenberg On Fri, Jan 22, 2010 at 10:45 AM, GeorgeMedia wrote: > Just in case anyone is having the same issue I had with PHP scripts > running from the command line stopping on them, I discovered my > problem. > > I was connecting to the linux server via SSH client remotely. I'd log > into a bash shell, CD over to the directory and run my script in the > background like -- "php script.php &" (the & is to run as a background > process). > > The problem: I'm embarrassed to say that the problem was whenever my > SSH client disconnected or timed out it killed the process I had > running in that session. Don't know why it took me so long to connect > those easy dots. > > The solution: Log into your system like normal then open another bash > session inside your session (bash). Then execute your script and exit > out of the extra session (exit). > > On Jan 15, 11:17 am, GeorgeMedia wrote: > > I'm looking for a solid PHP library to access the gardenhose and just > > wondering if anyone is successfully implementing this using phirehose. > > It seems to be the only one out there... > > > > This fairly dated code seems to work for random periods of time then > > stops. > > > > http://hasin.wordpress.com/2009/06/20/collecting-data-from-streaming-... >
Re: [twitter-dev] Re: Anyone using phirehose?
Our client would make even less sense to you then. It's written in Scala! On Sun, Jan 17, 2010 at 9:56 PM, M. Edward (Ed) Borasky wrote: > As an aside, could Twitter release the streaming client they use under > some open source license, so we can use it as a prototype? I took a > look at the one Tom May of Gist wrote using Apache HttpClient and it > didn't make much sense to me - it was importing a bunch of Java > libraries and I'm not a Java programmer. > > On Jan 16, 10:18 pm, John Kalucki wrote: > > Given a reasonable stack, it shouldn't be all that hard to build > something > > robust. Our internal streaming client, which transits every tweet that > you > > see on the streaming api, seems to work just fine through various forms > of > > abuse, and it's, roughly, a few hundred lines wrapped around Apache > > httpclient. > > > > On the other hand, I suspect that dependability is all but impossible on > > some stacks, or will require some heroism on the part of a library > > developer. > > > > As a community, we need clients that trivially allow robustness in a > variety > > of stacks. We'll get there soon enough. > > > > On Sat, Jan 16, 2010 at 10:05 PM, M. Edward (Ed) Borasky > > wrote: > > > > > > > > > On Jan 16, 7:28 pm, John Kalucki wrote: > > > > I'd strongly suggest consuming the Streaming API only from persistent > > > > processes that write into some form of durable asynchronous queue (of > any > > > > type) for your application to consume. Running curl periodically is > > > unlikely > > > > to be a robust solution. > > > > > > Select one of the existing Streaming API clients out there and wrap > it in > > > a > > > > durable process. Write to rotated log files, a message queue, or > whatever > > > > other mechanism that you choose, to buffer the arrival of new > statuses > > > > before consumption by your application. This will allow you to > restart > > > your > > > > application at will without data loss. > > > > > I don't know that there are any open source libraries out there yet > > > that are robust enough to do that. At the moment, I'm working > > > exclusively in Perl, and "AnyEvent::Twitter::Stream" seems to be the > > > only Perl Streaming API consumer with any kind of mileage on it. As > > > you point out, real-time programming for robustness is a non-trivial > > > exercise. It would be nice if someone would build a C library and SWIG > > > ".i" files. ;-) > > > > > -- > > > M. Edward (Ed) Borasky > > >http://borasky-research.net/smart-at-znmeb > > > > > "A mathematician is a device for turning coffee into theorems." ~ Paul > > > Erdős >
Re: [twitter-dev] Re: Anyone using phirehose?
Given a reasonable stack, it shouldn't be all that hard to build something robust. Our internal streaming client, which transits every tweet that you see on the streaming api, seems to work just fine through various forms of abuse, and it's, roughly, a few hundred lines wrapped around Apache httpclient. On the other hand, I suspect that dependability is all but impossible on some stacks, or will require some heroism on the part of a library developer. As a community, we need clients that trivially allow robustness in a variety of stacks. We'll get there soon enough. On Sat, Jan 16, 2010 at 10:05 PM, M. Edward (Ed) Borasky wrote: > > > On Jan 16, 7:28 pm, John Kalucki wrote: > > I'd strongly suggest consuming the Streaming API only from persistent > > processes that write into some form of durable asynchronous queue (of any > > type) for your application to consume. Running curl periodically is > unlikely > > to be a robust solution. > > > > Select one of the existing Streaming API clients out there and wrap it in > a > > durable process. Write to rotated log files, a message queue, or whatever > > other mechanism that you choose, to buffer the arrival of new statuses > > before consumption by your application. This will allow you to restart > your > > application at will without data loss. > > I don't know that there are any open source libraries out there yet > that are robust enough to do that. At the moment, I'm working > exclusively in Perl, and "AnyEvent::Twitter::Stream" seems to be the > only Perl Streaming API consumer with any kind of mileage on it. As > you point out, real-time programming for robustness is a non-trivial > exercise. It would be nice if someone would build a C library and SWIG > ".i" files. ;-) > > -- > M. Edward (Ed) Borasky > http://borasky-research.net/smart-at-znmeb > > "A mathematician is a device for turning coffee into theorems." ~ Paul > Erdős >
Re: [twitter-dev] Re: Anyone using phirehose?
I'd strongly suggest consuming the Streaming API only from persistent processes that write into some form of durable asynchronous queue (of any type) for your application to consume. Running curl periodically is unlikely to be a robust solution. Select one of the existing Streaming API clients out there and wrap it in a durable process. Write to rotated log files, a message queue, or whatever other mechanism that you choose, to buffer the arrival of new statuses before consumption by your application. This will allow you to restart your application at will without data loss. -John Kalucki http://twitter.com/jkalucki Services, Twitter Inc. On Sat, Jan 16, 2010 at 10:27 AM, Jacopo Gio wrote: > Same for us, George. > > But what are the alternatives ? > > On Jan 15, 6:17 pm, GeorgeMedia wrote: > > I'm looking for a solid PHP library to access the gardenhose and just > > wondering if anyone is successfully implementing this using phirehose. > > It seems to be the only one out there... > > > > This fairly dated code seems to work for random periods of time then > > stops. > > > > http://hasin.wordpress.com/2009/06/20/collecting-data-from-streaming-... >
