Re: [PHP] PHP cron job optimization

[Igor Escobar]

@Eric ok ;)


Regards,
Igor Escobar
*Software Engineer
*
+ http://blog.igorescobar.com
+ http://www.igorescobar.com
+ @igorescobar 





On Mon, Sep 12, 2011 at 10:52 AM, Eric Butera  wrote:

> On Mon, Sep 12, 2011 at 9:37 AM, Igor Escobar 
> wrote:
> > Other good point is: always set a timeout connection when you're getting
> the
> > RSS data to avoid your thread get stuck unnecessary. Use cURL (is much
> more
> > faster then file_get_contents).
> >
> > Multithreading in PHP with cURL http://devzone.zend.com/article/3341
> >
> >
> > Regards,
> > Igor Escobar
> > *Software Engineer
> > *
> > + http://blog.igorescobar.com
> > + http://www.igorescobar.com
> > + @igorescobar 
> >
> >
> >
> >
> >
> > On Mon, Sep 12, 2011 at 10:05 AM, Igor Escobar  >wrote:
> >
> >> Use PHP threads. Do the job separately.. in parts... in other words...
> you
> >> can't read all them at once.
> >>
> >> You can read a little more about php multithreading here:
> >> http://blog.motane.lu/2009/01/02/multithreading-in-php/
> >>
> >> You can use a non-relational database like mongo or couchdb to manage
> where
> >> you stop and where you have to look back to the RSS feed as well.
> >>
> >> []'s
> >>
> >> Regards,
> >> Igor Escobar
> >> *Software Engineer
> >> *
> >> + http://blog.igorescobar.com
> >> + http://www.igorescobar.com
> >> + @igorescobar 
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Sat, Sep 10, 2011 at 10:37 PM, Stuart Dallas 
> wrote:
> >>
> >>> On 10 Sep 2011, at 09:35, muad shibani wrote:
> >>>
> >>> > I want to design an application that reads news from RSS sources.
> >>> > I have about 1000 RSS feed to collect from.
> >>> >
> >>> > I also will use Cron jobs every 15 minutes to collect the data.
> >>> > the question is: Is there a clever way to collect all those feed
> items
> >>> > without exhausting the server
> >>> > any Ideas
> >>>
> >>> I designed a job queuing system a while back when I had a similar
> problem.
> >>> You can read about it here: http://stut.net/2009/05/29/php-job-queue/.
> >>> Set that type of system up and add a job for each feed, set to run
> every 15
> >>> minutes. You can then watch the server and tune the number of
> concurrent job
> >>> processors so you get the optimum balance between load and speed.
> >>>
> >>> -Stuart
> >>>
> >>> --
> >>> Stuart Dallas
> >>> 3ft9 Ltd
> >>> http://3ft9.com/
> >>> --
> >>> PHP General Mailing List (http://www.php.net/)
> >>> To unsubscribe, visit: http://www.php.net/unsub.php
> >>>
> >>>
> >>
> >
>
> Thread != Multi Process.
>

Re: [PHP] PHP cron job optimization

[Eric Butera]

On Mon, Sep 12, 2011 at 9:37 AM, Igor Escobar  wrote:
> Other good point is: always set a timeout connection when you're getting the
> RSS data to avoid your thread get stuck unnecessary. Use cURL (is much more
> faster then file_get_contents).
>
> Multithreading in PHP with cURL http://devzone.zend.com/article/3341
>
>
> Regards,
> Igor Escobar
> *Software Engineer
> *
> + http://blog.igorescobar.com
> + http://www.igorescobar.com
> + @igorescobar 
>
>
>
>
>
> On Mon, Sep 12, 2011 at 10:05 AM, Igor Escobar wrote:
>
>> Use PHP threads. Do the job separately.. in parts... in other words... you
>> can't read all them at once.
>>
>> You can read a little more about php multithreading here:
>> http://blog.motane.lu/2009/01/02/multithreading-in-php/
>>
>> You can use a non-relational database like mongo or couchdb to manage where
>> you stop and where you have to look back to the RSS feed as well.
>>
>> []'s
>>
>> Regards,
>> Igor Escobar
>> *Software Engineer
>> *
>> + http://blog.igorescobar.com
>> + http://www.igorescobar.com
>> + @igorescobar 
>>
>>
>>
>>
>>
>>
>> On Sat, Sep 10, 2011 at 10:37 PM, Stuart Dallas  wrote:
>>
>>> On 10 Sep 2011, at 09:35, muad shibani wrote:
>>>
>>> > I want to design an application that reads news from RSS sources.
>>> > I have about 1000 RSS feed to collect from.
>>> >
>>> > I also will use Cron jobs every 15 minutes to collect the data.
>>> > the question is: Is there a clever way to collect all those feed items
>>> > without exhausting the server
>>> > any Ideas
>>>
>>> I designed a job queuing system a while back when I had a similar problem.
>>> You can read about it here: http://stut.net/2009/05/29/php-job-queue/.
>>> Set that type of system up and add a job for each feed, set to run every 15
>>> minutes. You can then watch the server and tune the number of concurrent job
>>> processors so you get the optimum balance between load and speed.
>>>
>>> -Stuart
>>>
>>> --
>>> Stuart Dallas
>>> 3ft9 Ltd
>>> http://3ft9.com/
>>> --
>>> PHP General Mailing List (http://www.php.net/)
>>> To unsubscribe, visit: http://www.php.net/unsub.php
>>>
>>>
>>
>

Thread != Multi Process.

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] PHP cron job optimization

[Igor Escobar]

Other good point is: always set a timeout connection when you're getting the
RSS data to avoid your thread get stuck unnecessary. Use cURL (is much more
faster then file_get_contents).

Multithreading in PHP with cURL http://devzone.zend.com/article/3341


Regards,
Igor Escobar
*Software Engineer
*
+ http://blog.igorescobar.com
+ http://www.igorescobar.com
+ @igorescobar 





On Mon, Sep 12, 2011 at 10:05 AM, Igor Escobar wrote:

> Use PHP threads. Do the job separately.. in parts... in other words... you
> can't read all them at once.
>
> You can read a little more about php multithreading here:
> http://blog.motane.lu/2009/01/02/multithreading-in-php/
>
> You can use a non-relational database like mongo or couchdb to manage where
> you stop and where you have to look back to the RSS feed as well.
>
> []'s
>
> Regards,
> Igor Escobar
> *Software Engineer
> *
> + http://blog.igorescobar.com
> + http://www.igorescobar.com
> + @igorescobar 
>
>
>
>
>
>
> On Sat, Sep 10, 2011 at 10:37 PM, Stuart Dallas  wrote:
>
>> On 10 Sep 2011, at 09:35, muad shibani wrote:
>>
>> > I want to design an application that reads news from RSS sources.
>> > I have about 1000 RSS feed to collect from.
>> >
>> > I also will use Cron jobs every 15 minutes to collect the data.
>> > the question is: Is there a clever way to collect all those feed items
>> > without exhausting the server
>> > any Ideas
>>
>> I designed a job queuing system a while back when I had a similar problem.
>> You can read about it here: http://stut.net/2009/05/29/php-job-queue/.
>> Set that type of system up and add a job for each feed, set to run every 15
>> minutes. You can then watch the server and tune the number of concurrent job
>> processors so you get the optimum balance between load and speed.
>>
>> -Stuart
>>
>> --
>> Stuart Dallas
>> 3ft9 Ltd
>> http://3ft9.com/
>> --
>> PHP General Mailing List (http://www.php.net/)
>> To unsubscribe, visit: http://www.php.net/unsub.php
>>
>>
>

Re: [PHP] PHP cron job optimization

[Igor Escobar]

Use PHP threads. Do the job separately.. in parts... in other words... you
can't read all them at once.

You can read a little more about php multithreading here:
http://blog.motane.lu/2009/01/02/multithreading-in-php/

You can use a non-relational database like mongo or couchdb to manage where
you stop and where you have to look back to the RSS feed as well.

[]'s

Regards,
Igor Escobar
*Software Engineer
*
+ http://blog.igorescobar.com
+ http://www.igorescobar.com
+ @igorescobar 





On Sat, Sep 10, 2011 at 10:37 PM, Stuart Dallas  wrote:

> On 10 Sep 2011, at 09:35, muad shibani wrote:
>
> > I want to design an application that reads news from RSS sources.
> > I have about 1000 RSS feed to collect from.
> >
> > I also will use Cron jobs every 15 minutes to collect the data.
> > the question is: Is there a clever way to collect all those feed items
> > without exhausting the server
> > any Ideas
>
> I designed a job queuing system a while back when I had a similar problem.
> You can read about it here: http://stut.net/2009/05/29/php-job-queue/. Set
> that type of system up and add a job for each feed, set to run every 15
> minutes. You can then watch the server and tune the number of concurrent job
> processors so you get the optimum balance between load and speed.
>
> -Stuart
>
> --
> Stuart Dallas
> 3ft9 Ltd
> http://3ft9.com/
> --
> PHP General Mailing List (http://www.php.net/)
> To unsubscribe, visit: http://www.php.net/unsub.php
>
>

Re: [PHP] PHP cron job optimization

[Stuart Dallas]

On 10 Sep 2011, at 09:35, muad shibani wrote:

> I want to design an application that reads news from RSS sources.
> I have about 1000 RSS feed to collect from.
> 
> I also will use Cron jobs every 15 minutes to collect the data.
> the question is: Is there a clever way to collect all those feed items
> without exhausting the server
> any Ideas

I designed a job queuing system a while back when I had a similar problem. You 
can read about it here: http://stut.net/2009/05/29/php-job-queue/. Set that 
type of system up and add a job for each feed, set to run every 15 minutes. You 
can then watch the server and tune the number of concurrent job processors so 
you get the optimum balance between load and speed.

-Stuart

-- 
Stuart Dallas
3ft9 Ltd
http://3ft9.com/
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] PHP cron job optimization

[Eric Butera]

On Sat, Sep 10, 2011 at 1:47 PM, Sean Greenslade  wrote:
> On Sat, Sep 10, 2011 at 4:35 AM, muad shibani wrote:
>
>> I want to design an application that reads news from RSS sources.
>> I have about 1000 RSS feed to collect from.
>>
>> I also will use Cron jobs every 15 minutes to collect the data.
>> the question is: Is there a clever way to collect all those feed items
>> without exhausting the server
>> any Ideas
>> Thank you in advance
>>
>
> Do them one at a time. Fetching web pages isn't a particularly taxing job
> for any decent server.
>
> My advice is just to try it. Even if you solution isn't 'elegant' or
> 'clever,' if it works and doesn't bog down the server, that's a win in my
> book.
>
> --
> --Zootboy
>
> Sent from my PC.
>

Traversing the internet is one of the most expensive IO operations
that can be performed.  The OP should try and split the work into
several processes and utilize async connections via a mechanism like
curl_multi [1].  Doing 1,000 connections one at a time is abysmal if
you're blocked for even 3 seconds per connection.

I'd work by making a two step process that has one cron job for
constantly fetching/updating feeds into a store & then another process
that constantly checks for updates inside aforementioned store.

[1] http://us3.php.net/manual/en/function.curl-multi-add-handle.php

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] PHP cron job optimization

[MUAD SHIBANI]

thanks a lot after I test it I will share the code .. Thanks again

On Sat, Sep 10, 2011 at 11:20 AM, Gary Golden  wrote:

> > I want to design an application that reads news from RSS sources.
> >> I have about 1000 RSS feed to collect from.
> >>
> >> I also will use Cron jobs every 15 minutes to collect the data.
> >> the question is: Is there a clever way to collect all those feed items
> >> without exhausting the server
> >> any Ideas
> >> Thank you in advance
> Just watch your memory and you'll be fine.
> As was stated fetching an rss feed is fast and cheap, so I would
> contrariwise think about  paralleling if I would you.
>
>


-- 
*
___

Muad Shibani*
*
*
Aden Yemen
Mobile: 00967 733045678
Mobile: 00967 711811232

www.muadshibani.com

Re: [PHP] PHP cron job optimization

[Gary Golden]

> I want to design an application that reads news from RSS sources.
>> I have about 1000 RSS feed to collect from.
>>
>> I also will use Cron jobs every 15 minutes to collect the data.
>> the question is: Is there a clever way to collect all those feed items
>> without exhausting the server
>> any Ideas
>> Thank you in advance
Just watch your memory and you'll be fine.
As was stated fetching an rss feed is fast and cheap, so I would
contrariwise think about  paralleling if I would you.



signature.asc
Description: OpenPGP digital signature

Re: [PHP] PHP cron job optimization

[Sean Greenslade]

On Sat, Sep 10, 2011 at 4:35 AM, muad shibani wrote:

> I want to design an application that reads news from RSS sources.
> I have about 1000 RSS feed to collect from.
>
> I also will use Cron jobs every 15 minutes to collect the data.
> the question is: Is there a clever way to collect all those feed items
> without exhausting the server
> any Ideas
> Thank you in advance
>

Do them one at a time. Fetching web pages isn't a particularly taxing job
for any decent server.

My advice is just to try it. Even if you solution isn't 'elegant' or
'clever,' if it works and doesn't bog down the server, that's a win in my
book.

-- 
--Zootboy

Sent from my PC.