Re: [PHP] Pulling from Multiple Databases
-- purposely top posting as well -- just think, here in ontario, we have UBB... where Bell is proposing a 25 cap on usage... you sir, are killing my usage! :) On Tue, 2011-02-01 at 12:18 -0700, Alexis wrote: > Bloody Hell!! > > How many lines is the footer in your email response!!! > > I make it almost ten times longer than the reply itselftalk about > abominable netiquette, and I have purposely put this response at the top > as after all footers do go at the bottom of an email :) > > Do you even NEED a footer with nothing but inane comments in it? > > Alexis > On 01/02/11 11:54, David Hutto wrote: > > I'd pass the db's to a threaded function that processes each db's info > > in an algorithmic order. > > > > -- The lawyer in me says argue...even if you're wrong. The scientist in > > me... says shut up, listen, and then argue. But the lawyer won on > > appeal, so now I have to argue due to a court order. Furthermore, if you > > could be a scientific celebrity, would you want einstein sitting around > > with you on saturday morning, while you're sitting in your undies, > > watching Underdog?...Or better yet, would Einstein want you to violate > > his Underdog time? Can you imagine Einstein sitting around in his > > underware? Thinking about the relativity between his pubic nardsac, and > > his Fruit of the Looms, while knocking a few Dorito's crumbs off his > > inner brilliant white thighs, and hailing E = mc**2, and licking the > > orangy, delicious, Doritoey crust that layered his genetically rippled > > fingertips? But then again, J. Edgar Hoover would want his pantyhose > > intertwined within the equation. However, I digress, momentarily. But > > Einstein gave freely, for humanity, not for gain, other than personal > > freedom. An equation that benefited all, and yet gain is a personal > > product. Also, if you can answer it, is gravity anymore than > > interplanetary static cling? > > -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: > > http://www.php.net/unsub.php > -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Pulling from Multiple Databases
> Do you even NEED a footer with nothing but inane comments in it? > It's probably an overcompensation for a small penis. > Alexis > On 01/02/11 11:54, David Hutto wrote: >> I'd pass the db's to a threaded function that processes each db's info >> in an algorithmic order. >> >> -- The lawyer in me says argue...even if you're wrong. The scientist in >> me... says shut up, listen, and then argue. But the lawyer won on >> appeal, so now I have to argue due to a court order. Furthermore, if you >> could be a scientific celebrity, would you want einstein sitting around >> with you on saturday morning, while you're sitting in your undies, >> watching Underdog?...Or better yet, would Einstein want you to violate >> his Underdog time? Can you imagine Einstein sitting around in his >> underware? Thinking about the relativity between his pubic nardsac, and >> his Fruit of the Looms, while knocking a few Dorito's crumbs off his >> inner brilliant white thighs, and hailing E = mc**2, and licking the >> orangy, delicious, Doritoey crust that layered his genetically rippled >> fingertips? But then again, J. Edgar Hoover would want his pantyhose >> intertwined within the equation. However, I digress, momentarily. But >> Einstein gave freely, for humanity, not for gain, other than personal >> freedom. An equation that benefited all, and yet gain is a personal >> product. Also, if you can answer it, is gravity anymore than >> interplanetary static cling? >> -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: >> http://www.php.net/unsub.php > > -- > PHP General Mailing List (http://www.php.net/) > To unsubscribe, visit: http://www.php.net/unsub.php > > -- The lawyer in me says argue...even if you're wrong. The scientist in me... says shut up, listen, and then argue. But the lawyer won on appeal, so now I have to argue due to a court order. Furthermore, if you could be a scientific celebrity, would you want einstein sitting around with you on saturday morning, while you're sitting in your undies, watching Underdog?...Or better yet, would Einstein want you to violate his Underdog time? Can you imagine Einstein sitting around in his underware? Thinking about the relativity between his pubic nardsac, and his Fruit of the Looms, while knocking a few Dorito's crumbs off his inner brilliant white thighs, and hailing E = mc**2, and licking the orangy, delicious, Doritoey crust that layered his genetically rippled fingertips? But then again, J. Edgar Hoover would want his pantyhose intertwined within the equation. However, I digress, momentarily. But Einstein gave freely, for humanity, not for gain, other than personal freedom. An equation that benefited all, and yet gain is a personal product. Also, if you can answer it, is gravity anymore than interplanetary static cling? -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Pulling from Multiple Databases
Bloody Hell!! How many lines is the footer in your email response!!! I make it almost ten times longer than the reply itselftalk about abominable netiquette, and I have purposely put this response at the top as after all footers do go at the bottom of an email :) Do you even NEED a footer with nothing but inane comments in it? Alexis On 01/02/11 11:54, David Hutto wrote: > I'd pass the db's to a threaded function that processes each db's info > in an algorithmic order. > > -- The lawyer in me says argue...even if you're wrong. The scientist in > me... says shut up, listen, and then argue. But the lawyer won on > appeal, so now I have to argue due to a court order. Furthermore, if you > could be a scientific celebrity, would you want einstein sitting around > with you on saturday morning, while you're sitting in your undies, > watching Underdog?...Or better yet, would Einstein want you to violate > his Underdog time? Can you imagine Einstein sitting around in his > underware? Thinking about the relativity between his pubic nardsac, and > his Fruit of the Looms, while knocking a few Dorito's crumbs off his > inner brilliant white thighs, and hailing E = mc**2, and licking the > orangy, delicious, Doritoey crust that layered his genetically rippled > fingertips? But then again, J. Edgar Hoover would want his pantyhose > intertwined within the equation. However, I digress, momentarily. But > Einstein gave freely, for humanity, not for gain, other than personal > freedom. An equation that benefited all, and yet gain is a personal > product. Also, if you can answer it, is gravity anymore than > interplanetary static cling? > -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: > http://www.php.net/unsub.php -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Pulling from Multiple Databases
I'd pass the db's to a threaded function that processes each db's info in an algorithmic order. -- The lawyer in me says argue...even if you're wrong. The scientist in me... says shut up, listen, and then argue. But the lawyer won on appeal, so now I have to argue due to a court order. Furthermore, if you could be a scientific celebrity, would you want einstein sitting around with you on saturday morning, while you're sitting in your undies, watching Underdog?...Or better yet, would Einstein want you to violate his Underdog time? Can you imagine Einstein sitting around in his underware? Thinking about the relativity between his pubic nardsac, and his Fruit of the Looms, while knocking a few Dorito's crumbs off his inner brilliant white thighs, and hailing E = mc**2, and licking the orangy, delicious, Doritoey crust that layered his genetically rippled fingertips? But then again, J. Edgar Hoover would want his pantyhose intertwined within the equation. However, I digress, momentarily. But Einstein gave freely, for humanity, not for gain, other than personal freedom. An equation that benefited all, and yet gain is a personal product. Also, if you can answer it, is gravity anymore than interplanetary static cling? -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Pulling from Multiple Databases
Bah, forgot to reply-all On Tue, Feb 1, 2011 at 11:59 AM, Jon Hood wrote: > Using pcntl_fork mostly accomplished what I wanted (I had to go back and > create the actual connection in each of the forked processes, otherwise, the > first process that ended would destroy the connection to the main DB). It > seems to be working, and while it's not the most elegant solution, and > certainly not the best performance, it's all I can do for now. > > Any chance I can bring up the topic of native multithreading support? What > are the chances of feature request > http://bugs.php.net/bug.php?id=46919getting added to a roadmap for, say, PHP > version 7? > > -Jon >
Re: [PHP] Pulling from Multiple Databases
On 1 February 2011 16:39, Jon Hood wrote: > (comments in-line) > > On Tue, Feb 1, 2011 at 10:34 AM, Richard Quadling > wrote: >> >> I use a data warehouse (a semi denormalized db) to hold data from >> around 200 different data sources (DB, Excel spreadsheets, Web, etc.) >> >> I use multiple scripts to update the DB, each one tuned to a >> particular frequency. >> > > A different script for each database is a possibility. It's a little extra > load on the app server, but it should be able to handle it. Maybe with > pcntl_fork? I haven't explored this option much. > >> >> My main app's queries are always against the data warehouse. >> >> That way, the live front end isn't worried about getting the source data. >> >> If the data is needed live, then I'd be looking to see if I can get a >> live data feed from the source system. Essentially a push of the data >> to my data warehouse - or a staging structure to allow locally defined >> triggers to clean/process the data upon arrival. >> >> Automation is the key for me here. Rather than trying to do everything >> for the request, respond to the changes in the data or live with the >> fact that the data may be stale. >> >> Can you give us any clues as to the sort of app you are building? The >> sort of data you are working on? Are you running your own servers? > > Data are needed live. 3 of the databases are MySQL. 14 are XML files that > change frequently. 3 are JSON. 1 is Microsoft SQL Server 2005. The main app > is running on linux (distribution doesn't matter - currently Debian, but I > can change it to whatever if there's a reason). Most is financial data that > needs ordered. > > I'm going to explore the pcntl_fork option some more... > > Thanks! > Jon > If you are in control of the data, there are some things that may be useful. 1 - For tables that require syncing, I've added a timestamp column. This is an automatically updated column whenever the data changes. In the code handling the sync, I know that I don't need to retrieve any data if the most recent timestamp is the same as the one I last got. 2 - For physical files, and assuming that the last modified datetimestamp is maintained, then again, you have an indicator to know if you need to actually process any data. 3 - For JSON ... if it is coming to you over the web, check headers to see if the server is providing you a cached version. You may also be able to save yourself the processing time if you know you've got a stale response. In a best case scenario, you poll all the source, realize that none of them have any new data and you supply the data you already have (caching the data is pretty much essential). In a worse case scenario, you have to wait until all the data is polled and stored. Probably no worse than you are already at. Richard. -- Richard Quadling Twitter : EE : Zend @RQuadling : e-e.com/M_248814.html : bit.ly/9O8vFY -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Pulling from Multiple Databases
On Tue, Feb 1, 2011 at 10:59 AM, Jon Hood wrote: > I have a website that is currently pulling from more than 30 databases, > combining the data, and displaying it to the user. As more and more > databases are added, the script continues to get slower and slower, and > I've > realized that I need to either find a way to pull these data in parallel. > So > - what is the preferred method of pulling data from multiple locations in > parallel? > > Thanks, > Jon > Well, you could turn the calls into REST-based web requests that produce json. Then you could use curl_multi to grab the results in parallel and quick decode the json. Adam P.S. - Sorry for the duplicate, Jon, I forgot to copy the list. -- Nephtali: A simple, flexible, fast, and security-focused PHP framework http://nephtaliproject.com
Re: [PHP] Pulling from Multiple Databases
(comments in-line) On Tue, Feb 1, 2011 at 10:34 AM, Richard Quadling wrote: > I use a data warehouse (a semi denormalized db) to hold data from > around 200 different data sources (DB, Excel spreadsheets, Web, etc.) > > I use multiple scripts to update the DB, each one tuned to a > particular frequency. > > A different script for each database is a possibility. It's a little extra load on the app server, but it should be able to handle it. Maybe with pcntl_fork? I haven't explored this option much. > My main app's queries are always against the data warehouse. > > That way, the live front end isn't worried about getting the source data. > > If the data is needed live, then I'd be looking to see if I can get a > live data feed from the source system. Essentially a push of the data > to my data warehouse - or a staging structure to allow locally defined > triggers to clean/process the data upon arrival. > > Automation is the key for me here. Rather than trying to do everything > for the request, respond to the changes in the data or live with the > fact that the data may be stale. > > Can you give us any clues as to the sort of app you are building? The > sort of data you are working on? Are you running your own servers? > Data are needed live. 3 of the databases are MySQL. 14 are XML files that change frequently. 3 are JSON. 1 is Microsoft SQL Server 2005. The main app is running on linux (distribution doesn't matter - currently Debian, but I can change it to whatever if there's a reason). Most is financial data that needs ordered. I'm going to explore the pcntl_fork option some more... Thanks! Jon
Re: [PHP] Pulling from Multiple Databases
On 1 February 2011 15:59, Jon Hood wrote: > I have a website that is currently pulling from more than 30 databases, > combining the data, and displaying it to the user. As more and more > databases are added, the script continues to get slower and slower, and I've > realized that I need to either find a way to pull these data in parallel. So > - what is the preferred method of pulling data from multiple locations in > parallel? > > Thanks, > Jon > I use a data warehouse (a semi denormalized db) to hold data from around 200 different data sources (DB, Excel spreadsheets, Web, etc.) I use multiple scripts to update the DB, each one tuned to a particular frequency. My main app's queries are always against the data warehouse. That way, the live front end isn't worried about getting the source data. If the source data changes frequently, then you can poll more frequently. I'm on Windows and the Windows Scheduler works great for me. If the load on the machine during the polling is high, then offload it to a backend machine. No need for this machine to be forward facing. For the straight SQL sources, I'm looking at using the data warehouse itself (SQL Server 2008 R2) to use it's own job server to handle the data retrieval. That way, all the "data processing" is in the SQL server, rather than in a load of scripts. For those sources like Excel, then there are data conversion tools I can use - Excel exists in the ODBC space, so I can use that as another data source (I think - I've not tried this). But whatever I do, I don't try to get live data to the app on every request. It simply takes way too much time. If the data is needed live, then I'd be looking to see if I can get a live data feed from the source system. Essentially a push of the data to my data warehouse - or a staging structure to allow locally defined triggers to clean/process the data upon arrival. Automation is the key for me here. Rather than trying to do everything for the request, respond to the changes in the data or live with the fact that the data may be stale. Can you give us any clues as to the sort of app you are building? The sort of data you are working on? Are you running your own servers? -- Richard Quadling Twitter : EE : Zend @RQuadling : e-e.com/M_248814.html : bit.ly/9O8vFY -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] Pulling from Multiple Databases
[snip] I have a website that is currently pulling from more than 30 databases, combining the data, and displaying it to the user. As more and more databases are added, the script continues to get slower and slower, and I've realized that I need to either find a way to pull these data in parallel. So - what is the preferred method of pulling data from multiple locations in parallel? [/snip] Stage the data in a view or views. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
