[Qgis-user] alternatives to join attributes
Good morning. I have a lot of data I need to visualize in QGIS (142 time steps, each with over 200,000 points). I need a way to quickly and easily load this data in QGIS. All of my data is in separate .csv files (one file per time step). I load each file via the csv plug-in and then use join attributes to connect my csv data to the polygon data that defines my grid. Join attributes takes forever (trying to do 4,000 points took an hour). Any one know of any other, faster way to do this? Or a way to speed up my join attributes? I know 200,000 points is a lot to ask and can easily cut that down a bit, but an hour to do 4000 points seems very slow. At that rate, it would take 3 weeks to load the entire time series into QGIS. I'm running the latest version of QGIS (Mimas) on a Mac OsX. Thanks, Katie ___ Qgis-user mailing list Qgis-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-user
Re: [Qgis-user] alternatives to join attributes
Have you thought about loading the data into PostGIS? You would load the data into the db and then be able to query and display only the data that you want at any time. David. On Thu, Dec 17, 2009 at 8:46 AM, Carbonari, Katie (IS) katie.carbon...@ngc.com wrote: Good morning. I have a lot of data I need to visualize in QGIS (142 time steps, each with over 200,000 points). I need a way to quickly and easily load this data in QGIS. All of my data is in separate .csv files (one file per time step). I load each file via the csv plug-in and then use join attributes to connect my csv data to the polygon data that defines my grid. Join attributes takes forever (trying to do 4,000 points took an hour). Any one know of any other, faster way to do this? Or a way to speed up my join attributes? I know 200,000 points is a lot to ask and can easily cut that down a bit, but an hour to do 4000 points seems very slow. At that rate, it would take 3 weeks to load the entire time series into QGIS. I'm running the latest version of QGIS (Mimas) on a Mac OsX. Thanks, Katie ___ Qgis-user mailing list Qgis-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-user ___ Qgis-user mailing list Qgis-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-user
Re: [Qgis-user] alternatives to join attributes
Further to the same question: is there a really good PostGIS for Dummies type of explanation of how to get information into a PostGIS database? Google's bringing up some hits, but not one good reference that has the information I'm looking for. Scott On Thu, Dec 17, 2009 at 10:10 AM, Carbonari, Katie (IS) katie.carbon...@ngc.com wrote: I tried doing that at one point, but I've never used databases before and didn't get how to include them into QGIS. I tried going under Add PostGIS Table but wasn't really sure how to connect to my database. I have my dbf files stored locally on my machine, yet the gui wants me to make a PostgreSQL Connection; what exactly is that and is there a more simple way to access my dbf files? Thanks, katie -Original Message- From: David Fawcett [mailto:david.fawc...@gmail.comdavid.fawc...@gmail.com ] Sent: Thu 12/17/2009 9:57 AM To: Carbonari, Katie (IS) Cc: qgis-user@lists.osgeo.org Subject: Re: [Qgis-user] alternatives to join attributes Have you thought about loading the data into PostGIS? You would load the data into the db and then be able to query and display only the data that you want at any time. David. On Thu, Dec 17, 2009 at 8:46 AM, Carbonari, Katie (IS) katie.carbon...@ngc.com wrote: Good morning. I have a lot of data I need to visualize in QGIS (142 time steps, each with over 200,000 points). I need a way to quickly and easily load this data in QGIS. All of my data is in separate .csv files (one file per time step). I load each file via the csv plug-in and then use join attributes to connect my csv data to the polygon data that defines my grid. Join attributes takes forever (trying to do 4,000 points took an hour). Any one know of any other, faster way to do this? Or a way to speed up my join attributes? I know 200,000 points is a lot to ask and can easily cut that down a bit, but an hour to do 4000 points seems very slow. At that rate, it would take 3 weeks to load the entire time series into QGIS. I'm running the latest version of QGIS (Mimas) on a Mac OsX. Thanks, Katie ___ Qgis-user mailing list Qgis-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-user ___ Qgis-user mailing list Qgis-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-user -- Scott Rollins, Virginia Beach, VA organ...@gmail.com ___ Qgis-user mailing list Qgis-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-user
Re: [Qgis-user] alternatives to join attributes
Am 17.12.2009 15:46, schrieb Carbonari, Katie (IS): Good morning. I have a lot of data I need to visualize in QGIS (142 time steps, each with over 200,000 points). I need a way to quickly and easily load this data in QGIS. All of my data is in separate .csv files (one file per time step). I load each file via the csv plug-in and then use join attributes to connect my csv data to the polygon data that defines my grid. Join attributes takes forever (trying to do 4,000 points took an hour). Any one know of any other, faster way to do this? Or a way to speed up my join attributes? I know 200,000 points is a lot to ask and can easily cut that down a bit, but an hour to do 4000 points seems very slow. At that rate, it would take 3 weeks to load the entire time series into QGIS. I'm running the latest version of QGIS (Mimas) on a Mac OsX. Good afternoon ;-) I think with this amount of data is would be the best method to work with a spatially enhanced database - most prominently PostgreSQL/PostGIS and to do all the datamanagement there - that is the reason why databases exist... So, 1. export your polygon data to Postgres/PostGIS 2. push your csv data also into the database 3. create the necessary objects as tables from point 1+2 4. map it in qgis The same applies if you do it with spatiallite, which is also possible, but with that databasesystem I have no expierence... Hope this helps, greetings, Albin -- - | Albin Blaschka, Mag. rer.nat - Salzburg, Austria | http://www.albinblaschka.info http://www.thinkanimal.info | It's hard to live in the mountains, hard, but not hopeless! - ___ Qgis-user mailing list Qgis-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-user
Re: [Qgis-user] alternatives to join attributes
Am 17.12.2009 16:16, schrieb Scott Rollins: Further to the same question: is there a really good PostGIS for Dummies type of explanation of how to get information into a PostGIS database? Google's bringing up some hits, but not one good reference that has the information I'm looking for. Maybe something like this? http://www.bostongis.com/ and, more specific http://www.bostongis.com/PrinterFriendly.aspx?content_name=postgis_tut01 HTH, Albin -- - | Albin Blaschka, Mag. rer.nat - Salzburg, Austria | http://www.albinblaschka.info http://www.thinkanimal.info | It's hard to live in the mountains, hard, but not hopeless! - ___ Qgis-user mailing list Qgis-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-user
Re: [Qgis-user] alternatives to join attributes
and http://revenant.ca/www/postgis/workshop/ Postgis is really the way to go for you. You can easily join your data within Postgis with views. And you can do query/analysis in other tools besides QGIS, also in plain SQL. Andreas On Thu, December 17, 2009 4:21 pm, Albin Blaschka wrote: Am 17.12.2009 16:16, schrieb Scott Rollins: Further to the same question: is there a really good PostGIS for Dummies type of explanation of how to get information into a PostGIS database? Google's bringing up some hits, but not one good reference that has the information I'm looking for. Maybe something like this? http://www.bostongis.com/ and, more specific http://www.bostongis.com/PrinterFriendly.aspx?content_name=postgis_tut01 HTH, Albin -- - | Albin Blaschka, Mag. rer.nat - Salzburg, Austria | http://www.albinblaschka.info http://www.thinkanimal.info | It's hard to live in the mountains, hard, but not hopeless! - ___ Qgis-user mailing list Qgis-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-user -- Andreas Neumann http://www.carto.net/neumann/ http://www.svgopen.org/ ___ Qgis-user mailing list Qgis-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-user
Re: [Qgis-user] alternatives to join attributes
On Thu, Dec 17, 2009 at 10:19 AM, Edmond, Brian S brianedm...@missouristate.edu wrote: Katie, Your foray into PostgreSQL and PostGIS will not be wasted if you plan to do more GIS in the future. Even if you do it only for this project, it should help you tremendously. Katie, It can be a little daunting to figure out where to start with PostGIS, but when you are dealing with large amounts of spatial features and related attribute records, moving to a spatially-enabled relational database is really the best solution. Like Brian said, learning PostGIS will not be wasted time. Even if you end up using a different spatial database in the future, most of the knowledge transfers well. If you aren't already using William Kyngesburye's GIS frameworks and binaries for the Mac, I recommend it. It is quite a reasonable way to install QGIS, PostGIS, MapServer, and all of the software dependencies that they need. http://www.kyngchaos.com/macosx:index http://www.kyngchaos.com/macosx:build:index You can download the PostGIS workshop from FOSS4G 2009 at: http://2009.foss4g.org/workshops/#workshop_03 David. ___ Qgis-user mailing list Qgis-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-user
RE: [Qgis-user] alternatives to join attributes
Katie, Your foray into PostgreSQL and PostGIS will not be wasted if you plan to do more GIS in the future. Even if you do it only for this project, it should help you tremendously. I routinely plot 7K+ unique points as a PostGIS layer using QGIS and the time to plot them is almost instantaneous. Furthermore, a whole world of spatial capabilities will open to you with all of the PostGIS functions available to do analysis on your data. I do have an iMac (at work), but I run my setup on a Kubuntu machine (at home) so I might be able to help you a little bit there. Start by installing PostgreSQL and getting some familiarity with it. It will help if you know SQL and have worked with databases before (they are all essentially the same from a user POV). The desktop tool pgAdmin will help you guify PostgreSQL a bit, but you'll still need to do a few things manually or from the command line. I checked and there is a Mac OS port for pgAdmin. (I prefer using phpPgAdmin, but it would require that you have a web server and PHP installed on your Mac.) Good luck! B From: qgis-user-boun...@lists.osgeo.org [mailto:qgis-user-boun...@lists.osgeo.org] On Behalf Of Carbonari, Katie (IS) Sent: Thursday, December 17, 2009 9:11 AM To: David Fawcett Cc: qgis-user@lists.osgeo.org Subject: RE: [Qgis-user] alternatives to join attributes I tried doing that at one point, but I've never used databases before and didn't get how to include them into QGIS. I tried going under Add PostGIS Table but wasn't really sure how to connect to my database. I have my dbf files stored locally on my machine, yet the gui wants me to make a PostgreSQL Connection; what exactly is that and is there a more simple way to access my dbf files? Thanks, katie -Original Message- From: David Fawcett [mailto:david.fawc...@gmail.com] Sent: Thu 12/17/2009 9:57 AM To: Carbonari, Katie (IS) Cc: qgis-user@lists.osgeo.org Subject: Re: [Qgis-user] alternatives to join attributes Have you thought about loading the data into PostGIS? You would load the data into the db and then be able to query and display only the data that you want at any time. David. On Thu, Dec 17, 2009 at 8:46 AM, Carbonari, Katie (IS) katie.carbon...@ngc.com wrote: Good morning. I have a lot of data I need to visualize in QGIS (142 time steps, each with over 200,000 points). I need a way to quickly and easily load this data in QGIS. All of my data is in separate .csv files (one file per time step). I load each file via the csv plug-in and then use join attributes to connect my csv data to the polygon data that defines my grid. Join attributes takes forever (trying to do 4,000 points took an hour). Any one know of any other, faster way to do this? Or a way to speed up my join attributes? I know 200,000 points is a lot to ask and can easily cut that down a bit, but an hour to do 4000 points seems very slow. At that rate, it would take 3 weeks to load the entire time series into QGIS. I'm running the latest version of QGIS (Mimas) on a Mac OsX. Thanks, Katie ___ Qgis-user mailing list Qgis-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-user ___ Qgis-user mailing list Qgis-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-user
