Re: [mapserver-users] Best way to import 4.5TB of imagery?
Hi Stephen, Thanks, for the reply. I previously got 4 sample images from the USDA, and was able to get them to work just fine. There was no processing required. The sample images I got were all from Utah, and they are NAD83, UTM zone 12. I added the 4 sample images to a shape file using gdaltindex. I used UPSG 26912 and mapserver served them up very quickly for such large files. Now I have this entire data set, and it stretches from UTM zone 10, to UTM zone 19. The data is divided into directories by two letter state abbreviations, and under that into subdirectories. I'm just wondering how to add this to my mapfile. Do I need a different entry for each UTM Zone? How is it possible to get a single layer entry that includes multiple projections? This is looking like a huge job and I just want to know the best strategy for getting this done. Thanks, James -Original Message- From: mapserver-users-boun...@lists.osgeo.org [mailto:mapserver-users-boun...@lists.osgeo.org] On Behalf Of Stephen Woodbridge Sent: Friday, June 07, 2013 8:41 AM To: mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] Best way to import 4.5TB of imagery? On 6/7/2013 10:31 AM, James_in_Utah wrote: Hi, We just got 3 hard drive, loaded with 4.5TB of NAIP imagery for all of CONUS. I think there's a total of about 400,000 jpgs. The data is in directories, by states. Under each state, there are subfolders, probably reference by longitude. Other than going through folder by folder, adding each image to a shape file using gdaltindex, what's the best strategy for loading a couple of hundred thousand files up to our server and making the imagery available via our mapserver? Should I maintain the current directory structure when I copy the imagery to the server, or just dump all of it into a single directory? Do I want to stay with 1 shape file, or break it up by state? We eventually want a contiguous layer for all of CONUS to be served up to our users. James, Since imagery data is served via gdal, you might want to also ask this question on the gdal list. There are issues with jpg related to the fact that if you only want a small part of the image you still have to uncompress the whole image. So part of the answer might be that you need to pre-process all the imagery into something like a jpg compress tiled geotif or something else. You also need to consider what projection your imagery is in and what projection you want to display it in. Because if you need to preprocess the data, that would also be a good time to reproject it. Anyway the gdal list can probably ask additional questions to help sort all that out. -Steve W ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users smime.p7s Description: S/MIME cryptographic signature ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Best way to import 4.5TB of imagery?
On 6/10/2013 12:57 PM, Evans, James R Civ USAF ACC 84 RADES/SCZE wrote: Hi Stephen, Thanks, for the reply. I previously got 4 sample images from the USDA, and was able to get them to work just fine. There was no processing required. The sample images I got were all from Utah, and they are NAD83, UTM zone 12. I added the 4 sample images to a shape file using gdaltindex. I used UPSG 26912 and mapserver served them up very quickly for such large files. Now I have this entire data set, and it stretches from UTM zone 10, to UTM zone 19. The data is divided into directories by two letter state abbreviations, and under that into subdirectories. I'm just wondering how to add this to my mapfile. Do I need a different entry for each UTM Zone? How is it possible to get a single layer entry that includes multiple projections? This is looking like a huge job and I just want to know the best strategy for getting this done. So now you have a problem. You data is in UTM spread over 10 different projections. What do you plane to do when have your image is zone 10 and half is in zone 11 or if you zoom out and you images has 3-4 zones displayed? All data in an image must be rendered in the same projection. While I don't doubt that your test with 4 images worked fine, did you you test this a multiple zoom levels and at some point you will probably want to create a super overlay image so you do not have to open multiple files to just pull a tiny overlay out of each one. Your use cases will determine how you want to deal with the data. For example does it HAVE to be in a UTM projection, or can you work with a Spherical Mercator or geographic projection? The end solution will be much easier if you can work with one common projection over your whole data set. Otherwise, you will have to deal with the transitions from one zone to the next or maybe set up 10 separate servers that only serve one zone. Having pushed larges amounts (4-25TB) of imagery data more than once it is important to make these decisions up front and and prototype up something like a 4-10 degree square across a UTM boundary and make sure that the results are going to be what you expect before you process all the data. -Steve Thanks, James -Original Message- From: mapserver-users-boun...@lists.osgeo.org [mailto:mapserver-users-boun...@lists.osgeo.org] On Behalf Of Stephen Woodbridge Sent: Friday, June 07, 2013 8:41 AM To: mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] Best way to import 4.5TB of imagery? On 6/7/2013 10:31 AM, James_in_Utah wrote: Hi, We just got 3 hard drive, loaded with 4.5TB of NAIP imagery for all of CONUS. I think there's a total of about 400,000 jpgs. The data is in directories, by states. Under each state, there are subfolders, probably reference by longitude. Other than going through folder by folder, adding each image to a shape file using gdaltindex, what's the best strategy for loading a couple of hundred thousand files up to our server and making the imagery available via our mapserver? Should I maintain the current directory structure when I copy the imagery to the server, or just dump all of it into a single directory? Do I want to stay with 1 shape file, or break it up by state? We eventually want a contiguous layer for all of CONUS to be served up to our users. James, Since imagery data is served via gdal, you might want to also ask this question on the gdal list. There are issues with jpg related to the fact that if you only want a small part of the image you still have to uncompress the whole image. So part of the answer might be that you need to pre-process all the imagery into something like a jpg compress tiled geotif or something else. You also need to consider what projection your imagery is in and what projection you want to display it in. Because if you need to preprocess the data, that would also be a good time to reproject it. Anyway the gdal list can probably ask additional questions to help sort all that out. -Steve W ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Best way to import 4.5TB of imagery?
So, I'm guessing there's no easy way to automate this? Even looking at the states, some of the states are in two zones, and Texas is across 3 zones. At least the naming convention of the files indicate the UTM zone. For instance: m_2408002_ne_17_1_20100422_201001123.jp2, is in zone 17. As far as I can tell, all the files in a particular directory are all in the same UTM zone. I could create a layer for each UTM zone across CONUS, but that's not going to be particularly useful to my users. I'm thinking of making a layer for each state. For the stats that cross zones, there will probably be two layers. For Texas, there would be Texas_east, Texas_middle, and Texas_west. I will probably limit visibility until zoomed in sufficiently to see the whole state on the screen anyway, since the continental view of this data is pretty crappy anyway. So now it seems like it will be a lot of grunt work just copying these directories up to the server, and going through and creating a shape file index for each state. For states in more than one UTM, there would be more than one shape. Then I'll have to add a layer for to my mapfile for each shapefile, using the correct projection. Is there an easier way? I'm starting with Oklahoma, which is also in three UTM zones. I'll get that working before moving on. Any suggestions on making this pretty would be welcomed. :-) -Original Message- From: Stephen Woodbridge [mailto:wood...@swoodbridge.com] Sent: Monday, June 10, 2013 12:34 PM To: Evans, James R Civ USAF ACC 84 RADES/SCZE Cc: mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] Best way to import 4.5TB of imagery? On 6/10/2013 12:57 PM, Evans, James R Civ USAF ACC 84 RADES/SCZE wrote: Hi Stephen, Thanks, for the reply. I previously got 4 sample images from the USDA, and was able to get them to work just fine. There was no processing required. The sample images I got were all from Utah, and they are NAD83, UTM zone 12. I added the 4 sample images to a shape file using gdaltindex. I used UPSG 26912 and mapserver served them up very quickly for such large files. Now I have this entire data set, and it stretches from UTM zone 10, to UTM zone 19. The data is divided into directories by two letter state abbreviations, and under that into subdirectories. I'm just wondering how to add this to my mapfile. Do I need a different entry for each UTM Zone? How is it possible to get a single layer entry that includes multiple projections? This is looking like a huge job and I just want to know the best strategy for getting this done. So now you have a problem. You data is in UTM spread over 10 different projections. What do you plane to do when have your image is zone 10 and half is in zone 11 or if you zoom out and you images has 3-4 zones displayed? All data in an image must be rendered in the same projection. While I don't doubt that your test with 4 images worked fine, did you you test this a multiple zoom levels and at some point you will probably want to create a super overlay image so you do not have to open multiple files to just pull a tiny overlay out of each one. Your use cases will determine how you want to deal with the data. For example does it HAVE to be in a UTM projection, or can you work with a Spherical Mercator or geographic projection? The end solution will be much easier if you can work with one common projection over your whole data set. Otherwise, you will have to deal with the transitions from one zone to the next or maybe set up 10 separate servers that only serve one zone. Having pushed larges amounts (4-25TB) of imagery data more than once it is important to make these decisions up front and and prototype up something like a 4-10 degree square across a UTM boundary and make sure that the results are going to be what you expect before you process all the data. -Steve Thanks, James -Original Message- From: mapserver-users-boun...@lists.osgeo.org [mailto:mapserver-users-boun...@lists.osgeo.org] On Behalf Of Stephen Woodbridge Sent: Friday, June 07, 2013 8:41 AM To: mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] Best way to import 4.5TB of imagery? On 6/7/2013 10:31 AM, James_in_Utah wrote: Hi, We just got 3 hard drive, loaded with 4.5TB of NAIP imagery for all of CONUS. I think there's a total of about 400,000 jpgs. The data is in directories, by states. Under each state, there are subfolders, probably reference by longitude. Other than going through folder by folder, adding each image to a shape file using gdaltindex, what's the best strategy for loading a couple of hundred thousand files up to our server and making the imagery available via our mapserver? Should I maintain the current directory structure when I copy the imagery to the server, or just dump all of it into a single directory? Do I want to stay with 1 shape file, or break it up by state? We eventually want
Re: [mapserver-users] Best way to import 4.5TB of imagery?
On Mon, Jun 10, 2013 at 3:48 PM, Evans, James R Civ USAF ACC 84 RADES/SCZE james.ev...@hill.af.mil wrote: So, I'm guessing there's no easy way to automate this? Even looking at the states, some of the states are in two zones, and Texas is across 3 zones. At least the naming convention of the files indicate the UTM zone. James, Note it is pretty easy to write a script (ie. in Python) that would walk the directory tree and sort the images into distinct (per utm zone) collections. So, yes, the processing of this data is very automatable. For instance: m_2408002_ne_17_1_20100422_201001123.jp2, is in zone 17. As far as I can tell, all the files in a particular directory are all in the same UTM zone. I could create a layer for each UTM zone across CONUS, but that's not going to be particularly useful to my users. I'm thinking of making a layer for each state. For the stats that cross zones, there will probably be two layers. For Texas, there would be Texas_east, Texas_middle, and Texas_west. I will probably limit visibility until zoomed in sufficiently Is it important to you to distinguish things by state? If not, why not one layer per utm zone, and then join them in a layer group 'UTM NAIP' that the users would either turn on or off? to see the whole state on the screen anyway, since the continental view of this data is pretty crappy anyway. Limiting visibility to reasonable resolutions should be fine and would help you avoiding needing additional preprocessing to create an overview of the whole collection. So now it seems like it will be a lot of grunt work just copying these directories up to the server, and going through and creating a shape file index for each state. For states in more than one UTM, there would be more than one shape. Then I'll have to add a layer for to my mapfile for each shapefile, using the correct projection. Is there an easier way? I'm starting with Oklahoma, which is also in three UTM zones. I'll get that working before moving on. Any suggestions on making this pretty would be welcomed. :-) Well, I still think you should take states out of the equation unless that is important to your users. Good luck, Best regards, -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush| Geospatial Software Developer ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Best way to import 4.5TB of imagery?
On 6/10/2013 6:48 PM, Evans, James R Civ USAF ACC 84 RADES/SCZE wrote: So, I'm guessing there's no easy way to automate this? Even looking at the states, some of the states are in two zones, and Texas is across 3 zones. At least the naming convention of the files indicate the UTM zone. For instance: m_2408002_ne_17_1_20100422_201001123.jp2, is in zone 17. As far as I can tell, all the files in a particular directory are all in the same UTM zone. I could create a layer for each UTM zone across CONUS, but that's not going to be particularly useful to my users. I'm thinking of making a layer for each state. For the stats that cross zones, there will probably be two layers. For Texas, there would be Texas_east, Texas_middle, and Texas_west. I will probably limit visibility until zoomed in sufficiently to see the whole state on the screen anyway, since the continental view of this data is pretty crappy anyway. So now it seems like it will be a lot of grunt work just copying these directories up to the server, and going through and creating a shape file index for each state. For states in more than one UTM, there would be more than one shape. Then I'll have to add a layer for to my mapfile for each shapefile, using the correct projection. Is there an easier way? I'm starting with Oklahoma, which is also in three UTM zones. I'll get that working before moving on. Any suggestions on making this pretty would be welcomed. :-) If you are using something like OpenLayers for your client you can code smarts into that to change mapfiles based on the zone. I think I would start with a Perl script that takes every image file and creates a record in a postgresql/postgis database that represents the extents of the image. path/to/image/file.jpg | wgs84_geom | utm_geom | zone or something like that. Then you can create mapfiles that displays these rectangles in wgs84, or work out your logic for display the polygons in shifting zones. This is quick and easy to setup and lets you proceed with the disgn of that with waiting on imagery and it wil be faster. Beyond that, pushing around 4.5 TB of imagery is not going to be fast. Good luck, -Steve W -Original Message- From: Stephen Woodbridge [mailto:wood...@swoodbridge.com] Sent: Monday, June 10, 2013 12:34 PM To: Evans, James R Civ USAF ACC 84 RADES/SCZE Cc: mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] Best way to import 4.5TB of imagery? On 6/10/2013 12:57 PM, Evans, James R Civ USAF ACC 84 RADES/SCZE wrote: Hi Stephen, Thanks, for the reply. I previously got 4 sample images from the USDA, and was able to get them to work just fine. There was no processing required. The sample images I got were all from Utah, and they are NAD83, UTM zone 12. I added the 4 sample images to a shape file using gdaltindex. I used UPSG 26912 and mapserver served them up very quickly for such large files. Now I have this entire data set, and it stretches from UTM zone 10, to UTM zone 19. The data is divided into directories by two letter state abbreviations, and under that into subdirectories. I'm just wondering how to add this to my mapfile. Do I need a different entry for each UTM Zone? How is it possible to get a single layer entry that includes multiple projections? This is looking like a huge job and I just want to know the best strategy for getting this done. So now you have a problem. You data is in UTM spread over 10 different projections. What do you plane to do when have your image is zone 10 and half is in zone 11 or if you zoom out and you images has 3-4 zones displayed? All data in an image must be rendered in the same projection. While I don't doubt that your test with 4 images worked fine, did you you test this a multiple zoom levels and at some point you will probably want to create a super overlay image so you do not have to open multiple files to just pull a tiny overlay out of each one. Your use cases will determine how you want to deal with the data. For example does it HAVE to be in a UTM projection, or can you work with a Spherical Mercator or geographic projection? The end solution will be much easier if you can work with one common projection over your whole data set. Otherwise, you will have to deal with the transitions from one zone to the next or maybe set up 10 separate servers that only serve one zone. Having pushed larges amounts (4-25TB) of imagery data more than once it is important to make these decisions up front and and prototype up something like a 4-10 degree square across a UTM boundary and make sure that the results are going to be what you expect before you process all the data. -Steve Thanks, James -Original Message- From: mapserver-users-boun...@lists.osgeo.org [mailto:mapserver-users-boun...@lists.osgeo.org] On Behalf Of Stephen Woodbridge Sent: Friday, June 07, 2013 8:41 AM To: mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] Best way to import
Re: [mapserver-users] Best way to import 4.5TB of imagery?
Sounds all very complicated. I would advise that you choose a single projection that will work across your entire are and then re-project your imagery first, and then build your image datastore around that. Evans, James R Civ USAF ACC 84 RADES/SCZE james.ev...@hill.af.mil 11/06/2013 10:48 a.m. So, I'm guessing there's no easy way to automate this? Even looking at the states, some of the states are in two zones, and Texas is across 3 zones. At least the naming convention of the files indicate the UTM zone. For instance: m_2408002_ne_17_1_20100422_201001123.jp2, is in zone 17. As far as I can tell, all the files in a particular directory are all in the same UTM zone. I could create a layer for each UTM zone across CONUS, but that's not going to be particularly useful to my users. I'm thinking of making a layer for each state. For the stats that cross zones, there will probably be two layers. For Texas, there would be Texas_east, Texas_middle, and Texas_west. I will probably limit visibility until zoomed in sufficiently to see the whole state on the screen anyway, since the continental view of this data is pretty crappy anyway. So now it seems like it will be a lot of grunt work just copying these directories up to the server, and going through and creating a shape file index for each state. For states in more than one UTM, there would be more than one shape. Then I'll have to add a layer for to my mapfile for each shapefile, using the correct projection. Is there an easier way? I'm starting with Oklahoma, which is also in three UTM zones. I'll get that working before moving on. Any suggestions on making this pretty would be welcomed. :-) -Original Message- From: Stephen Woodbridge [mailto:wood...@swoodbridge.com] Sent: Monday, June 10, 2013 12:34 PM To: Evans, James R Civ USAF ACC 84 RADES/SCZE Cc: mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] Best way to import 4.5TB of imagery? On 6/10/2013 12:57 PM, Evans, James R Civ USAF ACC 84 RADES/SCZE wrote: Hi Stephen, Thanks, for the reply. I previously got 4 sample images from the USDA, and was able to get them to work just fine. There was no processing required. The sample images I got were all from Utah, and they are NAD83, UTM zone 12. I added the 4 sample images to a shape file using gdaltindex. I used UPSG 26912 and mapserver served them up very quickly for such large files. Now I have this entire data set, and it stretches from UTM zone 10, to UTM zone 19. The data is divided into directories by two letter state abbreviations, and under that into subdirectories. I'm just wondering how to add this to my mapfile. Do I need a different entry for each UTM Zone? How is it possible to get a single layer entry that includes multiple projections? This is looking like a huge job and I just want to know the best strategy for getting this done. So now you have a problem. You data is in UTM spread over 10 different projections. What do you plane to do when have your image is zone 10 and half is in zone 11 or if you zoom out and you images has 3-4 zones displayed? All data in an image must be rendered in the same projection. While I don't doubt that your test with 4 images worked fine, did you you test this a multiple zoom levels and at some point you will probably want to create a super overlay image so you do not have to open multiple files to just pull a tiny overlay out of each one. Your use cases will determine how you want to deal with the data. For example does it HAVE to be in a UTM projection, or can you work with a Spherical Mercator or geographic projection? The end solution will be much easier if you can work with one common projection over your whole data set. Otherwise, you will have to deal with the transitions from one zone to the next or maybe set up 10 separate servers that only serve one zone. Having pushed larges amounts (4-25TB) of imagery data more than once it is important to make these decisions up front and and prototype up something like a 4-10 degree square across a UTM boundary and make sure that the results are going to be what you expect before you process all the data. -Steve Thanks, James -Original Message- From: mapserver-users-boun...@lists.osgeo.org [mailto:mapserver-users-boun...@lists.osgeo.org] On Behalf Of Stephen Woodbridge Sent: Friday, June 07, 2013 8:41 AM To: mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] Best way to import 4.5TB of imagery? On 6/7/2013 10:31 AM, James_in_Utah wrote: Hi, We just got 3 hard drive, loaded with 4.5TB of NAIP imagery for all of CONUS. I think there's a total of about 400,000 jpgs. The data is in directories, by states. Under each state, there are subfolders, probably reference by longitude. Other than going through folder by folder, adding each image to a shape file using gdaltindex, what's the best strategy for loading a couple of hundred thousand
Re: [mapserver-users] Best way to import 4.5TB of imagery?
On 6/10/2013 7:07 PM, Robert Sanson wrote: Sounds all very complicated. I would advise that you choose a single projection that will work across your entire are and then re-project your imagery first, and then build your image datastore around that. I would agree with this that if htere are not other contraints on the problem this is often the best way to go. But that will not owrk if his users HAVE to have the data in UTM projection for some reason. -Steve Evans, James R Civ USAF ACC 84 RADES/SCZE james.ev...@hill.af.mil 11/06/2013 10:48 a.m. So, I'm guessing there's no easy way to automate this? Even looking at the states, some of the states are in two zones, and Texas is across 3 zones. At least the naming convention of the files indicate the UTM zone. For instance: m_2408002_ne_17_1_20100422_201001123.jp2, is in zone 17. As far as I can tell, all the files in a particular directory are all in the same UTM zone. I could create a layer for each UTM zone across CONUS, but that's not going to be particularly useful to my users. I'm thinking of making a layer for each state. For the stats that cross zones, there will probably be two layers. For Texas, there would be Texas_east, Texas_middle, and Texas_west. I will probably limit visibility until zoomed in sufficiently to see the whole state on the screen anyway, since the continental view of this data is pretty crappy anyway. So now it seems like it will be a lot of grunt work just copying these directories up to the server, and going through and creating a shape file index for each state. For states in more than one UTM, there would be more than one shape. Then I'll have to add a layer for to my mapfile for each shapefile, using the correct projection. Is there an easier way? I'm starting with Oklahoma, which is also in three UTM zones. I'll get that working before moving on. Any suggestions on making this pretty would be welcomed. :-) -Original Message- From: Stephen Woodbridge [mailto:wood...@swoodbridge.com] Sent: Monday, June 10, 2013 12:34 PM To: Evans, James R Civ USAF ACC 84 RADES/SCZE Cc: mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] Best way to import 4.5TB of imagery? On 6/10/2013 12:57 PM, Evans, James R Civ USAF ACC 84 RADES/SCZE wrote: Hi Stephen, Thanks, for the reply. I previously got 4 sample images from the USDA, and was able to get them to work just fine. There was no processing required. The sample images I got were all from Utah, and they are NAD83, UTM zone 12. I added the 4 sample images to a shape file using gdaltindex. I used UPSG 26912 and mapserver served them up very quickly for such large files. Now I have this entire data set, and it stretches from UTM zone 10, to UTM zone 19. The data is divided into directories by two letter state abbreviations, and under that into subdirectories. I'm just wondering how to add this to my mapfile. Do I need a different entry for each UTM Zone? How is it possible to get a single layer entry that includes multiple projections? This is looking like a huge job and I just want to know the best strategy for getting this done. So now you have a problem. You data is in UTM spread over 10 different projections. What do you plane to do when have your image is zone 10 and half is in zone 11 or if you zoom out and you images has 3-4 zones displayed? All data in an image must be rendered in the same projection. While I don't doubt that your test with 4 images worked fine, did you you test this a multiple zoom levels and at some point you will probably want to create a super overlay image so you do not have to open multiple files to just pull a tiny overlay out of each one. Your use cases will determine how you want to deal with the data. For example does it HAVE to be in a UTM projection, or can you work with a Spherical Mercator or geographic projection? The end solution will be much easier if you can work with one common projection over your whole data set. Otherwise, you will have to deal with the transitions from one zone to the next or maybe set up 10 separate servers that only serve one zone. Having pushed larges amounts (4-25TB) of imagery data more than once it is important to make these decisions up front and and prototype up something like a 4-10 degree square across a UTM boundary and make sure that the results are going to be what you expect before you process all the data. -Steve Thanks, James -Original Message- From: mapserver-users-boun...@lists.osgeo.org [mailto:mapserver-users-boun...@lists.osgeo.org] On Behalf Of Stephen Woodbridge Sent: Friday, June 07, 2013 8:41 AM To: mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] Best way to import 4.5TB of imagery? On 6/7/2013 10:31 AM, James_in_Utah wrote: Hi, We just got 3 hard drive, loaded with 4.5TB of NAIP imagery for all of CONUS. I think there's a total of about 400,000 jpgs. The data is in directories, by states. Under each
Re: [mapserver-users] Best way to import 4.5TB of imagery?
For some reason I didn't see the reply from Robert Sanson. Anyway, we have no requirement to stay with NAD83. I have global mapper, and I think it will do a bulk reprojection, from my local hard drive to a network drive. Maybe I will take a look at that tomorrow. Is there a better tool for that? Some GDAL utility? Anyway, thanks for all the great suggestions. James -Original Message- From: mapserver-users-boun...@lists.osgeo.org on behalf of Stephen Woodbridge Sent: Mon 6/10/2013 6:36 PM To: mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] Best way to import 4.5TB of imagery? On 6/10/2013 7:07 PM, Robert Sanson wrote: Sounds all very complicated. I would advise that you choose a single projection that will work across your entire are and then re-project your imagery first, and then build your image datastore around that. I would agree with this that if htere are not other contraints on the problem this is often the best way to go. But that will not owrk if his users HAVE to have the data in UTM projection for some reason. -Steve Evans, James R Civ USAF ACC 84 RADES/SCZE james.ev...@hill.af.mil 11/06/2013 10:48 a.m. So, I'm guessing there's no easy way to automate this? Even looking at the states, some of the states are in two zones, and Texas is across 3 zones. At least the naming convention of the files indicate the UTM zone. For instance: m_2408002_ne_17_1_20100422_201001123.jp2, is in zone 17. As far as I can tell, all the files in a particular directory are all in the same UTM zone. I could create a layer for each UTM zone across CONUS, but that's not going to be particularly useful to my users. I'm thinking of making a layer for each state. For the stats that cross zones, there will probably be two layers. For Texas, there would be Texas_east, Texas_middle, and Texas_west. I will probably limit visibility until zoomed in sufficiently to see the whole state on the screen anyway, since the continental view of this data is pretty crappy anyway. So now it seems like it will be a lot of grunt work just copying these directories up to the server, and going through and creating a shape file index for each state. For states in more than one UTM, there would be more than one shape. Then I'll have to add a layer for to my mapfile for each shapefile, using the correct projection. Is there an easier way? I'm starting with Oklahoma, which is also in three UTM zones. I'll get that working before moving on. Any suggestions on making this pretty would be welcomed. :-) -Original Message- From: Stephen Woodbridge [mailto:wood...@swoodbridge.com] Sent: Monday, June 10, 2013 12:34 PM To: Evans, James R Civ USAF ACC 84 RADES/SCZE Cc: mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] Best way to import 4.5TB of imagery? On 6/10/2013 12:57 PM, Evans, James R Civ USAF ACC 84 RADES/SCZE wrote: Hi Stephen, Thanks, for the reply. I previously got 4 sample images from the USDA, and was able to get them to work just fine. There was no processing required. The sample images I got were all from Utah, and they are NAD83, UTM zone 12. I added the 4 sample images to a shape file using gdaltindex. I used UPSG 26912 and mapserver served them up very quickly for such large files. Now I have this entire data set, and it stretches from UTM zone 10, to UTM zone 19. The data is divided into directories by two letter state abbreviations, and under that into subdirectories. I'm just wondering how to add this to my mapfile. Do I need a different entry for each UTM Zone? How is it possible to get a single layer entry that includes multiple projections? This is looking like a huge job and I just want to know the best strategy for getting this done. So now you have a problem. You data is in UTM spread over 10 different projections. What do you plane to do when have your image is zone 10 and half is in zone 11 or if you zoom out and you images has 3-4 zones displayed? All data in an image must be rendered in the same projection. While I don't doubt that your test with 4 images worked fine, did you you test this a multiple zoom levels and at some point you will probably want to create a super overlay image so you do not have to open multiple files to just pull a tiny overlay out of each one. Your use cases will determine how you want to deal with the data. For example does it HAVE to be in a UTM projection, or can you work with a Spherical Mercator or geographic projection? The end solution will be much easier if you can work with one common projection over your whole data set. Otherwise, you will have to deal with the transitions from one zone to the next or maybe set up 10 separate servers that only serve one zone. Having pushed larges amounts (4-25TB) of imagery data more than once it is important to make these decisions up front and and prototype up something like a 4-10
Re: [mapserver-users] Best way to import 4.5TB of imagery?
I have used gdalwarp to do this in the past and find or a perl script. I have not used global mapper for anything so I can't compare. -Steve On 6/10/2013 10:00 PM, Evans, James R Civ USAF ACC 84 RADES/SCZE wrote: For some reason I didn't see the reply from Robert Sanson. Anyway, we have no requirement to stay with NAD83. I have global mapper, and I think it will do a bulk reprojection, from my local hard drive to a network drive. Maybe I will take a look at that tomorrow. Is there a better tool for that? Some GDAL utility? Anyway, thanks for all the great suggestions. James -Original Message- From: mapserver-users-boun...@lists.osgeo.org on behalf of Stephen Woodbridge Sent: Mon 6/10/2013 6:36 PM To: mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] Best way to import 4.5TB of imagery? On 6/10/2013 7:07 PM, Robert Sanson wrote: Sounds all very complicated. I would advise that you choose a single projection that will work across your entire are and then re-project your imagery first, and then build your image datastore around that. I would agree with this that if htere are not other contraints on the problem this is often the best way to go. But that will not owrk if his users HAVE to have the data in UTM projection for some reason. -Steve Evans, James R Civ USAF ACC 84 RADES/SCZE james.ev...@hill.af.mil 11/06/2013 10:48 a.m. So, I'm guessing there's no easy way to automate this? Even looking at the states, some of the states are in two zones, and Texas is across 3 zones. At least the naming convention of the files indicate the UTM zone. For instance: m_2408002_ne_17_1_20100422_201001123.jp2, is in zone 17. As far as I can tell, all the files in a particular directory are all in the same UTM zone. I could create a layer for each UTM zone across CONUS, but that's not going to be particularly useful to my users. I'm thinking of making a layer for each state. For the stats that cross zones, there will probably be two layers. For Texas, there would be Texas_east, Texas_middle, and Texas_west. I will probably limit visibility until zoomed in sufficiently to see the whole state on the screen anyway, since the continental view of this data is pretty crappy anyway. So now it seems like it will be a lot of grunt work just copying these directories up to the server, and going through and creating a shape file index for each state. For states in more than one UTM, there would be more than one shape. Then I'll have to add a layer for to my mapfile for each shapefile, using the correct projection. Is there an easier way? I'm starting with Oklahoma, which is also in three UTM zones. I'll get that working before moving on. Any suggestions on making this pretty would be welcomed. :-) -Original Message- From: Stephen Woodbridge [mailto:wood...@swoodbridge.com] Sent: Monday, June 10, 2013 12:34 PM To: Evans, James R Civ USAF ACC 84 RADES/SCZE Cc: mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] Best way to import 4.5TB of imagery? On 6/10/2013 12:57 PM, Evans, James R Civ USAF ACC 84 RADES/SCZE wrote: Hi Stephen, Thanks, for the reply. I previously got 4 sample images from the USDA, and was able to get them to work just fine. There was no processing required. The sample images I got were all from Utah, and they are NAD83, UTM zone 12. I added the 4 sample images to a shape file using gdaltindex. I used UPSG 26912 and mapserver served them up very quickly for such large files. Now I have this entire data set, and it stretches from UTM zone 10, to UTM zone 19. The data is divided into directories by two letter state abbreviations, and under that into subdirectories. I'm just wondering how to add this to my mapfile. Do I need a different entry for each UTM Zone? How is it possible to get a single layer entry that includes multiple projections? This is looking like a huge job and I just want to know the best strategy for getting this done. So now you have a problem. You data is in UTM spread over 10 different projections. What do you plane to do when have your image is zone 10 and half is in zone 11 or if you zoom out and you images has 3-4 zones displayed? All data in an image must be rendered in the same projection. While I don't doubt that your test with 4 images worked fine, did you you test this a multiple zoom levels and at some point you will probably want to create a super overlay image so you do not have to open multiple files to just pull a tiny overlay out of each one. Your use cases will determine how you want to deal with the data. For example does it HAVE to be in a UTM projection, or can you work with a Spherical Mercator or geographic projection? The end solution will be much easier if you can work with one common projection over your whole data set. Otherwise, you will have to deal with the transitions from one zone to the next or maybe set up 10 separate servers that only serve one zone. Having pushed larges
Re: [mapserver-users] Best way to import 4.5TB of imagery?
Hi, If you reproject images one by one to any common coordinate system you will get black collars to most of them because images will be rotated. In order to make them transparent by using nodata you must keep images uncompressed. However, you may find it teasing to use the jpeg-in-tiff compression because that way you can save 90 of disk space and your users will not notice any difference in the quality and the speed can still be pretty good. For reprojecting it would be best to reproject into a regular South-North and West-East oriented grid in the target SRID. Then you could safely compress the reprojected tiles. That is a bit more complicated but still possible to do with scripts. If you want to keep your coverage up-to-date and you should slip in new images from here and there it will make a bit more trouble because usually the target tiles are mosaiced from many originals. This is what I am doing right now with a couple of terabytes of Finnish aerial photos. With your data I would do what Frank W. already suggested. Convert your jpeg files into tiled tiffs with jpeg compression and use creation option PHOTOMETRIC=YCBCR. Create also overviews which are compressed in the same way. Make one layer for each UTM zone by combining them with tileindex and group all the zones together with a layer group. You can also start from the original images by creating tileindexes from them and convert them into more speedy tiled jpeg-in-tiff images once you have time for that. -Jukka Rahkonen- Evans, James wrote: For some reason I didn't see the reply from Robert Sanson. Anyway, we have no requirement to stay with NAD83. I have global mapper, and I think it will do a bulk reprojection, from my local hard drive to a network drive. Maybe I will take a look at that tomorrow. Is there a better tool for that? Some GDAL utility? Anyway, thanks for all the great suggestions. James -Original Message- From: mapserver-users-boun...@lists.osgeo.org on behalf of Stephen Woodbridge Sent: Mon 6/10/2013 6:36 PM To: mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] Best way to import 4.5TB of imagery? On 6/10/2013 7:07 PM, Robert Sanson wrote: Sounds all very complicated. I would advise that you choose a single projection that will work across your entire are and then re-project your imagery first, and then build your image datastore around that. I would agree with this that if htere are not other contraints on the problem this is often the best way to go. But that will not owrk if his users HAVE to have the data in UTM projection for some reason. -Steve Evans, James R Civ USAF ACC 84 RADES/SCZE james.ev...@hill.af.mil 11/06/2013 10:48 a.m. So, I'm guessing there's no easy way to automate this? Even looking at the states, some of the states are in two zones, and Texas is across 3 zones. At least the naming convention of the files indicate the UTM zone. For instance: m_2408002_ne_17_1_20100422_201001123.jp2, is in zone 17. As far as I can tell, all the files in a particular directory are all in the same UTM zone. I could create a layer for each UTM zone across CONUS, but that's not going to be particularly useful to my users. I'm thinking of making a layer for each state. For the stats that cross zones, there will probably be two layers. For Texas, there would be Texas_east, Texas_middle, and Texas_west. I will probably limit visibility until zoomed in sufficiently to see the whole state on the screen anyway, since the continental view of this data is pretty crappy anyway. So now it seems like it will be a lot of grunt work just copying these directories up to the server, and going through and creating a shape file index for each state. For states in more than one UTM, there would be more than one shape. Then I'll have to add a layer for to my mapfile for each shapefile, using the correct projection. Is there an easier way? I'm starting with Oklahoma, which is also in three UTM zones. I'll get that working before moving on. Any suggestions on making this pretty would be welcomed. :-) -Original Message- From: Stephen Woodbridge [mailto:wood...@swoodbridge.com] Sent: Monday, June 10, 2013 12:34 PM To: Evans, James R Civ USAF ACC 84 RADES/SCZE Cc: mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] Best way to import 4.5TB of imagery? On 6/10/2013 12:57 PM, Evans, James R Civ USAF ACC 84 RADES/SCZE wrote: Hi Stephen, Thanks, for the reply. I previously got 4 sample images from the USDA, and was able to get them to work just fine. There was no processing required. The sample images I got were all from Utah, and they are NAD83, UTM zone 12. I added the 4 sample images to a shape file using gdaltindex. I used UPSG 26912 and mapserver served them up very quickly for such large files. Now I have this entire data set, and it stretches from UTM
[mapserver-users] Best way to import 4.5TB of imagery?
Hi, We just got 3 hard drive, loaded with 4.5TB of NAIP imagery for all of CONUS. I think there's a total of about 400,000 jpgs. The data is in directories, by states. Under each state, there are subfolders, probably reference by longitude. Other than going through folder by folder, adding each image to a shape file using gdaltindex, what's the best strategy for loading a couple of hundred thousand files up to our server and making the imagery available via our mapserver? Should I maintain the current directory structure when I copy the imagery to the server, or just dump all of it into a single directory? Do I want to stay with 1 shape file, or break it up by state? We eventually want a contiguous layer for all of CONUS to be served up to our users. Thanks, James -- View this message in context: http://osgeo-org.1560.x6.nabble.com/Best-way-to-import-4-5TB-of-imagery-tp5058745.html Sent from the Mapserver - User mailing list archive at Nabble.com. ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Best way to import 4.5TB of imagery?
On 6/7/2013 10:31 AM, James_in_Utah wrote: Hi, We just got 3 hard drive, loaded with 4.5TB of NAIP imagery for all of CONUS. I think there's a total of about 400,000 jpgs. The data is in directories, by states. Under each state, there are subfolders, probably reference by longitude. Other than going through folder by folder, adding each image to a shape file using gdaltindex, what's the best strategy for loading a couple of hundred thousand files up to our server and making the imagery available via our mapserver? Should I maintain the current directory structure when I copy the imagery to the server, or just dump all of it into a single directory? Do I want to stay with 1 shape file, or break it up by state? We eventually want a contiguous layer for all of CONUS to be served up to our users. James, Since imagery data is served via gdal, you might want to also ask this question on the gdal list. There are issues with jpg related to the fact that if you only want a small part of the image you still have to uncompress the whole image. So part of the answer might be that you need to pre-process all the imagery into something like a jpg compress tiled geotif or something else. You also need to consider what projection your imagery is in and what projection you want to display it in. Because if you need to preprocess the data, that would also be a good time to reproject it. Anyway the gdal list can probably ask additional questions to help sort all that out. -Steve W ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
