Re: [mapserver-users] logging GDAL queries
Hi Ivan, On Mon, Aug 16, 2010 at 6:02 AM, Lucena, Ivan ivan.luc...@pmldnet.comwrote: Sebastian, I have this little piece of code in Python that shows how 18 levels of pyramids would look like: ... ... 10 185 265 1 1 1049401 11 92 132 1 1 1049402 ** What you don't have: ** 12 46 66 1 1 1049403 13 23 33 1 1 1049404 14 11 16 1 1 1049405 15 5 8 1 1 1049406 16 2 4 1 1 1049407 17 1 2 1 1 1049408 Where: ... ... You said that “it performs very well at zoom level under 12” and performs poorly with zoom level between 13 and 17. are MapServer/GoogleMaps zoom level exactly the same as Oracle PyramidLevels ? For example... in Oracle , Pyramid level 0 means maximum zoom... that in MapServer/GoogleMaps it would something like 20 ??? The problem is that you don't have those levels built into your GeoRaster object. So, before doing any change to the system settings, could you please try to increase the number of pyramids to 17? Building the pyramids is a internal process on the server. Even if you use gdaladdo, that will just call a function from the GeoRaster PL/SQL extension. It wouldn't take as long as before because it will add up to the existing levels. The PL/SQL call should looks like that: declare gr sdo_georaster; begin select georaster into gr from fluvd04q200pj where id = 1 for update; sdo_geor.generatePyramid(gr, 'rlevel=17 resampling=NN'); update fluvd04q200pj set georaster = gr where id = 1; commit; end; / from Oracle documentation rLevel (for example, rLevel=2): Specifies the maximum reduction level: the number of pyramid levels to create at a smaller (reduced) size than the original object. If you do not specify this keyword, pyramid levels are generated until the smaller of the number of rows or columns is between 64 and 128. The dimension sizes at each lower resolution level are equal to the truncated integer values of the dimension sizes at the next higher resolution level, divided by 2. I've ignored the Oracle documentation and I've run it anyway... It toked 0.5 seconds to run ! In my case that code would not create any pyramids as I have already run sdo_geor.generatePyramid(gr, 'resampling=NN'); which would build the maximun levels of pyramids... And the gdaladdo command should look like that: gdaladdo georaster:geoserver,geoserver,MFPRODUK_11G,fluvd04q200pj,georaster,id=1 2 4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 32768 65536 Best regards, Ivan Sebastian E. Ovide wrote: Ivan, Yes! and No! That query creates a *cursor* to go through the whole raster, meaning, all the rows on the RasterDataTable that satisfy that query in the order stipulated by it. It does require a some memory but it looks like you got it. mmm... that makes me little bit nervous... I am testing it with a small map. Its is only UK with 5mx5m pixel Just wondering how this approach would scale to larger maps... US is 40 times bigger than UK... and they have more accurated maps and more than a single layer Just wondering how many manual configuration I'll be forced to do (Oracle created Spatial extention exactly for this reason... so the developer do not to think about handling blobs for him self anymore... reducing costs and risks) By the way, how long does it take to produce a geotiff file? Have loaded that image on QuantumGIS using the oracle_raster plugin? Hoes does it perform? it is not possible to create a TIFF as it would be too big (max size is 4GB)... and UK would be arounf 16GB (300MB in Oracle as I'm using deflate) I am testing Oracle MapViewer on the same table and it HAS the same problem The strange thing is that using MapBuilder (Oracle software to prepare the maps to use with MapViewer) the maps are displayed very fast at ANY zoom level !... therefore there is a way to do it fast (as MapBuilder does...) I was expecting to find in the logs some query that reads a subset of the whole image (just the tile/metatile that MapServer is serving) for example using SDO_GEOR.getRasterSubset.. That would be very slow. The GDAL driver access the BLOB directly. not sure about that... well I guess that at least it releases some work from the Oracle server... Anyway, as I have a spatial index, why don't just query only the right blobs? I have more questions: Are you using Mapserver FastCGI with a recent version? nop... I have not tried that yet... I'll try on Monday... I have tried as cgi and recently as mod_python. If you do, them the process will remains in memory, the GDALDataset associated with that GeoRaster will be keet, the *cursor* you remain alive and all the zooms and pans are going to work faster. Like for QGIS. Unless, of course, Mapserver is feeling the need to re-project or produce a new overview tiles on the fly. Can you see that on the log? Regards, Ivan
Re: [mapserver-users] logging GDAL queries
Hi Sebastian, Did you fix the problem with the pyramid levels? Do you see any improved where the performance was poor before? Thanks for your valuable test. I appreciate. Regards, Ivan ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] logging GDAL queries
Hi Ivan, I did create those extra pyramid levels Overviews: 95162x135711, 47581x67855, 23790x33927, 11895x16963, 5947x8481, 2973x4240, 1486x2120, 743x1060, 371x530, 185x265, 92x132, 46x66, 23x33, 11x16, 5x8, 2x4, 1x2 the performance has not improved. Why should it ? the extra pyramid levels contains 46x66, 23x33, 11x16, 5x8, 2x4, 1x2 but the problems that I am having are for 95162x135711, 47581x67855, 23790x33927, 11895x16963... that are low pyramid levels... (ie: high zoom=low pyramid level)... thanks On Mon, Aug 16, 2010 at 1:30 PM, Ivan Lucena ivan.luc...@pmldnet.comwrote: Hi Sebastian, Did you fix the problem with the pyramid levels? Do you see any improved where the performance was poor before? Thanks for your valuable test. I appreciate. Regards, Ivan -- Sebastian E. Ovide ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] logging GDAL queries
Sebastian, the performance has not improved. Why should it ? the extra pyramid levels contains 46x66, 23x33, 11x16, 5x8, 2x4, 1x2 but the problems that I am having are for 95162x135711, 47581x67855, 23790x33927, 11895x16963... that are low pyramid levels... (ie: high zoom=low pyramid level)... If you are still having problem with MapViewer I suggest you to post your log at [http://forums.oracle.com/forums/forum.jspa?forumID=727]. Any improvement on MapServer+GeoRaster through GDAL would take a little bit of time to adapt the initial ETL design. Regards, Ivan ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] logging GDAL queries
Thank Ivan, I appreciate your help. Regards On Mon, Aug 16, 2010 at 7:23 PM, Ivan Lucena ivan.luc...@pmldnet.comwrote: Sebastian, the performance has not improved. Why should it ? the extra pyramid levels contains 46x66, 23x33, 11x16, 5x8, 2x4, 1x2 but the problems that I am having are for 95162x135711, 47581x67855, 23790x33927, 11895x16963... that are low pyramid levels... (ie: high zoom=low pyramid level)... If you are still having problem with MapViewer I suggest you to post your log at [http://forums.oracle.com/forums/forum.jspa?forumID=727]. Any improvement on MapServer+GeoRaster through GDAL would take a little bit of time to adapt the initial ETL design. Regards, Ivan -- Sebastian E. Ovide ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] logging GDAL queries
Sebastian, I have this little piece of code in Python that shows how 18 levels of pyramids would look like: rows = 271423 cols = 190325 bxs = 256 bys = 256 t = 0 for i in range(0,18,1): c = int( round( cols / 2 ** i ) ) r = int( round( rows / 2 ** i ) ) if ( c = ( bxs / 2 ) ) and ( r = ( bys / 2 ) ): w = c h = r else: w = bxs h = bys bc = int( max(1, c / w ) ) br = int( max(1, r / h ) ) t += bc * br print i, c, r, bc, br, t And here is the output: 0 190325 271423 743 1060 787580 1 95162 135711 371 530 984210 2 47581 67855 185 265 1033235 3 23790 33927 92 132 1045379 4 11895 16963 46 66 1048415 5 5947 8481 23 33 1049174 6 2973 4240 11 16 1049350 7 1486 2120 5 8 1049390 8 743 1060 2 4 1049398 9 371 530 1 2 1049400 10 185 265 1 1 1049401 11 92 132 1 1 1049402 ** What you don't have: ** 12 46 66 1 1 1049403 13 23 33 1 1 1049404 14 11 16 1 1 1049405 15 5 8 1 1 1049406 16 2 4 1 1 1049407 17 1 2 1 1 1049408 Where: i = level c = size of the level in columns r = size of the level in rows bc = number of blocks per column br = number of blocks per row t = total number of blocks You said that “it performs very well at zoom level under 12” and performs poorly with zoom level between 13 and 17. The problem is that you don't have those levels built into your GeoRaster object. So, before doing any change to the system settings, could you please try to increase the number of pyramids to 17? Building the pyramids is a internal process on the server. Even if you use gdaladdo, that will just call a function from the GeoRaster PL/SQL extension. It wouldn't take as long as before because it will add up to the existing levels. The PL/SQL call should looks like that: declare gr sdo_georaster; begin select georaster into gr from fluvd04q200pj where id = 1 for update; sdo_geor.generatePyramid(gr, 'rlevel=17 resampling=NN'); update fluvd04q200pj set georaster = gr where id = 1; commit; end; / And the gdaladdo command should look like that: gdaladdo georaster:geoserver,geoserver,MFPRODUK_11G,fluvd04q200pj,georaster,id=1 2 4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 32768 65536 Best regards, Ivan Sebastian E. Ovide wrote: Ivan, Yes! and No! That query creates a *cursor* to go through the whole raster, meaning, all the rows on the RasterDataTable that satisfy that query in the order stipulated by it. It does require a some memory but it looks like you got it. mmm... that makes me little bit nervous... I am testing it with a small map. Its is only UK with 5mx5m pixel Just wondering how this approach would scale to larger maps... US is 40 times bigger than UK... and they have more accurated maps and more than a single layer Just wondering how many manual configuration I'll be forced to do (Oracle created Spatial extention exactly for this reason... so the developer do not to think about handling blobs for him self anymore... reducing costs and risks) By the way, how long does it take to produce a geotiff file? Have loaded that image on QuantumGIS using the oracle_raster plugin? Hoes does it perform? it is not possible to create a TIFF as it would be too big (max size is 4GB)... and UK would be arounf 16GB (300MB in Oracle as I'm using deflate) I am testing Oracle MapViewer on the same table and it HAS the same problem The strange thing is that using MapBuilder (Oracle software to prepare the maps to use with MapViewer) the maps are displayed very fast at ANY zoom level !... therefore there is a way to do it fast (as MapBuilder does...) I was expecting to find in the logs some query that reads a subset of the whole image (just the tile/metatile that MapServer is serving) for example using SDO_GEOR.getRasterSubset.. That would be very slow. The GDAL driver access the BLOB directly. not sure about that... well I guess that at least it releases some work from the Oracle server... Anyway, as I have a spatial index, why don't just query only the right blobs? I have more questions: Are you using Mapserver FastCGI with a recent version? nop... I have not tried that yet... I'll try on Monday... I have tried as cgi and recently as mod_python. If you do, them the process will remains in memory, the GDALDataset associated with that GeoRaster will be keet, the *cursor* you remain alive and all the zooms and pans are going to work faster. Like for QGIS. Unless, of course, Mapserver is feeling the need to re-project or produce a new overview tiles on the fly. Can you see that on the log? Regards, Ivan regards -- Sebastian E. Ovide ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] logging GDAL queries
Hi Ivan, the pyramid levels in Oracle are not exactly the same as the zoom levels in MapServer... Each level whould have 1/4 number of pixels as the previous level...ie: X/2*Y/2 The size of the pyramid image at each level is determined by the original image size and the pyramid level, according to the following formulas: r(n) = (int)(r(0) / 2^n) c(n) = (int)(c(0) / 2^n) In my case I have generated all the possible levels of pyramids... The Idea of generating the pyramids is to speeding up the zoomed out images.. ie: if you want to display the whole UK, you do not need to get the 100kx160k pixels... you can read an upper level in the pyramid reading only 1kx1k pixels... in the same way, if you need to do a zoom in at street level, you can read from the level 0, that means reading the maximum resolution and it is fast as you read only a fraction of the whole image... with MapViewer+GDAL it getting very slow when generating tiles at street level it should not be at that point you do not need pyramid any more... If I didn't have created pyramids, it still would be fast at street level but it would be very slow at city level and it would be impossible to display at country level... does it make sense ? On Fri, Aug 13, 2010 at 6:57 PM, Ivan Lucena ivan.luc...@pmldnet.comwrote: Sebastian, You said: yep I did... and the weir thing is that generating the images from far away zoom under 12... it performs very well... not sure why it doesn't in the levels 13-17 for that reason I was trying to find the SQL queries But your GeoRaster Pyramid level is maxLevel11/maxLevel By the size of you image you should have 20 level. Can you run sdo_geor.generatePyramid() again changing only the number of levels, keeping the same method. That should run faster, keeping the existing pyramid levels. Please let me know how it works now. Regards, Ivan ---Original Message--- From: Ivan Lucena ivan.luc...@pmldnet.com To: Sebastian E. Ovide sebastian.ov...@gmail.com Cc: mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] logging GDAL queries Sent: Aug 13 '10 11:16 Sebastian, ---Original Message--- From: Sebastian E. Ovide sebastian.ov...@gmail.com To: Ivan Lucena ivan.luc...@pmldnet.com Cc: mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] logging GDAL queries Sent: Aug 13 '10 09:27 On Fri, Aug 13, 2010 at 1:59 PM, Ivan Lucena [LINK: mailto:ivan.luc...@pmldnet.com] ivan.luc...@pmldnet.com wrote: Sebastian, You are using a very small block size, maybe because you are also using ArcGIS products and they only support Oracle GeoRaster with 256x256 blocking. Is that right? nop... I guess that 256x256 if the default gdal_translate block size ? That is the default on GDAL/GeoRaster driver but you can change it using -co BLOCKXSIZE and -co BLOCKYSIZE. But that would require you to load the image again (and generate Pyramids again...) Because of that, to access your GeoRaster object, the GDAL driver is requesting a very large array of BLOB (100 approximately). That can make your sever such-up memory from the machine where it is running and it is probably using virtual memory (disk) what is really slow. do you mean where MapServer is running ? top - 14:41:31 up 7 days, 6:20, 10 users, load average: 2.81, 1.92, 1.20 Tasks: 340 total, 2 running, 338 sleeping, 0 stopped, 0 zombie Cpu(s): 11.8%us, 10.4%sy, 7.7%ni, 62.1%id, 0.0%wa, 0.0%hi, 8.0%si, 0.0%st Mem: 12331240k total, 5992136k used, 6339104k free, 189416k buffers Swap: 9084716k total, 703708k used, 8381008k free, 1481992k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24850 www-data 20 0 533m 330m 18m R 14 2.7 0:00.94 mapserv 24852 www-data 20 0 532m 329m 18m S 19 2.7 0:01.22 mapserv 24849 www-data 20 0 529m 326m 18m S 16 2.7 0:01.00 mapserv 24851 www-data 20 0 517m 314m 18m S 13 2.6 0:00.96 mapserv 24853 www-data 20 0 499m 296m 18m S8 2.5 0:00.75 mapserv 24854 www-data 20 0 470m 267m 18m S5 2.2 0:00.53 mapserv Yeah. I am sorry. I called Mapserver a client :) It is a Oracle client in that case. You probably have another nice machine running Oracle. That is not the problem. It also uses some memory on the client side just to hold the array too but if you are running everything (Mapserver,Oracle) on the same machine that will adds up. Oracle is running on another Machine... and is very likely that it is the bottle neck... Maybe not but I believe you have tools on Enterprise Manager to show the volume and speed of traffic. We
Re: [mapserver-users] logging GDAL queries
Hi Ivan, Because of that, to access your GeoRaster object, the GDAL driver is requesting a very large array of BLOB (100 approximately). T I do not know how apServer/GDAL works, but I cannot understand this query: [Fri Aug 13 13:53:00 2010] [error] [client 10.0.1.40] SELECT RASTERBLOCK [Fri Aug 13 13:53:00 2010] [error] [client 10.0.1.40] FROM RDT_FLUVD04Q200PJ [Fri Aug 13 13:53:00 2010] [error] [client 10.0.1.40] WHERE RASTERID = :1 AND [Fri Aug 13 13:53:00 2010] [error] [client 10.0.1.40]PYRAMIDLEVEL = :3 [Fri Aug 13 13:53:00 2010] [error] [client 10.0.1.40] ORDER BY [Fri Aug 13 13:53:00 2010] [error] [client 10.0.1.40]BANDBLOCKNUMBER ASC, [Fri Aug 13 13:53:00 2010] [error] [client 10.0.1.40]ROWBLOCKNUMBER ASC, [Fri Aug 13 13:53:00 2010] [error] [client 10.0.1.40] COLUMNBLOCKNUMBER ASC It looks like MapServer/GDAL is trying to read the whole pyramid level X of the table RDT_FLUVD04Q200PJ ? Am I right ? at level 0 it would read the whole raster I was expecting to find in the logs some query that reads a subset of the whole image (just the tile/metatile that MapServer is serving) for example using SDO_GEOR.getRasterSubset.. any ideas ? thanks -- Sebastian E. Ovide ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] logging GDAL queries
Sebastian, Sebastian E. Ovide wrote: Hi Ivan, Because of that, to access your GeoRaster object, the GDAL driver is requesting a very large array of BLOB (100 approximately). T I do not know how apServer/GDAL works, but I cannot understand this query: [Fri Aug 13 13:53:00 2010] [error] [client 10.0.1.40] SELECT RASTERBLOCK [Fri Aug 13 13:53:00 2010] [error] [client 10.0.1.40] FROM RDT_FLUVD04Q200PJ [Fri Aug 13 13:53:00 2010] [error] [client 10.0.1.40] WHERE RASTERID = :1 AND [Fri Aug 13 13:53:00 2010] [error] [client 10.0.1.40] PYRAMIDLEVEL = :3 [Fri Aug 13 13:53:00 2010] [error] [client 10.0.1.40] ORDER BY [Fri Aug 13 13:53:00 2010] [error] [client 10.0.1.40] BANDBLOCKNUMBER ASC, [Fri Aug 13 13:53:00 2010] [error] [client 10.0.1.40] ROWBLOCKNUMBER ASC, [Fri Aug 13 13:53:00 2010] [error] [client 10.0.1.40] COLUMNBLOCKNUMBER ASC It looks like MapServer/GDAL is trying to read the whole pyramid level X of the table RDT_FLUVD04Q200PJ ? Am I right ? at level 0 it would read the whole raster Yes! and No! That query creates a *cursor* to go through the whole raster, meaning, all the rows on the RasterDataTable that satisfy that query in the order stipulated by it. It does require a some memory but it looks like you got it. Right? I assuming that because you said that gdal_translate loaded those 51 Gb in 5 minutes. By the way, how long does it take to produce a geotiff file? Have loaded that image on QuantumGIS using the oracle_raster plugin? Hoes does it perform? I was expecting to find in the logs some query that reads a subset of the whole image (just the tile/metatile that MapServer is serving) for example using SDO_GEOR.getRasterSubset.. That would be very slow. The GDAL driver access the BLOB directly. I have more questions: Are you using Mapserver FastCGI with a recent version? If you do, them the process will remains in memory, the GDALDataset associated with that GeoRaster will be keet, the *cursor* you remain alive and all the zooms and pans are going to work faster. Like for QGIS. Unless, of course, Mapserver is feeling the need to re-project or produce a new overview tiles on the fly. Can you see that on the log? Regards, Ivan ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] logging GDAL queries
Ivan, Yes! and No! That query creates a *cursor* to go through the whole raster, meaning, all the rows on the RasterDataTable that satisfy that query in the order stipulated by it. It does require a some memory but it looks like you got it. mmm... that makes me little bit nervous... I am testing it with a small map. Its is only UK with 5mx5m pixel Just wondering how this approach would scale to larger maps... US is 40 times bigger than UK... and they have more accurated maps and more than a single layer Just wondering how many manual configuration I'll be forced to do (Oracle created Spatial extention exactly for this reason... so the developer do not to think about handling blobs for him self anymore... reducing costs and risks) By the way, how long does it take to produce a geotiff file? Have loaded that image on QuantumGIS using the oracle_raster plugin? Hoes does it perform? it is not possible to create a TIFF as it would be too big (max size is 4GB)... and UK would be arounf 16GB (300MB in Oracle as I'm using deflate) I am testing Oracle MapViewer on the same table and it HAS the same problem The strange thing is that using MapBuilder (Oracle software to prepare the maps to use with MapViewer) the maps are displayed very fast at ANY zoom level !... therefore there is a way to do it fast (as MapBuilder does...) I was expecting to find in the logs some query that reads a subset of the whole image (just the tile/metatile that MapServer is serving) for example using SDO_GEOR.getRasterSubset.. That would be very slow. The GDAL driver access the BLOB directly. not sure about that... well I guess that at least it releases some work from the Oracle server... Anyway, as I have a spatial index, why don't just query only the right blobs? I have more questions: Are you using Mapserver FastCGI with a recent version? nop... I have not tried that yet... I'll try on Monday... I have tried as cgi and recently as mod_python. If you do, them the process will remains in memory, the GDALDataset associated with that GeoRaster will be keet, the *cursor* you remain alive and all the zooms and pans are going to work faster. Like for QGIS. Unless, of course, Mapserver is feeling the need to re-project or produce a new overview tiles on the fly. Can you see that on the log? Regards, Ivan regards -- Sebastian E. Ovide ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] logging GDAL queries
Hi, I can't say anything about georaster, but 4 GB is not a limit for GDAL because it supports BigTIFFs. -Jukka Rahkonen- -Alkuperäinen viesti- Lähettäjä: mapserver-users-boun...@lists.osgeo.org puolesta: Sebastian E. Ovide Lähetetty: la 14.8.2010 16:02 Vastaanottaja: Lucena, Ivan Kopio: mapserver-users@lists.osgeo.org Aihe: Re: [mapserver-users] logging GDAL queries Ivan, Yes! and No! That query creates a *cursor* to go through the whole raster, meaning, all the rows on the RasterDataTable that satisfy that query in the order stipulated by it. It does require a some memory but it looks like you got it. mmm... that makes me little bit nervous... I am testing it with a small map. Its is only UK with 5mx5m pixel Just wondering how this approach would scale to larger maps... US is 40 times bigger than UK... and they have more accurated maps and more than a single layer Just wondering how many manual configuration I'll be forced to do (Oracle created Spatial extention exactly for this reason... so the developer do not to think about handling blobs for him self anymore... reducing costs and risks) By the way, how long does it take to produce a geotiff file? Have loaded that image on QuantumGIS using the oracle_raster plugin? Hoes does it perform? it is not possible to create a TIFF as it would be too big (max size is 4GB)... and UK would be arounf 16GB (300MB in Oracle as I'm using deflate) I am testing Oracle MapViewer on the same table and it HAS the same problem The strange thing is that using MapBuilder (Oracle software to prepare the maps to use with MapViewer) the maps are displayed very fast at ANY zoom level !... therefore there is a way to do it fast (as MapBuilder does...) I was expecting to find in the logs some query that reads a subset of the whole image (just the tile/metatile that MapServer is serving) for example using SDO_GEOR.getRasterSubset.. That would be very slow. The GDAL driver access the BLOB directly. not sure about that... well I guess that at least it releases some work from the Oracle server... Anyway, as I have a spatial index, why don't just query only the right blobs? I have more questions: Are you using Mapserver FastCGI with a recent version? nop... I have not tried that yet... I'll try on Monday... I have tried as cgi and recently as mod_python. If you do, them the process will remains in memory, the GDALDataset associated with that GeoRaster will be keet, the *cursor* you remain alive and all the zooms and pans are going to work faster. Like for QGIS. Unless, of course, Mapserver is feeling the need to re-project or produce a new overview tiles on the fly. Can you see that on the log? Regards, Ivan regards -- Sebastian E. Ovide ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] logging GDAL queries
Hi All I am using MapServer + GDAL to render GeoRasters and I am having very poor performance reading GeoRasters at a high zoom level... over 13... it would take even 20 seconds per tile... Is it possible to log the GDAL queries that MapServer is doing? (and the SQL queries that GDAL is doing ?) thanks -- Sebastian E. Ovide ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] logging GDAL queries
Add CONFIG CPL_DEBUG ON to your MAP object in the MAP file. Probably wise to add DEBUG 5 to your MAP and LAYER as well. Also add something like: CONFIG MS_ERRORFILE stderr and then check your Apache error log. Best regards, Bart -- Looking for flexible support on OpenLayers or GeoExt? Please check out http://www.osgis.nl/support.html Bart van den Eijnden OSGIS bart...@osgis.nl On Aug 13, 2010, at 10:33 AM, Sebastian E. Ovide wrote: Hi All I am using MapServer + GDAL to render GeoRasters and I am having very poor performance reading GeoRasters at a high zoom level... over 13... it would take even 20 seconds per tile... Is it possible to log the GDAL queries that MapServer is doing? (and the SQL queries that GDAL is doing ?) thanks -- Sebastian E. Ovide ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] logging GDAL queries
Sebastian, Sebastian E. Ovide wrote: Hi All I am using MapServer + GDAL to render GeoRasters and I am having very poor performance reading GeoRasters at a high zoom level... over 13... it would take even 20 seconds per tile... Did you generate pyramids? Is it possible to log the GDAL queries that MapServer is doing? (and the SQL queries that GDAL is doing ?) thanks -- Sebastian E. Ovide ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] logging GDAL queries
yep I did... and the weir thing is that generating the images from far away zoom under 12... it performs very well... not sure why it doesn't in the levels 13-17 for that reason I was trying to find the SQL queries On Fri, Aug 13, 2010 at 12:01 PM, Lucena, Ivan ivan.luc...@pmldnet.comwrote: Sebastian, Sebastian E. Ovide wrote: Hi All I am using MapServer + GDAL to render GeoRasters and I am having very poor performance reading GeoRasters at a high zoom level... over 13... it would take even 20 seconds per tile... Did you generate pyramids? Is it possible to log the GDAL queries that MapServer is doing? (and the SQL queries that GDAL is doing ?) thanks -- Sebastian E. Ovide ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users -- Sebastian E. Ovide ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] logging GDAL queries
Interesting. Can you post your gdalinfo report with -mdd oracle option? What are the versions your are running (GDAL,Oracle)? Older version of the driver used to run a new query every time you change levels. Sebastian E. Ovide wrote: yep I did... and the weir thing is that generating the images from far away zoom under 12... it performs very well... not sure why it doesn't in the levels 13-17 for that reason I was trying to find the SQL queries On Fri, Aug 13, 2010 at 12:01 PM, Lucena, Ivan ivan.luc...@pmldnet.com mailto:ivan.luc...@pmldnet.com wrote: Sebastian, Sebastian E. Ovide wrote: Hi All I am using MapServer + GDAL to render GeoRasters and I am having very poor performance reading GeoRasters at a high zoom level... over 13... it would take even 20 seconds per tile... Did you generate pyramids? Is it possible to log the GDAL queries that MapServer is doing? (and the SQL queries that GDAL is doing ?) thanks -- Sebastian E. Ovide ___ mapserver-users mailing list mapserver-users@lists.osgeo.org mailto:mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users -- Sebastian E. Ovide ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] logging GDAL queries
sure... se...@seanspc:~/mapserver/mapserver-5.6.5$ gdalinfo --version GDAL 1.7.2, released 2010/04/23 se...@seanspc:~/mapserver/mapserver-5.6.5$ gdalinfo georaster:geoserver,geoserver,MFPRODUK_11G,fluvd04q200pj,georaster,id=1 -mdd oracle Driver: GeoRaster/Oracle Spatial GeoRaster Files: none associated Size is 190325, 271423 Coordinate System is: PROJCS[Popular Visualisation CRS / Mercator (deprecated), GEOGCS[Popular Visualisation CRS, DATUM[Popular_Visualisation_Datum, SPHEROID[Popular Visualisation Sphere,6378137,0, AUTHORITY[EPSG,7059]], TOWGS84[0,0,0,0,0,0,0], AUTHORITY[EPSG,6055]], PRIMEM[Greenwich,0, AUTHORITY[EPSG,8901]], UNIT[degree,0.01745329251994328, AUTHORITY[EPSG,9122]], AUTHORITY[EPSG,4055]], UNIT[metre,1, AUTHORITY[EPSG,9001]], PROJECTION[Mercator_1SP], PARAMETER[central_meridian,0], PARAMETER[scale_factor,1], PARAMETER[false_easting,0], PARAMETER[false_northing,0], EXTENSION[PROJ4,+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgri...@null +wktext +no_defs], AUTHORITY[EPSG,3785], AXIS[X,EAST], AXIS[Y,NORTH]] Origin = (-703633.464883987908252,7805920.616936270147562) Pixel Size = (4.999,-5.000) Metadata (oracle): TABLE_NAME=fluvd04q200pj COLUMN_NAME=georaster RDT_TABLE_NAME=RDT_FLUVD04Q200PJ RASTER_ID=62 METADATA=georasterMetadata xmlns= http://xmlns.oracle.com/spatial/georaster; objectInfo rasterType20001/rasterType isBlankfalse/isBlank defaultRed1/defaultRed defaultGreen1/defaultGreen defaultBlue1/defaultBlue /objectInfo rasterInfo cellRepresentationUNDEFINED/cellRepresentation cellDepth8BIT_U/cellDepth NODATA255/NODATA totalDimensions2/totalDimensions dimensionSize type=ROW size271423/size /dimensionSize dimensionSize type=COLUMN size190325/size /dimensionSize ULTCoordinate row0/row column0/column /ULTCoordinate blocking typeREGULAR/type totalRowBlocks1061/totalRowBlocks totalColumnBlocks744/totalColumnBlocks rowBlockSize256/rowBlockSize columnBlockSize256/columnBlockSize /blocking interleavingBIP/interleaving pyramid typeDECREASE/type resamplingNN/resampling maxLevel11/maxLevel /pyramid compression typeDEFLATE/type /compression /rasterInfo spatialReferenceInfo isReferencedtrue/isReferenced isRectifiedtrue/isRectified SRID3785/SRID spatialResolution dimensionType=X resolution5/resolution /spatialResolution spatialResolution dimensionType=Y resolution5/resolution /spatialResolution modelCoordinateLocationCENTER/modelCoordinateLocation modelTypeFunctionalFitting/modelType polynomialModel rowOff=0 columnOff=0 xOff=0 yOff=0 zOff=0 rowScale=1 columnScale=1 xScale=1 yScale=1 zScale=1 pPolynomial pType=1 nVars=2 order=1 nCoefficients=3 polynomialCoefficients1561183.623387254 0 -0.2/polynomialCoefficients /pPolynomial qPolynomial pType=1 nVars=0 order=0 nCoefficients=1 polynomialCoefficients1/polynomialCoefficients /qPolynomial rPolynomial pType=1 nVars=2 order=1 nCoefficients=3 polynomialCoefficients140726.1929767976 0.2 0/polynomialCoefficients /rPolynomial sPolynomial pType=1 nVars=0 order=0 nCoefficients=1 polynomialCoefficients1/polynomialCoefficients /sPolynomial /polynomialModel /spatialReferenceInfo layerInfo layerDimensionBAND/layerDimension objectLayer layerNumber0/layerNumber layerDimensionOrdinate0/layerDimensionOrdinate layerID/ colorMap colors cell value=1 red=255 green=0 blue=0 alpha=100/ cell value=2 red=0 green=255 blue=0 alpha=100/ cell value=3 red=0 green=0 blue=255 alpha=100/ cell value=4 red=150 green=50 blue=100 alpha=100/ cell value=255 red=0 green=0 blue=0 alpha=0/ /colors /colorMap /objectLayer /layerInfo /georasterMetadata Image Structure Metadata: INTERLEAVE=PIXEL COMPRESSION=DEFLATE Corner Coordinates: Upper Left ( -703633.465, 7805920.617) ( 6d19'15.05W, 57d 2'52.80N) Lower Left ( -703633.465, 6448805.617) ( 6d19'15.05W, 49d49'30.04N) Upper Right ( 247991.535, 7805920.617) ( 2d13'39.89E, 57d 2'52.80N) Lower Right ( 247991.535, 6448805.617) ( 2d13'39.89E, 49d49'30.04N) Center ( -227820.965, 7127363.117) ( 2d 2'47.58W, 53d35'23.19N) Band 1 Block=256x256 Type=Byte, ColorInterp=Gray NoData Value=255 Overviews: 95162x135711, 47581x67855, 23790x33927, 11895x16963, 5947x8481, 2973x4240, 1486x2120, 743x1060, 371x530, 185x265, 92x132 On Fri, Aug 13, 2010 at 12:32 PM, Lucena, Ivan ivan.luc...@pmldnet.comwrote: Interesting. Can you post your gdalinfo report with -mdd oracle option? What are the
Re: [mapserver-users] logging GDAL queries
Thanks Bart... I can see the debug info and the queries now On Fri, Aug 13, 2010 at 9:36 AM, Bart van den Eijnden bart...@osgis.nlwrote: Add CONFIG CPL_DEBUG ON to your MAP object in the MAP file. Probably wise to add DEBUG 5 to your MAP and LAYER as well. Also add something like: CONFIG MS_ERRORFILE stderr and then check your Apache error log. Best regards, Bart -- Looking for flexible support on OpenLayers or GeoExt? Please check out http://www.osgis.nl/support.html Bart van den Eijnden OSGIS bart...@osgis.nl On Aug 13, 2010, at 10:33 AM, Sebastian E. Ovide wrote: Hi All I am using MapServer + GDAL to render GeoRasters and I am having very poor performance reading GeoRasters at a high zoom level... over 13... it would take even 20 seconds per tile... Is it possible to log the GDAL queries that MapServer is doing? (and the SQL queries that GDAL is doing ?) thanks -- Sebastian E. Ovide ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users -- Sebastian E. Ovide ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] logging GDAL queries
Sebastian, You are using a very small block size, maybe because you are also using ArcGIS products and they only support Oracle GeoRaster with 256x256 blocking. Is that right? Because of that, to access your GeoRaster object, the GDAL driver is requesting a very large array of BLOB (100 approximately). That can make your sever such-up memory from the machine where it is running and it is probably using virtual memory (disk) what is really slow. It also uses some memory on the client side just to hold the array too but if you are running everything (Mapserver,Oracle) on the same machine that will adds up. We are aware of that problem but it would be very helpful if you could file a bug report on GDAL's trac [http://trac.osgeo.org/gdal/] Thanks. Just as curiosity. Did you loaded that image with gdal_translate? How long it took and long does it takes to generate pyramids? Ha, you forgot to say what version of Oracle you are using. Best regards, Ivan ---Original Message--- From: Sebastian E. Ovide sebastian.ov...@gmail.com To: Lucena, Ivan ivan.luc...@pmldnet.com Cc: mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] logging GDAL queries Sent: Aug 13 '10 06:44 sure... se...@seanspc:~/mapserver/mapserver-5.6.5$ gdalinfo --version GDAL 1.7.2, released 2010/04/23 se...@seanspc:~/mapserver/mapserver-5.6.5$ gdalinfo georaster:geoserver,geoserver,MFPRODUK_11G,fluvd04q200pj,georaster,id=1 -mdd oracle Driver: GeoRaster/Oracle Spatial GeoRaster Files: none associated Size is 190325, 271423 Coordinate System is: PROJCS[Popular Visualisation CRS / Mercator (deprecated), GEOGCS[Popular Visualisation CRS, DATUM[Popular_Visualisation_Datum, SPHEROID[Popular Visualisation Sphere,6378137,0, AUTHORITY[EPSG,7059]], TOWGS84[0,0,0,0,0,0,0], AUTHORITY[EPSG,6055]], PRIMEM[Greenwich,0, AUTHORITY[EPSG,8901]], UNIT[degree,0.01745329251994328, AUTHORITY[EPSG,9122]], AUTHORITY[EPSG,4055]], UNIT[metre,1, AUTHORITY[EPSG,9001]], PROJECTION[Mercator_1SP], PARAMETER[central_meridian,0], PARAMETER[scale_factor,1], PARAMETER[false_easting,0], PARAMETER[false_northing,0], EXTENSION[PROJ4,+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgri...@null +wktext +no_defs], AUTHORITY[EPSG,3785], AXIS[X,EAST], AXIS[Y,NORTH]] Origin = (-703633.464883987908252,7805920.616936270147562) Pixel Size = (4.999,-5.000) Metadata (oracle): TABLE_NAME=fluvd04q200pj COLUMN_NAME=georaster RDT_TABLE_NAME=RDT_FLUVD04Q200PJ RASTER_ID=62 METADATA=georasterMetadata xmlns=[LINK: http://xmlns.oracle.com/spatial/georaster] http://xmlns.oracle.com/spatial/georaster; objectInfo rasterType20001/rasterType isBlankfalse/isBlank defaultRed1/defaultRed defaultGreen1/defaultGreen defaultBlue1/defaultBlue /objectInfo rasterInfo cellRepresentationUNDEFINED/cellRepresentation cellDepth8BIT_U/cellDepth NODATA255/NODATA totalDimensions2/totalDimensions dimensionSize type=ROW size271423/size /dimensionSize dimensionSize type=COLUMN size190325/size /dimensionSize ULTCoordinate row0/row column0/column /ULTCoordinate blocking typeREGULAR/type totalRowBlocks1061/totalRowBlocks totalColumnBlocks744/totalColumnBlocks rowBlockSize256/rowBlockSize columnBlockSize256/columnBlockSize /blocking interleavingBIP/interleaving pyramid typeDECREASE/type resamplingNN/resampling maxLevel11/maxLevel /pyramid compression typeDEFLATE/type /compression /rasterInfo spatialReferenceInfo isReferencedtrue/isReferenced isRectifiedtrue/isRectified SRID3785/SRID spatialResolution dimensionType=X resolution5/resolution /spatialResolution spatialResolution dimensionType=Y resolution5/resolution /spatialResolution modelCoordinateLocationCENTER/modelCoordinateLocation modelTypeFunctionalFitting/modelType polynomialModel rowOff=0 columnOff=0 xOff=0 yOff=0 zOff=0 rowScale=1 columnScale=1 xScale=1 yScale=1 zScale=1 pPolynomial pType=1 nVars=2 order=1 nCoefficients=3 polynomialCoefficients1561183.623387254 0 -0.2/polynomialCoefficients /pPolynomial qPolynomial pType=1 nVars=0 order=0 nCoefficients=1 polynomialCoefficients1/polynomialCoefficients /qPolynomial rPolynomial pType=1 nVars=2 order=1 nCoefficients=3 polynomialCoefficients140726.1929767976 0.2 0/polynomialCoefficients /rPolynomial sPolynomial pType=1 nVars=0
Re: [mapserver-users] logging GDAL queries
On Fri, Aug 13, 2010 at 1:59 PM, Ivan Lucena ivan.luc...@pmldnet.comwrote: Sebastian, You are using a very small block size, maybe because you are also using ArcGIS products and they only support Oracle GeoRaster with 256x256 blocking. Is that right? nop... I guess that 256x256 if the default gdal_translate block size ? Because of that, to access your GeoRaster object, the GDAL driver is requesting a very large array of BLOB (100 approximately). That can make your sever such-up memory from the machine where it is running and it is probably using virtual memory (disk) what is really slow. do you mean where MapServer is running ? top - 14:41:31 up 7 days, 6:20, 10 users, load average: 2.81, 1.92, 1.20 Tasks: 340 total, 2 running, 338 sleeping, 0 stopped, 0 zombie Cpu(s): 11.8%us, 10.4%sy, 7.7%ni, 62.1%id, 0.0%wa, 0.0%hi, 8.0%si, 0.0%st Mem: 12331240k total, 5992136k used, 6339104k free, 189416k buffers Swap: 9084716k total, 703708k used, 8381008k free, 1481992k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24850 www-data 20 0 533m 330m 18m R 14 2.7 0:00.94 mapserv 24852 www-data 20 0 532m 329m 18m S 19 2.7 0:01.22 mapserv 24849 www-data 20 0 529m 326m 18m S 16 2.7 0:01.00 mapserv 24851 www-data 20 0 517m 314m 18m S 13 2.6 0:00.96 mapserv 24853 www-data 20 0 499m 296m 18m S8 2.5 0:00.75 mapserv 24854 www-data 20 0 470m 267m 18m S5 2.2 0:00.53 mapserv It also uses some memory on the client side just to hold the array too but if you are running everything (Mapserver,Oracle) on the same machine that will adds up. Oracle is running on another Machine... and is very likely that it is the bottle neck... We are aware of that problem but it would be very helpful if you could file a bug report on GDAL's trac [http://trac.osgeo.org/gdal/] Thanks. I guess that the problem is that MapServer triys to read the whole map instead of reading only tile is that correct ? Just as curiosity. Did you loaded that image with gdal_translate? How long it took and long does it takes to generate pyramids? Ha, you forgot to say what version of Oracle you are using. I loaded it with gdal in around 5 minutes... the pyramid (all levels) has been created overnight Best regards, Ivan ---Original Message--- From: Sebastian E. Ovide sebastian.ov...@gmail.com To: Lucena, Ivan ivan.luc...@pmldnet.com Cc: mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] logging GDAL queries Sent: Aug 13 '10 06:44 sure... se...@seanspc:~/mapserver/mapserver-5.6.5$ gdalinfo --version GDAL 1.7.2, released 2010/04/23 se...@seanspc:~/mapserver/mapserver-5.6.5$ gdalinfo georaster:geoserver,geoserver,MFPRODUK_11G,fluvd04q200pj,georaster,id=1 -mdd oracle Driver: GeoRaster/Oracle Spatial GeoRaster Files: none associated Size is 190325, 271423 Coordinate System is: PROJCS[Popular Visualisation CRS / Mercator (deprecated), GEOGCS[Popular Visualisation CRS, DATUM[Popular_Visualisation_Datum, SPHEROID[Popular Visualisation Sphere,6378137,0, AUTHORITY[EPSG,7059]], TOWGS84[0,0,0,0,0,0,0], AUTHORITY[EPSG,6055]], PRIMEM[Greenwich,0, AUTHORITY[EPSG,8901]], UNIT[degree,0.01745329251994328, AUTHORITY[EPSG,9122]], AUTHORITY[EPSG,4055]], UNIT[metre,1, AUTHORITY[EPSG,9001]], PROJECTION[Mercator_1SP], PARAMETER[central_meridian,0], PARAMETER[scale_factor,1], PARAMETER[false_easting,0], PARAMETER[false_northing,0], EXTENSION[PROJ4,+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgri...@null +wktext +no_defs], AUTHORITY[EPSG,3785], AXIS[X,EAST], AXIS[Y,NORTH]] Origin = (-703633.464883987908252,7805920.616936270147562) Pixel Size = (4.999,-5.000) Metadata (oracle): TABLE_NAME=fluvd04q200pj COLUMN_NAME=georaster RDT_TABLE_NAME=RDT_FLUVD04Q200PJ RASTER_ID=62 METADATA=georasterMetadata xmlns=[LINK: http://xmlns.oracle.com/spatial/georaster] http://xmlns.oracle.com/spatial/georaster; objectInfo rasterType20001/rasterType isBlankfalse/isBlank defaultRed1/defaultRed defaultGreen1/defaultGreen defaultBlue1/defaultBlue /objectInfo rasterInfo cellRepresentationUNDEFINED/cellRepresentation cellDepth8BIT_U/cellDepth NODATA255/NODATA totalDimensions2/totalDimensions dimensionSize type=ROW size271423/size /dimensionSize dimensionSize type=COLUMN size190325/size /dimensionSize ULTCoordinate row0/row column0/column /ULTCoordinate blocking typeREGULAR
Re: [mapserver-users] logging GDAL queries
Sebastian, ---Original Message--- From: Sebastian E. Ovide sebastian.ov...@gmail.com To: Ivan Lucena ivan.luc...@pmldnet.com Cc: mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] logging GDAL queries Sent: Aug 13 '10 09:27 On Fri, Aug 13, 2010 at 1:59 PM, Ivan Lucena [LINK: mailto:ivan.luc...@pmldnet.com] ivan.luc...@pmldnet.com wrote: Sebastian, You are using a very small block size, maybe because you are also using ArcGIS products and they only support Oracle GeoRaster with 256x256 blocking. Is that right? nop... I guess that 256x256 if the default gdal_translate block size ? That is the default on GDAL/GeoRaster driver but you can change it using -co BLOCKXSIZE and -co BLOCKYSIZE. But that would require you to load the image again (and generate Pyramids again...) Because of that, to access your GeoRaster object, the GDAL driver is requesting a very large array of BLOB (100 approximately). That can make your sever such-up memory from the machine where it is running and it is probably using virtual memory (disk) what is really slow. do you mean where MapServer is running ? top - 14:41:31 up 7 days, 6:20, 10 users, load average: 2.81, 1.92, 1.20 Tasks: 340 total, 2 running, 338 sleeping, 0 stopped, 0 zombie Cpu(s): 11.8%us, 10.4%sy, 7.7%ni, 62.1%id, 0.0%wa, 0.0%hi, 8.0%si, 0.0%st Mem: 12331240k total, 5992136k used, 6339104k free, 189416k buffers Swap: 9084716k total, 703708k used, 8381008k free, 1481992k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 24850 www-data 20 0 533m 330m 18m R 14 2.7 0:00.94 mapserv 24852 www-data 20 0 532m 329m 18m S 19 2.7 0:01.22 mapserv 24849 www-data 20 0 529m 326m 18m S 16 2.7 0:01.00 mapserv 24851 www-data 20 0 517m 314m 18m S 13 2.6 0:00.96 mapserv 24853 www-data 20 0 499m 296m 18m S 8 2.5 0:00.75 mapserv 24854 www-data 20 0 470m 267m 18m S 5 2.2 0:00.53 mapserv Yeah. I am sorry. I called Mapserver a client :) It is a Oracle client in that case. You probably have another nice machine running Oracle. That is not the problem. It also uses some memory on the client side just to hold the array too but if you are running everything (Mapserver,Oracle) on the same machine that will adds up. Oracle is running on another Machine... and is very likely that it is the bottle neck... Maybe not but I believe you have tools on Enterprise Manager to show the volume and speed of traffic. We are aware of that problem but it would be very helpful if you could file a bug report on GDAL's trac [[LINK: http://trac.osgeo.org/gdal/] http://trac.osgeo.org/gdal/] Thanks. I guess that the problem is that MapServer triys to read the whole map instead of reading only tile is that correct ? Just as curiosity. Did you loaded that image with gdal_translate? How long it took and long does it takes to generate pyramids? Ha, you forgot to say what version of Oracle you are using. I loaded it with gdal in around 5 minutes... the pyramid (all levels) has been created overnight You original file was 51 Gb and it was loaded in 5 minutes ! See, forget all about network traffic. That is just great. Note that it doesn't matter if you generate pyramids on PL/SQL or gdaladdo, the GDAL driver will call the same PL/SQL statement that you would write yourself. So I believe that the problem is that Mapserver need to call GDALOpen() several times on that GeoRaster and that makes the GDAL driver to run the main-big-query several times. That needs to be addressed some how. Regards, Ivan ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] logging GDAL queries
Sebastian, You said: yep I did... and the weir thing is that generating the images from far away zoom under 12... it performs very well... not sure why it doesn't in the levels 13-17 for that reason I was trying to find the SQL queries But your GeoRaster Pyramid level is maxLevel11/maxLevel By the size of you image you should have 20 level. Can you run sdo_geor.generatePyramid() again changing only the number of levels, keeping the same method. That should run faster, keeping the existing pyramid levels. Please let me know how it works now. Regards, Ivan ---Original Message--- From: Ivan Lucena ivan.luc...@pmldnet.com To: Sebastian E. Ovide sebastian.ov...@gmail.com Cc: mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] logging GDAL queries Sent: Aug 13 '10 11:16 Sebastian, ---Original Message--- From: Sebastian E. Ovide sebastian.ov...@gmail.com To: Ivan Lucena ivan.luc...@pmldnet.com Cc: mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] logging GDAL queries Sent: Aug 13 '10 09:27 On Fri, Aug 13, 2010 at 1:59 PM, Ivan Lucena [LINK: mailto:ivan.luc...@pmldnet.com] ivan.luc...@pmldnet.com wrote: Sebastian, You are using a very small block size, maybe because you are also using ArcGIS products and they only support Oracle GeoRaster with 256x256 blocking. Is that right? nop... I guess that 256x256 if the default gdal_translate block size ? That is the default on GDAL/GeoRaster driver but you can change it using -co BLOCKXSIZE and -co BLOCKYSIZE. But that would require you to load the image again (and generate Pyramids again...) Because of that, to access your GeoRaster object, the GDAL driver is requesting a very large array of BLOB (100 approximately). That can make your sever such-up memory from the machine where it is running and it is probably using virtual memory (disk) what is really slow. do you mean where MapServer is running ? top - 14:41:31 up 7 days, 6:20, 10 users, load average: 2.81, 1.92, 1.20 Tasks: 340 total, 2 running, 338 sleeping, 0 stopped, 0 zombie Cpu(s): 11.8%us, 10.4%sy, 7.7%ni, 62.1%id, 0.0%wa, 0.0%hi, 8.0%si, 0.0%st Mem: 12331240k total, 5992136k used, 6339104k free, 189416k buffers Swap: 9084716k total, 703708k used, 8381008k free, 1481992k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 24850 www-data 20 0 533m 330m 18m R 14 2.7 0:00.94 mapserv 24852 www-data 20 0 532m 329m 18m S 19 2.7 0:01.22 mapserv 24849 www-data 20 0 529m 326m 18m S 16 2.7 0:01.00 mapserv 24851 www-data 20 0 517m 314m 18m S 13 2.6 0:00.96 mapserv 24853 www-data 20 0 499m 296m 18m S 8 2.5 0:00.75 mapserv 24854 www-data 20 0 470m 267m 18m S 5 2.2 0:00.53 mapserv Yeah. I am sorry. I called Mapserver a client :) It is a Oracle client in that case. You probably have another nice machine running Oracle. That is not the problem. It also uses some memory on the client side just to hold the array too but if you are running everything (Mapserver,Oracle) on the same machine that will adds up. Oracle is running on another Machine... and is very likely that it is the bottle neck... Maybe not but I believe you have tools on Enterprise Manager to show the volume and speed of traffic. We are aware of that problem but it would be very helpful if you could file a bug report on GDAL's trac [[LINK: http://trac.osgeo.org/gdal/] http://trac.osgeo.org/gdal/] Thanks. I guess that the problem is that MapServer triys to read the whole map instead of reading only tile is that correct ? Just as curiosity. Did you loaded that image with gdal_translate? How long it took and long does it takes to generate pyramids? Ha, you forgot to say what version of Oracle you are using. I loaded it with gdal in around 5 minutes... the pyramid (all levels) has been created overnight You original file was 51 Gb and it was loaded in 5 minutes ! See, forget all about network traffic. That is just great. Note that it doesn't matter if you generate pyramids on PL/SQL or gdaladdo, the GDAL driver will call the same PL/SQL statement that you would write yourself. So I believe that the problem is that Mapserver need to call GDALOpen() several times on that GeoRaster and that makes the GDAL driver to run the main-big-query several times. That needs to be addressed some how. Regards, Ivan ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo
