Hi Adam, For working with very large point clouds Pete gave a workshop covering using SPDLib on a HPC. The notes are available from:
https://spectraldifferences.files.wordpress.com/2013/08/spd_worksheet_intro.pdf <https://spectraldifferences.files.wordpress.com/2013/08/spd_worksheet_intro.pdf> One of the things you want to be doing for large datasets is using a temp directory for the indexing rather than doing the whole thing in memory (this is covered in the worksheet). I realise the conda install is a bit of a mess at the moment - we’ve been trying to get it working using the conda-forge builds rather than building everything ourselves which should improve things but still a work in process at the moment. Unfortunately both Pete and myself have limited time to work on SPDLib as we are busy on other projects. We’re open to collaborative projects involving SPDLib, which might be a good way of moving development forward faster. If there is a link with NERC-ARF LiDAR data (https://nerc-arf-dan.pml.ac.uk/ <https://nerc-arf-dan.pml.ac.uk/>) that would be of particular interest to me. Happy to discuss off-list. Thanks, Dan > On 1 Aug 2016, at 08:49, Adam Steer <[email protected]> wrote: > > Hi SPDlib folks > > I’ve just started to tinker with SPDlib, with the aim of working out how to > use point clouds as a data service - this example: > > https://spectraldifferences.wordpress.com/2014/08/06/dem-spdlib/ > > …is pretty representative of what we want to do - get non-specialist users to > be able to work with point data we currently hold, and will hold in future. > > I wonder - has anyone tested the limits of SPD yet? I’m right now attempting > to prepare an SPD file with 400 million points, which replicates a > postgres-pointcloud database and a set of flat netCDF files with the same > dataset. > > I’m having a ‘fun’ start with SPD, it’s currently quite a mess - I’ve got a > functioning v3.3, following Daniel Clewleys instructions just posted (conda > create -n spdlib_v3_3 -c osgeo -c rios -c conda-forge spdlib=3.3.0). > > …and then see how well SPDlib functions for point retrieval and doing useful > jobs with data subsets. It’s been used for Australian data before, but for > only small blocks. We need to get much bigger! > > To do so, I’m actually working on the some of the same data as Narsimha, but > with a slightly different focus. And here’s my first result: > > spdtranslate --if LAS --of SPD -b 10 -x FIRST_RETURN -i > ACT2015-C3-ELL_6886094_55_0002_0002_l13.las -o > ACT2015-C3-ELL_6886094_55_0002_0002_l13.spd > ------------- > 1.5G > /mnt/vol1/pointcloud-data/ACT_Lidar_8pt/Ellipsoidal/las1.3/ACT2015-C3-ELL_6886092_55_0002_0002_l13.las > 975M > /mnt/vol1/pointcloud-data/ACT_Lidar_8pt/Ellipsoidal/las1.3/ACT2015-C3-ELL_6886092_55_0002_0002_l13.spd > > (blocksize choice is quite naive, data are ~8pt/m - I’ll investigate what is > in the blocks tomorrow :) ) > > …so a slight storage improvement, and now it’s indexed as well! > > Anyway - that is a distraction. I’d like to hear what experiences people have > with properly large point clouds. So far it’s chewing up a lot of memory > converting relatively small datasets (~90 million points, without pulse data) > to spd, and I’d like to scale up to many hundreds of millions to billions of > points. > > Also, is there any benefit at present to going through a ground up SPDlib 1.4 > installation? > > Cheers > > Adam > > ps - if anyone is going to FOSS4G in a few weeks and is happy to talk about > SPDlib and huge datasets, drop me a line off list - it would be great to see > what other organisations are up to. > > > -- > Adam Steer > Earth systems data service specialist > National Computational Infrastructure > Leonard Huxley Building, Mills Road > The Australian National University > Acton ACT 2600 Australia > > [email protected] > +61 2 6125 1413 > http://nci.org.au > > > > ------------------------------------------------------------------------------ > _______________________________________________ > spdlib-develop mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/spdlib-develop
------------------------------------------------------------------------------
_______________________________________________ spdlib-develop mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/spdlib-develop
