Hi Pete, Dan Thanks for your responses - I’ll work through your documentation in the next couple of days and see what questions arise.
I’m using Australian LiDAR, which is a bit patchy at the moment. When things get organised (not so far off) we are going to have many hundreds of billions of points to manage - both for on-and offshore environments :) Much appreciated, Adam > On 1 Aug 2016, at 6:46 PM, Daniel Clewley <[email protected]> wrote: > > Hi Adam, > > For working with very large point clouds Pete gave a workshop covering using > SPDLib on a HPC. The notes are available from: > > https://spectraldifferences.files.wordpress.com/2013/08/spd_worksheet_intro.pdf > > One of the things you want to be doing for large datasets is using a temp > directory for the indexing rather than doing the whole thing in memory (this > is covered in the worksheet). > > I realise the conda install is a bit of a mess at the moment - we’ve been > trying to get it working using the conda-forge builds rather than building > everything ourselves which should improve things but still a work in process > at the moment. Unfortunately both Pete and myself have limited time to work > on SPDLib as we are busy on other projects. > > We’re open to collaborative projects involving SPDLib, which might be a good > way of moving development forward faster. If there is a link with NERC-ARF > LiDAR data (https://nerc-arf-dan.pml.ac.uk/) that would be of particular > interest to me. Happy to discuss off-list. > > Thanks, > > Dan > >> On 1 Aug 2016, at 08:49, Adam Steer <[email protected]> wrote: >> >> Hi SPDlib folks >> >> I’ve just started to tinker with SPDlib, with the aim of working out how to >> use point clouds as a data service - this example: >> >> https://spectraldifferences.wordpress.com/2014/08/06/dem-spdlib/ >> >> …is pretty representative of what we want to do - get non-specialist users >> to be able to work with point data we currently hold, and will hold in >> future. >> >> I wonder - has anyone tested the limits of SPD yet? I’m right now attempting >> to prepare an SPD file with 400 million points, which replicates a >> postgres-pointcloud database and a set of flat netCDF files with the same >> dataset. >> >> I’m having a ‘fun’ start with SPD, it’s currently quite a mess - I’ve got a >> functioning v3.3, following Daniel Clewleys instructions just posted (conda >> create -n spdlib_v3_3 -c osgeo -c rios -c conda-forge spdlib=3.3.0). >> >> …and then see how well SPDlib functions for point retrieval and doing useful >> jobs with data subsets. It’s been used for Australian data before, but for >> only small blocks. We need to get much bigger! >> >> To do so, I’m actually working on the some of the same data as Narsimha, but >> with a slightly different focus. And here’s my first result: >> >> spdtranslate --if LAS --of SPD -b 10 -x FIRST_RETURN -i >> ACT2015-C3-ELL_6886094_55_0002_0002_l13.las -o >> ACT2015-C3-ELL_6886094_55_0002_0002_l13.spd >> ------------- >> 1.5G >> /mnt/vol1/pointcloud-data/ACT_Lidar_8pt/Ellipsoidal/las1.3/ACT2015-C3-ELL_6886092_55_0002_0002_l13.las >> 975M >> /mnt/vol1/pointcloud-data/ACT_Lidar_8pt/Ellipsoidal/las1.3/ACT2015-C3-ELL_6886092_55_0002_0002_l13.spd >> >> (blocksize choice is quite naive, data are ~8pt/m - I’ll investigate what is >> in the blocks tomorrow :) ) >> >> …so a slight storage improvement, and now it’s indexed as well! >> >> Anyway - that is a distraction. I’d like to hear what experiences people >> have with properly large point clouds. So far it’s chewing up a lot of >> memory converting relatively small datasets (~90 million points, without >> pulse data) to spd, and I’d like to scale up to many hundreds of millions to >> billions of points. >> >> Also, is there any benefit at present to going through a ground up SPDlib >> 1.4 installation? >> >> Cheers >> >> Adam >> >> ps - if anyone is going to FOSS4G in a few weeks and is happy to talk about >> SPDlib and huge datasets, drop me a line off list - it would be great to see >> what other organisations are up to. >> >> >> -- >> Adam Steer >> Earth systems data service specialist >> National Computational Infrastructure >> Leonard Huxley Building, Mills Road >> The Australian National University >> Acton ACT 2600 Australia >> >> [email protected] >> +61 2 6125 1413 >> http://nci.org.au >> >> >> >> ------------------------------------------------------------------------------ >> _______________________________________________ >> spdlib-develop mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/spdlib-develop > ------------------------------------------------------------------------------ _______________________________________________ spdlib-develop mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/spdlib-develop
