Re: [spdlib-develop] Hi, and a quick question - how many points have you stuffed into SPDLib?

Pete Bunting Mon, 01 Aug 2016 02:11:16 -0700

Hi Adam,

Sorry just heading off on vacation so can’t give a fuller answer, but yes we’ve done quite a bit of work with larger datasets with SPDLib. The processing chain I use is in the attached scripts, it refers to some custom scripts (e.g, Submit2bsub.py) these are for helping me work our HPC - you can just run the shell script the previous step outputs either linearly or using something like GNU parallel to multi-thread it. The basic method is that I tiling the data (e.g., 1x1km) with an overlap (50m) and then process the tiles and mosaic the results. The script BuildSPDProcessingBSUB.py builds all the processing steps for each tile and has all the command options / sequence that I tend to use.

Many thanks,

Pete

spdbatchgen.py -i ./convert2UPDTemplate.sh -o convert2UPD_all.sh  -p ./UPD -d 
./LAS -e .las


python ~/Scripts/Submit2bsub.py  -i convert2UPD_all.sh -o convert2UPD -m 4000 
-t 00:60 -n Liwale_Convert2UPD

###########################################
# WAIT FOR JOBS TO BE EXECUTED
###########################################

sh ./Scripts/rmZeroErrOutFiles.sh ./Liwale_Convert2UPD_

find ./UPD/ -name *.spd > InputUPDFileList.lst

spddeftiles --extent --input ./InputUPDFileList.lst

# Extent [xMin, xMax, yMin, yMax]: [422386.601809, 458428.0088, 8897338.64652, 
8909308.85486]
#                                   422300         458500       8897000        
8909500
##############################################################
# RUN TO HERE AND THEN NEED TO MANUALLY ENTER THE EXTENTS
# OUTPUTTED FROM THE spddeftiles COMMAND
##############################################################

spddeftiles --tiles --output ./LiwaleLSUTiles.xml --xmin 422300 \
--xmax 458500 --ymin 8897000 --ymax 8909500 --overlap 100 \
--ysize 1000 --xsize 1000

mkdir Tiles

spdtiling --builddirs -t ./LiwaleLSUTiles.xml -o ./Tiles/

spdbatchgen.py -i TileUPDFilesTemplate.sh -o TileUPDFiles.sh -p . -d ./UPD/ -e 
.spd --output_type single

python ~/Scripts/Submit2bsub.py -i TileUPDFiles.sh -o TileLiDARFlightlines -m 
2000 -t 00:60 -n TileLiDARUPD

###########################################
# WAIT FOR JOBS TO BE EXECUTED
###########################################

sh ./Scripts/rmZeroErrOutFiles.sh ./TileLiDARUPD_

spdtiling --rmdirs -o ./Tiles/ -t ./LiwaleLSUTiles.xml

python ./BuildSPDProcessingBSUB.py -i ./Tiles/ -t ./LiwaleLSUTiles.xml

python ./SubmitSPDProcessing2BSUB.py  -i ./Tiles/ -t ./LiwaleLSUTiles.xml

###########################################
# WAIT FOR JOBS TO BE EXECUTED
###########################################

python ./SubmitSPDMetrics2BSUB.py  -i ./Tiles/ -t ./LiwaleLSUTiles.xml

###########################################
# WAIT FOR JOBS TO BE EXECUTED
###########################################

bsub < runMASKMosaic.lsf
bsub < runDTMMosaic.lsf
bsub < runDTMHSMosaic.lsf
bsub < runCHMMosaic.lsf
bsub < runCHMMosaic5mMin.lsf
bsub < runCHMMosaic2mMin.lsf
bsub < runHeightMetricsMosaic.lsf

find ./Tiles/ -name *_10m_rmn_pmfmccgrd_h.spd > FinalSPDFiles.lst

spdtiling --upxml --input  ./FinalSPDFiles.lst --tiles ./LiwaleLSUTiles.xml

spdtiling --xml2shp --deleteshp --wkt ./UTM37S.wkt --tiles ./LiwaleLSUTiles.xml 
-o ./LiwaleLSU_TilePolys.shp

###########################################
# DONE
###########################################

runDTMMosaic.lsf
Description: Binary data

#! /usr/bin/env python

############################################################################
# Copyright (c) 2013 Dr. Peter Bunting, Aberystwyth University
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
#
#
# Purpose:  A class to submit jobs to loadleveler
# Author: Pete Bunting
# Email: petebunt...@mac.com
# Date: 19/04/2013
# Version: 1.0
#
# History:
# Version 1.0 - Created.
#
#############################################################################

import os.path
import sys
from time import strftime
import argparse
import os
# Import python XML Parser
import xml.etree.ElementTree as ET

class BuildLoadLevelerCommands (object):
    
    def createLoadLevelerScripts(self, inputDIR, inputTilesXML):
        inputTilesXML = os.path.abspath(inputTilesXML)
        inputDIR = os.path.abspath(inputDIR)
        tree = ET.parse(inputTilesXML)
        root = tree.getroot()
        if root.tag == "tiles":
            rows = int(root.attrib['rows'])
            cols = int(root.attrib['columns'])
            print("ROWS = ", rows)
            print("COLS = ", cols)
            
            for row in range(rows):
                for col in range(cols):
                    dirRowPath = os.path.join(inputDIR, "Rows"+str(row+1)) 
                    dirRowColPath = os.path.join(dirRowPath, "Cols"+str(col+1)) 
                    lidarOutBase = "als_dr_opt3_row"+str(row+1)+"col"+str(col+1)+"_2014_utm37s"
                    llFileName = "TileProcessig_row"+str(row+1)+"col"+str(col+1)+".lsf"

                    
                    if(os.path.exists(dirRowColPath)):
                        ##### CREATE LoadLeveler File #####
                        llFilePath = os.path.join(dirRowColPath, llFileName)
                        llFile = open(llFilePath, 'w')
                        
                        memory = 8000
                        ##### ADD LoadLeveler header info #####
                        llFile.write("#!/bin/bash --login\n")
                        llFile.write("#BSUB -J als_row"+str(row+1)+"col"+str(col+1) + str("\n"))
                        llFile.write("#BSUB -o als_row"+str(row+1)+"col"+str(col+1) + str(".out\n"))
                        llFile.write("#BSUB -e als_row"+str(row+1)+"col"+str(col+1) + str(".err\n"))
                        llFile.write("#BSUB -W 04:00\n")
                        llFile.write("#BSUB -M " + str(int(memory)*1024) + "\n")
                        llFile.write("#BSUB -P SAM0102\n")
                        llFile.write("#BSUB -n 1\n")
                        llFile.write("#BSUB -R span[ptile=1]\n\n")
                        llFile.write("ulimit -v " + str(int(memory)*1024)  + " -m " + str(int(memory)*1024)  + "\n\n")

                        ##### ADD MERGE COMMAND #####
                        llFile.write("############# SPDMERGE #############\n")
                        llFile.write("if [ ! -e \"" + os.path.join(dirRowColPath, lidarOutBase + ".spd") + "\" ]\n")
                        llFile.write("then\n")
                        dirRowColTilesPath = os.path.join(dirRowColPath, "Tiles")
                        tileFiles = os.listdir(dirRowColTilesPath)
                        if(len(tileFiles) == 1):
                            cmd = "    cp " + os.path.join(dirRowColTilesPath, tileFiles[0]) + " " + os.path.join(dirRowColPath, lidarOutBase + ".spd")
                            llFile.write(cmd + "\n")
                        elif(len(tileFiles) > 1):
                            cmd = "    spdmerge -f SPD --source --ignorechecks --keepextent -o " + os.path.join(dirRowColPath, lidarOutBase + ".spd")
                            for file in tileFiles:
                                cmd = cmd + " " + os.path.join(dirRowColTilesPath, file)
                            llFile.write(cmd + "\n")
                        else:
                            print("There are no tiles: ", dirRowColTilesPath)
                            print("You need to either remove directory or running the tiling commands.")
                            sys.exit(-1)
                        llFile.write("fi\n")
                        llFile.write("###########################################\n\n\n")
                    
                        ##### ADD SPDTRANSLATE COMMAND #####
                        llFile.write("############# SPDTRANSLATE #############\n")
                        llFile.write("if [ ! -e \"" + os.path.join(dirRowColPath, "tmp") + "\" ]\n")
                        llFile.write("then\n")
                        llFile.write("    mkdir " + os.path.join(dirRowColPath, "tmp") + "\n")
                        llFile.write("fi\n")
                        
                        llFile.write("if [ -e \"" + os.path.join(dirRowColPath, lidarOutBase + ".spd") + "\" ] && [ ! -e \"" + os.path.join(dirRowColPath, lidarOutBase + "_10m.spd") + "\" ]\n")
                        llFile.write("then\n")
                        llFile.write("    spdtranslate --if SPD --of SPD --temppath " + os.path.join(dirRowColPath, "tmp/") + " --numofrows 50 -b 10 --keepextent -i " + os.path.join(dirRowColPath, lidarOutBase + ".spd") + " -o " + os.path.join(dirRowColPath, lidarOutBase + "_10m.spd") + "\n")
                        llFile.write("fi\n")
                        llFile.write("###########################################\n\n\n")
                        
                        ##### ADD RMNOISE COMMAND #####
                        llFile.write("############# SPDRMNOISE #############\n")
                        llFile.write("if [ -e \"" + os.path.join(dirRowColPath, lidarOutBase + "_10m.spd") + "\" ] && [ ! -e \"" + os.path.join(dirRowColPath, lidarOutBase + "_10m_rmn_tmp.spd") + "\" ]\n")
                        llFile.write("then\n")
                        llFile.write("    spdrmnoise -c 50 -r 50 --relup 100 --grelup 400 -i " + os.path.join(dirRowColPath, lidarOutBase + "_10m.spd") + " -o " + os.path.join(dirRowColPath, lidarOutBase + "_10m_rmn_tmp.spd") + "\n")
                        llFile.write("fi\n")
                        llFile.write("###########################################\n\n\n")
                        
                        ##### ADD SPDTRANSLATE COMMAND #####
                        llFile.write("############# SPDTRANSLATE #############\n")
                        llFile.write("if [ ! -e \"" + os.path.join(dirRowColPath, "tmp") + "\" ]\n")
                        llFile.write("then\n")
                        llFile.write("    mkdir " + os.path.join(dirRowColPath, "tmp") + "\n")
                        llFile.write("fi\n")
                        
                        llFile.write("if [ -e \"" + os.path.join(dirRowColPath, lidarOutBase + "_10m_rmn_tmp.spd") + "\" ] && [ ! -e \"" + os.path.join(dirRowColPath, lidarOutBase + "_10m_rmn.spd") + "\" ]\n")
                        llFile.write("then\n")
                        llFile.write("    spdtranslate --if SPD --of SPD --temppath " + os.path.join(dirRowColPath, "tmp/") + " --numofrows 50 -b 10 --keepextent -i " + os.path.join(dirRowColPath, lidarOutBase + "_10m_rmn_tmp.spd") + " -o " + os.path.join(dirRowColPath, lidarOutBase + "_10m_rmn.spd") + "\n")
                        llFile.write("fi\n")
                        llFile.write("###########################################\n\n\n")
                        
                        ##### ADD PMF COMMAND #####
                        llFile.write("############# SPDPMFGRD #############\n")
                        llFile.write("if [ -e \"" + os.path.join(dirRowColPath, lidarOutBase + "_10m_rmn.spd") + "\" ] && [ ! -e \"" + os.path.join(dirRowColPath, lidarOutBase + "_10m_rmn_pmfgrd.spd") + "\" ]\n")
                        llFile.write("then\n")
                        llFile.write("    spdpmfgrd -c 50 -r 50 -b 1 --grd 1 -i " + os.path.join(dirRowColPath, lidarOutBase + "_10m_rmn.spd") + " -o " + os.path.join(dirRowColPath, lidarOutBase + "_10m_rmn_pmfgrd.spd") + "\n")
                        llFile.write("fi\n")
                        llFile.write("###########################################\n\n\n")
                        
                        ##### ADD MCC COMMAND #####
                        llFile.write("############# SPDMCCGRD #############\n")
                        llFile.write("if [ -e \"" + os.path.join(dirRowColPath, lidarOutBase + "_10m_rmn_pmfgrd.spd") + "\" ] && [ ! -e \"" + os.path.join(dirRowColPath, lidarOutBase + "_10m_rmn_pmfmccgrd.spd") + "\" ]\n")
                        llFile.write("then\n")
                        llFile.write("    spdmccgrd -c 50 -r 50 -b 1 --class 3 --initcurvetol 1 -i " + os.path.join(dirRowColPath, lidarOutBase + "_10m_rmn_pmfgrd.spd") + " -o " + os.path.join(dirRowColPath, lidarOutBase + "_10m_rmn_pmfmccgrd.spd") + "\n")
                        llFile.write("fi\n")
                        llFile.write("###########################################\n\n\n")
                        
                        ##### ADD SPDDEFHEIGHT COMMAND #####
                        llFile.write("############# SPDDEFHEIGHT #############\n")
                        llFile.write("if [ -e \"" + os.path.join(dirRowColPath, lidarOutBase + "_10m_rmn_pmfmccgrd.spd") + "\" ] && [ ! -e \"" + os.path.join(dirRowColPath, lidarOutBase + "_10m_rmn_pmfmccgrd_h.spd") + "\" ]\n")
                        llFile.write("then\n")
                        llFile.write("    spddefheight --interp -c 50 -r 50 --in NATURAL_NEIGHBOR -i " + os.path.join(dirRowColPath, lidarOutBase + "_10m_rmn_pmfmccgrd.spd") + " -o " + os.path.join(dirRowColPath, lidarOutBase + "_10m_rmn_pmfmccgrd_h.spd") + "\n")
                        llFile.write("fi\n")
                        llFile.write("###########################################\n\n\n")

                        ##### ADD DTM COMMAND #####
                        llFile.write("############# DTM #############\n")
                        llFile.write("if [ -e \"" + os.path.join(dirRowColPath, lidarOutBase + "_10m_rmn_pmfmccgrd_h.spd") + "\" ] && [ ! -e \"" + os.path.join(dirRowColPath, lidarOutBase + "_rmn_pmfmccgrd_1m_DTM.kea") + "\" ]\n")
                        llFile.write("then\n")
                        llFile.write("    spdinterp --dtm --topo --in NATURAL_NEIGHBOR -f KEA -b 1 -c 50 -r 50 --overlap 10 -i " + os.path.join(dirRowColPath, lidarOutBase + "_10m_rmn_pmfmccgrd_h.spd") + " -o " + os.path.join(dirRowColPath, lidarOutBase + "_rmn_pmfmccgrd_1m_DTM.kea") + "\n")
                        llFile.write("    gdalcalcstats " + os.path.join(dirRowColPath, lidarOutBase + "_rmn_pmfmccgrd_1m_DTM.kea") + " -ignore 0\n")
                        llFile.write("fi\n")
                        llFile.write("###########################################\n\n\n")
                        
                        ##### ADD CHM COMMAND #####
                        llFile.write("############# CHM #############\n")
                        llFile.write("if [ -e \"" + os.path.join(dirRowColPath, lidarOutBase + "_10m_rmn_pmfmccgrd_h.spd") + "\" ] && [ ! -e \"" + os.path.join(dirRowColPath, lidarOutBase + "_rmn_pmfmccgrd_1m_CHM.kea") + "\" ]\n")
                        llFile.write("then\n")
                        llFile.write("    spdinterp --dsm --height --in NATURAL_NEIGHBOR -f KEA -b 1 -c 50 -r 50 --overlap 10 -i " + os.path.join(dirRowColPath, lidarOutBase + "_10m_rmn_pmfmccgrd_h.spd") + " -o " + os.path.join(dirRowColPath, lidarOutBase + "_rmn_pmfmccgrd_1m_CHM.kea") + "\n")
                        llFile.write("    gdalcalcstats " + os.path.join(dirRowColPath, lidarOutBase + "_rmn_pmfmccgrd_1m_CHM.kea") + " -ignore 0\n")
                        llFile.write("fi\n")
                        llFile.write("###########################################\n\n\n")
                        
                        ##### ADD CHM 2 m MIN COMMAND #####
                        llFile.write("############# CHM Tidy 2m Min #############\n")
                        llFile.write("if [ -e \"" + os.path.join(dirRowColPath, lidarOutBase + "_10m_rmn_pmfmccgrd_h.spd") + "\" ] && [ ! -e \"" + os.path.join(dirRowColPath, lidarOutBase + "_rmn_pmfmccgrd_1m_CHM_2mMin.kea") + "\" ]\n")
                        llFile.write("then\n")
                        llFile.write("    python /home/pete.bunting/Tanzania/LiDAR/RunCHMTidy.py -i " + os.path.join(dirRowColPath, lidarOutBase + "_rmn_pmfmccgrd_1m_CHM.kea") + " -o " + os.path.join(dirRowColPath, lidarOutBase + "_rmn_pmfmccgrd_1m_CHM_2mMin.kea") + " -t 2\n")
                        llFile.write("    gdalcalcstats " + os.path.join(dirRowColPath, lidarOutBase + "_rmn_pmfmccgrd_1m_CHM_2mMin.kea") + " -ignore 0\n")
                        llFile.write("fi\n")
                        llFile.write("###########################################\n\n\n")
                        
                        ##### ADD CHM 5 m MIN COMMAND #####
                        llFile.write("############# CHM Tidy 5m Min #############\n")
                        llFile.write("if [ -e \"" + os.path.join(dirRowColPath, lidarOutBase + "_10m_rmn_pmfmccgrd_h.spd") + "\" ] && [ ! -e \"" + os.path.join(dirRowColPath, lidarOutBase + "_rmn_pmfmccgrd_1m_CHM_5mMin.kea") + "\" ]\n")
                        llFile.write("then\n")
                        llFile.write("    python /home/pete.bunting/Tanzania/LiDAR/RunCHMTidy.py -i " + os.path.join(dirRowColPath, lidarOutBase + "_rmn_pmfmccgrd_1m_CHM.kea") + " -o " + os.path.join(dirRowColPath, lidarOutBase + "_rmn_pmfmccgrd_1m_CHM_5mMin.kea") + " -t 2\n")
                        llFile.write("    gdalcalcstats " + os.path.join(dirRowColPath, lidarOutBase + "_rmn_pmfmccgrd_1m_CHM_5mMin.kea") + " -ignore 0\n")
                        llFile.write("fi\n")
                        llFile.write("###########################################\n\n\n")
                        
                        ##### ADD STATS COMMAND #####
                        llFile.write("############# STATS #############\n")
                        llFile.write("if [ -e \"" + os.path.join(dirRowColPath, lidarOutBase + "_10m_rmn_pmfmccgrd_h.spd") + "\" ] && [ ! -e \"" + os.path.join(dirRowColPath, lidarOutBase + "_rmn_pmfmccgrd_1m_STATS.kea") + "\" ]\n")
                        llFile.write("then\n")
                        llFile.write("    spdstats --image -c 50 -r 50 -b 1 -f KEA -i " + os.path.join(dirRowColPath, lidarOutBase + "_10m_rmn_pmfmccgrd_h.spd") + " -o " + os.path.join(dirRowColPath, lidarOutBase + "_rmn_pmfmccgrd_1m_STATS.kea") + "\n")
                        llFile.write("    gdalcalcstats " + os.path.join(dirRowColPath, lidarOutBase + "_rmn_pmfmccgrd_1m_STATS.kea") + " -ignore 0\n")
                        llFile.write("fi\n")
                        llFile.write("###########################################\n\n\n")
                        
                        ##### ADD MASK COMMAND #####
                        llFile.write("############# MASK #############\n")
                        llFile.write("if [ -e \"" + os.path.join(dirRowColPath, lidarOutBase + "_10m_rmn_pmfmccgrd_h.spd") + "\" ] && [ ! -e \"" + os.path.join(dirRowColPath, lidarOutBase + "_rmn_pmfmccgrd_1m_MASK.kea") + "\" ]\n")
                        llFile.write("then\n")
                        llFile.write("    spdmaskgen -c 50 -r 50 -b 1 -f KEA -p 1 -i " + os.path.join(dirRowColPath, lidarOutBase + "_10m_rmn_pmfmccgrd_h.spd") + " -o " + os.path.join(dirRowColPath, lidarOutBase + "_rmn_pmfmccgrd_1m_MASK.kea") + "\n")
                        llFile.write("    gdalcalcstats " + os.path.join(dirRowColPath, lidarOutBase + "_rmn_pmfmccgrd_1m_MASK.kea") + " -ignore 0\n")
                        llFile.write("fi\n")
                        llFile.write("###########################################\n\n\n")
                                                                        
                        ##### ADD DTM HILLSHADE COMMAND #####
                        llFile.write("############# DTM HILLSHADE #############\n")
                        llFile.write("if [ -e \"" + os.path.join(dirRowColPath, lidarOutBase + "_rmn_pmfmccgrd_1m_DTM.kea") + "\" ] && [ ! -e \"" + os.path.join(dirRowColPath, lidarOutBase + "_rmn_pmfmccgrd_1m_DTM_HS.kea") + "\" ]\n")
                        llFile.write("then\n")
                        llFile.write("    gdaldem hillshade -of KEA " + os.path.join(dirRowColPath, lidarOutBase + "_rmn_pmfmccgrd_1m_DTM.kea") + "  " + os.path.join(dirRowColPath, lidarOutBase + "_rmn_pmfmccgrd_1m_DTM_HS.kea") + "\n")
                        llFile.write("    gdalcalcstats " + os.path.join(dirRowColPath, lidarOutBase + "_rmn_pmfmccgrd_1m_DTM_HS.kea") + " -ignore 0\n")
                        llFile.write("fi\n")
                        llFile.write("###########################################\n\n\n")
                        
                        llFile.close()

    
    def run(self):
        parser = argparse.ArgumentParser()
        parser.add_argument("-i", "--dir", dest="inputdir", type=str, help="Input directory")
        parser.add_argument("-t", "--tiles", dest="tilesXMLfile", type=str, help="Input tile XML file.")

        args = parser.parse_args()    

        if args.inputdir is None:
            print("No input directory specified.")
            parser.print_help()
            sys.exit()
            
        if args.tilesXMLfile is None:
            print("No input tiles XML file specified.")
            parser.print_help()
            sys.exit()
            
        self.createLoadLevelerScripts(args.inputdir, args.tilesXMLfile)

if __name__ == '__main__':
    obj = BuildLoadLevelerCommands()
    obj.run()

TileUPDFilesTemplate.sh
Description: Binary data

convert2UPDTemplate.sh
Description: Binary data

#! /usr/bin/env python

############################################################################
# Copyright (c) 2013 Dr. Peter Bunting, Aberystwyth University
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
#
#
# Purpose:  A class to submit jobs to loadleveler
# Author: Pete Bunting
# Email: petebunt...@mac.com
# Date: 19/04/2013
# Version: 1.0
#
# History:
# Version 1.0 - Created.
#
#############################################################################

import os.path
import sys
from time import strftime
import argparse
import os
# Import python XML Parser
import xml.etree.ElementTree as ET

class SubmitLoadLevelerCommands (object):
    
    def submitLoadLevelerScripts(self, inputDIR, inputTilesXML):
        tree = ET.parse(inputTilesXML)
        root = tree.getroot()
        if root.tag == "tiles":
            rows = int(root.attrib['rows'])
            cols = int(root.attrib['columns'])
            print("ROWS = ", rows)
            print("COLS = ", cols)
            
            for row in range(rows):
                for col in range(cols):
                    dirRowPath = os.path.join(inputDIR, "Rows"+str(row+1)) 
                    dirRowColPath = os.path.join(dirRowPath, "Cols"+str(col+1)) 
                    llFileName = "TileProcessig_row"+str(row+1)+"col"+str(col+1)+".lsf"
                    
                    if(os.path.exists(dirRowColPath)):
                        ##### CREATE LoadLeveler File #####
                        llFilePath = os.path.join(dirRowColPath, llFileName)
                        command = str("bsub < ") + llFilePath
                        print(command)
                        os.system(command)
                        
                        

    
    def run(self):
        parser = argparse.ArgumentParser()
        parser.add_argument("-i", "--dir", dest="inputdir", type=str, help="Input directory")
        parser.add_argument("-t", "--tiles", dest="tilesXMLfile", type=str, help="Input tile XML file.")

        args = parser.parse_args()    

        if args.inputdir is None:
            print("No input directory specified.")
            parser.print_help()
            sys.exit()
            
        if args.tilesXMLfile is None:
            print("No input tiles XML file specified.")
            parser.print_help()
            sys.exit()
            
        self.submitLoadLevelerScripts(args.inputdir, args.tilesXMLfile)

if __name__ == '__main__':
    obj = SubmitLoadLevelerCommands()
    obj.run()

****************************************************
* Dr Pete Bunting
* Reader in Remote Sensing
* Earth Observation and Ecosystem Dynamics Group
* Department of Geography and Earth Sciences
* Aberystwyth University
* Aberystwyth
* Ceredigion
* SY23 3DB
* UK
*
* Ph: +44 (0) 1970 622615
* Mob: +44 (0) 7917 842743
* Email: p...@aber.ac.uk
* ORCID: http://orcid.org/0000-0002-7435-0148
****************************************************

"Please consider the environment before printing this email or any documents attached”

On 1 Aug 2016, at 08:49, Adam Steer <adam.st...@anu.edu.au> wrote:

Hi SPDlib folks

I’ve just started to tinker with SPDlib, with the aim of working out how to use point clouds as a data service - this example:

https://spectraldifferences.wordpress.com/2014/08/06/dem-spdlib/

…is pretty representative of what we want to do - get non-specialist users to be able to work with point data we currently hold, and will hold in future.

I wonder - has anyone tested the limits of SPD yet? I’m right now attempting to prepare an SPD file with 400 million points, which replicates a postgres-pointcloud database and a set of flat netCDF files with the same dataset.

I’m having a ‘fun’ start with SPD, it’s currently quite a mess - I’ve got a functioning v3.3, following Daniel Clewleys instructions just posted (conda create -n spdlib_v3_3 -c osgeo -c rios -c conda-forge spdlib=3.3.0).

…and then see how well SPDlib functions for point retrieval and doing useful jobs with data subsets. It’s been used for Australian data before, but for only small blocks. We need to get much bigger!

To do so, I’m actually working on the some of the same data as Narsimha, but with a slightly different focus. And here’s my first result:

spdtranslate --if LAS --of SPD -b 10 -x FIRST_RETURN -i ACT2015-C3-ELL_6886094_55_0002_0002_l13.las -o ACT2015-C3-ELL_6886094_55_0002_0002_l13.spd
-------------
1.5G /mnt/vol1/pointcloud-data/ACT_Lidar_8pt/Ellipsoidal/las1.3/ACT2015-C3-ELL_6886092_55_0002_0002_l13.las
975M /mnt/vol1/pointcloud-data/ACT_Lidar_8pt/Ellipsoidal/las1.3/ACT2015-C3-ELL_6886092_55_0002_0002_l13.spd

(blocksize choice is quite naive, data are ~8pt/m - I’ll investigate what is in the blocks tomorrow :) )

…so a slight storage improvement, and now it’s indexed as well!

Anyway - that is a distraction. I’d like to hear what experiences people have with properly large point clouds. So far it’s chewing up a lot of memory converting relatively small datasets (~90 million points, without pulse data) to spd, and I’d like to scale up to many hundreds of millions to billions of points.

Also, is there any benefit at present to going through a ground up SPDlib 1.4 installation?

Cheers

Adam

ps - if anyone is going to FOSS4G in a few weeks and is happy to talk about SPDlib and huge datasets, drop me a line off list - it would be great to see what other organisations are up to.

--
Adam Steer
Earth systems data service specialist
National Computational Infrastructure
Leonard Huxley Building, Mills Road
The Australian National University
Acton ACT 2600 Australia

adam.st...@anu.edu.au
+61 2 6125 1413
http://nci.org.au

------------------------------------------------------------------------------
_______________________________________________
spdlib-develop mailing list
spdlib-develop@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/spdlib-develop

------------------------------------------------------------------------------

_______________________________________________
spdlib-develop mailing list
spdlib-develop@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/spdlib-develop

Re: [spdlib-develop] Hi, and a quick question - how many points have you stuffed into SPDLib?

Reply via email to