regards,
Larry
On 10/5/07, *Stefan Steiniger* <[EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>> wrote:
well done!
i am not sure if this makes sense, but for drawing alone one
could use a
"skip n points" mechanism or use an indexing scheme (e.g.
quadtrees with
skipping the lower(?) levels to show). I guess this would need some
testing to identify the number of points that should be drawn
maximally
and that need to be drawn. But for the selection this is not
possible
(except the drawing). Not sure if modifying "enable check" is a
solution
I guess Martin has some nice thoughts on that ;)
stefan
btw: @Martin: is you FOSS4G - DTM-TIN paper available? (I read he
used
also massive data - millions of heigh points - with postgis)
PS: have you seen this presentation on high res image browsing,
which
comes quite close to our problem, although a it is bit different
domain:
http://www.ted.com/index.php/talks/view/id/129
(sorry don't want to hijack this topic but it is related)
Larry Becker schrieb:
> I'm retesting with Arnd's new XYZ-data which contained a 62MB
shape file
> and a 33MB dbf file.
>
> 1. OJ loaded the file in 15 seconds and finished drawing it in 55
> seconds. 335MB committed memory.
>
> 2. Finished selecting about 2/3 of the points in 75 seconds
Finished
> drawing the selection handles in another two minutes. Up to
452MB of
> committed memory. Right clicked on the view and it took about 2
minutes
> to respond.
>
> Clearly the selection of 600000+ objects is causing havoc in
the UI
> EnableChecks. Also, the selection handle rendering optimization
that I
> made is not effective for points, so it must draw 600000 tiny
yellow
> rectangles each redraw.
>
> So is it about memory? I increased OJ's memory allocation to
750MB and
> reloaded. Same load and redraw stats.
>
> 3. I used the Edit->Select layer items. It took 2 minutes to
display
> the selected items message (1000000) on the status bar. While
selection
> handles were drawing, committed memory cycled between 650 and
700 MB.
> Clearly, frequent pauses in drawing were caused by the JVM
garbage
> collector. I waited for the handles to finish drawing so that I
could
> test the menu response time. Finally, it finished. I clicked
on the
> Edit menu. After a few seconds the workbench window went totally
blank
> for 3 minutes, then came back. I didn't try it again.
>
> Conclusion: I concur with Arnd that OpenJump is unworkable
for point
> files in excess of 500000. The redraw of a million points is
bad,
> redraw of a million selection handles is worse, and the UI is
totally
> unresponsive with large numbers of objects selected.
>
> I believe that we may have reached the design limits of the UI
feedback
> mechanisms. In particular EnableCheck class methods make
multiple
> references to the number of features. The selection feedback
on the
> status bar even computes the number of points (requiring a
million
> iterations for each access).
>
> I repeated the experiment with JUMP 1.2 (750MB memory) which
does not
> have some of the UI enhancements like the status bar display of
number
> of selected items and points. The first thing I noticed is that
JUMP
> renders much slower than OpenJump. No way am I going to wait for
it to
> finish. I did a drag select around the whole view (since
there is no
> select layer items on the Edit menu). It is taking a long
time. I'm
> not timing any of this since the only thing I'm interested in
is the
> menu response time. I found that an easy test to see if the
UI is
> responsive is to mouse over a toolbar icon and see if it
highlights.
> Oops, JUMP just gave an Out of Memory Error. End of that
experiment.
>
> I still believe the main problem is the UI checks. The UI is
totally
> responsive until you make a large selection. Then it gets
progressively
> slower proportional to the size of the selection. Fixing this
is the
> number one priority.
>
> I loaded the file in SkyJUMP which has metrics for screen
redraw. It
> rendered the million points in 50 seconds. There is room for
improvement
> here, but this is not really the main problem. Any optimization
done to
> optimize point drawing would also apply to selection handles.
Fixing
> this is the number two priority.
>
> regards,
> Larry
>
>
>
>
>
>
>
> On 10/4/07, *Larry Becker* <[EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>
> <mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>>
wrote:
>
> OK, for my PC (P4 3.2Ghz with OJ set to use 256MB) it takes 5
> seconds to load your 351k points u1 layer, and 17 seconds to
draw.
> Pretty bad draw performance compared with other geometry
types, but
> not really relevant to the discussion since waiting for the
redraw
> is not necessary.
>
> 1. I managed to copy and paste about 100k of these to a new
layer in
> about 10 seconds. Replicate gives similar results.
>
> 2. Selecting about 1/3 of the points takes about 5 seconds
to do,
> and about 8 more to render the handles, although as I
mentioned
> before, you can proceed as soon as the the status bar
indicates the
> number selected.
>
> 3. Selecting all points takes about 20 seconds, and
another 30 to
> render.
>
> Preliminary results for 350000 points: slow but definitely
workable.
>
> I loaded 3 copies of the u1 layer giving a total of over 1
million
> points. Memory usage is at 200 MB committed. I suspect it
would be
> quite a bit more if Stefan's data had any attributes.
>
> 4. Redraw is now about 45 seconds, but just ignoring it works
for me.
>
> 5. Selecting about 1/3 failed after 90 seconds and
resulted in an
> Out of Memory Error.
>
> After allocating 512MB of memory to OJ, I restarted. 390000
points
> selected after 20 seconds. The selection handles finished
rendering
> 40 seconds later. 388MB in use.
>
> 6. Right clicking on the selection takes about 5 seconds to
bring up
> the menu.
>
> 7. Replicate took about 20 seconds to replicate to a new
layer.
>
> These are my current findings. I'll update the post when
I test
> with Arnd's data. So far I don't see anything too
bad. About what
> you might expect really. The rendering code is not
optimized for
> points, and the selection code is dealing with its worst case
since
> the bounding box for points is a point.
>
> Larry
>
> On 10/4/07, *Stefan Steiniger* < [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>
> <mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>> wrote:
>
> oha..
> that is interesting. Accidentaly i stored the points as
multipoints.
> Now i updated my jump @office and stored the points as
point
> (and wrote
> the file again to the ftp). The file size is now already
a bit
> smaller,
> and also the commited memory...
>
> stefan
>
> Stefan Steiniger wrote:
>
> > Good if Arnd provides the data, but here the link to
my data
> in case
> > it fails:
> > ftp://ftp.geo.uzh.ch/pub/sstein/data/u1_points.zip
> >
> > an option to get one million points is to copy the
350k points and
> > then to move them (if that works ;).
> > Btw. the lascers can data have no attributes (may be
therefore
> smaller
> > than Arnd's; the covered area is relatively small <
200m)
> > (shapefile size is 30mb and zip xyz-ascii was 15mb)
> >
> > stefan
> >
> > Arnd Kielhorn wrote:
> >
> > > Hello Larry,
> > >
> > > first of all, thank You very much!
> > >
> > > I work with delaunay triangulation and building a
3D-Model
> with our
> > > plugin.
> > >
> > > If we both could make a direct connection (via FTP or
a free
> tool
> > > called teamviewer) I could sent You one file with
> laserscanning data
> > > with 1 million points (XYZ). It is about 30 MB in the
PIROL-CSV
> > > format or about 100 MB as shape file.
> > >
> > > Kindly regards
> > > Arnd
> > >
> > > Larry Becker schrieb:
> > >
> > >> I'm going to start investigating the point
performance
> problem as
> > >> soon as I get some test data. I have been trying to
locate
> a large
> > >> point dataset on the web, but no luck. Perhaps
the best
> approach
> > >> might be to add a new item to the Generate menu,
"Random
> Points".
> > >>
> > >> @Arnd: One thing that people often forget about JUMP
is that
> > >> rendering is a background process. You don't have
to wait
> for it
> > >> to finish to start the next operation.
> > >> What kind of terrain maps are you generating?
Are you
> generating
> > >> contour lines from a grid of points? You mentioned
deleting
> > >> selected points being impossibly slow. I don't see
why this
> should
> > >> be so. If I can duplicate the problem, it should be
an easy
> fix.
> > >>
> > >> regards,
> > >> Larry
> > >>
> > >> On 10/2/07, *Larry Becker* < [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>
> <mailto:[EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>>
> > >> <mailto: [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>
> <mailto:[EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>>>> wrote:
> > >>
> > >> HI Arnd,
> > >>
> > >> Would it be helpful to work around the
problem by
> saving the
> > >> points as a shapefile and using a utility like
shp2tile
> to break
> > >> the point data up into multiple files?
> > >>
> > >> see: http://imaptools.com/download-software.html
<http://imaptools.com/download-software.html>
> > >>
> > >> regards,
> > >> Larry
> > >>
> > >>
> > >> On 10/2/07, *Arnd Kielhorn* < [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>
> <mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
> > >> <mailto: [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]> <mailto:[EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>>>>
> wrote:
> > >>
> > >> Hello Paul,
> > >>
> > >> thank You for the tip with the z-value, but
how I
> can use it
> > >> as a
> > >> z-value on the point itself?
> > >>
> > >> The file fomrat is txt, csv (from PIROL
CSV-Format) like
> > >> following example
> > >> $Rechts Hoch heigh
> > >> $grad grad m
> > >> $double double double
> > >> 3434000.00 5800000.00 93.78
> > >> 3434001.00 5800000.00 93.84
> > >> ... ... ...
> > >>
> > >> or the ESRI ascii grid format asc (reading
plugin
> from SIGLE).
> > >> The most files I got have 1 million points
and such
> csv files
> > >> have about
> > >> 30 MB.
> > >> In my start file for OJ I got 768 MB,
because I have
> 1 GB RAM.
> > >>
> > >> Kindly regards
> > >> Arnd
> > >>
> > >>
> > >> Paul Austin schrieb:
> > >> > Arnd,
> > >> >
> > >> > What file format are you using?
> > >> >
> > >> > As JUMP loads the whole file into memory I
would
> recommend
> > >> setting the
> > >> > heap size on the JAVA VM to be -Xmx512M if
you are
> dealing
> > >> with large
> > >> > datasets.
> > >> >
> > >> > You mentioned that the height is an
attribute on
> the feature,
> > >> if you use
> > >> > the height as a z-value on the point
itself there
> will be
> > >> some memory
> > >> > savings.
> > >> >
> > >> > Paul
> > >> >
> > >> > Arnd Kielhorn wrote:
> > >> >
> > >> >> Hello Larry,
> > >> >>
> > >> >> as in the further discussion just named
the main
> problema
> > >> are (for
> > >> >> example I just have to work with point
layers
> with only a
> > >> heigh
> > >> >> attribute but with 1 million points; but
also
> e.g. 300,000
> > >> points are
> > >> >> enough):
> > >> >> - (re)drawing of the points
> > >> >> - strongly restricted or impossible
operation
> (e.g. deleting
> > >> some
> > >> >> points/vertices after marking them; e.g.
copying
> hundreds or
> > >> a few
> > >> >> thousand in a new layer to have layer
with lesser
> points)
> > >> >> Very often OJ hangs on and I have to
restart it.
> > >> >> I use such point layers for creating a
digital
> terrain maps.
> > >> >>
> > >> >> Kindly regards
> > >> >> Arnd
> > >> >>
> > >> >> Larry Becker schrieb:
> > >> >>
> > >> >>> Hi Arnd,
> > >> >>>
> > >> >>> I haven't worked with large point
datasets
> much. What
> > >> it is
> > >> >>> exactly that is slow? Is it redraw
time? Or
> perhaps the
> > >> particular
> > >> >>> operation that you are doing?
> > >> >>>
> > >> >>> I know that point layers do not
benefit from
> the recent
> > >> decimation
> > >> >>> optimization, so they take longer to
draw than
> equivalent
> > >> polygons
> > >> >>> and polyline layers. I think Michaël
tried a point
> > >> decimation
> > >> >>> optimization, but wasn't satisfied with
the results.
> > >> >>>
> > >> >>> regards,
> > >> >>> Larry Becker
> > >> >>>
> > >> >>> On 10/1/07, *Stefan Steiniger* <
> [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
<mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
> > >> <mailto: [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]> <mailto:[EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>>>
> > >> >>> <mailto: [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>
> <mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
<mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
> <mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>>>>
> > >> wrote:
> > >> >>>
> > >> >>> mhm
> > >> >>> from my point of view it is both OJ
and JRE.
> It would
> > >> be probably
> > >> >>> better
> > >> >>> when the points are stored and
processes in
> a database.
> > >> >>>
> > >> >>> stefan
> > >> >>>
> > >> >>> Arnd Kielhorn schrieb:
> > >> >>> > Hello,
> > >> >>> >
> > >> >>> > yes I know, first of all OJ is a
GIS for
> vector data.
> > >> But the
> > >> >>> > functionality is also very good
for point
> datasets.
> > >> But working
> > >> >>> with
> > >> >>> > great point datasets (more 500,000
points)
> bring
> > >> problems with the
> > >> >>> > memory and the garbage collector
works not
> satisfied.
> > >> Every
> > >> >>> command take
> > >> >>> > a lot of time or could not carried
out.
> (CPU P4 2.4
> > >> GHz, 1 GB RAM)
> > >> >>> > Is it only a thing of the JRE or
also from
> OJ?
> > >> >>> > So, it is useful to know if there
is a
> chance to
> > >> increase the
> > >> >>> > performence of OJ in this case and
what
> could be the
> > >> possible
> > >> >>> milestones
> > >> >>> > to reach this aim?
> > >> >>> > I am very eager to read everyones
opion.
> > >> >>> >
> > >> >>> > Kindly regards
> > >> >>> > Arnd
> > >> >>> >
> _______________________________________________
> > >> >>> > jump-users mailing list
> > >> >>> > [email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > >> <mailto: [email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>
> > >> >>> <mailto:
[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > >> <mailto: [email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>>
> > >> >>> >
> > >>
http://lists.refractions.net/mailman/listinfo/jump-users
> > >> <
> http://lists.refractions.net/mailman/listinfo/jump-users>
> > >> >>> >
> > >>
> > >> <http://amusingprogrammer.blogspot.com/
> <http://amusingprogrammer.blogspot.com/
<http://amusingprogrammer.blogspot.com/>>>
> > >>
>
------------------------------------------------------------------------
>
> > >>
> > >>
> > >> _______________________________________________
> > >> jump-users mailing list
> > >> [email protected]
<mailto:[email protected]>
> <mailto: [email protected]
<mailto:[email protected]>>
> > >>
http://lists.refractions.net/mailman/listinfo/jump-users
> > >>
> > >
> > >
> > >
> > > _______________________________________________
> > > jump-users mailing list
> > > [email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > >
http://lists.refractions.net/mailman/listinfo/jump-users
> <http://lists.refractions.net/mailman/listinfo/jump-users
<http://lists.refractions.net/mailman/listinfo/jump-users>>
> > >
> > >
> >
> > _______________________________________________
> > jump-users mailing list
> > [email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> >
http://lists.refractions.net/mailman/listinfo/jump-users
> >
> >
>
> _______________________________________________
> jump-users mailing list
> [email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> http://lists.refractions.net/mailman/listinfo/jump-users
>
>
>
>
> --
> http://amusingprogrammer.blogspot.com/
> <http://amusingprogrammer.blogspot.com/>
>
>
>
>
> --
> http://amusingprogrammer.blogspot.com/
>
>
>
------------------------------------------------------------------------
>
> _______________________________________________
> jump-users mailing list
> [email protected]
<mailto:[email protected]>
> http://lists.refractions.net/mailman/listinfo/jump-users
<http://lists.refractions.net/mailman/listinfo/jump-users>
_______________________________________________
jump-users mailing list
[email protected]
<mailto:[email protected]>
http://lists.refractions.net/mailman/listinfo/jump-users
--
http://amusingprogrammer.blogspot.com/
------------------------------------------------------------------------
_______________________________________________
jump-users mailing list
[email protected]
http://lists.refractions.net/mailman/listinfo/jump-users
