Hi,
There are mails about slow parsing of JML in July 2017 and I experienced slow
parsing of dates from Spatialite in February 2019, discussion is also on the
mailing list. It seems there is a need to test your fixes also with JML and
SpatiaLite, not only with shapefiles.
-Jukka-
Lähettäjä: ede <e...@users.sourceforge.net>
Lähetetty: keskiviikko 19. elokuuta 2020 11.26
Vastaanottaja: [jump-pilot:bugs] <4...@bugs.jump-pilot.p.re.sourceforge.net>
Aihe: [jump-pilot:bugs] Re: #497 Shapefile export slowed down because of
FlexibleDateParser
On 19.08.2020 09:25, michael michaud wrote:
Indeed, from my tests, things are getting worst :
well. it shouldn't after all i'm trying to improve things :)
I save my dataset to a shapefile after it has been freshly extracted from a
database (it contains a geometry and a timestamp)
1.15 : 1st save : 55s
2nd save < 1s> r6382 : 1st save : 188s!
2nd save < 1s
let me check. actually r6382 should be faster now. in my tests it is.
what i do is.
load your data as JML (exported earlier), as it is text based and needs to be
parsed. then save it as SHP and compare times and FlexdateParser behaviour.
what's important is if the loader creates FlexibleFeatures or not. and even
then, if then lazy conversion is optional, as the loader may of course set the
attribute as an object of the Date class.
Another thing about your explanation : I remember that you introduced the
LazyConversion of attributes with the json parser. I think the reason was
related to the type system of json (which has only strings, numbers and
boolean).
wrt. GeoJSON that's true. but later on someone noticed that opening JML was
very slow and we found date parsing to be the root cause, so lazy conversion
was introduced.
In the case of shapefile or database where data is precisely typed, I think the
lazy conversion is finally very costly.
that's incorrect ;) brute forcing date formats is what is very_ costly here.
lazy conversion is merely distributing cost from time A to time B. if we do not
need to convert at some place we should remove conversion there.
Data should be stored with the right type at the first place. It would avoid
all these useless conversions of dates. What do you think ? Seems that lazy
conversion made for the json should stay the exception, not the general way to
handle data types.
the concept is sound. there is something else goin on.
here check out the commit
https://sourceforge.net/p/jump-pilot/code/5482/
for testing purposes you could simply replace FlexibleFeature with the
BasicFeature again in your local copy to disable lazy conversion and observe
the difference.
what i can't find right now where JML/GML parses/parsed the date which made it
slow. i'll have a look in the bug description again. i think it was Jukka via
mailing list.
so far.. ede
________________________________
[bugs:#497]<https://sourceforge.net/p/jump-pilot/bugs/497/> Shapefile export
slowed down because of FlexibleDateParser
Status: open
Created: Mon Aug 17, 2020 11:47 AM UTC by michael michaud
Last Updated: Tue Aug 18, 2020 09:42 PM UTC
Owner: michael michaud
Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb)
with version 1.15 lasted more than 20 minutes.
Most of the time is used by FlexibleDateParser.parse().
________________________________
Sent from sourceforge.net because you indicated interest in
https://sourceforge.net/p/jump-pilot/bugs/497/
To unsubscribe from further messages, please visit
https://sourceforge.net/auth/subscriptions/
________________________________
[bugs:#497]<https://sourceforge.net/p/jump-pilot/bugs/497/> Shapefile export
slowed down because of FlexibleDateParser
Status: open
Created: Mon Aug 17, 2020 11:47 AM UTC by michael michaud
Last Updated: Wed Aug 19, 2020 07:39 AM UTC
Owner: michael michaud
Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb)
with version 1.15 lasted more than 20 minutes.
Most of the time is used by FlexibleDateParser.parse().
________________________________
Sent from sourceforge.net because you indicated interest in
https://sourceforge.net/p/jump-pilot/bugs/497/
To unsubscribe from further messages, please visit
https://sourceforge.net/auth/subscriptions/
---
** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**
**Status:** open
**Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
**Last Updated:** Wed Aug 19, 2020 10:29 AM UTC
**Owner:** michael michaud
Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb)
with version 1.15 lasted more than 20 minutes.
Most of the time is used by FlexibleDateParser.parse().
---
Sent from sourceforge.net because jump-pilot-devel@lists.sourceforge.net is
subscribed to https://sourceforge.net/p/jump-pilot/bugs/
To unsubscribe from further messages, a project admin can change settings at
https://sourceforge.net/p/jump-pilot/admin/bugs/options. Or, if this is a
mailing list, you can unsubscribe from the mailing list.
_______________________________________________
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
