Le dimanche 01 septembre 2013 02:04:08 Gunnar Liljas a écrit : > You're opening a new session in a loop? That seems quite extreme! I think > we need to get a better picture of what you're doing.
See the project enclosed to my first post.
In my real application, I have to parse few thousand of XML files and store
them into a database for future display.
I have 3 levels on my database :
- the physical system a file represents,
- the file parsed
- the records of the file
Systems → files → records
Each XML file generate around 5.000 rows in database (~3MB of data).
Overall system is around 10.000.000 rows.
This is why I handle each file into a single session and transaction, to avoid
memory problem and long commit at the end.
On each file (thus on each session), I query the database to see if the
corresponding system already exists (and create it if not), then create the
file row, then the thousands of related record rows.
The first query (SELECT on system) to test system existence take more and more
time along the process, from 10ms the first time to more than 10min after few
thousand processed files. The remaining process (file and records creation, so
only INSERT query) take less than 5s.
So this timing problem leads to a more than 1 day overall process on thousands
files, instead 2 or 3hours.
And this problem is strange because this query is always the first one after
the session creation, so in my mind this session is empty.
--
Nicolas Vinot
signature.asc
Description: This is a digitally signed message part.
