[basex-talk] "Out of Memory" when inserting data from one DB to another

[BIRKNER Michael]

Hi to all,

I get an "Out of Memory" error (using the BaseX GUI on Ubuntu Linux) when I try 
to insert quite a lot of data into a BaseX database. The use case: I have a 
database (size is about 2600 MB, 13718400 nodes) with information in <record> 
elements that should be added to <record> elements in another database. The 
<record>s have a 1 to 1 connection identified by an ID that is available in 
both databases.

An example (simplified) of the DB with the information I want to add to the 
other DB:

<collection>
  <record>
    <id>1</id>
    <data>Some data</data>
    <data>More data</data>
    <data>More data</data>
    ...
  </record>
  <record>
    <id>2</id>
    <data>Some data</data>
    <data>More data</data>
    <data>More data</data>
    ...
  </record>
  <record>
    <id>3</id>
    <data>Some data</data>
    <data>More data</data>
    <data>More data</data>
    ...
  </record>
  ... many many more <record>s
</collection>

Here an example (simplified) of the DB to which the above <data> elements 
should be added:

<collection>
  <record>
    <id>1</id>
    <mainData>Main data</mainData>
    <mainData>More main data</mainData>
    <mainData>More main data</mainData>
    ...
    <!-- Insert <data> elements of record with id 1 of the other database here 
-->
  </record>
  <record>
    <id>2</id>
    <mainData>Main data</mainData>
    <mainData>More main data</mainData>
    <mainData>More main data</mainData>
    ...
    <!-- Insert <data> elements of record with id 2 of the other database here 
-->
  </record>
  <record>
    <id>3</id>
    <mainData>Main data</mainData>
    <mainData>More main data</mainData>
    <mainData>More main data</mainData>
    ...
    <!-- Insert <data> elements of record with id 3 of the other database here 
-->
  </record>
  ... many many more <record>s
</collection>

This is the XQuery I use to insert the given <data> elements from the <record> 
in one database to the corresponding <record> in the other database. It results 
in an "Out of Memory" error:

let $infoRecs := db:open('db-with-data')/collection/record
let $mainRecs := db:open('db-to-insert-data')/collection/record
for $infoRec in $infoRecs
  let $id := data($infoRec/id)
  let $mainRec := $mainRecs[id=$id]
  let $dataToInsert := $infoRec/*[not(name()='id')]
  return insert node ($dataToInsert) into $mainRec

I assume that the error is a result of the large amount of data that is 
processed. My question is if a strategy exists to work with such an amount of 
data without getting an "Out of Memory" error?

Thanks very much to everyone in advance for any hint and advice. If you need 
more information about DB setup or options just let me know.

Best regrads,
Michael


Dieses Mail ist ausschließlich für die Verwendung durch die/den darin genannten 
AdressatInnen bestimmt und kann vertrauliche bzw rechtlich geschützte 
Informationen enthalten, deren Verwendung ohne Genehmigung durch den/ die 
AbsenderIn rechtswidrig sein kann.
Falls Sie dieses Mail irrtümlich erhalten haben, informieren Sie uns bitte und 
löschen Sie die Nachricht.
UID: ATU 16209706 I https://wien.arbeiterkammer.at/datenschutz