Dear basex and xquery experts,
I have to solve the following problem:
On a daily basis I get data packages in csv format (approx.. 20.000 records)
I imported the 1st package to basex-DB and want to import further changed or
new records from the daily deliveries.
I tried several solutions but cannot get a solution which is performant enough
to be executed every day.
While importing complete packages takes a few seconds – comparing against
records in the database requires hours.
So I decided to use the index variant which is described by Christian Grün.
My core code looks as follows:
let $db = db:open(‘mydb’)
let $ndx = db:open(‘mydb-index’)
let $xml := csv:parse($csv, map { "header": true(), "separator": ';' })
let $xml := (
for $x in subsequence($xml//record,2)
count $c
return
if( local:recordUpdated( $x, ‘mydb’, $ndx)) then (
element {'record'} {
attribute {'recno'} {$c},
$x/child::*}
) else ()
)
(: Aufnahme der Differenzen in die DB :)
db:replace( $dbname, $tpalow||"/"||$seqno, $xml)
declare function local:recordUpdated( $rec, $dbname, $ndx as item()*) as
xs:boolean {
let $id := $ndx//vp[@ptn=$rec/_14_TPA-Key_VP/text() and
@tarif=$rec/_28_TPA-Key_Tarif/text()]/id
return
if( $id) then (
let $oldrec := db:open-id($dbname, $id)/..
return not(deep-equal($oldrec/child::*, $rec/child::*))
) else true()
};
Both databases ‘mydb’ and ‘mydb-index’ are created with attribute and text
indexes.
After each run the index is newly created to point to the newest records for
the given key values.
Can anybody point out why this process takes more than a second per each single
record to compare?
Thanks for your time
