Re: [basex-talk] Performance & scaling

2024-07-29 Thread Eliot Kimber
ers as map(*) := map:merge( for $serverMap as map(*)? in $data?* let $pid := $serverMap?pid let $cpuPercent as xs:string := ((if (exists($cpuData?data)) then $cpuData?data($pid)?pcpu else ()), '-1.0')[1] return map{ $serverMap?port : map:put($serverMap, &#

[basex-talk] Performance & scaling

2024-07-29 Thread Csaba Fekete
Hi I have read the Advanced User Guide section in the documentation and I got a lot of useful information but I'm still a little bit stuck. I have large (1 GB+) mostly single document databases and complex queries that often take too long to execute. Apart from optimizing the queries, what are my

Re: [basex-talk] Performance issue with BaseX CLI

2024-04-22 Thread ANDRADE Antonio
ailman.uni-konstanz.de Objet : Re: [basex-talk] Performance issue with BaseX CLI Hi again, I had a quick look into the monitoring code, and I noticed two things: 1. It looks to me (correct me if I’m wrong) as if the code of the project was initially written for Saxon and then ported to Bas

Re: [basex-talk] Performance issue with BaseX CLI

2024-04-22 Thread Christian Grün
>> *BaseX* >> >> >> >> *Start* >> >> *Stop* >> >> *Elapse time* >> >> *Start* >> >> *Stop* >> >> *Elapse time* >> >> Check Monitoring 2022 FRH >> >> 06:16:54 >> >> 06:19:30 &g

Re: [basex-talk] Performance issue with BaseX CLI

2024-04-22 Thread Christian Grün
:31:47 > > 00:06:01 > > 10:05:55 > > 11:39:07 > > 01:33:12 > > > > > > *De :* Liam R. E. Quin > *Envoyé :* samedi 20 avril 2024 05:00 > *À :* ANDRADE Antonio ; > basex-talk@mailman.uni-konstanz.de > *Objet :* Re: [basex-talk] Performance issue wi

Re: [basex-talk] Performance issue with BaseX CLI

2024-04-22 Thread Liam R. E. Quin
On Mon, 2024-04-22 at 08:54 +0200, ANDRADE Antonio wrote: > At this moment, the XML document is not intended to be stored. This > is why it is not loaded into a database before processing. BaseX is designed to operate primarily on documents in the database, which is why i suggest trying that. Oth

Re: [basex-talk] Performance issue with BaseX CLI

2024-04-21 Thread ANDRADE Antonio
an.uni-konstanz.de Objet : Re: [basex-talk] Performance issue with BaseX CLI On Fri, 2024-04-19 at 10:45 +0200, ANDRADE Antonio wrote: Hie, For the purposes of European Water Framework Directive reporting, I compared the performances of the Saxon and BaseX XQuery engines. First, you should consider

Re: [basex-talk] Performance issue with BaseX CLI

2024-04-19 Thread Liam R. E. Quin
On Fri, 2024-04-19 at 10:45 +0200, ANDRADE Antonio wrote: > Hie, >   > For the purposes of European Water Framework Directive reporting, I > compared the performances of the Saxon and BaseX XQuery engines. First, you should consider (as i think Martin said) the Java runtime startup time, typically

Re: [basex-talk] Performance issue with BaseX CLI

2024-04-19 Thread ANDRADE Antonio
. The Python/bash difference for the calling script does not seem to explain the observed performance differences. De : Hans-Juergen Rennau Envoyé : vendredi 19 avril 2024 11:25 À : basex-talk@mailman.uni-konstanz.de; ANDRADE Antonio Objet : Re: [basex-talk] Performance issue with BaseX CLI

Re: [basex-talk] Performance issue with BaseX CLI

2024-04-19 Thread Martin Honnen
Am 19.04.2024 um 10:45 schrieb ANDRADE Antonio: Hie, For the purposes of European Water Framework Directive reporting, I compared the performances of the Saxon and BaseX XQuery engines. I observe a performance gap of a factor of 100 to 200 depending on the use case (see functions test_xquery_m

Re: [basex-talk] Performance issue with BaseX CLI

2024-04-19 Thread Hans-Juergen Rennau
Hi Antonio, my experience is very different - quite comparable performance, except for very specific cases, e.g. massive use of fn:idref(). Furthermore, the performance of BaseX is often so stupendous that an improvement by an order of magnitude (not to mention two) appears to me very difficult

[basex-talk] Performance issue with BaseX CLI

2024-04-19 Thread ANDRADE Antonio
Hie, For the purposes of European Water Framework Directive reporting, I compared the performances of the Saxon and BaseX XQuery engines. I observe a performance gap of a factor of 100 to 200 depending on the use case (see functions test_xquery_monitoring() and test_xquery_multischema_2022() in

Re: [basex-talk] Performance of db:get-value() for large maps

2023-07-03 Thread Eliot Kimber
AM To: Eliot Kimber Cc: basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Performance of db:get-value() for large maps [External Email] Hi Eliot, When stored XQuery values are requested, they are always fully materialized in main-memory. Depending on the size, that may take a while. T

Re: [basex-talk] Performance of db:get-value() for large maps

2023-06-30 Thread Christian Grün
Hi Eliot, When stored XQuery values are requested, they are always fully materialized in main-memory. Depending on the size, that may take a while. The following query can be used to create a map with 1 million entries and store it in a database. It takes around 1200 ms on my machine: let $data

[basex-talk] Performance of db:get-value() for large maps

2023-06-29 Thread Eliot Kimber
I’m working on constructing DITA key spaces for our content. My current implementation builds an XQuery map that contains the key space data as well as the XML data from which the key space was constructed, which can be quite a bit (1-2 megabytes of XML all told—key space construction effectivel

Re: [basex-talk] Performance of ft:search function

2022-04-29 Thread Christian Grün
Exactly: The longer you run a BaseX instance, the faster it gets. That’s particularly noticeable when using the client/server or HTTP architecture. There are various reasons for that: BaseX caches, OS & main-memory caching, JIT optimizations, … Tim Thompson schrieb am Fr., 29. Apr. 2022, 22:40

Re: [basex-talk] Performance of ft:search function

2022-04-29 Thread Tim Thompson
Oh, I see--thanks for the tip; I wasn't aware of the SET RUNS feature, which is really helpful! With 1000 runs, the average execution time is more in line with expectations: 38.96ms for expression #1 and 12.44ms for #2. But I notice that with successive executions, #1 gets faster: 38.96ms, 17.73ms,

Re: [basex-talk] Performance of ft:search function

2022-04-27 Thread Christian Grün
> > 2. Direct lookup against subindex > Time: 3.3ms > Expression: ft:search($index, $text)/../.. > > 3. Lookup against subindex file with reference to large index > Time: 2.9ms > Expression: > let $s := > ft:search($index, $text)/../.. > return db:open-id($db, $s/id)/../.. > > My question is: why

[basex-talk] Performance of ft:search function

2022-04-27 Thread Tim Thompson
Hello, I have a largish (5.4G) file with a full-text index that I am using to reconcile names in a local dataset. I've been experimenting with splitting the file into many smaller index files to improve performance. I group the entries by initial character and create a new index file for each dist

Re: [basex-talk] Performance issue on copy-modify-return when too many items are being replaced

2021-11-12 Thread France Baril
Forwarding to the mailing list in order to share knowledge. On Fri, Nov 12, 2021 at 1:41 PM BaseX Support wrote: > Hi France, > > I’d need to get my hands on your code to tell you exactly where it’s > best used, but I can give you some more details on the XQuery > specification: > > When creatin

Re: [basex-talk] Performance: first and second xQuery execution / local PC and server

2020-11-30 Thread BIRKNER Michael
Hi Christian, thank you very much for your answer! You were totally right: After some tests with lower Xmx value (3700m) I can confirm that the xquery executes much faster on the virtual server (about 15 minutes). I have no control over the VMware config ... I just was provided with the virtu

Re: [basex-talk] Performance: first and second xQuery execution / local PC and server

2020-11-27 Thread Christian Grün
Hi Michael, > I have a question concerning performance ... once again ... sorry ;-) No reason to be sorry ;) Instead I am tempted to say sorry that I may not really give you helpful advice on why your queries are so much slower in your virtual environment. I’d guess that this is not particularly

[basex-talk] Performance: first and second xQuery execution / local PC and server

2020-11-24 Thread BIRKNER Michael
Hello BaseX-Community, I have a question concerning performance ... once again ... sorry ;-) First thing I noticed is that when I run a query that involves quite a lot of data (more information below), the first execution of the query takes much more time as the second execution. In my example,

Re: [basex-talk] Performance loss between version 9.2.4 and 9.3.2 when executing specific xQuery

2020-05-11 Thread BIRKNER Michael
ps://arbeiterkammer.at/100><https://arbeiterkammer.at/100><https://w.ak.at/zukunftsprogramm> Von: Christian Grün Gesendet: Montag, 11. Mai 2020 13:02 An: BIRKNER Michael Cc: basex-talk@mailman.uni-konstanz.de Betreff: Re: [basex-talk] Performance loss

Re: [basex-talk] Performance loss between version 9.2.4 and 9.3.2 when executing specific xQuery

2020-05-11 Thread Christian Grün
Hi Michael, I checked your use case in greater depth, and I found the change in our code that caused the slowdown [1]. A) The nutshell answer : Just use the attached query! B) The extensive technical answer: • In previous versions of BaseX, most paths in FLWOR expressions were »inlined« in the

Re: [basex-talk] Performance loss between version 9.2.4 and 9.3.2 when executing specific xQuery

2020-05-08 Thread Christian Grün
------ > *Von:* Christian Grün > *Gesendet:* Freitag, 8. Mai 2020 14:24 > *An:* BIRKNER Michael > *Cc:* basex-talk@mailman.uni-konstanz.de > *Betreff:* Re: [basex-talk] Performance loss between version 9.2.4 and > 9.3.2 when executing specific xQuery > &

Re: [basex-talk] Performance loss between version 9.2.4 and 9.3.2 when executing specific xQuery

2020-05-08 Thread BIRKNER Michael
Von: Christian Grün Gesendet: Freitag, 8. Mai 2020 14:24 An: BIRKNER Michael Cc: basex-talk@mailman.uni-konstanz.de Betreff: Re: [basex-talk] Performance loss between version 9.2.4 and 9.3.2 when executing specific xQuery And I’m always delighted to be confr

Re: [basex-talk] Performance loss between version 9.2.4 and 9.3.2 when executing specific xQuery

2020-05-08 Thread BIRKNER Michael
__ Von: Christian Grün Gesendet: Freitag, 8. Mai 2020 14:24 An: BIRKNER Michael Cc: basex-talk@mailman.uni-konstanz.de Betreff: Re: [basex-talk] Performance loss between version 9.2.4 and 9.3.2 when executing specific xQuery And I’m always delighted to be confronted with library use case. BaseX

Re: [basex-talk] Performance loss between version 9.2.4 and 9.3.2 when executing specific xQuery

2020-05-08 Thread Christian Grün
eich> > -- > > *Die AK setzt sich seit 100 Jahren für Gerechtigkeit ein. Damals. Heute. > Für immer.* > > *arbeiterkammer.at/100 <https://arbeiterkammer.at/100>** > <https://arbeiterkammer.at/100>* <https://w.ak.at/zuk

Re: [basex-talk] Performance loss between version 9.2.4 and 9.3.2 when executing specific xQuery

2020-05-08 Thread Marco Lettere
And even more curiously, we are also working in this very same time on handling fetches from OAI-PMH sources! :-D M. On 08/05/20 13:37, Imsieke, Gerrit, le-tex wrote: Just saying that I find it sooo interesting to learn at which places and for which purposes BaseX is being employed. Have a nice

Re: [basex-talk] Performance loss between version 9.2.4 and 9.3.2 when executing specific xQuery

2020-05-08 Thread Imsieke, Gerrit, le-tex
Just saying that I find it sooo interesting to learn at which places and for which purposes BaseX is being employed. Have a nice weekend! On 08.05.2020 13:31, BIRKNER Michael wrote: Hi Christian, thank you for your answers. As you can guess the queries I sent in my original email are just si

Re: [basex-talk] Performance loss between version 9.2.4 and 9.3.2 when executing specific xQuery

2020-05-08 Thread BIRKNER Michael
Jahren für Gerechtigkeit ein. Damals. Heute. Für immer. arbeiterkammer.at/100<https://arbeiterkammer.at/100><https://arbeiterkammer.at/100><https://w.ak.at/zukunftsprogramm> ____ Von: Christian Grün Gesendet: Freitag, 8. Mai 2020 12:37 An: BIRKNER Michael Cc: b

Re: [basex-talk] Performance loss between version 9.2.4 and 9.3.2 when executing specific xQuery

2020-05-08 Thread Christian Grün
I tried to reproduce your use case by creating some sample data (with a few millions of entries), but both the query plan and the performance were similar in 9.2.4 and the current 9.3.3 beta version. And I am still trying to understand your example query. Is it correct that the attribute of your e

Re: [basex-talk] Performance loss between version 9.2.4 and 9.3.2 when executing specific xQuery

2020-05-08 Thread Christian Grün
Thanks, Michael, for the valuable observation. It might be that another newly integrated optimization proves to be detrimental to your existing query. I’ll try to find the culprit. Just a minor question: In db2, a single document seems to be stored. Does this mean that only one record is assigned

[basex-talk] Performance loss between version 9.2.4 and 9.3.2 when executing specific xQuery

2020-05-08 Thread BIRKNER Michael
Hi, I am observing a performance loss between BaseX versions 9.2.4 (which I was using so far) and 9.3.2 (to which I updated recently) when executing an xQuery like this: --- (: Open 2 databases and get all s :) let $recsFromDb1 := db:open('db1')/record let $recsFromDb2 := db:open('db2')/record

Re: [basex-talk] Performance (index?) on large intermediate structures

2018-08-29 Thread Christian Grün
Dear Sebastian, In fact XQuery maps are very fast, and they are a great data structure for speeding up repeated lookups if you encounter cases in which this is not done automatically by the optimizer. If you manage to send us a little example query that demonstrates the pattern you are working on

[basex-talk] Performance (index?) on large intermediate structures

2018-08-29 Thread Sebastian Albert
Dear BaseX community I am currently working with BaseX to write a rather long XQuery statement with quite large intermediate structures (XML documents / fragments) bound to variables (in FLWOR expressions), which I query by XPath using attributes etc. further down. Having nested loops, this is bec

Re: [basex-talk] Performance of Query joining two documents

2018-02-07 Thread William Velasquez
Thank you very much! Using your advice about performance I solved not only this, but some other ugly queries here. Cheers, WILLIAM DAVID VELÁSQUEZ CREATIVO DE SOFTWARE Creativos Digitales S.A.S. Calle 30A # 83 - 53 Local 1033 Tel: 322 1730 - 311 709 8421 Medellín, Colombia http://c

Re: [basex-talk] Performance of Query joining two documents

2018-02-05 Thread Christian Grün
Hi William, Your query will be evaluated in appr. 100-200 ms if you do some little rewritings. Here is one variant: declare function local:compare($actual, $prev) { for $c in $actual/portfolio/asset return { $c/* }, { $prev/portfolio/asset[assetid/text() = $c/asseti

Re: [basex-talk] Performance issues and query optimizer

2017-10-20 Thread Christian Grün
Hi France, > 1. (For each //element without @id, add @id) then db:optimise and redirect > to next step > 2. For each /*, process content to XLIFF (uses the new ids so requires > preceding redirect to commit changes). > > Now, is it suggested to work with updindex/autooptimize to true or as shown >

Re: [basex-talk] Performance issues and query optimizer

2017-10-20 Thread France Baril
​The approach will definitely help with optimizing individual queries. The other level I'd like to know about is if content is indeed indexed when I think it should be and which indexing approach is best. We have batch processes in multiple layers and I'm not sure that our approach to them is opti

Re: [basex-talk] Performance issues and query optimizer

2017-10-20 Thread Marco Lettere
Just as an example: On our database with attribtue index the following query db:open("lookup")//entry[@zip = "53040" and @city ="BETTOLLE"] returns the following output in the info view: Compiling: - pre-evaluate db:open("lookup") to document-node() - atomic evaluation of (@*:zip = "53040")

Re: [basex-talk] Performance issues and query optimizer

2017-10-20 Thread France Baril
Can I avoid doing that by waiting on version 9.0 and enforcing rewriting? On Fri, Oct 20, 2017 at 2:35 PM, Marco Lettere wrote: > Hi France, > check out the info window of the GUI in a test-run it will tell. > Anyway I think that if the database name is in a declared variable it > should definte

Re: [basex-talk] Performance issues and query optimizer

2017-10-20 Thread Marco Lettere
Hi France, check out the info window of the GUI in a test-run it will tell. Anyway I think that if the database name is in a declared variable it should defintely be able to grasp the index. If $dbname is passed into another function as a function parameter then it could be that the info for acc

[basex-talk] Performance issues and query optimizer

2017-10-20 Thread France Baril
Hi, We are working hard right now on performance issues. I just read this about an upcoming release: Enforce Rewritings In various cases, existing index structures will not be utilized by the query optimizer. This is usually the case if the name of the database is not a static string (e.g., beca

Re: [basex-talk] Performance analysis in web DBA

2017-08-02 Thread Christian Grün
Hi Erik, The DBA editor is limited to the execution of queries. The query plan can be retrieved via XQuery, though: xquery:parse('1') Best, Christian On Thu, Aug 3, 2017 at 6:57 AM, Erik Peterson wrote: > Is there a way to get the query plan and stats in the web dba...like the > native cli

[basex-talk] Performance analysis in web DBA

2017-08-02 Thread Erik Peterson
Is there a way to get the query plan and stats in the web dba...like the native client has with the info view?

Re: [basex-talk] Performance: associating one with many, or many with one?

2017-03-29 Thread Christian Grün
Dear Jay, > And what I'm wondering is that, in a situation where there's one of > something (a musician), and a bunch of something else they're associated > with (song writting or performance credits), which node should get the > reference? I would go with the first approach (listing the track na

Re: [basex-talk] Performance: associating one with many, or many with one?

2017-03-29 Thread Fabrice ETANCHAUD
...@mailman.uni-konstanz.de] De la part de Jay Straw Envoyé : mercredi 29 mars 2017 03:16 À : basex-talk@mailman.uni-konstanz.de Objet : [basex-talk] Performance: associating one with many, or many with one? Hi List! I'm building an application in base-x that's sort of like a local music wi

[basex-talk] Performance: associating one with many, or many with one?

2017-03-28 Thread Jay Straw
Hi List! I'm building an application in base-x that's sort of like a local music wiki: bands, members, venues, promoters, and albums and songs are some of what I'm representing in XML. And what I'm wondering is that, in a situation where there's one of something (a musician), and a bunch of somet

Re: [basex-talk] Performance and heavy load

2015-08-28 Thread Martín Ferrari
O.MemoryStream(System.Text.Encoding.UTF8.GetBytes(AddMetadata(fileContents, DateTime.Now)));baseXClientSession.Replace("prueba" + tempId.ToString("D8") + ".xml", ms);} } Cheers, Martín.From: ferrari_mar...@hotmail.com To: m...@centrum.cz

Re: [basex-talk] Performance and heavy load

2015-08-19 Thread Martin
; > lock (sessionList){ > sessionList.Add(sessionEntry);}} > else{ > sessionEntry.BaseXSession.Timeout = timeout;} > return sessionEntr

Re: [basex-talk] Performance and heavy load

2015-08-19 Thread Martín Ferrari
}} else{sessionEntry.BaseXSession.Timeout = timeout;} return sessionEntry;} Cheers, Martín. > Date: Wed, 19 Aug 2015 13:41:34 +0200 > To: basex-talk@mailman.uni-konstanz.de > From: m...@centrum.cz > Subject: Re: [b

Re: [basex-talk] Performance and heavy load

2015-08-19 Thread Martin
Hi, I would like to know more about "keep the session opened" as you state it -- I am using Java/Groovy client populating a large database (over half a million resources) and if I keep the session opened, so it could be reused within the thread, after a while it starts to cause problems. The only

Re: [basex-talk] Performance and heavy load

2015-08-12 Thread Martín Ferrari
s! Martín. From: ferrari_mar...@hotmail.com To: christian.gr...@gmail.com CC: basex-talk@mailman.uni-konstanz.de Subject: RE: [basex-talk] Performance and heavy load Date: Thu, 30 Jul 2015 05:53:06 + Well, I've played around a bit more. I've set:AUTOFLUSH=falseTEXTINDEX=falseA

Re: [basex-talk] Performance with joins?

2015-08-04 Thread Christian Grün
> Executes in 257.13 ms. But it shouldn't be necessary to explicitly specify > the text() node, right (or would even be bad practice to do so[1])? The results of both variants should be equivalent indeed (unless a BIB_ID does not have more than one text node). I wouldn't call it bad practice to e

Re: [basex-talk] Performance with joins?

2015-08-04 Thread Tim Thompson
Adding a text() step to the predicate does dramatically reduce the execution time. This query: declare namespace marc="http://www.loc.gov/MARC21/slim";; for $m in collection( "latin_hold_20150730" )/marc:collection/marc:record, $r in collection( "latin_hold_20150730" )/root/row[BIB_ID/text()

Re: [basex-talk] Performance with joins?

2015-08-04 Thread Tim Thompson
I was mistaken; I hadn't enabled indexes on this database. However, after creating attribute and text indexes, the query actually seems to take longer to execute (483622.95 ms on last run), although the query plan itself doesn't seem to have changed: Compiling: - pre-evaluating collection("latin_h

Re: [basex-talk] Performance with joins?

2015-08-04 Thread Christian Grün
Dear Tim, The query plan indicates that no index is applied. Your query may be evaluated faster when rewriting "BIB_ID" to "BIB_ID/text()". I will see if this can automatically be done by the query compiler. Best, Christian On Tue, Aug 4, 2015 at 4:42 AM, Tim Thompson wrote: > Hello, > > I hav

[basex-talk] Performance with joins?

2015-08-03 Thread Tim Thompson
Hello, I have a database containing two resources/documents: they both represent the same set of library catalog records (7728 "records" in each), but they each contain different data that I want to join. The first resource looks like this: http://www.loc.gov/MARC21/slim";>

Re: [basex-talk] Performance and heavy load

2015-07-30 Thread Christian Grün
t; Date: Thu, 30 Jul 2015 00:46:17 + > CC: basex-talk@mailman.uni-konstanz.de > > Subject: Re: [basex-talk] Performance and heavy load > > Hi Christian, > I've dug more into this problem. We've installed BaseX 8.2.3 on our > Linux box. It looks like insertio

Re: [basex-talk] Performance and heavy load

2015-07-29 Thread Martín Ferrari
nks, Martín. From: ferrari_mar...@hotmail.com To: christian.gr...@gmail.com Date: Thu, 30 Jul 2015 00:46:17 + CC: basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Performance and heavy load Hi Christian, I've dug more into this problem. We've installed BaseX 8.

Re: [basex-talk] Performance and heavy load

2015-07-29 Thread Martín Ferrari
Resource(s) added in 123.96 ms. 125.52 ms Thanks! Martín. > From: christian.gr...@gmail.com > Date: Tue, 28 Jul 2015 15:12:48 +0200 > Subject: Re: [basex-talk] Performance and heavy load > To: ferrari_mar...@hotmail.com > CC: basex-talk@mailman.uni-konstanz.de > > Out

Re: [basex-talk] Performance and heavy load

2015-07-28 Thread Martín Ferrari
eX, but it looks it could work. We still have a lot of load, so I'll let you know how it goes when we enable it again in production. Thanks, Martín. > From: christian.gr...@gmail.com > Date: Tue, 28 Jul 2015 15:12:48 +0200 > Subject: Re: [basex-talk] Performance and heavy loa

Re: [basex-talk] Performance and heavy load

2015-07-28 Thread Martín Ferrari
r...@arcor.de; ferrari_mar...@hotmail.com > CC: basex-talk@mailman.uni-konstanz.de > Subject: RE: [basex-talk] Performance and heavy load > Date: Tue, 28 Jul 2015 09:40:19 + > > An another idea : > If you never replace a file, > You may expect better performance setting up a R

Re: [basex-talk] Performance and heavy load

2015-07-28 Thread Christian Grün
Out of interest: Do you use a recent version of BaseX? On Tue, Jul 28, 2015 at 3:34 AM, Martín Ferrari wrote: > Hi guys, > I'm quite new to BaseX. I've read a bit already, but perhaps you can > help so I can investigate further. We are having a performance problem with > our BaseX server. We

Re: [basex-talk] Performance and heavy load

2015-07-28 Thread Fabrice Etanchaud
d'origine- De : basex-talk-boun...@mailman.uni-konstanz.de [mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de Fabrice Etanchaud Envoyé : mardi 28 juillet 2015 11:36 À : Maximilian Gärber; Martín Ferrari Cc : basex-talk@mailman.uni-konstanz.de Objet : Re: [basex-talk] Performanc

Re: [basex-talk] Performance and heavy load

2015-07-28 Thread Fabrice Etanchaud
-talk@mailman.uni-konstanz.de Objet : Re: [basex-talk] Performance and heavy load Hi Martin, how do you spread the log files? All into one db or do you create new dbs? If you keep on adding all files to the same database, the add times will slow down over time. Please keep in mind that you can

Re: [basex-talk] Performance and heavy load

2015-07-28 Thread Maximilian Gärber
Hi Martin, how do you spread the log files? All into one db or do you create new dbs? If you keep on adding all files to the same database, the add times will slow down over time. Please keep in mind that you can query multiple databases at once, so I would rather have more databases. With 8.3 s

[basex-talk] Performance and heavy load

2015-07-27 Thread Martín Ferrari
Hi guys,I'm quite new to BaseX. I've read a bit already, but perhaps you can help so I can investigate further. We are having a performance problem with our BaseX server. We're running it on a VM, and hitting it from around 5 web servers. Under no stress, I get this timing from the log for a

Re: [basex-talk] performance of preceding/following axis

2015-06-18 Thread Schopper, Daniel
promising, or simply put: you made my day :) Best, Daniel -Ursprüngliche Nachricht- Von: Christian Grün [mailto:christian.gr...@gmail.com] Gesendet: Donnerstag, 18. Juni 2015 18:49 An: Schopper, Daniel Cc: basex-talk@mailman.uni-konstanz.de Betreff: Re: [basex-talk] performance of pr

Re: [basex-talk] performance of preceding/following axis

2015-06-18 Thread Christian Grün
Hi Daniel, > //w[@type = "NN"][(subsequence(preceding::w, 1, 3), > subsequence(following::w, 1, 3))/@type = "ADJA"] The preceding axis can be quite costly. You could try to use preceding-sibling and following-sibling instead (if it makes sense in your scenario). Another option could be to replace

Re: [basex-talk] performance of preceding/following axis

2015-06-18 Thread Dirk Kirsten
Hello Daniel, I don't have much time right now, but maybe a few pointers to get you started. I didn't test any of this, so take it with a grain of salt. However, I guess your subsequence solution is not performing optimal, as I would guess that there really is a new sequence created. So for 50.

[basex-talk] performance of preceding/following axis

2015-06-18 Thread Schopper, Daniel
Hi, I'm trying to use BaseX for linguistic queries on a TEI document containing annotated tokens (i.e. tei:w-elements with attributes). I'm specifically interested in distance queries that allow to search for combinations of token features within a given window (e.g. all nouns that have an adjec

Re: [basex-talk] Performance issue related with Casting.

2015-05-08 Thread Christian Grün
Hi Ankit, > Now the problem is that, It is working fine with this, but when I declare > variable $ids as xs:string* sequence. The reason is the existence of the Java function: We cannot always determine statically what types the results will have. As a result, we cannot statically detect that $i

[basex-talk] Performance issue related with Casting.

2015-05-07 Thread ankit kumar
Hi, I am invoking a java method inside My XQuery script, My java methods returns a string array which contains the Id of elements. Now I am taking that list in a xquery variable. It looks like this. let $ids := javacontext:getIds(); // Java Method return /sample_xml/category/product[@id = $ids]

Re: [basex-talk] Performance related query.

2015-03-27 Thread ankit kumar
Thanks Christian, I am able to solve the issue. Thanks, Ankit On 27 March 2015 at 17:10, Christian Grün wrote: > Hi Ankit, > > The query info output indicates that only the first query is rewritten for > index access (→ db:attribute). > > If you always work with the same database instance, the

Re: [basex-talk] Performance related query.

2015-03-27 Thread Christian Grün
Hi Ankit, The query info output indicates that only the first query is rewritten for index access (→ db:attribute). If you always work with the same database instance, the following version of your query should be evaluated faster: declare function lib:getCategory($products) { let $catRefs

Re: [basex-talk] Performance related query.

2015-03-27 Thread ankit kumar
Hi, Optimized query without module import count(db:attribute("large_products", distinct-values(db:open-pre("large_products",0)/products/*[@catid]/@catid))/self::id/parent::p:category[parent::products/parent::document-node()]) optimized query with module import count(let $catRefs_4 := distinc

Re: [basex-talk] Performance related query.

2015-03-27 Thread Christian Grün
Hi Ankit, have you already compared the query info outoput? Best, Christian On Fri, Mar 27, 2015 at 10:35 AM, ankit kumar wrote: > Hi, > > I am getting performance issue while using my own xquery library. I have > written an xquery module which contains a single function which return all > th

[basex-talk] Performance related query.

2015-03-27 Thread ankit kumar
Hi, I am getting performance issue while using my own xquery library. I have written an xquery module which contains a single function which return all the categories belong to a set of products as given below. Also there is one constrain that i cannot pass all the category to the getCategory()

Re: [basex-talk] Performance [former 'of PUL commit']

2015-02-25 Thread Christian Grün
> Thus if I understood correclty: > What you propose here is to not have the UPDINDEX=true but rather re-run an > "create index *; optimize (all)" If you have already checked performance, it's completely fine to use UPDINDEX.

Re: [basex-talk] Performance [former 'of PUL commit']

2015-02-25 Thread Marco Lettere
On 25/02/2015 14:51, Christian Grün wrote: A cold one which is growing (with scheduled batch ops) but indexed and UPDINDEX=true (for rare update operations that still might occur) and a hot one for realtime data which has no index but whose size will be reduced and kept constant. This makes perf

Re: [basex-talk] Performance [former 'of PUL commit']

2015-02-25 Thread Christian Grün
> A cold one which is growing (with scheduled batch ops) but indexed and > UPDINDEX=true (for rare update operations that still might occur) and a hot > one for realtime data which has no index but whose size will be reduced and > kept constant. This makes perfect sense. You could also think about

Re: [basex-talk] Performance [former 'of PUL commit']

2015-02-25 Thread Marco Lettere
Hello all, putting the PARALLEL flag to 1 solved the issue of inconsistency between the query + scheduling time and the actual operation completion time. Now we have it stabilized. Thanks. Peaks are now rarely possible when an update from the insertion client serializes itself with one or more

Re: [basex-talk] Performance of PUL commit

2015-02-24 Thread Christian Grün
Hi Marco, > There are two clients which read and write through RESTXQ to/from 2 > different logical DBs on the same server. > Is it possible that behaving concurrently we cause some sort of race > condition that serializes operations for a long time period? Yes, this is absolutely possible. I gue

Re: [basex-talk] Performance of PUL commit

2015-02-24 Thread Marco Lettere
Hi all, I've added a few logs to [1] turning it into [2]. The objective was to understand where we are leaking arbitrarily large amount of time in update operations. I can confirm that querying the DB and interacting with HTTP is nearly constant. Al the time is wasted between the "Starting updat

Re: [basex-talk] Performance of PUL commit

2015-02-22 Thread Marco Lettere
I'm planning to add some logging here [1] and here [2]. Do you think it will unleash some details? M. [1] https://github.com/BaseXdb/basex/blob/master/basex-api/src/main/java/org/basex/http/restxq/RestXqResponse.java#L55 [2] https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/

Re: [basex-talk] Performance of PUL commit

2015-02-22 Thread Christian Grün
> I'd like just to understand to what relates the huge difference (d) between > "querying the db and scheduling an update to the PUL" which is what the > internal function does and the time logged by the basexhttp server as the > completion time of the whole RestXQ call which is T + d. Hm, difficu

Re: [basex-talk] Performance of PUL commit

2015-02-22 Thread Marco Lettere
On 22/02/2015 19:50, Christian Grün wrote: Hi Marco, db:add or db:replace (BTW does this make a difference from a performance point of view?). It's faster to add documents, as this operation won't check if a document already exists. In future versions of BaseX, however, we will possibly me

Re: [basex-talk] Performance of PUL commit

2015-02-22 Thread Christian Grün
Hi Marco, > db:add or db:replace > (BTW does this make a difference from a performance point of view?). It's faster to add documents, as this operation won't check if a document already exists. In future versions of BaseX, however, we will possibly merge these two functions. > The time T out

Re: [basex-talk] Performance of PUL commit

2015-02-20 Thread Marco Lettere
Just one more hint, if we parallelly to the ongoing insertions post one directly with basexclient, we don't experience this lag ... M. On 20/02/2015 15:53, Marco Lettere wrote: It's me again sorry. ;-) One question that puzzles me in these last hours .. This is the scenario on 2 DBs (s

[basex-talk] Performance of PUL commit

2015-02-20 Thread Marco Lettere
It's me again sorry. ;-) One question that puzzles me in these last hours .. This is the scenario on 2 DBs (same server) with the following sizes: 136746 docs ~1 Gb 194608 docs~0.5 Gb declare %rest:path('{$project}/some/url/') %rest:POST('{$body}') %rest:consumes("application/xml")

Re: [basex-talk] Performance on batch processes with http requests.

2015-02-19 Thread France Baril
Hi, here is an example: A process that aggregates a few 100 topics and transforms the aggregated content to a large HTML file for reviewers to see all content together works fine. Try to do it for 32 languages, and you run out of memory. I'm trying to build a small sample. Our real processes also

Re: [basex-talk] Performance on batch processes with http requests.

2015-02-13 Thread Christian Grün
> We run out of memory. Who/what is responsible for the OOM)? Could you give us some more information on the exact step in the process that causes the bottleneck? > Web controller is jquery, but we redirect from xquery. jquery just says: > translate these files in these languages. The restxq fu

Re: [basex-talk] Performance on batch processes with http requests.

2015-02-11 Thread France Baril
Hi, - Our issue is with performance. - Performing all the transformations lead to 2 issues: - We run out of memory. - We sometimes need to query the content that has been transformed after it has gone through some of the transformations. - Web controller is jquery, but w

Re: [basex-talk] Performance on batch processes with http requests.

2015-02-10 Thread Christian Grün
Hi France, I guess there is no simple answer to your question; it mostly depends on the architecture of your approach what would be the best solution and further steps. And I'm not quite sure what's the major challenge? Is it performance, is it technical restrictions, is it the overall concept? A

[basex-talk] Performance on batch processes with http requests.

2015-02-09 Thread France Baril
Hi, I have an item that I would like to bring to attention. We have developed a web controller to let users manages translation processes for BaseX content. Our process is something like this: - Users select content to translation (1 to 500 small files) + languages to translation to (1 to

Re: [basex-talk] Performance issue due to evaluation during function composition ?

2015-01-09 Thread Christian Grün
Hi Jean-Marc, It's not that trivial to optimize your query, but I'd like to let you know that pre-evaluation of function literals works if function declarations are swapped [1]. Christian [1] https://github.com/BaseXdb/basex/issues/1052 On Fri, Jan 2, 2015 at 10:19 AM, jean-marc Mercier wrote

Re: [basex-talk] Performance issue due to evaluation during function composition ?

2015-01-06 Thread jean-marc Mercier
Hi Christian, Thx a lot. Jean-Marc 2015-01-06 12:20 GMT+01:00 Christian Grün : > Hi Jean-Marc, > > Dirk has already outlined well what this is about. I have added a new > GitHub request [1]. > > Cheers, > Christian > > [1] https://github.com/BaseXdb/basex/issues/1052 > > > > On Fri, Jan 2, 2015

  1   2   >