Re: [basex-talk] Help - Regarding Performance Improvement

[ETANCHAUD Fabrice]

Hi Deepak,
My two cents advice :

I found efficient to :

  *   Use a append only approach : instead of replacing a document, just add it 
to your collections. You will have to find how to get the latest version of a 
given logical document, based on a business key to be identified. For that 
purpose, you can create a kind of buffer (set of) collection(s) containing all 
the incoming documents.
  *   On schedule, build new collections containing only the latest versions 
taken from the additive collections (a new backlog), empty your buffers, and 
restart.

Do you think this could be applied to your use case ?

Best regards from French west coast,
Fabrice.


De : BaseX-Talk  De la part de 
Deepak Dinakara
Envoyé : jeudi 28 décembre 2023 09:39
À : basex-talk@mailman.uni-konstanz.de
Objet : [basex-talk] Help - Regarding Performance Improvement


Ce mail provient d’un expéditeur extérieur à la MAIF. En cas de doute : ne 
répondez pas, ne cliquez pas sur les liens ou pièces jointes et signalez le 
message via les boutons « signaler » et/ou « signaler un hameçonnage ».
Hi,

Reaching out to get suggestions on improving performance.
Using basex to store and analyze around 350,000 to 500,000 XMLs.
Size of each XML varies between a few KBs to 5MB. Each day around 10k XMLs get 
added/patched.
I have the following queries
1) What is the optimal size or number of documents in a DB? Earlier I had 1 DB 
with different collections but inserts were too slow, took more than 30s just 
to replace a document. So split it up by some category to have around 30 DBs. 
Inserts are fine but again if there are too many documents in a category, 
patching that DB slows and querying across all DBs also gets slowed down. Any 
optimal number for DBs? Can I create many DBs like 1 for every 10K XMLs? I read 
through 
https://www.mail-archive.com/basex-talk@mailman.uni-konstanz.de/msg06310.html, 
having 100s of DBs cause query performance degradation? Is there any better 
solution?
2) Query performance has degraded with more documents in a DB. I also noticed 
that with/without token/attribute index, there is not much difference to query 
performance (they are just XML attribute queries). "Optimize" flag after 
inserts to recreate the index takes too much time and memory. I am not running 
it now since I didn't find significant improvement with/without index with my 
tests. Any suggestions for improving this?
3) Is it possible to just run queries against specific XMLs? I will have a 
pre-filter based on user selection and queries need to be run against only 
those XMLs. There are a number of filters users can apply and every time it can 
result in a different set of XMLs against which analysis has to be performed 
(Hence not feasible to create so many collections). Right now, I am querying 
against all XMLs even though I am interested only in a subset of XMLs and doing 
post filtering. I did go through 
https://mailman.uni-konstanz.de/pipermail/basex-talk/2010-July/000495.html, but 
again having a regex to include all the interested file paths(sometimes entire 
set of documents) will slow it down.

Thank you,
Deepak

Re: [basex-talk] Yaml serialization

[ETANCHAUD Fabrice]

Dear all,

I agree with Marco,
Yaml serialization/deserialization would be a big plus.
I am currently using a great datawarehouse transformation tool that makes heavy 
use of yaml for configuration and documentation (dbt),
and as a BaseX's old friend, I naturally use XQuery to process these yaml 
files, but it requires a call to a unix command line tool in order to transform 
to json first.

And thank you for the summer edition !

Best regards from french west coast,
Fabrice


De : BaseX-Talk  de la part de 
Marco Lettere 
Envoyé : mercredi 12 août 2020 12:04
À : BaseX 
Objet : [basex-talk] Yaml serialization

Dear BaseX team,

just out of curiosity, could you imagine what kind of effort it would be
to provide YAML as another serialization method for BaseX in addition to
XML, HTML, JSON, CSV?

The datamodel seems pretty close to the JSON one so maybe a very similar
transformation to XML could be implemented ...

Ciao,

Marco.

[basex-talk] [FULLTEXT] How to search for underscore seperated words ?

[ETANCHAUD Fabrice]

Hi all BaseX users,

When I search for 'YET_ANOTHER_SILLY_KEYWORD', ft:search gives me all text 
nodes containing any of the YET ANOTHER SILLY KEYWORD words.
I tried the different search modes, but could not find a way to get ft:search 
return only the text nodes containing the concatenation.
Does anybody have a solution ?

Best regards,



[MAIF]

Fabrice ETANCHAUD

Unité Etudes Informatiques - MAIF VIE

50 avenue Salvador Allende 79029 Niort Cedex 9

fabrice.etanch...@maif.fr

MAIF.FR | Retrouvez-nous aussi sur [Facebook] 
 [Twitter]  
[Youtube] 

Re: [basex-talk] RESTXQ - java.lang.OutOfMemoryError: Java heap space

[ETANCHAUD Fabrice]

Hi Sebastian,

There is a prof:gc() function, but that should only be used for debugging 
purposes.
Is your Pending Update List applied to the collections after each call (do you 
see your changes in the db inbetween calls) ?
I had memory overflow issues during xquery update functions calls, but always 
when issued in the same script, because of saturated PUL.

Best,
Fabrice


Profiling Module - BaseX 
Documentation
Signatures: prof:current-ms() as xs:integer: Summary: Returns the number of 
milliseconds passed since 1970/01/01 UTC. The granularity of the value depends 
on the underlying operating system and may be larger.
docs.basex.org



De : BaseX-Talk  de la part de 
Sebastian Guerrero 
Envoyé : jeudi 28 mai 2020 20:39
À : BaseX 
Objet : [basex-talk] RESTXQ - java.lang.OutOfMemoryError: Java heap space

Hi BaseX team!

A quick question.

Is there some known bug/common setting missing for RESTXQ and memory problems?

I have this simple module ( into /webapp, a .xqm file ) :

module namespace exe = 'http://site.com/execute';
declare
%updating
%rest:path("update")
%rest:consumes("application/x-www-form-urlencoded")
%rest:POST function exe:update() {
xquery:eval-update(request:parameter("query"))
};

I use it to execute some updates against some databases from different clients.

Everything works fine by a while, but after some time I get this error [1]:

java.lang.OutOfMemoryError: Java heap space

I noticed that every call to update() the memory grows and grows until it 
reaches the OutOfMemoryError. [2]

If I stop the HTTP server, the memory is released immediately. [3]

What I'm doing wrong?
Is there some command to execute the GC?
Is this a problem with "xquery:eval-update()"?
Am I using it in the wrong way?

Best regards,
Sebastian.
[1] https://imgur.com/DrcbwQg
[2] https://imgur.com/fonmrhm
[3] https://imgur.com/SYFBFK8

Re: [basex-talk] Weird: mixed content trimmed unexpectedly

[ETANCHAUD Fabrice]

Hi Omar,

did you try
http://localhost:8984/rest/LeapinLists/test.xml?chop=false
 ?

Best regards,
Fabrice


De : BaseX-Talk  de la part de Omar 
Siam 
Envoyé : lundi 9 décembre 2019 16:51
À : basex-talk@mailman.uni-konstanz.de 
Objet : Re: [basex-talk] Weird: mixed content trimmed unexpectedly


Hello,

I had some expierence of this of my own. Saxon and other XML tools are better 
at guessing what the user wants. Probably there is an option (CHOP?) that one 
can set to tell BaseX not to trim whitespace at the edges of text nodes. But to 
my knowledge the standard way is to add @xml:space="preserve" to the outermost 
element where this is necessary. That's how I do it.

 test for  me 

Best regards

Omar Siam

Am 09.12.2019 um 16:32 schrieb Arjan Loeffen:
Dear BaseX People,

after many (happy!) projects using BaseX I have found that

curl -i -X PUT --basic --user admin ^
   -H "Content-Type: application/xml" -d " test for  me 
" "http://localhost:8984/rest/LeapinLists/test.xml;

stores

test forme

To the database. Notice the spaces in the XML string. What may have caused this 
behavior? Any suggestions on what to do here?

I tried it on Basex 9.3, the latest edition right now.

--
Arjan Loeffen
Armatiek BV & Armatiek Solutions BV
06-12918997

Re: [basex-talk] file pattern when creating DB

[ETANCHAUD Fabrice]

Great !
XQuery 3.1 is such a powerful language,
Thanks to Christian and his team's members !
I have been using BaseX for ten years,
and never saw such helping team and community as BaseX's ones !

Christian, did you think about bringing BaseX to Slack ?

Best regards,
fabrice



De : radim-havli...@post.cz 
Envoyé : lundi 9 décembre 2019 10:49
À : ETANCHAUD Fabrice 
Cc : BaseX 
Objet : Re: [basex-talk] file pattern when creating DB

Thank you very much! That was exactly what I was looking for.
Best regards,

Radim Havlicek

-- Původní e-mail --
Od: ETANCHAUD Fabrice 
Komu: BaseX 
Datum: 9. 12. 2019 10:28:57
Předmět: Re: [basex-talk] file pattern when creating DB
Radim, here is a simple snippet :

db:create('test-db', 'F:\Utilisateurs\14484H\data\test', (), map { 
'createfilter' : '*.*' })

Fabrice


De : radim-havli...@post.cz 
Envoyé : lundi 9 décembre 2019 10:19
À : ETANCHAUD Fabrice 
Cc : basex-talk@mailman.uni-konstanz.de 
Objet : RE: [basex-talk] file pattern when creating DB

Hi,
I've seen the link before but I was unable to compose the particular query to 
create the database with filter *, and still I am.

-- Původní e-mail --
Od: ETANCHAUD Fabrice 
Komu: radim-havli...@post.cz , 
basex-talk@mailman.uni-konstanz.de 
Datum: 9. 12. 2019 10:14:37
Předmět: RE: [basex-talk] file pattern when creating DB
Hi Radim,

you can pass parsing options to db:create [1], and set createfilter to the 
desired glob pattern.

Best regards,
Fabrice


[1] : http://docs.basex.org/wiki/Database_Module#db:create



De : BaseX-Talk  de la part de 
radim-havli...@post.cz 
Envoyé : lundi 9 décembre 2019 09:42
À : basex-talk@mailman.uni-konstanz.de 
Objet : [basex-talk] file pattern when creating DB

Hello,
I would like to ask about filters.

I'd like to call XQuery
db:create("DB", "c:/Temp/defaultDocuments/", "/")
but with filter pattern * becase without that pattern it adds *.xml files only. 
I would like to add all files from that directory and sub-directories. Is it 
possible to put the filter pattern there?

I discovered ADD TO / "C:/Temp/defaultDocuments" does exactly what I want, but 
I would much rather use XQuery like db:create or
db:add("DB", "Temp/defaultDocuments", "/") but the db:add filters the files 
with *.xml.

Is it posssible to add all files with any XQuery?
Thank in advance for your reply. Best regards,

Radim Havlicek

Re: [basex-talk] file pattern when creating DB

[ETANCHAUD Fabrice]

Radim, here is a simple snippet :

db:create('test-db', 'F:\Utilisateurs\14484H\data\test', (), map { 
'createfilter' : '*.*' })

Fabrice


De : radim-havli...@post.cz 
Envoyé : lundi 9 décembre 2019 10:19
À : ETANCHAUD Fabrice 
Cc : basex-talk@mailman.uni-konstanz.de 
Objet : RE: [basex-talk] file pattern when creating DB

Hi,
I've seen the link before but I was unable to compose the particular query to 
create the database with filter *, and still I am.

-- Původní e-mail --
Od: ETANCHAUD Fabrice 
Komu: radim-havli...@post.cz , 
basex-talk@mailman.uni-konstanz.de 
Datum: 9. 12. 2019 10:14:37
Předmět: RE: [basex-talk] file pattern when creating DB
Hi Radim,

you can pass parsing options to db:create [1], and set createfilter to the 
desired glob pattern.

Best regards,
Fabrice


[1] : http://docs.basex.org/wiki/Database_Module#db:create



De : BaseX-Talk  de la part de 
radim-havli...@post.cz 
Envoyé : lundi 9 décembre 2019 09:42
À : basex-talk@mailman.uni-konstanz.de 
Objet : [basex-talk] file pattern when creating DB

Hello,
I would like to ask about filters.

I'd like to call XQuery
db:create("DB", "c:/Temp/defaultDocuments/", "/")
but with filter pattern * becase without that pattern it adds *.xml files only. 
I would like to add all files from that directory and sub-directories. Is it 
possible to put the filter pattern there?

I discovered ADD TO / "C:/Temp/defaultDocuments" does exactly what I want, but 
I would much rather use XQuery like db:create or
db:add("DB", "Temp/defaultDocuments", "/") but the db:add filters the files 
with *.xml.

Is it posssible to add all files with any XQuery?
Thank in advance for your reply. Best regards,

Radim Havlicek

Re: [basex-talk] file pattern when creating DB

[ETANCHAUD Fabrice]

Hi Radim,

you can pass parsing options to db:create [1], and set createfilter to the 
desired glob pattern.

Best regards,
Fabrice


[1] : http://docs.basex.org/wiki/Database_Module#db:create



De : BaseX-Talk  de la part de 
radim-havli...@post.cz 
Envoyé : lundi 9 décembre 2019 09:42
À : basex-talk@mailman.uni-konstanz.de 
Objet : [basex-talk] file pattern when creating DB

Hello,
I would like to ask about filters.

I'd like to call XQuery
db:create("DB", "c:/Temp/defaultDocuments/", "/")
but with filter pattern * becase without that pattern it adds *.xml files only. 
I would like to add all files from that directory and sub-directories. Is it 
possible to put the filter pattern there?

I discovered ADD TO / "C:/Temp/defaultDocuments" does exactly what I want, but 
I would much rather use XQuery like db:create or
db:add("DB", "Temp/defaultDocuments", "/") but the db:add filters the files 
with *.xml.

Is it posssible to add all files with any XQuery?
Thank in advance for your reply. Best regards,

Radim Havlicek

Re: [basex-talk] Reg : ArrayIndexOutOfBoundsException

[ETANCHAUD Fabrice]

Hi Chandrasekhar, BaseX 7.9 is more than five years old,
what prevents you from upgrading to the latest version ?

BaseX is far more stable now, I can't remember the last time I faced this issue.

Best regards,
Fabrice


De : BaseX-Talk  de la part de 
chandra Sekhar 
Envoyé : vendredi 9 août 2019 09:26
À : basex-talk@mailman.uni-konstanz.de 
Objet : [basex-talk] Reg : ArrayIndexOutOfBoundsException

Dear basex team,

We are getting the below error while accessing the collection.

Exception ::org.basex.core.BaseXException: Improper use? Potential bug? Your 
feedback is welcome:
Contact: 
basex-talk@mailman.uni-konstanz.de
Version: BaseX 7.9
Java: Oracle Corporation, 1.7.0_151
OS: Linux, amd64
Stack Trace:
java.lang.ArrayIndexOutOfBoundsException

Collection size : 37 GB

And did INSPECT command on the collection,It is showing success.

Regards,
Chandrasekhar

Re: [basex-talk] command script / batch-processing of xqueries

[Etanchaud Fabrice]

Hello Günter,

You do can run several xquery files in a single command script with a syntax 
like this :


...
...
...
...


IMHO Xquery scripts are like transactions,
You might overflow memory if you put all your updating functions in a single 
huge transaction.
And in can remember db:create being executed after db:add, so your Pending 
update list might be meaningless at the end :

http://docs.basex.org/wiki/XQuery_Update#Pending_Update_List


Best regards,
Fabrice Etanchaud
Horanet Smart Objects

-Message d'origine-
De : Günter Dunz-Wolff [mailto:kle...@mail.dunzwolff.de] 
Envoyé : lundi 18 avril 2016 18:55
À : Etanchaud Fabrice <fabrice.etanch...@horanet.com>
Cc : Christian Grün <christian.gr...@gmail.com>; BaseX 
<basex-talk@mailman.uni-konstanz.de>
Objet : Re: [basex-talk] command script / batch-processing of xqueries

Hello Fabrice,

thanks for the info. Did see the SET, but didn't notice 
http://docs.basex.org/wiki/Options#Create_Options

> You can use several XQUERY or RUN commands,

I'd like to run a lot of existing XQuery-files (where dbs are created and 
updated) with ONE command.

Perhaps I should define a new xq with xi:include the others and RUN this one 
file?

Thanks again,
Günter

> Am 18.04.2016 um 17:16 schrieb Etanchaud Fabrice 
> <fabrice.etanch...@horanet.com>:
> 
> Hello Günter,
> 
> Did you have a look at
> 
> http://docs.basex.org/wiki/Commands#Command_Scripts
> 
> You can use several XQUERY or RUN commands, and the SET command to 
> configure the db or index options.
> 
> Best regards,
> Fabrice Etanchaud
> Horanet Smart Objects
> 
> -Message d'origine-
> De : basex-talk-boun...@mailman.uni-konstanz.de 
> [mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de 
> Günter Dunz-Wolff Envoyé : lundi 18 avril 2016 16:52 À : BaseX 
> <basex-talk@mailman.uni-konstanz.de>; Christian Grün 
> <christian.gr...@gmail.com> Objet : [basex-talk] command script / 
> batch-processing of xqueries
> 
> Hi all, hi Christian,
> 
> I'm running some xqueries to create and update basex dbs over and over again. 
> They all depend on the basic db 'mydata'. Whenever mydata is updated, I have 
> to update the other dbs.
> 
> 1. xq-file: db:create('mydata', path/to/my/xml .
> 2. xq-file: db:create('searchindex', let $data0 := db:open('mydata') .
> 3. xq-file: db:create('indd-xml', let $data0 := db:open('mydata')
> 4. and so on
> 
> First question:
> Is there any way to put the whole process into one command-script? Or is 
> there any kind of other batch-processing?
> 
> Second question:
> So far I'm creating the db 'mydata' with the GUI. I want to integrate the 
> creating into the batch-processing via XQuery. In generell I know the 
> create-command, but where to put the options: use XInclude, chop-whitespace 
> OFF and Fulltext Language German internal, Stemming OFF, Case Sensitive and 
> Diacritics ON? Can't find it in the docs.
> 
> Thanks for any help,
> 
> Günter
> 
> 
> 
> 
> 
> 
> 

Re: [basex-talk] command script / batch-processing of xqueries

[Etanchaud Fabrice]

Hello Günter,

Did you have a look at 

http://docs.basex.org/wiki/Commands#Command_Scripts

You can use several XQUERY or RUN commands,
and the SET command to configure the db or index options.

Best regards,
Fabrice Etanchaud
Horanet Smart Objects

-Message d'origine-
De : basex-talk-boun...@mailman.uni-konstanz.de 
[mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de Günter 
Dunz-Wolff
Envoyé : lundi 18 avril 2016 16:52
À : BaseX ; Christian Grün 

Objet : [basex-talk] command script / batch-processing of xqueries

Hi all, hi Christian,

I'm running some xqueries to create and update basex dbs over and over again. 
They all depend on the basic db 'mydata'. Whenever mydata is updated, I have to 
update the other dbs.

1. xq-file: db:create('mydata', path/to/my/xml .
2. xq-file: db:create('searchindex', let $data0 := db:open('mydata') .
3. xq-file: db:create('indd-xml', let $data0 := db:open('mydata')
4. and so on

First question:
Is there any way to put the whole process into one command-script? Or is there 
any kind of other batch-processing?

Second question:
So far I'm creating the db 'mydata' with the GUI. I want to integrate the 
creating into the batch-processing via XQuery. In generell I know the 
create-command, but where to put the options: use XInclude, chop-whitespace OFF 
and Fulltext Language German internal, Stemming OFF, Case Sensitive and 
Diacritics ON? Can't find it in the docs.

Thanks for any help,

Günter

Re: [basex-talk] Expected behaviour with full disk

[Etanchaud Fabrice]

Hello James,

Did you try to set ADDCACHE ?
This way you might run out of disk space during disk caching, before database 
corruption occurs.
Be careful, you may have to serialize the XML in order to use this option with 
db:add/db:replace (I read you are in the REST-XQ layer).

Otherwise, the only way to prevent database corruption might be to check disk 
size regularly and reject new data by program…

http://docs.basex.org/wiki/Options#ADDCACHE
http://docs.basex.org/wiki/Database_Module#db:add


Best regards,
Fabrice Etanchaud
Horanet Smart Objects
FR-85200 Fontenay le Comte

De : basex-talk-boun...@mailman.uni-konstanz.de 
[mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de James Ball
Envoyé : vendredi 1 avril 2016 16:48
À : BaseX 
Objet : [basex-talk] Expected behaviour with full disk

Hello,

I’m having some problems with BaseX in situations where there it runs out of 
disk space.

I’m delivering XML files to BaseX 8.4 using RestXQ.

A request is made and the XML is added to a database depending on some 
criteria. I receive many hundreds of thousands of requests and everything works 
well unless I run out of disk space.
J
What I would expect to happen is that BaseX would try to add the document, find 
it couldn’t due to the disk being full and return a server error (e.g. 500) and 
the transaction would be aborted leaving the database as it was before the 
request. Any further requests would all get errors returned.

What I’m seeing is that when a request is made that causes the disk to become 
full BaseX gets part way through adding the document, fails and returns an 
error but also leaves the database corrupted. Usually the server has to be 
restarted and then any database that has been affected has to be deleted - it’s 
rare to be able to save anything from it.

The errors vary but it is usually a NullPointer exception and then 
java.nio.channels.OverlappingFileLockException. for further requests.

I’ve tried turning off logging and setting parallel to 1 to see if that made 
any difference but it doesn’t.

Is what I’m seeing unique to my setup or do others see this?

Is there anything I can do to make BaseX just stop accepting requests when disk 
space runs out to protect my databases?

Many thanks, James

Re: [basex-talk] Binding external variable in GUI

[Etanchaud Fabrice]

Hi Buddy,

Did you try the $x button between the run query and run tests ones ?
Best regards,
Fabrice



De : basex-talk-boun...@mailman.uni-konstanz.de 
[mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de 
buddyonweb-softw...@yahoo.com
Envoyé : lundi 15 février 2016 16:53
À : BaseX 
Objet : [basex-talk] Binding external variable in GUI

At the risk of a) sounding like a real newbie  b) being accused of not Googling 
for an answer (which I did - but apparently overlooked)
I am wondering how I can bind an external variable using just XQuery syntax in 
the BaseX GUI.

I have found how to set the variable via command line options, and how to do it 
in Java, but I merely
wanted to run a quick interactive test in the GUI  to test an idea of mine 
(before writing a bunch of Java code).
What I have done is in a tab in the GUI, I have included the module, and before 
running a function, want to set that external variable.
Surprisingly I found myself at a loss on how to do what appears to be a simple 
task.

So, thanks in advance for any helpand for the rest of this community for 
not laughing at me too loudly

Buddy

Re: [basex-talk] Streaming file loads

[Etanchaud Fabrice]

Hello Christophe,

Are you using java ?
Maybe you should have a look at  the add() method in :

https://github.com/BaseXdb/basex/blob/master/basex-examples/src/main/java/org/basex/examples/api/BaseXClient.java

I can also remember that command line tools can read standard input stream with 
the '-' argument.

Cordialement,

Fabrice Etanchaud
Horanet Smart Objects


-Message d'origine-
De : basex-talk-boun...@mailman.uni-konstanz.de 
[mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de Christophe 
Marchand
Envoyé : mardi 19 janvier 2016 12:20
À : basex-talk@mailman.uni-konstanz.de
Objet : [basex-talk] Streaming file loads

Hello !

We have to load many files (>20.000), large size (>30Gb) into BaseX. 
Today, that's done vis a BXS script which accesses files thru a network share.

I would like to optimize network transfert ; tar / compression at source, 
uncompress / untar at destination. But I would like to avoid to create 
temporary files (at source, on NAS server, and on destination server, BaseX 
host).

So, is there an API which I could use to load a "file" thru a stream ? I think 
I can write an inputstream that redirects data to outpustreams, cutting input 
data in many output streams, one per file. But may I chain this outputstream to 
an inputstream usable by BaseX loader ?

Best regards,
Christophe

Re: [basex-talk] Question about baseX add "/path/to/filesystem/directory"

[Etanchaud Fabrice]

Hi JMA,

Did you try setting SKIPCORRUPT to true ?

Here is an example :

Command:
SET SKIPCORRUPT true
Command:
CREATE DB test C:/Data/adresses/test
Result:
Skipped:
- C:/Data/adresses/test/communes (2).xml
Database 'test' created in 1806.47 ms.

So you will see the list of skipped files in the output stream.

May your sub directories contain archives ?
That could explain these invisible files.

I never had problem with the ADD command even on huge directory structures.

Bonjour de Vendée
Cordialement,
Fabrice


-Message d'origine-
De : basex-talk-boun...@mailman.uni-konstanz.de 
[mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de 
ansel...@free.fr
Envoyé : vendredi 11 décembre 2015 06:19
À : basex-talk@mailman.uni-konstanz.de
Objet : [basex-talk] Question about baseX add "/path/to/filesystem/directory"

Hello

I'm experiencing a trouble with the  add "/path/to/filesystem/directory" on a 
large directory structure with thousands of files. 
The result is a strange error : 

> add "/data/PARTAGE/DOSSIERS EC/PC"
"Configurations2/accelerator/current.xml" (Line 1): No text allowed before root 
element.


And in the directory /data/PARTAGE/DOSSIERS EC/PC, no Configurations2 
directory...
I think a file is having a problem but with a thousand of files, how to know 
which of them ? 

On another directory, the add function works but finds xml files that aren't in 
the directory structure (find . -name "xxx" doesn't find them as basex finds 
!!!)...

Is the "add function" fragile on a directory ?

Many thanks in advance

JMA 

Re: [basex-talk] Returning text matches

[Etanchaud Fabrice]

Dear Liam,

I am afraid that full text index will not find "blue socks", because it does 
not cross text() node boundaries:

http://docs.basex.org/wiki/Full-Text#Mixed_Content

Best regards,
Fabrice


-Message d'origine-
De : basex-talk-boun...@mailman.uni-konstanz.de 
[mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de Liam R. E. 
Quin
Envoyé : jeudi 10 décembre 2015 21:37
À : Christian Grün ; Ron Katriel 
Cc : basex-talk@mailman.uni-konstanz.de
Objet : Re: [basex-talk] Returning text matches

On Thu, 2015-12-10 at 17:24 +0100, Christian Grün wrote:
> Hi Ron,
> 
> You can use ft:mark and ft:extract to highlights matches in a 
> full-text result [1].

And what happens if a full text match crosses an element boundary, e.g.
a search for "blue socks" matching,
He wore dark blue socks that day. could not return, He wore 
dark blue socks that day.

(Yes, I should test it, sorry! but the docs should probably mention it.
it was a big part of the XPath/XQuery Full Text design early on)

Liam

--
Liam R. E. Quin 
The World Wide Web Consortium (W3C)

Re: [basex-talk] AUTOOPTIMIZE - was true now false

[Etanchaud Fabrice]

Hello James,

Did you set AUTOOPTIMIZE to true BEFORE database creation ?
How did you set the option ? in the db:create call, or as a SET/CREATE DB pair ?

I don't have this problem with 8.3.1

Best regards,
Fabrice


De : basex-talk-boun...@mailman.uni-konstanz.de 
[mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de James Ball
Envoyé : lundi 30 novembre 2015 16:57
À : basex-talk@mailman.uni-konstanz.de
Objet : [basex-talk] AUTOOPTIMIZE - was true now false

Hello,

I'm trying out AUTOOPTIMIZE for the first time and I'm not getting what I 
expect.

Running: BaseX 8.3 beta ec9b627

A new database is created with a single document which has TEXTINDEX, 
ATTRINDEX, UPDINDEX and AUTOOPTIMISE all set to true.

This is what INFO DATABASE shows (as I expect):

Indexes
 Up-to-date: true
 TEXTINDEX: true
 ATTRINDEX: true
 FTINDEX: false
 TEXTINCLUDE:
 ATTRINCLUDE:
 FTINCLUDE:
 LANGUAGE: English
 STEMMING: false
 CASESENS: false
 DIACRITICS: false
 STOPWORDS:
 UPDINDEX: true
 AUTOOPTIMIZE: true
 MAXCATS: 100
 MAXLEN: 96

Now I add a new document (or more than one) to the database. I expect that 
Up-to-date will still be true and that AUTOOPTIMIZE will still be true. But 
what I see is both are now false:

Indexes
 Up-to-date: false
 TEXTINDEX: true
 ATTRINDEX: true
 FTINDEX: false
 TEXTINCLUDE:
 ATTRINCLUDE:
 FTINCLUDE:
 LANGUAGE: English
 STEMMING: false
 CASESENS: false
 DIACRITICS: false
 STOPWORDS:
 UPDINDEX: true
 AUTOOPTIMIZE: false
 MAXCATS: 100
 MAXLEN: 96

Perhaps this is the expected behaviour and I've misunderstood AUTOOPTIMIZE? Or 
is there a bug?

Many thanks, James

[basex-talk] RESTXQ : existing annotation for response headers ?

[Etanchaud Fabrice]

Dear all,

When I want to set response headers (for example to force a "save as..."),
I currently have to write a specific response before the response body :

  

  
  

  

Is there an annotation to set header values directly in the function 
declaration ?

Best regards,
Fabrice

Re: [basex-talk] RESTXQ : existing annotation for response headers ?

[Etanchaud Fabrice]

Hello Dirk and Christian,

Thank you for your - as usual ! - fast reply.
web:response-header is exactly the function I was about to reinvent...

and you are right Christian, I should have realized that annotation would not 
help,
since my header value is dynamically computed...

Have a nice week !

Regards,
Fabrice


-Message d'origine-
De : Christian Grün [mailto:christian.gr...@gmail.com] 
Envoyé : lundi 30 novembre 2015 15:15
À : Etanchaud Fabrice <fabrice.etanch...@horanet.com>
Cc : basex-talk@mailman.uni-konstanz.de
Objet : Re: [basex-talk] RESTXQ : existing annotation for response headers ?

Hi Fabrice,

There is currently no way to specify these headers via annotations. It may 
indeed be tricky, because there are many cases in which the headers are 
dynamically generated (such as in your example:
concat(..)).

Did you try the web:response-header function [1]?

Cheers,
Christian

[1] http://docs.basex.org/wiki/Web_Module#web:response-header


On Mon, Nov 30, 2015 at 3:07 PM, Etanchaud Fabrice 
<fabrice.etanch...@horanet.com> wrote:
> Dear all,
>
>
>
> When I want to set response headers (for example to force a “save 
> as…”),
>
> I currently have to write a specific response before the response body :
>
>
>
>   
>
> 
>
>value="text/comma-separated-values;
> charset=utf-8"/>
>
>   
>
> 
>
>   
>
>
>
> Is there an annotation to set header values directly in the function 
> declaration ?
>
>
>
> Best regards,
>
> Fabrice

Re: [basex-talk] db:create, namespaces not stripped when input is variable?

[Etanchaud Fabrice]

Hello Constantine,

IMHO parsing options only apply when there is actually a parsing phase.
I remember having the same problem with whitespace chopping.
Would you get expected result by serializing the xml before giving it to the 
db:create function ?

Best regards,
Fabrice


De : basex-talk-boun...@mailman.uni-konstanz.de 
[mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de Hondros, 
Constantine (ELS-AMS)
Envoyé : jeudi 26 novembre 2015 18:49
À : 'basex-talk@mailman.uni-konstanz.de' 
Objet : [basex-talk] db:create, namespaces not stripped when input is variable?

Hello all,
Any particular reason behind this observed behaviour? (Basex 8.3).

..
[1]
..
declare variable $test :=
http://www.elsevier.com/xml/ani/common; 
xmlns="http://www.elsevier.com/xml/ani/ani;>
  M.S.
;

db:create('test', $test, 'blah', map {'stripns':true()})

..
[2]
..

db:open('test')/*

..
[3]
..

http://www.elsevier.com/xml/ani/ani; 
xmlns:ce="http://www.elsevier.com/xml/ani/common;>
  M.S.




I would have expected the namespaces to be shredded ...

Cheers,
Constantine





Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The 
Netherlands, Registration No. 33156677, Registered in The Netherlands.

Re: [basex-talk] Applicative Cache

[Etanchaud Fabrice]

I am afraid that this would not work.
If you do not create the mainmem database at server's initialization,
but afterwards from a client,
then the database lives as long as the client's connection.
So using the client module in a RESTXQ request will not help, because 
connection is closed at the end of the request.

Currently, I use a basexclient for that, and keep the client console open, to 
prevent the mainmem db from closing.

Best regards,
Fabrice

-Message d'origine-
De : Christian Grün [mailto:christian.gr...@gmail.com] 
Envoyé : vendredi 20 novembre 2015 15:28
À : Marco Lettere <marco.lett...@dedalus.eu>
Cc : Etanchaud Fabrice <fabrice.etanch...@horanet.com>; 
basex-talk@mailman.uni-konstanz.de
Objet : Re: [basex-talk] Applicative Cache

> What about running from within a RESTXQ a 
> client:connect/client:execute against the coupled db server that has 
> been configured to startup like suggested by Fabrice (-c"SET MAINMEM 
> TRUE;CREATE DB mainmem")?

…Marco’s approach ;) I agree, this would be possible! Looking forward to your 
experiences.

Christian

Re: [basex-talk] sending variables to xquery module via client api (python)

[Etanchaud Fabrice]

Hi Bryan,

Could http://docs.basex.org/wiki/Options#BINDINGS give you a solution ?

Best regards,
Fabrice

De : basex-talk-boun...@mailman.uni-konstanz.de 
[mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de Cholfin, Bryan
Envoyé : lundi 16 novembre 2015 23:08
À : basex-talk@mailman.uni-konstanz.de
Objet : [basex-talk] sending variables to xquery module via client api (python)

Hi-

Hoping someone can help me with this. I'm trying to write a python script that 
calls an xquery module in the database. It seems to be pretty easy to use a 
line such as:

session.execute("run gettext.xqm")

and get back the response I want. What I want to do is pass a bound variable to 
the module. The client api has a provision to bind a variable to a query 
object, but I'm not making a query object. I'm simply executing the stored 
query.  I don't find anything in the documentation to suggest that a variable 
can be added to the run command this way.

Bryan Cholfin

Associate Editor for Digital Projects and Metadata
Harvard University Press
617.496.9116
79 Garden Street | Cambridge, MA 02138-1400
www.hup.harvard.edu

Re: [basex-talk] Applicative Cache

[Etanchaud Fabrice]

Hi Marco,

I did a few tests in that configuration last year (not with 8.X version),
And it worked, as long as I did not "OPEN" another database.
But I tested it again with 8.3, and did not succeed in sharing a server side 
mainmem db.

Server :

java -cp "%CP%" %BASEX_JVM% org.basex.BaseXServer %* -c"SET MAINMEM TRUE;CREATE 
DB mainmem"

Client :

> xquery db:open('mainmem')
Stopped at ., 1/9:
[bxerr:BXDB0002] Database 'mainmem' was not found.

Best regards,
Fabrice

-Message d'origine-
De : basex-talk-boun...@mailman.uni-konstanz.de 
[mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de Marco Lettere
Envoyé : lundi 9 novembre 2015 16:29
À : BaseX 
Objet : [basex-talk] Applicative Cache

Hi all,
I went though the emails talking about mainmemory databases and related but 
I've not been able to find an answer to this loud thinking ...
Would it be possible to create an in memory database with a RESTXQ call and 
keep it accessible for other RESTXQ invocations as if it was a disk db 
throughout the lifecycle of a basexhttp server?
Application level cacheing of (possibly distributed) dbs or management of 
transient volatile data items like tokens are the scenarios I have in mind.
Just wondering whether this makes any sense. Whether it's feasible or whether 
someone already did this with other means.
Thanks a lot!
Marco.

Re: [basex-talk] Best way to batch up a set of DB creates, updates and queries?

[Etanchaud Fabrice]

Hello Constantine,

Facing the same problem, as commands can be written in xml,
I found the following solution useful (as usual thank you Christian), to inject 
configuration where needed :

Declare variable $my-option as xs:string external;
Declare variable $my-value as xs:string external;


{$my-value}
...


Best regards,

Fabrice
Horanet Smart Objects
F-85200 Fontenay-le-Comte


De : basex-talk-boun...@mailman.uni-konstanz.de 
[mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de Hondros, 
Constantine (ELS-AMS)
Envoyé : vendredi 30 octobre 2015 10:34
À : 'basex-talk@mailman.uni-konstanz.de' 
Objet : Re: [basex-talk] Best way to batch up a set of DB creates, updates and 
queries?

Please let me also clarify, I'm looking for a solution that can be run from the 
GUI - so I'm not looking for suggestions using shell scripts or Ant which would 
be easy to implement.

Thanks,
C.

From: 
basex-talk-boun...@mailman.uni-konstanz.de
 [mailto:basex-talk-boun...@mailman.uni-konstanz.de] On Behalf Of Hondros, 
Constantine (ELS-AMS)
Sent: 29 October 2015 18:12
To: 'basex-talk@mailman.uni-konstanz.de'
Subject: [basex-talk] Best way to batch up a set of DB creates, updates and 
queries?

I'm looking for the best way to batch up a series of updating operations into a 
single pipeline that can be run at once.

To give you an idea, we have to create multiple databases, perform a number of 
updating post-processes on each, before finally pulling certain fields out.

I've got a lot of parameters that need to externalised somehow - database 
creation options, location of DTDs, XSDs, Catalogs, and unless I am trying this 
the wrong way I am finding it quite difficult.

I can't write it all XQuery - I have a mixture of updating and non-updating 
functions.

I can't get much traction out of the BXS scripting syntax - for example, I 
can't seem to create and use variables using this.

What do other people out there use to glue these sorts of operations together?

TIA,
Constantine



Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The 
Netherlands, Registration No. 33156677, Registered in The Netherlands.



Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The 
Netherlands, Registration No. 33156677, Registered in The Netherlands.

Re: [basex-talk] Best way to batch up a set of DB creates, updates and queries?

[Etanchaud Fabrice]

My fault,
I missed the final step :
You have to execute the resulting xml.

declare variable $my-option as xs:string external;
declare variable $my-value as xs:string external;

file:write('myScript.bxs',


{$my-value}
...
)

And then, in the GUI, load the bxs file and run it.

If there is a server running, you could try to client:execute() the resulting 
xml.
I did not find a xquery function that allow you to run a batch.

Best regards,
Fabrice

De : Hondros, Constantine (ELS-AMS) [mailto:c.hond...@elsevier.com]
Envoyé : vendredi 30 octobre 2015 14:07
À : Etanchaud Fabrice <fabrice.etanch...@horanet.com>; 
'basex-talk@mailman.uni-konstanz.de' <basex-talk@mailman.uni-konstanz.de>
Objet : RE: [basex-talk] Best way to batch up a set of DB creates, updates and 
queries?

Hello Fabrice,

Thanks, but I'm struggling to understand your example - you've got a mixture of 
xquery and Basex script syntax. Can you give me a slightly larger snippet, as I 
don't see how this can work.

Thanks!

From: Etanchaud Fabrice [mailto:fabrice.etanch...@horanet.com]
Sent: 30 October 2015 12:38
To: Hondros, Constantine (ELS-AMS); 'basex-talk@mailman.uni-konstanz.de'
Subject: RE: [basex-talk] Best way to batch up a set of DB creates, updates and 
queries?

Hello Constantine,

Facing the same problem, as commands can be written in xml,
I found the following solution useful (as usual thank you Christian), to inject 
configuration where needed :

Declare variable $my-option as xs:string external;
Declare variable $my-value as xs:string external;


{$my-value}
...


Best regards,

Fabrice
Horanet Smart Objects
F-85200 Fontenay-le-Comte


De : 
basex-talk-boun...@mailman.uni-konstanz.de<mailto:basex-talk-boun...@mailman.uni-konstanz.de>
 [mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de Hondros, 
Constantine (ELS-AMS)
Envoyé : vendredi 30 octobre 2015 10:34
À : 'basex-talk@mailman.uni-konstanz.de' 
<basex-talk@mailman.uni-konstanz.de<mailto:basex-talk@mailman.uni-konstanz.de>>
Objet : Re: [basex-talk] Best way to batch up a set of DB creates, updates and 
queries?

Please let me also clarify, I'm looking for a solution that can be run from the 
GUI - so I'm not looking for suggestions using shell scripts or Ant which would 
be easy to implement.

Thanks,
C.

From: 
basex-talk-boun...@mailman.uni-konstanz.de<mailto:basex-talk-boun...@mailman.uni-konstanz.de>
 [mailto:basex-talk-boun...@mailman.uni-konstanz.de] On Behalf Of Hondros, 
Constantine (ELS-AMS)
Sent: 29 October 2015 18:12
To: 'basex-talk@mailman.uni-konstanz.de'
Subject: [basex-talk] Best way to batch up a set of DB creates, updates and 
queries?

I'm looking for the best way to batch up a series of updating operations into a 
single pipeline that can be run at once.

To give you an idea, we have to create multiple databases, perform a number of 
updating post-processes on each, before finally pulling certain fields out.

I've got a lot of parameters that need to externalised somehow - database 
creation options, location of DTDs, XSDs, Catalogs, and unless I am trying this 
the wrong way I am finding it quite difficult.

I can't write it all XQuery - I have a mixture of updating and non-updating 
functions.

I can't get much traction out of the BXS scripting syntax - for example, I 
can't seem to create and use variables using this.

What do other people out there use to glue these sorts of operations together?

TIA,
Constantine



Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The 
Netherlands, Registration No. 33156677, Registered in The Netherlands.



Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The 
Netherlands, Registration No. 33156677, Registered in The Netherlands.



Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The 
Netherlands, Registration No. 33156677, Registered in The Netherlands.