Re: [basex-talk] Yaml serialization

2020-08-12 Thread ETANCHAUD Fabrice
Dear all,

I agree with Marco,
Yaml serialization/deserialization would be a big plus.
I am currently using a great datawarehouse transformation tool that makes heavy 
use of yaml for configuration and documentation (dbt),
and as a BaseX's old friend, I naturally use XQuery to process these yaml 
files, but it requires a call to a unix command line tool in order to transform 
to json first.

And thank you for the summer edition !

Best regards from french west coast,
Fabrice


De : BaseX-Talk  de la part de 
Marco Lettere 
Envoyé : mercredi 12 août 2020 12:04
À : BaseX 
Objet : [basex-talk] Yaml serialization

Dear BaseX team,

just out of curiosity, could you imagine what kind of effort it would be
to provide YAML as another serialization method for BaseX in addition to
XML, HTML, JSON, CSV?

The datamodel seems pretty close to the JSON one so maybe a very similar
transformation to XML could be implemented ...

Ciao,

Marco.



[basex-talk] [FULLTEXT] How to search for underscore seperated words ?

2020-06-05 Thread ETANCHAUD Fabrice
Hi all BaseX users,

When I search for 'YET_ANOTHER_SILLY_KEYWORD', ft:search gives me all text 
nodes containing any of the YET ANOTHER SILLY KEYWORD words.
I tried the different search modes, but could not find a way to get ft:search 
return only the text nodes containing the concatenation.
Does anybody have a solution ?

Best regards,



[MAIF]

Fabrice ETANCHAUD

Unité Etudes Informatiques - MAIF VIE

50 avenue Salvador Allende 79029 Niort Cedex 9

fabrice.etanch...@maif.fr

MAIF.FR | Retrouvez-nous aussi sur [Facebook] 
 [Twitter]  
[Youtube] 




Re: [basex-talk] RESTXQ - java.lang.OutOfMemoryError: Java heap space

2020-05-29 Thread ETANCHAUD Fabrice
Hi Sebastian,

There is a prof:gc() function, but that should only be used for debugging 
purposes.
Is your Pending Update List applied to the collections after each call (do you 
see your changes in the db inbetween calls) ?
I had memory overflow issues during xquery update functions calls, but always 
when issued in the same script, because of saturated PUL.

Best,
Fabrice


Profiling Module - BaseX 
Documentation
Signatures: prof:current-ms() as xs:integer: Summary: Returns the number of 
milliseconds passed since 1970/01/01 UTC. The granularity of the value depends 
on the underlying operating system and may be larger.
docs.basex.org



De : BaseX-Talk  de la part de 
Sebastian Guerrero 
Envoyé : jeudi 28 mai 2020 20:39
À : BaseX 
Objet : [basex-talk] RESTXQ - java.lang.OutOfMemoryError: Java heap space

Hi BaseX team!

A quick question.

Is there some known bug/common setting missing for RESTXQ and memory problems?

I have this simple module ( into /webapp, a .xqm file ) :

module namespace exe = 'http://site.com/execute';
declare
%updating
%rest:path("update")
%rest:consumes("application/x-www-form-urlencoded")
%rest:POST function exe:update() {
xquery:eval-update(request:parameter("query"))
};

I use it to execute some updates against some databases from different clients.

Everything works fine by a while, but after some time I get this error [1]:

java.lang.OutOfMemoryError: Java heap space

I noticed that every call to update() the memory grows and grows until it 
reaches the OutOfMemoryError. [2]

If I stop the HTTP server, the memory is released immediately. [3]

What I'm doing wrong?
Is there some command to execute the GC?
Is this a problem with "xquery:eval-update()"?
Am I using it in the wrong way?

Best regards,
Sebastian.
[1] https://imgur.com/DrcbwQg
[2] https://imgur.com/fonmrhm
[3] https://imgur.com/SYFBFK8


Re: [basex-talk] Full-text index: searches for common words in another node. Does it take a lot of time?

2020-05-19 Thread ETANCHAUD Fabrice
Thank you Sebastian !

Yes, BaseX is an incredible piece of software, reducing development time by 
magnitudes.

The problems I faced were elsewhere :

  *   disruptive technology : pure XML technology is poorly shared among IT 
people, few engineers have a starter level of XPath, XSLT, XQuery. I found that 
most colleagues did not want to improve their skills in that domain, keeping 
with java/jaxb/sax/sql (and sometimes even hibernate...), finding all kinds of 
reasons not to embrace this solution.
  *   not hype (not 'big' data) : excepted MarkLogic (but in an ashamed fashion 
in my opinion), XML is sadly absent from the 'big' data landscape, even though 
we did not wait for big data tools (map/reduce, json...) to handle lots of data 
!
  *   management by the way was reluctant to give that solution a try...
  *   who's that guy that do the entire team's job in one week with a solution 
no one else can maintain ?

So yes I found in love with BaseX and XML too, but even if I had great great 
pleasures, it was (and still is) a kind of secret love, a team and management 
breaker.
I certainly have my part in that situation, with my viceral aversion for things 
like governance, mediocracy...

All the best from french west coast,
Fabrice Etanchaud


De : Sebastian Guerrero 
Envoyé : lundi 18 mai 2020 20:32
À : ETANCHAUD Fabrice 
Cc : basex-talk@mailman.uni-konstanz.de 
Objet : Re: [basex-talk] Full-text index: searches for common words in another 
node. Does it take a lot of time?

Hi Fabrice!

Thanks a lot for your advice. Yes, it's a good idea. And yes, it works.

I created a separated index ( a new database ) for 'mark-identification':

for $db in ('US00','US01','US02')
let $index := {
  for $cases in 
db:open($db)/trademark-applications-daily/application-information/file-segments/action-keys/case-file
  group by $text := $cases/case-file-header/mark-identification
  return
  
{$text}

{for $node in $cases return { db:node-id($node) }}

  
}
return db:create($db || '-mark-text', $index, $db || '-mark-text.xml')

Of course with a full-text index for 'value'.

So, to search I use this piece of code:

  let $text := 'corporation'
  for $db in ('US00','US01','US02')
  for $id in ft:search($db || '-mark-text', $text)/ancestor::text/nodes/id
  let $case-file := db:open-id($db, $id)
  return $case-file

And now it only takes 185ms in order to get the results and there is no scan 
for the 'party-name' values.

- "I really appreciated working with basex that time, because others were in a 
kind of java/relational mapping hell... Me, I just had to add xml documents, 
reindex, and sometimes purge deleted items.": Oh dear, I can't explain to you 
how much I'm in love with BaseX right now. Yes, trying to manage this volume of 
data and translate to a SQL database is like a Kafkaesque nightmare, not a 
healthy idea.

Thank you very much!
Cheers,
Sebastian.

On Mon, May 18, 2020 at 12:43 PM ETANCHAUD Fabrice 
mailto:fabrice.etanch...@maif.fr>> wrote:
Hi Sebastian,
Yes I think your search on mark-identification suffers from the huge number of 
party-names.
>From what I remember, reverse index (from full text tokens to node ids) is 
>shared across all element's names.
so filtering on the element's name is done at last.

When I was using basex to handle DOCDB patent db, I used to explode a document 
in sub-documents containing only keys and text to be indexed with respect to 
language and xml element, and then build seperate databases.
That way I could create a dedicated full text index on a single (element names, 
language) combination.

Did that help ?

I really appreciated working with basex that time, because others were in a 
kind of java/relational mapping hell... Me, I just had to add xml documents, 
reindex, and sometimes purge deleted items.

Best,
Fabrice


De : BaseX-Talk 
mailto:basex-talk-boun...@mailman.uni-konstanz.de>>
 de la part de Sebastian Guerrero mailto:chap...@gmail.com>>
Envoyé : lundi 18 mai 2020 17:23
À : BaseX 
mailto:basex-talk@mailman.uni-konstanz.de>>
Objet : [basex-talk] Full-text index: searches for common words in another 
node. Does it take a lot of time?

Hi everybody.

I'm here again with my doubts. Thank you for your patience. ^^

I have a database of trademarks with a full-text index for two nodes: 
*:mark-identification,*:party-name. [1]

Where "mark-identification" contains the name of the trademark, and 
"party-name" contains the name of the owner of the trademark.

I use the full-text index in order to search trademarks by its name, for 
example:

for $results in //case-file[case-file-header/mark-identification/text() 
contains text {'basex'}]
return $results//mark-identification

returns all trademarks with "basex" on its name. It works like a thunderlight: 
15ms to get 3 records among 2,134,434,598 nodes. Really a dream. [2]

Bu

Re: [basex-talk] Full-text index: searches for common words in another node. Does it take a lot of time?

2020-05-18 Thread ETANCHAUD Fabrice
Hi Sebastian,
Yes I think your search on mark-identification suffers from the huge number of 
party-names.
>From what I remember, reverse index (from full text tokens to node ids) is 
>shared across all element's names.
so filtering on the element's name is done at last.

When I was using basex to handle DOCDB patent db, I used to explode a document 
in sub-documents containing only keys and text to be indexed with respect to 
language and xml element, and then build seperate databases.
That way I could create a dedicated full text index on a single (element names, 
language) combination.

Did that help ?

I really appreciated working with basex that time, because others were in a 
kind of java/relational mapping hell... Me, I just had to add xml documents, 
reindex, and sometimes purge deleted items.

Best,
Fabrice


De : BaseX-Talk  de la part de 
Sebastian Guerrero 
Envoyé : lundi 18 mai 2020 17:23
À : BaseX 
Objet : [basex-talk] Full-text index: searches for common words in another 
node. Does it take a lot of time?

Hi everybody.

I'm here again with my doubts. Thank you for your patience. ^^

I have a database of trademarks with a full-text index for two nodes: 
*:mark-identification,*:party-name. [1]

Where "mark-identification" contains the name of the trademark, and 
"party-name" contains the name of the owner of the trademark.

I use the full-text index in order to search trademarks by its name, for 
example:

for $results in //case-file[case-file-header/mark-identification/text() 
contains text {'basex'}]
return $results//mark-identification

returns all trademarks with "basex" on its name. It works like a thunderlight: 
15ms to get 3 records among 2,134,434,598 nodes. Really a dream. [2]

But, for example, if I change the searched text from "basex" by a common word 
in "party-name", for example, "corporation" ( has 1096187x occurrences on the 
full-text index as showed in [1], it's a very common word in owners of 
trademarks ):

for $results in //case-file[case-file-header/mark-identification/text() 
contains text {'corporation'}]
return $results//mark-identification

It takes a long time to get 6,715 records: 62,000ms [3]

If I search for "live" ( a common word for trademarks name, but not for owners 
names ) I get 5,875 records in 2,773 ms, which has not a relationship with the 
62,000ms to get the 6k records for "corporation". [4]

So...

  *   Is this an expected behaviour?
  *   Is there a way to specify which "section" of the full-text index should 
be used to perform the search? ( I don't know... maybe something similar to 
"using stemming" but "using index 'mark-identification'" )

Please apologize me if I'm asking by something not-logical,

Best regards,
Sebastian

[1] https://imgur.com/uLla1Xt
[2] https://imgur.com/Fkcvv2O
[3] https://imgur.com/Hk71CNe
[4] https://imgur.com/P72k574



Re: [basex-talk] Weird: mixed content trimmed unexpectedly

2019-12-09 Thread ETANCHAUD Fabrice
Hi Omar,

did you try
http://localhost:8984/rest/LeapinLists/test.xml?chop=false
 ?

Best regards,
Fabrice


De : BaseX-Talk  de la part de Omar 
Siam 
Envoyé : lundi 9 décembre 2019 16:51
À : basex-talk@mailman.uni-konstanz.de 
Objet : Re: [basex-talk] Weird: mixed content trimmed unexpectedly


Hello,

I had some expierence of this of my own. Saxon and other XML tools are better 
at guessing what the user wants. Probably there is an option (CHOP?) that one 
can set to tell BaseX not to trim whitespace at the edges of text nodes. But to 
my knowledge the standard way is to add @xml:space="preserve" to the outermost 
element where this is necessary. That's how I do it.

 test for  me 

Best regards

Omar Siam

Am 09.12.2019 um 16:32 schrieb Arjan Loeffen:
Dear BaseX People,

after many (happy!) projects using BaseX I have found that

curl -i -X PUT --basic --user admin ^
   -H "Content-Type: application/xml" -d " test for  me 
" "http://localhost:8984/rest/LeapinLists/test.xml;

stores

test forme

To the database. Notice the spaces in the XML string. What may have caused this 
behavior? Any suggestions on what to do here?

I tried it on Basex 9.3, the latest edition right now.

--
Arjan Loeffen
Armatiek BV & Armatiek Solutions BV
06-12918997


Re: [basex-talk] file pattern when creating DB

2019-12-09 Thread ETANCHAUD Fabrice
Great !
XQuery 3.1 is such a powerful language,
Thanks to Christian and his team's members !
I have been using BaseX for ten years,
and never saw such helping team and community as BaseX's ones !

Christian, did you think about bringing BaseX to Slack ?

Best regards,
fabrice



De : radim-havli...@post.cz 
Envoyé : lundi 9 décembre 2019 10:49
À : ETANCHAUD Fabrice 
Cc : BaseX 
Objet : Re: [basex-talk] file pattern when creating DB

Thank you very much! That was exactly what I was looking for.
Best regards,

Radim Havlicek

-- Původní e-mail --
Od: ETANCHAUD Fabrice 
Komu: BaseX 
Datum: 9. 12. 2019 10:28:57
Předmět: Re: [basex-talk] file pattern when creating DB
Radim, here is a simple snippet :

db:create('test-db', 'F:\Utilisateurs\14484H\data\test', (), map { 
'createfilter' : '*.*' })

Fabrice


De : radim-havli...@post.cz 
Envoyé : lundi 9 décembre 2019 10:19
À : ETANCHAUD Fabrice 
Cc : basex-talk@mailman.uni-konstanz.de 
Objet : RE: [basex-talk] file pattern when creating DB

Hi,
I've seen the link before but I was unable to compose the particular query to 
create the database with filter *, and still I am.

-- Původní e-mail --
Od: ETANCHAUD Fabrice 
Komu: radim-havli...@post.cz , 
basex-talk@mailman.uni-konstanz.de 
Datum: 9. 12. 2019 10:14:37
Předmět: RE: [basex-talk] file pattern when creating DB
Hi Radim,

you can pass parsing options to db:create [1], and set createfilter to the 
desired glob pattern.

Best regards,
Fabrice


[1] : http://docs.basex.org/wiki/Database_Module#db:create



De : BaseX-Talk  de la part de 
radim-havli...@post.cz 
Envoyé : lundi 9 décembre 2019 09:42
À : basex-talk@mailman.uni-konstanz.de 
Objet : [basex-talk] file pattern when creating DB

Hello,
I would like to ask about filters.

I'd like to call XQuery
db:create("DB", "c:/Temp/defaultDocuments/", "/")
but with filter pattern * becase without that pattern it adds *.xml files only. 
I would like to add all files from that directory and sub-directories. Is it 
possible to put the filter pattern there?

I discovered ADD TO / "C:/Temp/defaultDocuments" does exactly what I want, but 
I would much rather use XQuery like db:create or
db:add("DB", "Temp/defaultDocuments", "/") but the db:add filters the files 
with *.xml.

Is it posssible to add all files with any XQuery?
Thank in advance for your reply. Best regards,

Radim Havlicek


Re: [basex-talk] file pattern when creating DB

2019-12-09 Thread ETANCHAUD Fabrice
Radim, here is a simple snippet :

db:create('test-db', 'F:\Utilisateurs\14484H\data\test', (), map { 
'createfilter' : '*.*' })

Fabrice


De : radim-havli...@post.cz 
Envoyé : lundi 9 décembre 2019 10:19
À : ETANCHAUD Fabrice 
Cc : basex-talk@mailman.uni-konstanz.de 
Objet : RE: [basex-talk] file pattern when creating DB

Hi,
I've seen the link before but I was unable to compose the particular query to 
create the database with filter *, and still I am.

-- Původní e-mail --
Od: ETANCHAUD Fabrice 
Komu: radim-havli...@post.cz , 
basex-talk@mailman.uni-konstanz.de 
Datum: 9. 12. 2019 10:14:37
Předmět: RE: [basex-talk] file pattern when creating DB
Hi Radim,

you can pass parsing options to db:create [1], and set createfilter to the 
desired glob pattern.

Best regards,
Fabrice


[1] : http://docs.basex.org/wiki/Database_Module#db:create



De : BaseX-Talk  de la part de 
radim-havli...@post.cz 
Envoyé : lundi 9 décembre 2019 09:42
À : basex-talk@mailman.uni-konstanz.de 
Objet : [basex-talk] file pattern when creating DB

Hello,
I would like to ask about filters.

I'd like to call XQuery
db:create("DB", "c:/Temp/defaultDocuments/", "/")
but with filter pattern * becase without that pattern it adds *.xml files only. 
I would like to add all files from that directory and sub-directories. Is it 
possible to put the filter pattern there?

I discovered ADD TO / "C:/Temp/defaultDocuments" does exactly what I want, but 
I would much rather use XQuery like db:create or
db:add("DB", "Temp/defaultDocuments", "/") but the db:add filters the files 
with *.xml.

Is it posssible to add all files with any XQuery?
Thank in advance for your reply. Best regards,

Radim Havlicek


Re: [basex-talk] file pattern when creating DB

2019-12-09 Thread ETANCHAUD Fabrice
Hi Radim,

you can pass parsing options to db:create [1], and set createfilter to the 
desired glob pattern.

Best regards,
Fabrice


[1] : http://docs.basex.org/wiki/Database_Module#db:create



De : BaseX-Talk  de la part de 
radim-havli...@post.cz 
Envoyé : lundi 9 décembre 2019 09:42
À : basex-talk@mailman.uni-konstanz.de 
Objet : [basex-talk] file pattern when creating DB

Hello,
I would like to ask about filters.

I'd like to call XQuery
db:create("DB", "c:/Temp/defaultDocuments/", "/")
but with filter pattern * becase without that pattern it adds *.xml files only. 
I would like to add all files from that directory and sub-directories. Is it 
possible to put the filter pattern there?

I discovered ADD TO / "C:/Temp/defaultDocuments" does exactly what I want, but 
I would much rather use XQuery like db:create or
db:add("DB", "Temp/defaultDocuments", "/") but the db:add filters the files 
with *.xml.

Is it posssible to add all files with any XQuery?
Thank in advance for your reply. Best regards,

Radim Havlicek


Re: [basex-talk] Reg : ArrayIndexOutOfBoundsException

2019-08-09 Thread ETANCHAUD Fabrice
Hi Chandrasekhar, BaseX 7.9 is more than five years old,
what prevents you from upgrading to the latest version ?

BaseX is far more stable now, I can't remember the last time I faced this issue.

Best regards,
Fabrice


De : BaseX-Talk  de la part de 
chandra Sekhar 
Envoyé : vendredi 9 août 2019 09:26
À : basex-talk@mailman.uni-konstanz.de 
Objet : [basex-talk] Reg : ArrayIndexOutOfBoundsException

Dear basex team,

We are getting the below error while accessing the collection.

Exception ::org.basex.core.BaseXException: Improper use? Potential bug? Your 
feedback is welcome:
Contact: 
basex-talk@mailman.uni-konstanz.de
Version: BaseX 7.9
Java: Oracle Corporation, 1.7.0_151
OS: Linux, amd64
Stack Trace:
java.lang.ArrayIndexOutOfBoundsException

Collection size : 37 GB

And did INSPECT command on the collection,It is showing success.

Regards,
Chandrasekhar


Re: [basex-talk] command script / batch-processing of xqueries

2016-04-19 Thread Etanchaud Fabrice
Hello Günter,

You do can run several xquery files in a single command script with a syntax 
like this :


...
...
...
...


IMHO Xquery scripts are like transactions,
You might overflow memory if you put all your updating functions in a single 
huge transaction.
And in can remember db:create being executed after db:add, so your Pending 
update list might be meaningless at the end :

http://docs.basex.org/wiki/XQuery_Update#Pending_Update_List


Best regards,
Fabrice Etanchaud
Horanet Smart Objects

-Message d'origine-
De : Günter Dunz-Wolff [mailto:kle...@mail.dunzwolff.de] 
Envoyé : lundi 18 avril 2016 18:55
À : Etanchaud Fabrice <fabrice.etanch...@horanet.com>
Cc : Christian Grün <christian.gr...@gmail.com>; BaseX 
<basex-talk@mailman.uni-konstanz.de>
Objet : Re: [basex-talk] command script / batch-processing of xqueries

Hello Fabrice,

thanks for the info. Did see the SET, but didn't notice 
http://docs.basex.org/wiki/Options#Create_Options

> You can use several XQUERY or RUN commands,

I'd like to run a lot of existing XQuery-files (where dbs are created and 
updated) with ONE command.

Perhaps I should define a new xq with xi:include the others and RUN this one 
file?

Thanks again,
Günter

> Am 18.04.2016 um 17:16 schrieb Etanchaud Fabrice 
> <fabrice.etanch...@horanet.com>:
> 
> Hello Günter,
> 
> Did you have a look at
> 
> http://docs.basex.org/wiki/Commands#Command_Scripts
> 
> You can use several XQUERY or RUN commands, and the SET command to 
> configure the db or index options.
> 
> Best regards,
> Fabrice Etanchaud
> Horanet Smart Objects
> 
> -Message d'origine-
> De : basex-talk-boun...@mailman.uni-konstanz.de 
> [mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de 
> Günter Dunz-Wolff Envoyé : lundi 18 avril 2016 16:52 À : BaseX 
> <basex-talk@mailman.uni-konstanz.de>; Christian Grün 
> <christian.gr...@gmail.com> Objet : [basex-talk] command script / 
> batch-processing of xqueries
> 
> Hi all, hi Christian,
> 
> I'm running some xqueries to create and update basex dbs over and over again. 
> They all depend on the basic db 'mydata'. Whenever mydata is updated, I have 
> to update the other dbs.
> 
> 1. xq-file: db:create('mydata', path/to/my/xml .
> 2. xq-file: db:create('searchindex', let $data0 := db:open('mydata') .
> 3. xq-file: db:create('indd-xml', let $data0 := db:open('mydata')
> 4. and so on
> 
> First question:
> Is there any way to put the whole process into one command-script? Or is 
> there any kind of other batch-processing?
> 
> Second question:
> So far I'm creating the db 'mydata' with the GUI. I want to integrate the 
> creating into the batch-processing via XQuery. In generell I know the 
> create-command, but where to put the options: use XInclude, chop-whitespace 
> OFF and Fulltext Language German internal, Stemming OFF, Case Sensitive and 
> Diacritics ON? Can't find it in the docs.
> 
> Thanks for any help,
> 
> Günter
> 
> 
> 
> 
> 
> 
> 




Re: [basex-talk] command script / batch-processing of xqueries

2016-04-18 Thread Etanchaud Fabrice
Hello Günter,

Did you have a look at 

http://docs.basex.org/wiki/Commands#Command_Scripts

You can use several XQUERY or RUN commands,
and the SET command to configure the db or index options.

Best regards,
Fabrice Etanchaud
Horanet Smart Objects

-Message d'origine-
De : basex-talk-boun...@mailman.uni-konstanz.de 
[mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de Günter 
Dunz-Wolff
Envoyé : lundi 18 avril 2016 16:52
À : BaseX ; Christian Grün 

Objet : [basex-talk] command script / batch-processing of xqueries

Hi all, hi Christian,

I'm running some xqueries to create and update basex dbs over and over again. 
They all depend on the basic db 'mydata'. Whenever mydata is updated, I have to 
update the other dbs.

1. xq-file: db:create('mydata', path/to/my/xml .
2. xq-file: db:create('searchindex', let $data0 := db:open('mydata') .
3. xq-file: db:create('indd-xml', let $data0 := db:open('mydata')
4. and so on

First question:
Is there any way to put the whole process into one command-script? Or is there 
any kind of other batch-processing?

Second question:
So far I'm creating the db 'mydata' with the GUI. I want to integrate the 
creating into the batch-processing via XQuery. In generell I know the 
create-command, but where to put the options: use XInclude, chop-whitespace OFF 
and Fulltext Language German internal, Stemming OFF, Case Sensitive and 
Diacritics ON? Can't find it in the docs.

Thanks for any help,

Günter









Re: [basex-talk] Expected behaviour with full disk

2016-04-01 Thread Etanchaud Fabrice
Hello James,

Did you try to set ADDCACHE ?
This way you might run out of disk space during disk caching, before database 
corruption occurs.
Be careful, you may have to serialize the XML in order to use this option with 
db:add/db:replace (I read you are in the REST-XQ layer).

Otherwise, the only way to prevent database corruption might be to check disk 
size regularly and reject new data by program…

http://docs.basex.org/wiki/Options#ADDCACHE
http://docs.basex.org/wiki/Database_Module#db:add


Best regards,
Fabrice Etanchaud
Horanet Smart Objects
FR-85200 Fontenay le Comte

De : basex-talk-boun...@mailman.uni-konstanz.de 
[mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de James Ball
Envoyé : vendredi 1 avril 2016 16:48
À : BaseX 
Objet : [basex-talk] Expected behaviour with full disk

Hello,

I’m having some problems with BaseX in situations where there it runs out of 
disk space.

I’m delivering XML files to BaseX 8.4 using RestXQ.

A request is made and the XML is added to a database depending on some 
criteria. I receive many hundreds of thousands of requests and everything works 
well unless I run out of disk space.
J
What I would expect to happen is that BaseX would try to add the document, find 
it couldn’t due to the disk being full and return a server error (e.g. 500) and 
the transaction would be aborted leaving the database as it was before the 
request. Any further requests would all get errors returned.

What I’m seeing is that when a request is made that causes the disk to become 
full BaseX gets part way through adding the document, fails and returns an 
error but also leaves the database corrupted. Usually the server has to be 
restarted and then any database that has been affected has to be deleted - it’s 
rare to be able to save anything from it.

The errors vary but it is usually a NullPointer exception and then 
java.nio.channels.OverlappingFileLockException. for further requests.

I’ve tried turning off logging and setting parallel to 1 to see if that made 
any difference but it doesn’t.

Is what I’m seeing unique to my setup or do others see this?

Is there anything I can do to make BaseX just stop accepting requests when disk 
space runs out to protect my databases?

Many thanks, James


Re: [basex-talk] Binding external variable in GUI

2016-02-15 Thread Etanchaud Fabrice
Hi Buddy,

Did you try the $x button between the run query and run tests ones ?
Best regards,
Fabrice



De : basex-talk-boun...@mailman.uni-konstanz.de 
[mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de 
buddyonweb-softw...@yahoo.com
Envoyé : lundi 15 février 2016 16:53
À : BaseX 
Objet : [basex-talk] Binding external variable in GUI

At the risk of a) sounding like a real newbie  b) being accused of not Googling 
for an answer (which I did - but apparently overlooked)
I am wondering how I can bind an external variable using just XQuery syntax in 
the BaseX GUI.

I have found how to set the variable via command line options, and how to do it 
in Java, but I merely
wanted to run a quick interactive test in the GUI  to test an idea of mine 
(before writing a bunch of Java code).
What I have done is in a tab in the GUI, I have included the module, and before 
running a function, want to set that external variable.
Surprisingly I found myself at a loss on how to do what appears to be a simple 
task.

So, thanks in advance for any helpand for the rest of this community for 
not laughing at me too loudly

Buddy



Re: [basex-talk] Streaming file loads

2016-01-19 Thread Etanchaud Fabrice
Hello Christophe,

Are you using java ?
Maybe you should have a look at  the add() method in :

https://github.com/BaseXdb/basex/blob/master/basex-examples/src/main/java/org/basex/examples/api/BaseXClient.java

I can also remember that command line tools can read standard input stream with 
the '-' argument.

Cordialement,

Fabrice Etanchaud
Horanet Smart Objects


-Message d'origine-
De : basex-talk-boun...@mailman.uni-konstanz.de 
[mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de Christophe 
Marchand
Envoyé : mardi 19 janvier 2016 12:20
À : basex-talk@mailman.uni-konstanz.de
Objet : [basex-talk] Streaming file loads

Hello !

We have to load many files (>20.000), large size (>30Gb) into BaseX. 
Today, that's done vis a BXS script which accesses files thru a network share.

I would like to optimize network transfert ; tar / compression at source, 
uncompress / untar at destination. But I would like to avoid to create 
temporary files (at source, on NAS server, and on destination server, BaseX 
host).

So, is there an API which I could use to load a "file" thru a stream ? I think 
I can write an inputstream that redirects data to outpustreams, cutting input 
data in many output streams, one per file. But may I chain this outputstream to 
an inputstream usable by BaseX loader ?

Best regards,
Christophe


Re: [basex-talk] Question about baseX add "/path/to/filesystem/directory"

2015-12-11 Thread Etanchaud Fabrice
Hi JMA,

Did you try setting SKIPCORRUPT to true ?

Here is an example :

Command:
SET SKIPCORRUPT true
Command:
CREATE DB test C:/Data/adresses/test
Result:
Skipped:
- C:/Data/adresses/test/communes (2).xml
Database 'test' created in 1806.47 ms.

So you will see the list of skipped files in the output stream.

May your sub directories contain archives ?
That could explain these invisible files.

I never had problem with the ADD command even on huge directory structures.

Bonjour de Vendée
Cordialement,
Fabrice


-Message d'origine-
De : basex-talk-boun...@mailman.uni-konstanz.de 
[mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de 
ansel...@free.fr
Envoyé : vendredi 11 décembre 2015 06:19
À : basex-talk@mailman.uni-konstanz.de
Objet : [basex-talk] Question about baseX add "/path/to/filesystem/directory"

Hello

I'm experiencing a trouble with the  add "/path/to/filesystem/directory" on a 
large directory structure with thousands of files. 
The result is a strange error : 

> add "/data/PARTAGE/DOSSIERS EC/PC"
"Configurations2/accelerator/current.xml" (Line 1): No text allowed before root 
element.


And in the directory /data/PARTAGE/DOSSIERS EC/PC, no Configurations2 
directory...
I think a file is having a problem but with a thousand of files, how to know 
which of them ? 

On another directory, the add function works but finds xml files that aren't in 
the directory structure (find . -name "xxx" doesn't find them as basex finds 
!!!)...

Is the "add function" fragile on a directory ?

Many thanks in advance

JMA 


Re: [basex-talk] Returning text matches

2015-12-11 Thread Etanchaud Fabrice
Dear Liam,

I am afraid that full text index will not find "blue socks", because it does 
not cross text() node boundaries:

http://docs.basex.org/wiki/Full-Text#Mixed_Content

Best regards,
Fabrice


-Message d'origine-
De : basex-talk-boun...@mailman.uni-konstanz.de 
[mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de Liam R. E. 
Quin
Envoyé : jeudi 10 décembre 2015 21:37
À : Christian Grün ; Ron Katriel 
Cc : basex-talk@mailman.uni-konstanz.de
Objet : Re: [basex-talk] Returning text matches

On Thu, 2015-12-10 at 17:24 +0100, Christian Grün wrote:
> Hi Ron,
> 
> You can use ft:mark and ft:extract to highlights matches in a 
> full-text result [1].

And what happens if a full text match crosses an element boundary, e.g.
a search for "blue socks" matching,
He wore dark blue socks that day. could not return, He wore 
dark blue socks that day.

(Yes, I should test it, sorry! but the docs should probably mention it.
it was a big part of the XPath/XQuery Full Text design early on)

Liam

--
Liam R. E. Quin 
The World Wide Web Consortium (W3C)



Re: [basex-talk] AUTOOPTIMIZE - was true now false

2015-12-01 Thread Etanchaud Fabrice
Hello James,

Did you set AUTOOPTIMIZE to true BEFORE database creation ?
How did you set the option ? in the db:create call, or as a SET/CREATE DB pair ?

I don't have this problem with 8.3.1

Best regards,
Fabrice


De : basex-talk-boun...@mailman.uni-konstanz.de 
[mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de James Ball
Envoyé : lundi 30 novembre 2015 16:57
À : basex-talk@mailman.uni-konstanz.de
Objet : [basex-talk] AUTOOPTIMIZE - was true now false

Hello,

I'm trying out AUTOOPTIMIZE for the first time and I'm not getting what I 
expect.

Running: BaseX 8.3 beta ec9b627

A new database is created with a single document which has TEXTINDEX, 
ATTRINDEX, UPDINDEX and AUTOOPTIMISE all set to true.

This is what INFO DATABASE shows (as I expect):

Indexes
 Up-to-date: true
 TEXTINDEX: true
 ATTRINDEX: true
 FTINDEX: false
 TEXTINCLUDE:
 ATTRINCLUDE:
 FTINCLUDE:
 LANGUAGE: English
 STEMMING: false
 CASESENS: false
 DIACRITICS: false
 STOPWORDS:
 UPDINDEX: true
 AUTOOPTIMIZE: true
 MAXCATS: 100
 MAXLEN: 96

Now I add a new document (or more than one) to the database. I expect that 
Up-to-date will still be true and that AUTOOPTIMIZE will still be true. But 
what I see is both are now false:

Indexes
 Up-to-date: false
 TEXTINDEX: true
 ATTRINDEX: true
 FTINDEX: false
 TEXTINCLUDE:
 ATTRINCLUDE:
 FTINCLUDE:
 LANGUAGE: English
 STEMMING: false
 CASESENS: false
 DIACRITICS: false
 STOPWORDS:
 UPDINDEX: true
 AUTOOPTIMIZE: false
 MAXCATS: 100
 MAXLEN: 96

Perhaps this is the expected behaviour and I've misunderstood AUTOOPTIMIZE? Or 
is there a bug?

Many thanks, James


[basex-talk] RESTXQ : existing annotation for response headers ?

2015-11-30 Thread Etanchaud Fabrice
Dear all,

When I want to set response headers (for example to force a "save as..."),
I currently have to write a specific response before the response body :

  

  
  

  

Is there an annotation to set header values directly in the function 
declaration ?

Best regards,
Fabrice


Re: [basex-talk] RESTXQ : existing annotation for response headers ?

2015-11-30 Thread Etanchaud Fabrice
Hello Dirk and Christian,

Thank you for your - as usual ! - fast reply.
web:response-header is exactly the function I was about to reinvent...

and you are right Christian, I should have realized that annotation would not 
help,
since my header value is dynamically computed...

Have a nice week !

Regards,
Fabrice


-Message d'origine-
De : Christian Grün [mailto:christian.gr...@gmail.com] 
Envoyé : lundi 30 novembre 2015 15:15
À : Etanchaud Fabrice <fabrice.etanch...@horanet.com>
Cc : basex-talk@mailman.uni-konstanz.de
Objet : Re: [basex-talk] RESTXQ : existing annotation for response headers ?

Hi Fabrice,

There is currently no way to specify these headers via annotations. It may 
indeed be tricky, because there are many cases in which the headers are 
dynamically generated (such as in your example:
concat(..)).

Did you try the web:response-header function [1]?

Cheers,
Christian

[1] http://docs.basex.org/wiki/Web_Module#web:response-header


On Mon, Nov 30, 2015 at 3:07 PM, Etanchaud Fabrice 
<fabrice.etanch...@horanet.com> wrote:
> Dear all,
>
>
>
> When I want to set response headers (for example to force a “save 
> as…”),
>
> I currently have to write a specific response before the response body :
>
>
>
>   
>
> 
>
>value="text/comma-separated-values;
> charset=utf-8"/>
>
>   
>
> 
>
>   
>
>
>
> Is there an annotation to set header values directly in the function 
> declaration ?
>
>
>
> Best regards,
>
> Fabrice


Re: [basex-talk] db:create, namespaces not stripped when input is variable?

2015-11-27 Thread Etanchaud Fabrice
Hello Constantine,

IMHO parsing options only apply when there is actually a parsing phase.
I remember having the same problem with whitespace chopping.
Would you get expected result by serializing the xml before giving it to the 
db:create function ?

Best regards,
Fabrice


De : basex-talk-boun...@mailman.uni-konstanz.de 
[mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de Hondros, 
Constantine (ELS-AMS)
Envoyé : jeudi 26 novembre 2015 18:49
À : 'basex-talk@mailman.uni-konstanz.de' 
Objet : [basex-talk] db:create, namespaces not stripped when input is variable?

Hello all,
Any particular reason behind this observed behaviour? (Basex 8.3).

..
[1]
..
declare variable $test :=
http://www.elsevier.com/xml/ani/common; 
xmlns="http://www.elsevier.com/xml/ani/ani;>
  M.S.
;

db:create('test', $test, 'blah', map {'stripns':true()})

..
[2]
..

db:open('test')/*

..
[3]
..

http://www.elsevier.com/xml/ani/ani; 
xmlns:ce="http://www.elsevier.com/xml/ani/common;>
  M.S.




I would have expected the namespaces to be shredded ...

Cheers,
Constantine





Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The 
Netherlands, Registration No. 33156677, Registered in The Netherlands.


Re: [basex-talk] Applicative Cache

2015-11-20 Thread Etanchaud Fabrice
I am afraid that this would not work.
If you do not create the mainmem database at server's initialization,
but afterwards from a client,
then the database lives as long as the client's connection.
So using the client module in a RESTXQ request will not help, because 
connection is closed at the end of the request.

Currently, I use a basexclient for that, and keep the client console open, to 
prevent the mainmem db from closing.

Best regards,
Fabrice

-Message d'origine-
De : Christian Grün [mailto:christian.gr...@gmail.com] 
Envoyé : vendredi 20 novembre 2015 15:28
À : Marco Lettere <marco.lett...@dedalus.eu>
Cc : Etanchaud Fabrice <fabrice.etanch...@horanet.com>; 
basex-talk@mailman.uni-konstanz.de
Objet : Re: [basex-talk] Applicative Cache

> What about running from within a RESTXQ a 
> client:connect/client:execute against the coupled db server that has 
> been configured to startup like suggested by Fabrice (-c"SET MAINMEM 
> TRUE;CREATE DB mainmem")?

…Marco’s approach ;) I agree, this would be possible! Looking forward to your 
experiences.

Christian


Re: [basex-talk] sending variables to xquery module via client api (python)

2015-11-17 Thread Etanchaud Fabrice
Hi Bryan,

Could http://docs.basex.org/wiki/Options#BINDINGS give you a solution ?

Best regards,
Fabrice

De : basex-talk-boun...@mailman.uni-konstanz.de 
[mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de Cholfin, Bryan
Envoyé : lundi 16 novembre 2015 23:08
À : basex-talk@mailman.uni-konstanz.de
Objet : [basex-talk] sending variables to xquery module via client api (python)

Hi-

Hoping someone can help me with this. I'm trying to write a python script that 
calls an xquery module in the database. It seems to be pretty easy to use a 
line such as:

session.execute("run gettext.xqm")

and get back the response I want. What I want to do is pass a bound variable to 
the module. The client api has a provision to bind a variable to a query 
object, but I'm not making a query object. I'm simply executing the stored 
query.  I don't find anything in the documentation to suggest that a variable 
can be added to the run command this way.

Bryan Cholfin

Associate Editor for Digital Projects and Metadata
Harvard University Press
617.496.9116
79 Garden Street | Cambridge, MA 02138-1400
www.hup.harvard.edu


Re: [basex-talk] Applicative Cache

2015-11-09 Thread Etanchaud Fabrice
Hi Marco,

I did a few tests in that configuration last year (not with 8.X version),
And it worked, as long as I did not "OPEN" another database.
But I tested it again with 8.3, and did not succeed in sharing a server side 
mainmem db.

Server :

java -cp "%CP%" %BASEX_JVM% org.basex.BaseXServer %* -c"SET MAINMEM TRUE;CREATE 
DB mainmem"

Client :

> xquery db:open('mainmem')
Stopped at ., 1/9:
[bxerr:BXDB0002] Database 'mainmem' was not found.

Best regards,
Fabrice

-Message d'origine-
De : basex-talk-boun...@mailman.uni-konstanz.de 
[mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de Marco Lettere
Envoyé : lundi 9 novembre 2015 16:29
À : BaseX 
Objet : [basex-talk] Applicative Cache

Hi all,
I went though the emails talking about mainmemory databases and related but 
I've not been able to find an answer to this loud thinking ...
Would it be possible to create an in memory database with a RESTXQ call and 
keep it accessible for other RESTXQ invocations as if it was a disk db 
throughout the lifecycle of a basexhttp server?
Application level cacheing of (possibly distributed) dbs or management of 
transient volatile data items like tokens are the scenarios I have in mind.
Just wondering whether this makes any sense. Whether it's feasible or whether 
someone already did this with other means.
Thanks a lot!
Marco.


Re: [basex-talk] Best way to batch up a set of DB creates, updates and queries?

2015-10-30 Thread Etanchaud Fabrice
Hello Constantine,

Facing the same problem, as commands can be written in xml,
I found the following solution useful (as usual thank you Christian), to inject 
configuration where needed :

Declare variable $my-option as xs:string external;
Declare variable $my-value as xs:string external;


{$my-value}
...


Best regards,

Fabrice
Horanet Smart Objects
F-85200 Fontenay-le-Comte


De : basex-talk-boun...@mailman.uni-konstanz.de 
[mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de Hondros, 
Constantine (ELS-AMS)
Envoyé : vendredi 30 octobre 2015 10:34
À : 'basex-talk@mailman.uni-konstanz.de' 
Objet : Re: [basex-talk] Best way to batch up a set of DB creates, updates and 
queries?

Please let me also clarify, I'm looking for a solution that can be run from the 
GUI - so I'm not looking for suggestions using shell scripts or Ant which would 
be easy to implement.

Thanks,
C.

From: 
basex-talk-boun...@mailman.uni-konstanz.de
 [mailto:basex-talk-boun...@mailman.uni-konstanz.de] On Behalf Of Hondros, 
Constantine (ELS-AMS)
Sent: 29 October 2015 18:12
To: 'basex-talk@mailman.uni-konstanz.de'
Subject: [basex-talk] Best way to batch up a set of DB creates, updates and 
queries?

I'm looking for the best way to batch up a series of updating operations into a 
single pipeline that can be run at once.

To give you an idea, we have to create multiple databases, perform a number of 
updating post-processes on each, before finally pulling certain fields out.

I've got a lot of parameters that need to externalised somehow - database 
creation options, location of DTDs, XSDs, Catalogs, and unless I am trying this 
the wrong way I am finding it quite difficult.

I can't write it all XQuery - I have a mixture of updating and non-updating 
functions.

I can't get much traction out of the BXS scripting syntax - for example, I 
can't seem to create and use variables using this.

What do other people out there use to glue these sorts of operations together?

TIA,
Constantine



Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The 
Netherlands, Registration No. 33156677, Registered in The Netherlands.



Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The 
Netherlands, Registration No. 33156677, Registered in The Netherlands.


Re: [basex-talk] Best way to batch up a set of DB creates, updates and queries?

2015-10-30 Thread Etanchaud Fabrice
My fault,
I missed the final step :
You have to execute the resulting xml.

declare variable $my-option as xs:string external;
declare variable $my-value as xs:string external;

file:write('myScript.bxs',


{$my-value}
...
)

And then, in the GUI, load the bxs file and run it.

If there is a server running, you could try to client:execute() the resulting 
xml.
I did not find a xquery function that allow you to run a batch.

Best regards,
Fabrice

De : Hondros, Constantine (ELS-AMS) [mailto:c.hond...@elsevier.com]
Envoyé : vendredi 30 octobre 2015 14:07
À : Etanchaud Fabrice <fabrice.etanch...@horanet.com>; 
'basex-talk@mailman.uni-konstanz.de' <basex-talk@mailman.uni-konstanz.de>
Objet : RE: [basex-talk] Best way to batch up a set of DB creates, updates and 
queries?

Hello Fabrice,

Thanks, but I'm struggling to understand your example - you've got a mixture of 
xquery and Basex script syntax. Can you give me a slightly larger snippet, as I 
don't see how this can work.

Thanks!

From: Etanchaud Fabrice [mailto:fabrice.etanch...@horanet.com]
Sent: 30 October 2015 12:38
To: Hondros, Constantine (ELS-AMS); 'basex-talk@mailman.uni-konstanz.de'
Subject: RE: [basex-talk] Best way to batch up a set of DB creates, updates and 
queries?

Hello Constantine,

Facing the same problem, as commands can be written in xml,
I found the following solution useful (as usual thank you Christian), to inject 
configuration where needed :

Declare variable $my-option as xs:string external;
Declare variable $my-value as xs:string external;


{$my-value}
...


Best regards,

Fabrice
Horanet Smart Objects
F-85200 Fontenay-le-Comte


De : 
basex-talk-boun...@mailman.uni-konstanz.de<mailto:basex-talk-boun...@mailman.uni-konstanz.de>
 [mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de Hondros, 
Constantine (ELS-AMS)
Envoyé : vendredi 30 octobre 2015 10:34
À : 'basex-talk@mailman.uni-konstanz.de' 
<basex-talk@mailman.uni-konstanz.de<mailto:basex-talk@mailman.uni-konstanz.de>>
Objet : Re: [basex-talk] Best way to batch up a set of DB creates, updates and 
queries?

Please let me also clarify, I'm looking for a solution that can be run from the 
GUI - so I'm not looking for suggestions using shell scripts or Ant which would 
be easy to implement.

Thanks,
C.

From: 
basex-talk-boun...@mailman.uni-konstanz.de<mailto:basex-talk-boun...@mailman.uni-konstanz.de>
 [mailto:basex-talk-boun...@mailman.uni-konstanz.de] On Behalf Of Hondros, 
Constantine (ELS-AMS)
Sent: 29 October 2015 18:12
To: 'basex-talk@mailman.uni-konstanz.de'
Subject: [basex-talk] Best way to batch up a set of DB creates, updates and 
queries?

I'm looking for the best way to batch up a series of updating operations into a 
single pipeline that can be run at once.

To give you an idea, we have to create multiple databases, perform a number of 
updating post-processes on each, before finally pulling certain fields out.

I've got a lot of parameters that need to externalised somehow - database 
creation options, location of DTDs, XSDs, Catalogs, and unless I am trying this 
the wrong way I am finding it quite difficult.

I can't write it all XQuery - I have a mixture of updating and non-updating 
functions.

I can't get much traction out of the BXS scripting syntax - for example, I 
can't seem to create and use variables using this.

What do other people out there use to glue these sorts of operations together?

TIA,
Constantine



Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The 
Netherlands, Registration No. 33156677, Registered in The Netherlands.



Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The 
Netherlands, Registration No. 33156677, Registered in The Netherlands.



Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The 
Netherlands, Registration No. 33156677, Registered in The Netherlands.