Re: [basex-talk] Tomcat & services

[Christian Grün]

>
> At the same time the unfortunate consequence is that jobs.xml, which to
> our feeling is something that lives with code and not with data, has to be
> copied manually to the external dbpath in scenarios where you redeploy a
> webapp after some fixes or tuning. So our suggestion is to have something
> like a "jobs" variable in web.xml that points to an servlet internal
> jobs.xml (maybe in the WEB-INF folder?) to be used instead of the default
> one.
>

I see! We’ll keep this in mind and think about some alternatives.

Re: [basex-talk] Tomcat & services

[Marco Lettere]

On 15/10/2018 17:56, Christian Grün wrote:

Thanks for your summary on the behavior of Tomcat.

> In general we think that when working with tomcat one usually has to 
link an external dbpath in order to avoid getting his data removed 
when redeploying the webapp. The requirement to put jobs.xml inside 
the datapath somehow breaks the atomicity of a deployable webapp.
> A workaround could be to have the jobs.xml somehow redefined as a 
web.xml variable. Might this be possible?


To understand your use case better:
1. Do you currently define an external database path when working with 
Tomcat?
Yes. It's necessary. Otherwise, for each redeploy of the webapp, the 
data is lost.
2. Would you prefer to have a servlet-specific jobs.xml file and an 
external database directory, or should both of them be placed 
internally or externally?
I think there is nothing against how dbpath is handled now. One has just 
to uncomment the dbpath variable inside the web.xml of the webapps that 
require persistent data and change it. That's good enough for us.
At the same time the unfortunate consequence is that jobs.xml, which to 
our feeling is something that lives with code and not with data, has to 
be copied manually to the external dbpath in scenarios where you 
redeploy a webapp after some fixes or tuning. So our suggestion is to 
have something like a "jobs" variable in web.xml that points to an 
servlet internal jobs.xml (maybe in the WEB-INF folder?) to be used 
instead of the default one.


In any case I don't know if this solves the current misbehaviour we have 
with tomcat which simply and rudely ignores the content of jobs.xml 
altogether.


I hope things are slightly clearer now.
M.





Just a guess, but maybe it would generally be a good idea to be able 
to define an external database directory for Tomcat? Or would you 
indeed like to have servlet-specific databases removod if



On Mon, Oct 15, 2018 at 5:47 PM Marco Lettere > wrote:


Ok, actually on my personal PC jobs are correctly listed also with
basexhttp (need to investigate why on my colleagues PC this
doesn't work).

Back to the more important issue...
I put the following jobs.xml[1] file in the data folder and get
[2] as first lines of log in dba after reboot.
I put the same file into the "dbpath" folder of my app running as
tomcat webapp but the log looks like [3].

The feeling is that services are scheduled only in accordance to
the startup of the http server which doesn't occur when running
inside Tomcat.

In general we think that when working with tomcat one usually has
to link an external dbpath in order to avoid getting his data
removed when redeploying the webapp. The requirement to put
jobs.xml inside the datapath somehow breaks the atomicity of a
deployable webapp.
A workaround could be to have the jobs.xml somehow redefined as a
web.xml variable. Might this be possible?

Thanks for your support.
M.

[1]

  admin:write-log('!')
  admin:write-log("1=1")


[2]
17:39:05.1000:0:0:0:0:0:0:1:42740   admin   302 745.90  [GET]
http://localhost:8984/dba
17:39:02.715SERVER  admin   INFONaN !
17:39:02.715SERVER  admin   INFONaN 1=1
17:39:02.468SERVER  admin   OK  NaN HTTP Server was started
(port: 8984).
17:39:02.454SERVER  admin   OK  NaN Server was started 
(port:
1984).


[3]
17:29:04.9330:0:0:0:0:0:0:1:52332   admin   302 1497.81 
[GET]
http://localhost:8081/contract-repository/dba
17:28:43.218SERVER  admin   OK  NaN Server was started 
(port:
1984).



On 15/10/2018 16:59, Christian Grün wrote:

Hi Marco,

I tried to reproduce the problem:

1. I downloadedhttp://files.basex.org/releases/9.0.2/BaseX902.zip
2. I unzipped the archive
3. I added a jobs.xml file in the basex/data directory:


   admin:write-log('!')


4. I started basexhttp
5. As expected, the log entry is listed in the DBA logs panel.
6. jobs:services() returns admin:write-log('!')

Could you do the same and report back to us what you did differently?

Thanks in advance
Christian





Hi Christian,
thank you as usual for your attention.
With standalone I mean basexhttp which is what we usually do. This time
we have requirement for running inside tomcat...
The jobs.xml file looks like:


if (db:exists('onedb')) then () else
db:create('ondedb')


but we have also tried with:


admin:write-log(current-dateTime())


which presents the exact same issues: basexhttp runs ok but
jobs:services() doesn't list them whereas tomcat doesn't run the scripts
but services are listed.
Thank you again,
Marco.

On 15/10/2018 15:25, Christian Grün wrote:

Hi Marco,

I see y

Re: [basex-talk] Tomcat & services

[Christian Grün]

Thanks for your summary on the behavior of Tomcat.

> In general we think that when working with tomcat one usually has to link
an external dbpath in order to avoid getting his data removed when
redeploying the webapp. The requirement to put jobs.xml inside the datapath
somehow breaks the atomicity of a deployable webapp.
> A workaround could be to have the jobs.xml somehow redefined as a web.xml
variable. Might this be possible?

To understand your use case better:
1. Do you currently define an external database path when working with
Tomcat?
2. Would you prefer to have a servlet-specific jobs.xml file and an
external database directory, or should both of them be placed internally or
externally?



Just a guess, but maybe it would generally be a good idea to be able to
define an external database directory for Tomcat? Or would you indeed like
to have servlet-specific databases removod if


On Mon, Oct 15, 2018 at 5:47 PM Marco Lettere  wrote:

> Ok, actually on my personal PC jobs are correctly listed also with
> basexhttp (need to investigate why on my colleagues PC this doesn't work).
>
> Back to the more important issue...
> I put the following jobs.xml[1] file in the data folder and get [2] as
> first lines of log in dba after reboot.
> I put the same file into the "dbpath" folder of my app running as tomcat
> webapp but the log looks like [3].
>
> The feeling is that services are scheduled only in accordance to the
> startup of the http server which doesn't occur when running inside Tomcat.
>
> In general we think that when working with tomcat one usually has to link
> an external dbpath in order to avoid getting his data removed when
> redeploying the webapp. The requirement to put jobs.xml inside the datapath
> somehow breaks the atomicity of a deployable webapp.
> A workaround could be to have the jobs.xml somehow redefined as a web.xml
> variable. Might this be possible?
>
> Thanks for your support.
> M.
>
> [1]
> 
>   admin:write-log('!')
>   admin:write-log("1=1")
> 
>
> [2]
> 17:39:05.100 0:0:0:0:0:0:0:1:42740 admin 302 745.90 [GET]
> http://localhost:8984/dba
> 17:39:02.715 SERVER admin INFO NaN !
> 17:39:02.715 SERVER admin INFO NaN 1=1
> 17:39:02.468 SERVER admin OK NaN HTTP Server was started (port: 8984).
> 17:39:02.454 SERVER admin OK NaN Server was started (port: 1984).
> [3]
> 17:29:04.933 0:0:0:0:0:0:0:1:52332 admin 302 1497.81 [GET]
> http://localhost:8081/contract-repository/dba
> 17:28:43.218 SERVER admin OK NaN Server was started (port: 1984).
>
> On 15/10/2018 16:59, Christian Grün wrote:
>
> Hi Marco,
>
> I tried to reproduce the problem:
>
> 1. I downloaded http://files.basex.org/releases/9.0.2/BaseX902.zip
> 2. I unzipped the archive
> 3. I added a jobs.xml file in the basex/data directory:
>
> 
>   admin:write-log('!')
> 
>
> 4. I started basexhttp
> 5. As expected, the log entry is listed in the DBA logs panel.
> 6. jobs:services() returns admin:write-log('!')
>
> Could you do the same and report back to us what you did differently?
>
> Thanks in advance
> Christian
>
>
>
>
>
> Hi Christian,
> thank you as usual for your attention.
> With standalone I mean basexhttp which is what we usually do. This time
> we have requirement for running inside tomcat...
> The jobs.xml file looks like:
>
> 
> if (db:exists('onedb')) then () else
> db:create('ondedb')
> 
>
> but we have also tried with:
>
> 
> admin:write-log(current-dateTime())
> 
>
> which presents the exact same issues: basexhttp runs ok but
> jobs:services() doesn't list them whereas tomcat doesn't run the scripts
> but services are listed.
> Thank you again,
> Marco.
>
> On 15/10/2018 15:25, Christian Grün wrote:
>
> Hi Marco,
>
> I see your question didn’t get an answer yet. Maybe we can first try
> to find out what Jetty does (or does not):
>
>
> When running standalone with Jetty, the scripts are executed even if, in
> this case, we are getting empty results when running jobs:services() in
> the DBA after startup.
>
> Some questions back:
> 1. By 'standalone', do you mean that you are running 'basexhttp', or
> do you run Jetty and deploy BaseX as WAR file?
> 2. How does the jobs.xml file in the database directory look like?
>
> Cheers,
> Christian
>
>
>

Re: [basex-talk] Tomcat & services

[Marco Lettere]

Ok, actually on my personal PC jobs are correctly listed also with 
basexhttp (need to investigate why on my colleagues PC this doesn't work).


Back to the more important issue...
I put the following jobs.xml[1] file in the data folder and get [2] as 
first lines of log in dba after reboot.
I put the same file into the "dbpath" folder of my app running as tomcat 
webapp but the log looks like [3].


The feeling is that services are scheduled only in accordance to the 
startup of the http server which doesn't occur when running inside Tomcat.


In general we think that when working with tomcat one usually has to 
link an external dbpath in order to avoid getting his data removed when 
redeploying the webapp. The requirement to put jobs.xml inside the 
datapath somehow breaks the atomicity of a deployable webapp.
A workaround could be to have the jobs.xml somehow redefined as a 
web.xml variable. Might this be possible?


Thanks for your support.
M.

[1]

  admin:write-log('!')
  admin:write-log("1=1")


[2]
17:39:05.100 	0:0:0:0:0:0:0:1:42740 	admin 	302 	745.90 	[GET] 
http://localhost:8984/dba

17:39:02.715SERVER  admin   INFONaN !
17:39:02.715SERVER  admin   INFONaN 1=1
17:39:02.468SERVER  admin   OK  NaN HTTP Server was started (port: 
8984).
17:39:02.454SERVER  admin   OK  NaN Server was started (port: 1984).


[3]
17:29:04.933 	0:0:0:0:0:0:0:1:52332 	admin 	302 	1497.81 	[GET] 
http://localhost:8081/contract-repository/dba

17:28:43.218SERVER  admin   OK  NaN Server was started (port: 1984).



On 15/10/2018 16:59, Christian Grün wrote:

Hi Marco,

I tried to reproduce the problem:

1. I downloaded http://files.basex.org/releases/9.0.2/BaseX902.zip
2. I unzipped the archive
3. I added a jobs.xml file in the basex/data directory:


   admin:write-log('!')


4. I started basexhttp
5. As expected, the log entry is listed in the DBA logs panel.
6. jobs:services() returns admin:write-log('!')

Could you do the same and report back to us what you did differently?

Thanks in advance
Christian





Hi Christian,
thank you as usual for your attention.
With standalone I mean basexhttp which is what we usually do. This time
we have requirement for running inside tomcat...
The jobs.xml file looks like:


if (db:exists('onedb')) then () else
db:create('ondedb')


but we have also tried with:


admin:write-log(current-dateTime())


which presents the exact same issues: basexhttp runs ok but
jobs:services() doesn't list them whereas tomcat doesn't run the scripts
but services are listed.
Thank you again,
Marco.

On 15/10/2018 15:25, Christian Grün wrote:

Hi Marco,

I see your question didn’t get an answer yet. Maybe we can first try
to find out what Jetty does (or does not):


When running standalone with Jetty, the scripts are executed even if, in
this case, we are getting empty results when running jobs:services() in
the DBA after startup.

Some questions back:
1. By 'standalone', do you mean that you are running 'basexhttp', or
do you run Jetty and deploy BaseX as WAR file?
2. How does the jobs.xml file in the database directory look like?

Cheers,
Christian

Re: [basex-talk] Importing XML (DITA) validated with Relax NG schemas?

[Jason Davis]

Hi Christian, Radu,

I’ve tried adding the dita-ng.jar to the lib/custom dir of basex and then 
manually modified the startup script to load it first. I can even confirm that 
it’s the first jar on the path using:

proc:property('java.class.path')

However, the database still fails to parse the XML with default attributes 
applied. I find myself having to cobble together an undesirable workaround 
whereby I manually supply the default attribute values myself in order to get 
my project to work with BaseX. Do you have any further suggestions for how I 
might get this to work?

Thanks,
Jason 

On 10/3/18, 3:24 AM, "Christian Grün"  wrote:

Hi Jason (cc to the list),

> I set the CP variable like so:
> 
CP=$MAIN/BaseX.jar:$MAIN/lib/custom/dita-ng.jar:$MAIN/lib/*:$MAIN/lib/custom/*:$CLASSPATH
>
> This appears to be slightly different than the example you linked 
Christian. I’m using BaseX 9.0.2. Does this make a difference?

The start scripts in the official distributions are created from the
GitHub examples I linked, so they are slightly different.

> I added an echo $CLASSPATH line under the CP variable. When I run the 
script, the echo statement is blank.

In the script, no value will be bound to the $CLASSPATH variable.
Instead, you can assign values to this variable by yourself, which
will then be appended to the $CP variable. If you didn’t do so, and if
your Linux environment does not have any other values assigned to this
variable (which is the default), the output will necessarily be empty.

> Is there a way to see how the classpath is set when running this script?

To answer the "how": It will be set via the line that you will find
some lines below in the script, and the -cp Java argument:

  exec java -cp "$CP" $BASEX_JVM org.basex.BaseX "$@"

If you want to know which value is bound to $CP, try "echo $CP". In
Java, the full user class path at runtime will be bound to the
"java.class.path" system property. It can e.g. be retrieved via
proc:property('java.class.path') [1].

Christian

[1] http://docs.basex.org/wiki/Process_Module#proc:property

Re: [basex-talk] Tomcat & services

[Christian Grün]

Hi Marco,

I tried to reproduce the problem:

1. I downloaded http://files.basex.org/releases/9.0.2/BaseX902.zip
2. I unzipped the archive
3. I added a jobs.xml file in the basex/data directory:


  admin:write-log('!')


4. I started basexhttp
5. As expected, the log entry is listed in the DBA logs panel.
6. jobs:services() returns admin:write-log('!')

Could you do the same and report back to us what you did differently?

Thanks in advance
Christian




> Hi Christian,
> thank you as usual for your attention.
> With standalone I mean basexhttp which is what we usually do. This time
> we have requirement for running inside tomcat...
> The jobs.xml file looks like:
>
> 
> if (db:exists('onedb')) then () else
> db:create('ondedb')
> 
>
> but we have also tried with:
>
> 
> admin:write-log(current-dateTime())
> 
>
> which presents the exact same issues: basexhttp runs ok but
> jobs:services() doesn't list them whereas tomcat doesn't run the scripts
> but services are listed.
> Thank you again,
> Marco.
>
> On 15/10/2018 15:25, Christian Grün wrote:
> > Hi Marco,
> >
> > I see your question didn’t get an answer yet. Maybe we can first try
> > to find out what Jetty does (or does not):
> >
> >> When running standalone with Jetty, the scripts are executed even if, in
> >> this case, we are getting empty results when running jobs:services() in
> >> the DBA after startup.
> > Some questions back:
> > 1. By 'standalone', do you mean that you are running 'basexhttp', or
> > do you run Jetty and deploy BaseX as WAR file?
> > 2. How does the jobs.xml file in the database directory look like?
> >
> > Cheers,
> > Christian
>
>

Re: [basex-talk] Tomcat & services

[Marco Lettere]

Hi Christian,
thank you as usual for your attention.
With standalone I mean basexhttp which is what we usually do. This time 
we have requirement for running inside tomcat...

The jobs.xml file looks like:


if (db:exists('onedb')) then () else 
db:create('ondedb')



but we have also tried with:


admin:write-log(current-dateTime())


which presents the exact same issues: basexhttp runs ok but 
jobs:services() doesn't list them whereas tomcat doesn't run the scripts 
but services are listed.

Thank you again,
Marco.

On 15/10/2018 15:25, Christian Grün wrote:

Hi Marco,

I see your question didn’t get an answer yet. Maybe we can first try
to find out what Jetty does (or does not):


When running standalone with Jetty, the scripts are executed even if, in
this case, we are getting empty results when running jobs:services() in
the DBA after startup.

Some questions back:
1. By 'standalone', do you mean that you are running 'basexhttp', or
do you run Jetty and deploy BaseX as WAR file?
2. How does the jobs.xml file in the database directory look like?

Cheers,
Christian

Re: [basex-talk] Tomcat & services

[Christian Grün]

Hi Marco,

I see your question didn’t get an answer yet. Maybe we can first try
to find out what Jetty does (or does not):

> When running standalone with Jetty, the scripts are executed even if, in
> this case, we are getting empty results when running jobs:services() in
> the DBA after startup.

Some questions back:
1. By 'standalone', do you mean that you are running 'basexhttp', or
do you run Jetty and deploy BaseX as WAR file?
2. How does the jobs.xml file in the database directory look like?

Cheers,
Christian

Re: [basex-talk] Database Updates

[Christian Grün]

Hi Dave,

here is one way to do it:

  declare context item := db:open('catalog');
  declare variable $update := 'daily-update.xml';

  for $new in doc($update)//ENTRY
  let $old := //ENTRY[SKU = $new/SKU]
  return if($old) then (
replace node $old with $new
  ) else (
insert node $new into /CATALOG/ENTRIES
  )

Hope this helps,
Christian


On Sun, Oct 14, 2018 at 12:10 AM Dave Kopecek  wrote:
>
> Here are some tweaked examples & better problem definition:
>
> Given a database created from catalog.xml below, what's the best way to 
> update & add records received daily as in file daily-update.xml?
>
> For every  node in daily-update.xml
> -- If ENTRY/SKU found in catalog.xml replace the entire  node in 
> catalog.xml with the one from daily-update.xml
> -- If ENTRY/SKU not found in catalog.xml add the  node from 
> daily-update.xml to catalog.xml
>
> catalog.xml
> 
> 
>
>   
>  1
>  Empire Burlesque
>  Dylan
>  10.90
>   
>   
>  2
>  Hide your heart
>  Bonnie Tyler
>  9.90
>   
>
> 
>
> daily-update.xml
> 
> 
>
>   
>  1
>  Empire Burlesque
>  Bob Dylan
>  29.99
>  some value
>   
>   
>  5
>  Tupelo Honey
>  Van Morrison
>  8.20
>   
>
> 
>
> catalog.xml after daily update:
> 
>
>   
>  1
>  Empire Burlesque
>  Bob Dylan
>  29.99
>  some value
>   
>   
>  2
>  Hide your heart
>  Bonnie Tyler
>  9.90
>   
>   
>  5
>  Tupelo Honey
>  Van Morrison
>  8.20
>   
>
> 
>
> Thanks,
> -Dave
>
> On Fri, Aug 17, 2018 at 11:27 AM Dave Kopecek  wrote:
>>
>> Hi All,
>>
>> Given a database created from catalog.xml below. What's the best way to 
>> update & add records received daily as in file daily-update.xml ?
>>
>> Coming from relational DBs & new to this. I ultimately need to 
>> script/automate this. Hoping there's some magic command I'm missing & 
>> looking for best way to approach the problem.
>>
>> Thanks,
>> -Dave
>>
>>
>> catalog.xml
>> 
>> 
>>
>>   
>>  1
>>  Empire Burlesque
>>  Dylan
>>  10.90
>>   
>>   
>>  2
>>  Hide your heart
>>  Bonnie Tyler
>>  9.90
>>   
>>
>> 
>>
>> daily-update.xml
>> 
>> 
>>
>>   
>>  1
>>  19.90
>>  I wasn't here before the update
>>   
>>   
>>  1971
>>  Tupelo Honey
>>  Van Morrison
>>  8.20
>>   
>>
>> 
>>
>
>
> --
> DAVE KOPECEK OFFICE 607-431-8565 CELL 607-267-3449
> 6 CROSS STREET, DELHI NY 13753

Re: [basex-talk] Database Updates

[Fabrice ETANCHAUD]

Hi Dave,

Like you, because of my RDBMS background, I had to feel the ‘document’ paradigm 
in order to obtain good performance results.

I suggest you do not try to update your catalog document in place using XQuery 
Update facility.

A simple solution is to add each daily-update, and annotate it with a 
my-sort-value value you can sort on (the date in your use case).
When looking for a given SKU, use an index to obtain the list of entry 
elements, ordered by descending my-sort-value, and take the first item.
When you hit performance or storage limits, create a collection containing only 
the latest version of each entry element.

But as your data is not really document oriented, did you consider using JSON 
repositories like couchbase ?
You could choose to store each entry element in a separate document with SKU as 
key.
Coupled with just a little preprocessing step - that could be written easly in 
XQuery - transforming your entries element in a JSON  array of entry objects, 
you will obtain incredible performance.
Couchbase as a ‘SQL like’ language to query your bucket of documents called 
N1QL.

Best regards,
Fabrice


De : BaseX-Talk [mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part 
de Dave Kopecek
Envoyé : dimanche 14 octobre 2018 00:10
À : basex-talk@mailman.uni-konstanz.de
Objet : Re: [basex-talk] Database Updates

Here are some tweaked examples & better problem definition:

Given a database created from catalog.xml below, what's the best way to update 
& add records received daily as in file daily-update.xml?

For every  node in daily-update.xml
-- If ENTRY/SKU found in catalog.xml replace the entire  node in 
catalog.xml with the one from daily-update.xml
-- If ENTRY/SKU not found in catalog.xml add the  node from 
daily-update.xml to catalog.xml

catalog.xml


   
  
 1
 Empire Burlesque
 Dylan
 10.90
  
  
 2
 Hide your heart
 Bonnie Tyler
 9.90
  
   


daily-update.xml


   
  
 1
 Empire Burlesque
 Bob Dylan
 29.99
 some value
  
  
 5
 Tupelo Honey
 Van Morrison
 8.20
  
   


catalog.xml after daily update:

   
  
 1
 Empire Burlesque
 Bob Dylan
 29.99
 some value
  
  
 2
 Hide your heart
 Bonnie Tyler
 9.90
  
  
 5
 Tupelo Honey
 Van Morrison
 8.20
  
   


Thanks,
-Dave

On Fri, Aug 17, 2018 at 11:27 AM Dave Kopecek 
mailto:dave.kope...@gmail.com>> wrote:
Hi All,

Given a database created from catalog.xml below. What's the best way to update 
& add records received daily as in file daily-update.xml ?

Coming from relational DBs & new to this. I ultimately need to script/automate 
this. Hoping there's some magic command I'm missing & looking for best way to 
approach the problem.

Thanks,
-Dave


catalog.xml


   
  
 1
 Empire Burlesque
 Dylan
 10.90
  
  
 2
 Hide your heart
 Bonnie Tyler
 9.90
  
   


daily-update.xml


   
  
 1
 19.90
 I wasn't here before the update
  
  
 1971
 Tupelo Honey
 Van Morrison
 8.20
  
   




--
DAVE KOPECEK OFFICE 607-431-8565 CELL 607-267-3449
6 CROSS STREET, DELHI NY 13753