Re: [basex-talk] BaseX 9.1.2: Maintenance Release

[George Sofianos]

Thanks Christian,

I think I managed to make it work, if I'm not missing anything. The 
problem was (the last time I tried it, over a year ago), that I need to 
work with relative paths, and location uris would not work properly. I 
managed to set it up today, by using a QueryProcessor constructor that 
includes a base-uri parameter.


The problem is, that when you call the QueryProcessor [1] the base URI 
should be set to a fake uri or file path, for example 
"/home/user/application/queries/fake". This way, both location URIs 
(import module declarations), and document requests to relative urls - 
doc(concat("../myxml.xml")) work. If base URI is just set as 
"/home/user/application/queries" - which I thought it would be correct - 
then only doc() function will look at the correct base-uri directory, 
while the import module statements, will try to import files from the 
parent of the base directory. I think it has something to do with how 
baseURI function works in StaticContext.java [2], but I'm not sure.


Maybe a Java example showing the correct usage of the base-uri, or a 
wiki page will help others in the future.


[1]: 
https://github.com/BaseXdb/basex/blob/master/basex-examples/src/main/java/org/basex/examples/local/BindVariables.java


[2]: 
https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/query/StaticContext.java


I'm happy I got this out of the way, my next email will be about some 
issues I found with parallel xquery processing :)


Regards,

George

On 1/23/19 3:04 PM, Christian Grün wrote:

Hi George,

Sorry, we have no plans to reintroduce the old option (to much has 
changed in our architecture). Maybe there are other solutions for your 
use case in the latest version?


Best
Christian

Re: [basex-talk] BaseX/GUI v9.1.2 memory use

[Bridger Dyson-Smith]

Mea culpa! I may have steered the train of conversation in a non-productive
direction: my home is parsed due to how I launch the GUI. In any event, I
haven't had any issues with parsing project files or symbolic links,
either.

Apologies,
Bridger

On Wed, Jan 23, 2019 at 8:30 AM Johannes Echterhoff <
echterh...@interactive-instruments.de> wrote:

> For what it’s worth: in our developments we had a directory as part of the
> project files where we stored all queries that were executed during
> development and testing (these queries were dynamically created by other
> queries). Saving these queries is just part of our process. However, after
> a while that resulted in hundreds to thousands of query files. That
> resulted in a massive memory demand, and we noticed increased CPU usage,
> both of which we could not explain at first. Eventually we found out that
> the BaseX GUI automatically parsed all these queries, each time
> initializing a custom module we wrote, where the initialization of that
> module created a temporary directory with a number of files. The solution
> for us was to simply move the directory where we save the executed queries
> somewhere outside of the project files opened in the BaseX GUI.
>
>
>
> Best regards,
>
> Johannes
>
>
>
>
>
> *Von:* BaseX-Talk [mailto:basex-talk-boun...@mailman.uni-konstanz.de] *Im
> Auftrag von *Graydon Saunders
> *Gesendet:* Mittwoch, 23. Januar 2019 14:19
> *An:* Christian Grün 
> *Cc:* BaseX 
> *Betreff:* Re: [basex-talk] BaseX/GUI v9.1.2 memory use
>
>
>
> Never had a problem with the GUI parsing project files.  No issues with
> symbolic links.
>
>
>
> I have generally found basex very reliable but never try to update the
> installed version; it's always move the old, unpack the new.
>
> On Wed, Jan 23, 2019, 08:09 Christian Grün 
> wrote:
>
> Thanks for your assessments. So maybe we should ensure that every file and
> directory will only be parsed once.
>
>
>
> Did anyone of you have problems with the automatic parsing of project
> files in the BaseX GUI, or with symbolic links in particular?
>
>
>
>
>
>
>
>
>
>
>
> Am Mi., 23. Jan. 2019, 09:12 hat Marco Lettere 
> geschrieben:
>
> Same for us over here. The ability to follow symlinks is a very powerful
> feature that we use to externalize folders (data, restxq for instance). So
> please don't remove it altogether!
>
> M.
>
>
>
> On 22/01/19 23:50, Graydon Saunders wrote:
>
> I've been handling updates by making data/ a symbolic link to a data
> directory that's a sibling of the basex directory.  (Move the old, unpack
> the new, go into new and replace data/ with a symbolic link up and over.)
>
>
>
> Would hate to see that stop working.
>
> On Tue, Jan 22, 2019, 17:36 Christian Grün 
> wrote:
>
> Good to hear that! I can’t recollect that something particular has changed
> in version 9.1.2, regarding the scanning of project files, but I’ll have
> some thoughts how we can trace and interrupt such loops (or ignore symbolic
> links instead).
>
>
>
>
>
> Am Di., 22. Jan. 2019, 23:22 hat Bridger Dyson-Smith <
> bdysonsm...@gmail.com> geschrieben:
>
> Glad that helped :)
>
>
>
> I see this when I start from a fresh install vs expanding the ZIP into the
> same directory.
>
>
>
> On Tue, Jan 22, 2019, 5:17 PM Rick Graham 
> Thanks Bridger!
>
>
>
> Indeed, I quit basexgui and manually edited .basexgui to set the project
> directory to a newly created empty directory.  basexgui seems normal/stable
> after that.
>
>
>
> I rarely, as in almost never, use wine but I didn't have this issue with
> previous versions of BaseX.  Something seems unexpected here.
>
>
>
>
>
> On Tue, Jan 22, 2019 at 11:04 PM Bridger Dyson-Smith <
> bdysonsm...@gmail.com> wrote:
>
> Hi Rick, et al,
>
> I think (but am not 100% sure) that the GUI defaults to looking through
> your home directory on startup. So, somewhere in
> `~/rick/.wine/dosdevices/...` you have symbolic links that are looped.
>
>
>
> I think you might be able to circumvent this problem by finding
> `.basexgui` - it would probably be close to wherever you started the GUI
> from on your filesystem. I think you can edit some of the PATHS there and
> that may help?
>
>
>
> Again, I'm not sure. HTH!
>
> Best,
>
> Bridger
>
>
>
> On Tue, Jan 22, 2019 at 4:56 PM Rick Graham  wrote:
>
> The command-line seemed to be operating normally.
>
>
>
> What exactly is/are my project directories?
>
>
>
> I attached to the running GUI instance `strace -f -e trace=stat -p 13368`
> and it has infinite repetitions of:
>
>
>
> [pid 13436]
> 

Re: [basex-talk] BaseX/GUI v9.1.2 memory use

[Johannes Echterhoff]

For what it’s worth: in our developments we had a directory as part of the 
project files where we stored all queries that were executed during development 
and testing (these queries were dynamically created by other queries). Saving 
these queries is just part of our process. However, after a while that resulted 
in hundreds to thousands of query files. That resulted in a massive memory 
demand, and we noticed increased CPU usage, both of which we could not explain 
at first. Eventually we found out that the BaseX GUI automatically parsed all 
these queries, each time initializing a custom module we wrote, where the 
initialization of that module created a temporary directory with a number of 
files. The solution for us was to simply move the directory where we save the 
executed queries somewhere outside of the project files opened in the BaseX GUI.

Best regards,
Johannes


Von: BaseX-Talk [mailto:basex-talk-boun...@mailman.uni-konstanz.de] Im Auftrag 
von Graydon Saunders
Gesendet: Mittwoch, 23. Januar 2019 14:19
An: Christian Grün 
Cc: BaseX 
Betreff: Re: [basex-talk] BaseX/GUI v9.1.2 memory use

Never had a problem with the GUI parsing project files.  No issues with 
symbolic links.

I have generally found basex very reliable but never try to update the 
installed version; it's always move the old, unpack the new.
On Wed, Jan 23, 2019, 08:09 Christian Grün 
mailto:christian.gr...@gmail.com>> wrote:
Thanks for your assessments. So maybe we should ensure that every file and 
directory will only be parsed once.

Did anyone of you have problems with the automatic parsing of project files in 
the BaseX GUI, or with symbolic links in particular?





Am Mi., 23. Jan. 2019, 09:12 hat Marco Lettere 
mailto:m.lett...@gmail.com>> geschrieben:
Same for us over here. The ability to follow symlinks is a very powerful 
feature that we use to externalize folders (data, restxq for instance). So 
please don't remove it altogether!
M.

On 22/01/19 23:50, Graydon Saunders wrote:
I've been handling updates by making data/ a symbolic link to a data directory 
that's a sibling of the basex directory.  (Move the old, unpack the new, go 
into new and replace data/ with a symbolic link up and over.)

Would hate to see that stop working.
On Tue, Jan 22, 2019, 17:36 Christian Grün 
mailto:christian.gr...@gmail.com>> wrote:
Good to hear that! I can’t recollect that something particular has changed in 
version 9.1.2, regarding the scanning of project files, but I’ll have some 
thoughts how we can trace and interrupt such loops (or ignore symbolic links 
instead).


Am Di., 22. Jan. 2019, 23:22 hat Bridger Dyson-Smith 
mailto:bdysonsm...@gmail.com>> geschrieben:
Glad that helped :)

I see this when I start from a fresh install vs expanding the ZIP into the same 
directory.

On Tue, Jan 22, 2019, 5:17 PM Rick Graham 
mailto:rickhg1...@gmail.com> wrote:
Thanks Bridger!

Indeed, I quit basexgui and manually edited .basexgui to set the project 
directory to a newly created empty directory.  basexgui seems normal/stable 
after that.

I rarely, as in almost never, use wine but I didn't have this issue with 
previous versions of BaseX.  Something seems unexpected here.


On Tue, Jan 22, 2019 at 11:04 PM Bridger Dyson-Smith 
mailto:bdysonsm...@gmail.com>> wrote:
Hi Rick, et al,
I think (but am not 100% sure) that the GUI defaults to looking through your 
home directory on startup. So, somewhere in `~/rick/.wine/dosdevices/...` you 
have symbolic links that are looped.

I think you might be able to circumvent this problem by finding `.basexgui` - 
it would probably be close to wherever you started the GUI from on your 
filesystem. I think you can edit some of the PATHS there and that may help?

Again, I'm not sure. HTH!
Best,
Bridger

On Tue, Jan 22, 2019 at 4:56 PM Rick Graham 
mailto:rickhg1...@gmail.com>> wrote:
The command-line seemed to be operating normally.

What exactly is/are my project directories?

I attached to the running GUI instance `strace -f -e trace=stat -p 13368` and 
it has infinite repetitions of:

[pid 13436] 
stat("/home/rick/.wine/dosdevices/z:/sys/class/thermal/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone0/device/subsystem/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/device:75/subsystem/devices/PNP0C0A:02",
 0x7f7beb2796e0) = -1 ELOOP (Too many levels of symbolic links)

What's going on here?

On Tue, Jan 22, 2019 at 10:21 PM Christian Grün 
mailto:christian.gr...@gmail.com>> wrote:
at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:167)
at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173)
at 

Re: [basex-talk] BaseX/GUI v9.1.2 memory use

[Christian Grün]

Thanks for your assessments. So maybe we should ensure that every file and
directory will only be parsed once.

Did anyone of you have problems with the automatic parsing of project files
in the BaseX GUI, or with symbolic links in particular?





Am Mi., 23. Jan. 2019, 09:12 hat Marco Lettere 
geschrieben:

> Same for us over here. The ability to follow symlinks is a very powerful
> feature that we use to externalize folders (data, restxq for instance). So
> please don't remove it altogether!
> M.
>
> On 22/01/19 23:50, Graydon Saunders wrote:
>
> I've been handling updates by making data/ a symbolic link to a data
> directory that's a sibling of the basex directory.  (Move the old, unpack
> the new, go into new and replace data/ with a symbolic link up and over.)
>
> Would hate to see that stop working.
>
> On Tue, Jan 22, 2019, 17:36 Christian Grün 
> wrote:
>
>> Good to hear that! I can’t recollect that something particular has
>> changed in version 9.1.2, regarding the scanning of project files, but I’ll
>> have some thoughts how we can trace and interrupt such loops (or ignore
>> symbolic links instead).
>>
>>
>>
>> Am Di., 22. Jan. 2019, 23:22 hat Bridger Dyson-Smith <
>> bdysonsm...@gmail.com> geschrieben:
>>
>>> Glad that helped :)
>>>
>>> I see this when I start from a fresh install vs expanding the ZIP into
>>> the same directory.
>>>
>>> On Tue, Jan 22, 2019, 5:17 PM Rick Graham >>
 Thanks Bridger!

 Indeed, I quit basexgui and manually edited .basexgui to set the
 project directory to a newly created empty directory.  basexgui seems
 normal/stable after that.

 I rarely, as in almost never, use wine but I didn't have this issue
 with previous versions of BaseX.  Something seems unexpected here.


 On Tue, Jan 22, 2019 at 11:04 PM Bridger Dyson-Smith <
 bdysonsm...@gmail.com> wrote:

> Hi Rick, et al,
> I think (but am not 100% sure) that the GUI defaults to looking
> through your home directory on startup. So, somewhere in
> `~/rick/.wine/dosdevices/...` you have symbolic links that are looped.
>
> I think you might be able to circumvent this problem by finding
> `.basexgui` - it would probably be close to wherever you started the GUI
> from on your filesystem. I think you can edit some of the PATHS there and
> that may help?
>
> Again, I'm not sure. HTH!
> Best,
> Bridger
>
> On Tue, Jan 22, 2019 at 4:56 PM Rick Graham 
> wrote:
>
>> The command-line seemed to be operating normally.
>>
>> What exactly is/are my project directories?
>>
>> I attached to the running GUI instance `strace -f -e trace=stat -p
>> 13368` and it has infinite repetitions of:
>>
>> [pid 13436]
>>> stat("/home/rick/.wine/dosdevices/z:/sys/class/thermal/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone0/device/subsystem/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/device:75/subsystem/devices/PNP0C0A:02",
>>> 0x7f7beb2796e0) = -1 ELOOP (Too many levels of symbolic links)
>>
>>
>> What's going on here?
>>
>> On Tue, Jan 22, 2019 at 10:21 PM Christian Grün <
>> christian.gr...@gmail.com> wrote:
>>
>>> at org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:167)
> at
> org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173)
> at
> org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173)
> at
> org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173)
>

>>> Looks like a endless loop that is caused by parsing the files in
>>> your project directory. Do you possibly have any symbolic links?
>>>
>>> Can you reproduce the problem with a completely fresh BaseX zip
>>> archive?
>>>
>>>
>>>
>

Re: [basex-talk] BaseX 9.1.2: Maintenance Release

[Christian Grün]

Hi George,

Sorry, we have no plans to reintroduce the old option (to much has changed
in our architecture). Maybe there are other solutions for your use case in
the latest version?

Best
Christian




Am Mi., 23. Jan. 2019, 11:42 hat George Sofianos 
geschrieben:

> Hi Christian,
>
> thanks for the new version. Will it be available in maven soon?
>
> Also another question that I hate to make, but I have to, are there any
> plans to re-introduce QUERYPATH? This has been keeping me from upgrading
> to a more recent version than 8.4.4, but we have another need that
> requires 9.1.2 at the moment.
>
> Regards,
>
> George
>
> On 1/22/19 11:06 AM, Christian Grün wrote:
> > Hi all,
> >
> > A new BaseX maintenance release is available. Minor bugs have been
> > fixed, and some performance tweaks have been added. In particular,
> > access to large WebDAV directories should be faster now.
> >
> > As usual, you find the latest version on our homepage basex.org. Maven
> > artifacts have been added, various other distributions will follow
> > soon.
> >
> > Have fun everyone,
> > Christian, BaseX Team
>

Re: [basex-talk] BaseX 9.1.2: Maintenance Release

[George Sofianos]

Oops, I just noticed it already is available. I need to stop looking 
into mvnrepository first. Sorry about that!


George

On 1/23/19 12:42 PM, George Sofianos wrote:

Hi Christian,

thanks for the new version. Will it be available in maven soon?

Also another question that I hate to make, but I have to, are there 
any plans to re-introduce QUERYPATH? This has been keeping me from 
upgrading to a more recent version than 8.4.4, but we have another 
need that requires 9.1.2 at the moment.


Regards,

George

Re: [basex-talk] BaseX 9.1.2: Maintenance Release

[George Sofianos]

Hi Christian,

thanks for the new version. Will it be available in maven soon?

Also another question that I hate to make, but I have to, are there any 
plans to re-introduce QUERYPATH? This has been keeping me from upgrading 
to a more recent version than 8.4.4, but we have another need that 
requires 9.1.2 at the moment.


Regards,

George

On 1/22/19 11:06 AM, Christian Grün wrote:

Hi all,

A new BaseX maintenance release is available. Minor bugs have been
fixed, and some performance tweaks have been added. In particular,
access to large WebDAV directories should be faster now.

As usual, you find the latest version on our homepage basex.org. Maven
artifacts have been added, various other distributions will follow
soon.

Have fun everyone,
Christian, BaseX Team

Re: [basex-talk] BaseX/GUI v9.1.2 memory use

[Marco Lettere]

Same for us over here. The ability to follow symlinks is a very powerful 
feature that we use to externalize folders (data, restxq for instance). 
So please don't remove it altogether!

M.

On 22/01/19 23:50, Graydon Saunders wrote:
I've been handling updates by making data/ a symbolic link to a data 
directory that's a sibling of the basex directory.  (Move the old, 
unpack the new, go into new and replace data/ with a symbolic link up 
and over.)


Would hate to see that stop working.

On Tue, Jan 22, 2019, 17:36 Christian Grün > wrote:


Good to hear that! I can’t recollect that something particular has
changed in version 9.1.2, regarding the scanning of project files,
but I’ll have some thoughts how we can trace and interrupt such
loops (or ignore symbolic links instead).



Am Di., 22. Jan. 2019, 23:22 hat Bridger Dyson-Smith
mailto:bdysonsm...@gmail.com>> geschrieben:

Glad that helped :)

I see this when I start from a fresh install vs expanding the
ZIP into the same directory.

On Tue, Jan 22, 2019, 5:17 PM Rick Graham
mailto:rickhg1...@gmail.com> wrote:

Thanks Bridger!

Indeed, I quit basexgui and manually edited .basexgui to
set the project directory to a newly created empty
directory.  basexgui seems normal/stable after that.

I rarely, as in almost never, use wine but I didn't have
this issue with previous versions of BaseX.  Something
seems unexpected here.


On Tue, Jan 22, 2019 at 11:04 PM Bridger Dyson-Smith
mailto:bdysonsm...@gmail.com>> wrote:

Hi Rick, et al,
I think (but am not 100% sure) that the GUI defaults
to looking through your home directory on startup. So,
somewhere in `~/rick/.wine/dosdevices/...` you have
symbolic links that are looped.

I think you might be able to circumvent this problem
by finding `.basexgui` - it would probably be close to
wherever you started the GUI from on your filesystem.
I think you can edit some of the PATHS there and that
may help?

Again, I'm not sure. HTH!
Best,
Bridger

On Tue, Jan 22, 2019 at 4:56 PM Rick Graham
mailto:rickhg1...@gmail.com>>
wrote:

The command-line seemed to be operating normally.

What exactly is/are my project directories?

I attached to the running GUI instance `strace -f
-e trace=stat -p 13368` and it has infinite
repetitions of:

[pid 13436]

stat("/home/rick/.wine/dosdevices/z:/sys/class/thermal/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone0/device/subsystem/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/device:75/subsystem/devices/PNP0C0A:02",
0x7f7beb2796e0) = -1 ELOOP (Too many levels of
symbolic links)


What's going on here?

On Tue, Jan 22, 2019 at 10:21 PM Christian Grün
mailto:christian.gr...@gmail.com>> wrote:

at

org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:167)
at

org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173)
at

org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173)
at

org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173)


Looks like a endless loop that is caused by
parsing the files in your project directory.
Do you possibly have any symbolic links?

Can you reproduce the problem with a
completely fresh BaseX zip archive?

Re: [basex-talk] BaseX 9.1.2: Maintenance Release

[Alexander Holupirek]

For all the macOS/homebrew [1] users,

BaseX 9.1.2 is now available:

$ brew update
Updated Homebrew from 80c2ed310 to 550202d00.
Updated 2 taps (homebrew/core and homebrew/cask).
==> New Formulae
grpcurl
==> Updated Formulae
basex ✔ 

$ brew upgrade basex
==> Upgrading 1 outdated package:
basex 9.1.1 -> 9.1.2
==> Upgrading basex 
==> Downloading http://files.basex.org/releases/9.1.2/BaseX912.zip
  /usr/local/Cellar/basex/9.1.2: 129 files, 10.5MB, built in 2 seconds

Cheers,
Alex

[1] https://brew.sh

> On 22. Jan 2019, at 10:06, Christian Grün  wrote:
> 
> Hi all,
> 
> A new BaseX maintenance release is available. Minor bugs have been
> fixed, and some performance tweaks have been added. In particular,
> access to large WebDAV directories should be faster now.
> 
> As usual, you find the latest version on our homepage basex.org. Maven
> artifacts have been added, various other distributions will follow
> soon.
> 
> Have fun everyone,
> Christian, BaseX Team