Re: [basex-talk] HTTPServer + gzip compression

[Stefan Koch]

Hi,Sorry for self-bumping - but topic might have been overlooked because of holydays.While gzip may be my personal wishlist - It would be could if others could chime in and say what they think about it. HTTPS however is pretty important - independent of my personal wishlist. If there is an embedded jetty already - it should support HTTPS without using proxy workarounds, IMO. See recent issue with a lot of repostitories stopping HTTP support, breaking a couple of  build chains. (https://www.alphabot.com/security/blog/2020/java/Your-Java-builds-might-break-starting-January-13th.html)HTTP is gonna cause  more issues/bugs in the future ... For security reasons HTTPS should be mandatory and default, IMO. Trusted SSL certificates are free nowadays (self signed still works for services) ...Keep up the great work,Stefan-Ursprüngliche Nachricht-Von: Stefan Koch [mailto:k...@buit-solutions.com]Gesendet: Montag, 23. Dezember 2019 19:16An: basex-talk@mailman.uni-konstanz.deBetreff: Re: [basex-talk] HTTPServer + gzip compressionHi Christian, Checked the GZIP Handler options. https://www.eclipse.org/jetty/documentation/current/gzip-filter.htmlIt should support: includedMethods(GET, POST, PUT) - PATCH is not used by BaseX REST API - not needed.However these are response settings. Jetty does support request compression - if BaseX works for XML-Webservices, you wanna POST/PUT large XML docs that are compressed, I guess. As far as I could figure out: "inflateBufferSize" should be set to 1 (0 is default) (it's in the official jetty-gzip.xml not in the link from above). No default option in jetty, shouldn't be default in BaseX REST client, I guess. This should enable request compression. HTTP is dead, long live HTTPS!For productive use this is needed (otherwise you have to run NGINX as proxy or do similar workarounds). Currently only HTTP is supported. Downloaded jetty - in /etc there are the config xmls - relevant config options are: jetty-https,jetty-ssl, jetty-ssl-context (you wanna have a keystore/certificate for SSL). productive use: jetty-threadpool, jetty-threadlimit and jetty-connectionlimit might come in handy for the admins - but have no clue about that.Not really a priority, though. There is a bunch of other stuff, but IMO not really important. Best wishes and nice holydays,Stefan-Ursprüngliche Nachricht-Von: Christian Grün [mailto:christian.gr...@gmail.com]Gesendet: Dienstag, 17. Dezember 2019 07:59An: Stefan Koch Cc: BaseX Betreff: Re: [basex-talk] HTTPServer + gzip compressionHi Stefan,We could consider an extension of the current solution.> So only for GET - not POST.> Our current Webservices rely heavily on POST (+Response) though :(Have you checked the remaining default options? What other customsettings may you need to rely on?Cheers,Christian> -Ursprüngliche Nachricht-> Von: Stefan Koch [mailto:k...@buit-solutions.com]> Gesendet: Mittwoch, 20. November 2019 14:45> An: basex-talk@mailman.uni-konstanz.de> Betreff: Re: [basex-talk] HTTPServer + gzip compression>>> Hi,>> great to hear! So it's coming in 9.3>> We have been running base.war via tomcat with gzip instead - but it wasn't optimal for our solution.>>> Thx for fixing>> Stefan>>>> -Ursprüngliche Nachricht-> Von: Christian Grün [mailto:christian.gr...@gmail.com]> Gesendet: Mittwoch, 20. November 2019 13:26> An: Omar Siam > Cc: BaseX > Betreff: Re: [basex-talk] HTTPServer + gzip compression>>> Thanks for the confirmation!>>>> Omar Siam  schrieb am Mi., 20. Nov. 2019, 12:41:>>>> Yes, it works! 1000 entries of XML in a JSON wrapper from my data are>> now transfered as 193 KB instead of 1.6 MB. Not bad. Unfortunately on>> localhost that doesn't make much of a difference in overall timing as>> measured in Chrome's development tools. For real internet connections I>> think this is quite an improvement.>>>> Best regards>>>> Omar Siam>>>> Am 19.11.2019 um 01:28 schrieb Christian Grün:>> > The snapshot has been updated.>> > Is it working now?>> >>> >>> >>> > On Mon, Nov 18, 2019 at 6:41 PM Omar Siam  wrote:>> >> I just wanted to try gzip http compression and downloaded the current build BaseX93-20191116. The option is not there anymore. What happened?>> >>>> >> Best regards>> >>>> >> Omar Siam

Re: [basex-talk] HTTPServer + gzip compression

[Stefan Koch]

Hi Christian, Checked the GZIP Handler options. https://www.eclipse.org/jetty/documentation/current/gzip-filter.htmlIt should support: includedMethods(GET, POST, PUT) - PATCH is not used by BaseX REST API - not needed.However these are response settings. Jetty does support request compression - if BaseX works for XML-Webservices, you wanna POST/PUT large XML docs that are compressed, I guess. As far as I could figure out: "inflateBufferSize" should be set to 1 (0 is default) (it's in the official jetty-gzip.xml not in the link from above). No default option in jetty, shouldn't be default in BaseX REST client, I guess. This should enable request compression. HTTP is dead, long live HTTPS!For productive use this is needed (otherwise you have to run NGINX as proxy or do similar workarounds). Currently only HTTP is supported. Downloaded jetty - in /etc there are the config xmls - relevant config options are: jetty-https,jetty-ssl, jetty-ssl-context (you wanna have a keystore/certificate for SSL). productive use: jetty-threadpool, jetty-threadlimit and jetty-connectionlimit might come in handy for the admins - but have no clue about that.Not really a priority, though. There is a bunch of other stuff, but IMO not really important. Best wishes and nice holydays,Stefan-Ursprüngliche Nachricht-Von: Christian Grün [mailto:christian.gr...@gmail.com]Gesendet: Dienstag, 17. Dezember 2019 07:59An: Stefan Koch Cc: BaseX Betreff: Re: [basex-talk] HTTPServer + gzip compressionHi Stefan,We could consider an extension of the current solution.> So only for GET - not POST.> Our current Webservices rely heavily on POST (+Response) though :(Have you checked the remaining default options? What other customsettings may you need to rely on?Cheers,Christian> -Ursprüngliche Nachricht-> Von: Stefan Koch [mailto:k...@buit-solutions.com]> Gesendet: Mittwoch, 20. November 2019 14:45> An: basex-talk@mailman.uni-konstanz.de> Betreff: Re: [basex-talk] HTTPServer + gzip compression>>> Hi,>> great to hear! So it's coming in 9.3>> We have been running base.war via tomcat with gzip instead - but it wasn't optimal for our solution.>>> Thx for fixing>> Stefan>>>> -Ursprüngliche Nachricht-> Von: Christian Grün [mailto:christian.gr...@gmail.com]> Gesendet: Mittwoch, 20. November 2019 13:26> An: Omar Siam > Cc: BaseX > Betreff: Re: [basex-talk] HTTPServer + gzip compression>>> Thanks for the confirmation!>>>> Omar Siam  schrieb am Mi., 20. Nov. 2019, 12:41:>>>> Yes, it works! 1000 entries of XML in a JSON wrapper from my data are>> now transfered as 193 KB instead of 1.6 MB. Not bad. Unfortunately on>> localhost that doesn't make much of a difference in overall timing as>> measured in Chrome's development tools. For real internet connections I>> think this is quite an improvement.>>>> Best regards>>>> Omar Siam>>>> Am 19.11.2019 um 01:28 schrieb Christian Grün:>> > The snapshot has been updated.>> > Is it working now?>> >>> >>> >>> > On Mon, Nov 18, 2019 at 6:41 PM Omar Siam  wrote:>> >> I just wanted to try gzip http compression and downloaded the current build BaseX93-20191116. The option is not there anymore. What happened?>> >>>> >> Best regards>> >>>> >> Omar Siam

Re: [basex-talk] HTTPServer + gzip compression

[Stefan Koch]

Hi,GZIP once again. It works fine with 9.3. However: http://docs.basex.org/wiki/Options#GZIP"Sane defaults" - so if you click on the link: _methods.include(HttpMethod.GET.asString());So only for GET - not POST. Our current Webservices rely heavily on POST (+Response) though :( So my question is - do you plan to change this and support POST responses with GZIP (Feature Request!) in the near future or is this not in your current pipeline?We can work with a external server und run the BaseX.war instead - but it would be cool if POST responses would get supported. Kind regards and keep up the awesome work,Stefan-Ursprüngliche Nachricht-Von: Stefan Koch [mailto:k...@buit-solutions.com]Gesendet: Mittwoch, 20. November 2019 14:45An: basex-talk@mailman.uni-konstanz.deBetreff: Re: [basex-talk] HTTPServer + gzip compressionHi,great to hear! So it's coming in 9.3 We have been running base.war via tomcat with gzip instead - but it wasn't optimal for our solution.Thx for fixingStefan -Ursprüngliche Nachricht-Von: Christian Grün [mailto:christian.gr...@gmail.com]Gesendet: Mittwoch, 20. November 2019 13:26An: Omar Siam Cc: BaseX Betreff: Re: [basex-talk] HTTPServer + gzip compressionThanks for the confirmation!  Omar Siam <omar.s...@oeaw.ac.at> schrieb am Mi., 20. Nov. 2019, 12:41:Yes, it works! 1000 entries of XML in a JSON wrapper from my data are now transfered as 193 KB instead of 1.6 MB. Not bad. Unfortunately on localhost that doesn't make much of a difference in overall timing as measured in Chrome's development tools. For real internet connections I think this is quite an improvement.Best regardsOmar SiamAm 19.11.2019 um 01:28 schrieb Christian Grün:> The snapshot has been updated.> Is it working now?>>>> On Mon, Nov 18, 2019 at 6:41 PM Omar Siam <omar.s...@oeaw.ac.at> wrote:>> I just wanted to try gzip http compression and downloaded the current build BaseX93-20191116. The option is not there anymore. What happened?>>>> Best regards>>>> Omar Siam

Re: [basex-talk] HTTPServer + gzip compression

[Stefan Koch]

Hi,great to hear! So it's coming in 9.3 We have been running base.war via tomcat with gzip instead - but it wasn't optimal for our solution.Thx for fixingStefan -Ursprüngliche Nachricht-Von: Christian Grün [mailto:christian.gr...@gmail.com]Gesendet: Mittwoch, 20. November 2019 13:26An: Omar Siam Cc: BaseX Betreff: Re: [basex-talk] HTTPServer + gzip compressionThanks for the confirmation!  Omar Siam  schrieb am Mi., 20. Nov. 2019, 12:41:Yes, it works! 1000 entries of XML in a JSON wrapper from my data are now transfered as 193 KB instead of 1.6 MB. Not bad. Unfortunately on localhost that doesn't make much of a difference in overall timing as measured in Chrome's development tools. For real internet connections I think this is quite an improvement.Best regardsOmar SiamAm 19.11.2019 um 01:28 schrieb Christian Grün:> The snapshot has been updated.> Is it working now? On Mon, Nov 18, 2019 at 6:41 PM Omar Siam  wrote:>> I just wanted to try gzip http compression and downloaded the current build BaseX93-20191116. The option is not there anymore. What happened? Best regards Omar Siam

Re: [basex-talk] HTTPServer + gzip compression

[Stefan Koch]

Hi Christian,thx for your reply. Got it  :)Solution 2 would be cool - maybe feature request? But I can live with a workaround.kind regards,Stefan -Ursprüngliche Nachricht-Von: Christian Grün [mailto:christian.gr...@gmail.com]Gesendet: Donnerstag, 3. Oktober 2019 11:51An: Stefan Koch Cc: BaseX Betreff: Re: [basex-talk] HTTPServer + gzip compressionHi Stefan,There’s a StackOverflow entry that has previously been referenced onthis list (by Michael Seiferle, see [2]). Based on this thread, inwhich Joakim Erdfelt (the magnificent Jetty core developer) explainswhy the existing approaches for enabling GZIP compression don’t workanymore, I see three choices:1. to wrap all HTTP responses in a GZIP output stream;2. to initialize GZipHandler in our basexhttp code; or3. enable GZIP compression outside BaseX,Alternative 1 would give us most control, but it would raise newquestions that would need to be solved. Alternative 2 may be thebetter approach: It only works if basexhttp is used, but we couldbenefit from existing optimizations and tweaks from the Jettyimplementation [2]. Alternative 3 is already available: You can useanother light-weight web server as proxy (caddy, nginx), or you canstart Jetty as described by Joakim.> Good idea to ask the Jetty devs - I'll try that.Thanks; feel free to keep us updated,Christian[1] https://stackoverflow.com/questions/38635262/jetty-9-and-gziphandler[2] https://www.eclipse.org/jetty/documentation/current/gzip-filter.html

Re: [basex-talk] HTTPServer + gzip compression

[Stefan Koch]

Hi Christian,Thx for your reply.I'm using the embedded Jett via basexhttp service.Not using RESTXQ - basic REST is what we need for this project.Good idea to ask the Jetty devs - I'll try that.kind regards,Stefan-Ursprüngliche Nachricht-Von: Christian Grün [mailto:christian.gr...@gmail.com]Gesendet: Dienstag, 1. Oktober 2019 14:47An: Stefan Koch Cc: BaseX Betreff: Re: [basex-talk] HTTPServer + gzip compressionHi Stefan,Do you work with an embedded Jetty instance (i.e., did you runbasexhttp), or do you use BaseX as servlet? In the latter case, thismay need to be tackled by the Jetty developers. Did you address thison their mailing list?Another alternative could be to include GZIP support in RESTXQ, andsend gzipped responses whenever the client send a correspondingAccept-Encoding header. I’ll have some more thoughts on that.Best,ChristianOn Tue, Sep 24, 2019 at 8:09 PM Stefan Koch  wrote:>> Hi,> I'm struggling to get gzip compression working.> I'm using the REST module.> http://docs.basex.org/wiki/REST> Version: BaseX 9.2.2> Tried to add the handler in jetty.xml as descriped here:> https://www.eclipse.org/jetty/documentation/current/gzip-filter.html> But it didn't work. Tried gzip filter via web.xml - but it is deprecated> since jetty 9.3.> Gzip handler is the correct way to do it.> Searched the mailling list, similar problem was reported here:> https://mailman.uni-konstanz.de/pipermail/basex-talk/2019-February/014160.html> Unfortunately no solution :(> Not much experience with embedded jetty - but adding gzip compression is> straight forward in tomcat or apache - no clue why it doesn't> work.Shouldn't be that hard - and it's kinda a default feature.> Did anybody get it to work? Any tipps?> Thanks> Stefan Koch>> B Solutions> Dipl.-Kfm. Rudolf Markus Petri> Lietzenburger Str. 77> 10719 Berlin> Tel: 0049 30 8867 6099> Fax: 0049 30 8867 6159> Mail: k...@buit-solutions.com> Web: www.buit-solutions.com

[basex-talk] HTTPServer + gzip compression

[Stefan Koch]

 Hi,
I'm struggling to get gzip compression working.
I'm using the REST module.
http://docs.basex.org/wiki/REST
Version: BaseX 9.2.2
Tried to add the handler in jetty.xml as descriped here:
https://www.eclipse.org/jetty/documentation/current/gzip-filter.html
But it didn't work. Tried gzip filter via web.xml - but it is deprecated
since jetty 9.3.
Gzip handler is the correct way to do it.
Searched the mailling list, similar problem was reported here:
https://mailman.uni-konstanz.de/pipermail/basex-talk/2019-February/014160.html
Unfortunately no solution :(
Not much experience with embedded jetty - but adding gzip compression is
straight forward in tomcat or apache - no clue why it doesn't
work.Shouldn't be that hard - and it's kinda a default feature.
Did anybody get it to work? Any tipps?
Thanks
Stefan Koch
 
B Solutions
Dipl.-Kfm. Rudolf Markus Petri
Lietzenburger Str. 77
10719 Berlin
Tel: 0049 30 8867 6099
Fax: 0049 30 8867 6159
Mail: k...@buit-solutions.com
Web: www.buit-solutions.com

[basex-talk] debian release for BaseX 8.2.3

[Stefan Koch]

Title: Message
Hello,when will the debian package for 8.2.3 be released (currently only is available)Thx,Stefan

[basex-talk] Debian/Ubuntu package

[Stefan Koch]

Dear Basex Team,

any plans when the debian package for version 7.7.1 will be released?

Kind regards,

Stefan


___
BaseX-Talk mailing list
BaseX-Talk@mailman.uni-konstanz.de
https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

[basex-talk] db:list via basexclients doesn't show all databases

[Stefan Koch]

Hi,

I was wondering why our backupskript isn't exporting all databases, one 
is missing.
The I found out why - our  backuptool connects via Java and uses db:list 
to get the names of alle databases.


If I connect via basexclient -  there is one missing, if I check via 
basex (from the shell) the database is shown.

I added the database also via basex and not like the others via the client.

Is this intended behaviour? How can I tell the client, that there is one 
more database?


Regards,

Stefan
___
BaseX-Talk mailing list
BaseX-Talk@mailman.uni-konstanz.de
https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk