Re: [basex-talk] Stopping the http server

[France Baril]

I'm not sure I understand how to do that. If the call to baseX.jar start
has everything embedded together, how would I split it up to control each
part?

I'm a programmer, not a sysadmin, so it might be a basic question, but I'm
not finding anything in the doc.

[basex-talk] How to restrict access to restxq functions just for the same domain

[Ioan Fericel]

Hi,

I seemed that this is the behavior if disabling REST Service in web.xml 
(BaseX.war deploymented on Tomcat), but this does not happen. The 
surprise is that the functions are accessible from anywhere, if access 
is not protected by other methods.


I'm doing wrong, what should be setting? I use BaseX 8.3.

Thanks for any guidance!
Ioan

[basex-talk] Basex on maven central

[Lance Java]

It seems that maven central is not up to date with the latest basex
version. I see here that there's only version 7.3.1 whereas on the basex
website the latest version is 8.4.4

http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.basex%22

Are there any plans to keep maven central up to date with basex releases?

Re: [basex-talk] Xquery recursion and db:add() - stack overflow

[Lars Johnsen]

Thanks for pointer!

Code is rewritten using hof:until() and tested towards a particular set at
our national provider of library data.

The script still accumulates data, so it will probably still run into
memory troubles with larger datasets, but the stack-overflow should be
taken care of.

For anyone interested, the code is attached below, and using hof:until() as
the higher order function. To make it work, fill in URLs for a choosen
OAI-endpoint, and maybe change som of the request parameters - this one
fetches marc21 posts and uses sets. Some error checking may also be
implemented.

Cheers,
Lars


declare namespace oai = "http://www.openarchives.org/OAI/2.0/;;

(:URL for  resumption tokens :)
declare variable $URL := "oai-URL?verb=ListRecordsresumptionToken=";

(:URL for initial request:)
declare variable $URL2 :=
"oai-URL?verb=ListRecordsmetadataPrefix=marc21set=";

(: Variable for OAI-set - if not used, remove "set=" in URL2 :)
declare variable $oai-set := "aset";


(: basex http :)
declare variable $http-option := ;


(: --

Fetch data from OAI-endpoint using a start map containing resumption token
and the first set of data.
The map has two keys, 'resume' and 'chunk', where 'chunk' is an accumulator
holding data from the current and previous requests.
hof:until() does not return an aggregated list of maps, so data must be
collected somehow

--:)

declare function local:getResumption($startmap) {

  let $token := map:get($startmap, 'resume')
  return if (empty($token)) then
$startmap
  else
let $http-request := http:send-request($http-option, $URL || $token)
let $result := if ($http-request instance of node()) then
$http-request
  else
{$http-request}
return  map {
  'resume':  $result//oai:resumptionToken/text(),
  'chunk': (
map:get($startmap, 'chunk'),
$result//oai:metadata
  )
}
 };


(: Issue initial request :)

let $first := http:send-request($http-option, $URL2 || $oai-set)

(:  Create startmap :)

let $init := map {
  'chunk': $first//oai:metadata,
  'resume': $first//oai:resumptionToken/text()
}

let $oai :=  hof:until(

  function($x) {
empty(map:get($x, 'resume'))
  },

  function($y) {
local:getResumption($y)
 },
 $init
)

(: Amend with additional code like db:add() of file:write() here :)

return element oai {map:get($oai, 'chunk')}


2016-05-12 15:07 GMT+02:00 Dirk Kirsten :

> Hello Lars,
>
> just a thought (and really just a pointer, I am neither a purely
> functional guy and also I feel like I am missing something obious...):
> Maybe you could rewrite the recursive approach using higher order
> functions. Consider a query like the following
>
> hof:scan-left(1 to 100,
>   map { "token": "starttoken" },
>   function($result, $index) {
> let $req := http:send-request(,
> "http://google.com?q=;  || $result("token"))
> return map {
>   "result": $req,
>   "token" : $req//http:header[@name = "Date"]/@value/data()
> }
> })
>
> It will issue 100 requests to google and use some specific token from the
> query before (in this case I used the date). This will output a sequence of
> the map entries and in a subsequent step you could return only the actual
> result values.
>
> Best regards, Dirk
>
> On 05/12/2016 12:55 PM, Lars Johnsen wrote:
>
> Thanks Johan and Matti for useful suggestions.
>
> Cutting down on the chunks seems to be a viable alternative.
>
> It would have been nice, though,  to have a robust harvester in XQuery
> that could take on anything, although the recursive version works fine as
> long as the dataset consist of a couple of  thousand entries.
>
> Best,
> Lars
>
> 2016-05-12 8:16 GMT+02:00 Lassila, Matti :
>
>> Hello,
>>
>> If your case allows using external tools for harvesting, I can highly
>> recommend metha (https://github.com/miku/metha) which is a fairly full
>> featured command line OAI-PMH harvester.
>>
>> Best regards,
>>
>> Matti L.
>>
>> On 11/05/16 18:31 , "basex-talk-boun...@mailman.uni-konstanz.de on behalf
>> of Johan Mörén" > johan.mo...@gmail.com> wrote:
>>
>> >Maybe there is some other way to get the data over. I'll have a talk with
>> >the guys providing the OAI-endpoint.
>>
>>
>
> --
> Dirk Kirsten, BaseX GmbH, http://basexgmbh.de
> |-- Firmensitz: Blarerstrasse 56, 78462 Konstanz
> |-- Registergericht Freiburg, HRB: 708285, Geschäftsführer:
> |   Dr. Christian Grün, Dr. Alexander Holupirek, Michael Seiferle
> `-- Phone: 0049 7531 91 68 276, Fax: 0049 7531 20 05 22
>
>

Re: [basex-talk] Xquery recursion and db:add() - stack overflow

[Dirk Kirsten]

Hello Lars,

just a thought (and really just a pointer, I am neither a purely
functional guy and also I feel like I am missing something obious...):
Maybe you could rewrite the recursive approach using higher order
functions. Consider a query like the following

hof:scan-left(1 to 100,
  map { "token": "starttoken" },
  function($result, $index) {
let $req := http:send-request(,
"http://google.com?q=; || $result("token"))
return map {
  "result": $req,
  "token" : $req//http:header[@name = "Date"]/@value/data()
}
})

It will issue 100 requests to google and use some specific token from
the query before (in this case I used the date). This will output a
sequence of the map entries and in a subsequent step you could return
only the actual result values.

Best regards, Dirk


On 05/12/2016 12:55 PM, Lars Johnsen wrote:
> Thanks Johan and Matti for useful suggestions. 
>
> Cutting down on the chunks seems to be a viable alternative. 
>
> It would have been nice, though,  to have a robust harvester in XQuery
> that could take on anything, although the recursive version works fine
> as long as the dataset consist of a couple of  thousand entries.
>
> Best,
> Lars
>
> 2016-05-12 8:16 GMT+02:00 Lassila, Matti  >:
>
> Hello,
>
> If your case allows using external tools for harvesting, I can highly
> recommend metha (https://github.com/miku/metha) which is a fairly full
> featured command line OAI-PMH harvester.
>
> Best regards,
>
> Matti L.
>
> On 11/05/16 18:31 , "basex-talk-boun...@mailman.uni-konstanz.de
>  on behalf
> of Johan Mörén"   on behalf of
> johan.mo...@gmail.com > wrote:
>
> >Maybe there is some other way to get the data over. I'll have a
> talk with
> >the guys providing the OAI-endpoint.
>
>

-- 
Dirk Kirsten, BaseX GmbH, http://basexgmbh.de
|-- Firmensitz: Blarerstrasse 56, 78462 Konstanz
|-- Registergericht Freiburg, HRB: 708285, Geschäftsführer:
|   Dr. Christian Grün, Dr. Alexander Holupirek, Michael Seiferle
`-- Phone: 0049 7531 91 68 276, Fax: 0049 7531 20 05 22

Re: [basex-talk] Xquery recursion and db:add() - stack overflow

[Lars Johnsen]

Thanks Johan and Matti for useful suggestions.

Cutting down on the chunks seems to be a viable alternative.

It would have been nice, though,  to have a robust harvester in XQuery that
could take on anything, although the recursive version works fine as long
as the dataset consist of a couple of  thousand entries.

Best,
Lars

2016-05-12 8:16 GMT+02:00 Lassila, Matti :

> Hello,
>
> If your case allows using external tools for harvesting, I can highly
> recommend metha (https://github.com/miku/metha) which is a fairly full
> featured command line OAI-PMH harvester.
>
> Best regards,
>
> Matti L.
>
> On 11/05/16 18:31 , "basex-talk-boun...@mailman.uni-konstanz.de on behalf
> of Johan Mörén"  johan.mo...@gmail.com> wrote:
>
> >Maybe there is some other way to get the data over. I'll have a talk with
> >the guys providing the OAI-endpoint.
>
>

Re: [basex-talk] Xquery recursion and db:add() - stack overflow

[Lassila, Matti]

Hello,

If your case allows using external tools for harvesting, I can highly
recommend metha (https://github.com/miku/metha) which is a fairly full
featured command line OAI-PMH harvester.

Best regards,

Matti L.

On 11/05/16 18:31 , "basex-talk-boun...@mailman.uni-konstanz.de on behalf
of Johan Mörén"  wrote:

>Maybe there is some other way to get the data over. I'll have a talk with
>the guys providing the OAI-endpoint.