date:20160622

[MarkLogic Dev General] Speeding up access to large XML file

2016-06-22 Thread Hans Hübner

Hi,

I have a beginner's question regarding indexing:  As far as I understand,
MarkLogic indexes all data according to global indexing rules.  Thus, in
general, all accesses to data should be "fast" when the universal index can
be used.

I am trying to combine data from two XML files into a report, and
throughput is not sufficient.  I'm now looking for ways to improve the
performance of my XQuery, either by adding indexes to MarkLogic or by
improving the query so that it runs faster.  Here's what I have:

for $key in doc("file1.xml")/a/b/c/d/e/f/g/h/i[j = 123]/k/text()
let $data-from-file-2 := doc("file2.xml")/a/b[c/d = $key]/e/f/text()
return
  
   $key => $data-from-file-2
  

Thus, I'm iterating over some subset of the nodes in file1.xml, selecting
data from file2.xml for each of the nodes selected.  Both file1.xml and
file2.xml are in a files residing in MarkLogic.  They have a pre-determined
format and contain roughly 100,000 elements matching each of the two paths.

It seems that the performance of the second of the two XPaths
(doc("file2.xml")/a/b[c/d = $key]/e/f/text()) is most important.  Ideally,
I would like this lookup to complete in under a millisecond, but it seems
to need 75ms right now.

Any help speeding this up would be greatly appreciated!

Thanks,
Hans

-- 
LambdaWerk GmbH
Oranienburger Straße 87/89
10178 Berlin
Phone: +49 30 555 7335 0
Fax: +49 30 555 7335 99

HRB 169991 B Amtsgericht Charlottenburg
USt-ID: DE301399951
Geschäftsführer:  Hans Hübner

http://lambdawerk.com/
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] MLCP import transform and search

2016-06-22 Thread Aries Li

Hi Frank,
Glad to hear it works now.
Yeah, by default the transform module only query against the database 
associated with the XDBC server MLCP is talking to.

To query other database, you can try passing in the 5th parameter $forest-ids 
to cts:search.
$forest-ids can be fetched by calling 
xdmp:database-forests(xdmp:database("your_database"))

Hope this helps.

Aries
From: general-boun...@developer.marklogic.com 
[mailto:general-boun...@developer.marklogic.com] On Behalf Of 
fhcj.jan...@nl.abnamro.com
Sent: Wednesday, June 22, 2016 1:51 PM
To: MarkLogic Developer Discussion
Subject: Re: [MarkLogic Dev General] MLCP import transform and search

Hi Aries,

Thanks it is working now.
Only when I want to get documents from a sub-database it's still empty.


Hi Rob,

We are orienting.
But if you load every day the same document with bitemporal, after 7 day there 
are 7 till 13 documents. In which only the bitemporal dateTime elements are 
different.
We want to reduce this to 1 document.

This because one of the project to use ML is to store billions of documents.
1 of the options is to check if the current document and the document from the 
day before are the same.
If so, we can delete the current document, before loading.




From:Rob Szkutak 
>
To:MarkLogic Developer Discussion 
>
Date:22-06-2016 19:51
Subject:Re: [MarkLogic Dev General] MLCP import transform and search
Sent by:
general-boun...@developer.marklogic.com




Hi,

To answer your question directly, yes, you can run a search within an MLCP 
transform.

Regarding your specific transform, how big is the "docs" collection? If you're 
trying to paste a large number of other documents and their URIs into the 
incoming document you're probably going to run into some pretty big problems.

This leads me to ask, why are you trying to paste an entire collection of 
documents and URIs into an incoming document in the first place? There's 
probably a better way to do this.

Best,
Rob

Rob Szkutak
Senior Consultant
MarkLogic Corporation
rob.szku...@marklogic.com
www.marklogic.com


From: 
general-boun...@developer.marklogic.com
 [general-boun...@developer.marklogic.com] on behalf of Aries Li 
[aries...@marklogic.com]
Sent: Wednesday, June 22, 2016 12:45 PM
To: MarkLogic Developer Discussion
Subject: Re: [MarkLogic Dev General] MLCP import transform and search

Hi Frank,
If you login QConsole with $USER that was passed into mlcp, can you get result 
by running the cts query in the transform module?
I expect mlcp transform module can run the query with privileges from $USER.

Aries
From: 
general-boun...@developer.marklogic.com
 [mailto:general-boun...@developer.marklogic.com] On Behalf Of 
fhcj.jan...@nl.abnamro.com
Sent: Wednesday, June 22, 2016 8:48 AM
To: general@developer.marklogic.com
Subject: [MarkLogic Dev General] MLCP import transform and search

Hi,

Is it possible in MLCP IMPORT within a TRANSFORM  to search on the (same) 
database?

To enrich the data (from same database) I use the transform option.
But in below transformation the raw data is loaded in the database with empty 
elements a,b and c (structure is ok)
Questions:
- is it possible to do a MLCP IMPORT with a search in a transform (xquery) and 
get data from same (sub-)database
- or do I have to adjust my security role (Which role do I need, currently I 
use role admin )
- or  .


MLCP call:

mlcp.sh import \
  -mode local \
  -host localhost \
  -port $PORT \
  -username $USER \
  -password $PASS \
  -input_file_path /home/cent/Desktop/$SRCS \
  -document_type xml  \
  -input_file_type aggregates \
  -aggregate_record_element record \
  -database Dummy \
  -content_encoding "UTF-8" \
  -namespace http://ml.com/$SRCS \
  -output_collections $COLL \
  -transform_module /src/transform/transform.xqy \
  -transform_namespace http://ml.com/mlcp-transform \
  -transform_param $PARM \
  -tolerate_errors true \
  -xml_repair_level full




/src/transform/transform.xqy:

xquery version "1.0-ml";
module namespace ingest = "http://ml.com/mlcp-transform;;
declare function ingest:transform(
 $content as map:map,
 $context as map:map
)
{
 let $doc := map:get($content, "value")

(: Normally the below search function are more specific based on element values 
in $doc :)

 let $a := cts:search(fn:doc(), cts:collection-query("docs"), "unfiltered")
 let $b := cts:uris((),(),cts:collection-query("docs"))
 let $c

Re: [MarkLogic Dev General] MLCP import transform and search

2016-06-22 Thread fhcj . jansen

Hi Aries,

Thanks it is working now. 
Only when I want to get documents from a sub-database it's still empty.


Hi Rob,

We are orienting. 
But if you load every day the same document with bitemporal, after 7 day 
there are 7 till 13 documents. In which only the bitemporal dateTime 
elements are different.
We want to reduce this to 1 document. 

This because one of the project to use ML is to store billions of 
documents.
1 of the options is to check if the current document and the document from 
the day before are the same. 
If so, we can delete the current document, before loading.
 



From:   Rob Szkutak 
To: MarkLogic Developer Discussion 
Date:   22-06-2016 19:51
Subject:Re: [MarkLogic Dev General] MLCP import transform and 
search
Sent by:general-boun...@developer.marklogic.com



Hi, 

To answer your question directly, yes, you can run a search within an MLCP 
transform.

Regarding your specific transform, how big is the "docs" collection? If 
you're trying to paste a large number of other documents and their URIs 
into the incoming document you're probably going to run into some pretty 
big problems.

This leads me to ask, why are you trying to paste an entire collection of 
documents and URIs into an incoming document in the first place? There's 
probably a better way to do this.

Best,
Rob

Rob Szkutak 
Senior Consultant
MarkLogic Corporation
rob.szku...@marklogic.com
www.marklogic.com


From: general-boun...@developer.marklogic.com 
[general-boun...@developer.marklogic.com] on behalf of Aries Li 
[aries...@marklogic.com]
Sent: Wednesday, June 22, 2016 12:45 PM
To: MarkLogic Developer Discussion
Subject: Re: [MarkLogic Dev General] MLCP import transform and search

Hi Frank,
If you login QConsole with $USER that was passed into mlcp, can you get 
result by running the cts query in the transform module?
I expect mlcp transform module can run the query with privileges from 
$USER.
 
Aries
From: general-boun...@developer.marklogic.com [
mailto:general-boun...@developer.marklogic.com] On Behalf Of 
fhcj.jan...@nl.abnamro.com
Sent: Wednesday, June 22, 2016 8:48 AM
To: general@developer.marklogic.com
Subject: [MarkLogic Dev General] MLCP import transform and search
 
Hi, 

Is it possible in MLCP IMPORT within a TRANSFORM  to search on the (same) 
database? 

To enrich the data (from same database) I use the transform option. 
But in below transformation the raw data is loaded in the database with 
empty elements a,b and c (structure is ok) 
Questions: 
- is it possible to do a MLCP IMPORT with a search in a transform (xquery) 
and get data from same (sub-)database 
- or do I have to adjust my security role (Which role do I need, currently 
I use role admin ) 
- or  . 


MLCP call: 

mlcp.sh import \ 
   -mode local \ 
   -host localhost \ 
   -port $PORT \ 
   -username $USER \ 
   -password $PASS \ 
   -input_file_path /home/cent/Desktop/$SRCS \ 
   -document_type xml  \ 
   -input_file_type aggregates \ 
   -aggregate_record_element record \ 
   -database Dummy \ 
   -content_encoding "UTF-8" \ 
   -namespace http://ml.com/$SRCS \ 
   -output_collections $COLL \ 
   -transform_module /src/transform/transform.xqy \ 
   -transform_namespace http://ml.com/mlcp-transform \ 
   -transform_param $PARM \ 
   -tolerate_errors true \ 
   -xml_repair_level full 




/src/transform/transform.xqy: 

xquery version "1.0-ml"; 
module namespace ingest = "http://ml.com/mlcp-transform;; 
declare function ingest:transform( 
  $content as map:map, 
  $context as map:map 
) 
{ 
  let $doc := map:get($content, "value") 

(: Normally the below search function are more specific based on element 
values in $doc :) 
 
  let $a := cts:search(fn:doc(), cts:collection-query("docs"), 
"unfiltered") 
  let $b := cts:uris((),(),cts:collection-query("docs")) 
  let $c := fn:collection("docs")[1] 
(: removed transformation part   :) 
(:   but $a, $b and $c are empty :) 
(:   Same function in QControl gives results :)   

  let $_1 :=  
  {$doc} 
  {$a} 
  {$b} 
  {$c} 

  let $_2 := map:put($content, "value", $_1) 
  return $content 
}; 



Kind Regards, 
Frank Jansen 

This message has been sent by ABN AMRO Bank N.V., which has its seat at 
Gustav Mahlerlaan 10 (1082 PP) Amsterdam, the Netherlands, and is 
registered in the Commercial Register of Amsterdam under number 34334259.
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general





This message has been sent by ABN AMRO Bank N.V., which has its seat at Gustav 
Mahlerlaan 10 (1082 PP) Amsterdam, the Netherlands, and is registered in the 
Commercial

Re: [MarkLogic Dev General] MLCP import transform and search

2016-06-22 Thread Rob Szkutak

Hi,

To answer your question directly, yes, you can run a search within an MLCP 
transform.

Regarding your specific transform, how big is the "docs" collection? If you're 
trying to paste a large number of other documents and their URIs into the 
incoming document you're probably going to run into some pretty big problems.

This leads me to ask, why are you trying to paste an entire collection of 
documents and URIs into an incoming document in the first place? There's 
probably a better way to do this.

Best,
Rob

Rob Szkutak
Senior Consultant
MarkLogic Corporation
rob.szku...@marklogic.com
www.marklogic.com


From: general-boun...@developer.marklogic.com 
[general-boun...@developer.marklogic.com] on behalf of Aries Li 
[aries...@marklogic.com]
Sent: Wednesday, June 22, 2016 12:45 PM
To: MarkLogic Developer Discussion
Subject: Re: [MarkLogic Dev General] MLCP import transform and search

Hi Frank,
If you login QConsole with $USER that was passed into mlcp, can you get result 
by running the cts query in the transform module?
I expect mlcp transform module can run the query with privileges from $USER.

Aries
From: general-boun...@developer.marklogic.com 
[mailto:general-boun...@developer.marklogic.com] On Behalf Of 
fhcj.jan...@nl.abnamro.com
Sent: Wednesday, June 22, 2016 8:48 AM
To: general@developer.marklogic.com
Subject: [MarkLogic Dev General] MLCP import transform and search

Hi,

Is it possible in MLCP IMPORT within a TRANSFORM  to search on the (same) 
database?

To enrich the data (from same database) I use the transform option.
But in below transformation the raw data is loaded in the database with empty 
elements a,b and c (structure is ok)
Questions:
- is it possible to do a MLCP IMPORT with a search in a transform (xquery) and 
get data from same (sub-)database
- or do I have to adjust my security role (Which role do I need, currently I 
use role admin )
- or  .


MLCP call:

mlcp.sh import \
   -mode local \
   -host localhost \
   -port $PORT \
   -username $USER \
   -password $PASS \
   -input_file_path /home/cent/Desktop/$SRCS \
   -document_type xml  \
   -input_file_type aggregates \
   -aggregate_record_element record \
   -database Dummy \
   -content_encoding "UTF-8" \
   -namespace http://ml.com/$SRCS \
   -output_collections $COLL \
   -transform_module /src/transform/transform.xqy \
   -transform_namespace http://ml.com/mlcp-transform \
   -transform_param $PARM \
   -tolerate_errors true \
   -xml_repair_level full




/src/transform/transform.xqy:

xquery version "1.0-ml";
module namespace ingest = "http://ml.com/mlcp-transform;;
declare function ingest:transform(
  $content as map:map,
  $context as map:map
)
{
  let $doc := map:get($content, "value")

(: Normally the below search function are more specific based on element values 
in $doc :)

  let $a := cts:search(fn:doc(), cts:collection-query("docs"), "unfiltered")
  let $b := cts:uris((),(),cts:collection-query("docs"))
  let $c := fn:collection("docs")[1]
(: removed transformation part   :)
(:   but $a, $b and $c are empty :)
(:   Same function in QControl gives results :)

  let $_1 := 
  {$doc}
  {$a}
  {$b}
  {$c}
   
  let $_2 := map:put($content, "value", $_1)
  return $content
};



Kind Regards,
Frank Jansen

This message has been sent by ABN AMRO Bank N.V., which has its seat at Gustav 
Mahlerlaan 10 (1082 PP) Amsterdam, the Netherlands, and is registered in the 
Commercial Register of Amsterdam under number 34334259.
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] MLCP import transform and search

2016-06-22 Thread Aries Li

Hi Frank,
If you login QConsole with $USER that was passed into mlcp, can you get result 
by running the cts query in the transform module?
I expect mlcp transform module can run the query with privileges from $USER.

Aries
From: general-boun...@developer.marklogic.com 
[mailto:general-boun...@developer.marklogic.com] On Behalf Of 
fhcj.jan...@nl.abnamro.com
Sent: Wednesday, June 22, 2016 8:48 AM
To: general@developer.marklogic.com
Subject: [MarkLogic Dev General] MLCP import transform and search

Hi,

Is it possible in MLCP IMPORT within a TRANSFORM  to search on the (same) 
database?

To enrich the data (from same database) I use the transform option.
But in below transformation the raw data is loaded in the database with empty 
elements a,b and c (structure is ok)
Questions:
- is it possible to do a MLCP IMPORT with a search in a transform (xquery) and 
get data from same (sub-)database
- or do I have to adjust my security role (Which role do I need, currently I 
use role admin )
- or  .


MLCP call:

mlcp.sh import \
   -mode local \
   -host localhost \
   -port $PORT \
   -username $USER \
   -password $PASS \
   -input_file_path /home/cent/Desktop/$SRCS \
   -document_type xml  \
   -input_file_type aggregates \
   -aggregate_record_element record \
   -database Dummy \
   -content_encoding "UTF-8" \
   -namespace http://ml.com/$SRCS \
   -output_collections $COLL \
   -transform_module /src/transform/transform.xqy \
   -transform_namespace http://ml.com/mlcp-transform \
   -transform_param $PARM \
   -tolerate_errors true \
   -xml_repair_level full




/src/transform/transform.xqy:

xquery version "1.0-ml";
module namespace ingest = "http://ml.com/mlcp-transform;;
declare function ingest:transform(
  $content as map:map,
  $context as map:map
)
{
  let $doc := map:get($content, "value")

(: Normally the below search function are more specific based on element values 
in $doc :)

  let $a := cts:search(fn:doc(), cts:collection-query("docs"), "unfiltered")
  let $b := cts:uris((),(),cts:collection-query("docs"))
  let $c := fn:collection("docs")[1]
(: removed transformation part   :)
(:   but $a, $b and $c are empty :)
(:   Same function in QControl gives results :)

  let $_1 := 
  {$doc}
  {$a}
  {$b}
  {$c}
   
  let $_2 := map:put($content, "value", $_1)
  return $content
};



Kind Regards,
Frank Jansen

This message has been sent by ABN AMRO Bank N.V., which has its seat at Gustav 
Mahlerlaan 10 (1082 PP) Amsterdam, the Netherlands, and is registered in the 
Commercial Register of Amsterdam under number 34334259.
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

[MarkLogic Dev General] MLCP import transform and search

2016-06-22 Thread fhcj . jansen

Hi,

Is it possible in MLCP IMPORT within a TRANSFORM  to search on the (same) 
database?

To enrich the data (from same database) I use the transform option.
But in below transformation the raw data is loaded in the database with 
empty elements a,b and c (structure is ok)
Questions:
- is it possible to do a MLCP IMPORT with a search in a transform (xquery) 
and get data from same (sub-)database
- or do I have to adjust my security role (Which role do I need, currently 
I use role admin ) 
- or  .


MLCP call:

mlcp.sh import \
   -mode local \
   -host localhost \
   -port $PORT \
   -username $USER \
   -password $PASS \
   -input_file_path /home/cent/Desktop/$SRCS \
   -document_type xml  \
   -input_file_type aggregates \
   -aggregate_record_element record \
   -database Dummy \
   -content_encoding "UTF-8" \
   -namespace http://ml.com/$SRCS \
   -output_collections $COLL \
   -transform_module /src/transform/transform.xqy \
   -transform_namespace http://ml.com/mlcp-transform \
   -transform_param $PARM \
   -tolerate_errors true \
   -xml_repair_level full




/src/transform/transform.xqy:

xquery version "1.0-ml";
module namespace ingest = "http://ml.com/mlcp-transform;;
declare function ingest:transform(
  $content as map:map,
  $context as map:map
) 
{
  let $doc := map:get($content, "value")

(: Normally the below search function are more specific based on element 
values in $doc :) 
 
  let $a := cts:search(fn:doc(), cts:collection-query("docs"), 
"unfiltered")
  let $b := cts:uris((),(),cts:collection-query("docs"))
  let $c := fn:collection("docs")[1]
(: removed transformation part   :)
(:   but $a, $b and $c are empty :)
(:   Same function in QControl gives results :) 

  let $_1 := 
  {$doc}
  {$a}
  {$b}
  {$c}
   
  let $_2 := map:put($content, "value", $_1)
  return $content
};



Kind Regards, 
Frank Jansen 



This message has been sent by ABN AMRO Bank N.V., which has its seat at Gustav 
Mahlerlaan 10 (1082 PP) Amsterdam, the Netherlands, and is registered in the 
Commercial Register of Amsterdam under number 34334259.___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Validation error when loading pipeline with options via Management API

2016-06-22 Thread Rob Walpole

Hi Indy,

Yes I think it's some kind of namespace issue - however we have now found
another solution as I realised we don't need file system access to load the
pipeline - it can just as easily be loaded from the modules database which
we are populating using mlcp.

Thanks again
Rob

On 22 June 2016 at 10:28, Indrajeet Verma  wrote:

> Hi Rob,
>
> I have never tried loading pipeline via APIs however by looking into the
> error, it seems a namespace error as you can see pipeline is in different
> namespace and options is with different namespace as per docs.
>
> If possible, could you share your code and I may try to look. If not I am
> sure some other ML experts can help you on this.
>
> Regards,
> Indy
> On 22-Jun-2016 2:44 pm, "Rob Walpole"  wrote:
>
>> Hi Indy,
>>
>> Thanks for your reply. Yes we have tried loading the pipeline via the
>> admin UI and it works fine using the format you describe - however we
>> really want to use the management API as the admin UI requires file system
>> access on the server where MarkLogic and this will not always be available
>> to us.
>>
>> Rob
>>
>>
>> On 22 June 2016 at 10:05, Indrajeet Verma 
>> wrote:
>>
>>> Hi Rob,
>>>
>>> Have you tried to load pipeline via admin UI and seeing same error?
>>>
>>> I am using ML8.0-4 and below code works with the options however I have
>>> loaded pipeline via UI.
>>>
>>> 
>>> 
>>> When a document is zip,
>>> 
>>> http://marklogic.com/states/initial
>>> http://marklogic.com/states/transformed
>>> http://marklogic.com/states/error
>>> 
>>> 
>>>
>>> /MarkLogic/cpf/actions/mimetype-condition.xqy
>>> >> xmlns="/MarkLogic/cpf/actions/mimetype-condition.xqy">
>>> application/zip
>>> 
>>> 
>>> 
>>> action/extract-zip.xqy
>>> 
>>> 
>>> 
>>>
>>> Regards,
>>> Indy
>>>
>>> On Wed, Jun 22, 2016 at 2:11 PM, Rob Walpole 
>>> wrote:
>>>
 Hi,

 We are trying to load a CPF pipeline via the RESTful Management API and
 this fails with an invalid node error when there is an options element
 present within /pipeline-properties/state-transition/execute/condition

 The full error message is as follows:

 http://marklogic.com/xdmp/error;>
   400
   Bad Request
   MANAGE-INVALIDPAYLOAD
   MANAGE-INVALIDPAYLOAD: (err:FOER) Payload has errors in
 structure, content-type or values. XDMP-VALIDATEUNEXPECTED: (err:XQDY0027)
 validate strict { $pipeline } -- Invalid node: Found pp:options but
 expected any(lax,!(http://marklogic.com/manage/pipeline/properties))?
 at
 fn:doc("")/pp:pipeline-properties/pp:state-transition/pp:execute/pp:condition/pp:options
 using schema "manage-pipeline-properties.xsd"

 The problem node looks like this:

 /MarkLogic/cpf/actions/mimetype-condition.xqy

 application/xml

 We have also tried putting the options element in as..

 application/xml

 ..which is how it is described in the Content Processing Framework
 Guide but this fails with a different error as follows:

 http://marklogic.com/xdmp/error;>
   500
   Internal Server Error
   XDMP-VALIDATEUNEXPECTED
   XDMP-VALIDATEUNEXPECTED: (err:XQDY0027) validate strict {
 $pipeline } -- Invalid node: Found p:options but expected any(lax,!(
 http://marklogic.com/cpf/pipelines))? at
 /p:pipeline/p:state-transition/p:execute/p:condition/p:options using schema
 "pipelines.xsd"

 If we remove the options element the pipeline loads via the API with no
 problems.

 We are using MarkLogic Server Enterprise Edition 8.0-4.2

 Many thanks
 Rob Walpole

 ___
 General mailing list
 General@developer.marklogic.com
 Manage your subscription at:
 http://developer.marklogic.com/mailman/listinfo/general

>>>
>>> ___
>>> General mailing list
>>> General@developer.marklogic.com
>>> Manage your subscription at:
>>> http://developer.marklogic.com/mailman/listinfo/general
>>>
>>>
>>
>> ___
>> General mailing list
>> General@developer.marklogic.com
>> Manage your subscription at:
>> http://developer.marklogic.com/mailman/listinfo/general
>>
>>
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at:
> http://developer.marklogic.com/mailman/listinfo/general
>
>
___
General mailing list
General@developer.marklogic.com
Manage your subscription at:

Re: [MarkLogic Dev General] Validation error when loading pipeline with options via Management API

2016-06-22 Thread Indrajeet Verma

Hi Rob,

I have never tried loading pipeline via APIs however by looking into the
error, it seems a namespace error as you can see pipeline is in different
namespace and options is with different namespace as per docs.

If possible, could you share your code and I may try to look. If not I am
sure some other ML experts can help you on this.

Regards,
Indy
On 22-Jun-2016 2:44 pm, "Rob Walpole"  wrote:

> Hi Indy,
>
> Thanks for your reply. Yes we have tried loading the pipeline via the
> admin UI and it works fine using the format you describe - however we
> really want to use the management API as the admin UI requires file system
> access on the server where MarkLogic and this will not always be available
> to us.
>
> Rob
>
>
> On 22 June 2016 at 10:05, Indrajeet Verma 
> wrote:
>
>> Hi Rob,
>>
>> Have you tried to load pipeline via admin UI and seeing same error?
>>
>> I am using ML8.0-4 and below code works with the options however I have
>> loaded pipeline via UI.
>>
>> 
>> 
>> When a document is zip,
>> 
>> http://marklogic.com/states/initial
>> http://marklogic.com/states/transformed
>> http://marklogic.com/states/error
>> 
>> 
>>
>> /MarkLogic/cpf/actions/mimetype-condition.xqy
>> > xmlns="/MarkLogic/cpf/actions/mimetype-condition.xqy">
>> application/zip
>> 
>> 
>> 
>> action/extract-zip.xqy
>> 
>> 
>> 
>>
>> Regards,
>> Indy
>>
>> On Wed, Jun 22, 2016 at 2:11 PM, Rob Walpole 
>> wrote:
>>
>>> Hi,
>>>
>>> We are trying to load a CPF pipeline via the RESTful Management API and
>>> this fails with an invalid node error when there is an options element
>>> present within /pipeline-properties/state-transition/execute/condition
>>>
>>> The full error message is as follows:
>>>
>>> http://marklogic.com/xdmp/error;>
>>>   400
>>>   Bad Request
>>>   MANAGE-INVALIDPAYLOAD
>>>   MANAGE-INVALIDPAYLOAD: (err:FOER) Payload has errors in
>>> structure, content-type or values. XDMP-VALIDATEUNEXPECTED: (err:XQDY0027)
>>> validate strict { $pipeline } -- Invalid node: Found pp:options but
>>> expected any(lax,!(http://marklogic.com/manage/pipeline/properties))?
>>> at
>>> fn:doc("")/pp:pipeline-properties/pp:state-transition/pp:execute/pp:condition/pp:options
>>> using schema "manage-pipeline-properties.xsd"
>>> 
>>>
>>> The problem node looks like this:
>>>
>>> 
>>> /MarkLogic/cpf/actions/mimetype-condition.xqy
>>> 
>>> application/xml
>>> 
>>> 
>>>
>>> We have also tried putting the options element in as..
>>>
>>> 
>>> application/xml
>>> 
>>>
>>> ..which is how it is described in the Content Processing Framework Guide
>>> but this fails with a different error as follows:
>>>
>>> http://marklogic.com/xdmp/error;>
>>>   500
>>>   Internal Server Error
>>>   XDMP-VALIDATEUNEXPECTED
>>>   XDMP-VALIDATEUNEXPECTED: (err:XQDY0027) validate strict {
>>> $pipeline } -- Invalid node: Found p:options but expected any(lax,!(
>>> http://marklogic.com/cpf/pipelines))? at
>>> /p:pipeline/p:state-transition/p:execute/p:condition/p:options using schema
>>> "pipelines.xsd"
>>> 
>>>
>>> If we remove the options element the pipeline loads via the API with no
>>> problems.
>>>
>>> We are using MarkLogic Server Enterprise Edition 8.0-4.2
>>>
>>> Many thanks
>>> Rob Walpole
>>>
>>> ___
>>> General mailing list
>>> General@developer.marklogic.com
>>> Manage your subscription at:
>>> http://developer.marklogic.com/mailman/listinfo/general
>>>
>>>
>>
>> ___
>> General mailing list
>> General@developer.marklogic.com
>> Manage your subscription at:
>> http://developer.marklogic.com/mailman/listinfo/general
>>
>>
>
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at:
> http://developer.marklogic.com/mailman/listinfo/general
>
>
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Validation error when loading pipeline with options via Management API

2016-06-22 Thread Rob Walpole

Hi Indy,

Thanks for your reply. Yes we have tried loading the pipeline via the admin
UI and it works fine using the format you describe - however we really want
to use the management API as the admin UI requires file system access on
the server where MarkLogic and this will not always be available to us.

Rob


On 22 June 2016 at 10:05, Indrajeet Verma  wrote:

> Hi Rob,
>
> Have you tried to load pipeline via admin UI and seeing same error?
>
> I am using ML8.0-4 and below code works with the options however I have
> loaded pipeline via UI.
>
> 
> 
> When a document is zip,
> 
> http://marklogic.com/states/initial
> http://marklogic.com/states/transformed
> http://marklogic.com/states/error
> 
> 
>
> /MarkLogic/cpf/actions/mimetype-condition.xqy
>  xmlns="/MarkLogic/cpf/actions/mimetype-condition.xqy">
> application/zip
> 
> 
> 
> action/extract-zip.xqy
> 
> 
> 
>
> Regards,
> Indy
>
> On Wed, Jun 22, 2016 at 2:11 PM, Rob Walpole 
> wrote:
>
>> Hi,
>>
>> We are trying to load a CPF pipeline via the RESTful Management API and
>> this fails with an invalid node error when there is an options element
>> present within /pipeline-properties/state-transition/execute/condition
>>
>> The full error message is as follows:
>>
>> http://marklogic.com/xdmp/error;>
>>   400
>>   Bad Request
>>   MANAGE-INVALIDPAYLOAD
>>   MANAGE-INVALIDPAYLOAD: (err:FOER) Payload has errors in
>> structure, content-type or values. XDMP-VALIDATEUNEXPECTED: (err:XQDY0027)
>> validate strict { $pipeline } -- Invalid node: Found pp:options but
>> expected any(lax,!(http://marklogic.com/manage/pipeline/properties))? at
>> fn:doc("")/pp:pipeline-properties/pp:state-transition/pp:execute/pp:condition/pp:options
>> using schema "manage-pipeline-properties.xsd"
>> 
>>
>> The problem node looks like this:
>>
>> 
>> /MarkLogic/cpf/actions/mimetype-condition.xqy
>> 
>> application/xml
>> 
>> 
>>
>> We have also tried putting the options element in as..
>>
>> 
>> application/xml
>> 
>>
>> ..which is how it is described in the Content Processing Framework Guide
>> but this fails with a different error as follows:
>>
>> http://marklogic.com/xdmp/error;>
>>   500
>>   Internal Server Error
>>   XDMP-VALIDATEUNEXPECTED
>>   XDMP-VALIDATEUNEXPECTED: (err:XQDY0027) validate strict {
>> $pipeline } -- Invalid node: Found p:options but expected any(lax,!(
>> http://marklogic.com/cpf/pipelines))? at
>> /p:pipeline/p:state-transition/p:execute/p:condition/p:options using schema
>> "pipelines.xsd"
>> 
>>
>> If we remove the options element the pipeline loads via the API with no
>> problems.
>>
>> We are using MarkLogic Server Enterprise Edition 8.0-4.2
>>
>> Many thanks
>> Rob Walpole
>>
>> ___
>> General mailing list
>> General@developer.marklogic.com
>> Manage your subscription at:
>> http://developer.marklogic.com/mailman/listinfo/general
>>
>>
>
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at:
> http://developer.marklogic.com/mailman/listinfo/general
>
>
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Validation error when loading pipeline with options via Management API

2016-06-22 Thread Indrajeet Verma

Hi Rob,

Have you tried to load pipeline via admin UI and seeing same error?

I am using ML8.0-4 and below code works with the options however I have
loaded pipeline via UI.



When a document is zip,

http://marklogic.com/states/initial
http://marklogic.com/states/transformed
http://marklogic.com/states/error



/MarkLogic/cpf/actions/mimetype-condition.xqy

application/zip



action/extract-zip.xqy




Regards,
Indy

On Wed, Jun 22, 2016 at 2:11 PM, Rob Walpole  wrote:

> Hi,
>
> We are trying to load a CPF pipeline via the RESTful Management API and
> this fails with an invalid node error when there is an options element
> present within /pipeline-properties/state-transition/execute/condition
>
> The full error message is as follows:
>
> http://marklogic.com/xdmp/error;>
>   400
>   Bad Request
>   MANAGE-INVALIDPAYLOAD
>   MANAGE-INVALIDPAYLOAD: (err:FOER) Payload has errors in
> structure, content-type or values. XDMP-VALIDATEUNEXPECTED: (err:XQDY0027)
> validate strict { $pipeline } -- Invalid node: Found pp:options but
> expected any(lax,!(http://marklogic.com/manage/pipeline/properties))? at
> fn:doc("")/pp:pipeline-properties/pp:state-transition/pp:execute/pp:condition/pp:options
> using schema "manage-pipeline-properties.xsd"
> 
>
> The problem node looks like this:
>
> 
> /MarkLogic/cpf/actions/mimetype-condition.xqy
> 
> application/xml
> 
> 
>
> We have also tried putting the options element in as..
>
> 
> application/xml
> 
>
> ..which is how it is described in the Content Processing Framework Guide
> but this fails with a different error as follows:
>
> http://marklogic.com/xdmp/error;>
>   500
>   Internal Server Error
>   XDMP-VALIDATEUNEXPECTED
>   XDMP-VALIDATEUNEXPECTED: (err:XQDY0027) validate strict {
> $pipeline } -- Invalid node: Found p:options but expected any(lax,!(
> http://marklogic.com/cpf/pipelines))? at
> /p:pipeline/p:state-transition/p:execute/p:condition/p:options using schema
> "pipelines.xsd"
> 
>
> If we remove the options element the pipeline loads via the API with no
> problems.
>
> We are using MarkLogic Server Enterprise Edition 8.0-4.2
>
> Many thanks
> Rob Walpole
>
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at:
> http://developer.marklogic.com/mailman/listinfo/general
>
>
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

[MarkLogic Dev General] Validation error when loading pipeline with options via Management API

2016-06-22 Thread Rob Walpole

Hi,

We are trying to load a CPF pipeline via the RESTful Management API and
this fails with an invalid node error when there is an options element
present within /pipeline-properties/state-transition/execute/condition

The full error message is as follows:

http://marklogic.com/xdmp/error;>
  400
  Bad Request
  MANAGE-INVALIDPAYLOAD
  MANAGE-INVALIDPAYLOAD: (err:FOER) Payload has errors in
structure, content-type or values. XDMP-VALIDATEUNEXPECTED: (err:XQDY0027)
validate strict { $pipeline } -- Invalid node: Found pp:options but
expected any(lax,!(http://marklogic.com/manage/pipeline/properties))? at
fn:doc("")/pp:pipeline-properties/pp:state-transition/pp:execute/pp:condition/pp:options
using schema "manage-pipeline-properties.xsd"


The problem node looks like this:


/MarkLogic/cpf/actions/mimetype-condition.xqy

application/xml



We have also tried putting the options element in as..


application/xml


..which is how it is described in the Content Processing Framework Guide
but this fails with a different error as follows:

http://marklogic.com/xdmp/error;>
  500
  Internal Server Error
  XDMP-VALIDATEUNEXPECTED
  XDMP-VALIDATEUNEXPECTED: (err:XQDY0027) validate strict {
$pipeline } -- Invalid node: Found p:options but expected any(lax,!(
http://marklogic.com/cpf/pipelines))? at
/p:pipeline/p:state-transition/p:execute/p:condition/p:options using schema
"pipelines.xsd"


If we remove the options element the pipeline loads via the API with no
problems.

We are using MarkLogic Server Enterprise Edition 8.0-4.2

Many thanks
Rob Walpole
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

[MarkLogic Dev General] Speeding up access to large XML file

Re: [MarkLogic Dev General] MLCP import transform and search

Re: [MarkLogic Dev General] MLCP import transform and search

Re: [MarkLogic Dev General] MLCP import transform and search

Re: [MarkLogic Dev General] MLCP import transform and search

[MarkLogic Dev General] MLCP import transform and search

Re: [MarkLogic Dev General] Validation error when loading pipeline with options via Management API

Re: [MarkLogic Dev General] Validation error when loading pipeline with options via Management API

Re: [MarkLogic Dev General] Validation error when loading pipeline with options via Management API

Re: [MarkLogic Dev General] Validation error when loading pipeline with options via Management API

[MarkLogic Dev General] Validation error when loading pipeline with options via Management API

11 matches

Site Navigation

Mail list logo

Footer information