[MarkLogic Dev General] wildcarded search with cts:field-value-query
hi all, I'm facing an issue using cts:field-value-query with wildcarded search in ML 7.0-5 Following search should match any field starting with 'dev' like development, etc. let $term := 'dev' return cts:search( fn:collection()/l:manifestation[l:issn], cts:field-value-query( 'field-quick-search', $term || '*', ('case-insensitive', 'diacritic-insensitive', 'unstemmed', 'wildcarded') ) ) But its returns nothing Fields 'field-quick-search' seems to be well configured to work with wildcarded search: Name: field-quick-search Path: *:title, *:subTitle, *:doiSuffix, *:code, *:userCode, *:issn, @id Includes: work, expression, manifestation Excludes (nothing) Index Settings: +word, +fast phrase, +fast case sensitive, +fast diacritic sensitive, +trailing wildcard, +three character, +two character The same search with cts:field-word-query works : let $term := 'dev' return cts:search( fn:collection()/l:manifestation[l:issn], cts:field-word-query( 'field-quick-search', $term || '*', ('case-insensitive', 'diacritic-insensitive', 'unstemmed', 'wildcarded') ) ) Any thoughts on what I might be doing wrong? Thanks Lancelot ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
Re: [MarkLogic Dev General] wildcarded search with cts:field-value-query
Hi Lancelot , cts:field-value-query() matches the full text content containing a given value in the specified field. Example: if a element is containing the development then you have to pass development as search string parameter incts:field-value-query(). Please refer :https://docs.marklogic.com/cts:field-value-query cts:field-word-query() matchestext content containing a given phrase in the specified field. Example : if a element iscontaining the development then you can pass the dev as search string parameter incts:field-word-query() So, that's why you are able to get result while usingcts:field-word-query() function. Please refer :https://docs.marklogic.com/cts:field-word-query Regards, Asit Nautiyal From: general-boun...@developer.marklogic.com [general-boun...@developer.marklogic.com] on behalf of Lanz [lancelot.meuril...@gmail.com] Sent: Monday, May 11, 2015 4:17 PM To: general Subject: [MarkLogic Dev General] wildcarded search with cts:field-value-query hi all, I'm facing an issue using cts:field-value-query with wildcarded search in ML 7.0-5 Following search should match any field starting with 'dev' like development, etc. let $term := 'dev' return cts:search( fn:collection()/l:manifestation[l:issn], cts:field-value-query( 'field-quick-search', $term || '*', ('case-insensitive', 'diacritic-insensitive', 'unstemmed', 'wildcarded') ) ) But its returns nothing Fields 'field-quick-search' seems to be well configured to work with wildcarded search: Name: field-quick-search Path: *:title, *:subTitle, *:doiSuffix, *:code, *:userCode, *:issn, @id Includes: work, _expression_, manifestation Excludes (nothing) Index Settings: word, fast phrase, fast case sensitive, fast diacritic sensitive, trailing wildcard, three character, two character The same search with cts:field-word-query works : let $term := 'dev' return cts:search( fn:collection()/l:manifestation[l:issn], cts:field-word-query( 'field-quick-search', $term || '*', ('case-insensitive', 'diacritic-insensitive', 'unstemmed', 'wildcarded') ) ) Any thoughts on what I might be doingwrong? Thanks Lancelot ::DISCLAIMER:: The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents (with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of authorized representative of HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any email and/or attachments, please check them for viruses and other defects. ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
Re: [MarkLogic Dev General] Search API : defining a new operator
Hi Eric,Thank You for the information. I have chosen the Search operator option (http://docs.marklogic.com/guide/search-dev/query-options#) and following is my xml file search:options xmlns:search=http://marklogic.com/appservices/search; search:operator name=sort search:state name=relevance search:sort-order search:score/ /search:sort-order /search:state search:state name=city search:sort-order direction=descending type=xs:string search:element ns=my-ns name=city/ /search:sort-order search:sort-order search:score/ /search:sort-order /search:state /search:operator/search:options I am using the following url to searchhttp://localhost:8040/v1/search?q=red%20sort:city But I am not able retrieve any result(without sort:city displays the result) is this the right way to use the search option?regards,Puneet From: erik.hen...@marklogic.com To: general@developer.marklogic.com Date: Sun, 10 May 2015 14:11:20 + Subject: Re: [MarkLogic Dev General] Search API : defining a new operator Hi, Puneet: The search:query element can not contain a search:operator child element. Only the search:options element can contain a search:operator child element. The valid options elements are described here: http://docs.marklogic.com/guide/rest-dev/appendixb The valid query elements are described here: http://docs.marklogic.com/guide/search-dev/structured-query#id_59265 The REST API supports sending both query and options in a single request: http://docs.marklogic.com/guide/rest-dev/search#id_69918 Instead of putting the sort order in a string (which has to be parsed), you might want to specify it directly in the options with the sort-order element: http://docs.marklogic.com/guide/rest-dev/appendixb#id_33716 Hoping that helps, Erik Hennum From: general-boun...@developer.marklogic.com [general-boun...@developer.marklogic.com] on behalf of Yinyan guy [yinyan...@hotmail.com] Sent: Saturday, May 09, 2015 10:34 PM To: MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] Search API : defining a new operator Hi Joe, Thanks for your response. I realized I was not on the right track but I have slightly improvised . Now I am configuring 2 search option First one defining the sort operator using following XML ?xml version=1.0? options xmlns=http://marklogic.com/appservices/search; operator name=sort state name=relevance sort-order score/ /sort-order /state state name=postedon sort-order direction=descending type=xs:date json-propertypostedon/json-property /sort-order sort-order score/ /sort-order /state /operator /options This one works without the error. Now when I am trying to define the state using a separate XML as follows ?xml version=1.0? query xmlns:search=http://marklogic.com/appservices/search; operator-state operator-namesort/operator-name state-namepostedon/state-name /operator-state /query I am getting the following error {errorResponse:{statusCode:400, status:Bad Request, messageCode:RESTAPI-INVALIDCONTENT, message:RESTAPI-INVALIDCONTENT: (err:FOER) Invalid content: Unexpected Payload: \t\t\tsort\t\tpostedon\t}} User-Agent: curl/7.30.0 Host: localhost:8040 Accept: */* Content-type: application/xml Content-Length: 194 * upload completely sent off: 194 out of 194 bytes HTTP/1.1 400 Bad Request Content-type: application/json; charset=UTF-8 The data that I have is in JSON form and has postedon as date of the type 2015-04-11, Now the error seems to be about the structure but I am not able to understand. regards, Puneet From: joe.br...@marklogic.com To: general@developer.marklogic.com Date: Sun, 10 May 2015 05:05:54 + Subject: Re: [MarkLogic Dev General] Search API : defining a new operator Hi Puneet, What's the HTTP status code from your search request? Are you getting an error in the response body, or simply a valid response with no results? (If you're getting an error, you can find more details in the MarkLogic error log at http://localhost:8001/get-error-log.xqy?filename=ErrorLog.txt) Thanks. -jb From: Yinyan guy yinyan...@hotmail.com Reply-To: MarkLogic Developer Discussion general@developer.marklogic.com Date: Saturday, May 9, 2015 at 9:12 PM To: general@developer.marklogic.com general@developer.marklogic.com Subject: Re: [MarkLogic Dev General] Search API : defining a new operator Hi, I was able to define the operator with following XML options xmlns=http://marklogic.com/appservices/search; operator name=sort state name=postedon sort-order type=xs:date direction=ascending json-propertypostedon/json-property /sort-order /state /operator /options I can confirm this when I check in http://localhost:8040/v1/config/query But now I am not able to use it in my Search API, I am using the following url to verify the sort order
Re: [MarkLogic Dev General] wildcarded search with cts:field-value-query
Hi Lancelot, Is the text you try to match the entire value of the element it is part of? Like the difference between: titleDevelopment is .../title and: titleThe development is .../title In the first case, it will match both queries (thanks to the wildcard at the end of the search term). In the second case, it will match only the word search (which matches words as in a search engine), not the value search (which matches elements values, like entire values). Regards, -- Florent Georges http://fgeorges.org/ http://h2oconsulting.be/ On 11 May 2015 at 12:47, Lanz wrote: hi all, I'm facing an issue using cts:field-value-query with wildcarded search in ML 7.0-5 Following search should match any field starting with 'dev' like development, etc. let $term := 'dev' return cts:search( fn:collection()/l:manifestation[l:issn], cts:field-value-query( 'field-quick-search', $term || '*', ('case-insensitive', 'diacritic-insensitive', 'unstemmed', 'wildcarded') ) ) But its returns nothing Fields 'field-quick-search' seems to be well configured to work with wildcarded search: Name: field-quick-search Path: *:title, *:subTitle, *:doiSuffix, *:code, *:userCode, *:issn, @id Includes: work, expression, manifestation Excludes (nothing) Index Settings: +word, +fast phrase, +fast case sensitive, +fast diacritic sensitive, +trailing wildcard, +three character, +two character The same search with cts:field-word-query works : let $term := 'dev' return cts:search( fn:collection()/l:manifestation[l:issn], cts:field-word-query( 'field-quick-search', $term || '*', ('case-insensitive', 'diacritic-insensitive', 'unstemmed', 'wildcarded') ) ) Any thoughts on what I might be doing wrong? Thanks Lancelot ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
Re: [MarkLogic Dev General] Re-indexing new documents
Newly added files will be indexed as they are added. It's not clear from your question why you would reindex the database. Here's some general information on reindexing: https://help.marklogic.com/knowledgebase/article/View/18/15/how-reindexing-works-and-its-impact-on-performance On Mon, May 11, 2015 at 8:42 AM, Shashidhar Rao raoshashidhar...@gmail.com wrote: Hi , We have loaded some 500GB of data and the re-indexing is almost done. These include fast character searches, word searches, word positions just to name a few. Now we need to store another 14 GB of data. When re-indexing is done again will Marklogic index only the newly added files or will it index all the documents again. Please help. Thanks ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
[MarkLogic Dev General] Re-indexing new documents
Hi , We have loaded some 500GB of data and the re-indexing is almost done. These include fast character searches, word searches, word positions just to name a few. Now we need to store another 14 GB of data. When re-indexing is done again will Marklogic index only the newly added files or will it index all the documents again. Please help. Thanks ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
Re: [MarkLogic Dev General] Re-indexing new documents
Hi Sashidhar, MarkLogic re-indexing for whole content is happen only when configuration are changes. If there is no changes are made to database after loading 500GB data then you can load another 14GB of data without no issue. Re-indexing only needed when you made any changes to database configuration settings like creating new range-indexing etc. Regards, Asit Nautiyal From: general-boun...@developer.marklogic.com [general-boun...@developer.marklogic.com] on behalf of Shashidhar Rao [raoshashidhar...@gmail.com] Sent: Monday, May 11, 2015 6:12 PM To: general@developer.marklogic.com Subject: [MarkLogic Dev General] Re-indexing new documents Hi , We have loaded some 500GB of data and the re-indexing is almost done. These include fast character searches, word searches, word positions just to name a few. Now we need to store another 14 GB of data. When re-indexing is done again will Marklogic index only the newly added files or will it index all the documents again. Please help. Thanks ::DISCLAIMER:: The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents (with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of authorized representative of HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any email and/or attachments, please check them for viruses and other defects. ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
[MarkLogic Dev General] List of Management API resources and possible parameters
The Management API docs https://docs.marklogic.com/guide/monitoring/monitoringAPI#id_99441 mention how to list resources like forests or databases. Is there a listing of all possible resources and their allowed parameters ? In particular, I'm looking for a ready-made way to query what indexes are present for a given database. Regards, Danny ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
Re: [MarkLogic Dev General] Re-indexing new documents
Hi Christopher, I wanted to check for this condition. Already 500GB loaded and re-indexing done. Now I first load 12 GB of data which has different elements without creating any range index. When loading is finished I enable the reindexer enable to false. Now I create the range indexes for these newly loaded data. Now I enable the reindexer enable to true and click ok. On the Admin status page, when re-index starts it always gives out the number of fragments and the time to complete. My question was on the number of fragments in this situation. Will the number of fragments will be 4 million plus newly loaded data number or just the newly added fragments say in this case .2 million. Thanks ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
[MarkLogic Dev General] (no subject)
Thank you Asit for your explanation, I confused the two mechanisms It's clear now Regards, Lancelot On Mon, May 11, 2015 at 1:06 PM, Asitmohan Nautiyal asitmoha...@hcl.com wrote: Hi Lancelot , cts:field-value-query() matches the full text content containing a given value in the specified field. Example: if a element is containing the development then you have to pass development as search string parameter in cts:field-value-query(). Please refer : https://docs.marklogic.com/cts:field-value-query cts:field-word-query() matches text content containing a given phrase in the specified field. Example : if a element is containing the development then you can pass the dev as search string parameter in cts:field-word-query() So, that's why you are able to get result while using cts:field-word-query() function. Please refer : https://docs.marklogic.com/cts:field-word-query Regards, Asit Nautiyal -- *From:* general-boun...@developer.marklogic.com [ general-boun...@developer.marklogic.com] on behalf of Lanz [ lancelot.meuril...@gmail.com] *Sent:* Monday, May 11, 2015 4:17 PM *To:* general *Subject:* [MarkLogic Dev General] wildcarded search with cts:field-value-query hi all, I'm facing an issue using cts:field-value-query with wildcarded search in ML 7.0-5 Following search should match any field starting with 'dev' like development, etc. let $term := 'dev' return cts:search( fn:collection()/l:manifestation[l:issn], cts:field-value-query( 'field-quick-search', $term || '*', ('case-insensitive', 'diacritic-insensitive', 'unstemmed', 'wildcarded') ) ) But its returns nothing Fields 'field-quick-search' seems to be well configured to work with wildcarded search: Name: field-quick-search Path: *:title, *:subTitle, *:doiSuffix, *:code, *:userCode, *:issn, @id Includes: work, expression, manifestation Excludes (nothing) Index Settings: +word, +fast phrase, +fast case sensitive, +fast diacritic sensitive, +trailing wildcard, +three character, +two character The same search with cts:field-word-query works : let $term := 'dev' return cts:search( fn:collection()/l:manifestation[l:issn], cts:field-word-query( 'field-quick-search', $term || '*', ('case-insensitive', 'diacritic-insensitive', 'unstemmed', 'wildcarded') ) ) Any thoughts on what I might be doing wrong? Thanks Lancelot ::DISCLAIMER:: The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents (with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of authorized representative of HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any email and/or attachments, please check them for viruses and other defects. ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
Re: [MarkLogic Dev General] Search API : defining a new operator
Hi, Puneet: Your request doesn't specify your options. If you haven't installed your options as the default, you should install your options under another name and refer to them with the options URI parameter or POST your options as part of the request: http://docs.marklogic.com/guide/rest-dev/search#id_83871 If you POST query options, it would be better to specify the sort order in the options instead of parsing them from the string. You should consider whether you can run an unfiltered search (as with the default options). Hoping that helps, Erik Hennum From: general-boun...@developer.marklogic.com [general-boun...@developer.marklogic.com] on behalf of Yinyan guy [yinyan...@hotmail.com] Sent: Monday, May 11, 2015 2:40 AM To: MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] Search API : defining a new operator Hi Eric, Thank You for the information. I have chosen the Search operator option (http://docs.marklogic.com/guide/search-dev/query-options#) and following is my xml file search:options xmlns:search=http://marklogic.com/appservices/search; search:operator name=sort search:state name=relevance search:sort-order search:score/ /search:sort-order /search:state search:state name=city search:sort-order direction=descending type=xs:string search:element ns=my-ns name=city/ /search:sort-order search:sort-order search:score/ /search:sort-order /search:state /search:operator /search:options I am using the following url to search http://localhost:8040/v1/search?q=red%20sort:city But I am not able retrieve any result(without sort:city displays the result) is this the right way to use the search option? regards, Puneet From: erik.hen...@marklogic.com To: general@developer.marklogic.com Date: Sun, 10 May 2015 14:11:20 + Subject: Re: [MarkLogic Dev General] Search API : defining a new operator Hi, Puneet: The search:query element can not contain a search:operator child element. Only the search:options element can contain a search:operator child element. The valid options elements are described here: http://docs.marklogic.com/guide/rest-dev/appendixb The valid query elements are described here: http://docs.marklogic.com/guide/search-dev/structured-query#id_59265 The REST API supports sending both query and options in a single request: http://docs.marklogic.com/guide/rest-dev/search#id_69918 Instead of putting the sort order in a string (which has to be parsed), you might want to specify it directly in the options with the sort-order element: http://docs.marklogic.com/guide/rest-dev/appendixb#id_33716 Hoping that helps, Erik Hennum From: general-boun...@developer.marklogic.com [general-boun...@developer.marklogic.com] on behalf of Yinyan guy [yinyan...@hotmail.com] Sent: Saturday, May 09, 2015 10:34 PM To: MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] Search API : defining a new operator Hi Joe, Thanks for your response. I realized I was not on the right track but I have slightly improvised . Now I am configuring 2 search option First one defining the sort operator using following XML ?xml version=1.0? options xmlns=http://marklogic.com/appservices/search; operator name=sort state name=relevance sort-order score/ /sort-order /state state name=postedon sort-order direction=descending type=xs:date json-propertypostedon/json-property /sort-order sort-order score/ /sort-order /state /operator /options This one works without the error. Now when I am trying to define the state using a separate XML as follows ?xml version=1.0? query xmlns:search=http://marklogic.com/appservices/search; operator-state operator-namesort/operator-name state-namepostedon/state-name /operator-state /query I am getting the following error {errorResponse:{statusCode:400, status:Bad Request, messageCode:RESTAPI-INVALIDCONTENT, message:RESTAPI-INVALIDCONTENT: (err:FOER) Invalid content: Unexpected Payload: \t\t\tsort\t\tpostedon\t}} User-Agent: curl/7.30.0 Host: localhost:8040 Accept: */* Content-type: application/xml Content-Length: 194 * upload completely sent off: 194 out of 194 bytes HTTP/1.1 400 Bad Request Content-type: application/json; charset=UTF-8 The data that I have is in JSON form and has postedon as date of the type 2015-04-11, Now the error seems to be about the structure but I am not able to understand. regards, Puneet From: joe.br...@marklogic.com To: general@developer.marklogic.com Date: Sun, 10 May 2015 05:05:54 + Subject: Re: [MarkLogic Dev General] Search API : defining a new operator Hi Puneet, What's the HTTP status code from your search request? Are you getting an error in the response body, or simply a valid response with no results? (If you're getting an error, you
Re: [MarkLogic Dev General] List of Management API resources and possible parameters
Found it ... http://docs.marklogic.com/REST/management . On Mon, May 11, 2015 at 9:07 AM, Danny Sinang d.sin...@gmail.com wrote: The Management API docs https://docs.marklogic.com/guide/monitoring/monitoringAPI#id_99441 mention how to list resources like forests or databases. Is there a listing of all possible resources and their allowed parameters ? In particular, I'm looking for a ready-made way to query what indexes are present for a given database. Regards, Danny ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
Re: [MarkLogic Dev General] XDMP-INMMLISTFULL error
Hi Christopher, It's an error , below There is currently an XDMP-FORESTERR: Error in reindex of forest PROD_DB_1: XDMP-REINDEX: Error reindexing fn:doc(/home/data/TD078999.XML): XDMP-FRAGTOOLARGE: Fragment of /home/data/TD078999.XML too large for in-memory storage: XDMP-INMMLISTFULL: In-memory list storage full; list: table=100%, wordsused=50%, wordsfree=25%, overhead=25%; tree: table=0%, wordsused=6%, wordsfree=94%, overhead=0% exception. Information on this page may be missing. Any suggestion on how to resolve this error? This value 32555 MB if I increase it says invalid number. I have seen the link you have sent but I am just not getting the number. Thanks On Mon, May 11, 2015 at 8:06 PM, Shashidhar Rao raoshashidhar...@gmail.com wrote: Hi, I have seen various post regarding this error but it only says it needs to increase the size. The *in memory list size* is currently in my case. If I try to increase the size it says it is not valid size. What size will be valid? Please help. Thanks ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
Re: [MarkLogic Dev General] Analyze-data.xqy with JSON data
Hi there, Sounds like my work. I have a copy of it here: https://gist.github.com/grtjn, and it is also included in the slush template. Not sure which is latest, maybe neither. I’m a bit surprised it doesn’t work as is, but will have a look. I can imagine though, it will not provide a lot of suggestions for json data, I really created it a year ago, when EA1 wasn’t even out yet.. :) Stay tuned, Geert From: Yinyan guy yinyan...@hotmail.commailto:yinyan...@hotmail.com Reply-To: MarkLogic Developer Discussion general@developer.marklogic.commailto:general@developer.marklogic.com Date: Monday, May 4, 2015 at 3:17 AM To: MarkLogic Developer Discussion general@developer.marklogic.commailto:general@developer.marklogic.com Subject: Re: [MarkLogic Dev General] Analyze-data.xqy with JSON data Got it! Thank You Eric. I did deploy this as an extension using the ./local deploy modules with definition in /rest-api/ext/analyze-data.xqy Regards, Puneet From: erik.hen...@marklogic.commailto:erik.hen...@marklogic.com To: general@developer.marklogic.commailto:general@developer.marklogic.com Date: Mon, 4 May 2015 00:47:55 + Subject: Re: [MarkLogic Dev General] Analyze-data.xqy with JSON data Hi, Puneet: Services available under /v1/resources are extensions that were installed under /v1/config/resources and not services implemented and delivered as part of the REST API itself. You should contact whoever wrote the analyze-data extension about modifying it to support JSON. Hoping that helps, Erik Hennum From: general-boun...@developer.marklogic.commailto:general-boun...@developer.marklogic.com [general-boun...@developer.marklogic.commailto:general-boun...@developer.marklogic.com] on behalf of Yinyan guy [yinyan...@hotmail.commailto:yinyan...@hotmail.com] Sent: Sunday, May 03, 2015 4:21 PM To: MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] Analyze-data.xqy with JSON data HI Eric, Thank You for your response!. I am using following call http://localhost:8040/v1/resources/analyze-data; This has worked for me in this past for XML documents and when I was using ML7. Now I am using it on ML8 and with JSON documents. regards, Puneet From: erik.hen...@marklogic.commailto:erik.hen...@marklogic.com To: general@developer.marklogic.commailto:general@developer.marklogic.com Date: Sun, 3 May 2015 13:36:29 + Subject: Re: [MarkLogic Dev General] Analyze-data.xqy with JSON data Hi, Puneet: Can you provide an example of the REST API call that you're making to analyze data? Erik Hennum From: general-boun...@developer.marklogic.commailto:general-boun...@developer.marklogic.com [general-boun...@developer.marklogic.commailto:general-boun...@developer.marklogic.com] on behalf of Yinyan guy [yinyan...@hotmail.commailto:yinyan...@hotmail.com] Sent: Saturday, May 02, 2015 6:35 AM To: general@developer.marklogic.commailto:general@developer.marklogic.com Subject: [MarkLogic Dev General] Analyze-data.xqy with JSON data Hi All, I am exploring the use of the analyze-data for index suggestion on my data. I am using ML8 and my data is JSON. On running analyze-data using the REST API, I got 400: Bad Request with message XDMP-UNEXPECTED: (err:XPST0003) Unexpected token syntax error, unexpected Rpar_, expecting Function30_ or Percent_ I guess it works only with xml data. is there a JSON version in works? Looking forward to hear from you. regards, Puneet ___ General mailing list General@developer.marklogic.commailto:General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.commailto:General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
Re: [MarkLogic Dev General] Re-indexing new documents
The reindexer will not reindex all documents every time there is a change. It will try to minimize the work to only those fragments that need the indexing. However, when you know you need an index, why not add it before you add the new content? Even though the reindexer will minimize the work, it will still need to reinsert the fragment, and there will likely be merges too. The reindexer would still need to be enabled to reindex the existing 12 GB (if such reindexing is needed) but the data you load after adding the index would be indexed when it is added to the db. On Mon, May 11, 2015 at 9:58 AM, Shashidhar Rao raoshashidhar...@gmail.com wrote: Hi Christopher, I wanted to check for this condition. Already 500GB loaded and re-indexing done. Now I first load 12 GB of data which has different elements without creating any range index. When loading is finished I enable the reindexer enable to false. Now I create the range indexes for these newly loaded data. Now I enable the reindexer enable to true and click ok. On the Admin status page, when re-index starts it always gives out the number of fragments and the time to complete. My question was on the number of fragments in this situation. Will the number of fragments will be 4 million plus newly loaded data number or just the newly added fragments say in this case .2 million. Thanks ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
Re: [MarkLogic Dev General] wildcarded search with cts:field-value-query
Hi Florent, Thanks I understand the difference, actually we want to match word (and part of word) wherever they it is located in the value of the field. The cts:field-word-query is the right option in our case. Regards Lancelot On Mon, May 11, 2015 at 1:11 PM, Florent Georges li...@fgeorges.org wrote: Hi Lancelot, Is the text you try to match the entire value of the element it is part of? Like the difference between: titleDevelopment is .../title and: titleThe development is .../title In the first case, it will match both queries (thanks to the wildcard at the end of the search term). In the second case, it will match only the word search (which matches words as in a search engine), not the value search (which matches elements values, like entire values). Regards, -- Florent Georges http://fgeorges.org/ http://h2oconsulting.be/ On 11 May 2015 at 12:47, Lanz wrote: hi all, I'm facing an issue using cts:field-value-query with wildcarded search in ML 7.0-5 Following search should match any field starting with 'dev' like development, etc. let $term := 'dev' return cts:search( fn:collection()/l:manifestation[l:issn], cts:field-value-query( 'field-quick-search', $term || '*', ('case-insensitive', 'diacritic-insensitive', 'unstemmed', 'wildcarded') ) ) But its returns nothing Fields 'field-quick-search' seems to be well configured to work with wildcarded search: Name: field-quick-search Path: *:title, *:subTitle, *:doiSuffix, *:code, *:userCode, *:issn, @id Includes: work, expression, manifestation Excludes (nothing) Index Settings: +word, +fast phrase, +fast case sensitive, +fast diacritic sensitive, +trailing wildcard, +three character, +two character The same search with cts:field-word-query works : let $term := 'dev' return cts:search( fn:collection()/l:manifestation[l:issn], cts:field-word-query( 'field-quick-search', $term || '*', ('case-insensitive', 'diacritic-insensitive', 'unstemmed', 'wildcarded') ) ) Any thoughts on what I might be doing wrong? Thanks Lancelot ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
Re: [MarkLogic Dev General] XDMP-INMMLISTFULL error
What log level are these? Info? Debug? Are these truly errors? More info is here: https://help.marklogic.com/knowledgebase/article/View/23/16/understanding-xdmp-inmmfull-messages On Mon, May 11, 2015 at 10:36 AM, Shashidhar Rao raoshashidhar...@gmail.com wrote: Hi, I have seen various post regarding this error but it only says it needs to increase the size. The in memory list size is currently in my case. If I try to increase the size it says it is not valid size. What size will be valid? Please help. Thanks ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
Re: [MarkLogic Dev General] Re-indexing new documents
Thanks Chrisopher for your suggestion. On Mon, May 11, 2015 at 7:28 PM, Shashidhar Rao raoshashidhar...@gmail.com wrote: Hi Christopher, I wanted to check for this condition. Already 500GB loaded and re-indexing done. Now I first load 12 GB of data which has different elements without creating any range index. When loading is finished I enable the reindexer enable to false. Now I create the range indexes for these newly loaded data. Now I enable the reindexer enable to true and click ok. On the Admin status page, when re-index starts it always gives out the number of fragments and the time to complete. My question was on the number of fragments in this situation. Will the number of fragments will be 4 million plus newly loaded data number or just the newly added fragments say in this case .2 million. Thanks ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
[MarkLogic Dev General] XDMP-INMMLISTFULL error
Hi, I have seen various post regarding this error but it only says it needs to increase the size. The *in memory list size* is currently in my case. If I try to increase the size it says it is not valid size. What size will be valid? Please help. Thanks ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
Re: [MarkLogic Dev General] Analyze-data.xqy with JSON data
It goes wrong here: https://github.com/marklogic/slush-marklogic-node/blob/master/app/templates/rest-api/ext/analyze-data.xqy#L516 Replace: local-name($node) With: local-name-from-QName(node-name($node)) And you should have something that works again. I also suggest to turn on $use-path-indexes here: https://github.com/marklogic/slush-marklogic-node/blob/master/app/templates/rest-api/ext/analyze-data.xqy#L19 Cheers, Geert From: Geert Josten geert.jos...@marklogic.commailto:geert.jos...@marklogic.com Reply-To: MarkLogic Developer Discussion general@developer.marklogic.commailto:general@developer.marklogic.com Date: Monday, May 11, 2015 at 4:10 PM To: MarkLogic Developer Discussion general@developer.marklogic.commailto:general@developer.marklogic.com Subject: Re: [MarkLogic Dev General] Analyze-data.xqy with JSON data Hi there, Sounds like my work. I have a copy of it here: https://gist.github.com/grtjn, and it is also included in the slush template. Not sure which is latest, maybe neither. I’m a bit surprised it doesn’t work as is, but will have a look. I can imagine though, it will not provide a lot of suggestions for json data, I really created it a year ago, when EA1 wasn’t even out yet.. :) Stay tuned, Geert From: Yinyan guy yinyan...@hotmail.commailto:yinyan...@hotmail.com Reply-To: MarkLogic Developer Discussion general@developer.marklogic.commailto:general@developer.marklogic.com Date: Monday, May 4, 2015 at 3:17 AM To: MarkLogic Developer Discussion general@developer.marklogic.commailto:general@developer.marklogic.com Subject: Re: [MarkLogic Dev General] Analyze-data.xqy with JSON data Got it! Thank You Eric. I did deploy this as an extension using the ./local deploy modules with definition in /rest-api/ext/analyze-data.xqy Regards, Puneet From: erik.hen...@marklogic.commailto:erik.hen...@marklogic.com To: general@developer.marklogic.commailto:general@developer.marklogic.com Date: Mon, 4 May 2015 00:47:55 + Subject: Re: [MarkLogic Dev General] Analyze-data.xqy with JSON data Hi, Puneet: Services available under /v1/resources are extensions that were installed under /v1/config/resources and not services implemented and delivered as part of the REST API itself. You should contact whoever wrote the analyze-data extension about modifying it to support JSON. Hoping that helps, Erik Hennum From:general-boun...@developer.marklogic.commailto:general-boun...@developer.marklogic.com [general-boun...@developer.marklogic.commailto:general-boun...@developer.marklogic.com] on behalf of Yinyan guy [yinyan...@hotmail.commailto:yinyan...@hotmail.com] Sent: Sunday, May 03, 2015 4:21 PM To: MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] Analyze-data.xqy with JSON data HI Eric, Thank You for your response!. I am using following call http://localhost:8040/v1/resources/analyze-data; This has worked for me in this past for XML documents and when I was using ML7. Now I am using it on ML8 and with JSON documents. regards, Puneet From: erik.hen...@marklogic.commailto:erik.hen...@marklogic.com To: general@developer.marklogic.commailto:general@developer.marklogic.com Date: Sun, 3 May 2015 13:36:29 + Subject: Re: [MarkLogic Dev General] Analyze-data.xqy with JSON data Hi, Puneet: Can you provide an example of the REST API call that you're making to analyze data? Erik Hennum From:general-boun...@developer.marklogic.commailto:general-boun...@developer.marklogic.com [general-boun...@developer.marklogic.commailto:general-boun...@developer.marklogic.com] on behalf of Yinyan guy [yinyan...@hotmail.commailto:yinyan...@hotmail.com] Sent: Saturday, May 02, 2015 6:35 AM To: general@developer.marklogic.commailto:general@developer.marklogic.com Subject: [MarkLogic Dev General] Analyze-data.xqy with JSON data Hi All, I am exploring the use of the analyze-data for index suggestion on my data. I am using ML8 and my data is JSON. On running analyze-data using the REST API, I got 400: Bad Request with message XDMP-UNEXPECTED: (err:XPST0003) Unexpected token syntax error, unexpected Rpar_, expecting Function30_ or Percent_ I guess it works only with xml data. is there a JSON version in works? Looking forward to hear from you. regards, Puneet ___ General mailing list General@developer.marklogic.commailto:General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.commailto:General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.com Manage your
Re: [MarkLogic Dev General] Forest Warning
Hi Wayne, Thank your for correcting me however I was just suggesting some options to improve performance after solving that issue as he said, he is only one forest in the DB with TB data. Regards, Indy On Mon, May 11, 2015 at 11:57 PM, Wayne Feick wayne.fe...@marklogic.com wrote: Creating more forests won't help, because they'll all have the same in-memory limits and the rebalancer will run into the same issue as the reindexer if it tries to move the document to a different forest. Wayne. On 05/11/2015 11:18 AM, Indrajeet Verma wrote: Shashi, I am sure, this file size must be large. This is not good that you have created only one forest. There should be more than that. I would recommend around 8-10 forests. However these should be based on core of CPUs. per 2 core 1 forest but if your server is having 32 cores, 8-10 forest should be sufficient. Please correct somebody if I am wrong. As per my understanding you should to do following things to solve your problem, 1. Delete this large file size. I would not recommend to increase memory size just for the heck. Your performance will be degraded. 2. Create more forest and attach to the database. if you are using 7+, this automatically re-balance the data. 3. After re-balancing the content, you might perform manual merge to claim memory immediately. 3. Split your files into smaller size Regards, Indy On Mon, May 11, 2015 at 11:30 PM, Wayne Feick wayne.fe...@marklogic.com wrote: Looping in some additional information from private email. Since your list size is already configured to the maximum (32768), you could try to identify some index settings that you don't actually need and turn them off. If that isn't an option, you could try breaking it up into multiple documents, and then deleting the original document with xdmp:document-delete(). Wayne. On 05/11/2015 10:52 AM, Wayne Feick wrote: Hi Shashidhar, It sounds like the document was close to the limit when it was originally ingested, and that turning on additional index setting put it over the top. The error message says that your in-memory list storage is full, so if you go to the Admin UI and look at the database settings, you'll see an entry for in memory list size. Configure a larger value there and you should be able to finish your reindex. Wayne. On 04/23/2015 01:28 AM, Shashidhar Rao wrote: Hi, Can somebody help me how to fix this issue There is currently an XDMP-FORESTERR: Error in reindex of forest PROD_DB_1: XDMP-REINDEX: Error reindexing fn:doc(/home/data/Folder2/US07625699-20091201-T2.XML): XDMP-FRAGTOOLARGE: Fragment of /home/data/Folder2/US07625699-20091201-T2.XML too large for in-memory storage: XDMP-INMMLISTFULL: In-memory list storage full; list: table=100%, wordsused=50%, wordsfree=25%, overhead=25%; tree: table=0%, wordsused=6%, wordsfree=94%, overhead=0% exception. Information on this page may be missing. It says US07625699-20091201-T2.XML too large. what are the other options any suggestions would be helpful. Is deleting this file an option as the last resort? Thanks ___ General mailing listgene...@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general -- Wayne Feick Principal Engineer MarkLogic corporationwayne.fe...@marklogic.com Phone: +1 650 655 2378www.marklogic.com This e-mail and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of this e-mail communication by others is strictly prohibited. If you are not the intended recipient, please notify us immediately by returning this message to the sender and delete all copies. Thank you for your cooperation. ___ General mailing listgene...@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general -- Wayne Feick Principal Engineer MarkLogic corporationwayne.fe...@marklogic.com Phone: +1 650 655 2378www.marklogic.com This e-mail and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of this e-mail communication by others is strictly prohibited. If you are not the intended recipient, please notify us immediately by returning this message to the sender and delete all copies. Thank you for your cooperation. ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing listgene...@developer.marklogic.com
Re: [MarkLogic Dev General] bulk delete xquery syntax
Thank you, got it working just like you said: admin:save-configuration(admin:appserver-delete($config, admin:appserver-get-id($config, $groupid, secure2_app) ) ) On 05/11/2015 02:42 PM, Danny Sokolsky wrote: Your save-configuration call has to wrap the config returned in your flwor expression. Something like: admin:save-configuration( admin:appserver-delete($config,admin:appserver-get-id($config, $groupid, secure2_app)) ) *From:*general-boun...@developer.marklogic.com [mailto:general-boun...@developer.marklogic.com] *On Behalf Of *cyanline llc *Sent:* Monday, May 11, 2015 11:40 AM *To:* MarkLogic Developer Discussion *Subject:* [MarkLogic Dev General] bulk delete xquery syntax Hi, Getting started with XQuery so that I can bulk delete a bunch of applications and associated files, but stuck on xquery syntax error. And, I'm also wondering, can I make this call and delete more than 1 app server simultaneously? As well as the app-server's associated DB, modules, and forests? This is the code I'm trying to run, and I found it in the link below: xquery version 1.0-ml; import module namespace admin = http://marklogic.com/xdmp/admin; http://marklogic.com/xdmp/admin at /MarkLogic/admin.xqy; let $config := admin:get-configuration() let $groupid := admin:group-get-id($config, Default) return admin:appserver-delete($config,admin:appserver-get-id($config, $groupid, secure2_app)) admin:save-configuration($config) (: returns the new configuration element -- use admin:save-configuration to save the changes to the configuration or pass the configuration to other Admin API functions to make other changes. :) - https://docs.marklogic.com/7.0/admin:appserver-delete and, the error: [1.0-ml] XDMP-UNEXPECTED: (err:XPST0003) Unexpected token syntax error, unexpected QName_, expecting $end or SemiColon_ Stack Trace At line 10 column 2: In xdmp:eval( xquery version quot;1.0-mlquot;;#10;#10; import module na..., (), options xmlns=xdmp:evalisolationdifferent-transaction/isolationignore-ampstrue/i.../options) 8. return 9. admin:appserver-delete($config,admin:appserver-get-id($config, $groupid, secure2_app)) 10. admin:save-configuration($config) ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
Re: [MarkLogic Dev General] XDMP-FRAGTOOLARGE
Hi Shashidhar, I’m wondering how large the original file was, probably not 32gb. I’m also wondering how it ended up getting inserted without trouble. Almost as if memory values have been tuned down afterwards. I’d decrease memory list size to a more reasonable value, and instead take a look at in memory tree size as well. It is also suggested to keep journal size larger than list size + tree size at minimum. You could of course delete the file, that should just work, but I can’t judge whether the file contains valuable information or not. It is also suggested to split that file into smaller parts. Best to do that at ingest time, but if that is not an option, fragmentation might help here. But word of warning fragmentation also influences how queries behave, and have some other side effects as well. We typically recommend against it.. Kind regards, Geert From: Shashidhar Rao raoshashidhar...@gmail.commailto:raoshashidhar...@gmail.com Date: Monday, May 11, 2015 at 6:52 PM To: Geert Josten geert.jos...@marklogic.commailto:geert.jos...@marklogic.com Subject: XDMP-FRAGTOOLARGE Hi Geert, I am getting this error. I tried posting but not getting any replies. Can you suggest anything to resolve this error. There is currently an XDMP-FORESTERR: Error in reindex of forest PROD_DB_1: XDMP-REINDEX: Error reindexing fn:doc(/home/data/TD078999.XML): XDMP-FRAGTOOLARGE: Fragment of /home/data/TD078999.XML too large for in-memory storage: : In-memory list storage full; list: table=100%, wordsused=50%, wordsfree=25%, overhead=25%; tree: table=0%, wordsused=6%, wordsfree=94%, overhead=0% exception. Information on this page may be missing. Any suggestion on how to resolve this error? My in memory list is 32699MB How can I increase this value or can I delete this file? Please help Thanks ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
Re: [MarkLogic Dev General] XDMP-FRAGTOOLARGE
Shashi, I think you should see your logic of splitting/loading of the XML document. What method/tool are you using to ingest content (mlcp and recordloader etc...) You can write your logic to split files based on a element which you want to create a root element. Also please take a look the suggestions of the journal size that Geert has suggested. I am sure there is some problem in your ingestion and XML size otherwise default configurations also works without any issues. of-course later you can optimize them if needed. Also you should see on your number of forests otherwise you can not show the actual power of ML to your customer. (mainly search performance and Big Data management etc..) Records, Indy Regards, Indy On Mon, May 11, 2015 at 11:49 PM, Geert Josten geert.jos...@marklogic.com wrote: Hi Shashidhar, I’m wondering how large the original file was, probably not 32gb. I’m also wondering how it ended up getting inserted without trouble. Almost as if memory values have been tuned down afterwards. I’d decrease memory list size to a more reasonable value, and instead take a look at in memory tree size as well. It is also suggested to keep journal size larger than list size + tree size at minimum. You could of course delete the file, that should just work, but I can’t judge whether the file contains valuable information or not. It is also suggested to split that file into smaller parts. Best to do that at ingest time, but if that is not an option, fragmentation might help here. But word of warning fragmentation also influences how queries behave, and have some other side effects as well. We typically recommend against it.. Kind regards, Geert From: Shashidhar Rao raoshashidhar...@gmail.com Date: Monday, May 11, 2015 at 6:52 PM To: Geert Josten geert.jos...@marklogic.com Subject: XDMP-FRAGTOOLARGE Hi Geert, I am getting this error. I tried posting but not getting any replies. Can you suggest anything to resolve this error. There is currently an XDMP-FORESTERR: Error in reindex of forest PROD_DB_1: XDMP-REINDEX: Error reindexing fn:doc(/home/data/TD078999.XML): XDMP-FRAGTOOLARGE: Fragment of /home/data/TD078999.XML too large for in-memory storage: : In-memory list storage full; list: table=100%, wordsused=50%, wordsfree=25%, overhead=25%; tree: table=0%, wordsused=6%, wordsfree=94%, overhead=0% exception. Information on this page may be missing. Any suggestion on how to resolve this error? My in memory list is 32699MB How can I increase this value or can I delete this file? Please help Thanks ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
Re: [MarkLogic Dev General] Forest Warning
Shashi, I am sure, this file size must be large. This is not good that you have created only one forest. There should be more than that. I would recommend around 8-10 forests. However these should be based on core of CPUs. per 2 core 1 forest but if your server is having 32 cores, 8-10 forest should be sufficient. Please correct somebody if I am wrong. As per my understanding you should to do following things to solve your problem, 1. Delete this large file size. I would not recommend to increase memory size just for the heck. Your performance will be degraded. 2. Create more forest and attach to the database. if you are using 7+, this automatically re-balance the data. 3. After re-balancing the content, you might perform manual merge to claim memory immediately. 3. Split your files into smaller size Regards, Indy On Mon, May 11, 2015 at 11:30 PM, Wayne Feick wayne.fe...@marklogic.com wrote: Looping in some additional information from private email. Since your list size is already configured to the maximum (32768), you could try to identify some index settings that you don't actually need and turn them off. If that isn't an option, you could try breaking it up into multiple documents, and then deleting the original document with xdmp:document-delete(). Wayne. On 05/11/2015 10:52 AM, Wayne Feick wrote: Hi Shashidhar, It sounds like the document was close to the limit when it was originally ingested, and that turning on additional index setting put it over the top. The error message says that your in-memory list storage is full, so if you go to the Admin UI and look at the database settings, you'll see an entry for in memory list size. Configure a larger value there and you should be able to finish your reindex. Wayne. On 04/23/2015 01:28 AM, Shashidhar Rao wrote: Hi, Can somebody help me how to fix this issue There is currently an XDMP-FORESTERR: Error in reindex of forest PROD_DB_1: XDMP-REINDEX: Error reindexing fn:doc(/home/data/Folder2/US07625699-20091201-T2.XML): XDMP-FRAGTOOLARGE: Fragment of /home/data/Folder2/US07625699-20091201-T2.XML too large for in-memory storage: XDMP-INMMLISTFULL: In-memory list storage full; list: table=100%, wordsused=50%, wordsfree=25%, overhead=25%; tree: table=0%, wordsused=6%, wordsfree=94%, overhead=0% exception. Information on this page may be missing. It says US07625699-20091201-T2.XML too large. what are the other options any suggestions would be helpful. Is deleting this file an option as the last resort? Thanks ___ General mailing listgene...@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general -- Wayne Feick Principal Engineer MarkLogic corporationwayne.fe...@marklogic.com Phone: +1 650 655 2378www.marklogic.com This e-mail and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of this e-mail communication by others is strictly prohibited. If you are not the intended recipient, please notify us immediately by returning this message to the sender and delete all copies. Thank you for your cooperation. ___ General mailing listgene...@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general -- Wayne Feick Principal Engineer MarkLogic corporationwayne.fe...@marklogic.com Phone: +1 650 655 2378www.marklogic.com This e-mail and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of this e-mail communication by others is strictly prohibited. If you are not the intended recipient, please notify us immediately by returning this message to the sender and delete all copies. Thank you for your cooperation. ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
Re: [MarkLogic Dev General] bulk delete xquery syntax
Your save-configuration call has to wrap the config returned in your flwor expression. Something like: admin:save-configuration( admin:appserver-delete($config,admin:appserver-get-id($config, $groupid, secure2_app)) ) From: general-boun...@developer.marklogic.com [mailto:general-boun...@developer.marklogic.com] On Behalf Of cyanline llc Sent: Monday, May 11, 2015 11:40 AM To: MarkLogic Developer Discussion Subject: [MarkLogic Dev General] bulk delete xquery syntax Hi, Getting started with XQuery so that I can bulk delete a bunch of applications and associated files, but stuck on xquery syntax error. And, I'm also wondering, can I make this call and delete more than 1 app server simultaneously? As well as the app-server's associated DB, modules, and forests? This is the code I'm trying to run, and I found it in the link below: xquery version 1.0-ml; import module namespace admin = http://marklogic.com/xdmp/admin;http://marklogic.com/xdmp/admin at /MarkLogic/admin.xqy; let $config := admin:get-configuration() let $groupid := admin:group-get-id($config, Default) return admin:appserver-delete($config,admin:appserver-get-id($config, $groupid, secure2_app)) admin:save-configuration($config) (: returns the new configuration element -- use admin:save-configuration to save the changes to the configuration or pass the configuration to other Admin API functions to make other changes. :) - https://docs.marklogic.com/7.0/admin:appserver-delete and, the error: [1.0-ml] XDMP-UNEXPECTED: (err:XPST0003) Unexpected token syntax error, unexpected QName_, expecting $end or SemiColon_ Stack Trace At line 10 column 2: In xdmp:eval( xquery version quot;1.0-mlquot;;#10;#10; import module na..., (), options xmlns=xdmp:evalisolationdifferent-transaction/isolationignore-ampstrue/i.../options) 8. return 9. admin:appserver-delete($config,admin:appserver-get-id($config, $groupid, secure2_app)) 10. admin:save-configuration($config) ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
Re: [MarkLogic Dev General] Forest Warning
Creating more forests won't help, because they'll all have the same in-memory limits and the rebalancer will run into the same issue as the reindexer if it tries to move the document to a different forest. Wayne. On 05/11/2015 11:18 AM, Indrajeet Verma wrote: Shashi, I am sure, this file size must be large. This is not good that you have created only one forest. There should be more than that. I would recommend around 8-10 forests. However these should be based on core of CPUs. per 2 core 1 forest but if your server is having 32 cores, 8-10 forest should be sufficient. Please correct somebody if I am wrong. As per my understanding you should to do following things to solve your problem, 1. Delete this large file size. I would not recommend to increase memory size just for the heck. Your performance will be degraded. 2. Create more forest and attach to the database. if you are using 7+, this automatically re-balance the data. 3. After re-balancing the content, you might perform manual merge to claim memory immediately. 3. Split your files into smaller size Regards, Indy On Mon, May 11, 2015 at 11:30 PM, Wayne Feick wayne.fe...@marklogic.com mailto:wayne.fe...@marklogic.com wrote: Looping in some additional information from private email. Since your list size is already configured to the maximum (32768), you could try to identify some index settings that you don't actually need and turn them off. If that isn't an option, you could try breaking it up into multiple documents, and then deleting the original document with xdmp:document-delete(). Wayne. On 05/11/2015 10:52 AM, Wayne Feick wrote: Hi Shashidhar, It sounds like the document was close to the limit when it was originally ingested, and that turning on additional index setting put it over the top. The error message says that your in-memory list storage is full, so if you go to the Admin UI and look at the database settings, you'll see an entry for in memory list size. Configure a larger value there and you should be able to finish your reindex. Wayne. On 04/23/2015 01:28 AM, Shashidhar Rao wrote: Hi, Can somebody help me how to fix this issue There is currently an XDMP-FORESTERR: Error in reindex of forest PROD_DB_1: XDMP-REINDEX: Error reindexing fn:doc(/home/data/Folder2/US07625699-20091201-T2.XML): XDMP-FRAGTOOLARGE: Fragment of /home/data/Folder2/US07625699-20091201-T2.XML too large for in-memory storage: XDMP-INMMLISTFULL: In-memory list storage full; list: table=100%, wordsused=50%, wordsfree=25%, overhead=25%; tree: table=0%, wordsused=6%, wordsfree=94%, overhead=0% exception. Information on this page may be missing. It says US07625699-20091201-T2.XML too large. what are the other options any suggestions would be helpful. Is deleting this file an option as the last resort? Thanks ___ General mailing list General@developer.marklogic.com mailto:General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general -- Wayne Feick Principal Engineer MarkLogic Corporation wayne.fe...@marklogic.com mailto:wayne.fe...@marklogic.com Phone: +1 650 655 2378 tel:%2B1%20650%20655%202378 www.marklogic.com http://www.marklogic.com This e-mail and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of this e-mail communication by others is strictly prohibited. If you are not the intended recipient, please notify us immediately by returning this message to the sender and delete all copies. Thank you for your cooperation. ___ General mailing list General@developer.marklogic.com mailto:General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general -- Wayne Feick Principal Engineer MarkLogic Corporation wayne.fe...@marklogic.com mailto:wayne.fe...@marklogic.com Phone: +1 650 655 2378 tel:%2B1%20650%20655%202378 www.marklogic.com http://www.marklogic.com This e-mail and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of this e-mail communication by others is strictly prohibited. If you are not the intended recipient, please notify us immediately by returning this message to the sender and delete all copies. Thank you for your cooperation.
[MarkLogic Dev General] bulk delete xquery syntax
Hi, Getting started with XQuery so that I can bulk delete a bunch of applications and associated files, but stuck on xquery syntax error. And, I'm also wondering, can I make this call and delete more than 1 app server simultaneously? As well as the app-server's associated DB, modules, and forests? This is the code I'm trying to run, and I found it in the link below: xquery version 1.0-ml; import module namespace admin = http://marklogic.com/xdmp/admin; at /MarkLogic/admin.xqy; let $config := admin:get-configuration() let $groupid := admin:group-get-id($config, Default) return admin:appserver-delete($config,admin:appserver-get-id($config, $groupid, secure2_app)) admin:save-configuration($config) (: returns the new configuration element -- use admin:save-configuration to save the changes to the configuration or pass the configuration to other Admin API functions to make other changes. :) - https://docs.marklogic.com/7.0/admin:appserver-delete and, the error: [1.0-ml] XDMP-UNEXPECTED: (err:XPST0003) Unexpected token syntax error, unexpected QName_, expecting $end or SemiColon_ Stack Trace At line 10 column 2: In xdmp:eval( xquery version quot;1.0-mlquot;;#10;#10; import module na..., (), options xmlns=xdmp:evalisolationdifferent-transaction/isolationignore-ampstrue/i.../options) 8. return 9. admin:appserver-delete($config,admin:appserver-get-id($config, $groupid, secure2_app)) 10. admin:save-configuration($config) ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
Re: [MarkLogic Dev General] wildcarded search with cts:field-value-query
And you should be able to express that with the value query by making your wildcarded expression something like: dev* * The first star means matches zero or more chars in that word, and the second * means matches zero or more words in the value. So that could match the following values: developer world devops but would not match: a developer’s world -Danny From: general-boun...@developer.marklogic.com [mailto:general-boun...@developer.marklogic.com] On Behalf Of Lanz Sent: Monday, May 11, 2015 7:54 AM To: MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] wildcarded search with cts:field-value-query Hi Florent, Thanks I understand the difference, actually we want to match word (and part of word) wherever they it is located in the value of the field. The cts:field-word-query is the right option in our case. Regards Lancelot On Mon, May 11, 2015 at 1:11 PM, Florent Georges li...@fgeorges.orgmailto:li...@fgeorges.org wrote: Hi Lancelot, Is the text you try to match the entire value of the element it is part of? Like the difference between: titleDevelopment is .../title and: titleThe development is .../title In the first case, it will match both queries (thanks to the wildcard at the end of the search term). In the second case, it will match only the word search (which matches words as in a search engine), not the value search (which matches elements values, like entire values). Regards, -- Florent Georges http://fgeorges.org/ http://h2oconsulting.be/ On 11 May 2015 at 12:47, Lanz wrote: hi all, I'm facing an issue using cts:field-value-query with wildcarded search in ML 7.0-5 Following search should match any field starting with 'dev' like development, etc. let $term := 'dev' return cts:search( fn:collection()/l:manifestation[l:issn], cts:field-value-query( 'field-quick-search', $term || '*', ('case-insensitive', 'diacritic-insensitive', 'unstemmed', 'wildcarded') ) ) But its returns nothing Fields 'field-quick-search' seems to be well configured to work with wildcarded search: Name: field-quick-search Path: *:title, *:subTitle, *:doiSuffix, *:code, *:userCode, *:issn, @id Includes: work, expression, manifestation Excludes (nothing) Index Settings: +word, +fast phrase, +fast case sensitive, +fast diacritic sensitive, +trailing wildcard, +three character, +two character The same search with cts:field-word-query works : let $term := 'dev' return cts:search( fn:collection()/l:manifestation[l:issn], cts:field-word-query( 'field-quick-search', $term || '*', ('case-insensitive', 'diacritic-insensitive', 'unstemmed', 'wildcarded') ) ) Any thoughts on what I might be doing wrong? Thanks Lancelot ___ General mailing list General@developer.marklogic.commailto:General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.commailto:General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
Re: [MarkLogic Dev General] Analyze-data.xqy with JSON data
Thanks Geert! I was looking forward to your response.I will use the tips below and let you know the outcome.regards,Puneet From: geert.jos...@marklogic.com To: general@developer.marklogic.com Date: Mon, 11 May 2015 14:48:14 + Subject: Re: [MarkLogic Dev General] Analyze-data.xqy with JSON data It goes wrong here: https://github.com/marklogic/slush-marklogic-node/blob/master/app/templates/rest-api/ext/analyze-data.xqy#L516 Replace: local-name($node) With: local-name-from-QName(node-name($node)) And you should have something that works again. I also suggest to turn on $use-path-indexes here: https://github.com/marklogic/slush-marklogic-node/blob/master/app/templates/rest-api/ext/analyze-data.xqy#L19 Cheers, Geert From: Geert Josten geert.jos...@marklogic.com Reply-To: MarkLogic Developer Discussion general@developer.marklogic.com Date: Monday, May 11, 2015 at 4:10 PM To: MarkLogic Developer Discussion general@developer.marklogic.com Subject: Re: [MarkLogic Dev General] Analyze-data.xqy with JSON data Hi there, Sounds like my work. I have a copy of it here: https://gist.github.com/grtjn, and it is also included in the slush template. Not sure which is latest, maybe neither. I’m a bit surprised it doesn’t work as is, but will have a look. I can imagine though, it will not provide a lot of suggestions for json data, I really created it a year ago, when EA1 wasn’t even out yet.. :) Stay tuned, Geert From: Yinyan guy yinyan...@hotmail.com Reply-To: MarkLogic Developer Discussion general@developer.marklogic.com Date: Monday, May 4, 2015 at 3:17 AM To: MarkLogic Developer Discussion general@developer.marklogic.com Subject: Re: [MarkLogic Dev General] Analyze-data.xqy with JSON data Got it! Thank You Eric. I did deploy this as an extension using the ./local deploy modules with definition in /rest-api/ext/analyze-data.xqy Regards, Puneet From: erik.hen...@marklogic.com To: general@developer.marklogic.com Date: Mon, 4 May 2015 00:47:55 + Subject: Re: [MarkLogic Dev General] Analyze-data.xqy with JSON data Hi, Puneet: Services available under /v1/resources are extensions that were installed under /v1/config/resources and not services implemented and delivered as part of the REST API itself. You should contact whoever wrote the analyze-data extension about modifying it to support JSON. Hoping that helps, Erik Hennum From:general-boun...@developer.marklogic.com [general-boun...@developer.marklogic.com] on behalf of Yinyan guy [yinyan...@hotmail.com] Sent: Sunday, May 03, 2015 4:21 PM To: MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] Analyze-data.xqy with JSON data HI Eric, Thank You for your response!. I am using following call http://localhost:8040/v1/resources/analyze-data; This has worked for me in this past for XML documents and when I was using ML7. Now I am using it on ML8 and with JSON documents. regards, Puneet From: erik.hen...@marklogic.com To: general@developer.marklogic.com Date: Sun, 3 May 2015 13:36:29 + Subject: Re: [MarkLogic Dev General] Analyze-data.xqy with JSON data Hi, Puneet: Can you provide an example of the REST API call that you're making to analyze data? Erik Hennum From:general-boun...@developer.marklogic.com [general-boun...@developer.marklogic.com] on behalf of Yinyan guy [yinyan...@hotmail.com] Sent: Saturday, May 02, 2015 6:35 AM To: general@developer.marklogic.com Subject: [MarkLogic Dev General] Analyze-data.xqy with JSON data Hi All, I am exploring the use of the analyze-data for index suggestion on my data. I am using ML8 and my data is JSON. On running analyze-data using the REST API, I got 400: Bad Request with message XDMP-UNEXPECTED: (err:XPST0003) Unexpected token syntax error, unexpected Rpar_, expecting Function30_ or Percent_ I guess it works only with xml data. is there a JSON version in works? Looking forward to hear from you. regards, Puneet ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
[MarkLogic Dev General] xdmp:node-replace
I am trying to use xdmp:node-replace to change a node value. Assume 'theTranslation' is a value I am matching in another document, and 'firmname' exists as a node or as text, depending on how I reference it, how do I use node-replace to change the value of theFirmname? xdmp:node-replace($theFirmname/text(),$translation/text()) The theTranslation firmname to insert is: text Freehill, Hogan Mahar, LLP The firmname to be changed is: text http://10.6.0.95:8000/qconsole/ Freehill, Hogan Mahar Or I could do it this way, (and I specified firmname to be a node value not text Will return a conflicting updates exception. xdmp:node-replace($theFirmname,firmname$translation/text()/firmname) [1.0-ml] XDMP-CONFLICTINGUPDATES: xdmp:eval(xquery version quot;1.0-mlquot;;#10;declare namespace html = ..., (), options xmlns=xdmp:evaldatabase4793418423724769125/databasemodules529627520719988.../options) -- Conflicting updates xdmp:node-replace(fn:doc(/olympus/pacer-xml/1155991_2:13-cv-00188)/case/parties/party[1]/representedBy/counsel/firmname, firmname$translation/text()/firmname) and xdmp:node-replace(fn:doc(/olympus/pacer-xml/1155991_2:13-cv-00188)/case/parties/party[1]/representedBy/counsel/firmname, firmname$translation/text()/firmname) I am feeling flustered. Is there an easier way to do this? ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
Re: [MarkLogic Dev General] xdmp:node-replace
Hi You get the error with just a single node-replace call? Are you sure each argument is a single node? If you left function-mapping on, it might not create the error you think it would, so you could try adding the option declare option xdmp:mapping false; to turn it off, then see if the error makes more sense. /ch ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
Re: [MarkLogic Dev General] Forest Warning
Looping in some additional information from private email. Since your list size is already configured to the maximum (32768), you could try to identify some index settings that you don't actually need and turn them off. If that isn't an option, you could try breaking it up into multiple documents, and then deleting the original document with xdmp:document-delete(). Wayne. On 05/11/2015 10:52 AM, Wayne Feick wrote: Hi Shashidhar, It sounds like the document was close to the limit when it was originally ingested, and that turning on additional index setting put it over the top. The error message says that your in-memory list storage is full, so if you go to the Admin UI and look at the database settings, you'll see an entry for in memory list size. Configure a larger value there and you should be able to finish your reindex. Wayne. On 04/23/2015 01:28 AM, Shashidhar Rao wrote: Hi, Can somebody help me how to fix this issue There is currently an XDMP-FORESTERR: Error in reindex of forest PROD_DB_1: XDMP-REINDEX: Error reindexing fn:doc(/home/data/Folder2/US07625699-20091201-T2.XML): XDMP-FRAGTOOLARGE: Fragment of /home/data/Folder2/US07625699-20091201-T2.XML too large for in-memory storage: XDMP-INMMLISTFULL: In-memory list storage full; list: table=100%, wordsused=50%, wordsfree=25%, overhead=25%; tree: table=0%, wordsused=6%, wordsfree=94%, overhead=0% exception. Information on this page may be missing. It says US07625699-20091201-T2.XML too large. what are the other options any suggestions would be helpful. Is deleting this file an option as the last resort? Thanks ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general -- Wayne Feick Principal Engineer MarkLogic Corporation wayne.fe...@marklogic.com Phone: +1 650 655 2378 www.marklogic.com This e-mail and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of this e-mail communication by others is strictly prohibited. If you are not the intended recipient, please notify us immediately by returning this message to the sender and delete all copies. Thank you for your cooperation. ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general -- Wayne Feick Principal Engineer MarkLogic Corporation wayne.fe...@marklogic.com Phone: +1 650 655 2378 www.marklogic.com This e-mail and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of this e-mail communication by others is strictly prohibited. If you are not the intended recipient, please notify us immediately by returning this message to the sender and delete all copies. Thank you for your cooperation. ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
Re: [MarkLogic Dev General] Forest Warning
Hi Shashidhar, It sounds like the document was close to the limit when it was originally ingested, and that turning on additional index setting put it over the top. The error message says that your in-memory list storage is full, so if you go to the Admin UI and look at the database settings, you'll see an entry for in memory list size. Configure a larger value there and you should be able to finish your reindex. Wayne. On 04/23/2015 01:28 AM, Shashidhar Rao wrote: Hi, Can somebody help me how to fix this issue There is currently an XDMP-FORESTERR: Error in reindex of forest PROD_DB_1: XDMP-REINDEX: Error reindexing fn:doc(/home/data/Folder2/US07625699-20091201-T2.XML): XDMP-FRAGTOOLARGE: Fragment of /home/data/Folder2/US07625699-20091201-T2.XML too large for in-memory storage: XDMP-INMMLISTFULL: In-memory list storage full; list: table=100%, wordsused=50%, wordsfree=25%, overhead=25%; tree: table=0%, wordsused=6%, wordsfree=94%, overhead=0% exception. Information on this page may be missing. It says US07625699-20091201-T2.XML too large. what are the other options any suggestions would be helpful. Is deleting this file an option as the last resort? Thanks ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general -- Wayne Feick Principal Engineer MarkLogic Corporation wayne.fe...@marklogic.com Phone: +1 650 655 2378 www.marklogic.com This e-mail and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of this e-mail communication by others is strictly prohibited. If you are not the intended recipient, please notify us immediately by returning this message to the sender and delete all copies. Thank you for your cooperation. ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general