Re: [MarkLogic Dev General] Forest Warning
Hi Wayne, Thank your for correcting me however I was just suggesting some options to improve performance after solving that issue as he said, he is only one forest in the DB with TB data. Regards, Indy On Mon, May 11, 2015 at 11:57 PM, Wayne Feick wayne.fe...@marklogic.com wrote: Creating more forests won't help, because they'll all have the same in-memory limits and the rebalancer will run into the same issue as the reindexer if it tries to move the document to a different forest. Wayne. On 05/11/2015 11:18 AM, Indrajeet Verma wrote: Shashi, I am sure, this file size must be large. This is not good that you have created only one forest. There should be more than that. I would recommend around 8-10 forests. However these should be based on core of CPUs. per 2 core 1 forest but if your server is having 32 cores, 8-10 forest should be sufficient. Please correct somebody if I am wrong. As per my understanding you should to do following things to solve your problem, 1. Delete this large file size. I would not recommend to increase memory size just for the heck. Your performance will be degraded. 2. Create more forest and attach to the database. if you are using 7+, this automatically re-balance the data. 3. After re-balancing the content, you might perform manual merge to claim memory immediately. 3. Split your files into smaller size Regards, Indy On Mon, May 11, 2015 at 11:30 PM, Wayne Feick wayne.fe...@marklogic.com wrote: Looping in some additional information from private email. Since your list size is already configured to the maximum (32768), you could try to identify some index settings that you don't actually need and turn them off. If that isn't an option, you could try breaking it up into multiple documents, and then deleting the original document with xdmp:document-delete(). Wayne. On 05/11/2015 10:52 AM, Wayne Feick wrote: Hi Shashidhar, It sounds like the document was close to the limit when it was originally ingested, and that turning on additional index setting put it over the top. The error message says that your in-memory list storage is full, so if you go to the Admin UI and look at the database settings, you'll see an entry for in memory list size. Configure a larger value there and you should be able to finish your reindex. Wayne. On 04/23/2015 01:28 AM, Shashidhar Rao wrote: Hi, Can somebody help me how to fix this issue There is currently an XDMP-FORESTERR: Error in reindex of forest PROD_DB_1: XDMP-REINDEX: Error reindexing fn:doc(/home/data/Folder2/US07625699-20091201-T2.XML): XDMP-FRAGTOOLARGE: Fragment of /home/data/Folder2/US07625699-20091201-T2.XML too large for in-memory storage: XDMP-INMMLISTFULL: In-memory list storage full; list: table=100%, wordsused=50%, wordsfree=25%, overhead=25%; tree: table=0%, wordsused=6%, wordsfree=94%, overhead=0% exception. Information on this page may be missing. It says US07625699-20091201-T2.XML too large. what are the other options any suggestions would be helpful. Is deleting this file an option as the last resort? Thanks ___ General mailing listgene...@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general -- Wayne Feick Principal Engineer MarkLogic corporationwayne.fe...@marklogic.com Phone: +1 650 655 2378www.marklogic.com This e-mail and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of this e-mail communication by others is strictly prohibited. If you are not the intended recipient, please notify us immediately by returning this message to the sender and delete all copies. Thank you for your cooperation. ___ General mailing listgene...@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general -- Wayne Feick Principal Engineer MarkLogic corporationwayne.fe...@marklogic.com Phone: +1 650 655 2378www.marklogic.com This e-mail and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of this e-mail communication by others is strictly prohibited. If you are not the intended recipient, please notify us immediately by returning this message to the sender and delete all copies. Thank you for your cooperation. ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing listgene...@developer.marklogic.com
Re: [MarkLogic Dev General] Forest Warning
Shashi, I am sure, this file size must be large. This is not good that you have created only one forest. There should be more than that. I would recommend around 8-10 forests. However these should be based on core of CPUs. per 2 core 1 forest but if your server is having 32 cores, 8-10 forest should be sufficient. Please correct somebody if I am wrong. As per my understanding you should to do following things to solve your problem, 1. Delete this large file size. I would not recommend to increase memory size just for the heck. Your performance will be degraded. 2. Create more forest and attach to the database. if you are using 7+, this automatically re-balance the data. 3. After re-balancing the content, you might perform manual merge to claim memory immediately. 3. Split your files into smaller size Regards, Indy On Mon, May 11, 2015 at 11:30 PM, Wayne Feick wayne.fe...@marklogic.com wrote: Looping in some additional information from private email. Since your list size is already configured to the maximum (32768), you could try to identify some index settings that you don't actually need and turn them off. If that isn't an option, you could try breaking it up into multiple documents, and then deleting the original document with xdmp:document-delete(). Wayne. On 05/11/2015 10:52 AM, Wayne Feick wrote: Hi Shashidhar, It sounds like the document was close to the limit when it was originally ingested, and that turning on additional index setting put it over the top. The error message says that your in-memory list storage is full, so if you go to the Admin UI and look at the database settings, you'll see an entry for in memory list size. Configure a larger value there and you should be able to finish your reindex. Wayne. On 04/23/2015 01:28 AM, Shashidhar Rao wrote: Hi, Can somebody help me how to fix this issue There is currently an XDMP-FORESTERR: Error in reindex of forest PROD_DB_1: XDMP-REINDEX: Error reindexing fn:doc(/home/data/Folder2/US07625699-20091201-T2.XML): XDMP-FRAGTOOLARGE: Fragment of /home/data/Folder2/US07625699-20091201-T2.XML too large for in-memory storage: XDMP-INMMLISTFULL: In-memory list storage full; list: table=100%, wordsused=50%, wordsfree=25%, overhead=25%; tree: table=0%, wordsused=6%, wordsfree=94%, overhead=0% exception. Information on this page may be missing. It says US07625699-20091201-T2.XML too large. what are the other options any suggestions would be helpful. Is deleting this file an option as the last resort? Thanks ___ General mailing listgene...@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general -- Wayne Feick Principal Engineer MarkLogic corporationwayne.fe...@marklogic.com Phone: +1 650 655 2378www.marklogic.com This e-mail and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of this e-mail communication by others is strictly prohibited. If you are not the intended recipient, please notify us immediately by returning this message to the sender and delete all copies. Thank you for your cooperation. ___ General mailing listgene...@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general -- Wayne Feick Principal Engineer MarkLogic corporationwayne.fe...@marklogic.com Phone: +1 650 655 2378www.marklogic.com This e-mail and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of this e-mail communication by others is strictly prohibited. If you are not the intended recipient, please notify us immediately by returning this message to the sender and delete all copies. Thank you for your cooperation. ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
Re: [MarkLogic Dev General] Forest Warning
Creating more forests won't help, because they'll all have the same in-memory limits and the rebalancer will run into the same issue as the reindexer if it tries to move the document to a different forest. Wayne. On 05/11/2015 11:18 AM, Indrajeet Verma wrote: Shashi, I am sure, this file size must be large. This is not good that you have created only one forest. There should be more than that. I would recommend around 8-10 forests. However these should be based on core of CPUs. per 2 core 1 forest but if your server is having 32 cores, 8-10 forest should be sufficient. Please correct somebody if I am wrong. As per my understanding you should to do following things to solve your problem, 1. Delete this large file size. I would not recommend to increase memory size just for the heck. Your performance will be degraded. 2. Create more forest and attach to the database. if you are using 7+, this automatically re-balance the data. 3. After re-balancing the content, you might perform manual merge to claim memory immediately. 3. Split your files into smaller size Regards, Indy On Mon, May 11, 2015 at 11:30 PM, Wayne Feick wayne.fe...@marklogic.com mailto:wayne.fe...@marklogic.com wrote: Looping in some additional information from private email. Since your list size is already configured to the maximum (32768), you could try to identify some index settings that you don't actually need and turn them off. If that isn't an option, you could try breaking it up into multiple documents, and then deleting the original document with xdmp:document-delete(). Wayne. On 05/11/2015 10:52 AM, Wayne Feick wrote: Hi Shashidhar, It sounds like the document was close to the limit when it was originally ingested, and that turning on additional index setting put it over the top. The error message says that your in-memory list storage is full, so if you go to the Admin UI and look at the database settings, you'll see an entry for in memory list size. Configure a larger value there and you should be able to finish your reindex. Wayne. On 04/23/2015 01:28 AM, Shashidhar Rao wrote: Hi, Can somebody help me how to fix this issue There is currently an XDMP-FORESTERR: Error in reindex of forest PROD_DB_1: XDMP-REINDEX: Error reindexing fn:doc(/home/data/Folder2/US07625699-20091201-T2.XML): XDMP-FRAGTOOLARGE: Fragment of /home/data/Folder2/US07625699-20091201-T2.XML too large for in-memory storage: XDMP-INMMLISTFULL: In-memory list storage full; list: table=100%, wordsused=50%, wordsfree=25%, overhead=25%; tree: table=0%, wordsused=6%, wordsfree=94%, overhead=0% exception. Information on this page may be missing. It says US07625699-20091201-T2.XML too large. what are the other options any suggestions would be helpful. Is deleting this file an option as the last resort? Thanks ___ General mailing list General@developer.marklogic.com mailto:General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general -- Wayne Feick Principal Engineer MarkLogic Corporation wayne.fe...@marklogic.com mailto:wayne.fe...@marklogic.com Phone: +1 650 655 2378 tel:%2B1%20650%20655%202378 www.marklogic.com http://www.marklogic.com This e-mail and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of this e-mail communication by others is strictly prohibited. If you are not the intended recipient, please notify us immediately by returning this message to the sender and delete all copies. Thank you for your cooperation. ___ General mailing list General@developer.marklogic.com mailto:General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general -- Wayne Feick Principal Engineer MarkLogic Corporation wayne.fe...@marklogic.com mailto:wayne.fe...@marklogic.com Phone: +1 650 655 2378 tel:%2B1%20650%20655%202378 www.marklogic.com http://www.marklogic.com This e-mail and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of this e-mail communication by others is strictly prohibited. If you are not the intended recipient, please notify us immediately by returning this message to the sender and delete all copies. Thank you for your cooperation.
Re: [MarkLogic Dev General] Forest Warning
Looping in some additional information from private email. Since your list size is already configured to the maximum (32768), you could try to identify some index settings that you don't actually need and turn them off. If that isn't an option, you could try breaking it up into multiple documents, and then deleting the original document with xdmp:document-delete(). Wayne. On 05/11/2015 10:52 AM, Wayne Feick wrote: Hi Shashidhar, It sounds like the document was close to the limit when it was originally ingested, and that turning on additional index setting put it over the top. The error message says that your in-memory list storage is full, so if you go to the Admin UI and look at the database settings, you'll see an entry for in memory list size. Configure a larger value there and you should be able to finish your reindex. Wayne. On 04/23/2015 01:28 AM, Shashidhar Rao wrote: Hi, Can somebody help me how to fix this issue There is currently an XDMP-FORESTERR: Error in reindex of forest PROD_DB_1: XDMP-REINDEX: Error reindexing fn:doc(/home/data/Folder2/US07625699-20091201-T2.XML): XDMP-FRAGTOOLARGE: Fragment of /home/data/Folder2/US07625699-20091201-T2.XML too large for in-memory storage: XDMP-INMMLISTFULL: In-memory list storage full; list: table=100%, wordsused=50%, wordsfree=25%, overhead=25%; tree: table=0%, wordsused=6%, wordsfree=94%, overhead=0% exception. Information on this page may be missing. It says US07625699-20091201-T2.XML too large. what are the other options any suggestions would be helpful. Is deleting this file an option as the last resort? Thanks ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general -- Wayne Feick Principal Engineer MarkLogic Corporation wayne.fe...@marklogic.com Phone: +1 650 655 2378 www.marklogic.com This e-mail and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of this e-mail communication by others is strictly prohibited. If you are not the intended recipient, please notify us immediately by returning this message to the sender and delete all copies. Thank you for your cooperation. ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general -- Wayne Feick Principal Engineer MarkLogic Corporation wayne.fe...@marklogic.com Phone: +1 650 655 2378 www.marklogic.com This e-mail and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of this e-mail communication by others is strictly prohibited. If you are not the intended recipient, please notify us immediately by returning this message to the sender and delete all copies. Thank you for your cooperation. ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
Re: [MarkLogic Dev General] Forest Warning
Hi Shashidhar, It sounds like the document was close to the limit when it was originally ingested, and that turning on additional index setting put it over the top. The error message says that your in-memory list storage is full, so if you go to the Admin UI and look at the database settings, you'll see an entry for in memory list size. Configure a larger value there and you should be able to finish your reindex. Wayne. On 04/23/2015 01:28 AM, Shashidhar Rao wrote: Hi, Can somebody help me how to fix this issue There is currently an XDMP-FORESTERR: Error in reindex of forest PROD_DB_1: XDMP-REINDEX: Error reindexing fn:doc(/home/data/Folder2/US07625699-20091201-T2.XML): XDMP-FRAGTOOLARGE: Fragment of /home/data/Folder2/US07625699-20091201-T2.XML too large for in-memory storage: XDMP-INMMLISTFULL: In-memory list storage full; list: table=100%, wordsused=50%, wordsfree=25%, overhead=25%; tree: table=0%, wordsused=6%, wordsfree=94%, overhead=0% exception. Information on this page may be missing. It says US07625699-20091201-T2.XML too large. what are the other options any suggestions would be helpful. Is deleting this file an option as the last resort? Thanks ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general -- Wayne Feick Principal Engineer MarkLogic Corporation wayne.fe...@marklogic.com Phone: +1 650 655 2378 www.marklogic.com This e-mail and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of this e-mail communication by others is strictly prohibited. If you are not the intended recipient, please notify us immediately by returning this message to the sender and delete all copies. Thank you for your cooperation. ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
[MarkLogic Dev General] Forest Warning
Hi, Can somebody help me how to fix this issue There is currently an XDMP-FORESTERR: Error in reindex of forest PROD_DB_1: XDMP-REINDEX: Error reindexing fn:doc(/home/data/Folder2/US07625699-20091201-T2.XML): XDMP-FRAGTOOLARGE: Fragment of /home/data/Folder2/US07625699-20091201-T2.XML too large for in-memory storage: XDMP-INMMLISTFULL: In-memory list storage full; list: table=100%, wordsused=50%, wordsfree=25%, overhead=25%; tree: table=0%, wordsused=6%, wordsfree=94%, overhead=0% exception. Information on this page may be missing. It says US07625699-20091201-T2.XML too large. what are the other options any suggestions would be helpful. Is deleting this file an option as the last resort? Thanks ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
