Re: [Sedna-discussion] question regarding data files [extremely big]
Hi Ivan,
Things get worse now. On Friday I restored the database from the two-days
old backup (because one-day old backup had the same problem with growing tmp
file).
However, today the problem reappeared. Meanwhile, I've found a number of
commands that lead trigger the tmp file to grow fast. These bad commands
are:
document($documents)/documents/collection[@name=vpDita/technicalSummary]
/document[@name=ts_vp_BUK7Y4R8-60E.html]
document($documents)/documents/collection[@name=parametricHeaders]/docum
ent[@name=30905.xml]
document($documents)/documents/collection[@name=basicTypes/released]/doc
ument[@name=1N4148.xml]
//and of course
document($documents)
I just wanted to be sure that it didn't depend on a specific document name.
Other commands seem to work fine:
doc(..,..)
doc($collections)
doc($indexes)
doc($modules)
doc($db_security_data)
Do you have any ideas why this may happen and how to solve it? The data
files are huge (15GB) but I've shared several event.log files where the
problem occurs: https://www.dropbox.com/s/w5n0lowdlxhiapi/sedna_problem.zip
Best regards,
Ivan Lagunov
From: Ivan Lagunov [mailto:lagi...@gmail.com]
Sent: Friday, February 15, 2013 12:47 PM
To: 'Ivan Shcheklein'
Cc: Robby Pelssers; 'sedna-discussion'
Subject: RE: [Sedna-discussion] question regarding data files [extremely
big]
Hi Ivan,
I've tried both your suggestions. Everything leads to the same issue
repeating. Even exporting data with se_exp doesn't work - no data is being
exported but the tmp file starts growing. I'll try to restore from the last
backup now.
Best regards,
Ivan Lagunov
From: Ivan Shcheklein [mailto:shchekl...@gmail.com]
Sent: Friday, February 15, 2013 12:07 PM
To: Ivan Lagunov
Cc: Robby Pelssers; sedna-discussion
Subject: Re: [Sedna-discussion] question regarding data files [extremely
big]
Hi Ivan,
Seems that this query is the problem:
document($documents)/documents/collection[@name=vpDita/technicalSummary]
/document[@name=ts_vp_BUK7Y4R8-60E.html]
Can you try to run the same database in isolated environment (no external
workload) and try to run doc('$documents') first, then this query?
Also, there is a possibility that database is corrupted (after the first 'No
space left' error happened). Try to restart it from the backup or make
se_exp export/restore.
Ivan
On Fri, Feb 15, 2013 at 2:51 PM, Ivan Lagunov lagi...@gmail.com wrote:
Hi Ivan,
Here is the zipped event.log file.
Best regards,
Ivan Lagunov
From: Ivan Lagunov [mailto:lagi...@gmail.com]
Sent: Friday, February 15, 2013 11:50 AM
To: Robby Pelssers; 'Ivan Shcheklein'
Cc: sedna-discussion@lists.sourceforge.net
Subject: RE: [Sedna-discussion] question regarding data files [extremely
big]
Hi Ivan,
Could you check the attached event.log and help us resolve the issue?
The database was restarted at about 11:00 due to the fatal error:
SYS 15/02/2013 10:58:11 (SM nxp pid=27220) [uhdd.c:uWriteFile:237]: write
(code = 28): No space left on device
FATAL 15/02/2013 10:58:11 (SM nxp pid=27220)
[bm_core.cpp:write_block_addr:233]: Cannot write block
INFO 15/02/2013 11:00:01 (GOV pid=28048)
[gov_functions.cpp:log_out_system_information:87]: SEDNA version is 3.5.161
(64bit Release)
INFO 15/02/2013 11:00:01 (GOV pid=28048)
[gov_functions.cpp:log_out_system_information:95]: System: Linux
2.6.18-238.12.1.el5 x86_64
It must have crashed at that moment probably because we have a script that
automatically checks and starts the database every 5 minutes.
Then the tmp file started growing back to 46GB at about 11:08:
LOG 15/02/2013 11:08:22 (SM nxp pid=28081)
[blk_mngmt.cpp:extend_tmp_file:629]: Temp file has been extended, size:
c80
LOG 15/02/2013 11:08:24 (SM nxp pid=28081)
[blk_mngmt.cpp:extend_tmp_file:629]: Temp file has been extended, size:
12c0
LOG 15/02/2013 11:08:25 (SM nxp pid=28081)
[blk_mngmt.cpp:extend_tmp_file:629]: Temp file has been extended, size:
1900
So it happens regularly now, the Sedna goes down, then being restarted and
the file starts growing to 46GB again. It looks like there is some huge
transaction that cannot be completed. Is there any way to locate it and kill
that transaction or resolve this somehow to prevent growing?
Best regards,
Ivan Lagunov
From: Robby Pelssers [mailto:robby.pelss...@nxp.com]
Sent: Friday, February 15, 2013 11:27 AM
To: Ivan Shcheklein
Cc: sedna-discussion@lists.sourceforge.net
Subject: Re: [Sedna-discussion] question regarding data files [extremely
big]
Hi Ivan,
Issue solved indeed. Thx again for the quick reply !!
Robby
From: Ivan Shcheklein [mailto:shchekl...@gmail.com]
Sent: Friday, February 15, 2013 10:52 AM
To: Robby Pelssers
Cc: sedna-discussion@lists.sourceforge.net
Subject: Re: [Sedna-discussion] question regarding data files [extremely
big]
Robby,
Thx for the quick reply. I was looking into the admin guide
http://www.sedna.org
Re: [Sedna-discussion] question regarding data files [extremely big]
Hi Ivan,
Most likely there is a bug. Can you give some data and queries to reproduce
this issue?
Ivan Shcheklein,
Sedna Team
On Mon, Feb 18, 2013 at 6:07 PM, Ivan Lagunov lagi...@gmail.com wrote:
Hi Ivan,
** **
Things get worse now. On Friday I restored the database from the two-days
old backup (because one-day old backup had the same problem with growing
tmp file).
** **
However, today the problem reappeared. Meanwhile, I’ve found a number of
commands that lead trigger the tmp file to grow fast. These “bad” commands
are:
** **
document($documents)/documents/collection[@name=vpDita/technicalSummary]/document[@name=ts_vp_BUK7Y4R8-60E.html]
document($documents)/documents/collection[@name=parametricHeaders]/document[@name=30905.xml]
document($documents)/documents/collection[@name=basicTypes/released]/document[@name=1N4148.xml]
//and of course
document($documents)
** **
I just wanted to be sure that it didn’t depend on a specific document
name. Other commands seem to work fine:
doc(“..”,”..”)
doc(“$collections”)
doc(“$indexes”)
doc(“$modules”)
doc(“$db_security_data”)
** **
Do you have any ideas why this may happen and how to solve it? The data
files are huge (15GB) but I’ve shared several event.log files where the
problem occurs:
https://www.dropbox.com/s/w5n0lowdlxhiapi/sedna_problem.zip
** **
Best regards,
Ivan Lagunov
** **
*From:* Ivan Lagunov [mailto:lagi...@gmail.com]
*Sent:* Friday, February 15, 2013 12:47 PM
*To:* 'Ivan Shcheklein'
*Cc:* Robby Pelssers; 'sedna-discussion'
*Subject:* RE: [Sedna-discussion] question regarding data files
[extremely big]
** **
Hi Ivan,
** **
I’ve tried both your suggestions. Everything leads to the same issue
repeating. Even exporting data with se_exp doesn’t work – no data is being
exported but the tmp file starts growing. I’ll try to restore from the last
backup now.
** **
Best regards,
Ivan Lagunov
** **
*From:* Ivan Shcheklein [mailto:shchekl...@gmail.comshchekl...@gmail.com]
*Sent:* Friday, February 15, 2013 12:07 PM
*To:* Ivan Lagunov
*Cc:* Robby Pelssers; sedna-discussion
*Subject:* Re: [Sedna-discussion] question regarding data files
[extremely big]
** **
Hi Ivan,
** **
Seems that this query is the problem:
** **
document($documents)/documents/collection[@name=vpDita/technicalSummary]/document[@name=ts_vp_BUK7Y4R8-60E.html]
** **
Can you try to run the same database in isolated environment (no external
workload) and try to run doc('$documents') first, then this query?
** **
Also, there is a possibility that database is corrupted (after the first
'No space left' error happened). Try to restart it from the backup or make
se_exp export/restore.
** **
Ivan
On Fri, Feb 15, 2013 at 2:51 PM, Ivan Lagunov lagi...@gmail.com wrote:**
**
Hi Ivan,
Here is the zipped event.log file.
Best regards,
Ivan Lagunov
*From:* Ivan Lagunov [mailto:lagi...@gmail.com]
*Sent:* Friday, February 15, 2013 11:50 AM
*To:* Robby Pelssers; 'Ivan Shcheklein'
*Cc:* sedna-discussion@lists.sourceforge.net
*Subject:* RE: [Sedna-discussion] question regarding data files
[extremely big]
Hi Ivan,
Could you check the attached event.log and help us resolve the issue?
The database was restarted at about 11:00 due to the fatal error:
SYS 15/02/2013 10:58:11 (SM nxp pid=27220) [uhdd.c:uWriteFile:237]:
write (code = 28): No space left on device
FATAL 15/02/2013 10:58:11 (SM nxp pid=27220)
[bm_core.cpp:write_block_addr:233]: Cannot write block
INFO 15/02/2013 11:00:01 (GOV pid=28048)
[gov_functions.cpp:log_out_system_information:87]: SEDNA version is 3.5.161
(64bit Release)
INFO 15/02/2013 11:00:01 (GOV pid=28048)
[gov_functions.cpp:log_out_system_information:95]: System: Linux
2.6.18-238.12.1.el5 x86_64
It must have crashed at that moment probably because we have a script that
automatically checks and starts the database every 5 minutes.
Then the tmp file started growing back to 46GB at about 11:08:
LOG 15/02/2013 11:08:22 (SM nxp pid=28081)
[blk_mngmt.cpp:extend_tmp_file:629]: Temp file has been extended, size:
c80
LOG 15/02/2013 11:08:24 (SM nxp pid=28081)
[blk_mngmt.cpp:extend_tmp_file:629]: Temp file has been extended, size:
12c0
LOG 15/02/2013 11:08:25 (SM nxp pid=28081)
[blk_mngmt.cpp:extend_tmp_file:629]: Temp file has been extended, size:
1900
So it happens regularly now, the Sedna goes down, then being restarted and
the file starts growing to 46GB again. It looks like there is some huge
transaction that cannot be completed. Is there any way to locate it and
kill that transaction or resolve this somehow to prevent growing
Re: [Sedna-discussion] question regarding data files [extremely big]
Hello! Some time ago I reported the bug related huge tmp file sizehttp://sourceforge.net/p/sedna/bugs/90/. On Mon, Feb 18, 2013 at 6:07 PM, Ivan Lagunov lagi...@gmail.com wrote: Hi Ivan, ** ** Things get worse now. On Friday I restored the database from the two-days old backup (because one-day old backup had the same problem with growing tmp file). ** ** However, today the problem reappeared. Meanwhile, I’ve found a number of commands that lead trigger the tmp file to grow fast. These “bad” commands are: ** ** document($documents)/documents/collection[@name=vpDita/technicalSummary]/document[@name=ts_vp_BUK7Y4R8-60E.html] document($documents)/documents/collection[@name=parametricHeaders]/document[@name=30905.xml] document($documents)/documents/collection[@name=basicTypes/released]/document[@name=1N4148.xml] //and of course document($documents) [...] -- Ruvim -- The Go Parallel Website, sponsored by Intel - in partnership with Geeknet, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials, tech docs, whitepapers, evaluation guides, and opinion stories. Check out the most recent posts - join the conversation now. http://goparallel.sourceforge.net/___ Sedna-discussion mailing list Sedna-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sedna-discussion
Re: [Sedna-discussion] question regarding data files [extremely big]
Ruvim, thank you. Until now I thought that the reason for your bug was different. We'll take a look at it once again. On Mon, Feb 18, 2013 at 6:33 PM, Ruvim Pinka ruvim.pi...@gmail.com wrote: Hello! Some time ago I reported the bug related huge tmp file sizehttp://sourceforge.net/p/sedna/bugs/90/. On Mon, Feb 18, 2013 at 6:07 PM, Ivan Lagunov lagi...@gmail.com wrote: Hi Ivan, ** ** Things get worse now. On Friday I restored the database from the two-days old backup (because one-day old backup had the same problem with growing tmp file). ** ** However, today the problem reappeared. Meanwhile, I’ve found a number of commands that lead trigger the tmp file to grow fast. These “bad” commands are: ** ** document($documents)/documents/collection[@name=vpDita/technicalSummary]/document[@name=ts_vp_BUK7Y4R8-60E.html] document($documents)/documents/collection[@name=parametricHeaders]/document[@name=30905.xml] document($documents)/documents/collection[@name=basicTypes/released]/document[@name=1N4148.xml] //and of course document($documents) [...] -- Ruvim -- The Go Parallel Website, sponsored by Intel - in partnership with Geeknet, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials, tech docs, whitepapers, evaluation guides, and opinion stories. Check out the most recent posts - join the conversation now. http://goparallel.sourceforge.net/ ___ Sedna-discussion mailing list Sedna-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sedna-discussion -- The Go Parallel Website, sponsored by Intel - in partnership with Geeknet, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials, tech docs, whitepapers, evaluation guides, and opinion stories. Check out the most recent posts - join the conversation now. http://goparallel.sourceforge.net/___ Sedna-discussion mailing list Sedna-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sedna-discussion
Re: [Sedna-discussion] question regarding data files [extremely big]
Hi Ruvim, Thanks for the addition! However, it looks a bit different. You faced a finite increase of tmp file size whereas we face infinite increasing. After a single query it keeps growing in size until no free space left on the disk. Best regards, Ivan Lagunov From: Ruvim Pinka [mailto:ruvim.pi...@gmail.com] Sent: Monday, February 18, 2013 3:34 PM To: Ivan Lagunov Cc: sedna-discussion Subject: Re: [Sedna-discussion] question regarding data files [extremely big] Hello! Some time ago I reported the bug related huge tmp file size http://sourceforge.net/p/sedna/bugs/90/ . On Mon, Feb 18, 2013 at 6:07 PM, Ivan Lagunov lagi...@gmail.com wrote: Hi Ivan, Things get worse now. On Friday I restored the database from the two-days old backup (because one-day old backup had the same problem with growing tmp file). However, today the problem reappeared. Meanwhile, I’ve found a number of commands that lead trigger the tmp file to grow fast. These “bad” commands are: document($documents)/documents/collection[@name=vpDita/technicalSummary]/document[@name=ts_vp_BUK7Y4R8-60E.html] document($documents)/documents/collection[@name=parametricHeaders]/document[@name=30905.xml] document($documents)/documents/collection[@name=basicTypes/released]/document[@name=1N4148.xml] //and of course document($documents) [...] -- Ruvim -- The Go Parallel Website, sponsored by Intel - in partnership with Geeknet, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials, tech docs, whitepapers, evaluation guides, and opinion stories. Check out the most recent posts - join the conversation now. http://goparallel.sourceforge.net/___ Sedna-discussion mailing list Sedna-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sedna-discussion
Re: [Sedna-discussion] question regarding data files [extremely big]
Hi Robby, Actually, you can. Just restart database - setmp should be restored to its default size. To get statistics, try: - $schema_name document – the descriptive schema of the document or collection named name; - $document_name document – statistical information about the document named name; - $collection_name document – statistical information about the collection named name. details there: http://sedna.org/progguide/ProgGuidesu8.html#x14-580002.5.6 You can also try to determine if your data (sedata) file is fragmented. Use se_exp to export and then import data into clean database. Compare new sedata size with the old one. Ivan Shcheklein, Sedna Team On Fri, Feb 15, 2013 at 1:16 PM, Robby Pelssers robby.pelss...@nxp.comwrote: Hi all, Seems like some files are growing out of proportion for the database nxp. It kind of surprises me as I don't expect we have that much data. But to find a possible bottleneck or get more insight into which collection might be this big, how can I get some statistics? And I guess we can't clean up one of these files, right? Thx in advance, Robby drwxr-xr-x 2 pxprod1 spider4096 Feb 15 05:40 ./ drwxr-xr-x 3 pxprod1 spider 983040 Feb 15 08:59 ../ -rw-rw 1 pxprod1 spider 1313106 Feb 15 10:04 nxp.15516.llog -rw-rw 1 pxprod1 spider 15309275136 Feb 15 10:04 nxp.sedata -rw-rw 1 pxprod1 spider 31352422400 Feb 15 10:08 nxp.setmp pxprod1@nlscli71:/appl/spider_prod/sedna/pxprod1/sedna35/data/nxp_files pxprod1@nlscli71:/appl/spider_prod/sedna/pxprod1/sedna35/data/nxp_filesdu -hs * 1.3Mnxp.15516.llog 15G nxp.sedata 41G nxp.setmp -- Free Next-Gen Firewall Hardware Offer Buy your Sophos next-gen firewall before the end March 2013 and get the hardware for free! Learn more. http://p.sf.net/sfu/sophos-d2d-feb ___ Sedna-discussion mailing list Sedna-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sedna-discussion -- Free Next-Gen Firewall Hardware Offer Buy your Sophos next-gen firewall before the end March 2013 and get the hardware for free! Learn more. http://p.sf.net/sfu/sophos-d2d-feb___ Sedna-discussion mailing list Sedna-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sedna-discussion
Re: [Sedna-discussion] question regarding data files [extremely big]
Hi Ivan, Thx for the quick reply. I was looking into the admin guide http://www.sedna.org/adminguide/AdminGuidesu3.html and noticed that it’s possible to set the max file size during creation. That would probably be a safer future proof way to prevent our disk from filling up. But what would be the scenario to accomplish this as our database is obviously already created. I just want to make sure I don’t miss out on anything here as obviously we want the data, xquery libraries and indexes to be fully restored after this procedure. Robby From: Ivan Shcheklein [mailto:shchekl...@gmail.com] Sent: Friday, February 15, 2013 10:23 AM To: Robby Pelssers Cc: sedna-discussion@lists.sourceforge.net Subject: Re: [Sedna-discussion] question regarding data files [extremely big] Hi Robby, Actually, you can. Just restart database - setmp should be restored to its default size. To get statistics, try: * $schema_name document – the descriptive schema of the document or collection named name; * $document_name document – statistical information about the document named name; * $collection_name document – statistical information about the collection named name. details there: http://sedna.org/progguide/ProgGuidesu8.html#x14-580002.5.6 You can also try to determine if your data (sedata) file is fragmented. Use se_exp to export and then import data into clean database. Compare new sedata size with the old one. Ivan Shcheklein, Sedna Team On Fri, Feb 15, 2013 at 1:16 PM, Robby Pelssers robby.pelss...@nxp.commailto:robby.pelss...@nxp.com wrote: Hi all, Seems like some files are growing out of proportion for the database nxp. It kind of surprises me as I don't expect we have that much data. But to find a possible bottleneck or get more insight into which collection might be this big, how can I get some statistics? And I guess we can't clean up one of these files, right? Thx in advance, Robby drwxr-xr-x 2 pxprod1 spider4096 Feb 15 05:40 ./ drwxr-xr-x 3 pxprod1 spider 983040 Feb 15 08:59 ../ -rw-rw 1 pxprod1 spider 1313106 Feb 15 10:04 nxp.15516.llog -rw-rw 1 pxprod1 spider 15309275136 Feb 15 10:04 nxp.sedata -rw-rw 1 pxprod1 spider 31352422400 Feb 15 10:08 nxp.setmp pxprod1@nlscli71:/appl/spider_prod/sedna/pxprod1/sedna35/data/nxp_files pxprod1@nlscli71:/appl/spider_prod/sedna/pxprod1/sedna35/data/nxp_filesdu -hs * 1.3Mnxp.15516.llog 15G nxp.sedata 41G nxp.setmp -- Free Next-Gen Firewall Hardware Offer Buy your Sophos next-gen firewall before the end March 2013 and get the hardware for free! Learn more. http://p.sf.net/sfu/sophos-d2d-feb ___ Sedna-discussion mailing list Sedna-discussion@lists.sourceforge.netmailto:Sedna-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sedna-discussion -- Free Next-Gen Firewall Hardware Offer Buy your Sophos next-gen firewall before the end March 2013 and get the hardware for free! Learn more. http://p.sf.net/sfu/sophos-d2d-feb___ Sedna-discussion mailing list Sedna-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sedna-discussion
Re: [Sedna-discussion] question regarding data files [extremely big]
Robby, Thx for the quick reply. I was looking into the admin guide http://www.sedna.org/adminguide/AdminGuidesu3.html and noticed that it’s possible to set the max file size during creation. Yes, it's possible, but I think transactions will be rolled back if they failed to extend setmp. In this case you will need to restart (not recreate, just restart with se_smsd, se_sm commands) database anyway. Probably, we should try to implement vacuum() function which at least trims setmp without the need to restart the database. But what would be the scenario to accomplish this as our database is obviously already created. To clean setmp you don't need to recreate database - just restart it with se_smsd, se_sm. I just want to make sure I don’t miss out on anything here as obviously we want the data, xquery libraries and indexes to be fully restored after this procedure. Just try it (if you really want to see if it's possible to shrink 'sedata' file at all). se_exp should work fine in most cases. Just one note from the documentation: ... current version of the se_exp utility doesn’t support exporting/importing triggers, documents with multiple roots and empty documents (empty documents nodes and document nodes with multiple root elements are allowed by XQuery data model but cannot be serialized as is without modifications). Ivan ** *From:* Ivan Shcheklein [mailto:shchekl...@gmail.com] *Sent:* Friday, February 15, 2013 10:23 AM *To:* Robby Pelssers *Cc:* sedna-discussion@lists.sourceforge.net *Subject:* Re: [Sedna-discussion] question regarding data files [extremely big] ** ** Hi Robby, ** ** Actually, you can. Just restart database - setmp should be restored to its default size. ** ** To get statistics, try: - *$schema_name* document – the descriptive schema of the document or collection named *name*; - *$document_name* document – statistical information about the document named *name*; - *$collection_name* document – statistical information about the collection named *name*. details there: http://sedna.org/progguide/ProgGuidesu8.html#x14-580002.5.6 ** ** You can also try to determine if your data (sedata) file is fragmented. Use se_exp to export and then import data into clean database. Compare new sedata size with the old one. ** ** ** ** Ivan Shcheklein, Sedna Team ** ** On Fri, Feb 15, 2013 at 1:16 PM, Robby Pelssers robby.pelss...@nxp.com wrote: Hi all, Seems like some files are growing out of proportion for the database nxp. It kind of surprises me as I don't expect we have that much data. But to find a possible bottleneck or get more insight into which collection might be this big, how can I get some statistics? And I guess we can't clean up one of these files, right? Thx in advance, Robby drwxr-xr-x 2 pxprod1 spider4096 Feb 15 05:40 ./ drwxr-xr-x 3 pxprod1 spider 983040 Feb 15 08:59 ../ -rw-rw 1 pxprod1 spider 1313106 Feb 15 10:04 nxp.15516.llog -rw-rw 1 pxprod1 spider 15309275136 Feb 15 10:04 nxp.sedata -rw-rw 1 pxprod1 spider 31352422400 Feb 15 10:08 nxp.setmp pxprod1@nlscli71:/appl/spider_prod/sedna/pxprod1/sedna35/data/nxp_files pxprod1@nlscli71:/appl/spider_prod/sedna/pxprod1/sedna35/data/nxp_filesdu -hs * 1.3Mnxp.15516.llog 15G nxp.sedata 41G nxp.setmp -- Free Next-Gen Firewall Hardware Offer Buy your Sophos next-gen firewall before the end March 2013 and get the hardware for free! Learn more. http://p.sf.net/sfu/sophos-d2d-feb ___ Sedna-discussion mailing list Sedna-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sedna-discussion ** ** -- Free Next-Gen Firewall Hardware Offer Buy your Sophos next-gen firewall before the end March 2013 and get the hardware for free! Learn more. http://p.sf.net/sfu/sophos-d2d-feb___ Sedna-discussion mailing list Sedna-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sedna-discussion
Re: [Sedna-discussion] question regarding data files [extremely big]
Hi Ivan, Issue solved indeed. Thx again for the quick reply !! Robby From: Ivan Shcheklein [mailto:shchekl...@gmail.com] Sent: Friday, February 15, 2013 10:52 AM To: Robby Pelssers Cc: sedna-discussion@lists.sourceforge.net Subject: Re: [Sedna-discussion] question regarding data files [extremely big] Robby, Thx for the quick reply. I was looking into the admin guide http://www.sedna.org/adminguide/AdminGuidesu3.html and noticed that it’s possible to set the max file size during creation. Yes, it's possible, but I think transactions will be rolled back if they failed to extend setmp. In this case you will need to restart (not recreate, just restart with se_smsd, se_sm commands) database anyway. Probably, we should try to implement vacuum() function which at least trims setmp without the need to restart the database. But what would be the scenario to accomplish this as our database is obviously already created. To clean setmp you don't need to recreate database - just restart it with se_smsd, se_sm. I just want to make sure I don’t miss out on anything here as obviously we want the data, xquery libraries and indexes to be fully restored after this procedure. Just try it (if you really want to see if it's possible to shrink 'sedata' file at all). se_exp should work fine in most cases. Just one note from the documentation: ... current version of the se_exp utility doesn’t support exporting/importing triggers, documents with multiple roots and empty documents (empty documents nodes and document nodes with multiple root elements are allowed by XQuery data model but cannot be serialized as is without modifications). Ivan From: Ivan Shcheklein [mailto:shchekl...@gmail.commailto:shchekl...@gmail.com] Sent: Friday, February 15, 2013 10:23 AM To: Robby Pelssers Cc: sedna-discussion@lists.sourceforge.netmailto:sedna-discussion@lists.sourceforge.net Subject: Re: [Sedna-discussion] question regarding data files [extremely big] Hi Robby, Actually, you can. Just restart database - setmp should be restored to its default size. To get statistics, try: * $schema_name document – the descriptive schema of the document or collection named name; * $document_name document – statistical information about the document named name; * $collection_name document – statistical information about the collection named name. details there: http://sedna.org/progguide/ProgGuidesu8.html#x14-580002.5.6 You can also try to determine if your data (sedata) file is fragmented. Use se_exp to export and then import data into clean database. Compare new sedata size with the old one. Ivan Shcheklein, Sedna Team On Fri, Feb 15, 2013 at 1:16 PM, Robby Pelssers robby.pelss...@nxp.commailto:robby.pelss...@nxp.com wrote: Hi all, Seems like some files are growing out of proportion for the database nxp. It kind of surprises me as I don't expect we have that much data. But to find a possible bottleneck or get more insight into which collection might be this big, how can I get some statistics? And I guess we can't clean up one of these files, right? Thx in advance, Robby drwxr-xr-x 2 pxprod1 spider4096 Feb 15 05:40 ./ drwxr-xr-x 3 pxprod1 spider 983040 Feb 15 08:59 ../ -rw-rw 1 pxprod1 spider 1313106 Feb 15 10:04 nxp.15516.llog -rw-rw 1 pxprod1 spider 15309275136 Feb 15 10:04 nxp.sedata -rw-rw 1 pxprod1 spider 31352422400 Feb 15 10:08 nxp.setmp pxprod1@nlscli71:/appl/spider_prod/sedna/pxprod1/sedna35/data/nxp_files pxprod1@nlscli71:/appl/spider_prod/sedna/pxprod1/sedna35/data/nxp_filesdu -hs * 1.3Mnxp.15516.llog 15G nxp.sedata 41G nxp.setmp -- Free Next-Gen Firewall Hardware Offer Buy your Sophos next-gen firewall before the end March 2013 and get the hardware for free! Learn more. http://p.sf.net/sfu/sophos-d2d-feb ___ Sedna-discussion mailing list Sedna-discussion@lists.sourceforge.netmailto:Sedna-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sedna-discussion -- Free Next-Gen Firewall Hardware Offer Buy your Sophos next-gen firewall before the end March 2013 and get the hardware for free! Learn more. http://p.sf.net/sfu/sophos-d2d-feb___ Sedna-discussion mailing list Sedna-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sedna-discussion
Re: [Sedna-discussion] question regarding data files [extremely big]
Hi Ivan, Could you check the attached event.log and help us resolve the issue? The database was restarted at about 11:00 due to the fatal error: SYS 15/02/2013 10:58:11 (SM nxp pid=27220) [uhdd.c:uWriteFile:237]: write (code = 28): No space left on device FATAL 15/02/2013 10:58:11 (SM nxp pid=27220) [bm_core.cpp:write_block_addr:233]: Cannot write block INFO 15/02/2013 11:00:01 (GOV pid=28048) [gov_functions.cpp:log_out_system_information:87]: SEDNA version is 3.5.161 (64bit Release) INFO 15/02/2013 11:00:01 (GOV pid=28048) [gov_functions.cpp:log_out_system_information:95]: System: Linux 2.6.18-238.12.1.el5 x86_64 It must have crashed at that moment probably because we have a script that automatically checks and starts the database every 5 minutes. Then the tmp file started growing back to 46GB at about 11:08: LOG 15/02/2013 11:08:22 (SM nxp pid=28081) [blk_mngmt.cpp:extend_tmp_file:629]: Temp file has been extended, size: c80 LOG 15/02/2013 11:08:24 (SM nxp pid=28081) [blk_mngmt.cpp:extend_tmp_file:629]: Temp file has been extended, size: 12c0 LOG 15/02/2013 11:08:25 (SM nxp pid=28081) [blk_mngmt.cpp:extend_tmp_file:629]: Temp file has been extended, size: 1900 So it happens regularly now, the Sedna goes down, then being restarted and the file starts growing to 46GB again. It looks like there is some huge transaction that cannot be completed. Is there any way to locate it and kill that transaction or resolve this somehow to prevent growing? Best regards, Ivan Lagunov From: Robby Pelssers [mailto:robby.pelss...@nxp.com] Sent: Friday, February 15, 2013 11:27 AM To: Ivan Shcheklein Cc: sedna-discussion@lists.sourceforge.net Subject: Re: [Sedna-discussion] question regarding data files [extremely big] Hi Ivan, Issue solved indeed. Thx again for the quick reply !! Robby From: Ivan Shcheklein [mailto:shchekl...@gmail.com] Sent: Friday, February 15, 2013 10:52 AM To: Robby Pelssers Cc: sedna-discussion@lists.sourceforge.net Subject: Re: [Sedna-discussion] question regarding data files [extremely big] Robby, Thx for the quick reply. I was looking into the admin guide http://www.sedna.org/adminguide/AdminGuidesu3.html and noticed that it's possible to set the max file size during creation. Yes, it's possible, but I think transactions will be rolled back if they failed to extend setmp. In this case you will need to restart (not recreate, just restart with se_smsd, se_sm commands) database anyway. Probably, we should try to implement vacuum() function which at least trims setmp without the need to restart the database. But what would be the scenario to accomplish this as our database is obviously already created. To clean setmp you don't need to recreate database - just restart it with se_smsd, se_sm. I just want to make sure I don't miss out on anything here as obviously we want the data, xquery libraries and indexes to be fully restored after this procedure. Just try it (if you really want to see if it's possible to shrink 'sedata' file at all). se_exp should work fine in most cases. Just one note from the documentation: ... current version of the se_exp utility doesn't support exporting/importing triggers, documents with multiple roots and empty documents (empty documents nodes and document nodes with multiple root elements are allowed by XQuery data model but cannot be serialized as is without modifications). Ivan From: Ivan Shcheklein [mailto:shchekl...@gmail.com] Sent: Friday, February 15, 2013 10:23 AM To: Robby Pelssers Cc: sedna-discussion@lists.sourceforge.net Subject: Re: [Sedna-discussion] question regarding data files [extremely big] Hi Robby, Actually, you can. Just restart database - setmp should be restored to its default size. To get statistics, try: * $schema_name document - the descriptive schema of the document or collection named name; * $document_name document - statistical information about the document named name; * $collection_name document - statistical information about the collection named name. details there: http://sedna.org/progguide/ProgGuidesu8.html#x14-580002.5.6 You can also try to determine if your data (sedata) file is fragmented. Use se_exp to export and then import data into clean database. Compare new sedata size with the old one. Ivan Shcheklein, Sedna Team On Fri, Feb 15, 2013 at 1:16 PM, Robby Pelssers robby.pelss...@nxp.com wrote: Hi all, Seems like some files are growing out of proportion for the database nxp. It kind of surprises me as I don't expect we have that much data. But to find a possible bottleneck or get more insight into which collection might be this big, how can I get some statistics? And I guess we can't clean up one of these files, right? Thx in advance, Robby drwxr-xr-x 2 pxprod1 spider4096 Feb 15 05:40 ./ drwxr-xr-x 3 pxprod1 spider 983040 Feb 15 08
Re: [Sedna-discussion] question regarding data files [extremely big]
Hi again, I've received an automatic reply about the blocked zip attachment. I've also shared the event.log in my dropbox account: https://www.dropbox.com/s/pg63nuqb26937dx/event.log Best regards, Ivan Lagunov From: Ivan Lagunov [mailto:lagi...@gmail.com] Sent: Friday, February 15, 2013 11:51 AM To: Robby Pelssers; 'Ivan Shcheklein' Cc: sedna-discussion@lists.sourceforge.net Subject: RE: [Sedna-discussion] question regarding data files [extremely big] Hi Ivan, Here is the zipped event.log file. Best regards, Ivan Lagunov From: Ivan Lagunov [mailto:lagi...@gmail.com] Sent: Friday, February 15, 2013 11:50 AM To: Robby Pelssers; 'Ivan Shcheklein' Cc: sedna-discussion@lists.sourceforge.net Subject: RE: [Sedna-discussion] question regarding data files [extremely big] Hi Ivan, Could you check the attached event.log and help us resolve the issue? The database was restarted at about 11:00 due to the fatal error: SYS 15/02/2013 10:58:11 (SM nxp pid=27220) [uhdd.c:uWriteFile:237]: write (code = 28): No space left on device FATAL 15/02/2013 10:58:11 (SM nxp pid=27220) [bm_core.cpp:write_block_addr:233]: Cannot write block INFO 15/02/2013 11:00:01 (GOV pid=28048) [gov_functions.cpp:log_out_system_information:87]: SEDNA version is 3.5.161 (64bit Release) INFO 15/02/2013 11:00:01 (GOV pid=28048) [gov_functions.cpp:log_out_system_information:95]: System: Linux 2.6.18-238.12.1.el5 x86_64 It must have crashed at that moment probably because we have a script that automatically checks and starts the database every 5 minutes. Then the tmp file started growing back to 46GB at about 11:08: LOG 15/02/2013 11:08:22 (SM nxp pid=28081) [blk_mngmt.cpp:extend_tmp_file:629]: Temp file has been extended, size: c80 LOG 15/02/2013 11:08:24 (SM nxp pid=28081) [blk_mngmt.cpp:extend_tmp_file:629]: Temp file has been extended, size: 12c0 LOG 15/02/2013 11:08:25 (SM nxp pid=28081) [blk_mngmt.cpp:extend_tmp_file:629]: Temp file has been extended, size: 1900 So it happens regularly now, the Sedna goes down, then being restarted and the file starts growing to 46GB again. It looks like there is some huge transaction that cannot be completed. Is there any way to locate it and kill that transaction or resolve this somehow to prevent growing? Best regards, Ivan Lagunov From: Robby Pelssers [mailto:robby.pelss...@nxp.com] Sent: Friday, February 15, 2013 11:27 AM To: Ivan Shcheklein Cc: sedna-discussion@lists.sourceforge.net Subject: Re: [Sedna-discussion] question regarding data files [extremely big] Hi Ivan, Issue solved indeed. Thx again for the quick reply !! Robby From: Ivan Shcheklein [mailto:shchekl...@gmail.com] Sent: Friday, February 15, 2013 10:52 AM To: Robby Pelssers Cc: sedna-discussion@lists.sourceforge.net Subject: Re: [Sedna-discussion] question regarding data files [extremely big] Robby, Thx for the quick reply. I was looking into the admin guide http://www.sedna.org/adminguide/AdminGuidesu3.html and noticed that it's possible to set the max file size during creation. Yes, it's possible, but I think transactions will be rolled back if they failed to extend setmp. In this case you will need to restart (not recreate, just restart with se_smsd, se_sm commands) database anyway. Probably, we should try to implement vacuum() function which at least trims setmp without the need to restart the database. But what would be the scenario to accomplish this as our database is obviously already created. To clean setmp you don't need to recreate database - just restart it with se_smsd, se_sm. I just want to make sure I don't miss out on anything here as obviously we want the data, xquery libraries and indexes to be fully restored after this procedure. Just try it (if you really want to see if it's possible to shrink 'sedata' file at all). se_exp should work fine in most cases. Just one note from the documentation: ... current version of the se_exp utility doesn't support exporting/importing triggers, documents with multiple roots and empty documents (empty documents nodes and document nodes with multiple root elements are allowed by XQuery data model but cannot be serialized as is without modifications). Ivan From: Ivan Shcheklein [mailto:shchekl...@gmail.com] Sent: Friday, February 15, 2013 10:23 AM To: Robby Pelssers Cc: sedna-discussion@lists.sourceforge.net Subject: Re: [Sedna-discussion] question regarding data files [extremely big] Hi Robby, Actually, you can. Just restart database - setmp should be restored to its default size. To get statistics, try: * $schema_name document - the descriptive schema of the document or collection named name; * $document_name document - statistical information about the document named name; * $collection_name document - statistical information about the collection named name. details there: http
