Hi Jocelyn,
Latest development build is stable now. It contains only bug fixes.
We will release Sedna 3.5 with fixes soon.
Ivan
Thank you Ivan. Sorry for late reply, I was away on vacation.
> Is this bug fix in the latest stable release (i.e. 3.5)? I am not sure if
> I should put the "development" sedna into my production server. If it is
> not part of the current stable release, when do you anticipate to be part
> of it?
> Do you consider this development release to be stable?
> Thanks,
> Jocelyn
>
>
> ------------------------------
> *From:* Ivan Shcheklein [mailto:shchekl...@gmail.com]
> *Sent:* December 24, 2011 12:26 AM
> *To:* Raymond, Jocelyn
> *Cc:* sedna-discussion
>
> *Subject:* Re: [Sedna-discussion] Sedna 3.4.66 (Linux) FATAL issue
>
> Hi Jocelyn,
>
> Please check the latest dev build:
>
> http://modis.ispras.ru/FTPContent/sedna/development/
>
> It should fix the bug.
>
> Thank you for your feedback and using Sedna!
>
> Ivan Shcheklein,
> Sedna Team
>
>
> On Wed, Dec 21, 2011 at 7:49 PM, Raymond, Jocelyn <
> jocelyn.raym...@ualberta.ca> wrote:
>
>> **
>> Hi Ivan,
>> Thanks for the shell script below. I have tried and discovered the
>> cause. I don't think there is anything wrong with Sedna and probably one
>> of my mistake.
>> The problem appear to come from my hot backups. When I disabled the hot
>> backups, the lsof showed no increases (my development box). When I
>> re-enabled my hot backups shell script, it increased by one every time it
>> ran (I ran the hot backup every hour). I have attached my script but the
>> line that seems to cause this is in red below. So every time the -incr-mode
>> add is called, the lsof showed the increase of one and never went back
>> down. Even after I called the -incr-mode stop it didn't go down. It
>> also increased by two when -incr-mode start was called. Am I missing
>> something? Is there something else I should be calling when I want to
>> terminate a hot backup cycle? Do you have any suggestions?
>> Thanks for your help
>>
>>
>> incr(){
>> if [ ${#INCR_DIRECTORY} -ne 0 ]; then
>> $SEDNA_HB -time-dir -incr-mode add $DBNAME $INCR_DIRECTORY
>> if [ $? -ne 0 ]; then
>> email "Sedna Hot Backup (increment) *failed* today (see
>> $INCR_DIRECTORY)" "SEDNA HOT BACKUPS - FAILURE" $EMAILS
>> exit -1
>> fi
>> else
>> email "Sedna Hot Backup (increment) *failed* today because the cycle
>> wasn't previously started (see $STORED_PATH_IN)" "SEDNA HOT BACKUPS -
>> FAILURE" $EMAILS
>> fi
>>
>> }
>>
>>
>> ------------------------------
>> *From:* Ivan Shcheklein [mailto:shchekl...@gmail.com]
>> *Sent:* December 19, 2011 1:23 PM
>> *To:* Raymond, Jocelyn
>> *Cc:* se...@ispras.ru
>>
>> *Subject:* Re: [Sedna-discussion] Sedna 3.4.66 (Linux) FATAL issue
>>
>> Ok, thanks for the data. Though, we still don't know the reason.
>> Everything seems ok.
>>
>> How long do you use 3.4.66? Have you seen this problem before?
>>
>> Let's monitor se_gov for a while to collect number of open files at least.
>>
>> 1. Create /tmp/monitor.sh (vim monitor.sh)
>> 2. Copy the following script inside this file. It prints once in 10
>> seconds line: "current date number of open file by se_gov process
>> number of total open files".
>>
>> #!/bin/sh -e
>>
>> GOV_PID=`pgrep se_gov`
>>
>> while true; do
>> echo `date` `lsof -n -p $GOV_PID | wc -l` `lsof -n | wc -l`
>> sleep 10
>> done
>>
>> 3. Make it executable. And check that it works:
>>
>> chmod +x /tmp/monitor.sh
>> /tmp/monitor.sh
>>
>> it should print something like:
>>
>> Tue Dec 20 00:17:43 MSK 2011 81 2087
>>
>> 4. Make sure that Sedna is running (se_gov process exists)
>>
>> 5. Start it in background:
>>
>> nohup /tmp/monitor.sh >/tmp/log.out 2>&1 &
>>
>> 6. Make sure it's running properly (check that monitor.sh process exists
>> and log.out is being appended).
>>
>> Then we have to wait until next error happens or log.out (you can check
>> it with tail -f /tmp/log.out command) contains huge numbers.
>>
>> Anyway copy and send us log.out after one day.
>>
>> P.S. Don't forget to kill monitor.sh process after the experiment is
>> finished ). And let us know if you have problems with running that script.
>>
>> On Mon, Dec 19, 2011 at 8:02 PM, Raymond, Jocelyn <
>> jocelyn.raym...@ualberta.ca> wrote:
>>
>>> **
>>> From your instructions, I got the following (but don't really know how
>>> to analyse it):
>>> [sedna data]# ulimit -n
>>> 1024
>>> [sedna data]# sysctl -n fs.file-max
>>> 203350
>>> [sedna data]# se_stop
>>> SEDNA server has been shut down successfully
>>> [sedna data]# sudo lsof | wc -l
>>> 824
>>>
>>> Also, this very morning@06:11:02, our development sedna (same setup as
>>> prod) went down for what appear to be the same reason. Here is a sample of
>>> our log (attached). Here is the same command executed on our development
>>> machine:
>>> [raja sedna]# ulimit -n
>>> 1024
>>> [raja sedna]# sysctl -n fs.file-max
>>> 206005
>>> [raja sedna]# se_stop
>>> SEDNA server has been shut down successfully
>>> [raja sedna]# sudo lsof | wc -l
>>> 2533
>>> Let me know if you have any insights on this problem.
>>> Thank you again,
>>> Jocelyn
>>> University of Alberta
>>>
>>> ------------------------------
>>> *From:* Ivan Shcheklein [mailto:shchekl...@gmail.com]
>>> *Sent:* December 15, 2011 12:45 PM
>>> *To:* Raymond, Jocelyn
>>> *Cc:* sedna-discussion@lists.sourceforge.net
>>> *Subject:* Re: [Sedna-discussion] Sedna 3.4.66 (Linux) FATAL issue
>>>
>>> Hi Jocelyn,
>>>
>>>
>>> We have been using Sedna for quite some time now (since version 0.6
>>>> :-).
>>>>
>>>
>>> Yep. We remember this :) Thank you for using it!
>>>
>>>
>>>> SYS 15/12/2011 09:10:01 (GOV pid=24200) [ushm.c:uOpenShMem:71]:
>>>> shm_open (code = 24): Too many open files
>>>> FATAL 15/12/2011 09:10:01 (GOV pid=24200)
>>>> [hb_funcs.cpp:hbSendMsgToSm:64]: Failed to initialize SSMMsg service
>>>> (message service)
>>>>
>>>
>>> First of all, we need to determine the actual cause of that problem
>>> (Sedna or some other process which holds too many open files).
>>>
>>> Please, run the following (under Sedna user):
>>>
>>> ulimit -n
>>> sysctl -n fs.file-max
>>>
>>> se_stop
>>>
>>> sudo lsof | wc -l
>>>
>>>
>>>
>>>> I noticed that you have version 3.5 released. I will probably
>>>> upgrade our devl and prod early in 2012 unless you believe that this
>>>> problem could go away if I upgrade right now.
>>>>
>>>
>>> We didn't fix any file descriptors leaks. Though, it's better to upgrade
>>> it anyway. It's faster, more stable, etc :)
>>>
>>>
>>> Ivan Shcheklein,
>>> Sednan Team
>>>
>>
>>
>
------------------------------------------------------------------------------
Ridiculously easy VDI. With Citrix VDI-in-a-Box, you don't need a complex
infrastructure or vast IT resources to deliver seamless, secure access to
virtual desktops. With this all-in-one solution, easily deploy virtual
desktops for less than the cost of PCs and save 60% on VDI infrastructure
costs. Try it free! http://p.sf.net/sfu/Citrix-VDIinabox
_______________________________________________
Sedna-discussion mailing list
Sedna-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sedna-discussion
