Hi,
opensipsctl takes care that each command takes a separate fifo reply, so
here it should be no problem. But the problem may be when comes with
sending multiple commands (via FIFO) in the same time - this translates
into parallel writes to the same file and depends on the atomicity of
the write op.
But in the worst case, a mixture at the FIFO level may lead to bogus
command and not in any kind of crash....Do you use the "flat_rotate" MI
command ?
Regards,
Bogdan
thrillerbee wrote:
Bogdan,
It seems the issue is with 'opensipsctl fifo' - it's very sensitive to
simultaneous calls. Basically, I've combined all my scripts to
prevent 'opensipsctl fifo' from being called too frequently and that
seems (so far) to have mitigated the issue. Is there anything one
should know about how (not) to use /opensipsctl/?
Thanks.
On Mon, Nov 8, 2010 at 6:07 AM, Bogdan-Andrei Iancu
<bog...@voice-system.ro <mailto:bog...@voice-system.ro>> wrote:
Hi,
strange if you do not have any errors :(....
I just made a fix on both trunk and 1.6 to extend some checks in
flatstore and prevent crashing (even if the DB op will not be
executed).
Could you update from SVN and see if stops crashing ?
Regards,
Bogdan
thrillerbee wrote:
> Bogdan,
>
> I am not seeing any other errors in the logs. Is there anything
else
> I can look at? Both proxies are crashing everyday.
>
> Thanks.
>
> On Wed, Nov 3, 2010 at 4:21 AM, Bogdan-Andrei Iancu
> <bog...@voice-system.ro <mailto:bog...@voice-system.ro>
<mailto:bog...@voice-system.ro <mailto:bog...@voice-system.ro>>>
wrote:
>
> I got some clue about what is happening - as you are using
> flatstore for
> acc, the acc module does not check the success of the
"use_table" DB
> operation - in 90% of the cases (for mysql, postgres, etc) this
> function
> cannot fail, but it seams that for flatstore can. And if it
fails, the
> h->tail is set to NULL, leading to crash.
>
> Now, before the crash itself, do you see any other ERROR
messages
> (even
> long before the crash) related to flatstore module ? Try:
> grep "ERROR" opensips_log_file | grep "flatstore"
>
> Regards,
> Bogdan
>
> thrillerbee wrote:
> > Bogdan,
> >
> > One more detail that may help - I added another OpenSIPS
proxy in
> > parallel with this one (& load balancing between the two) and
> now both
> > OpenSIPS proxies crash at the same time (within a couple
seconds).
> >
> > I can provide more core dumps if it will help.
> >
> > Thanks.
> >
> >
> > On Tue, Nov 2, 2010 at 9:02 AM, thrillerbee
> <thriller...@gmail.com <mailto:thriller...@gmail.com>
<mailto:thriller...@gmail.com <mailto:thriller...@gmail.com>>
> > <mailto:thriller...@gmail.com
<mailto:thriller...@gmail.com> <mailto:thriller...@gmail.com
<mailto:thriller...@gmail.com>>>>
> wrote:
> >
> > Bogdan,
> >
> > Below is the info requested:
> > (gdb) frame 0
> > #0 0x00007f51999f221f in flat_db_insert (h=0x7f0978,
> > k=0x7f5198732a60, v=0x7f5198732d20, n=19) at
flatstore.c:165
> > 165 f = CON_FILE(h);
> > (gdb) print h
> > $1 = (const db_con_t *) 0x7f0978
> > (gdb) print h->tail
> > $2 = 0
> > (gdb) print ((struct flat_con*)(h->tail))->file
> > Cannot access memory at address 0x10
> >
> > Before each of the crashes yesterday, I saw these in
the logs:
> > Nov 1 14:17:40 core1 kernel: [4287745.452111]
opensips[22141]:
> > segfault at 10 ip 7f51999f221f sp 7fffbcd8d510 error 4 in
> > db_flatstore.so[7f51999ef000+5000]
> > Nov 1 23:52:58 core1 kernel: [4348562.990735]
opensips[26978]:
> > segfault at 10 ip 7f726cb9b21f sp 7ffff083f6f0 error 4 in
> > db_flatstore.so[7f726cb98000+5000]
> >
> > Are there any compiler flags I should use for debugging?
> > (gdb) info locals
> > f = <value optimized out>
> > i = <value optimized out>
> > l = <value optimized out>
> > p = <value optimized out>
> > __FUNCTION__ = "flat_db_insert"
> >
> > Thanks again.
> >
> >
> > On Tue, Nov 2, 2010 at 4:18 AM, Bogdan-Andrei Iancu
> > <bog...@voice-system.ro
<mailto:bog...@voice-system.ro> <mailto:bog...@voice-system.ro
<mailto:bog...@voice-system.ro>>
> <mailto:bog...@voice-system.ro
<mailto:bog...@voice-system.ro> <mailto:bog...@voice-system.ro
<mailto:bog...@voice-system.ro>>>>
> wrote:
> >
> > Hi,
> >
> > in frame 0, could you print:
> > h
> > h->tail
> > ((struct flat_con*)(h->tail))->file
> >
> > Also, before crash, do you see in the logs any errors
> from the
> > db_flatstore module ?
> >
> > Regards,
> > Bogdan
> >
> > thrillerbee wrote:
> > > Bogdan,
> > >
> > > It crashed again tonight. I've attached the
backtrace.
> > >
> > > Thanks.
> > >
> > > On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee
> > <thriller...@gmail.com
<mailto:thriller...@gmail.com> <mailto:thriller...@gmail.com
<mailto:thriller...@gmail.com>>
> <mailto:thriller...@gmail.com <mailto:thriller...@gmail.com>
<mailto:thriller...@gmail.com <mailto:thriller...@gmail.com>>>
> > > <mailto:thriller...@gmail.com
<mailto:thriller...@gmail.com>
> <mailto:thriller...@gmail.com <mailto:thriller...@gmail.com>>
> > <mailto:thriller...@gmail.com
<mailto:thriller...@gmail.com>
> <mailto:thriller...@gmail.com
<mailto:thriller...@gmail.com>>>>> wrote:
> > >
> > > Bogdan,
> > >
> > > Yes, I've attached 2 to my response - one
was a crash
> > from 10/29.
> > > The other occurred a few minutes ago.
> > >
> > > Thanks.
> > >
> > > On Fri, Oct 22, 2010 at 11:41 AM,
Bogdan-Andrei Iancu
> > > <bog...@voice-system.ro
<mailto:bog...@voice-system.ro>
> <mailto:bog...@voice-system.ro
<mailto:bog...@voice-system.ro>> <mailto:bog...@voice-system.ro
<mailto:bog...@voice-system.ro>
> <mailto:bog...@voice-system.ro <mailto:bog...@voice-system.ro>>>
> > <mailto:bog...@voice-system.ro
<mailto:bog...@voice-system.ro>
> <mailto:bog...@voice-system.ro <mailto:bog...@voice-system.ro>>
> > <mailto:bog...@voice-system.ro
<mailto:bog...@voice-system.ro>
> <mailto:bog...@voice-system.ro
<mailto:bog...@voice-system.ro>>>>> wrote:
> > >
> > > Any chance with the backtraces ?
> > >
> > > Regards,
> > > Bogdan
> > >
> > > Anca Vamanu wrote:
> > > > Hi,
> > > >
> > > > You need to inspect them with gdb,
run: gdb
> > > > path_to_opensips_executable
> path_to_corefile, and
> > then run
> > > 'bt full'
> > > > and send the output.
> > > >
> > > > Regards,
> > > > --
> > > > Anca Vamanu
> > > > www.voice-system.ro
<http://www.voice-system.ro>
> <http://www.voice-system.ro> <http://www.voice-system.ro>
> > <http://www.voice-system.ro>
> > > >
> > > >
> > > > On 10/14/2010 10:12 PM, thrillerbee wrote:
> > > >> I have this info from dmesg:
> > > >>
> > > >> [1985853.285221] opensips[30865]:
segfault
> at 10 ip
> > > 7f43899ce21f sp
> > > >> 7fff8de1cf40 error 4 in
> > db_flatstore.so[7f43899cb000+5000]
> > > >> [1985856.379671] opensips[30858]:
segfault
> at 10 ip
> > > 7f43899ce21f sp
> > > >> 7fff8de1cf40 error 4 in
> > db_flatstore.so[7f43899cb000+5000]
> > > >> [1985896.961279] opensips[30868]:
segfault
> at 10 ip
> > > 7f43899ce21f sp
> > > >> 7fff8de1cf40 error 4 in
> > db_flatstore.so[7f43899cb000+5000]
> > > >>
> > > >> [2000131.245512] opensips[17672]:
segfault
> at 10 ip
> > > 7fd0f21fb21f sp
> > > >> 7fff3a3b4f00 error 4 in
> > db_flatstore.so[7fd0f21f8000+5000]
> > > >> [2000161.735962] opensips[17668]:
segfault
> at 10 ip
> > > 7fd0f21fb21f sp
> > > >> 7fff3a3b4e40 error 4 in
> > db_flatstore.so[7fd0f21f8000+5000]
> > > >> [2000167.299402] opensips[17670]:
segfault
> at 10 ip
> > > 7fd0f21fb21f sp
> > > >> 7fff3a3b4f00 error 4 in
> > db_flatstore.so[7fd0f21f8000+5000]
> > > >>
> > > >> On Thu, Oct 14, 2010 at 1:57 PM,
thrillerbee
> > > <thriller...@gmail.com
<mailto:thriller...@gmail.com>
> <mailto:thriller...@gmail.com <mailto:thriller...@gmail.com>>
> > <mailto:thriller...@gmail.com
<mailto:thriller...@gmail.com>
> <mailto:thriller...@gmail.com
<mailto:thriller...@gmail.com>>> <mailto:thriller...@gmail.com
<mailto:thriller...@gmail.com>
> <mailto:thriller...@gmail.com <mailto:thriller...@gmail.com>>
> > <mailto:thriller...@gmail.com
<mailto:thriller...@gmail.com>
> <mailto:thriller...@gmail.com <mailto:thriller...@gmail.com>>>>
> > > >> <mailto:thriller...@gmail.com
<mailto:thriller...@gmail.com>
> <mailto:thriller...@gmail.com <mailto:thriller...@gmail.com>>
> > <mailto:thriller...@gmail.com
<mailto:thriller...@gmail.com>
> <mailto:thriller...@gmail.com <mailto:thriller...@gmail.com>>>
> > > <mailto:thriller...@gmail.com
<mailto:thriller...@gmail.com>
> <mailto:thriller...@gmail.com <mailto:thriller...@gmail.com>>
> > <mailto:thriller...@gmail.com
<mailto:thriller...@gmail.com>
> <mailto:thriller...@gmail.com
<mailto:thriller...@gmail.com>>>>>> wrote:
> > > >>
> > > >> When OpenSIPS crashes, three
corefiles are
> > generated
> > > that are
> > > >> 2.1GB in size. How do I use
these files to
> > understand
> > > what's
> > > >> causing the crash?
> > > >>
> > > >> Thanks.
> > > >>
> >
>
--
Bogdan-Andrei Iancu
OpenSIPS Bootcamp
15 - 19 November 2010, Edison, New Jersey, USA
www.voice-system.ro <http://www.voice-system.ro>
_______________________________________________
Users mailing list
Users@lists.opensips.org <mailto:Users@lists.opensips.org>
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
------------------------------------------------------------------------
_______________________________________________
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
--
Bogdan-Andrei Iancu
OpenSIPS Bootcamp
15 - 19 November 2010, Edison, New Jersey, USA
www.voice-system.ro
_______________________________________________
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
