Ok, "contribute it GPL'ed to the project", I was meant to make it available
to the project, I don't want to get into a licensing nightmare, just wanted
to give it back to you guys! :)
The purpose of this box is to overcome latency problems that arise when
trying to enqueue thousands of messages, for example from PHP.
There's a _real_ bottleneck there, and the proof is, sqlbox does a much
better job than the http interface for bulk enqueueing.
My goal is to implement a generic interface able of abstract the kannel
functionality into some languages librarys. In doing it, I can aim to
implement generic (PHP, Perl, Python, etc.) interfaces for sending messages.
For example the PHP extension I'm developing make the function
kannel_send_sms() available as a native function, so no PHP coding is
neccesary. Of course I can achieve the same results by writing my PHP
extension to use the HTTP interface, but I want to get "sqlbox or better"
performance at the same time, without having to have an sql instance
running.
>From all the mails I've received, it's getting obvious that shared memory is
not the way to go. As I said, I'm a complete newby on semaphores and shared
memory (well, at least now I know a little more about it).
I'll go back to the design table, thank you to all the people that answered
with suggestions and clearer views than mine's.
Regards,
Alejandro
On Tue, Mar 4, 2008 at 8:36 AM, Andreas Fink <[EMAIL PROTECTED]> wrote:
> On 03.03.2008, at 01:45, Alejandro Guerrieri wrote:
>
> Dear List,
>
> I'm trying to develop a new "box" that I'm planning to contribute it
> GPL'ed to the project when it's ready. My goal is a box that reads MT
> messages from a shared memory segment and injects them into kannel. This
> way, a number of interfaces could be easily done that write to that shared
> memory segment to send messages.
>
>
> Please note that kannel is not under GPL but under its own license which
> allows to do more than GPL. So integrating it into the core product would
> not work.
>
> Also whats the purpose of this? It won't work under unixes and most
> probably not under windows 2000 upwards neither as memory is protected
> between processes or in other words, they don't share common memory.
> Otherwise you would have to do it in the kernel which is pretty much hard
> core. Much easier to use a socket in such a case as we are not transporting
> gigabytes of memory. And HTTP is one such socket. I have not found an
> performance bottlenecks there yet.
>
> I'm pretty close to make it work, and I even developed a PHP module that
> implements the function "kannel_send_sms()" on PHP. I've called the box
> "phpbox", but I'll probably rename it to "membox", since it can be used with
> any language, as long as it writes messages to the shm.
>
>
> Well... I think your problem is somewhere else.
>
> The performance improvement over the HTTP interface and even over sqlbox
> are impressive (though it's only for MT, of course). It makes sense, since
> all negotiation is done on memory, so it could be a very fast method to
> inject MT messages.
>
>
>
> How many 10'000 messages per second do you need to transport that you have
> bottlenecks like that?
>
> I've done most of the job, I've borrowed the structure from
> sqlbox-standalone (Renee and Martin's ork, credited accordingly) and I've
> mostly succeeded in developing the box and php interface, but after I've
> started testing it with thousands of messages, I've discovered a race
> condition, probably having to do with semaphores.
>
> I'm copying relevant code at the bottom. It's two pieces of code: the
> server (aka phpbox, runs as a box) and the client (a test program written in
> C that enqueues 100K messages).
>
> Everything works fine so far, but when I try running two instances of the
> client program (and sometimes with a single instance), I'm starting getting
> the same message (in fact, it's the last message that replaces all the
> previous).
>
> To be clear, the client program does:
>
> for(c=1;c<=100000;c++) {
> asprintf(&ms, "Hello %d", c);
> send_sms(ms);
> }
>
> So I should receive:
>
> Hello 1
> Hello 2
> ...
> Hello 100000
>
> But instead, I receive:
>
> Hello 1
> Hello 2
> ...
> Hello 23456
> Hello 100000
> Hello 100000
> ...
> Hello 100000
> Hello 100000
>
> No messages are lost, but somewhat the shared memory area is being
> overwritten before it's being read, I suppose.
>
>
> Well thats why there is locking...
>
>
>
> I'm a total newby at semaphores, and I've borrowed the semaphore code from
> a few examples floating on the net, so I suppose I'm doing something wrong
> there.
>
>
> Semaphores are to sync inside a process in multiple threads. They dont
> work between multiple processes. You would need a file locking mechanism or
> the like.
>
>
> Could someone take a look at the code and tell me if there's something
> wrong, or how to rewrite it to use gwlib's semaphores? If you need the full
> (flawed) source code I'll gladly send it.
>
>
> I would say the whole design is wrong. Shared memory is different in most
> operating systems and frankly I don't think your real problem is there.
> Maybe you share the real issue you are facing?
>
>
>
>
>
>
> Andreas Fink
>
> Fink Consulting GmbH
> Global Networks Schweiz AG
> BebbiCell AG
>
> ---------------------------------------------------------------
> Tel: +41-61-6666330 Fax: +41-61-6666331 Mobile: +41-79-2457333
> Address: Clarastrasse 3, 4058 Basel, Switzerland
> E-Mail: [EMAIL PROTECTED]
> www.finkconsulting.com www.global-networks.ch www.bebbicell.ch
> ---------------------------------------------------------------
> ICQ: 8239353 MSN: [EMAIL PROTECTED] AIM: smsrelay Skype: andreasfink
> Yahoo: finkconsulting SMS: +41792457333
>
> Say *NO* to Power Line Communications:
> http://www.youtube.com/watch?v=pdcY0Eetvsw
>
>
>
>
>
>
--
Alejandro Guerrieri
Magicom
http://www.magicom-bcn.net/
LinkedIn: http://www.linkedin.com/in/aguerrieri
