Sebastian, My best guess about multi-tabs root cause is un-synchronized access to video/audio coding internals (decoding works well) and inside Flash and device drivers. This makes crashes unpredictable.
-- With best regards / с наилучшими пожеланиями, Alexei Fedotov / Алексей Федотов, http://dataved.ru/ +7 916 562 8095 [1] Start using Apache Openmeetings today, http://openmeetings.apache.org/ [2] Join Alexei Fedotov @linkedin, http://ru.linkedin.com/in/dataved/ [3] Join Alexei Fedotov @facebook, http://www.facebook.com/openmeetings On Tue, Jun 25, 2013 at 4:58 AM, [email protected] <[email protected]> wrote: > * Multi-tabbing (no other flash application should be running in different > browsers or tabs)* > "This is a well-known problem with Flash on Linux" > => Then the sentence might be more like: > " Running multiple instance of OpenMeetings or Flash applications on Linux > can cause issues with the stability of the Flash Plugin" > > My problem is more the general approach of all those release notes. > You generally say: Usage of multiple tabs is forbidden, while the actual > use case is much more specific. > > Same for recordings, the documented use cases is fairly more detailed then > the Release Notes indicate. > > Do you have a proposal for those Release Notes that would reflect more the > detailed level of discussion that we have done now? > > Thanks, > Sebastian > > > > > > 2013/6/25 Alexei Fedotov <[email protected]> > >> That was intended to be one sentence, (it is broken in the text) >> >> > For those who don't trust, let's try a recording session online and >> check. >> -- >> With best regards / с наилучшими пожеланиями, >> Alexei Fedotov / Алексей Федотов, >> http://dataved.ru/ >> +7 916 562 8095 >> >> [1] Start using Apache Openmeetings today, http://openmeetings.apache.org/ >> [2] Join Alexei Fedotov @linkedin, http://ru.linkedin.com/in/dataved/ >> [3] Join Alexei Fedotov @facebook, http://www.facebook.com/openmeetings >> >> >> On Mon, Jun 24, 2013 at 4:01 PM, Alexei Fedotov >> <[email protected]> wrote: >> > Artyom, >> > release notes are written for a user. Imagine you are one. Notes >> > should help you and tell >> > you that despite openmeetings.apache.org web site which mentions >> > recordings they do not work for >> > real conferences in 2.1.1. If you learn this by trial and error, >> > it will take a lot of your time. >> > >> > Release notes are not intended to blame that someone breaks something. >> > This is useless anyway because one can imagine the case when >> > Openmeetings breaks by itself (and we >> > get one with Adobe auto-updater). >> > >> > Sebastian, >> > The bug about recordings is not that good in description, still the >> > problem exists. We do recordings several times per week, and soon >> > locate the problem root cause more accurately. Vasily reports that >> > generic recording session works for no longer than 20 minutes (longer >> > recording sessions are rare exceptions). >> > a recording session online and check. I myself did several in a row. >> > Taking into account the standard conference planning cycle is 1 hour, >> > I think 20 minutes is not sufficient. >> > >> > For those who don't trust, let's try. The "try it" approach magically >> > worked for multi-tabs and Irina, we >> > got a crash in few seconds. She seems much more convinced in her last >> > letter, though this is only a part of the problem. Also multi-tabbed >> > openmeetings somehow corrupts windows microphone settings. >> > >> > As for memory consumption, locating a problem was more difficult, than >> > to believe into it now. IMHO Adobe developers have leaks in codec >> > implementations (eg releasing memory for some video frames), when we >> > do not free all resources explicitly (and cannot free system-deep >> > resources). SIP integration has its own server resource leak, this one >> > is unrelated. >> > >> > While I personally want having SIP among the features (and do test it >> > several times a day), I'm ok with dropping it. Let's just reflect this >> > drop in the last release notes. This is important because we announced >> > SIP integration several times, and the last time was in March. >> > >> > I'm not +1 with shipping a release which misses reported features >> > without a proper note, because this potentially makes customers >> > unhappy. >> > >> > -- >> > With best regards / с наилучшими пожеланиями, >> > Alexei Fedotov / Алексей Федотов, >> > http://dataved.ru/ >> > +7 916 562 8095 >> > >> > [1] Start using Apache Openmeetings today, >> http://openmeetings.apache.org/ >> > [2] Join Alexei Fedotov @linkedin, http://ru.linkedin.com/in/dataved/ >> > [3] Join Alexei Fedotov @facebook, http://www.facebook.com/openmeetings >> > >> > >> > On Mon, Jun 24, 2013 at 3:08 PM, [email protected] >> > <[email protected]> wrote: >> >> Hi there, >> >> >> >> regarding OPENMEETINGS-669 >> >> This has nothing todo with Release 2.1.1. I don't think it should be a >> note >> >> in the release notes. The comment in the Jira ticket *I have retested >> on OM >> >> 3.0. Recording works without errors.* >> >> => This error can potentially happen with any version of OpenMeetings. >> >> I can clarify the details of how and when this can happen, however the >> root >> >> cause of the issue is not any new introduced functionality of >> OpenMeetings >> >> 2.1.1 and it has _nothing_ todo with the length of the recording. I >> would >> >> be very happy if there would be a concrete steps to reproduce the issue >> in >> >> the Bug report. >> >> >> >> regarding https://bugzilla.mozilla.org/show_bug.cgi?id=885188 >> >> Does this bug happen also _without_ using the SIP integration? >> >> Also the Bug report clearly states that the inital report has nothing >> todo >> >> with the second crash report in the ticket. >> >> I doubt that downgrading Flash Player will change the memory >> consumption, >> >> however that is also just a theory. The same theory that the bug report >> >> asumes by saying *downgrading Flash solves the issue*. I don't think >> that >> >> there is a general request for *downgrade to 11.5 version* of Flash >> Player. >> >> The bug report actually does not even mention which version exactly was >> >> tested and identified as buggy. So which version of the Flash Player was >> >> this actually? And for instance which task exactly in the Flash Player >> has >> >> the memory leak? It could be for instance also a problem with our >> >> Audio/Video Codec configuration in the conference room or simply a >> problem >> >> with the cam/microphone driver on that particular device. >> >> >> >> * Multi-tabbing (no other flash application should be running in >> different >> >> browsers or tabs)* >> >> => I don't see any problem in running two instances of OpenMeetings >> next to >> >> each other, I do that all the time for testing. >> >> I don't think this Release Note should be added. What Irina mentions is >> a >> >> specific use-case, but even that, I do that all the time without >> issues. So >> >> there might be a problem but creating now the general restriction to >> have >> >> only a single tab opened all the time and close everything else seems a >> bit >> >> restricted. Maybe this phrase could be worded differently: >> >> *Running multiple Flash Applications or OpenMeetings instances in >> multiple >> >> Tabs on the same device can lead to unexpected behaviour* >> >> >> >> regarding OPENMEETINGS-677, OPENMEETINGS-678 >> >> What is it that we mention this in the Release Notes? Integration of >> >> SIP-Video is not on our list of current features. It would be a nice >> >> feature, but why would you add it to the Release Notes if there is an >> issue >> >> with it? It was not even announced yet that it would exist at all? >> >> >> >> Thanks, >> >> Sebastian >> >> >> >> >> >> >> >> 2013/6/24 Irina Arkhipets <[email protected]> >> >> >> >>> Hi All, >> >>> >> >>> Reharding the multi-tabbing: >> >>> >> >>> Here is a link to the crash report I've received recently when I tried >> to >> >>> sit in two different OpenMeetings rooms in the tabs of the same >> Mozilla FF >> >>> browser: >> >>> >> >>> >> https://crash-stats.mozilla.com/report/index/c448bb3d-18ef-45f6-aa66-7d8492130624 >> >>> >> >>> It looks like some known bug in the flash player: see the list of the >> >>> candidates in the bug report. >> >>> >> >>> It seems like I am able to sit in one OpenMeetings room from two tabs >> of >> >>> the same browser as well as play youtube videos at the same moment. >> >>> For me, the problems appear when I am trying to enter two different >> rooms >> >>> in the same browser at the moment. >> >>> >> >>> Regards, >> >>> Irina. >> >>> >> >>> >> >>> On Mon, Jun 24, 2013 at 4:50 PM, Artyom Horuzhenko <[email protected] >> >>> >wrote: >> >>> >> >>> > Video/audio SIP integration has never been a part of Openmeetings >> (as I >> >>> > know); >> >>> > Recordings didn't work before release, it's not new in 2.1.1; >> >>> > There are troubles with recent version of Adobe Flash in 2.1, it's >> not >> >>> new >> >>> > too >> >>> > >> >>> > 2013/6/24 Alexei Fedotov <[email protected]> >> >>> > >> >>> > > Artyom, >> >>> > > I have updated the release notes with your input, and additional >> input >> >>> > > from Irina. >> >>> > > >> >>> > > --- >> >>> > > Video/audio SIP integration is no longer a part of Apache release. >> >>> > > Check free red5sip product for this integration. >> >>> > > >> >>> > > We also introduced reliability testing for this release (great >> job, >> >>> > > Vasily Degtyarev!). For 1 hour long conferences the following does >> not >> >>> > > work with Openmeetings 2.1.1: >> >>> > > * Recordigs >> >>> > > https://issues.apache.org/jira/browse/OPENMEETINGS-669 >> >>> > > * Recent version with Adobe Flash (please downgrade to 11.5 >> version) >> >>> > > https://bugzilla.mozilla.org/show_bug.cgi?id=885188 >> >>> > > * Multi-tabbing (no other flash application should be running in >> >>> > > different browsers or tabs) >> >>> > > --- >> >>> > > >> >>> > > >> >>> > > -- >> >>> > > With best regards / с наилучшими пожеланиями, >> >>> > > Alexei Fedotov / Алексей Федотов, >> >>> > > http://dataved.ru/ >> >>> > > +7 916 562 8095 >> >>> > > >> >>> > > [1] Start using Apache Openmeetings today, >> >>> > http://openmeetings.apache.org/ >> >>> > > [2] Join Alexei Fedotov @linkedin, >> http://ru.linkedin.com/in/dataved/ >> >>> > > [3] Join Alexei Fedotov @facebook, >> >>> http://www.facebook.com/openmeetings >> >>> > > >> >>> > > >> >>> > > On Mon, Jun 24, 2013 at 12:55 PM, Artyom Horuzhenko < >> >>> [email protected]> >> >>> > > wrote: >> >>> > > > Alexei, we shouldn't worry about sip integeration in our releases >> >>> until >> >>> > > > red5sip is not an Openmeetings part. For example, let's imagine >> if >> >>> > Linux >> >>> > > > developers say: "We can't release new kernel because OpenOffice >> has >> >>> > > various >> >>> > > > document rendering bugs". And Adobe Flash is not good example >> because >> >>> > > > Openmeetings uses flash, but red5sip uses Openmeetings to provide >> >>> voip >> >>> > > > integration; all sip audio/video troubles are not cause of >> >>> Openmeetings >> >>> > > > bugs. >> >>> > > > >> >>> > > > >> >>> > > > 2013/6/24 Alexei Fedotov <[email protected]> >> >>> > > > >> >>> > > >> I'm constantly getting enlightened by many people that red4sip >> is >> >>> not >> >>> > > >> a part of Openmeetings. I'm well aware of that fact. :-) Adobe >> Flash >> >>> > > >> is not a part of Openmeetings as well, and cause us the problems >> >>> > > >> anyway. >> >>> > > >> >> >>> > > >> If you program does not run on Linux, release notes should say, >> >>> "does >> >>> > > >> not run on Linux". IMHO, it's important to reference ecosystem >> >>> > > >> requirements in release notes. >> >>> > > >> >> >>> > > >> >> >>> > > >> -- >> >>> > > >> With best regards / с наилучшими пожеланиями, >> >>> > > >> Alexei Fedotov / Алексей Федотов, >> >>> > > >> http://dataved.ru/ >> >>> > > >> +7 916 562 8095 >> >>> > > >> >> >>> > > >> [1] Start using Apache Openmeetings today, >> >>> > > http://openmeetings.apache.org/ >> >>> > > >> [2] Join Alexei Fedotov @linkedin, >> >>> http://ru.linkedin.com/in/dataved/ >> >>> > > >> [3] Join Alexei Fedotov @facebook, >> >>> > http://www.facebook.com/openmeetings >> >>> > > >> >> >>> > > >> >> >>> > > >> On Mon, Jun 24, 2013 at 11:59 AM, Alexei Fedotov >> >>> > > >> <[email protected]> wrote: >> >>> > > >> > The previous release was an important step for us, we >> graduated. >> >>> > Right >> >>> > > >> > now we should restore original "release" meaning by >> establishing >> >>> the >> >>> > > >> > quality. There are two steps in establishing the quality. >> >>> > > >> > >> >>> > > >> > 1. Define the quality. Vasily did a good job here. >> >>> > > >> > 2. Improving the quality. Here we can improve more. >> >>> > > >> > >> >>> > > >> > I want release notes being correct and agreed with >> participants. >> >>> > > >> > Taking Artyom comment into account, I wrote the following >> attempt >> >>> of >> >>> > > >> > release notes. >> >>> > > >> > >> >>> > > >> > The following no longer works with Openmeetings 2.1.1 (for 1 >> hour >> >>> > long >> >>> > > >> > conferences) >> >>> > > >> > * Recordigs >> >>> > > >> > * Recent version with Adobe Flash (please downgrade to 11.5 >> >>> version) >> >>> > > >> > * Multi-tabbing (no other flash application should be running >> in >> >>> > > >> > different browsers or tabs) >> >>> > > >> > * Video/audio SIP integration (because it requires another >> program >> >>> > we >> >>> > > >> > don't want to mention anymore) >> >>> > > >> > >> >>> > > >> > Vasily, could you please help adding bug numbers to each >> line of >> >>> > the >> >>> > > >> notes? >> >>> > > >> > The last item intentionally has a weird explanation, but >> that's >> >>> > what I >> >>> > > >> > get from Artyom's letter. Artyom, could you please correct the >> >>> last >> >>> > > >> > item in a way you think correctly reflects the state of SIP >> >>> > > >> > integration? >> >>> > > >> > >> >>> > > >> > I think we can ship release with reflected problems, this may >> help >> >>> > > >> > users not to seek things they won't get. With these release >> notes >> >>> > > >> > (improved with bug references and corrections) the release >> gets my >> >>> > +1. >> >>> > > >> > >> >>> > > >> > -- >> >>> > > >> > With best regards / с наилучшими пожеланиями, >> >>> > > >> > Alexei Fedotov / Алексей Федотов, >> >>> > > >> > http://dataved.ru/ >> >>> > > >> > +7 916 562 8095 >> >>> > > >> > >> >>> > > >> > [1] Start using Apache Openmeetings today, >> >>> > > >> http://openmeetings.apache.org/ >> >>> > > >> > [2] Join Alexei Fedotov @linkedin, >> >>> > http://ru.linkedin.com/in/dataved/ >> >>> > > >> > [3] Join Alexei Fedotov @facebook, >> >>> > > http://www.facebook.com/openmeetings >> >>> > > >> > >> >>> > > >> > >> >>> > > >> > On Mon, Jun 24, 2013 at 2:17 AM, [email protected] >> >>> > > >> > <[email protected]> wrote: >> >>> > > >> >> I think Artyom is right. Red5Sip is not the scope of the >> >>> > OpenMeetings >> >>> > > >> >> project. >> >>> > > >> >> Me might discuss on how we can make it the scope, if we are >> able >> >>> to >> >>> > > >> create >> >>> > > >> >> an Incubator project around Red5Sip. >> >>> > > >> >> >> >>> > > >> >> I think for the end user this discussion (scope or not) is a >> >>> rather >> >>> > > >> >> esoteric discussion as they are simply interested in a >> working >> >>> > > product. >> >>> > > >> >> >> >>> > > >> >> However for the release, Artyom & Alexei, you are both PMC >> >>> members, >> >>> > > have >> >>> > > >> >> you an opinion about the 2.1.1 release? >> >>> > > >> >> >> >>> > > >> >> Thanks, >> >>> > > >> >> Sebastian >> >>> > > >> >> >> >>> > > >> >> >> >>> > > >> >> 2013/6/24 Artyom Horuzhenko <[email protected]> >> >>> > > >> >> >> >>> > > >> >>> Hello Alexei, >> >>> > > >> >>> >> >>> > > >> >>> I've just changed sip integration instruction a little by >> adding >> >>> > > >> denoise >> >>> > > >> >>> and dsp_drop_silence attributes and removed silent >> detection in >> >>> > > red5sip >> >>> > > >> >>> because there were some problems with it. I think we >> shouldn't >> >>> > > include >> >>> > > >> any >> >>> > > >> >>> information about red5sip improvement into Openmeetings >> release >> >>> > > notes >> >>> > > >> >>> because red5sip is another application. >> >>> > > >> >>> >> >>> > > >> >>> >> >>> > > >> >>> 2013/6/23 Alexei Fedotov <[email protected]> >> >>> > > >> >>> >> >>> > > >> >>> > Sebastian, I have tested audio SIP integration by using >> SIP >> >>> > bridge >> >>> > > >> for >> >>> > > >> >>> > my own conferences. As a result, Artyom found out that >> >>> disabling >> >>> > > >> >>> > jitter-buffer in the Asterisk configuration improves >> audio. >> >>> > > Artyom, >> >>> > > >> >>> > where one can find your findings on the Apache web site? >> Does >> >>> it >> >>> > > >> worth >> >>> > > >> >>> > reflecting this in release notes? >> >>> > > >> >>> > >> >>> > > >> >>> > -- >> >>> > > >> >>> > With best regards / с наилучшими пожеланиями, >> >>> > > >> >>> > Alexei Fedotov / Алексей Федотов, >> >>> > > >> >>> > http://dataved.ru/ >> >>> > > >> >>> > +7 916 562 8095 >> >>> > > >> >>> > >> >>> > > >> >>> > [1] Start using Apache Openmeetings today, >> >>> > > >> >>> http://openmeetings.apache.org/ >> >>> > > >> >>> > [2] Join Alexei Fedotov @linkedin, >> >>> > > >> http://ru.linkedin.com/in/dataved/ >> >>> > > >> >>> > [3] Join Alexei Fedotov @facebook, >> >>> > > >> http://www.facebook.com/openmeetings >> >>> > > >> >>> > >> >>> > > >> >>> > >> >>> > > >> >>> > On Sun, Jun 23, 2013 at 7:09 AM, [email protected] >> >>> > > >> >>> > <[email protected]> wrote: >> >>> > > >> >>> > > We have 3 +1 votes from PMCs. So in theory it is >> possible. >> >>> > > >> >>> > > >> >>> > > >> >>> > > However it would be good if there is some more feedback >> in >> >>> the >> >>> > > >> >>> community >> >>> > > >> >>> > > about that release. >> >>> > > >> >>> > > >> >>> > > >> >>> > > Anybody else has been testing this release? >> >>> > > >> >>> > > >> >>> > > >> >>> > > Sebastian >> >>> > > >> >>> > > >> >>> > > >> >>> > > >> >>> > > >> >>> > > 2013/6/22 Maxim Solodovnik <[email protected]> >> >>> > > >> >>> > > >> >>> > > >> >>> > >> I believe we should release rc1. >> >>> > > >> >>> > >> >> >>> > > >> >>> > >> @Sebastian can you publish and announce the release? >> >>> > > >> >>> > >> >> >>> > > >> >>> > >> BTW I did paragliding from ~2300 meters in Turkey with >> >>> > > instructor >> >>> > > >> and >> >>> > > >> >>> > sky >> >>> > > >> >>> > >> acrobatics :) it was really amazing! :) >> >>> > > >> >>> > >> On Jun 22, 2013 3:36 AM, "[email protected]" < >> >>> > > >> >>> [email protected] >> >>> > > >> >>> > > >> >>> > > >> >>> > >> wrote: >> >>> > > >> >>> > >> >> >>> > > >> >>> > >> > I could only find one minor issue: >> >>> > > >> >>> > >> > >> https://issues.apache.org/jira/browse/OPENMEETINGS-681 >> >>> > > >> >>> > >> > >> >>> > > >> >>> > >> > I think that is still good to go out as a release. >> >>> > > >> >>> > >> > >> >>> > > >> >>> > >> > +1 >> >>> > > >> >>> > >> > >> >>> > > >> >>> > >> > Sebastian >> >>> > > >> >>> > >> > >> >>> > > >> >>> > >> > >> >>> > > >> >>> > >> > 2013/6/22 [email protected] < >> [email protected]> >> >>> > > >> >>> > >> > >> >>> > > >> >>> > >> > > I am sorry for the delay. >> >>> > > >> >>> > >> > > >> >>> > > >> >>> > >> > > The vote is now actually outdated. >> >>> > > >> >>> > >> > > Shall we restart the vote process for 2.1.1 ? >> >>> > > >> >>> > >> > > >> >>> > > >> >>> > >> > > Sebastian >> >>> > > >> >>> > >> > > >> >>> > > >> >>> > >> > > >> >>> > > >> >>> > >> > > 2013/6/10 [email protected] < >> [email protected] >> >>> > >> >>> > > >> >>> > >> > > >> >>> > > >> >>> > >> > > Release vote happens on dev list, never on private. >> >>> > Please >> >>> > > >> vote on >> >>> > > >> >>> > dev, >> >>> > > >> >>> > >> > as >> >>> > > >> >>> > >> > >> the link to the dev list will be what we publish >> as a >> >>> > > result >> >>> > > >> of >> >>> > > >> >>> the >> >>> > > >> >>> > >> > vote. >> >>> > > >> >>> > >> > >> >> >>> > > >> >>> > >> > >> I am sorry but I can't test before next week as I >> >>> don't >> >>> > > have >> >>> > > >> a >> >>> > > >> >>> > >> Internet >> >>> > > >> >>> > >> > >> connection good enough while I travel abroad. >> >>> > > >> >>> > >> > >> >> >>> > > >> >>> > >> > >> Thanks, >> >>> > > >> >>> > >> > >> Sebastian >> >>> > > >> >>> > >> > >> >> >>> > > >> >>> > >> > >> >> >>> > > >> >>> > >> > >> >> >>> > > >> >>> > >> > >> >> >>> > > >> >>> > >> > >> 2013/6/8 Maxim Solodovnik <[email protected]> >> >>> > > >> >>> > >> > >> >> >>> > > >> >>> > >> > >>> Dear OpenMeetings Community, >> >>> > > >> >>> > >> > >>> >> >>> > > >> >>> > >> > >>> I would like to start a vote about releasing >> Apache >> >>> > > >> OpenMeetings >> >>> > > >> >>> > >> 2.1.1 >> >>> > > >> >>> > >> > >>> RC1 >> >>> > > >> >>> > >> > >>> >> >>> > > >> >>> > >> > >>> Main changes are covered in the >> >>> > > >> >>> > >> > >>> >> >>> > > >> >>> > >> > >>> Readme: >> >>> > > >> >>> > >> > >>> >> >>> > > >> >>> >> >>> http://svn.apache.org/repos/asf/openmeetings/tags/2.1.1RC1/README >> >>> > > >> >>> > >> > >>> >> >>> > > >> >>> > >> > >>> >> >>> > > >> >>> > >> > >>> Full Changelog: >> >>> > > >> >>> > >> > >>> >> >>> > > >> >>> > >> >>> > > >> http://svn.apache.org/repos/asf/openmeetings/tags/2.1.1RC1/CHANGELOG >> >>> > > >> >>> > >> > >>> >> >>> > > >> >>> > >> > >>> Release artifacts: >> >>> > > >> >>> > >> > >>> >> >>> > > >> https://dist.apache.org/repos/dist/dev/openmeetings/2.1.1/rc1/ >> >>> > > >> >>> > >> > >>> >> >>> > > >> >>> > >> > >>> Tag: >> >>> > > >> >>> http://svn.apache.org/repos/asf/openmeetings/tags/2.1.1RC1/ >> >>> > > >> >>> > >> > >>> >> >>> > > >> >>> > >> > >>> PGP release keys (signed using >> >>> > > >> >>> > >> > >>> C467526E): >> >>> > > >> >>> > >> > >>> >> >>> > > >> >>> > >> >>> > > https://dist.apache.org/repos/dist/dev/openmeetings/2.1.1/rc1/KEYS >> >>> > > >> >>> > >> > >>> >> >>> > > >> >>> > >> > >>> Vote will be open for 72 hours. >> >>> > > >> >>> > >> > >>> >> >>> > > >> >>> > >> > >>> [ ] +1 approve >> >>> > > >> >>> > >> > >>> [ ] +0 no opinion >> >>> > > >> >>> > >> > >>> [ ] -1 disapprove (and reason why) >> >>> > > >> >>> > >> > >>> >> >>> > > >> >>> > >> > >>> My vote is +1. >> >>> > > >> >>> > >> > >>> >> >>> > > >> >>> > >> > >>> >> >>> > > >> >>> > >> > >>> >> >>> > > >> >>> > >> > >>> -- >> >>> > > >> >>> > >> > >>> WBR >> >>> > > >> >>> > >> > >>> Maxim aka solomax >> >>> > > >> >>> > >> > >>> >> >>> > > >> >>> > >> > >> >> >>> > > >> >>> > >> > >> >> >>> > > >> >>> > >> > >> >> >>> > > >> >>> > >> > >> -- >> >>> > > >> >>> > >> > >> Sebastian Wagner >> >>> > > >> >>> > >> > >> https://twitter.com/#!/dead_lock >> >>> > > >> >>> > >> > >> http://www.webbase-design.de >> >>> > > >> >>> > >> > >> http://www.wagner-sebastian.com >> >>> > > >> >>> > >> > >> [email protected] >> >>> > > >> >>> > >> > >> >> >>> > > >> >>> > >> > > >> >>> > > >> >>> > >> > > >> >>> > > >> >>> > >> > > >> >>> > > >> >>> > >> > > -- >> >>> > > >> >>> > >> > > Sebastian Wagner >> >>> > > >> >>> > >> > > https://twitter.com/#!/dead_lock >> >>> > > >> >>> > >> > > http://www.webbase-design.de >> >>> > > >> >>> > >> > > http://www.wagner-sebastian.com >> >>> > > >> >>> > >> > > [email protected] >> >>> > > >> >>> > >> > > >> >>> > > >> >>> > >> > >> >>> > > >> >>> > >> > >> >>> > > >> >>> > >> > >> >>> > > >> >>> > >> > -- >> >>> > > >> >>> > >> > Sebastian Wagner >> >>> > > >> >>> > >> > https://twitter.com/#!/dead_lock >> >>> > > >> >>> > >> > http://www.webbase-design.de >> >>> > > >> >>> > >> > http://www.wagner-sebastian.com >> >>> > > >> >>> > >> > [email protected] >> >>> > > >> >>> > >> > >> >>> > > >> >>> > >> >> >>> > > >> >>> > > >> >>> > > >> >>> > > >> >>> > > >> >>> > > >> >>> > > >> >>> > > -- >> >>> > > >> >>> > > Sebastian Wagner >> >>> > > >> >>> > > https://twitter.com/#!/dead_lock >> >>> > > >> >>> > > http://www.webbase-design.de >> >>> > > >> >>> > > http://www.wagner-sebastian.com >> >>> > > >> >>> > > [email protected] >> >>> > > >> >>> > >> >>> > > >> >>> >> >>> > > >> >> >> >>> > > >> >> >> >>> > > >> >> >> >>> > > >> >> -- >> >>> > > >> >> Sebastian Wagner >> >>> > > >> >> https://twitter.com/#!/dead_lock >> >>> > > >> >> http://www.webbase-design.de >> >>> > > >> >> http://www.wagner-sebastian.com >> >>> > > >> >> [email protected] >> >>> > > >> >> >>> > > >> >>> > >> >>> >> >> >> >> >> >> >> >> -- >> >> Sebastian Wagner >> >> https://twitter.com/#!/dead_lock >> >> http://www.webbase-design.de >> >> http://www.wagner-sebastian.com >> >> [email protected] >> > > > > -- > Sebastian Wagner > https://twitter.com/#!/dead_lock > http://www.webbase-design.de > http://www.wagner-sebastian.com > [email protected]
