On 20 March 2017 at 10:11, Tuukka Mustonen <[email protected]> wrote: > Thanks! > > I am not sure about the property name though - in normal retrying, > idempotent requests (but not non-idempotent) are indeed resent. It does no > harm - two subsequent idemponent requests result in same data as just one.
Whilst it's true that multiple idempotent requests don't change the server state, they do affect the test results. I'm not sure if the retries are counted in the stats or not. If not, then the calculated throughput will be reduced. If they are included, then more requests will be counted than were expected. Either way, if retries are enabled this needs to be allowed for when analysing results. > So, property "request_sent_retry_enabled" sounds like it would enable > retrying (or a subset of it) in general (to actually retry failures where > request was sent but response wasn't read properly). I think better name > would be something like "retry_all_methods"? > > Also, what remains a bit weird here is that in my tests HC 4 retrying is > broken in JMeter 3.1 (but worked in snapshot build) but it works for you? > > Tuukka > > > On Sat, Mar 18, 2017 at 7:35 PM, Philippe Mouawad < > [email protected]> wrote: > >> Hi Tuukka, >> Thanks for your feedback. >> This feature has now been added cleanly and documented to trunk within bug: >> https://bz.apache.org/bugzilla/show_bug.cgi?id=60888 >> >> # true if it's OK to retry requests that have been sent >> # This will retry Idempotent and non Idempotent requests >> # This should usually be false, but it can be useful >> # when testing against some Load Balancers like Amazon ELB >> #httpclient4.request_sent_retry_enabled=false >> >> See: >> - >> https://github.com/apache/httpclient/blob/4.5.x/ >> httpclient/src/main/java/org/apache/http/impl/client/ >> DefaultHttpRequestRetryHandler.java#L160 >> >> As you can guess, settings this to true means that a request could be >> received twice as we retry requests already sent. >> So this means that even Non Idempotent requests are retried. >> >> >> Regards >> Philippe >> >> On Wed, Mar 15, 2017 at 4:45 PM, Tuukka Mustonen < >> [email protected]> >> wrote: >> >> > Ok, I added test that uses POST/PATCH/DELETE/GET methods. >> > >> > With #6020 build of JMeter >> > and httpclient4.request_sent_retry_enabled=false, there were some >> retries, >> > but mostly not. I think it retried GET (and maybe DELETE) methods but not >> > anything else. >> > >> > With httpclient4.request_sent_retry_enabled=true, all failed requests >> were >> > retried and this resulted in zero errors. >> > >> > So yeah, this new option is something I would need/want with ALB (until >> > they fix the issue, I still haven't received reply if AWS considers it a >> > bug or a feature). >> > >> > Though I am still not sure what it actually does: >> > >> > *"However, can you crystalize what that new property is supposed to do? >> As >> > I wrote above I thought it was to retry also non-idempotent methods, but >> I >> > guess it's actually supposed to do something else?"* >> > >> > Tuukka >> > >> > >> > On Fri, Mar 10, 2017 at 12:08 PM, Tuukka Mustonen < >> > [email protected] >> > > wrote: >> > >> > > > You confirm it's this night nightly or yesterday's one ? >> > > > Did you use the property: >> > > > httpclient4.request_sent_retry_enabled=true >> > > >> > > It was this build: https://builds.apache.org/job/JMeter-trunk/6020/ >> > > >> > > No, I didn't enable that property - I thought that property was to also >> > > retry non-idempotent methods. >> > > >> > > > If you used httpclient4.request_sent_retry_enabled=true and it works >> > > while >> > > > it doesn't with httpclient4.request_sent_retry_enabled=false >> > > > It's a good reason to add it. >> > > > BUt if it already works with : >> > > > httpclient4.request_sent_retry_enabled=false >> > > > >> > > > Then it might not be useful. Can you give feedback on this ? >> > > >> > > Yeah, I didn't declare it at so I guess it's not needed then... >> > > >> > > However, can you crystalize what that new property is supposed to do? >> As >> > I >> > > wrote above I thought it was to retry also non-idempotent methods, but >> I >> > > guess it's actually supposed to do something else? >> > > >> > > > And we'll wait a week for your test on non idempotent methods. >> > > >> > > Ok. >> > > >> > > > Yes but it's not a good setting, in this case better use stalecheck. >> > > >> > > Ok. >> > > >> > > Tuukka >> > > >> > > >> > > On Fri, Mar 10, 2017 at 11:27 AM, Philippe Mouawad < >> > > [email protected]> wrote: >> > > >> > >> On Fri, Mar 10, 2017 at 8:08 AM, Tuukka Mustonen < >> > >> [email protected]> >> > >> wrote: >> > >> >> > >> > > > +DELETE also like sebb said. >> > >> > > > >> > >> > > True but iIt was not concerned by bug. >> > >> > >> > >> > Maybe I got something wrong, but you did paste DELETE originally as >> > >> > non-idemponent in list of non-retryable methods: >> > >> > >> > >> > Delete not being implementation of HttpEntityEnclosingRequest, it >> was >> > >> not >> > >> concerned by issue. >> > >> Yes I thought wrongly Delete was not idempotent >> > >> >> > >> > > - Idempotent HTTP methods which are by default those that do not >> > >> > implement >> > >> > > > HttpEntityEnclosingRequest, so not POST, PUT,DELETE, PATCH, GET >> > With >> > >> > body >> > >> > >> > >> > > 1/ To diagnose set those classes in debug mode: >> > >> > > org.apache.http.impl.client.DefaultRequestDirector >> > >> > > >> > >> > > You should see: >> > >> > > Retrying connect to >> > >> > > Retrying request to >> > >> > > >> > >> > > Log level can be set in log4j2.xml >> > >> > >> > >> > Ok, with latest nightly build, retrying works just fine, I got: >> > >> > >> > >> > 2017-03-10 08:50:57,856 INFO o.a.j.p.h.s.HTTPHC4Impl$3: I/O >> exception >> > >> > (org.apache.http.NoHttpResponseException) caught when processing >> > >> request >> > >> > to >> > >> > {}->http://client-api.perf.ew1.cosmos-ci.fsapi.com:80: The target >> > >> server >> > >> > failed to respond >> > >> > 2017-03-10 08:50:57,856 INFO o.a.j.p.h.s.HTTPHC4Impl$3: Retrying >> > >> request to >> > >> > {}->http://client-api.perf.ew1.cosmos-ci.fsapi.com:80 >> > >> > >> > >> > However, with exact same testplan/configuration/scenario, retrying >> > just >> > >> > does not work with HC4 on JMeter 3.1. Unfortunately, I wasn't able >> to >> > >> > configure logging on JMeter 3.1 so that it would print similar >> lines. >> > I >> > >> > tried adding to user.properties: >> > >> > >> > >> > log_level.org.apache.http.impl.client.DefaultRequestDirector=DEBUG >> > >> > >> > >> > But that didn't do anything. Also tried tuning bin/log4j.conf. Maybe >> > >> that >> > >> > class does not exist in JMeter 3.1? >> > >> > >> > >> > Anyway, I can confirm that retying with HC4 does not work on JMeter >> > 3.1 >> > >> but >> > >> > does work on latest nightly. >> > >> >> > >> >> > >> You confirm it's this night nightly or yesterday's one ? >> > >> Did you use the property: >> > >> httpclient4.request_sent_retry_enabled=true >> > >> >> > >> >> > >> > Not for me at least (I don't believe I have >> > >> > anything special in my JMeter 3.1 installation, just same plugins >> > that I >> > >> > downloaded into nightly build also). >> > >> > >> > >> > > I have added a property that you can set and try in >> user.properties: >> > >> > > httpclient4.request_sent_retry_enabled=true >> > >> > > Please give your feedback rapidly, so that we decide what to do if >> > it >> > >> > has an >> > >> > effect. >> > >> > >> > >> > Unfortunately, I don't have a test case with non-idempotent method >> at >> > >> hand >> > >> > just now. I can give it a try at the start of next week. >> > >> > >> > >> >> > >> If you used httpclient4.request_sent_retry_enabled=true and it works >> > >> while >> > >> it doesn't with httpclient4.request_sent_retry_enabled=false >> > >> It's a good reason to add it. >> > >> BUt if it already works with : >> > >> httpclient4.request_sent_retry_enabled=false >> > >> >> > >> Then it might not be useful. Can you give feedback on this ? >> > >> And we'll wait a week for your test on non idempotent methods. >> > >> >> > >> > >> > >> > > But setting httpclient4.validate_after_inactivity=0 means you >> > >> disable >> > >> > the >> > >> > > validation which is not a good idea as you'll end up having broken >> > >> > > connection. >> > >> > >> > >> > Maybe httpclient4.validate_after_inactivity=1 then? >> > >> >> > >> >> > >> Yes but it's not a good setting, in this case better use stalecheck. >> > >> >> > >> >> > >> > I assume that's pretty >> > >> > much the same as the old stale check? >> > >> > >> > >> >> > >> >> > >> >> > >> > >> > >> > Tuukka >> > >> > >> > >> > >> > >> > On Thu, Mar 9, 2017 at 10:38 PM, Philippe Mouawad < >> > >> > [email protected]> wrote: >> > >> > >> > >> > > On Thu, Mar 9, 2017 at 5:28 PM, Tuukka Mustonen < >> > >> > [email protected] >> > >> > > > >> > >> > > wrote: >> > >> > > >> > >> > > > Sorry for the late reply, >> > >> > > > >> > >> > > > About customizing retry method whitelist: >> > >> > > > >> > >> > > > > You case is a bit soecific no ? >> > >> > > > >> > >> > > > True, it's very undesirable scenario, but that's what you get >> with >> > >> AWS >> > >> > > ALB >> > >> > > > for now - I believe there are lots and lots of other users that >> > will >> > >> > face >> > >> > > > the same problem. >> > >> > > > >> > >> > > > I have added a property that you can set and try in >> > user.properties: >> > >> > > httpclient4.request_sent_retry_enabled=true >> > >> > > >> > >> > > Please give your feedback rapidly, so that we decide what to do if >> > it >> > >> has >> > >> > > an effect. >> > >> > > >> > >> > > Having said that, I am in discussion with AWS about if they will >> fix >> > >> it >> > >> > and >> > >> > > > with what timeline - current behavior is complete nonsense... >> > >> > > > >> > >> > > > > In this case, the bug is larger than what I thought as for now >> > we >> > >> > > > consider >> > >> > > > > PUT as non idempotent. >> > >> > > > >> > >> > > > +DELETE also like sebb said. >> > >> > > > >> > >> > > True but iIt was not concerned by bug. >> > >> > > >> > >> > > > >> > >> > > > > 1/ To diagnose set those classes in debug mode: >> > >> > > > > org.apache.http.impl.client.DefaultRequestDirector >> > >> > > > > >> > >> > > > > You should see: >> > >> > > > > Retrying connect to >> > >> > > > > Retrying request to >> > >> > > > > >> > >> > > > > Log level can be set in log4j2.xml >> > >> > > > >> > >> > > > Thanks! I'll try to find time to check that tomorrow... >> > >> > > > >> > >> > > > > 2/ I confirm stalecheck is still available. >> > >> > > > >> > >> > > > Can you clarify what is the difference to having >> > >> > > > httpclient4.validate_after_inactivity=0? >> > >> > > > >> > >> > > >> > >> > > It was an optimisation added by HC4 to replace stalecheck which >> was >> > >> very >> > >> > > costly. >> > >> > > But setting httpclient4.validate_after_inactivity=0 means you >> > >> disable >> > >> > the >> > >> > > validation which is not a good idea as you'll end up having broken >> > >> > > connection. >> > >> > > This value should be set to a value lower than the keepalive set >> on >> > >> > server >> > >> > > side. >> > >> > > >> > >> > > >> > >> > > > Tuukka >> > >> > > > >> > >> > > > >> > >> > > > On Thu, Mar 9, 2017 at 2:23 PM, Philippe Mouawad < >> > >> > > > [email protected] >> > >> > > > > wrote: >> > >> > > > >> > >> > > > > Thanks for link sebb. >> > >> > > > > >> > >> > > > > In this case, the bug is larger than what I thought as for now >> > we >> > >> > > > consider >> > >> > > > > PUT as non idempotent. >> > >> > > > > >> > >> > > > > I'll commit a new fix. >> > >> > > > > Regards >> > >> > > > > >> > >> > > > > On Thu, Mar 9, 2017 at 12:20 PM, sebb <[email protected]> >> wrote: >> > >> > > > > >> > >> > > > > > On 9 March 2017 at 06:42, Philippe Mouawad < >> > >> > > [email protected] >> > >> > > > > >> > >> > > > > > wrote: >> > >> > > > > > > On Thursday, March 9, 2017, Tuukka Mustonen < >> > >> > > > [email protected] >> > >> > > > > > >> > >> > > > > > > wrote: >> > >> > > > > > > >> > >> > > > > > >> Hi Philippe and thanks for speedy actions! >> > >> > > > > > >> >> > >> > > > > > >> 1) Retrying only idempotent requests makes sense, as >> usual. >> > >> But >> > >> > > this >> > >> > > > > > (retry >> > >> > > > > > >> on idempotent) seems to be only documented on wiki page ( >> > >> > > > > > >> https://wiki.apache.org/jmeter/JMeterSocketClosed). You >> > >> should >> > >> > > add >> > >> > > > it >> > >> > > > > > also >> > >> > > > > > >> to http://jmeter.apache.org/userm >> > >> anual/component_reference.html >> > >> > . >> > >> > > > > > >> >> > >> > > > > > >> 2) In this case, the requests are GETs (and without body) >> > so >> > >> > they >> > >> > > > > > should be >> > >> > > > > > >> retried. >> > >> > > > > > > >> > >> > > > > > > Yes. >> > >> > > > > > > I debugged it and I confirm they are unless you face the >> > >> > > exceptions I >> > >> > > > > > > mentionned. >> > >> > > > > > > >> > >> > > > > > >> >> > >> > > > > > >> So the bug you fixed shouldn't affect this scenario. >> > >> > > > > > > >> > >> > > > > > > Yes >> > >> > > > > > > >> > >> > > > > > >> >> > >> > > > > > >> There must be >> > >> > > > > > >> something else - any pointers to what logging >> module/level >> > I >> > >> > > should >> > >> > > > > > enable >> > >> > > > > > >> to inspect the (failing) retry logic? >> > >> > > > > > > >> > >> > > > > > > As you can see there is no logging in this HC4 class >> > >> > > > > > > Try the calling class. I' ll provide its name later >> > >> > > > > > > >> > >> > > > > > >> >> > >> > > > > > >> 3) AFAIK PUT is idempotent - shouldn't you retry also >> that? >> > >> > > > > > > >> > >> > > > > > > No it's not IMU, it changes state of server. >> > >> > > > > > >> > >> > > > > > PUT *is* idempotent. >> > >> > > > > > >> > >> > > > > > It may change the state of the server the first time it is >> > sent, >> > >> > but >> > >> > > > > > subsequent PUTs should not change the state further. >> > >> > > > > > So it can be repeated as required. >> > >> > > > > > >> > >> > > > > > Similarly for DELETE >> > >> > > > > > >> > >> > > > > > https://tools.ietf.org/html/rfc2616#section-9.1.2 >> > >> > > > > > >> > >> > > > > > >> >> > >> > > > > > >> 4) In this case, once I add tests for POST/PATCH/DELETE >> > etc. >> > >> I >> > >> > > also >> > >> > > > > > want to >> > >> > > > > > >> retry these non-idempotent requests (to tolerate ALBs >> ugly >> > >> > > > behavior). >> > >> > > > > > These >> > >> > > > > > >> are "just" performance tests so I don't care if I'm >> > creating >> > >> > > > duplicate >> > >> > > > > > >> data. Is there way to specify custom method whitelist to >> > >> retry >> > >> > any >> > >> > > > > HTTP >> > >> > > > > > >> method? I see requestSentRetryEnabled in HC4 source code >> - >> > >> can I >> > >> > > > > enable >> > >> > > > > > >> that somehow? >> > >> > > > > > > >> > >> > > > > > > Not for now. >> > >> > > > > > > You case is a bit soecific no ? >> > >> > > > > > > >> > >> > > > > > >> >> > >> > > > > > >> 5) *Related question: >> > >> > > > > > >> is http.connection.stalecheck$Boolean=false (in >> > >> hc.parameters >> > >> > > file) >> > >> > > > > > valid >> > >> > > > > > >> anymore on JMeter 3.x with improved retry/stale logic >> > >> > > > > > >> (and httpclient4.validate_after_inactivity)? The line is >> > >> still >> > >> > > > there >> > >> > > > > in >> > >> > > > > > >> bundled hc.parameters file...* >> > >> > > > > > > >> > >> > > > > > > i'll double check >> > >> > > > > > > >> > >> > > > > > >> >> > >> > > > > > >> Tuukka >> > >> > > > > > >> >> > >> > > > > > >> >> > >> > > > > > >> On Thu, Mar 9, 2017 at 12:25 AM, Philippe Mouawad < >> > >> > > > > > >> [email protected] <javascript:;>> wrote: >> > >> > > > > > >> >> > >> > > > > > >> > Hello, >> > >> > > > > > >> > The issue with Get with body should be fixed now: >> > >> > > > > > >> > - https://bz.apache.org/bugzilla/show_bug.cgi?id=60837 >> > >> > > > > > >> > >> > >> > > > > > >> > Regards >> > >> > > > > > >> > Philippe >> > >> > > > > > >> > >> > >> > > > > > >> > On Wed, Mar 8, 2017 at 8:53 PM, Philippe Mouawad < >> > >> > > > > > >> > [email protected] <javascript:;> >> > >> > > > > > >> > > wrote: >> > >> > > > > > >> > >> > >> > > > > > >> > > Hello Tuukka, >> > >> > > > > > >> > > >> > >> > > > > > >> > > In my recent tests I didn't face any issue with >> > >> > > > > > httpclient4.retrycount. >> > >> > > > > > >> > > For me it works but be aware that JMeter (HC4) does >> > not >> > >> > retry >> > >> > > > all >> > >> > > > > > >> > > requests, it only retries those it is allowed to : >> > >> > > > > > >> > > - Idempotent HTTP methods which are by default those >> > >> that do >> > >> > > not >> > >> > > > > > >> > implement >> > >> > > > > > >> > > HttpEntityEnclosingRequest, so not POST, PUT,DELETE, >> > >> PATCH, >> > >> > > GET >> > >> > > > > With >> > >> > > > > > >> body >> > >> > > > > > >> > > (<= That might be a bug thinking more about it) >> > >> > > > > > >> > > - Non retriable exceptions >> > (InterruptedIOException.class, >> > >> > > > > > >> > > UnknownHostException.class, ConnectException.class, >> > >> > > > > > >> SSLException.class) >> > >> > > > > > >> > > - + Some other reasons >> > >> > > > > > >> > > >> > >> > > > > > >> > > See: >> > >> > > > > > >> > > https://github.com/apache/httpclient/blob/4.5.x/ >> > >> > > > > > >> > > httpclient/src/main/java/org/ >> apache/http/impl/client/ >> > >> > > > > > >> > > DefaultHttpRequestRetryHandler.java#L129 >> > >> > > > > > >> > > >> > >> > > > > > >> > > >> > >> > > > > > >> > > Regards >> > >> > > > > > >> > > Philippe >> > >> > > > > > >> > > >> > >> > > > > > >> > > >> > >> > > > > > >> > > On Wed, Mar 8, 2017 at 2:14 PM, Tuukka Mustonen < >> > >> > > > > > >> > [email protected] <javascript:;> >> > >> > > > > > >> > > > wrote: >> > >> > > > > > >> > > >> > >> > > > > > >> > >> My problem is that AWS Application Load Balancer >> (ALB) >> > >> > > > terminates >> > >> > > > > > all >> > >> > > > > > >> > >> existing connections during configuration changes >> > >> > (including >> > >> > > > > > >> > >> auto-scaling). >> > >> > > > > > >> > >> As my perf tests trigger auto-scaling, I want to >> retry >> > >> > failed >> > >> > > > > > requests >> > >> > > > > > >> > >> that >> > >> > > > > > >> > >> ALB connection termination causes. >> > >> > > > > > >> > >> >> > >> > > > > > >> > >> This results in a bunch of NoHttpResponseException >> > >> whenever >> > >> > > > > scaling >> > >> > > > > > >> > occurs >> > >> > > > > > >> > >> (=when ALB terminates existing connections). >> > >> > > > > > >> > >> >> > >> > > > > > >> > >> (And yeah, this load balancer behavior is weird and >> > ugly >> > >> > but >> > >> > > > it's >> > >> > > > > > what >> > >> > > > > > >> > >> they >> > >> > > > > > >> > >> confirmed). >> > >> > > > > > >> > >> >> > >> > > > > > >> > >> Using HttpClient 4, In user.properties I have set: >> > >> > > > > > >> > >> >> > >> > > > > > >> > >> httpclient4.retrycount=1 >> > >> > > > > > >> > >> >> > >> > > > > > >> > >> But that does nothing. I even tried: >> > >> > > > > > >> > >> >> > >> > > > > > >> > >> httpclient4.retrycount=100000000 >> > >> > > > > > >> > >> >> > >> > > > > > >> > >> But zero effect. >> > >> > > > > > >> > >> >> > >> > > > > > >> > >> Switching to HttpClient 3.1 reproduces the problem. >> > >> > However, >> > >> > > > with >> > >> > > > > > >> > >> HttpClient 3 and: >> > >> > > > > > >> > >> >> > >> > > > > > >> > >> httpclient3.retrycount=1 >> > >> > > > > > >> > >> >> > >> > > > > > >> > >> The problem vanishes so I assume retrying then >> works. >> > >> > > > > > >> > >> >> > >> > > > > > >> > >> I couldn't find where retry-attempts are logged so I >> > >> have >> > >> > no >> > >> > > > > > "proof" >> > >> > > > > > >> > that >> > >> > > > > > >> > >> HttpClient 4 wouldn't actually retry, but a) >> > retrycount >> > >> > makes >> > >> > > > > > >> difference >> > >> > > > > > >> > >> on >> > >> > > > > > >> > >> HttpClient 3.1 but not on 4 b) I would assume my >> whole >> > >> > > process >> > >> > > > > > should >> > >> > > > > > >> > >> crash >> > >> > > > > > >> > >> with retrycount=100000000 if it was really applied. >> > >> > > > > > >> > >> >> > >> > > > > > >> > >> Related question: is http.connection.stalecheck$ >> > >> > > Boolean=false >> > >> > > > > (in >> > >> > > > > > >> > >> hc.parameters file) valid anymore on JMeter 3.x with >> > >> > improved >> > >> > > > > > >> > retry/stale >> > >> > > > > > >> > >> logic (and httpclient4.validate_after_inactivity)? >> > The >> > >> > line >> > >> > > is >> > >> > > > > > still >> > >> > > > > > >> > >> there >> > >> > > > > > >> > >> in bundled hc.parameters file... >> > >> > > > > > >> > >> >> > >> > > > > > >> > >> Tested on JMeter 3.1. >> > >> > > > > > >> > >> >> > >> > > > > > >> > >> Any open (or already fixed) tickets about this? >> > Couldn't >> > >> > find >> > >> > > > > > any... >> > >> > > > > > >> > >> >> > >> > > > > > >> > >> FYI: to describe better what happens during ALB >> > >> connection >> > >> > > > > > >> termination: >> > >> > > > > > >> > >> >> > >> > > > > > >> > >> ...Successful communication with keep-alive... >> > >> > > > > > >> > >> - ALB responds to a request, and sends Connection: >> > >> > keep-alive >> > >> > > > so >> > >> > > > > > >> JMeter >> > >> > > > > > >> > >> leaves the connection open >> > >> > > > > > >> > >> - JMeter sends a new request >> > >> > > > > > >> > >> - ALB may might or might not ack it (not sure if >> there >> > >> was >> > >> > > > > pattern) >> > >> > > > > > >> > >> - ALB closes connection on TCP level (FIN) >> > >> > > > > > >> > >> - Connection gets closed and so sent request failed >> > and >> > >> > needs >> > >> > > > to >> > >> > > > > be >> > >> > > > > > >> > >> retried >> > >> > > > > > >> > >> >> > >> > > > > > >> > >> I think this generates NoHttpResponseException in >> > >> JMeter. >> > >> > > > > > >> > >> >> > >> > > > > > >> > >> Tuukka >> > >> > > > > > >> > >> >> > >> > > > > > >> > > >> > >> > > > > > >> > > >> > >> > > > > > >> > > >> > >> > > > > > >> > > -- >> > >> > > > > > >> > > Cordialement. >> > >> > > > > > >> > > Philippe Mouawad. >> > >> > > > > > >> > > >> > >> > > > > > >> > > >> > >> > > > > > >> > > >> > >> > > > > > >> > >> > >> > > > > > >> > >> > >> > > > > > >> > -- >> > >> > > > > > >> > Cordialement. >> > >> > > > > > >> > Philippe Mouawad. >> > >> > > > > > >> > >> > >> > > > > > >> >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > -- >> > >> > > > > > > Cordialement. >> > >> > > > > > > Philippe Mouawad. >> > >> > > > > > >> > >> > > > > > ------------------------------ >> ------------------------------ >> > >> > > --------- >> > >> > > > > > To unsubscribe, e-mail: [email protected] >> > >> > > > > > For additional commands, e-mail: >> [email protected] >> > >> > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > > >> > >> > > > > -- >> > >> > > > > Cordialement. >> > >> > > > > Philippe Mouawad. >> > >> > > > > >> > >> > > > >> > >> > > >> > >> > > >> > >> > > >> > >> > > -- >> > >> > > Cordialement. >> > >> > > Philippe Mouawad. >> > >> > > >> > >> > >> > >> >> > >> >> > >> >> > >> -- >> > >> Cordialement. >> > >> Philippe Mouawad. >> > >> >> > > >> > > >> > >> >> >> >> -- >> Cordialement. >> Philippe Mouawad. >> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
