Hi, Thanks for the modified jar. Now I can see the http jetty config row in the configuration tab. I have modified the connection time out to be 600000 ms (10 min), but still a request is not served if it is pending for more than 60 seconds. I have restarted the stanbol server as well but the connection time out setting does not seems to be absorbed by the system? What could be the issue?
Regards, Manish On Tue, May 14, 2013 at 2:34 AM, Rupert Westenthaler < [email protected]> wrote: > On Mon, May 13, 2013 at 11:30 AM, Manish Aggarwal <[email protected]> > wrote: > > Hi Rupert, > > > > I downloaded org.apache.felix.http.jetty-2.2.1-20111227.163236-1.jar< > https://repository.apache.org/content/groups/snapshots/org/apache/felix/org.apache.felix.http.jetty/2.2.1-SNAPSHOT/org.apache.felix.http.jetty-2.2.1-20111227.163236-1.jar > > > > from > > the link provided and installed through bundles tab. It installed > smoothly > > and after installation, it started showing version 2.2.1.SNAPSHOT against > > Apache Felix Http Jettyorg.apache.felix.http.jetty under bundles tab. > > I was assuming to see "org.apache.felix.http.timeout" setting > > underconfiguration tab, but I am unable to find any timeout setting under > > configuration. > > So, how we gonna configure this timeout? > > > > Year that is also what I was doing. The bundle installed fine and was > active. However I was no longer seeing the row for configuring the > "Apache Felix Jetty Based Http Service" in the configuration tab. The > reason for that was the > > org.apache.felix.metatype fromDocuments: Error parsing document > bundle://1.1:0/OSGI-INF/metatype/metatype.xml > > exception posted earlier in this thread. Only after fixing the > referenced metatype.xml file I was able to configure the timeout via > the configuration tab of the felix webconsole. > > I copied the bundle containing the fixed metatype.xml to [1] > > best > Rupert > > [1] > http://www.srfg.at/~rwesten/stanbol/org.apache.felix.http.jetty-2.2.1-20111227.163236-1.jar > > > Regards, > > Manish > > > > > > > > On Fri, May 10, 2013 at 4:09 PM, Rupert Westenthaler < > > [email protected]> wrote: > > > >> Hi Manish > >> > >> The timeout is just for the EventJobManager. If you are using the > >> RESTful services you will also need to set the HTTP timeout. In case > >> you are using the Stanbol runnable jar you will need to set the > >> "org.apache.felix.http.timeout" of the Felix HTTO service (see the > >> documentation at [1] for details). > >> > >> However the downside is that those properties are only supported with > >> the current SNAPSHOT version (version 2.2.1). However Stanbol uses the > >> last released version 2.2.0 that does not allow to set the timeout. So > >> if you want to configure the timeout you will need to install the > >> current SNAPSHOT version to your Stanbol Launcher. You can get a > >> SNAPSHOT from [2]. > >> > >> I tried it with the currently available SNAPSHOT version, but I had no > >> luck, because I was getting a > >> > >> 10.05.2013 12:22:09.833 *ERROR* [56699287@qtp-1281956591-5] > >> org.apache.felix.metatype fromDocuments: Error parsing document > >> bundle://1.1:0/OSGI-INF/metatype/metatype.xml > >> (org.xmlpull.v1.XmlPullParserException: expected: /AD read: OCD > >> (position:END_TAG </OCD>@46:11 in > >> bundle://1.1:0/OSGI-INF/metatype/metatype.xml) ) > >> org.xmlpull.v1.XmlPullParserException: expected: /AD read: OCD > >> (position:END_TAG </OCD>@46:11 in > >> bundle://1.1:0/OSGI-INF/metatype/metatype.xml) > >> > >> when installing the bundle. So I extracted the jar, manually fixed the > >> missing close tags for ~5 <AD .../> and created an updated jar. After > >> that I was able to install the bundle and configure the timeout. > >> > >> best > >> Rupert > >> > >> [1] > >> > http://felix.apache.org/documentation/subprojects/apache-felix-http-service.html > >> [2] > >> > https://repository.apache.org/content/groups/snapshots/org/apache/felix/org.apache.felix.http.jetty/2.2.1-SNAPSHOT/ > >> > >> On Fri, May 10, 2013 at 11:24 AM, Manish Aggarwal <[email protected]> > >> wrote: > >> > Hi, > >> > > >> > yeah it was a typo and it worked after correcting it. > >> > > >> > > >> > Now, when I run integration test for 10 threads, (10 simultaneous > >> > requests), some request failed because of "Connection time out". I > >> observed > >> > that all the request after 60 second whose response are pending gets > >> failed > >> > because of time out error. > >> > I found this "Apache Stanbol Enhancer Job Manager: Event" setting and > I > >> > have set "Job Timeout" to be 600000ms (10 min), but still I am getting > >> this > >> > connection time out error. When this Job Timeout setting is absorbed? > Do > >> I > >> > need to restart the server after updating this values? Is there any > other > >> > setting that needs to be done to increase the time out? > >> > > >> > Detailed Log attached below: > >> > > >> > > >> > [INFO] Scanning for projects... > >> > [INFO] > >> > > >> > [INFO] > >> > > ------------------------------------------------------------------------ > >> > [INFO] Building Apache Stanbol Integration Tests 0.10.0-SNAPSHOT > >> > [INFO] > >> > > ------------------------------------------------------------------------ > >> > [INFO] > >> > [INFO] --- maven-enforcer-plugin:1.0.1:enforce (enforce-java) @ > >> > org.apache.stanbol.integration-tests --- > >> > [INFO] > >> > [INFO] --- maven-remote-resources-plugin:1.4:process (default) @ > >> > org.apache.stanbol.integration-tests --- > >> > [INFO] > >> > [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ > >> > org.apache.stanbol.integration-tests --- > >> > [debug] execute contextualize > >> > [INFO] Using 'UTF-8' encoding to copy filtered resources. > >> > [INFO] skip non existing resourceDirectory > >> > /Users/labuser/stanbol/integration-tests/src/main/resources > >> > [INFO] Copying 3 resources > >> > [INFO] > >> > [INFO] --- maven-dependency-plugin:2.3:copy-dependencies > >> > (copy-runnable-jar) @ org.apache.stanbol.integration-tests --- > >> > [INFO] org.apache.stanbol.launchers.full-0.10.0-SNAPSHOT.jar already > >> exists > >> > in destination. > >> > [INFO] > >> > [INFO] --- build-helper-maven-plugin:1.5:reserve-network-port > >> > (reserve-server-port) @ org.apache.stanbol.integration-tests --- > >> > [INFO] Reserved port 63594 for http.port > >> > [INFO] > >> > [INFO] --- maven-compiler-plugin:2.3.1:compile (default-compile) @ > >> > org.apache.stanbol.integration-tests --- > >> > [INFO] No sources to compile > >> > [INFO] > >> > [INFO] --- maven-resources-plugin:2.5:testResources > >> (default-testResources) > >> > @ org.apache.stanbol.integration-tests --- > >> > [debug] execute contextualize > >> > [INFO] Using 'UTF-8' encoding to copy filtered resources. > >> > [INFO] Copying 8 resources > >> > [INFO] Copying 3 resources > >> > [INFO] > >> > [INFO] --- maven-compiler-plugin:2.3.1:testCompile > (default-testCompile) > >> @ > >> > org.apache.stanbol.integration-tests --- > >> > [INFO] Nothing to compile - all classes are up to date > >> > [INFO] > >> > [INFO] --- maven-surefire-plugin:2.11:test (default-test) @ > >> > org.apache.stanbol.integration-tests --- > >> > [INFO] Surefire report directory: > >> > /Users/labuser/stanbol/integration-tests/target/surefire-reports > >> > > >> > ------------------------------------------------------- > >> > T E S T S > >> > ------------------------------------------------------- > >> > Running org.apache.stanbol.enhancer.it.MultiThreadedTest > >> > 04:47:07,378 INFO StanbolTestBase:67 - test.server.url is set: not > >> > starting server jar (http://10.0.0.1) > >> > 04:47:07,428 INFO StanbolTestBase:100 - Will wait up to 180 seconds > for > >> > server to become ready > >> > 04:47:13,975 INFO StanbolTestBase:172 - Got expected content for all > >> > configured requests, server is ready > >> > 04:47:15,261 INFO MultiThreadedTest:198 - Enhancement engines checked > >> for > >> > '/enhancer?executionmetadata=true', all present > >> > 04:47:15,287 INFO MultiThreadedTest:164 - Read Testdata from > >> > 'germanText.txt' > >> > 04:47:15,287 INFO MultiThreadedTest:168 - ... init from File > >> > 04:47:15,288 INFO MultiThreadedTest:185 - ... init via Classpath > >> > 04:47:15,288 INFO MultiThreadedTest:189 - - InputStream: > >> FileInputStream > >> > 04:47:15,289 INFO MultiThreadedTest:207 - - uncompressed source: > >> > germanText.txt > >> > 04:47:15,320 INFO MultiThreadedTest:240 - - Media-Type: text/plain > >> > 04:47:15,321 INFO MultiThreadedTest:512 - ... using charset UTF-8 > for > >> > parsing Text data > >> > 04:47:15,321 INFO MultiThreadedTest:353 - Start Multi Thread testing > of > >> > max. 500 requests using 10 threads on Endpoint > >> > /enhancer/chain/mychain?executionmetadata=true > >> > 04:47:15,329 INFO MultiThreadedTest:368 - ... sent 0 Requests (0 > >> > finished, 1 pending, 0 failed > >> > 04:47:15,373 INFO MultiThreadedTest:373 - > All 10 requests sent! > >> > 04:47:15,373 INFO MultiThreadedTest:374 - ... wait for all > requests to > >> > complete > >> > 04:47:18,375 INFO MultiThreadedTest:377 - ... 0 finished, 10 > pending, > >> 0 > >> > failed > >> > 04:47:21,378 INFO MultiThreadedTest:377 - ... 0 finished, 10 > pending, > >> 0 > >> > failed > >> > 04:47:24,380 INFO MultiThreadedTest:377 - ... 0 finished, 10 > pending, > >> 0 > >> > failed > >> > 04:47:27,381 INFO MultiThreadedTest:377 - ... 0 finished, 10 > pending, > >> 0 > >> > failed > >> > 04:47:30,382 INFO MultiThreadedTest:377 - ... 0 finished, 10 > pending, > >> 0 > >> > failed > >> > 04:47:33,384 INFO MultiThreadedTest:377 - ... 0 finished, 10 > pending, > >> 0 > >> > failed > >> > 04:47:36,385 INFO MultiThreadedTest:377 - ... 0 finished, 10 > pending, > >> 0 > >> > failed > >> > 04:47:39,387 INFO MultiThreadedTest:377 - ... 0 finished, 10 > pending, > >> 0 > >> > failed > >> > 04:47:42,388 INFO MultiThreadedTest:377 - ... 0 finished, 10 > pending, > >> 0 > >> > failed > >> > 04:47:45,390 INFO MultiThreadedTest:377 - ... 0 finished, 10 > pending, > >> 0 > >> > failed > >> > 04:47:48,391 INFO MultiThreadedTest:377 - ... 1 finished, 9 > pending, 0 > >> > failed > >> > 04:47:51,393 INFO MultiThreadedTest:377 - ... 1 finished, 9 > pending, 0 > >> > failed > >> > 04:47:54,394 INFO MultiThreadedTest:377 - ... 2 finished, 8 > pending, 0 > >> > failed > >> > 04:47:57,396 INFO MultiThreadedTest:377 - ... 2 finished, 8 > pending, 0 > >> > failed > >> > 04:48:00,397 INFO MultiThreadedTest:377 - ... 3 finished, 7 > pending, 0 > >> > failed > >> > 04:48:03,399 INFO MultiThreadedTest:377 - ... 3 finished, 7 > pending, 0 > >> > failed > >> > 04:48:06,401 INFO MultiThreadedTest:377 - ... 3 finished, 7 > pending, 0 > >> > failed > >> > 04:48:09,402 INFO MultiThreadedTest:377 - ... 4 finished, 6 > pending, 0 > >> > failed > >> > 04:48:12,403 INFO MultiThreadedTest:377 - ... 4 finished, 6 > pending, 0 > >> > failed > >> > 04:48:15,404 INFO MultiThreadedTest:377 - ... 5 finished, 5 > pending, 0 > >> > failed > >> > 04:48:15,995 WARN MultiThreadedTest:720 - Error while sending Request > >> > java.lang.AssertionError: POST request to > >> > http://10.0.0.1/enhancer/chain/mychain?executionmetadata=true: > expecting > >> > status 200 (content: <HEAD><TITLE>Connection Timed Out</TITLE></HEAD> > >> > <BODY BGCOLOR="white" FGCOLOR="black"><H1>Connection Timed > Out</H1><HR> > >> > <FONT FACE="Helvetica,Arial"><B> > >> > Description: Connection Timed Out</B></FONT> > >> > <HR> > >> > <!-- default "Connection Timed Out" response (504) --> > >> > </BODY> > >> > > >> > > >> > > >> > > >> > > >> > o) expected:<200> but was:<504> > >> > at org.junit.Assert.fail(Assert.java:88) > >> > at org.junit.Assert.failNotEquals(Assert.java:743) > >> > at org.junit.Assert.assertEquals(Assert.java:118) > >> > at org.junit.Assert.assertEquals(Assert.java:555) > >> > at > >> > > >> > org.apache.stanbol.commons.testing.http.RequestExecutor.assertStatus(RequestExecutor.java:175) > >> > at > >> > > >> > org.apache.stanbol.enhancer.it.MultiThreadedTestBase$AsyncExecuter.run(MultiThreadedTestBase.java:716) > >> > at > >> > > >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > >> > at > >> > > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > >> > at java.lang.Thread.run(Thread.java:680) > >> > 04:48:16,008 WARN MultiThreadedTest:720 - Error while sending Request > >> > java.lang.AssertionError: POST request to > >> > http://10.0.0.1/enhancer/chain/mychain?executionmetadata=true: > expecting > >> > status 200 (content: <HEAD><TITLE>Connection Timed Out</TITLE></HEAD> > >> > <BODY BGCOLOR="white" FGCOLOR="black"><H1>Connection Timed > Out</H1><HR> > >> > <FONT FACE="Helvetica,Arial"><B> > >> > Description: Connection Timed Out</B></FONT> > >> > <HR> > >> > <!-- default "Connection Timed Out" response (504) --> > >> > </BODY> > >> > > >> > > >> > > >> > > >> > > >> > ) expected:<200> but was:<504> > >> > at org.junit.Assert.fail(Assert.java:88) > >> > at org.junit.Assert.failNotEquals(Assert.java:743) > >> > at org.junit.Assert.assertEquals(Assert.java:118) > >> > at org.junit.Assert.assertEquals(Assert.java:555) > >> > at > >> > > >> > org.apache.stanbol.commons.testing.http.RequestExecutor.assertStatus(RequestExecutor.java:175) > >> > at > >> > > >> > org.apache.stanbol.enhancer.it.MultiThreadedTestBase$AsyncExecuter.run(MultiThreadedTestBase.java:716) > >> > at > >> > > >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > >> > at > >> > > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > >> > at java.lang.Thread.run(Thread.java:680) > >> > 04:48:16,031 WARN MultiThreadedTest:720 - Error while sending Request > >> > java.lang.AssertionError: POST request to > >> > http://10.0.0.1/enhancer/chain/mychain?executionmetadata=true: > expecting > >> > status 200 (content: <HEAD><TITLE>Connection Timed Out</TITLE></HEAD> > >> > <BODY BGCOLOR="white" FGCOLOR="black"><H1>Connection Timed > Out</H1><HR> > >> > <FONT FACE="Helvetica,Arial"><B> > >> > Description: Connection Timed Out</B></FONT> > >> > <HR> > >> > <!-- default "Connection Timed Out" response (504) --> > >> > </BODY> > >> > > >> > > >> > > >> > > >> > > >> > l) expected:<200> but was:<504> > >> > at org.junit.Assert.fail(Assert.java:88) > >> > at org.junit.Assert.failNotEquals(Assert.java:743) > >> > at org.junit.Assert.assertEquals(Assert.java:118) > >> > at org.junit.Assert.assertEquals(Assert.java:555) > >> > at > >> > > >> > org.apache.stanbol.commons.testing.http.RequestExecutor.assertStatus(RequestExecutor.java:175) > >> > at > >> > > >> > org.apache.stanbol.enhancer.it.MultiThreadedTestBase$AsyncExecuter.run(MultiThreadedTestBase.java:716) > >> > at > >> > > >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > >> > at > >> > > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > >> > at java.lang.Thread.run(Thread.java:680) > >> > 04:48:16,159 WARN MultiThreadedTest:720 - Error while sending Request > >> > java.lang.AssertionError: POST request to > >> > http://10.0.0.1/enhancer/chain/mychain?executionmetadata=true: > expecting > >> > status 200 (content: <HEAD><TITLE>Connection Timed Out</TITLE></HEAD> > >> > <BODY BGCOLOR="white" FGCOLOR="black"><H1>Connection Timed > Out</H1><HR> > >> > <FONT FACE="Helvetica,Arial"><B> > >> > Description: Connection Timed Out</B></FONT> > >> > <HR> > >> > <!-- default "Connection Timed Out" response (504) --> > >> > </BODY> > >> > > >> > > >> > > >> > > >> > > >> > ) expected:<200> but was:<504> > >> > at org.junit.Assert.fail(Assert.java:88) > >> > at org.junit.Assert.failNotEquals(Assert.java:743) > >> > at org.junit.Assert.assertEquals(Assert.java:118) > >> > at org.junit.Assert.assertEquals(Assert.java:555) > >> > at > >> > > >> > org.apache.stanbol.commons.testing.http.RequestExecutor.assertStatus(RequestExecutor.java:175) > >> > at > >> > > >> > org.apache.stanbol.enhancer.it.MultiThreadedTestBase$AsyncExecuter.run(MultiThreadedTestBase.java:716) > >> > at > >> > > >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > >> > at > >> > > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > >> > at java.lang.Thread.run(Thread.java:680) > >> > 04:48:16,159 WARN MultiThreadedTest:720 - Error while sending Request > >> > java.lang.AssertionError: POST request to > >> > http://10.0.0.1/enhancer/chain/mychain?executionmetadata=true: > expecting > >> > status 200 (content: <HEAD><TITLE>Connection Timed Out</TITLE></HEAD> > >> > <BODY BGCOLOR="white" FGCOLOR="black"><H1>Connection Timed > Out</H1><HR> > >> > <FONT FACE="Helvetica,Arial"><B> > >> > Description: Connection Timed Out</B></FONT> > >> > <HR> > >> > <!-- default "Connection Timed Out" response (504) --> > >> > </BODY> > >> > > >> > > >> > > >> > > >> > > >> > ) expected:<200> but was:<504> > >> > at org.junit.Assert.fail(Assert.java:88) > >> > at org.junit.Assert.failNotEquals(Assert.java:743) > >> > at org.junit.Assert.assertEquals(Assert.java:118) > >> > at org.junit.Assert.assertEquals(Assert.java:555) > >> > at > >> > > >> > org.apache.stanbol.commons.testing.http.RequestExecutor.assertStatus(RequestExecutor.java:175) > >> > at > >> > > >> > org.apache.stanbol.enhancer.it.MultiThreadedTestBase$AsyncExecuter.run(MultiThreadedTestBase.java:716) > >> > at > >> > > >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > >> > at > >> > > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > >> > at java.lang.Thread.run(Thread.java:680) > >> > 04:48:18,406 INFO MultiThreadedTest:377 - ... 10 finished, 0 > pending, > >> 5 > >> > failed > >> > 04:48:18,406 INFO MultiThreadedTest:380 - Multi Thread testing of 10 > >> > requests (failed: 5) using 10 threads completed > >> > 04:48:18,406 INFO MultiThreadedTest:658 - Statistics: > >> > 04:48:18,407 INFO MultiThreadedTest:660 - Chain: > >> > 04:48:18,407 INFO MultiThreadedTest:661 - Round Trip Time (Server + > >> > Transfer + Client): > >> > 04:48:18,407 INFO MultiThreadedTest:665 - max: 58704ms | min: > >> 29648ms > >> > | avr: 44424ms over 5 requests > >> > 04:48:18,407 INFO MultiThreadedTest:671 - Processing Time (server > >> side) > >> > 04:48:18,408 INFO MultiThreadedTest:676 - max: 54943ms | min: > >> 28165ms > >> > | avr: 41942ms over 5 requests > >> > 04:48:18,408 INFO MultiThreadedTest:681 - Bandwith Consumption > >> (received > >> > data) > >> > 04:48:18,408 INFO MultiThreadedTest:682 - max: 14KB | min: 14KB > | > >> > avr: 14KB received over 5 requests > >> > 04:48:18,408 INFO MultiThreadedTest:687 - Enhancement Engines > >> > 04:48:18,409 INFO MultiThreadedTest:689 - entityhubExtraction: max: > >> > 9711ms | min: 2046ms | avr: 5067ms over 5 requests > >> > 04:48:18,409 INFO MultiThreadedTest:689 - langdetect: max: 28ms | > min: > >> > 4ms | avr: 15ms over 5 requests > >> > 04:48:18,409 INFO MultiThreadedTest:689 - manthan: max: 44433ms | > min: > >> > 22246ms | avr: 36546ms over 5 requests > >> > 04:48:18,410 INFO MultiThreadedTest:689 - opennlp-ner: max: 345ms | > >> min: > >> > 3ms | avr: 79ms over 5 requests > >> > 04:48:18,410 INFO MultiThreadedTest:689 - opennlp-pos: max: 379ms | > >> min: > >> > 107ms | avr: 205ms over 5 requests > >> > 04:48:18,410 INFO MultiThreadedTest:689 - opennlp-token: max: 21ms > | > >> > min: 5ms | avr: 11ms over 5 requests > >> > 04:48:18,410 INFO MultiThreadedTest:689 - tika: max: 37ms | min: > 4ms | > >> > avr: 18ms over 5 requests > >> > 04:48:18,411 WARN MultiThreadedTest:383 - Content(s) of Faild tests: > >> > 04:48:18,411 WARN MultiThreadedTest:386 - Failed (1): > >> > 04:48:18,411 WARN MultiThreadedTest:387 - > Request: {}POST > >> > http://10.0.0.1/enhancer/chain/mychain?executionmetadata=trueHTTP/1.1 > >> > 04:48:18,412 WARN MultiThreadedTest:388 - > Response: {}HTTP/1.1 > 504 > >> > Connection Timed Out [Date: Thu, 09 May 2013 11:48:15 GMT, Connection: > >> > close, Via: 1.1 wsg.noida, Cache-Control: no-store, Content-Type: > >> > text/html, Content-Language: en, Content-Length: 667] > >> > 04:48:18,412 WARN MultiThreadedTest:390 - - Status: HTTP/1.1 504 > >> > Connection Timed Out > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > On Wed, May 8, 2013 at 7:06 PM, Rupert Westenthaler < > >> > [email protected]> wrote: > >> > > >> >> Hi Manish > >> >> > >> >> Also took me some time to figure this out, but you have a typo in the > >> >> name of the System property that parses the chain name > >> >> > >> >> On Wed, May 8, 2013 at 2:55 PM, Manish Aggarwal <[email protected] > > > >> >> wrote: > >> >> > Lab-Computer-28b68b:integration-tests labuser$ mvn -o test > >> >> > -Dtest=MultiThreadedTest > >> >> -Dstanbol.it.multithreadedtest.chain=sample-chain > >> >> > >> >> you use > >> >> > >> >> stanbol.it.multithreadedtest.chain > >> >> > >> >> but it should be > >> >> > >> >> stanbol.it.multithreadtest.chain > >> >> > >> >> best > >> >> Rupert > >> >> > >> >> > >> >> -- > >> >> | Rupert Westenthaler [email protected] > >> >> | Bodenlehenstraße 11 ++43-699-11108907 > >> >> | A-5500 Bischofshofen > >> >> > >> > >> > >> > >> -- > >> | Rupert Westenthaler [email protected] > >> | Bodenlehenstraße 11 ++43-699-11108907 > >> | A-5500 Bischofshofen > >> > > > > -- > | Rupert Westenthaler [email protected] > | Bodenlehenstraße 11 ++43-699-11108907 > | A-5500 Bischofshofen >
