Re: HTTP response is not being chunked
Yes, you are right - the 'Accept-Encoding: gzip' header is the cause.
Adding that to the curl request causes the response to block, while
removing it from Apache HTTP client request prevents the response blocking.
But I'm not sure why this would be the case. Gzip compression is
something that can be done of a streaming basis so there should be no
need to read the entire dataset.
Turning off compression will have a big performance impact!
Tim
On 16/01/2019 18:48, Luiz Eduardo Valmont wrote:
Hi Tim!
Sorry about the delay. I was summoned into a meeting that took a while.
Plus my PC is in the process of upgrading from Debian 9 to Debian
10/testing.
On the important matter.
There are certainly differences when it comes to headers. But I suspect the
culprit is the accept-encoding. I don't think it's the Camel headers.
I believe because it has to compress the content, it has to read everything
beforehand. Try adding "-H 'Accept-Encoding: gzip'" (double quotes
excluded) to your curl command and see if it changes the actual behaviour.
If it does, then you'll have to configure your client accordingly.
In any case, you can emulate the Async request in full by configuring the
headers with curl's "-H" parameter.
HTH
On Wed, 16 Jan 2019, 15:17 Tim Dudgeon
Hi Luiz,
here are the request headers. For Apache HTTPClient:
accept -> chemical/x-mdl-sdfile
accept-encoding -> gzip
breadcrumbId -> ID-6667e3dab68a-38877-1547658276076-0-15
CamelHttpMethod -> POST
CamelHttpPath ->
CamelHttpQuery -> null
CamelHttpServletRequest ->
org.apache.catalina.connector.RequestFacade@7551baa3
CamelHttpServletResponse ->
org.apache.catalina.connector.ResponseFacade@60e05546
CamelHttpUri ->
/chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
CamelHttpUrl ->
http://localhost:8092/chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
CamelServletContextPath -> /v1/converters/dataset_to_sdf
connection -> Keep-Alive
content-encoding -> gzip
Content-Type -> application/x-squonk-dataset-molecule+json
host -> localhost:8092
transfer-encoding -> chunked
user-agent -> Apache-HttpAsyncClient/4.1.4 (Java/1.8.0_191)
For curl:
accept -> chemical/x-mdl-sdfile
breadcrumbId -> ID-09b7601451a2-34027-1547658868812-0-15
CamelHttpMethod -> POST
CamelHttpPath ->
CamelHttpQuery -> null
CamelHttpServletRequest ->
org.apache.catalina.connector.RequestFacade@3af9ee39
CamelHttpServletResponse ->
org.apache.catalina.connector.ResponseFacade@1f33d20b
CamelHttpUri ->
/chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
CamelHttpUrl ->
http://localhost:8092/chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
CamelServletContextPath -> /v1/converters/dataset_to_sdf
content-encoding -> gzip
content-length -> 19397182
Content-Type -> application/x-squonk-dataset-molecule+json
expect -> 100-continue
host -> localhost:8092
user-agent -> curl/7.55.1
On 16/01/2019 16:55, Luiz Eduardo Valmont wrote:
Hi, Tim!
I'd suggest trying to intercept / dump the HttpAsyncClient request as
well
as the curl one. A simple "nc -l -p " in an *NIX OS will do.
My guess is that the perceived difference lies in the request headers.
Might be in the Accept.
Can you post both requests in full, body excluded?
Thanks!
On Wed, 16 Jan 2019, 14:46 Tim Dudgeon
I'm hitting a strange problem with a route that is using the servlet
component to send a large stream of data in its response.
This is part of a route that uses the REST DSL which is set up like
this:
restConfiguration().component("servlet").host("0.0.0.0");
The route sets the response body as an PipedInputStream and then (in a
separate thread) writes to that InputStream and eventually closes it.
The expectation is that as the response size is unknown the data will be
chunked and response returned to the client immediately and the client
will start to consume the data immediately.
When using curl as the client it seems to be working correctly, and I
see a 'Transfer-Encoding: chunked' header in the response.
However my real client is Apache HttpAsyncClient and with that what I'm
seeing is that the response is not returned until all the data is
written, and instead of a 'Transfer-Encoding: chunked' header I see a
'Content-Length: 19196336' header suggesting that Camel has waited for
the entire content to be written so that it can set the Content-Length.
I tried manually setting the 'Transfer-Encoding: chunked' in the
response message but it did not appear and instead I saw the
Content-Length header.
I've got HttpAsyncClient working in a test example to prove that it can
received the response asynchronously, so I think (but can't be sure)
that the problem is on the server (Camel) side.
What can explain this different behaviour from the same service when
using different clients?
Re: HTTP response is not being chunked
Hi Tim!
Sorry about the delay. I was summoned into a meeting that took a while.
Plus my PC is in the process of upgrading from Debian 9 to Debian
10/testing.
On the important matter.
There are certainly differences when it comes to headers. But I suspect the
culprit is the accept-encoding. I don't think it's the Camel headers.
I believe because it has to compress the content, it has to read everything
beforehand. Try adding "-H 'Accept-Encoding: gzip'" (double quotes
excluded) to your curl command and see if it changes the actual behaviour.
If it does, then you'll have to configure your client accordingly.
In any case, you can emulate the Async request in full by configuring the
headers with curl's "-H" parameter.
HTH
On Wed, 16 Jan 2019, 15:17 Tim Dudgeon Hi Luiz,
>
> here are the request headers. For Apache HTTPClient:
>
>accept -> chemical/x-mdl-sdfile
>accept-encoding -> gzip
>breadcrumbId -> ID-6667e3dab68a-38877-1547658276076-0-15
>CamelHttpMethod -> POST
>CamelHttpPath ->
>CamelHttpQuery -> null
>CamelHttpServletRequest ->
> org.apache.catalina.connector.RequestFacade@7551baa3
>CamelHttpServletResponse ->
> org.apache.catalina.connector.ResponseFacade@60e05546
>CamelHttpUri ->
> /chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
>CamelHttpUrl ->
>
> http://localhost:8092/chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
>CamelServletContextPath -> /v1/converters/dataset_to_sdf
>connection -> Keep-Alive
>content-encoding -> gzip
>Content-Type -> application/x-squonk-dataset-molecule+json
>host -> localhost:8092
>transfer-encoding -> chunked
>user-agent -> Apache-HttpAsyncClient/4.1.4 (Java/1.8.0_191)
>
> For curl:
>
>accept -> chemical/x-mdl-sdfile
>breadcrumbId -> ID-09b7601451a2-34027-1547658868812-0-15
>CamelHttpMethod -> POST
>CamelHttpPath ->
>CamelHttpQuery -> null
>CamelHttpServletRequest ->
> org.apache.catalina.connector.RequestFacade@3af9ee39
>CamelHttpServletResponse ->
> org.apache.catalina.connector.ResponseFacade@1f33d20b
>CamelHttpUri ->
> /chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
>CamelHttpUrl ->
>
> http://localhost:8092/chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
>CamelServletContextPath -> /v1/converters/dataset_to_sdf
>content-encoding -> gzip
>content-length -> 19397182
>Content-Type -> application/x-squonk-dataset-molecule+json
>expect -> 100-continue
>host -> localhost:8092
>user-agent -> curl/7.55.1
>
>
> On 16/01/2019 16:55, Luiz Eduardo Valmont wrote:
> > Hi, Tim!
> >
> > I'd suggest trying to intercept / dump the HttpAsyncClient request as
> well
> > as the curl one. A simple "nc -l -p " in an *NIX OS will do.
> >
> > My guess is that the perceived difference lies in the request headers.
> > Might be in the Accept.
> >
> > Can you post both requests in full, body excluded?
> >
> > Thanks!
> >
> > On Wed, 16 Jan 2019, 14:46 Tim Dudgeon >
> >> I'm hitting a strange problem with a route that is using the servlet
> >> component to send a large stream of data in its response.
> >> This is part of a route that uses the REST DSL which is set up like
> this:
> >>
> >> restConfiguration().component("servlet").host("0.0.0.0");
> >>
> >> The route sets the response body as an PipedInputStream and then (in a
> >> separate thread) writes to that InputStream and eventually closes it.
> >> The expectation is that as the response size is unknown the data will be
> >> chunked and response returned to the client immediately and the client
> >> will start to consume the data immediately.
> >>
> >> When using curl as the client it seems to be working correctly, and I
> >> see a 'Transfer-Encoding: chunked' header in the response.
> >>
> >> However my real client is Apache HttpAsyncClient and with that what I'm
> >> seeing is that the response is not returned until all the data is
> >> written, and instead of a 'Transfer-Encoding: chunked' header I see a
> >> 'Content-Length: 19196336' header suggesting that Camel has waited for
> >> the entire content to be written so that it can set the Content-Length.
> >> I tried manually setting the 'Transfer-Encoding: chunked' in the
> >> response message but it did not appear and instead I saw the
> >> Content-Length header.
> >> I've got HttpAsyncClient working in a test example to prove that it can
> >> received the response asynchronously, so I think (but can't be sure)
> >> that the problem is on the server (Camel) side.
> >>
> >> What can explain this different behaviour from the same service when
> >> using different clients?
> >>
> >>
>
Re: HTTP response is not being chunked
Hi Luiz,
here are the request headers. For Apache HTTPClient:
accept -> chemical/x-mdl-sdfile
accept-encoding -> gzip
breadcrumbId -> ID-6667e3dab68a-38877-1547658276076-0-15
CamelHttpMethod -> POST
CamelHttpPath ->
CamelHttpQuery -> null
CamelHttpServletRequest ->
org.apache.catalina.connector.RequestFacade@7551baa3
CamelHttpServletResponse ->
org.apache.catalina.connector.ResponseFacade@60e05546
CamelHttpUri ->
/chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
CamelHttpUrl ->
http://localhost:8092/chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
CamelServletContextPath -> /v1/converters/dataset_to_sdf
connection -> Keep-Alive
content-encoding -> gzip
Content-Type -> application/x-squonk-dataset-molecule+json
host -> localhost:8092
transfer-encoding -> chunked
user-agent -> Apache-HttpAsyncClient/4.1.4 (Java/1.8.0_191)
For curl:
accept -> chemical/x-mdl-sdfile
breadcrumbId -> ID-09b7601451a2-34027-1547658868812-0-15
CamelHttpMethod -> POST
CamelHttpPath ->
CamelHttpQuery -> null
CamelHttpServletRequest ->
org.apache.catalina.connector.RequestFacade@3af9ee39
CamelHttpServletResponse ->
org.apache.catalina.connector.ResponseFacade@1f33d20b
CamelHttpUri ->
/chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
CamelHttpUrl ->
http://localhost:8092/chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
CamelServletContextPath -> /v1/converters/dataset_to_sdf
content-encoding -> gzip
content-length -> 19397182
Content-Type -> application/x-squonk-dataset-molecule+json
expect -> 100-continue
host -> localhost:8092
user-agent -> curl/7.55.1
On 16/01/2019 16:55, Luiz Eduardo Valmont wrote:
Hi, Tim!
I'd suggest trying to intercept / dump the HttpAsyncClient request as well
as the curl one. A simple "nc -l -p " in an *NIX OS will do.
My guess is that the perceived difference lies in the request headers.
Might be in the Accept.
Can you post both requests in full, body excluded?
Thanks!
On Wed, 16 Jan 2019, 14:46 Tim Dudgeon
I'm hitting a strange problem with a route that is using the servlet
component to send a large stream of data in its response.
This is part of a route that uses the REST DSL which is set up like this:
restConfiguration().component("servlet").host("0.0.0.0");
The route sets the response body as an PipedInputStream and then (in a
separate thread) writes to that InputStream and eventually closes it.
The expectation is that as the response size is unknown the data will be
chunked and response returned to the client immediately and the client
will start to consume the data immediately.
When using curl as the client it seems to be working correctly, and I
see a 'Transfer-Encoding: chunked' header in the response.
However my real client is Apache HttpAsyncClient and with that what I'm
seeing is that the response is not returned until all the data is
written, and instead of a 'Transfer-Encoding: chunked' header I see a
'Content-Length: 19196336' header suggesting that Camel has waited for
the entire content to be written so that it can set the Content-Length.
I tried manually setting the 'Transfer-Encoding: chunked' in the
response message but it did not appear and instead I saw the
Content-Length header.
I've got HttpAsyncClient working in a test example to prove that it can
received the response asynchronously, so I think (but can't be sure)
that the problem is on the server (Camel) side.
What can explain this different behaviour from the same service when
using different clients?
Re: HTTP response is not being chunked
Hi, Tim!
I'd suggest trying to intercept / dump the HttpAsyncClient request as well
as the curl one. A simple "nc -l -p " in an *NIX OS will do.
My guess is that the perceived difference lies in the request headers.
Might be in the Accept.
Can you post both requests in full, body excluded?
Thanks!
On Wed, 16 Jan 2019, 14:46 Tim Dudgeon I'm hitting a strange problem with a route that is using the servlet
> component to send a large stream of data in its response.
> This is part of a route that uses the REST DSL which is set up like this:
>
> restConfiguration().component("servlet").host("0.0.0.0");
>
> The route sets the response body as an PipedInputStream and then (in a
> separate thread) writes to that InputStream and eventually closes it.
> The expectation is that as the response size is unknown the data will be
> chunked and response returned to the client immediately and the client
> will start to consume the data immediately.
>
> When using curl as the client it seems to be working correctly, and I
> see a 'Transfer-Encoding: chunked' header in the response.
>
> However my real client is Apache HttpAsyncClient and with that what I'm
> seeing is that the response is not returned until all the data is
> written, and instead of a 'Transfer-Encoding: chunked' header I see a
> 'Content-Length: 19196336' header suggesting that Camel has waited for
> the entire content to be written so that it can set the Content-Length.
> I tried manually setting the 'Transfer-Encoding: chunked' in the
> response message but it did not appear and instead I saw the
> Content-Length header.
> I've got HttpAsyncClient working in a test example to prove that it can
> received the response asynchronously, so I think (but can't be sure)
> that the problem is on the server (Camel) side.
>
> What can explain this different behaviour from the same service when
> using different clients?
>
>
HTTP response is not being chunked
I'm hitting a strange problem with a route that is using the servlet
component to send a large stream of data in its response.
This is part of a route that uses the REST DSL which is set up like this:
restConfiguration().component("servlet").host("0.0.0.0");
The route sets the response body as an PipedInputStream and then (in a
separate thread) writes to that InputStream and eventually closes it.
The expectation is that as the response size is unknown the data will be
chunked and response returned to the client immediately and the client
will start to consume the data immediately.
When using curl as the client it seems to be working correctly, and I
see a 'Transfer-Encoding: chunked' header in the response.
However my real client is Apache HttpAsyncClient and with that what I'm
seeing is that the response is not returned until all the data is
written, and instead of a 'Transfer-Encoding: chunked' header I see a
'Content-Length: 19196336' header suggesting that Camel has waited for
the entire content to be written so that it can set the Content-Length.
I tried manually setting the 'Transfer-Encoding: chunked' in the
response message but it did not appear and instead I saw the
Content-Length header.
I've got HttpAsyncClient working in a test example to prove that it can
received the response asynchronously, so I think (but can't be sure)
that the problem is on the server (Camel) side.
What can explain this different behaviour from the same service when
using different clients?
Re: Duplicate Messages (Multiple Consumers?)
Hi, Andrea! Thanks for the quick reply! I was looking at the disruptor, but the master component seems much more suited to the task at hand. For starters, it needs next to no configuration at all. It looks like a case of just prefixing the routes with "master:[namespace]". Awesome. If the current architecture of multiple routes present a problem, then it'll be just a case of simplifying the routes and refactoring certain activities into beans. But I believe that won't be necessary. Have a good one! =) On 16/01/2019 13:12, Andrea Cosentino wrote: You can take a look at the master component https://github.com/apache/camel/blob/master/components/camel-master/src/main/docs/master-component.adoc -- Andrea Cosentino -- Apache Camel PMC Chair Apache Karaf Committer Apache Servicemix PMC Member Email: ancosen1...@yahoo.com Twitter: @oscerd2 Github: oscerd On Wednesday, January 16, 2019, 4:08:41 PM GMT+1, Luiz Eduardo Guida Valmont wrote: Hi! Question: is it possible to configure a route so that at any given time there is at most one consumer? I thought of using idempotentConsumers. The expression for the message id would be any constant, really. This way, when another consumer tries to process the exchange, it'd be discarded. http://people.apache.org/~dkulp/camel/idempotent-consumer.html But this seems like bending the rules more than what I'd like to. Hence the question. If there's a way to do a semaphore like step, that'd be lovely. Thanks in advance! Luiz
Re: Duplicate Messages (Multiple Consumers?)
You can take a look at the master component https://github.com/apache/camel/blob/master/components/camel-master/src/main/docs/master-component.adoc -- Andrea Cosentino -- Apache Camel PMC Chair Apache Karaf Committer Apache Servicemix PMC Member Email: ancosen1...@yahoo.com Twitter: @oscerd2 Github: oscerd On Wednesday, January 16, 2019, 4:08:41 PM GMT+1, Luiz Eduardo Guida Valmont wrote: Hi! Question: is it possible to configure a route so that at any given time there is at most one consumer? I thought of using idempotentConsumers. The expression for the message id would be any constant, really. This way, when another consumer tries to process the exchange, it'd be discarded. http://people.apache.org/~dkulp/camel/idempotent-consumer.html But this seems like bending the rules more than what I'd like to. Hence the question. If there's a way to do a semaphore like step, that'd be lovely. Thanks in advance! Luiz
Duplicate Messages (Multiple Consumers?)
Hi! Question: is it possible to configure a route so that at any given time there is at most one consumer? I thought of using idempotentConsumers. The expression for the message id would be any constant, really. This way, when another consumer tries to process the exchange, it'd be discarded. http://people.apache.org/~dkulp/camel/idempotent-consumer.html But this seems like bending the rules more than what I'd like to. Hence the question. If there's a way to do a semaphore like step, that'd be lovely. Thanks in advance! Luiz
