Ignoring incomplete HTTP headers
I was wondering whether it was possible to configure Restlet to ignore incomplete HTTP headers when performing a request on a remote URI. While using Restlet to crawl some sites, I've come across a fair amount of URIs that declared an empty Pragma header (Pragma:). This causes the query to fail, while it could simply be logged as a warning and ignored. Is there any way to configure this behaviour, or is it impossible for Restlet to access remote resources who don't adhere to the HTTP specifications to the letter ? Nicolas -- http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2651541
Re: Wrong @Get method is called after negotiation
I've had the same issue. The only workaround I've found is a bit cumbersome but
works:
@Get(json|html)
public Representation getRepresentation(Variant variant) {
if(variant.getMediaType().equals(MediaType.APPLICATION_JSON)
}
I'm not sure whether that's a regression or the intended behaviour...
Nicolas
On 24 Aug 2010, at 08:41, webp...@tigris.org wrote:
In my ServerResource impl (on official restlet 2.0.0 GAE edition) I've got
two annotated methods:
@Get(json)
public Representation getJson() { ...
and
@Get(html)
public Representation getHtml() { ...
When testing the resource with:
curl -v -H 'Accept: application/json'
the result is that getHtml is called returning text/plain
So it seems the annotaion/negotiation isn't working... what could be causing
this?
--
http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2650559
--
http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2651853
Re: Maximum number of concurrent connections
Hello Thierry, Thanks for your reply. Just to make sure I understand properly though, this only applies to the internal connector, not others such as the java.net or Apache httpclient ? I unfortunately can't use the internal connector for our project: we require an HTTPS connection which, according to the documentation, isn't supported by the internal connector. It's not a big deal if the feature isn't supported across all connectors, it'd just be nice to know for sure so that I can at least mark our internal ticket as impossible for the moment and move on to other things :) Cheers, Nicolas On 20 Aug 2010, at 17:42, Thierry Boileau wrote: Hello Nicolas, I'm sorry for my late reply. The internal connector allows to set a maxTotalConnections parameter (described in the javadocs of the BaseHelper class, which is published here: [1]). I've also updated the page dedicated to the connectors [2]. Note that this apply to both client and server internal connectors. Best regards, Thierry boileau [1] http://www.restlet.org/documentation/2.0/jse/engine/org/restlet/engine/http/connector/BaseHelper.html [2] http://wiki.restlet.org/docs_2.0/13-restlet/37-restlet.html -- http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2649294 -- http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2650208
Client configuration
I'm having issues configuring or understanding how to configure client connectors. What I'm trying to do is: - disable persistant connections altogether. - set a specific timeout policy. Note: this is for the Apache httpclient connector, I've not tried with others. Connection Persistence: The only documentation I could find was: http://wiki.restlet.org/docs_2.0/13-restlet/27-restlet/324-restlet/130-restlet.html According to this, it appears that setting the context property clientKeepAlive to a specific value will disable connection persistence. The necessary value isn't clear, so I've tried close, off, no and false. Whatever the configuration, the client's HTTP headers always contain a Connection: Keep-Alive entry. Is there a way to disable connection persistence, at least for the Apache httpclient connector? If so, I'd be grateful for any help, I've struggled with this for a while and am no further than when I started. Timeout handling This is a documentation issue more than an implementation one. I'd like to set a timeout for both socket opening and content retrieval - a query shouldn't take more than x seconds to be established, nor should it take more than y seconds to complete. Note that in my specific case, setting x and y to the same value is perfectly acceptable. Looking at the documentation (http://www.restlet.org/documentation/2.0/jse/ext/org/restlet/ext/httpclient/HttpClientHelper.html), I'm a bit confused. - socketTimeout is probably the variable I want to set for connection establishment. - I'm unsure what idleTimeout does. The documentation could describe both persistent connections timeout (if the connection hasn't been used for x ms, close it) or stalled connections. Either way, it doesn't look like what I'm looking for. There are no other timeout related configuration variables that I can see in the documentation. Am I missing or misunderstanding something? Thanks ! Nicolas -- http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2645568
Maximum number of concurrent connections
Is there a way to set the maximum number of concurrent connections on the server side ? I've been trying to find a server connector property, but there doesn't seem to be any - or I might be misunderstanding the documentation. Thanks ! Nicolas -- http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2645577
RE: Re: SSL modifications since 2.0 RC 1 ?
Hi Bruno, You certainly put a lot of effort into that ! I have to admit that some of it went over my head - you obviously are a bit of an expert on the matter, which I'm not. If I understand you properly, the problem isn't Restlet, Safari, or even Java specific - it's a generic flaw (?) in client certificates management ? Which means that in my very specific, very selfish case, I should disable client certificate requests to make the problem go away. Which also means getting rid of the Simple connector, since this is hardcoded and can't be modified. Is that a fair summary, or did I misunderstand you even more badly than I thought ? Nicolas -- http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2601883
RE: Re: SSL modifications since 2.0 RC 1 ?
Hey Bruno, Cheers, that clarified lots of SSL things for me and allowed me to fix the RC 3 client as you suggested. I'm still a bit confused about OS X Safari refusing connections, though - and wondering whether the same issue affects properly signed certificates (as opposed to self-signed ones). Do you have any idea what could cause this ? Is this something I should file a bug report for - after all, while Safari doesn't have that huge a market share, the fact that Restlet can't seem to serve HTTPS requests to it seems rather critical to me... Thanks again ! Nicolas -- http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2597872
RE: Re: SSL modifications since 2.0 RC 1 ?
This might be an important bit of information. After running a few more tests, I realised that Safari would: - fail ignoring certificates if the server is running on OS X. - succeed in ignoring certificates if the server is running on Debian Lenny (5.0) This might be an OS X only issue after all. Nicolas -- http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2597997
RE: Re: SSL modifications since 2.0 RC 1 ?
First, thanks for taking the time to look into this, I really appreciate it.
Now, to answer your post.
2. By default, the SSLContext (on the server connectors) is created by
the DefaultSslContextFactory, which follows the defaults of the JSSE Ref
guide by default. It will use javax.net.ssl.keyStore,
javax.net.ssl.trustStore and related properties if specified, otherwise,
the keystore is unspecified (you'd always have to configure it anyway)
and the truststore is based on the CA certificates bundled with the JRE.
I'm not sure I follow. Do you mean that in order to configure my server side
keystore, I need to use the corresponding system properties ?
What I'm doing is initialising the connector with parameters initialised as
follows:
parameters = new VectorParameter();
parameters.add(new Parameter(keystorePath, keystorePath));
parameters.add(new Parameter(keystorePassword, keystorePassword));
parameters.add(new Parameter(keyPassword, keyPassword));
Has that changed ?
In addition, the Simple connector is set up to ask for an optional
client certificate (that's hard-coded in Simple). If the client sends
one it doesn't like, this would cause the connection to break.
Right, that might explain things.
On the client side, I'm using the internal connector without any specific
configuration, except for the following code executed in our test cases:
SSLContext sc = SSLContext.getInstance(SSL);
sc.init(null, new TrustManager[] {new X509TrustManager() {
public X509Certificate[] getAcceptedIssuers() {return null;}
public void checkServerTrusted(X509Certificate[] certs, String authType)
throws CertificateException {}
public void checkClientTrusted(X509Certificate[] certs, String authType)
throws CertificateException {}
}}, new SecureRandom());
HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
HttpsURLConnection.setDefaultHostnameVerifier(new HostnameVerifier() {
public boolean verify(String urlHostName, SSLSession session) {return true;}
});
I don't really see how that could result in the client sending an odd
certificate, but I'm a bit out of my depths here and I might very well have
overlooked something.
They're not, but not all connectors support them in the same way
unfortunately (in some you can enable, in some you can disable).
Ok, I guess that Simple doesn't support this then. That would explain why I
can't change my cipher suites.
Thanks !
Nicolas
--
http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2596684
RE: Re: SSL modifications since 2.0 RC 1 ?
Could you clarify what you mean by accept/ignore? I'm not sure what this means in the context of Firefox/Safari. Right, sorry, that wasn't very clear. Accept: the browser will load the page without complaint. Refuse: the browser will fail to load the page with an error message such as Safari can’t establish a secure connection to the server 'localhost'. Ignore: instructing the browser to ignore the fact that the certificate is self-signed - adding an exception in Firefox, for example. All the tests I've described before are on OS X. are you using client-certificates? Could you explain what configuration you use for the client? (In particular w.r.t the trust store?) I'm not using client-certificates, no. To be honest, I've no idea what a trust store is. The only thing I can think of that might be related to that is the code we use in our tests to disable SSL certificate validation: - set the default host name verifier to an instance that will accept anything. - set the trust manager to a pass-through implementation. I'd be happy to post the code if you need me to. for example, you would have to put your server's self-signed cert into the client's truststore for it to be trusted by the client). This confuses me a bit - I thought that's exactly what I did when I told Safari to add the certificate to the list of trusted certificates. Once I've done that, however, Safari still refuses to load pages. I'm not sure where the change in cipher suites comes from, but it shouldn't affect how the certificates are trusted. What confuses me is that even using the enabledCipherSuites and disabledCipherSuites, I can't change these. Are these instructions ignored in the new version ? Nicolas -- http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2595972
RE: Re: Issues with keepAlive
Hi Thierry, Ok, I've worked things out. I feel a bit silly now. The problem wasn't using a new client for each new request, but the fact that I wasn't emptying and releasing entity streams - since I wasn't releasing the connection, it couldn't be used by further requests. Problem solved ! Nicolas -- http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2448388
RE: Re: Issues with keepAlive
Hi Thierry, I'm sorry, I should have mentioned: the queries I perform are *not* in parallel. I basically do System.setProperty(http.maxConnections, 1); for(int i = 0; i 2000; i++) new Client(new Context(), Protocol.HTTP).handle(new Request(Method.GET, http://localhost:8182;)); I ran this test with both the default and Simple client connectors and observe up to 200 open sockets at the same time. On the other hand, running the same test using standard URLConnection instances only opens 1 socket. Let me know if there's anything else I can do to help you track this bug down. Cheers, Nicolas -- http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2448028
Issues with keepAlive
I've been trying to get persistent connections to work for my application running on Restlet 2.0m7, using the Simple server connector and the default client one. To make sure that it behaved properly, I wrote a quick test that sends thousands of queries to the server and counted the number of open sockets using lsof - if my understanding of persistent connections is correct, I should see at most 4 or 5 open sockets, but I'm observing hundreds of them. Mind you, I'm seeing less open sockets than I'm sending queries, but I can't decide whether it's because connections are re-used or because they get closed eventually. I've tried setting the http.maxConnections to 1 on the client side, but that doesn't change anything. I'd file that as a bug, but I'm not convinced I understand persistent connections well enough - maybe this is the normal behavior and I'm missing something. Can anyone enlighten me on the matter ? Nicolas -- http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2447687
Cancelling request entity
Hey, If my understanding is correct, Request.abort() allows server resources to not read a request entity that they know won't be accepted - unacceptable MIME type, too large... I have however been incapable of getting this to work. Attached is a sample case, which should be compiled and run against Restlet 2.0m7. What it does is: - create a simple server resource that only accepts POST and immediately calls getRequest().abort(); - submit a counting input stream to that resource: the stream will count the number of bytes that it returns. - print the number of bytes that the server resource has read. This number is always the size of the submitted stream, which seems to defeat the purpose of Request.abort(). I might very well have misunderstood the purpose of the abort() method, or be doing something wrong. Can anyone point me in the right direction ? For reference, I'm running these tests: - with Restlet 2.0m7. - using org.restlet.ext.simple for the server connector. Oh, and as a side note, Restlet 2.0m7 isn't yet available in the issue tracker's versions. Cheers, Nicolas Test.java Description: Binary data
RE: Restlet 2.0M4 on Mac OS X
Hey Paul, I had the exact same problem and submitted a bug report this weekend (http://restlet.tigris.org/issues/show_bug.cgi?id=868). To work around it, simply make sure that your default encoding is UTF-8 rather than macroman. You can achieve that by passing -Dfile.encoding=UTF-8 to your VM. I guess you could also programaticaly set the file.encoding System property, but you need to make sure this is done before the MetadataService is started. Nicolas -- http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2382375
RESTlet 2.0 M4 issues
I'm not submitting this in the issue tracker as I failed to gather enough information for a proper bug report - pretty much all I can say is that things that used to work flawlessly are now broken. The application we're working on has fairly extensive client side unit tests. Running them will generate thousands of queries to our RESTful service. This used to not be a problem with RESTlet 1.1.5, where aside from a few issue we needed to work around here and there, everything was fine. I've recently upgraded to RESTlet 2.0 M4 to give it a go, and I've found that we now have huge performance issue. As a disclaimer, it must be noted that I've just upgraded to RESTlet 2.0, but haven't ported anything - I'm still using the old, deprecated API. This might be the reason for all my woes. By default, we're using the Simple web server. Running our application on this, we very quickly start seeing OutOfMemoryExceptions on the server side. Additionally, I'm seeing the following CPU hotspots: - 12.1% : org.simpleframework.transport.reactor.ExecuteAction.init - 11.3% : org.simpleframework.http.core.Reader.init - 17.6% : org.simpleframework.transport.reactor.PartitionDistributor.process This might not look so bad if not for the fact that: - I'm not running anything else. - both of my computer's cores are on a average load of 3.28. - this is *after* I killed the unit tests. The server should be idle, it's not being queried at all. Looking at these hotspots, I figured it'd be an issue with the Simple webserver and swapped it for Jetty. Performances are much better, for a few seconds, but then I start seeing the following stacktrace server side: 2009-08-10 10:31:03,778] DEBUG org.mortbay.log EXCEPTION java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293) at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746) at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75) at org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:382) at org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:107) at org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:198) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:290) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:636) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520) Once this has happened, the server will often drop queries and take ages to answer to the ones it does reply to. I might very well be doing something wrong, and maybe the fact that I'm using the old 1.1.5 deprecated APIs is to blame, but I thought I'd mention this on the off-chance that it was something worst. These issues are preventing me from working on our application, so I'll temporarily switch back to 1.1.5. On the other hand, I'm keeping the 2.0 M4 project handy, so if you need me to run any tests to identify a possible issue, just let me know. Regards, Nicolas -- http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2381998
RE: problem returning a response status
Hi there, Disclaimer: you're running RESTlet 2.0, with which I have very little familiarity. With RESTlet 1.1, the way to specify a message within a status is to create a new instance of Status. try: setStatus(new Status(Status.SERVER_ERROR_INTERNAL, e.getMessage())); Nicolas -- http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2381308
RE: Re: JAX RS exceptions with illegal QueryParameter
From your answer, I think I might be misunderstanding the basic concept of a resource. The way I understand it is, if you have a /reports resource that lists all available reports and accepts a count parameter that limits the number of reports returned in a given query, /reports?count=10 and /reports?count=20 are the same resource. With that in mind, /reports?count=foobar isn't a 404 (the reports resource was found) but a 400 (query parameter 'count' has an illegal value). What you're saying, however, is that /reports?count=10 and /reports?count=20 are two different resources? This isn't what I understood at all, but it probably means I didn't understand it correctly. If that's the case, I apologise for wasting your time and will be doing some more reading on what a resource actually is. Regards, Nicolas -- http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2380725
RE: Re: JAX RS exceptions with illegal QueryParameter
Understood, and after thinking about it, I do agree with your definition. On the other hand (and this is purely academic), I do find that returning 404 if a query parameter is malformed is not helpful to the client: it's not incorrect, since the resource wasn't found, but answering 400 tells him the reason why the resource couldn't be found - because the server couldn't understand his query and he should try and fix it. Both seem correct to me, and I find that 400 gives more information to clients. Bit of a moot point though, since there appears to not be any way for me to do that using JAX RS :) Nicolas -- http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2380801
RE: Re: JAX RS exceptions with illegal QueryParameter
Yes, you're absolutely right, that's in the JAX RS specifications. I was just wondering whether RESTlet had an extension to the basic specifications that allowed more control over what error code was used (an @Error(400) annotation, for example). The different for the user, if we're talking about a web user, is slim at best. An error occurred but that's not really his problem, the UI should have sheltered him from such basic syntax errors. For the developer writing a client application, however, it can save a lot of time. Knowing precisely why your implementation fails is always preferable in my opinion than just having a generic failed error. I'm oversimplifying things here, but that's the difference between throwing a NumberFormatException and a generic Exception. Apologies if I'm taking the debate too far, I'm aware that RESTlet is following the specifications to the letter and am not complaining about that at all. To be honest, I should have asked that question to the JAX RS design group but couldn't find any public forum and thought that since Jérome Louvel sits at that comity and is working on the implementation of the specifications that I'm using, this would be the next best place to voice my questions... Regards, Nicolas -- http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2381083
RE: Re: JAX RS exceptions with illegal QueryParameter
I haven't tested it myself, but I'm not sure that ExceptionMappers actually kick-in for exceptions triggered by the underlying Engine. Regardless of that consideration though, the WebApplicationException thrown in such cases is just a generic one with a 404 status code. Even if I could write an ExceptionMapper to catch it, there'd be no way for me to differentiate it from other 404s. Well, let me rephrase that. I don't see how I'd go about differentiating it, but I'd love to hear your thoughts on the subject :) Regards, Nicola -- http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2381106
RE: Re: JAX RS exceptions with illegal QueryParameter
To be honest, I'm only using JAX RS on a side project to evaluate the technology. I'm loving it so far, once you get the hang of it it makes your code extremely clean and easy to maintain, provided you don't mind having more annotations than actual Java code. I sometimes feel a bit trapped by it though - like a lot of specifications, it makes a lot of decisions for you and I don't always agree with them. I haven't tested RESTlet 2.0 yet, but it sounds like a good mix of both, where the general case is handled well but you still get a chance to tweak things and do them your way when necessary. I guess that's probably the answer to my technical question :) Nicolas -- http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2381109
RE: Re: JAX RS ExceptionMappers by MessageBodyWriter
Actually, come to think of it, what I'm talking about *should* be possible. I just ran a test and if I throw any sort of exception in a writeTo method, the status code received by the client is a generic 500. RESTlet somehow manages to catch the error and deal with it (maybe it caches headers until the first byte of the response has been written?). Since RESTlet appears to be capable of dealing with such errors, should it not also try to find a relevant ExceptionMapper before triggering a generic 500 error? I apologise if I'm talking rubbish here, maybe I'm missing something obvious. Nicolas -- http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2379510
RE: Re: JAX RS ExceptionMappers by MessageBodyWriter
Ah, yes, very good point. I feel a bit silly now :) That's a bit of a pain, since it means that you have to manually check for the requested charset in each method that returns a Representation though. I liked the idea of just using the one specified in the request's media type and catching the possible UnsupportedEncodingException in an ExceptionMapper, but I guess that's not possible. I'm not sure using the isWriteable method is a good idea for that particular problem - you don't have MIME type information there and just because the current instance doesn't support the requested charset doesn't mean no other does. -- http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2377552
JAX RS ExceptionMappers by MessageBodyWriter
Disclaimer: the behaviour I'm about to describe might be entirely corret and true to the JAX RS specifications, I just haven't been able to confirm it. I'm a big fan of the ExceptionMapper mechanism, but I discovered that they are only triggered when a resource method (annotated by @GET, @POST...) throws an exception. Say that you have a MessageBodyWriter implementation that checks its MediaType argument to try and respect client charset wishes - if a client asks for us-ascii and I can support it, then I might as well rather than use the default utf-8. If the client were to request an unknown encoding (say, zorglub), an UnsupportedCharsetException will be triggered. This exception will not be caught by any ExceptionMapper I might have registered. If it *were* caught, it would allow application developers to not worry about exceptions at all: just throw whatever exception you might need to throw and rely and the mappers to catch them and return the correct HTTP status code to the client. In my example, I know that an UnsupportedCharsetException should always be mapped to UNSUPPORTED_MEDIA_TYPE, and I don't want to duplicate this mapping in all of my MessageBodyWriters or MessageBodyReaders. Is the fact that this doesn't happen a RESTlet issue, or a specification issue (decision I don't agree with but might not understand the reason for) ? Nicolas -- http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447dsMessageId=2377287
