I am attaching new patch.
On 10/19/2016 07:13 PM, Alex Rousskov wrote:
On 10/19/2016 08:49 AM, Christos Tsantilas wrote:
I am attaching a new patch.
I would like to discuss two issues:
* Logging of scheme-less URLs
This is defines a new proto the PROTO_TCP, and for this prints the url
in the form host:port.
The PROTO_TCP name sounds bad because we may want to log
tcp://host:port/ URLs in the future as discussed earlier.
PROTO_NONE and PROTO_UNKNOWN are already utilized for different _valid_
use cases. I suggest adding PROTO_AUTHORITY or PROTO_AUTHORITY_FORM
instead of adding PROTO_TCP.
I used the PROTO_AUTHORITY_FORM.
* flags.tunneling
removed now.
AFAIK, you added flags.tunneling specifically to work around the
tunneling request removal from the ConnStateData pipeline (trunk commit
r14418 "Cleanup: Refactor ConnStateData pipeline handling"). The
justification for that removal is given in the corresponding commit
message (thank you, Amos!):
Initial testing revealed CONNECT tunnels always being logged as ABORTED.
[..] I have now made the context be finished() just prior to the
TunnelStateData being destroyed. That way normal closure should show up
only as TUNNEL, but timeouts and I/O errors should still be recorded as
abnormal.
That explanation makes sense to me -- when we know that the request has
finished successfully, we should call its finished() method.
AFAICT, the flags.tunneling is currently needed specifically because the
following ConnStateData::checkLogging() condition is [sometimes] false
when the pipeline is empty and we are done tunneling:
// do not log connections that closed after a transaction (it is normal)
if (receivedFirstByte_ && inBuf.isEmpty())
return;
Instead of adding flags.tunneling, can we fix/adjust that condition
and/or the code that affects its components to make the condition true?
receivedFirstByte_: This member remains false if we splice and tunnel
intercepted connections during step1 because we read no bytes at that
time, right? However, it feels like this part of the condition should be
not receivedFirstByte_ but something like thereWasAtLeastOne (i.e., we
pushed at least one request into ConnStateData pipeline). Can we adjust
the pipeline code to remember that at least one request has been queued
and use that instead of receivedFirstByte_ here.
inBuf.isEmpty(): That part of the condition looks correct to me. If it
is false in the tunneling case being discussed, then we need to fix the
code to clean inBuf when tunneling starts.
In summary, can we do something like this instead:
// do not log connections that closed after a transaction (it is normal)
if (pipeline.used() && inBuf.isEmpty())
return;
I used the pipeline.nrequests member.
Still I have some worries about inBuf but looks that it is empty in all
cases we need it. In the worst case would not require a lot of work to
fix any related issue when this is found.
Thank you,
Alex.
const SBuf &
HttpRequest::effectiveRequestUri() const
{
- if (method.id() == Http::METHOD_CONNECT)
+ if (method.id() == Http::METHOD_CONNECT || url.getScheme() ==
AnyP::PROTO_TCP)
return url.authority(true); // host:port
return url.absolute();
}
Support tunneling of bumped non-HTTP traffic. Other SslBump fixes.
Use case: Skype groups appear to use TLS-encrypted MSNP protocol instead
of HTTPS. This change allows Squid admins using SslBump to tunnel Skype
groups and similar non-HTTP traffic bytes via "on_unsupported_protocol
tunnel all". Previously, the combination resulted in encrypted HTTP 400
(Bad Request) messages sent to the client (that does not speak HTTP).
Also this patch:
* fixes bug 4529: !EBIT_TEST(entry->flags, ENTRY_FWD_HDR_WAIT)
assertion in FwdState.cc.
* when splicing transparent connections during SslBump step1, avoid
access-logging an extra record and log %ssl::bump_mode as the expected
"splice" not "none".
* handles an XXX comment inside clientTunnelOnError for possible memory
leak of client streams related objects
* fixes TunnelStateData logging in the case of splicing after peek.
This is a Measurement Factory project.
=== modified file 'src/FwdState.cc'
--- src/FwdState.cc 2016-08-10 00:56:30 +0000
+++ src/FwdState.cc 2016-10-06 08:27:50 +0000
@@ -830,62 +830,59 @@
request->hier.startPeerClock();
// Do not fowrward bumped connections to parent proxy unless it is an
// origin server
if (serverDestinations[0]->getPeer() && !serverDestinations[0]->getPeer()->options.originserver && request->flags.sslBumped) {
debugs(50, 4, "fwdConnectStart: Ssl bumped connections through parent proxy are not allowed");
ErrorState *anErr = new ErrorState(ERR_CANNOT_FORWARD, Http::scServiceUnavailable, request);
fail(anErr);
self = NULL; // refcounted
return;
}
request->flags.pinned = false; // XXX: what if the ConnStateData set this to flag existing credentials?
// XXX: answer: the peer selection *should* catch it and give us only the pinned peer. so we reverse the =0 step below.
// XXX: also, logs will now lie if pinning is broken and leads to an error message.
if (serverDestinations[0]->peerType == PINNED) {
ConnStateData *pinned_connection = request->pinnedConnection();
debugs(17,7, "pinned peer connection: " << pinned_connection);
// pinned_connection may become nil after a pconn race
- if (pinned_connection) {
- serverConn = pinned_connection->borrowPinnedConnection(request, serverDestinations[0]->getPeer());
- if (Comm::IsConnOpen(serverConn)) {
- pinned_connection->stopPinnedConnectionMonitoring();
- flags.connected_okay = true;
- ++n_tries;
- request->flags.pinned = true;
- if (pinned_connection->pinnedAuth())
- request->flags.auth = true;
-
- closeHandler = comm_add_close_handler(serverConn->fd, fwdServerClosedWrapper, this);
-
- syncWithServerConn(pinned_connection->pinning.host);
-
- // the server may close the pinned connection before this request
- pconnRace = racePossible;
- dispatch();
- return;
- }
+ serverConn = pinned_connection ? pinned_connection->borrowPinnedConnection(request, serverDestinations[0]->getPeer()) : nullptr;
+ if (Comm::IsConnOpen(serverConn)) {
+ flags.connected_okay = true;
+ ++n_tries;
+ request->flags.pinned = true;
+
+ if (pinned_connection->pinnedAuth())
+ request->flags.auth = true;
+
+ closeHandler = comm_add_close_handler(serverConn->fd, fwdServerClosedWrapper, this);
+
+ syncWithServerConn(pinned_connection->pinning.host);
+
+ // the server may close the pinned connection before this request
+ pconnRace = racePossible;
+ dispatch();
+ return;
+ }
- } else
- serverConn = nullptr;
// Pinned connection failure.
debugs(17,2,HERE << "Pinned connection failed: " << pinned_connection);
ErrorState *anErr = new ErrorState(ERR_ZERO_SIZE_OBJECT, Http::scServiceUnavailable, request);
fail(anErr);
self = NULL; // refcounted
return;
}
// Use pconn to avoid opening a new connection.
const char *host = NULL;
if (!serverDestinations[0]->getPeer())
host = request->url.host();
Comm::ConnectionPointer temp;
// Avoid pconns after races so that the same client does not suffer twice.
// This does not increase the total number of connections because we just
// closed the connection that failed the race. And re-pinning assumes this.
if (pconnRace != raceHappened)
temp = pconnPop(serverDestinations[0], host);
=== modified file 'src/HttpRequest.cc'
--- src/HttpRequest.cc 2016-08-17 00:38:25 +0000
+++ src/HttpRequest.cc 2016-10-20 14:46:05 +0000
@@ -640,25 +640,25 @@
{
if (clientConnectionManager.valid() && clientConnectionManager->pinning.pinned)
return clientConnectionManager.get();
return NULL;
}
const SBuf
HttpRequest::storeId()
{
if (store_id.size() != 0) {
debugs(73, 3, "sent back store_id: " << store_id);
return StringToSBuf(store_id);
}
debugs(73, 3, "sent back effectiveRequestUrl: " << effectiveRequestUri());
return effectiveRequestUri();
}
const SBuf &
HttpRequest::effectiveRequestUri() const
{
- if (method.id() == Http::METHOD_CONNECT)
+ if (method.id() == Http::METHOD_CONNECT || url.getScheme() == AnyP::PROTO_AUTHORITY_FORM)
return url.authority(true); // host:port
return url.absolute();
}
=== modified file 'src/RequestFlags.h'
--- src/RequestFlags.h 2016-01-01 00:12:18 +0000
+++ src/RequestFlags.h 2016-10-04 07:46:28 +0000
@@ -99,38 +99,41 @@
bool chunkedReply :1;
/** set if stream error has occured */
bool streamError :1;
/** internal ssl-bump request to get server cert */
bool sslPeek :1;
/** set if X-Forwarded-For checking is complete
*
* do not read directly; use doneFollowXff for reading
*/
bool done_follow_x_forwarded_for :1;
/** set for ssl-bumped requests */
bool sslBumped :1;
/// carries a representation of an FTP command [received on ftp_port]
bool ftpNative :1;
bool destinationIpLookedUp:1;
/** request to reset the TCP stream */
bool resetTcp:1;
/** set if the request is ranged */
bool isRanged :1;
+ /// whether to forward via TunnelStateData (instead of FwdState)
+ bool forceTunnel :1;
+
/** clone the flags, resetting to default those which are not safe in
* a related (e.g. ICAP-adapted) request.
*/
RequestFlags cloneAdaptationImmune() const;
// if FOLLOW_X_FORWARDED_FOR is not set, we always return "done".
bool doneFollowXff() const {
return done_follow_x_forwarded_for || !FOLLOW_X_FORWARDED_FOR;
}
// if USE_HTTP_VIOLATIONS is not set, never allow this
bool noCacheHack() const {
return USE_HTTP_VIOLATIONS && nocacheHack;
}
};
#endif /* SQUID_REQUESTFLAGS_H_ */
=== modified file 'src/anyp/ProtocolType.h'
--- src/anyp/ProtocolType.h 2016-05-18 16:26:16 +0000
+++ src/anyp/ProtocolType.h 2016-10-20 14:31:27 +0000
@@ -21,40 +21,41 @@
*/
typedef enum {
PROTO_NONE = 0,
PROTO_HTTP,
PROTO_FTP,
PROTO_HTTPS,
PROTO_COAP,
PROTO_COAPS,
PROTO_GOPHER,
PROTO_WAIS,
PROTO_CACHE_OBJECT,
PROTO_ICP,
#if USE_HTCP
PROTO_HTCP,
#endif
PROTO_URN,
PROTO_WHOIS,
PROTO_ICY,
PROTO_TLS,
PROTO_SSL,
+ PROTO_AUTHORITY_FORM,
PROTO_UNKNOWN,
PROTO_MAX
} ProtocolType;
extern const char *ProtocolType_str[];
/** Display the registered Protocol Type (in upper case).
* If the protocol is not a registered AnyP::ProtocolType nothing will be displayed.
* The caller is responsible for any alternative text.
*/
inline std::ostream &
operator <<(std::ostream &os, ProtocolType const &p)
{
if (PROTO_NONE <= p && p < PROTO_MAX)
os << ProtocolType_str[p];
else
os << static_cast<int>(p);
return os;
}
=== modified file 'src/client_side.cc'
--- src/client_side.cc 2016-09-22 14:21:12 +0000
+++ src/client_side.cc 2016-10-20 14:31:46 +0000
@@ -1281,50 +1281,53 @@
/** Parse an HTTP request
*
* \note Sets result->flags.parsed_ok to 0 if failed to parse the request,
* to 1 if the request was correctly parsed.
* \param[in] csd a ConnStateData. The caller must make sure it is not null
* \param[in] hp an Http1::RequestParser
* \param[out] mehtod_p will be set as a side-effect of the parsing.
* Pointed-to value will be set to Http::METHOD_NONE in case of
* parsing failure
* \param[out] http_ver will be set as a side-effect of the parsing
* \return NULL on incomplete requests,
* a Http::Stream on success or failure.
*/
Http::Stream *
parseHttpRequest(ConnStateData *csd, const Http1::RequestParserPointer &hp)
{
/* Attempt to parse the first line; this will define where the method, url, version and header begin */
{
const bool parsedOk = hp->parse(csd->inBuf);
- if (csd->port->flags.isIntercepted() && Config.accessList.on_unsupported_protocol)
- csd->preservedClientData = csd->inBuf;
// sync the buffers after parsing.
csd->inBuf = hp->remaining();
if (hp->needsMoreData()) {
debugs(33, 5, "Incomplete request, waiting for end of request line");
return NULL;
}
+ if (csd->mayTunnelUnsupportedProto()) {
+ csd->preservedClientData = hp->parsed();
+ csd->preservedClientData.append(csd->inBuf);
+ }
+
if (!parsedOk) {
const bool tooBig =
hp->parseStatusCode == Http::scRequestHeaderFieldsTooLarge ||
hp->parseStatusCode == Http::scUriTooLong;
auto result = csd->abortRequestParsing(
tooBig ? "error:request-too-large" : "error:invalid-request");
// assume that remaining leftovers belong to this bad request
csd->consumeInput(csd->inBuf.length());
return result;
}
}
/* We know the whole request is in parser now */
debugs(11, 2, "HTTP Client " << csd->clientConnection);
debugs(11, 2, "HTTP Client REQUEST:\n---------\n" <<
hp->method() << " " << hp->requestUri() << " " << hp->messageProtocol() << "\n" <<
hp->mimeHeader() <<
"\n----------");
/* deny CONNECT via accelerated ports */
@@ -1547,77 +1550,62 @@
context->http->al->request = request;
HTTPMSGLOCK(context->http->al->request);
}
repContext->setReplyToError(request->method, err);
assert(context->http->out.offset == 0);
context->pullData();
return true;
}
}
}
return false;
}
#endif // USE_OPENSSL
/**
* Check on_unsupported_protocol checklist and return true if tunnel mode selected
* or false otherwise
*/
bool
-clientTunnelOnError(ConnStateData *conn, Http::Stream *context, HttpRequest *request, const HttpRequestMethod& method, err_type requestError, Http::StatusCode errStatusCode, const char *requestErrorBytes)
+clientTunnelOnError(ConnStateData *conn, Http::StreamPointer &context, HttpRequest::Pointer &request, const HttpRequestMethod& method, err_type requestError)
{
- if (conn->port->flags.isIntercepted() &&
- Config.accessList.on_unsupported_protocol && conn->pipeline.nrequests <= 1) {
- ACLFilledChecklist checklist(Config.accessList.on_unsupported_protocol, request, NULL);
+ if (conn->mayTunnelUnsupportedProto()) {
+ ACLFilledChecklist checklist(Config.accessList.on_unsupported_protocol, request.getRaw(), NULL);
checklist.requestErrorType = requestError;
checklist.src_addr = conn->clientConnection->remote;
checklist.my_addr = conn->clientConnection->local;
checklist.conn(conn);
allow_t answer = checklist.fastCheck();
if (answer == ACCESS_ALLOWED && answer.kind == 1) {
debugs(33, 3, "Request will be tunneled to server");
if (context) {
- // XXX: Either the context is finished() or it should stay queued.
- // The below may leak client streams BodyPipe objects. BUT, we need
- // to check if client-streams detatch is safe to do here (finished() will detatch).
assert(conn->pipeline.front() == context); // XXX: still assumes HTTP/1 semantics
- conn->pipeline.popMe(Http::StreamPointer(context));
+ context->finished(); // Will remove from conn->pipeline queue
}
Comm::SetSelect(conn->clientConnection->fd, COMM_SELECT_READ, NULL, NULL, 0);
- return conn->fakeAConnectRequest("unknown-protocol", conn->preservedClientData);
+ return conn->initiateTunneledRequest(request, Http::METHOD_NONE, "unknown-protocol", conn->preservedClientData);
} else {
debugs(33, 3, "Continue with returning the error: " << requestError);
}
}
- if (context) {
- conn->quitAfterError(request);
- clientStreamNode *node = context->getClientReplyContext();
- clientReplyContext *repContext = dynamic_cast<clientReplyContext *>(node->data.getRaw());
- assert (repContext);
-
- repContext->setReplyToError(requestError, errStatusCode, method, context->http->uri, conn->clientConnection->remote, NULL, requestErrorBytes, NULL);
-
- assert(context->http->out.offset == 0);
- context->pullData();
- } // else Probably an ERR_REQUEST_START_TIMEOUT error so just return.
return false;
}
void
clientProcessRequestFinished(ConnStateData *conn, const HttpRequest::Pointer &request)
{
/*
* DPW 2007-05-18
* Moved the TCP_RESET feature from clientReplyContext::sendMoreData
* to here because calling comm_reset_close() causes http to
* be freed before accessing.
*/
if (request != NULL && request->flags.resetTcp && Comm::IsConnOpen(conn->clientConnection)) {
debugs(33, 3, HERE << "Sending TCP RST on " << conn->clientConnection);
conn->flags.readMore = false;
comm_reset_close(conn->clientConnection);
}
}
void
@@ -2138,41 +2126,41 @@
*/
bool
ConnStateData::clientParseRequests()
{
bool parsed_req = false;
debugs(33, 5, HERE << clientConnection << ": attempting to parse");
// Loop while we have read bytes that are not needed for producing the body
// On errors, bodyPipe may become nil, but readMore will be cleared
while (!inBuf.isEmpty() && !bodyPipe && flags.readMore) {
/* Limit the number of concurrent requests */
if (concurrentRequestQueueFilled())
break;
// try to parse the PROXY protocol header magic bytes
if (needProxyProtocolHeader_ && !parseProxyProtocolHeader())
break;
- if (Http::Stream *context = parseOneRequest()) {
+ if (Http::StreamPointer context = parseOneRequest()) {
debugs(33, 5, clientConnection << ": done parsing a request");
AsyncCall::Pointer timeoutCall = commCbCall(5, 4, "clientLifetimeTimeout",
CommTimeoutCbPtrFun(clientLifetimeTimeout, context->http));
commSetConnTimeout(clientConnection, Config.Timeout.lifetime, timeoutCall);
context->registerWithConn();
processParsedRequest(context);
parsed_req = true; // XXX: do we really need to parse everything right NOW ?
if (context->mayUseConnection()) {
debugs(33, 3, HERE << "Not parsing new requests, as this request may need the connection");
break;
}
} else {
debugs(33, 5, clientConnection << ": not enough request data: " <<
inBuf.length() << " < " << Config.maxRequestHeaderSize);
Must(inBuf.length() < Config.maxRequestHeaderSize);
@@ -2358,57 +2346,46 @@
flags.readMore = false;
}
void
ConnStateData::noteBodyConsumerAborted(BodyPipe::Pointer )
{
// request reader may get stuck waiting for space if nobody consumes body
if (bodyPipe != NULL)
bodyPipe->enableAutoConsumption();
// kids extend
}
/** general lifetime handler for HTTP requests */
void
ConnStateData::requestTimeout(const CommTimeoutCbParams &io)
{
if (!Comm::IsConnOpen(io.conn))
return;
- if (Config.accessList.on_unsupported_protocol && !receivedFirstByte_) {
-#if USE_OPENSSL
- if (serverBump() && (serverBump()->act.step1 == Ssl::bumpPeek || serverBump()->act.step1 == Ssl::bumpStare)) {
- if (spliceOnError(ERR_REQUEST_START_TIMEOUT)) {
- receivedFirstByte();
- return;
- }
- } else if (!fd_table[io.conn->fd].ssl)
-#endif
- {
- const HttpRequestMethod method;
- if (clientTunnelOnError(this, NULL, NULL, method, ERR_REQUEST_START_TIMEOUT, Http::scNone, NULL)) {
- // Tunnel established. Set receivedFirstByte to avoid loop.
- receivedFirstByte();
- return;
- }
- }
+ if (mayTunnelUnsupportedProto() && !receivedFirstByte_) {
+ Http::StreamPointer context = pipeline.front();
+ Must(context && context->http);
+ HttpRequest::Pointer request = context->http->request;
+ if (clientTunnelOnError(this, context, request, HttpRequestMethod(), ERR_REQUEST_START_TIMEOUT))
+ return;
}
/*
* Just close the connection to not confuse browsers
* using persistent connections. Some browsers open
* a connection and then do not use it until much
* later (presumeably because the request triggering
* the open has already been completed on another
* connection)
*/
debugs(33, 3, "requestTimeout: FD " << io.fd << ": lifetime is expired.");
io.conn->close();
}
static void
clientLifetimeTimeout(const CommTimeoutCbParams &io)
{
ClientHttpRequest *http = static_cast<ClientHttpRequest *>(io.data);
debugs(33, DBG_IMPORTANT, "WARNING: Closing client connection due to lifetime timeout");
debugs(33, DBG_IMPORTANT, "\t" << http->uri);
http->logType.err.timedout = true;
@@ -2750,41 +2727,41 @@
Comm::SetSelect(details->fd, COMM_SELECT_READ, clientNegotiateSSL, connState, 0);
}
/**
* A callback function to use with the ACLFilledChecklist callback.
* In the case of ACCESS_ALLOWED answer initializes a bumped SSL connection,
* else reverts the connection to tunnel mode.
*/
static void
httpsSslBumpAccessCheckDone(allow_t answer, void *data)
{
ConnStateData *connState = (ConnStateData *) data;
// if the connection is closed or closing, just return.
if (!connState->isOpen())
return;
// Require both a match and a positive bump mode to work around exceptional
// cases where ACL code may return ACCESS_ALLOWED with zero answer.kind.
- if (answer == ACCESS_ALLOWED && (answer.kind != Ssl::bumpNone && answer.kind != Ssl::bumpSplice)) {
+ if (answer == ACCESS_ALLOWED && answer.kind != Ssl::bumpNone) {
debugs(33, 2, "sslBump needed for " << connState->clientConnection << " method " << answer.kind);
connState->sslBumpMode = static_cast<Ssl::BumpMode>(answer.kind);
} else {
debugs(33, 2, HERE << "sslBump not needed for " << connState->clientConnection);
connState->sslBumpMode = Ssl::bumpNone;
}
if (!connState->fakeAConnectRequest("ssl-bump", connState->inBuf))
connState->clientConnection->close();
}
/** handle a new HTTPS connection */
static void
httpsAccept(const CommAcceptCbParams ¶ms)
{
MasterXaction::Pointer xact = params.xaction;
const AnyP::PortCfgPointer s = xact->squidPort;
// NP: it is possible the port was reconfigured when the call or accept() was queued.
if (params.flag != Comm::OK) {
@@ -2882,56 +2859,43 @@
Security::ContextPointer ctx;
ctx.resetAndLock(SSL_get_SSL_CTX(ssl));
Ssl::configureUnconfiguredSslContext(ctx, signAlgorithm, *port);
} else {
Security::ContextPointer ctx(Ssl::generateSslContextUsingPkeyAndCertFromMemory(reply_message.getBody().c_str(), *port));
getSslContextDone(ctx, true);
}
return;
}
}
}
Security::ContextPointer nil;
getSslContextDone(nil);
}
void ConnStateData::buildSslCertGenerationParams(Ssl::CertificateProperties &certProperties)
{
certProperties.commonName = sslCommonName_.isEmpty() ? sslConnectHostOrIp.termedBuf() : sslCommonName_.c_str();
- // fake certificate adaptation requires bump-server-first mode
- if (!sslServerBump) {
- assert(port->signingCert.get());
- certProperties.signWithX509.resetAndLock(port->signingCert.get());
- if (port->signPkey.get())
- certProperties.signWithPkey.resetAndLock(port->signPkey.get());
- certProperties.signAlgorithm = Ssl::algSignTrusted;
- return;
- }
-
- // In case of an error while connecting to the secure server, use a fake
- // trusted certificate, with no mimicked fields and no adaptation
- // algorithms. There is nothing we can mimic so we want to minimize the
- // number of warnings the user will have to see to get to the error page.
- assert(sslServerBump->entry);
- if (sslServerBump->entry->isEmpty()) {
+ const bool triedToConnect = sslServerBump && sslServerBump->entry;
+ const bool connectedOK = triedToConnect && sslServerBump->entry->isEmpty();
+ if (connectedOK) {
if (X509 *mimicCert = sslServerBump->serverCert.get())
certProperties.mimicCert.resetAndLock(mimicCert);
ACLFilledChecklist checklist(NULL, sslServerBump->request.getRaw(),
clientConnection != NULL ? clientConnection->rfc931 : dash_str);
checklist.sslErrors = cbdataReference(sslServerBump->sslErrors());
for (sslproxy_cert_adapt *ca = Config.ssl_client.cert_adapt; ca != NULL; ca = ca->next) {
// If the algorithm already set, then ignore it.
if ((ca->alg == Ssl::algSetCommonName && certProperties.setCommonName) ||
(ca->alg == Ssl::algSetValidAfter && certProperties.setValidAfter) ||
(ca->alg == Ssl::algSetValidBefore && certProperties.setValidBefore) )
continue;
if (ca->aclList && checklist.fastCheck(ca->aclList) == ACCESS_ALLOWED) {
const char *alg = Ssl::CertAdaptAlgorithmStr[ca->alg];
const char *param = ca->param;
// For parameterless CN adaptation, use hostname from the
// CONNECT request.
@@ -2940,45 +2904,47 @@
param = sslConnectHostOrIp.termedBuf();
certProperties.commonName = param;
certProperties.setCommonName = true;
} else if (ca->alg == Ssl::algSetValidAfter)
certProperties.setValidAfter = true;
else if (ca->alg == Ssl::algSetValidBefore)
certProperties.setValidBefore = true;
debugs(33, 5, HERE << "Matches certificate adaptation aglorithm: " <<
alg << " param: " << (param ? param : "-"));
}
}
certProperties.signAlgorithm = Ssl::algSignEnd;
for (sslproxy_cert_sign *sg = Config.ssl_client.cert_sign; sg != NULL; sg = sg->next) {
if (sg->aclList && checklist.fastCheck(sg->aclList) == ACCESS_ALLOWED) {
certProperties.signAlgorithm = (Ssl::CertSignAlgorithm)sg->alg;
break;
}
}
- } else {// if (!sslServerBump->entry->isEmpty())
- // Use trusted certificate for a Squid-generated error
- // or the user would have to add a security exception
- // just to see the error page. We will close the connection
- // so that the trust is not extended to non-Squid content.
+ } else {// did not try to connect (e.g. client-first) or failed to connect
+ // In case of an error while connecting to the secure server, use a
+ // trusted certificate, with no mimicked fields and no adaptation
+ // algorithms. There is nothing we can mimic, so we want to minimize the
+ // number of warnings the user will have to see to get to the error page.
+ // We will close the connection, so that the trust is not extended to
+ // non-Squid content.
certProperties.signAlgorithm = Ssl::algSignTrusted;
}
assert(certProperties.signAlgorithm != Ssl::algSignEnd);
if (certProperties.signAlgorithm == Ssl::algSignUntrusted) {
assert(port->untrustedSigningCert.get());
certProperties.signWithX509.resetAndLock(port->untrustedSigningCert.get());
certProperties.signWithPkey.resetAndLock(port->untrustedSignPkey.get());
} else {
assert(port->signingCert.get());
certProperties.signWithX509.resetAndLock(port->signingCert.get());
if (port->signPkey.get())
certProperties.signWithPkey.resetAndLock(port->signPkey.get());
}
signAlgorithm = certProperties.signAlgorithm;
certProperties.signHash = Ssl::DefaultSignHash;
}
@@ -3167,85 +3133,77 @@
assert(!inBuf.isEmpty());
receivedFirstByte();
fd_note(clientConnection->fd, "Parsing TLS handshake");
bool unsupportedProtocol = false;
try {
if (!tlsParser.parseHello(inBuf)) {
// need more data to finish parsing
readSomeData();
return;
}
}
catch (const std::exception &ex) {
debugs(83, 2, "error on FD " << clientConnection->fd << ": " << ex.what());
unsupportedProtocol = true;
}
parsingTlsHandshake = false;
+ if (mayTunnelUnsupportedProto())
+ preservedClientData = inBuf;
+
// Even if the parser failed, each TLS detail should either be set
// correctly or still be "unknown"; copying unknown detail is a no-op.
clientConnection->tlsNegotiations()->retrieveParsedInfo(tlsParser.details);
// We should disable read/write handlers
Comm::SetSelect(clientConnection->fd, COMM_SELECT_READ, NULL, NULL, 0);
Comm::SetSelect(clientConnection->fd, COMM_SELECT_WRITE, NULL, NULL, 0);
- if (!sslServerBump) { // BumpClientFirst mode does not use this member
+ if (unsupportedProtocol) {
+ Http::StreamPointer context = pipeline.front();
+ Must(context && context->http);
+ HttpRequest::Pointer request = context->http->request;
+ debugs(83, 5, "Got something other than TLS Client Hello. Cannot SslBump.");
+ sslBumpMode = Ssl::bumpNone;
+ if (!clientTunnelOnError(this, context, request, HttpRequestMethod(), ERR_PROTOCOL_UNKNOWN))
+ clientConnection->close();
+ return;
+ }
+
+ if (!sslServerBump || sslServerBump->act.step1 == Ssl::bumpClientFirst) { // Either means client-first.
getSslContextStart();
return;
} else if (sslServerBump->act.step1 == Ssl::bumpServerFirst) {
// will call httpsPeeked() with certificate and connection, eventually
FwdState::fwdStart(clientConnection, sslServerBump->entry, sslServerBump->request.getRaw());
} else {
Must(sslServerBump->act.step1 == Ssl::bumpPeek || sslServerBump->act.step1 == Ssl::bumpStare);
- startPeekAndSplice(unsupportedProtocol);
+ startPeekAndSplice();
}
}
-bool
-ConnStateData::spliceOnError(const err_type err)
-{
- if (Config.accessList.on_unsupported_protocol) {
- assert(serverBump());
- ACLFilledChecklist checklist(Config.accessList.on_unsupported_protocol, serverBump()->request.getRaw(), NULL);
- checklist.requestErrorType = err;
- checklist.conn(this);
- allow_t answer = checklist.fastCheck();
- if (answer == ACCESS_ALLOWED && answer.kind == 1) {
- return splice();
- }
- }
- return false;
-}
-
void
-ConnStateData::startPeekAndSplice(const bool unsupportedProtocol)
+ConnStateData::startPeekAndSplice()
{
- if (unsupportedProtocol) {
- if (!spliceOnError(ERR_PROTOCOL_UNKNOWN))
- clientConnection->close();
- return;
- }
-
if (serverBump()) {
Security::TlsDetails::Pointer const &details = tlsParser.details;
if (details && !details->serverName.isEmpty()) {
serverBump()->clientSni = details->serverName;
resetSslCommonName(details->serverName.c_str());
}
}
startPeekAndSpliceDone();
}
void httpsSslBumpStep2AccessCheckDone(allow_t answer, void *data)
{
ConnStateData *connState = (ConnStateData *) data;
// if the connection is closed or closing, just return.
if (!connState->isOpen())
return;
debugs(33, 5, "Answer: " << answer << " kind:" << answer.kind);
@@ -3261,55 +3219,48 @@
if (bumpAction == Ssl::bumpTerminate) {
connState->clientConnection->close();
} else if (bumpAction != Ssl::bumpSplice) {
connState->startPeekAndSpliceDone();
} else if (!connState->splice())
connState->clientConnection->close();
}
bool
ConnStateData::splice()
{
// normally we can splice here, because we just got client hello message
if (fd_table[clientConnection->fd].ssl.get()) {
// Restore default read methods
fd_table[clientConnection->fd].read_method = &default_read_method;
fd_table[clientConnection->fd].write_method = &default_write_method;
}
- if (transparent()) {
- // set the current protocol to something sensible (was "HTTPS" for the bumping process)
- // we are sending a faked-up HTTP/1.1 message wrapper, so go with that.
- transferProtocol = Http::ProtocolVersion();
- return fakeAConnectRequest("intercepted TLS spliced", inBuf);
- } else {
- // XXX: assuming that there was an HTTP/1.1 CONNECT to begin with...
-
- // reset the current protocol to HTTP/1.1 (was "HTTPS" for the bumping process)
- transferProtocol = Http::ProtocolVersion();
- Http::StreamPointer context = pipeline.front();
- ClientHttpRequest *http = context->http;
- tunnelStart(http);
- return true;
- }
+ // XXX: assuming that there was an HTTP/1.1 CONNECT to begin with...
+ // reset the current protocol to HTTP/1.1 (was "HTTPS" for the bumping process)
+ transferProtocol = Http::ProtocolVersion();
+ assert(!pipeline.empty());
+ Http::StreamPointer context = pipeline.front();
+ ClientHttpRequest *http = context->http;
+ tunnelStart(http);
+ return true;
}
void
ConnStateData::startPeekAndSpliceDone()
{
// This is the Step2 of the SSL bumping
assert(sslServerBump);
Http::StreamPointer context = pipeline.front();
ClientHttpRequest *http = context ? context->http : NULL;
if (sslServerBump->step == Ssl::bumpStep1) {
sslServerBump->step = Ssl::bumpStep2;
// Run a accessList check to check if want to splice or continue bumping
ACLFilledChecklist *acl_checklist = new ACLFilledChecklist(Config.accessList.ssl_bump, sslServerBump->request.getRaw(), NULL);
acl_checklist->al = http ? http->al : NULL;
//acl_checklist->src_addr = params.conn->remote;
//acl_checklist->my_addr = s->s;
acl_checklist->banAction(allow_t(ACCESS_ALLOWED, Ssl::bumpNone));
acl_checklist->banAction(allow_t(ACCESS_ALLOWED, Ssl::bumpClientFirst));
@@ -3323,42 +3274,42 @@
fd_table[clientConnection->fd].dynamicTlsContext = unConfiguredCTX;
if (!httpsCreate(clientConnection, unConfiguredCTX))
return;
switchedToHttps_ = true;
auto ssl = fd_table[clientConnection->fd].ssl.get();
BIO *b = SSL_get_rbio(ssl);
Ssl::ClientBio *bio = static_cast<Ssl::ClientBio *>(b->ptr);
bio->setReadBufData(inBuf);
bio->hold(true);
// Here squid should have all of the client hello message so the
// Squid_SSL_accept should return 0;
// This block exist only to force openSSL parse client hello and detect
// ERR_SECURE_ACCEPT_FAIL error, which should be checked and splice if required.
int ret = 0;
if ((ret = Squid_SSL_accept(this, NULL)) < 0) {
debugs(83, 2, "SSL_accept failed.");
- const err_type err = ERR_SECURE_ACCEPT_FAIL;
- if (!spliceOnError(err))
+ HttpRequest::Pointer request = http->request;
+ if (!clientTunnelOnError(this, context, request, HttpRequestMethod(), ERR_SECURE_ACCEPT_FAIL))
clientConnection->close();
return;
}
// We need to reset inBuf here, to be used by incoming requests in the case
// of SSL bump
inBuf.clear();
debugs(83, 5, "Peek and splice at step2 done. Start forwarding the request!!! ");
FwdState::Start(clientConnection, sslServerBump->entry, sslServerBump->request.getRaw(), http ? http->al : NULL);
}
void
ConnStateData::doPeekAndSpliceStep()
{
auto ssl = fd_table[clientConnection->fd].ssl.get();
BIO *b = SSL_get_rbio(ssl);
assert(b);
Ssl::ClientBio *bio = static_cast<Ssl::ClientBio *>(b->ptr);
@@ -3375,75 +3326,159 @@
Must(sslServerBump != NULL);
if (Comm::IsConnOpen(serverConnection)) {
pinConnection(serverConnection, NULL, NULL, false);
debugs(33, 5, HERE << "bumped HTTPS server: " << sslConnectHostOrIp);
} else {
debugs(33, 5, HERE << "Error while bumping: " << sslConnectHostOrIp);
// copy error detail from bump-server-first request to CONNECT request
if (!pipeline.empty() && pipeline.front()->http != nullptr && pipeline.front()->http->request)
pipeline.front()->http->request->detailError(sslServerBump->request->errType, sslServerBump->request->errDetail);
}
getSslContextStart();
}
#endif /* USE_OPENSSL */
bool
-ConnStateData::fakeAConnectRequest(const char *reason, const SBuf &payload)
+ConnStateData::initiateTunneledRequest(HttpRequest::Pointer const &cause, Http::MethodType const method, const char *reason, const SBuf &payload)
{
// fake a CONNECT request to force connState to tunnel
SBuf connectHost;
+ unsigned short connectPort = 0;
+
+ if (pinning.serverConnection != nullptr) {
+ static char ip[MAX_IPSTRLEN];
+ connectHost.assign(pinning.serverConnection->remote.toStr(ip, sizeof(ip)));
+ connectPort = pinning.serverConnection->remote.port();
+ } else if (cause != NULL && cause->method == Http::METHOD_CONNECT) {
+ // We are inside a (not fully established) CONNECT request
+ connectHost = cause->url.host();
+ connectPort = cause->url.port();
+ } else {
+ debugs(33, 2, "Not able to compute URL, abort request tunneling for " << reason);
+ return false;
+ }
+
+ debugs(33, 2, "Request tunneling for " << reason);
+ ClientHttpRequest *http = buildFakeRequest(method, connectHost, connectPort, payload);
+ HttpRequest::Pointer request = http->request;
+ request->flags.forceTunnel = true;
+ http->calloutContext = new ClientRequestContext(http);
+ http->doCallouts();
+ clientProcessRequestFinished(this, request);
+ return true;
+}
+
+bool
+ConnStateData::fakeAConnectRequest(const char *reason, const SBuf &payload)
+{
+ debugs(33, 2, "fake a CONNECT request to force connState to tunnel for " << reason);
+
+ SBuf connectHost;
+ assert(transparent());
+ const unsigned short connectPort = clientConnection->local.port();
+
#if USE_OPENSSL
- if (serverBump() && !serverBump()->clientSni.isEmpty()) {
+ if (serverBump() && !serverBump()->clientSni.isEmpty())
connectHost.assign(serverBump()->clientSni);
- if (clientConnection->local.port() > 0)
- connectHost.appendf(":%d",clientConnection->local.port());
- } else
+ else
#endif
{
static char ip[MAX_IPSTRLEN];
- connectHost.assign(clientConnection->local.toUrl(ip, sizeof(ip)));
+ connectHost.assign(clientConnection->local.toStr(ip, sizeof(ip)));
}
- // Pre-pend this fake request to the TLS bits already in the buffer
- SBuf retStr;
- retStr.append("CONNECT ");
- retStr.append(connectHost);
- retStr.append(" HTTP/1.1\r\nHost: ");
- retStr.append(connectHost);
- retStr.append("\r\n\r\n");
- retStr.append(payload);
- inBuf = retStr;
- bool ret = handleReadData();
- if (ret)
- ret = clientParseRequests();
- if (!ret) {
- debugs(33, 2, "Failed to start fake CONNECT request for " << reason << " connection: " << clientConnection);
- return false;
- }
+ ClientHttpRequest *http = buildFakeRequest(Http::METHOD_CONNECT, connectHost, connectPort, payload);
+
+ http->calloutContext = new ClientRequestContext(http);
+ HttpRequest::Pointer request = http->request;
+ http->doCallouts();
+ clientProcessRequestFinished(this, request);
return true;
}
+ClientHttpRequest *
+ConnStateData::buildFakeRequest(Http::MethodType const method, SBuf &useHost, unsigned short usePort, const SBuf &payload)
+{
+ ClientHttpRequest *http = new ClientHttpRequest(this);
+ Http::Stream *stream = new Http::Stream(clientConnection, http);
+
+ StoreIOBuffer tempBuffer;
+ tempBuffer.data = stream->reqbuf;
+ tempBuffer.length = HTTP_REQBUF_SZ;
+
+ ClientStreamData newServer = new clientReplyContext(http);
+ ClientStreamData newClient = stream;
+ clientStreamInit(&http->client_stream, clientGetMoreData, clientReplyDetach,
+ clientReplyStatus, newServer, clientSocketRecipient,
+ clientSocketDetach, newClient, tempBuffer);
+
+ http->uri = SBufToCstring(useHost);
+ stream->flags.parsed_ok = 1; // Do we need it?
+ stream->mayUseConnection(true);
+
+ AsyncCall::Pointer timeoutCall = commCbCall(5, 4, "clientLifetimeTimeout",
+ CommTimeoutCbPtrFun(clientLifetimeTimeout, stream->http));
+ commSetConnTimeout(clientConnection, Config.Timeout.lifetime, timeoutCall);
+
+ stream->registerWithConn();
+
+ // Setup Http::Request object. Maybe should be replaced by a call to (modified)
+ // clientProcessRequest
+ HttpRequest::Pointer request = new HttpRequest();
+ AnyP::ProtocolType proto = (method == Http::METHOD_NONE) ? AnyP::PROTO_AUTHORITY_FORM : AnyP::PROTO_HTTP;
+ request->url.setScheme(proto, nullptr);
+ request->method = method;
+ request->url.host(useHost.c_str());
+ request->url.port(usePort);
+ http->request = request.getRaw();
+ HTTPMSGLOCK(http->request);
+
+ request->clientConnectionManager = this;
+
+ if (proto == AnyP::PROTO_HTTP)
+ request->header.putStr(Http::HOST, useHost.c_str());
+ request->flags.intercepted = ((clientConnection->flags & COMM_INTERCEPTION) != 0);
+ request->flags.interceptTproxy = ((clientConnection->flags & COMM_TRANSPARENT) != 0 );
+ request->sources |= ((switchedToHttps() || port->transport.protocol == AnyP::PROTO_HTTPS) ? HttpMsg::srcHttps : HttpMsg::srcHttp);
+#if USE_AUTH
+ if (getAuth() != NULL)
+ request->auth_user_request = getAuth();
+#endif
+ request->client_addr = clientConnection->remote;
+#if FOLLOW_X_FORWARDED_FOR
+ request->indirect_client_addr = clientConnection->remote;
+#endif /* FOLLOW_X_FORWARDED_FOR */
+ request->my_addr = clientConnection->local;
+ request->myportname = port->name;
+
+ inBuf = payload;
+ flags.readMore = false;
+
+ setLogUri(http, urlCanonicalClean(request.getRaw()));
+ return http;
+}
+
/// check FD after clientHttp[s]ConnectionOpened, adjust HttpSockets as needed
static bool
OpenedHttpSocket(const Comm::ConnectionPointer &c, const Ipc::FdNoteId portType)
{
if (!Comm::IsConnOpen(c)) {
Must(NHttpSockets > 0); // we tried to open some
--NHttpSockets; // there will be fewer sockets than planned
Must(HttpSockets[NHttpSockets] < 0); // no extra fds received
if (!NHttpSockets) // we could not open any listen sockets at all
fatalf("Unable to open %s",FdNote(portType));
return false;
}
return true;
}
/// find any unused HttpSockets[] slot and store fd there or return false
static bool
AddOpenedHttpSocket(const Comm::ConnectionPointer &conn)
@@ -4045,34 +4080,42 @@
void
ConnStateData::checkLogging()
{
// if we are parsing request body, its request is responsible for logging
if (bodyPipe)
return;
// a request currently using this connection is responsible for logging
if (!pipeline.empty() && pipeline.back()->mayUseConnection())
return;
/* Either we are waiting for the very first transaction, or
* we are done with the Nth transaction and are waiting for N+1st.
* XXX: We assume that if anything was added to inBuf, then it could
* only be consumed by actions already covered by the above checks.
*/
// do not log connections that closed after a transaction (it is normal)
// TODO: access_log needs ACLs to match received-no-bytes connections
- // XXX: TLS may return here even though we got no transactions yet
- // XXX: PROXY protocol may return here even though we got no
- // transactions yet
- if (receivedFirstByte_ && inBuf.isEmpty())
+ if (pipeline.nrequests && inBuf.isEmpty())
return;
/* Create a temporary ClientHttpRequest object. Its destructor will log. */
ClientHttpRequest http(this);
http.req_sz = inBuf.length();
char const *uri = "error:transaction-end-before-headers";
http.uri = xstrdup(uri);
setLogUri(&http, uri);
}
+bool
+ConnStateData::mayTunnelUnsupportedProto()
+{
+ return Config.accessList.on_unsupported_protocol
+#if USE_OPENSSL
+ &&
+ ((port->flags.isIntercepted() && port->flags.tunnelSslBumping)
+ || (serverBump() && pinning.serverConnection != NULL))
+#endif
+ ;
+}
=== modified file 'src/client_side.h'
--- src/client_side.h 2016-09-21 17:56:27 +0000
+++ src/client_side.h 2016-10-20 14:29:42 +0000
@@ -186,58 +186,53 @@
void connStateClosed(const CommCloseCbParams &io);
void requestTimeout(const CommTimeoutCbParams ¶ms);
// AsyncJob API
virtual void start();
virtual bool doneAll() const { return BodyProducer::doneAll() && false;}
virtual void swanSong();
/// Changes state so that we close the connection and quit after serving
/// the client-side-detected error response instead of getting stuck.
void quitAfterError(HttpRequest *request); // meant to be private
/// The caller assumes responsibility for connection closure detection.
void stopPinnedConnectionMonitoring();
#if USE_OPENSSL
/// the second part of old httpsAccept, waiting for future HttpsServer home
void postHttpsAccept();
/// Initializes and starts a peek-and-splice negotiation with the SSL client
- void startPeekAndSplice(const bool unknownProtocol);
+ void startPeekAndSplice();
/// Called when the initialization of peek-and-splice negotiation finidhed
void startPeekAndSpliceDone();
/// Called when a peek-and-splice step finished. For example after
/// server SSL certificates received and fake server SSL certificates
/// generated
void doPeekAndSpliceStep();
/// called by FwdState when it is done bumping the server
void httpsPeeked(Comm::ConnectionPointer serverConnection);
/// Splice a bumped client connection on peek-and-splice mode
bool splice();
- /// Check on_unsupported_protocol access list and splice if required
- /// \retval true on splice
- /// \retval false otherwise
- bool spliceOnError(const err_type err);
-
/// Start to create dynamic Security::ContextPointer for host or uses static port SSL context.
void getSslContextStart();
/**
* Done create dynamic ssl certificate.
*
* \param[in] isNew if generated certificate is new, so we need to add this certificate to storage.
*/
void getSslContextDone(Security::ContextPointer &, bool isNew = false);
/// Callback function. It is called when squid receive message from ssl_crtd.
static void sslCrtdHandleReplyWrapper(void *data, const Helper::Reply &reply);
/// Proccess response from ssl_crtd.
void sslCrtdHandleReply(const Helper::Reply &reply);
void switchToHttps(HttpRequest *request, Ssl::BumpMode bumpServerMode);
void parseTlsHandshake();
bool switchedToHttps() const { return switchedToHttps_; }
Ssl::ServerBump *serverBump() {return sslServerBump;}
inline void setServerBump(Ssl::ServerBump *srvBump) {
if (!sslServerBump)
sslServerBump = srvBump;
@@ -270,70 +265,79 @@
/// handle a control message received by context from a peer and call back
virtual void writeControlMsgAndCall(HttpReply *rep, AsyncCall::Pointer &call) = 0;
/// ClientStream calls this to supply response header (once) and data
/// for the current Http::Stream.
virtual void handleReply(HttpReply *header, StoreIOBuffer receivedData) = 0;
/// remove no longer needed leading bytes from the input buffer
void consumeInput(const size_t byteCount);
/* TODO: Make the methods below (at least) non-public when possible. */
/// stop parsing the request and create context for relaying error info
Http::Stream *abortRequestParsing(const char *const errUri);
/// generate a fake CONNECT request with the given payload
/// at the beginning of the client I/O buffer
bool fakeAConnectRequest(const char *reason, const SBuf &payload);
+ /// generates and sends to tunnel.cc a fake request with a given payload
+ bool initiateTunneledRequest(HttpRequest::Pointer const &cause, Http::MethodType const method, const char *reason, const SBuf &payload);
+
+ /// whether tunneling of unsupported protocol is allowed for this connection
+ bool mayTunnelUnsupportedProto();
+
+ /// build a fake http request
+ ClientHttpRequest *buildFakeRequest(Http::MethodType const method, SBuf &useHost, unsigned short usePort, const SBuf &payload);
+
/// client data which may need to forward as-is to server after an
/// on_unsupported_protocol tunnel decision.
SBuf preservedClientData;
/* Registered Runner API */
virtual void startShutdown();
virtual void endingShutdown();
protected:
void startDechunkingRequest();
void finishDechunkingRequest(bool withSuccess);
void abortChunkedRequestBody(const err_type error);
err_type handleChunkedRequestBody();
void startPinnedConnectionMonitoring();
void clientPinnedConnectionRead(const CommIoCbParams &io);
#if USE_OPENSSL
/// Handles a ready-for-reading TLS squid-to-server connection that
/// we thought was idle.
/// \return false if and only if the connection should be closed.
bool handleIdleClientPinnedTlsRead();
#endif
/// parse input buffer prefix into a single transfer protocol request
/// return NULL to request more header bytes (after checking any limits)
/// use abortRequestParsing() to handle parsing errors w/o creating request
virtual Http::Stream *parseOneRequest() = 0;
/// start processing a freshly parsed request
- virtual void processParsedRequest(Http::Stream *) = 0;
+ virtual void processParsedRequest(Http::StreamPointer &) = 0;
/// returning N allows a pipeline of 1+N requests (see pipeline_prefetch)
virtual int pipelinePrefetchMax() const;
/// timeout to use when waiting for the next request
virtual time_t idleTimeout() const = 0;
BodyPipe::Pointer bodyPipe; ///< set when we are reading request body
private:
/* ::Server API */
virtual bool connFinishedWithConn(int size);
virtual void checkLogging();
void clientAfterReadingRequests();
bool concurrentRequestQueueFilled() const;
void pinNewConnection(const Comm::ConnectionPointer &pinServer, HttpRequest *request, CachePeer *aPeer, bool auth);
/* PROXY protocol functionality */
=== modified file 'src/client_side_request.cc'
--- src/client_side_request.cc 2016-10-06 22:05:50 +0000
+++ src/client_side_request.cc 2016-10-13 15:08:07 +0000
@@ -1395,40 +1395,45 @@
calloutContext->checkNoCacheDone(answer);
}
void
ClientRequestContext::checkNoCacheDone(const allow_t &answer)
{
acl_checklist = NULL;
http->request->flags.cachable = (answer == ACCESS_ALLOWED);
http->doCallouts();
}
#if USE_OPENSSL
bool
ClientRequestContext::sslBumpAccessCheck()
{
if (!http->getConn()) {
http->al->ssl.bumpMode = Ssl::bumpEnd; // SslBump does not apply; log -
return false;
}
+ if (http->request->flags.forceTunnel) {
+ debugs(85, 5, "not needed; already decided to tunnel " << http->getConn());
+ return false;
+ }
+
// If SSL connection tunneling or bumping decision has been made, obey it.
const Ssl::BumpMode bumpMode = http->getConn()->sslBumpMode;
if (bumpMode != Ssl::bumpEnd) {
debugs(85, 5, HERE << "SslBump already decided (" << bumpMode <<
"), " << "ignoring ssl_bump for " << http->getConn());
if (!http->getConn()->serverBump())
http->sslBumpNeed(bumpMode); // for processRequest() to bump if needed and not already bumped
http->al->ssl.bumpMode = bumpMode; // inherited from bumped connection
return false;
}
// If we have not decided yet, decide whether to bump now.
// Bumping here can only start with a CONNECT request on a bumping port
// (bumping of intercepted SSL conns is decided before we get 1st request).
// We also do not bump redirected CONNECT requests.
if (http->request->method != Http::METHOD_CONNECT || http->redirect.status ||
!Config.accessList.ssl_bump ||
!http->getConn()->port->flags.tunnelSslBumping) {
http->al->ssl.bumpMode = Ssl::bumpEnd; // SslBump does not apply; log -
@@ -1473,47 +1478,51 @@
const Ssl::BumpMode bumpMode = answer == ACCESS_ALLOWED ?
static_cast<Ssl::BumpMode>(answer.kind) : Ssl::bumpNone;
http->sslBumpNeed(bumpMode); // for processRequest() to bump if needed
http->al->ssl.bumpMode = bumpMode; // for logging
http->doCallouts();
}
#endif
/*
* Identify requests that do not go through the store and client side stream
* and forward them to the appropriate location. All other requests, request
* them.
*/
void
ClientHttpRequest::processRequest()
{
debugs(85, 4, request->method << ' ' << uri);
- if (request->method == Http::METHOD_CONNECT && !redirect.status) {
+ const bool untouchedConnect = request->method == Http::METHOD_CONNECT && !redirect.status;
+
#if USE_OPENSSL
- if (sslBumpNeeded()) {
- sslBumpStart();
- return;
- }
+ if (untouchedConnect && sslBumpNeeded()) {
+ assert(!request->flags.forceTunnel);
+ sslBumpStart();
+ return;
+ }
#endif
+
+ if (untouchedConnect || request->flags.forceTunnel) {
getConn()->stopReading(); // tunnels read for themselves
tunnelStart(this);
return;
}
httpStart();
}
void
ClientHttpRequest::httpStart()
{
PROF_start(httpStart);
logType = LOG_TAG_NONE;
debugs(85, 4, logType.c_str() << " for '" << uri << "'");
/* no one should have touched this */
assert(out.offset == 0);
/* Use the Stream Luke */
clientStreamNode *node = (clientStreamNode *)client_stream.tail->data;
clientStreamRead(node, this, node->readBuffer);
@@ -1778,41 +1787,41 @@
// We need to check for SslBump even if the calloutContext->error is set
// because bumping may require delaying the error until after CONNECT.
if (!calloutContext->sslBumpCheckDone) {
calloutContext->sslBumpCheckDone = true;
if (calloutContext->sslBumpAccessCheck())
return;
/* else no ssl bump required*/
}
#endif
if (calloutContext->error) {
// XXX: prformance regression. c_str() reallocates
SBuf storeUriBuf(request->storeId());
const char *storeUri = storeUriBuf.c_str();
StoreEntry *e = storeCreateEntry(storeUri, storeUri, request->flags, request->method);
#if USE_OPENSSL
if (sslBumpNeeded()) {
// We have to serve an error, so bump the client first.
sslBumpNeed(Ssl::bumpClientFirst);
// set final error but delay sending until we bump
- Ssl::ServerBump *srvBump = new Ssl::ServerBump(request, e);
+ Ssl::ServerBump *srvBump = new Ssl::ServerBump(request, e, Ssl::bumpClientFirst);
errorAppendEntry(e, calloutContext->error);
calloutContext->error = NULL;
getConn()->setServerBump(srvBump);
e->unlock("ClientHttpRequest::doCallouts+sslBumpNeeded");
} else
#endif
{
// send the error to the client now
clientStreamNode *node = (clientStreamNode *)client_stream.tail->prev->data;
clientReplyContext *repContext = dynamic_cast<clientReplyContext *>(node->data.getRaw());
assert (repContext);
repContext->setReplyToStoreEntry(e, "immediate SslBump error");
errorAppendEntry(e, calloutContext->error);
calloutContext->error = NULL;
if (calloutContext->readNextRequest && getConn())
getConn()->flags.readMore = true; // resume any pipeline reads.
node = (clientStreamNode *)client_stream.tail->data;
clientStreamRead(node, this, node->readBuffer);
e->unlock("ClientHttpRequest::doCallouts-sslBumpNeeded");
return;
=== modified file 'src/http/Stream.h'
--- src/http/Stream.h 2016-07-02 06:47:55 +0000
+++ src/http/Stream.h 2016-10-13 16:48:30 +0000
@@ -58,40 +58,43 @@
*
*
* XXX: If an async call ends the ClientHttpRequest job, Http::Stream
* (and ConnStateData) may not know about it, leading to segfaults and
* assertions. This is difficult to fix
* because ClientHttpRequest lacks a good way to communicate its ongoing
* destruction back to the Http::Stream which pretends to "own" *http.
*/
class Stream : public RefCountable
{
MEMPROXY_CLASS(Stream);
public:
/// construct with HTTP/1.x details
Stream(const Comm::ConnectionPointer &aConn, ClientHttpRequest *aReq);
~Stream();
/// register this stream with the Server
void registerWithConn();
+ /// whether it is registered with a Server
+ bool connRegistered() const {return connRegistered_;};
+
/// whether the reply has started being sent
bool startOfOutput() const;
/// update stream state after a write, may initiate more I/O
void writeComplete(size_t size);
/// get more data to send
void pullData();
/// \return true if the HTTP request is for multiple ranges
bool multipartRangeRequest() const;
int64_t getNextRangeOffset() const;
bool canPackMoreRanges() const;
size_t lengthToSend(Range<int64_t> const &available) const;
clientStream_status_t socketState();
/// send an HTTP reply message headers and maybe some initial payload
void sendStartOfMessage(HttpReply *, StoreIOBuffer bodyData);
=== modified file 'src/http/one/RequestParser.cc'
--- src/http/one/RequestParser.cc 2016-10-04 14:25:15 +0000
+++ src/http/one/RequestParser.cc 2016-10-13 15:08:07 +0000
@@ -3,42 +3,43 @@
*
* Squid software is distributed under GPLv2+ license and includes
* contributions from numerous individuals and organizations.
* Please see the COPYING and CONTRIBUTORS files for details.
*/
#include "squid.h"
#include "Debug.h"
#include "http/one/RequestParser.h"
#include "http/one/Tokenizer.h"
#include "http/ProtocolVersion.h"
#include "profiler/Profiler.h"
#include "SquidConfig.h"
// the right debugs() level for parsing errors
inline static int
ErrorLevel() {
return Config.onoff.relaxed_header_parser < 0 ? DBG_IMPORTANT : 5;
}
-Http::One::RequestParser::RequestParser() :
- Parser()
+Http::One::RequestParser::RequestParser(bool preserveParsed) :
+ Parser(),
+ preserveParsed_(preserveParsed)
{}
Http1::Parser::size_type
Http::One::RequestParser::firstLineSize() const
{
// RFC 7230 section 2.6
/* method SP request-target SP "HTTP/" DIGIT "." DIGIT CRLF */
return method_.image().length() + uri_.length() + 12;
}
/**
* Attempt to parse the first line of a new request message.
*
* Governed by RFC 7230 section 3.5
* "
* In the interest of robustness, a server that is expecting to receive
* and parse a request-line SHOULD ignore at least one empty line (CRLF)
* received prior to the request-line.
* "
*
@@ -330,40 +331,53 @@
/* parsed everything before and after the URI */
if (!parseUriField(tok))
return -1;
if (!tok.atEnd()) {
debugs(33, ErrorLevel(), "invalid request-line: garbage after URI");
parseStatusCode = Http::scBadRequest;
return -1;
}
parseStatusCode = Http::scOkay;
buf_ = lineTok.remaining(); // incremental parse checkpoint
return 1;
}
bool
Http::One::RequestParser::parse(const SBuf &aBuf)
{
+ const bool result = doParse(aBuf);
+ if (preserveParsed_) {
+ assert(aBuf.length() >= remaining().length());
+ parsed_.append(aBuf.substr(0, aBuf.length() - remaining().length())); // newly parsed bytes
+ }
+
+ return result;
+}
+
+// raw is not a reference because a reference might point back to our own buf_ or parsed_
+bool
+Http::One::RequestParser::doParse(const SBuf &aBuf)
+{
buf_ = aBuf;
debugs(74, DBG_DATA, "Parse buf={length=" << aBuf.length() << ", data='" << aBuf << "'}");
// stage 1: locate the request-line
if (parsingStage_ == HTTP_PARSE_NONE) {
skipGarbageLines();
// if we hit something before EOS treat it as a message
if (!buf_.isEmpty())
parsingStage_ = HTTP_PARSE_FIRST;
else
return false;
}
// stage 2: parse the request-line
if (parsingStage_ == HTTP_PARSE_FIRST) {
PROF_start(HttpParserParseReqLine);
const int retcode = parseRequestFirstLine();
// first-line (or a look-alike) found successfully.
=== modified file 'src/http/one/RequestParser.h'
--- src/http/one/RequestParser.h 2016-08-25 17:26:02 +0000
+++ src/http/one/RequestParser.h 2016-10-13 14:32:32 +0000
@@ -13,60 +13,70 @@
#include "http/RequestMethod.h"
namespace Parser {
class Tokenizer;
}
namespace Http {
namespace One {
/** HTTP/1.x protocol request parser
*
* Works on a raw character I/O buffer and tokenizes the content into
* the major CRLF delimited segments of an HTTP/1 request message:
*
* \item request-line (method, URL, protocol, version)
* \item mime-header (set of RFC2616 syntax header fields)
*/
class RequestParser : public Http1::Parser
{
public:
- RequestParser();
+ explicit RequestParser(bool preserveParsed = false);
virtual ~RequestParser() {}
/* Http::One::Parser API */
virtual void clear() {*this = RequestParser();}
virtual Http1::Parser::size_type firstLineSize() const;
virtual bool parse(const SBuf &aBuf);
/// the HTTP method if this is a request message
const HttpRequestMethod & method() const {return method_;}
/// the request-line URI if this is a request message, or an empty string.
const SBuf &requestUri() const {return uri_;}
+ /// the accumulated parsed bytes
+ const SBuf &parsed() const { Must(preserveParsed_); return parsed_; }
+
private:
void skipGarbageLines();
int parseRequestFirstLine();
+ /// called from parse() to do the parsing
+ bool doParse(const SBuf &aBuf);
/* all these return false and set parseStatusCode on parsing failures */
bool parseMethodField(Http1::Tokenizer &);
bool parseUriField(Http1::Tokenizer &);
bool parseHttpVersionField(Http1::Tokenizer &);
bool skipDelimiter(const size_t count, const char *where);
bool skipTrailingCrs(Http1::Tokenizer &tok);
bool http0() const {return !msgProtocol_.major;}
static const CharacterSet &RequestTargetCharacters();
/// what request method has been found on the first line
HttpRequestMethod method_;
/// raw copy of the original client request-line URI field
SBuf uri_;
+
+ /// all parsed bytes (i.e., input prefix consumed by parse() calls)
+ /// meaningless unless preserveParsed_ is true
+ SBuf parsed_;
+ bool preserveParsed_; ///< whether to accumulate parsed bytes (in parsed_)
};
} // namespace One
} // namespace Http
#endif /* _SQUID_SRC_HTTP_ONE_REQUESTPARSER_H */
=== modified file 'src/servers/FtpServer.cc'
--- src/servers/FtpServer.cc 2016-07-02 06:47:55 +0000
+++ src/servers/FtpServer.cc 2016-10-13 09:21:57 +0000
@@ -135,41 +135,41 @@
assert(http != NULL);
HttpRequest *const request = http->request;
Must(http->storeEntry() || request);
const bool mayForward = !http->storeEntry() && handleRequest(request);
if (http->storeEntry() != NULL) {
debugs(33, 4, "got an immediate response");
clientSetKeepaliveFlag(http);
context->pullData();
} else if (mayForward) {
debugs(33, 4, "forwarding request to server side");
assert(http->storeEntry() == NULL);
clientProcessRequest(this, Http1::RequestParserPointer(), context.getRaw());
} else {
debugs(33, 4, "will resume processing later");
}
}
void
-Ftp::Server::processParsedRequest(Http::Stream *)
+Ftp::Server::processParsedRequest(Http::StreamPointer &)
{
Must(pipeline.count() == 1);
// Process FTP request asynchronously to make sure FTP
// data connection accept callback is fired first.
CallJobHere(33, 4, CbcPointer<Server>(this),
Ftp::Server, doProcessRequest);
}
/// imports more upload data from the data connection
void
Ftp::Server::readUploadData(const CommIoCbParams &io)
{
debugs(33, 5, io.conn << " size " << io.size);
Must(reader != NULL);
reader = NULL;
assert(Comm::IsConnOpen(dataConn));
assert(io.conn->fd == dataConn->fd);
=== modified file 'src/servers/FtpServer.h'
--- src/servers/FtpServer.h 2016-03-15 18:12:09 +0000
+++ src/servers/FtpServer.h 2016-10-13 07:58:08 +0000
@@ -75,41 +75,41 @@
// This is a pointer in hope to minimize future changes when MasterState
// becomes a part of MasterXaction. Guaranteed not to be nil.
MasterState::Pointer master; ///< info shared among our FTP client and server jobs
protected:
friend void StartListening();
// errors detected before it is possible to create an HTTP request wrapper
enum class EarlyErrorKind {
HugeRequest,
MissingLogin,
MissingUsername,
MissingHost,
UnsupportedCommand,
InvalidUri,
MalformedCommand
};
/* ConnStateData API */
virtual Http::Stream *parseOneRequest() override;
- virtual void processParsedRequest(Http::Stream *context) override;
+ virtual void processParsedRequest(Http::StreamPointer &context) override;
virtual void notePeerConnection(Comm::ConnectionPointer conn) override;
virtual void clientPinnedConnectionClosed(const CommCloseCbParams &io) override;
virtual void handleReply(HttpReply *header, StoreIOBuffer receivedData) override;
virtual int pipelinePrefetchMax() const override;
virtual void writeControlMsgAndCall(HttpReply *rep, AsyncCall::Pointer &call) override;
virtual time_t idleTimeout() const override;
/* BodyPipe API */
virtual void noteMoreBodySpaceAvailable(BodyPipe::Pointer) override;
virtual void noteBodyConsumerAborted(BodyPipe::Pointer ptr) override;
/* AsyncJob API */
virtual void start() override;
/* Comm callbacks */
static void AcceptCtrlConnection(const CommAcceptCbParams ¶ms);
void acceptDataConnection(const CommAcceptCbParams ¶ms);
void readUploadData(const CommIoCbParams &io);
void wroteEarlyReply(const CommIoCbParams &io);
void wroteReply(const CommIoCbParams &io);
=== modified file 'src/servers/Http1Server.cc'
--- src/servers/Http1Server.cc 2016-05-02 06:09:13 +0000
+++ src/servers/Http1Server.cc 2016-10-13 16:49:41 +0000
@@ -63,158 +63,181 @@
{
if (!handleRequestBodyData())
return;
// too late to read more body
if (!isOpen() || stoppedReceiving())
return;
readSomeData();
}
Http::Stream *
Http::One::Server::parseOneRequest()
{
PROF_start(HttpServer_parseOneRequest);
// parser is incremental. Generate new parser state if we,
// a) dont have one already
// b) have completed the previous request parsing already
if (!parser_ || !parser_->needsMoreData())
- parser_ = new Http1::RequestParser();
+ parser_ = new Http1::RequestParser(mayTunnelUnsupportedProto());
/* Process request */
Http::Stream *context = parseHttpRequest(this, parser_);
PROF_stop(HttpServer_parseOneRequest);
return context;
}
void clientProcessRequestFinished(ConnStateData *conn, const HttpRequest::Pointer &request);
-bool clientTunnelOnError(ConnStateData *conn, Http::Stream *context, HttpRequest *request, const HttpRequestMethod& method, err_type requestError, Http::StatusCode errStatusCode, const char *requestErrorBytes);
+bool clientTunnelOnError(ConnStateData *conn, Http::StreamPointer &context, HttpRequest::Pointer &request, const HttpRequestMethod& method, err_type requestError);
bool
-Http::One::Server::buildHttpRequest(Http::Stream *context)
+Http::One::Server::buildHttpRequest(Http::StreamPointer &context)
{
HttpRequest::Pointer request;
ClientHttpRequest *http = context->http;
if (context->flags.parsed_ok == 0) {
debugs(33, 2, "Invalid Request");
// determine which error page templates to use for specific parsing errors
err_type errPage = ERR_INVALID_REQ;
switch (parser_->parseStatusCode) {
case Http::scRequestHeaderFieldsTooLarge:
// fall through to next case
case Http::scUriTooLong:
errPage = ERR_TOO_BIG;
break;
case Http::scMethodNotAllowed:
errPage = ERR_UNSUP_REQ;
break;
case Http::scHttpVersionNotSupported:
errPage = ERR_UNSUP_HTTPVERSION;
break;
default:
if (parser_->method() == METHOD_NONE || parser_->requestUri().length() == 0)
// no method or url parsed, probably is wrong protocol
errPage = ERR_PROTOCOL_UNKNOWN;
// else use default ERR_INVALID_REQ set above.
break;
}
// setLogUri should called before repContext->setReplyToError
setLogUri(http, http->uri, true);
const char * requestErrorBytes = inBuf.c_str();
- if (!clientTunnelOnError(this, context, request.getRaw(), parser_->method(), errPage, parser_->parseStatusCode, requestErrorBytes)) {
+ if (!clientTunnelOnError(this, context, request, parser_->method(), errPage)) {
+ setReplyError(context, request, parser_->method(), errPage, parser_->parseStatusCode, requestErrorBytes);
// HttpRequest object not build yet, there is no reason to call
// clientProcessRequestFinished method
}
return false;
}
if ((request = HttpRequest::CreateFromUrl(http->uri, parser_->method())) == NULL) {
debugs(33, 5, "Invalid URL: " << http->uri);
// setLogUri should called before repContext->setReplyToError
setLogUri(http, http->uri, true);
const char * requestErrorBytes = inBuf.c_str();
- if (!clientTunnelOnError(this, context, request.getRaw(), parser_->method(), ERR_INVALID_URL, Http::scBadRequest, requestErrorBytes)) {
+ if (!clientTunnelOnError(this, context, request, parser_->method(), ERR_INVALID_URL)) {
+ setReplyError(context, request, parser_->method(), ERR_INVALID_URL, Http::scBadRequest, requestErrorBytes);
// HttpRequest object not build yet, there is no reason to call
// clientProcessRequestFinished method
}
return false;
}
/* RFC 2616 section 10.5.6 : handle unsupported HTTP major versions cleanly. */
/* We currently only support 0.9, 1.0, 1.1 properly */
/* TODO: move HTTP-specific processing into servers/HttpServer and such */
if ( (parser_->messageProtocol().major == 0 && parser_->messageProtocol().minor != 9) ||
(parser_->messageProtocol().major > 1) ) {
debugs(33, 5, "Unsupported HTTP version discovered. :\n" << parser_->messageProtocol());
// setLogUri should called before repContext->setReplyToError
setLogUri(http, http->uri, true);
const char * requestErrorBytes = NULL; //HttpParserHdrBuf(parser_);
- if (!clientTunnelOnError(this, context, request.getRaw(), parser_->method(), ERR_UNSUP_HTTPVERSION, Http::scHttpVersionNotSupported, requestErrorBytes)) {
+ if (!clientTunnelOnError(this, context, request, parser_->method(), ERR_UNSUP_HTTPVERSION)) {
+ setReplyError(context, request, parser_->method(), ERR_UNSUP_HTTPVERSION, Http::scHttpVersionNotSupported, requestErrorBytes);
clientProcessRequestFinished(this, request);
}
return false;
}
/* compile headers */
if (parser_->messageProtocol().major >= 1 && !request->parseHeader(*parser_.getRaw())) {
debugs(33, 5, "Failed to parse request headers:\n" << parser_->mimeHeader());
// setLogUri should called before repContext->setReplyToError
setLogUri(http, http->uri, true);
const char * requestErrorBytes = NULL; //HttpParserHdrBuf(parser_);
- if (!clientTunnelOnError(this, context, request.getRaw(), parser_->method(), ERR_INVALID_REQ, Http::scBadRequest, requestErrorBytes)) {
+ if (!clientTunnelOnError(this, context, request, parser_->method(), ERR_INVALID_REQ)) {
+ setReplyError(context, request, parser_->method(), ERR_INVALID_REQ, Http::scBadRequest, requestErrorBytes);
clientProcessRequestFinished(this, request);
}
return false;
}
// when absolute-URI is provided Host header should be ignored. However
// some code still uses Host directly so normalize it using the previously
// sanitized URL authority value.
// For now preserve the case where Host is completely absent. That matters.
if (const auto x = request->header.delById(Http::HOST)) {
debugs(33, 5, "normalize " << x << " Host header using " << request->url.authority());
SBuf tmp(request->url.authority());
request->header.putStr(Http::HOST, tmp.c_str());
}
http->request = request.getRaw();
HTTPMSGLOCK(http->request);
return true;
}
void
+Http::One::Server::setReplyError(Http::StreamPointer &context, HttpRequest::Pointer &request, const HttpRequestMethod& method, err_type requestError, Http::StatusCode errStatusCode, const char *requestErrorBytes)
+{
+ quitAfterError(request.getRaw());
+ if (!context->connRegistered()) {
+ debugs(33, 2, "Client stream deregister it self, nothing to do");
+ clientConnection->close();
+ return;
+ }
+ clientStreamNode *node = context->getClientReplyContext();
+ clientReplyContext *repContext = dynamic_cast<clientReplyContext *>(node->data.getRaw());
+ assert (repContext);
+
+ repContext->setReplyToError(requestError, errStatusCode, method, context->http->uri, clientConnection->remote, NULL, requestErrorBytes, NULL);
+
+ assert(context->http->out.offset == 0);
+ context->pullData();
+}
+
+void
Http::One::Server::proceedAfterBodyContinuation(Http::StreamPointer context)
{
debugs(33, 5, "Body Continuation written");
clientProcessRequest(this, parser_, context.getRaw());
}
void
-Http::One::Server::processParsedRequest(Http::Stream *context)
+Http::One::Server::processParsedRequest(Http::StreamPointer &context)
{
if (!buildHttpRequest(context))
return;
ClientHttpRequest *http = context->http;
HttpRequest::Pointer request = http->request;
if (request->header.has(Http::HdrType::EXPECT)) {
const String expect = request->header.getList(Http::HdrType::EXPECT);
const bool supportedExpect = (expect.caseCmp("100-continue") == 0);
if (!supportedExpect) {
clientStreamNode *node = context->getClientReplyContext();
quitAfterError(request.getRaw());
// setLogUri should called before repContext->setReplyToError
setLogUri(http, urlCanonicalClean(request.getRaw()));
clientReplyContext *repContext = dynamic_cast<clientReplyContext *>(node->data.getRaw());
assert (repContext);
repContext->setReplyToError(ERR_INVALID_REQ, Http::scExpectationFailed, request->method, http->uri,
clientConnection->remote, request.getRaw(), NULL, NULL);
assert(context->http->out.offset == 0);
@@ -222,41 +245,41 @@
clientProcessRequestFinished(this, request);
return;
}
if (Config.accessList.forceRequestBodyContinuation) {
ACLFilledChecklist bodyContinuationCheck(Config.accessList.forceRequestBodyContinuation, request.getRaw(), NULL);
if (bodyContinuationCheck.fastCheck() == ACCESS_ALLOWED) {
debugs(33, 5, "Body Continuation forced");
request->forcedBodyContinuation = true;
//sendControlMsg
HttpReply::Pointer rep = new HttpReply;
rep->sline.set(Http::ProtocolVersion(), Http::scContinue);
typedef UnaryMemFunT<Http1::Server, Http::StreamPointer> CbDialer;
const AsyncCall::Pointer cb = asyncCall(11, 3, "Http1::Server::proceedAfterBodyContinuation", CbDialer(this, &Http1::Server::proceedAfterBodyContinuation, Http::StreamPointer(context)));
sendControlMsg(HttpControlMsg(rep, cb));
return;
}
}
}
- clientProcessRequest(this, parser_, context);
+ clientProcessRequest(this, parser_, context.getRaw());
}
void
Http::One::Server::noteBodyConsumerAborted(BodyPipe::Pointer ptr)
{
ConnStateData::noteBodyConsumerAborted(ptr);
stopReceiving("virgin request body consumer aborted"); // closes ASAP
}
void
Http::One::Server::handleReply(HttpReply *rep, StoreIOBuffer receivedData)
{
// the caller guarantees that we are dealing with the current context only
Http::StreamPointer context = pipeline.front();
Must(context != nullptr);
const ClientHttpRequest *http = context->http;
Must(http != NULL);
// After sending Transfer-Encoding: chunked (at least), always send
// the last-chunk if there was no error, ignoring responseFinishedOrFailed.
=== modified file 'src/servers/Http1Server.h'
--- src/servers/Http1Server.h 2016-01-24 17:41:43 +0000
+++ src/servers/Http1Server.h 2016-10-13 07:56:02 +0000
@@ -13,56 +13,58 @@
namespace Http
{
namespace One
{
/// Manages a connection from an HTTP/1 or HTTP/0.9 client.
class Server: public ConnStateData
{
CBDATA_CLASS(Server);
public:
Server(const MasterXaction::Pointer &xact, const bool beHttpsServer);
virtual ~Server() {}
void readSomeHttpData();
protected:
/* ConnStateData API */
virtual Http::Stream *parseOneRequest();
- virtual void processParsedRequest(Http::Stream *context);
+ virtual void processParsedRequest(Http::StreamPointer &context);
virtual void handleReply(HttpReply *rep, StoreIOBuffer receivedData);
virtual void writeControlMsgAndCall(HttpReply *rep, AsyncCall::Pointer &call);
virtual time_t idleTimeout() const;
/* BodyPipe API */
virtual void noteMoreBodySpaceAvailable(BodyPipe::Pointer);
virtual void noteBodyConsumerAborted(BodyPipe::Pointer);
/* AsyncJob API */
virtual void start();
void proceedAfterBodyContinuation(Http::StreamPointer context);
private:
void processHttpRequest(Http::Stream *const context);
void handleHttpRequestData();
/// Handles parsing results. May generate and deliver an error reply
/// to the client if parsing is failed, or parses the url and build the
/// HttpRequest object using parsing results.
/// Return false if parsing is failed, true otherwise.
- bool buildHttpRequest(Http::Stream *context);
+ bool buildHttpRequest(Http::StreamPointer &context);
+
+ void setReplyError(Http::StreamPointer &context, HttpRequest::Pointer &request, const HttpRequestMethod& method, err_type requestError, Http::StatusCode errStatusCode, const char *requestErrorBytes);
Http1::RequestParserPointer parser_;
HttpRequestMethod method_; ///< parsed HTTP method
/// temporary hack to avoid creating a true HttpsServer class
const bool isHttpsServer;
};
} // namespace One
} // namespace Http
#endif /* SQUID_SRC_SERVERS_HTTP1SERVER_H */
=== modified file 'src/tunnel.cc'
--- src/tunnel.cc 2016-08-17 14:24:07 +0000
+++ src/tunnel.cc 2016-10-20 14:30:02 +0000
@@ -51,77 +51,81 @@
#include <cerrno>
/**
* TunnelStateData is the state engine performing the tasks for
* setup of a TCP tunnel from an existing open client FD to a server
* then shuffling binary data between the resulting FD pair.
*/
/*
* TODO 1: implement a read/write API on ConnStateData to send/receive blocks
* of pre-formatted data. Then we can use that as the client side of the tunnel
* instead of re-implementing it here and occasionally getting the ConnStateData
* read/write state wrong.
*
* TODO 2: then convert this into a AsyncJob, possibly a child of 'Server'
*/
class TunnelStateData
{
CBDATA_CLASS(TunnelStateData);
public:
- TunnelStateData();
+ TunnelStateData(ClientHttpRequest *);
~TunnelStateData();
TunnelStateData(const TunnelStateData &); // do not implement
TunnelStateData &operator =(const TunnelStateData &); // do not implement
class Connection;
static void ReadClient(const Comm::ConnectionPointer &, char *buf, size_t len, Comm::Flag errcode, int xerrno, void *data);
static void ReadServer(const Comm::ConnectionPointer &, char *buf, size_t len, Comm::Flag errcode, int xerrno, void *data);
static void WriteClientDone(const Comm::ConnectionPointer &, char *buf, size_t len, Comm::Flag flag, int xerrno, void *data);
static void WriteServerDone(const Comm::ConnectionPointer &, char *buf, size_t len, Comm::Flag flag, int xerrno, void *data);
/// Starts reading peer response to our CONNECT request.
void readConnectResponse();
/// Called when we may be done handling a CONNECT exchange with the peer.
void connectExchangeCheckpoint();
bool noConnections() const;
char *url;
CbcPointer<ClientHttpRequest> http;
HttpRequest::Pointer request;
AccessLogEntryPointer al;
Comm::ConnectionList serverDestinations;
const char * getHost() const {
return (server.conn != NULL && server.conn->getPeer() ? server.conn->getPeer()->host : request->url.host());
};
/// Whether we are writing a CONNECT request to a peer.
bool waitingForConnectRequest() const { return connectReqWriting; }
/// Whether we are reading a CONNECT response from a peer.
bool waitingForConnectResponse() const { return connectRespBuf; }
/// Whether we are waiting for the CONNECT request/response exchange with the peer.
bool waitingForConnectExchange() const { return waitingForConnectRequest() || waitingForConnectResponse(); }
/// Whether the client sent a CONNECT request to us.
bool clientExpectsConnectResponse() const {
+ // If we are forcing a tunnel after receiving a client CONNECT, then we
+ // have already responded to that CONNECT before tunnel.cc started.
+ if (request && request->flags.forceTunnel)
+ return false;
#if USE_OPENSSL
// We are bumping and we had already send "OK CONNECTED"
if (http.valid() && http->getConn() && http->getConn()->serverBump() && http->getConn()->serverBump()->step > Ssl::bumpStep1)
return false;
#endif
return !(request != NULL &&
(request->flags.interceptTproxy || request->flags.intercepted));
}
/// Sends "502 Bad Gateway" error response to the client,
/// if it is waiting for Squid CONNECT response, closing connections.
void informUserOfPeerError(const char *errMsg, size_t);
class Connection
{
public:
Connection() : len (0), buf ((char *)xmalloc(SQUID_TCP_SO_RCVBUF)), size_ptr(NULL), delayedLoops(0),
readPending(NULL), readPendingFunc(NULL) {}
@@ -268,53 +272,65 @@
tunnelState->client.writer = NULL;
if (tunnelState->noConnections()) {
// ConnStateData pipeline should contain the CONNECT we are performing
// but it may be invalid already (bug 4392)
if (tunnelState->http.valid() && tunnelState->http->getConn()) {
auto ctx = tunnelState->http->getConn()->pipeline.front();
if (ctx != nullptr)
ctx->finished();
}
delete tunnelState;
return;
}
if (!tunnelState->server.writer) {
tunnelState->server.conn->close();
return;
}
}
-TunnelStateData::TunnelStateData() :
- url(NULL),
- http(),
- request(NULL),
- status_ptr(NULL),
- logTag_ptr(NULL),
+TunnelStateData::TunnelStateData(ClientHttpRequest *clientRequest) :
connectRespBuf(NULL),
connectReqWriting(false),
started(squid_curtime)
{
debugs(26, 3, "TunnelStateData constructed this=" << this);
client.readPendingFunc = &tunnelDelayedClientRead;
server.readPendingFunc = &tunnelDelayedServerRead;
+
+ assert(clientRequest);
+ url = xstrdup(clientRequest->uri);
+ request = clientRequest->request;
+ server.size_ptr = &clientRequest->out.size;
+ client.size_ptr = &clientRequest->al->http.clientRequestSz.payloadData;
+ status_ptr = &clientRequest->al->http.code;
+ logTag_ptr = &clientRequest->logType;
+ al = clientRequest->al;
+ http = clientRequest;
+
+ client.conn = clientRequest->getConn()->clientConnection;
+ comm_add_close_handler(client.conn->fd, tunnelClientClosed, this);
+
+ AsyncCall::Pointer timeoutCall = commCbCall(5, 4, "tunnelTimeout",
+ CommTimeoutCbPtrFun(tunnelTimeout, this));
+ commSetConnTimeout(client.conn, Config.Timeout.lifetime, timeoutCall);
}
TunnelStateData::~TunnelStateData()
{
debugs(26, 3, "TunnelStateData destructed this=" << this);
assert(noConnections());
xfree(url);
serverDestinations.clear();
delete connectRespBuf;
}
TunnelStateData::Connection::~Connection()
{
if (readPending)
eventDelete(readPendingFunc, readPending);
safe_free(buf);
}
int
@@ -1058,62 +1074,44 @@
/*
* Check if this host is allowed to fetch MISSES from us (miss_access)
* default is to allow.
*/
ACLFilledChecklist ch(Config.accessList.miss, request, NULL);
ch.src_addr = request->client_addr;
ch.my_addr = request->my_addr;
if (ch.fastCheck() == ACCESS_DENIED) {
debugs(26, 4, HERE << "MISS access forbidden.");
err = new ErrorState(ERR_FORWARDING_DENIED, Http::scForbidden, request);
http->al->http.code = Http::scForbidden;
errorSend(http->getConn()->clientConnection, err);
return;
}
}
debugs(26, 3, request->method << ' ' << url << ' ' << request->http_ver);
++statCounter.server.all.requests;
++statCounter.server.other.requests;
- tunnelState = new TunnelStateData;
+ tunnelState = new TunnelStateData(http);
#if USE_DELAY_POOLS
- tunnelState->server.setDelayId(DelayId::DelayClient(http));
+ //server.setDelayId called from tunnelConnectDone after server side connection established
#endif
- tunnelState->url = xstrdup(url);
- tunnelState->request = request;
- tunnelState->server.size_ptr = &http->out.size;
- tunnelState->client.size_ptr = &http->al->http.clientRequestSz.payloadData;
- tunnelState->status_ptr = &http->al->http.code;
- tunnelState->logTag_ptr = &http->logType;
- tunnelState->client.conn = http->getConn()->clientConnection;
- tunnelState->http = http;
- tunnelState->al = http->al;
- //tunnelState->started is set in TunnelStateData ctor
-
- comm_add_close_handler(tunnelState->client.conn->fd,
- tunnelClientClosed,
- tunnelState);
-
- AsyncCall::Pointer timeoutCall = commCbCall(5, 4, "tunnelTimeout",
- CommTimeoutCbPtrFun(tunnelTimeout, tunnelState));
- commSetConnTimeout(tunnelState->client.conn, Config.Timeout.lifetime, timeoutCall);
peerSelect(&(tunnelState->serverDestinations), request, http->al,
NULL,
tunnelPeerSelectComplete,
tunnelState);
}
void
TunnelStateData::connectToPeer()
{
if (CachePeer *p = server.conn->getPeer()) {
if (p->secure.encryptTransport) {
AsyncCall::Pointer callback = asyncCall(5,4,
"TunnelStateData::ConnectedToPeer",
MyAnswerDialer(&TunnelStateData::connectedToPeer, this));
auto *connector = new Security::BlindPeerConnector(request, server.conn, callback, al);
AsyncJob::Start(connector); // will call our callback
return;
}
}
@@ -1166,47 +1164,74 @@
CommIoCbPtrFun(tunnelConnectReqWriteDone,
tunnelState));
tunnelState->server.write(mb.buf, mb.size, writeCall, mb.freeFunc());
tunnelState->connectReqWriting = true;
tunnelState->connectRespBuf = new MemBuf;
// SQUID_TCP_SO_RCVBUF: we should not accumulate more than regular I/O buffer
// can hold since any CONNECT response leftovers have to fit into server.buf.
// 2*SQUID_TCP_SO_RCVBUF: HttpMsg::parse() zero-terminates, which uses space.
tunnelState->connectRespBuf->init(SQUID_TCP_SO_RCVBUF, 2*SQUID_TCP_SO_RCVBUF);
tunnelState->readConnectResponse();
assert(tunnelState->waitingForConnectExchange());
AsyncCall::Pointer timeoutCall = commCbCall(5, 4, "tunnelTimeout",
CommTimeoutCbPtrFun(tunnelTimeout, tunnelState));
commSetConnTimeout(srv, Config.Timeout.read, timeoutCall);
}
+static Comm::ConnectionPointer
+borrowPinnedConnection(HttpRequest *request, Comm::ConnectionPointer &serverDestination)
+{
+ // pinned_connection may become nil after a pconn race
+ if (ConnStateData *pinned_connection = request ? request->pinnedConnection() : nullptr) {
+ Comm::ConnectionPointer serverConn = pinned_connection->borrowPinnedConnection(request, serverDestination->getPeer());
+ return serverConn;
+ }
+
+ return nullptr;
+}
+
static void
tunnelPeerSelectComplete(Comm::ConnectionList *peer_paths, ErrorState *err, void *data)
{
TunnelStateData *tunnelState = (TunnelStateData *)data;
+ bool bail = false;
if (peer_paths == NULL || peer_paths->size() < 1) {
debugs(26, 3, HERE << "No paths found. Aborting CONNECT");
+ bail = true;
+ }
+
+ if (!bail && tunnelState->serverDestinations[0]->peerType == PINNED) {
+ Comm::ConnectionPointer serverConn = borrowPinnedConnection(tunnelState->request.getRaw(), tunnelState->serverDestinations[0]);
+ debugs(26,7, "pinned peer connection: " << serverConn);
+ if (Comm::IsConnOpen(serverConn)) {
+ tunnelConnectDone(serverConn, Comm::OK, 0, (void *)tunnelState);
+ return;
+ }
+ bail = true;
+ }
+
+ if (bail) {
if (!err) {
err = new ErrorState(ERR_CANNOT_FORWARD, Http::scServiceUnavailable, tunnelState->request.getRaw());
}
*tunnelState->status_ptr = err->httpStatus;
err->callback = tunnelErrorComplete;
err->callback_data = tunnelState;
errorSend(tunnelState->client.conn, err);
return;
}
delete err;
GetMarkingsToServer(tunnelState->request.getRaw(), *tunnelState->serverDestinations[0]);
if (tunnelState->request != NULL)
tunnelState->request->hier.startPeerClock();
debugs(26, 3, HERE << "paths=" << peer_paths->size() << ", p[0]={" << (*peer_paths)[0] << "}, serverDest[0]={" <<
tunnelState->serverDestinations[0] << "}");
AsyncCall::Pointer call = commCbCall(26,3, "tunnelConnectDone", CommConnectCbPtrFun(tunnelConnectDone, tunnelState));
@@ -1220,96 +1245,78 @@
bool
TunnelStateData::noConnections() const
{
return !Comm::IsConnOpen(server.conn) && !Comm::IsConnOpen(client.conn);
}
#if USE_DELAY_POOLS
void
TunnelStateData::Connection::setDelayId(DelayId const &newDelay)
{
delayId = newDelay;
}
#endif
#if USE_OPENSSL
void
switchToTunnel(HttpRequest *request, Comm::ConnectionPointer &clientConn, Comm::ConnectionPointer &srvConn)
{
debugs(26,5, "Revert to tunnel FD " << clientConn->fd << " with FD " << srvConn->fd);
- /* Create state structure. */
- const SBuf url(request->effectiveRequestUri());
- debugs(26, 3, request->method << " " << url << " " << request->http_ver);
+ /* Create state structure. */
++statCounter.server.all.requests;
++statCounter.server.other.requests;
- TunnelStateData *tunnelState = new TunnelStateData;
- tunnelState->url = SBufToCstring(url);
- tunnelState->request = request;
- tunnelState->server.size_ptr = NULL; //Set later if Http::Stream is available
-
- // Temporary static variable to store the unneeded for our case status code
- static int status_code = 0;
- tunnelState->status_ptr = &status_code;
- tunnelState->client.conn = clientConn;
-
- if (auto conn = request->clientConnectionManager.get()) {
- Http::StreamPointer context = conn->pipeline.front();
- if (context && context->http) {
- tunnelState->logTag_ptr = &context->http->logType;
- tunnelState->server.size_ptr = &context->http->out.size;
- tunnelState->al = context->http->al;
+ auto conn = request->clientConnectionManager.get();
+ Must(conn);
+ Http::StreamPointer context = conn->pipeline.front();
+ Must(context && context->http);
-#if USE_DELAY_POOLS
- /* no point using the delayIsNoDelay stuff since tunnel is nice and simple */
- if (srvConn->getPeer() && srvConn->getPeer()->options.no_delay)
- tunnelState->server.setDelayId(DelayId::DelayClient(context->http));
-#endif
- }
- }
+ debugs(26, 3, request->method << " " << context->http->uri << " " << request->http_ver);
- comm_add_close_handler(tunnelState->client.conn->fd,
- tunnelClientClosed,
- tunnelState);
+ TunnelStateData *tunnelState = new TunnelStateData(context->http);
- AsyncCall::Pointer timeoutCall = commCbCall(5, 4, "tunnelTimeout",
- CommTimeoutCbPtrFun(tunnelTimeout, tunnelState));
- commSetConnTimeout(tunnelState->client.conn, Config.Timeout.lifetime, timeoutCall);
fd_table[clientConn->fd].read_method = &default_read_method;
fd_table[clientConn->fd].write_method = &default_write_method;
request->hier.note(srvConn, tunnelState->getHost());
tunnelState->server.conn = srvConn;
+
+#if USE_DELAY_POOLS
+ /* no point using the delayIsNoDelay stuff since tunnel is nice and simple */
+ if (srvConn->getPeer() && srvConn->getPeer()->options.no_delay)
+ tunnelState->server.setDelayId(DelayId::DelayClient(context->http));
+#endif
+
request->peer_host = srvConn->getPeer() ? srvConn->getPeer()->host : nullptr;
comm_add_close_handler(srvConn->fd, tunnelServerClosed, tunnelState);
debugs(26, 4, "determine post-connect handling pathway.");
if (srvConn->getPeer()) {
request->peer_login = srvConn->getPeer()->login;
request->peer_domain = srvConn->getPeer()->domain;
request->flags.auth_no_keytab = srvConn->getPeer()->options.auth_no_keytab;
request->flags.proxying = !(srvConn->getPeer()->options.originserver);
} else {
request->peer_login = nullptr;
request->peer_domain = nullptr;
request->flags.auth_no_keytab = false;
request->flags.proxying = false;
}
- timeoutCall = commCbCall(5, 4, "tunnelTimeout",
- CommTimeoutCbPtrFun(tunnelTimeout, tunnelState));
+ AsyncCall::Pointer timeoutCall = commCbCall(5, 4, "tunnelTimeout",
+ CommTimeoutCbPtrFun(tunnelTimeout, tunnelState));
commSetConnTimeout(srvConn, Config.Timeout.read, timeoutCall);
fd_table[srvConn->fd].read_method = &default_read_method;
fd_table[srvConn->fd].write_method = &default_write_method;
auto ssl = fd_table[srvConn->fd].ssl.get();
assert(ssl);
BIO *b = SSL_get_rbio(ssl);
Ssl::ServerBio *srvBio = static_cast<Ssl::ServerBio *>(b->ptr);
tunnelState->preReadServerData = srvBio->rBufData();
tunnelStartShoveling(tunnelState);
}
#endif //USE_OPENSSL
_______________________________________________
squid-dev mailing list
squid-dev@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-dev
