subject:"\[jira\] \[Commented\] $HADOOP\-9421$ Convert SASL to use ProtoBuf and add lengths for non\-blocking processing"


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13690066#comment-13690066
 ] 

Daryn Sharp commented on HADOOP-9421:
-

bq. If we have to replace Digest-MD5 for security reasons, we'll be SOL.

That's completely untrue.  There is nothing in the protocol that would prevent 
SCRAM being supported.

{noformat}
C - S connectionHeader(SASL)
C - S NEGOTIATE { [TOKEN, SCRAM, proto, serverId], ... }
C - S INITIATE [TOKEN] initial-response
{noformat}

bq. I merely want to leave the optional client initiate proto in the protocol 
for future optimizations

In light of everything I've described, please detail what future optimization 
is possible?

Please answer, how is the client capable of:
* Guessing a supported auth
* Guessing the supported mechanism for guessed auth
* Based on those guesses, reliably creating a SASL client to generate a SASL 
response
* Dealing with the mishaps when the client blows itself up trying an auth the 
server doesn't even support

Notably, describe how you would handle the problems I detailed regarding a 
client failing if it even attempts kerberos with a non-kerberos server.  It 
won't even succeed far enough to send the INITIATE.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

[
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13690136#comment-13690136
]

Luke Lu commented on HADOOP-9421:
-

bq. There is nothing in the protocol that would prevent SCRAM being supported.

I meant you'll be SOL to make the token optimization work. Your protocol
*requires* an extra round trip to support SCRAM.

bq. Guessing a supported auth/mechanism

For most common hadoop auth work load: distributed containers/tasks, you don't
need to guess, it's the delegation token with digest-md5/scram, as it's a
framework internal token bootstrapped by other public facing mechanisms. For
other use cases, you can use cached values. For the remaining use cases, the
client can send an empty INITIATE and use NEGOTIATE and REINITIATE with the
same total round-trip cost as yours in all cases. With the optional client
initiate, my protocol gives the choices to the practicing system designers
instead of the original protocol designers.

bq. Dealing with the mishaps when the client blows itself up trying an auth the
server doesn't even support

INITIATE contains the same extra info like protocol and serverId for preferred
auth. Server can simply send a NEGOTIATE if it decides that it cannot support
the preferred auth choice, the client can then decide to REINITIATE or abort.

bq. if it even attempts kerberos with a non-kerberos server. It won't even
succeed far enough to send the INITIATE

For this contrived case, the client can catch exceptions for the preferred auth
when generating the initial token, which would apply to fetching service ticket
for non-kerberos server, and send an empty INITIATE to NEGOTIATE and
REINITIATE. Again for integration clients that need to talk to multiple servers
with different auths it can simply use empty INITIATE to NEGOTIATE and cache
the server auth/mechs for later use to save a round-trip later.

Imagine a busy interactive web console that talk to multiple back-end Hadoop
servers, it's not feasible to keep a connection per user open to all these
servers, you often need to constantly creating new connections to the back-end
servers (a connection pool helps), my protocol allows the web console to save a
mandatory round-trip from yours, which can make the interactive user experience
much better due to lowered latencies.

In summary, my protocol gives that choice to real system designers. Your
protocol takes away that choice because you could not possibly think of all use
cases, where auth latency matters.

Convert SASL to use ProtoBuf and add lengths for non-blocking processing

Key: HADOOP-9421
URL: https://issues.apache.org/jira/browse/HADOOP-9421
Project: Hadoop Common
Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch,
HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch,
HADOOP-9421.patch, HADOOP-9421-v2-demo.patch

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

[
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13690241#comment-13690241
]

Daryn Sharp commented on HADOOP-9421:
-

bq. I meant you'll be SOL to make the token optimization work. Your protocol
requires an extra round trip to support SCRAM.

Ok, so now how would you handle improved serverId based tokens? Always take
the REINITIATE hit?

bq. For most common hadoop auth work load: distributed containers/tasks, you
don't need to guess, it's the delegation token with digest-md5/scram

In a mixed security env including not just within the cluster but across
different clusters with different versions, or during rolling upgrades when
supported mechanisms are upgraded, it will always be a guess. You'll always
have to do the REINITIATE?

bq. For this contrived case, the client can catch exceptions for the preferred
auth when generating the initial token, which would apply to fetching a service
ticket for non-kerberos server

In mixed security envs, or rolling upgrades, or use of IP failover are a
contrived use case? In all these cases, we'll also take the penalty of an
unnecessary roundtrip to the KDC for a non-existent service ticket?

bq. cache the server auth/mechs for later use to save a round-trip later
In this case, can't the client just blast the INITIATE before getting a
NEGOTIATE?

Actually, the client receiving a NEGOTIATE will allow the client to know in
advance of the server reply whether its guess will fail and it can begin
preparing the proper auth.

bq. In summary, my protocol gives that choice to real system designers. Your
protocol takes away that choice because you could not possibly think of all use
cases, where auth latency matters.

As I asked before, do you believe there will actually be a measurable
performance difference? Will having the client ignore the inflight NEGOTIATE
on a reconnect have a measurable latency? If so, is the extra round trip for
REINITIATEs a bad thing?

Convert SASL to use ProtoBuf and add lengths for non-blocking processing

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13690351#comment-13690351
 ] 

Daryn Sharp commented on HADOOP-9421:
-

Although I don't see a scenario in which this optimization is possible, I'll 
just add it.  I'll re-leverage the existing states to avoid adding the 
reinitate, and prevent a client seizing a connection thread.

With tcp nodelay, the two messages aren't combined, but I think that should be 
easy to fix.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

[
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13690496#comment-13690496
]

Luke Lu commented on HADOOP-9421:
-

bq. Ok, so now how would you handle improved serverId based tokens? Always take
the REINITIATE hit?

Maintain a persistent (serverId - mech) cache in /tmp/hadoop-$user/rpc. Again:
it should be choice of protocol users (system designers).

bq. In this case, can't the client just blast the INITIATE before getting a
NEGOTIATE?

Like I said before, ignoring NEGOTIATE in client increase client complexity to
ignore the right NEGOTIATE. What if server upgrade/security fix happened in
between?

bq. do you believe there will actually be a measurable performance difference?

Depending on the use case, e.g. a console managing multiple hadoop clusters in
multiple regions: the additional round trip could be in hundreds of ms! which
is extremely annoying.

bq. Will having the client ignore the inflight NEGOTIATE on a reconnect have a
measurable latency

Like I said, client will have to decide when to ignore the NEGOTIATE, which
increase client complexity while still feel dirty. I've shown that this
ugliness is not necessary.

bq. If so, is the extra round trip for REINITIATEs a bad thing

REINITIATE is only supposed to happen when mech cache goes stale.

bq. I'll just add it. I'll re-leverage the existing states to avoid adding the
reinitate.

IMO, explicit REINITIATE make things a lot easier to reason about.

bq. prevent a client seizing a connection thread.

What're you talking about? The new protocol allows completely non-blocking
client.

Convert SASL to use ProtoBuf and add lengths for non-blocking processing

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13690510#comment-13690510
 ] 

Luke Lu commented on HADOOP-9421:
-

bq. I'll just add it.

Thank you! It's important to note that we don't need to add the optimizations 
for this jira, which should focus on fixing the protocol. Once the protocol is 
fixed, we can explore new non-blocking client in different languages and enable 
a Cambrian explosion of new use cases! 

bq. REINITIATE is only supposed to happen when mech cache goes stale.

s/only//. System designer can make it happen rarely when mech cache is empty or 
stale and when serverId failover happens.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

[
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13690513#comment-13690513
]

Daryn Sharp commented on HADOOP-9421:
-

It's really not worth debating anymore, but if the client decided to guess and
blast a INITIATE, it could simply set a ignoreNegotiate boolean. A server
upgrade is exactly one of the reasons the client shouldn't be guessing.

During authentication the socket reader isn't fair. As long as there is
pending data from the client, it will keep reading until it gets the first RPC.
It's not a problem today because the client is forced along a path that either
ends in the connection being aborted, or the reading of an RPC. Your patch
would allow a client to spam INITIATE and keep the socket tied up indefinitely.

New patch arriving shortly...

Convert SASL to use ProtoBuf and add lengths for non-blocking processing

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13690539#comment-13690539
 ] 

Luke Lu commented on HADOOP-9421:
-

bq. if the client decided to guess and blast a INITIATE, it could simply set a 
ignoreNegotiate boolean.

When failover/upgrade happens, the INITIATE can fail and potentially two 
NEGOTIATE in row. You need to pick the right one. I think this increase client 
complexity unnecessarily.

bq. A server upgrade is exactly one of the reasons the client shouldn't be 
guessing.

Or one the reasons to REINTIATE? :)


 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

[
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13690558#comment-13690558
]

Luke Lu commented on HADOOP-9421:
-

bq. Server will not SASL respond to a client until it sends either NEGOTIATE or
INITIATE.

OK, use client NEGOTIATE to replace my empty INITIATE. That's cool.

bq. If the client sends INITIATE, and guesses wrong, the server responds with
NEGOTIATE. Again, the client now has one shot to send a valid INITIATE.

I believe the code could get tricky without a REINITIATE.

bq. Your patch would allow a client to spam INITIATE and keep the socket tied
up indefinitely.

My patch only sends one INITIATE after connection header, it either succeeds or
get a NEGOTIATE to REINITIATE, which doesn't have a transition to NEGOTIATE
again. The code is simple and succinct.

Convert SASL to use ProtoBuf and add lengths for non-blocking processing

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13690585#comment-13690585
 ] 

Luke Lu commented on HADOOP-9421:
-

I'm fine with the new protocol. I agree that REINITIATE is not necessary if 
server keeps the sentNegotiate state for the connection.

One bug in the patch: in Server case INITIATE, in the authMethod == null, the 
if (sentNegotiate) should be if (!sentNegotiate)...

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

2013-06-21 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13690593#comment-13690593
 ] 

Hadoop QA commented on HADOOP-9421:
---

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12589114/HADOOP-9421.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-common-project/hadoop-common.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/2690//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/2690//console

This message is automatically generated.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13690600#comment-13690600
 ] 

Luke Lu commented on HADOOP-9421:
-

In light of the bug, can you add a unit test for the 
INITIATE/NEGOTIATE/INITIATE path?

The sentNegotiate state will also need to be kept per flow if you want to 
multiplex the connection.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13690604#comment-13690604
 ] 

Daryn Sharp commented on HADOOP-9421:
-

That's not a bug.  The logic is if the client sent an INITIATE which was a bad 
guess, only throw an exception if it has already been sent a NEGOTIATE.  It's 
what allows you desire for the client to guess.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13690626#comment-13690626
 ] 

Luke Lu commented on HADOOP-9421:
-

You're right, I somehow read that it is trying to buildNegotiateResponse in 
that branch.

Though the patch always sends client NEGOTIATE, which means a slight 
performance regression on typical clusters, I think the protocol is in a good 
shape now.

Can you add some unit tests, as it's easy to break the code due to all the 
states? Cluster testing results would be appreciated as well.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13690638#comment-13690638
 ] 

Daryn Sharp commented on HADOOP-9421:
-

We agreed that guessing wasn't going to be implemented.  There's no penalty for 
token auth because of my other optimization.  Kerberos takes a tiny penalty but 
it's necessary to support subsequent IP failover changes.

My existing SASL unit tests exercise the full matrix of client and server 
auths, both with and without tokens.  Writing tests all the way down to the 
wire is going to further delay this patch.  I think Arun is pretty eager to get 
this in.  If we want wire tests, then I think we should defer to another jira 
and get 2.1 unblocked.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13690639#comment-13690639
 ] 

Daryn Sharp commented on HADOOP-9421:
-

Oh, and I did manually test on a secure cluster, using all combinations of 
secure/insecure server + client using kerberos/simple/tokens.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13690650#comment-13690650
 ] 

Luke Lu commented on HADOOP-9421:
-

Filed HADOOP-9662 for more unit tests.

+1 for the patch to unblock 2.1 scale testing.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13688991#comment-13688991
 ] 

Luke Lu commented on HADOOP-9421:
-

bq. As described, optimize token path.

[Your 
patch|https://issues.apache.org/jira/secure/attachment/12588738/HADOOP-9421.patch]
 still doesn't have the proper client initiation support:
# Only works with token auths that use digest-md5, will require major protocol 
change to optimize for SCRAM (modern digest-md5 replacement) or Kerberos and 
anything SASL mechanisms that hasInitialResponse.
# fallback prevention seems broken as the patch didn't modify 
TestSaslRpc#testSimpleServerWith*Token and still pass these tests.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

2013-06-20 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689025#comment-13689025
 ] 

Hadoop QA commented on HADOOP-9421:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12588781/HADOOP-9421.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-common-project/hadoop-common:

  org.apache.hadoop.ipc.TestSaslRPC

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/2682//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/2682//console

This message is automatically generated.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

2013-06-20 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689130#comment-13689130
 ] 

Hadoop QA commented on HADOOP-9421:
---

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12588814/HADOOP-9421.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-common-project/hadoop-common.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/2684//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/2684//console

This message is automatically generated.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

[
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689208#comment-13689208
]

Daryn Sharp commented on HADOOP-9421:
-

bq. Only works with token auths that use digest-md5, will require major
protocol change to optimize for SCRAM (modern digest-md5 replacement) or
Kerberos and anything SASL mechanisms that hasInitialResponse.

A major protocol change will not be required for other auths. The client is
properly coded to handle the server providing an initial challenge for any
auth, but the server currently only does it for tokens. When the server auths
become extensible, additional initial challenges can be added w/o changing the
client. Ie. It's forward compatible.

I did not generate an initial challenge for kerberos because the SASL mechanism
does not support it. An exception is thrown if you try.

This is intended to be a minimal change to provide a base implementation for
future work. I thought everybody would be satisfied by removal of an existing
round trip to offset the negotiate response?

I'll look at your modifications to the patch.

Convert SASL to use ProtoBuf and add lengths for non-blocking processing

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689283#comment-13689283
 ] 

Daryn Sharp commented on HADOOP-9421:
-

I don't understand the advantage of this patch.  At a minimum, here are major 
problems.
* Re-introduces the roundtrip I removed for tokens and usable by other auths in 
the future
* Appears to add yet another roundtrip for non-token auths
* Completely removes the ability for the client to chose the best or most 
preferred auth
* Ruins pluggable auths because the client now requires specific logic to 
guess if it can do the new auth
* Prevents elimination of token use_ip
* Prevents supporting tokens for multi-interface, multi-A record, or CNAMEs
* Breaks my ability to add IP failover support
* Allows clients to do complete DOS attacks by tying up the socket indefinitely 
with initiates

Given my correction of the misunderstandings of my prior patch, what are the 
disadvantages?

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

[
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689336#comment-13689336
]

Luke Lu commented on HADOOP-9421:
-

bq. I don't understand the advantage of this patch. At a minimum, here are
major problems.

I think you misunderstood the patch and the current flow as well. In the
[Current
flow|https://issues.apache.org/jira/browse/HADOOP-9421?focusedCommentId=13688055page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13688055],
step 1 and 2 is in the same C-S packet.

My patch is an improvement over yours in that challenge or negotiation is not
done blindly. It can support SCRAM/Kerberos round-trip reduction with no
protocol branching while yours cannot.

Conceptually I just replace the step 2 (still part of the connection header
packet) with wrapped RpcSaslProto to allow future extensibility. So my patch
feature wise is a superset of yours.

Convert SASL to use ProtoBuf and add lengths for non-blocking processing

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

[
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689451#comment-13689451
]

Luke Lu commented on HADOOP-9421:
-

Specifically:

bq. Re-introduces the roundtrip I removed for tokens and usable by other auths
in the future

Not true. The initiate proto wrapped in rpc header is part of the connection
header packet even though they're logically separate messages.

bq. Appears to add yet another roundtrip for non-token auths

Not true. See above.

bq. Completely removes the ability for the client to chose the best or most
preferred auth

Not true. In fact the client initiate proto allows future auths without
introduce new round-trip.

bq. Ruins pluggable auths because the client now requires specific logic to
guess if it can do the new auth

Not true. initiate allows but not requires client specific logic, hence
more extensible.

bq. Prevents elimination of token use_ip

Not applicable to any known token mechs: Digest-MD5 or SCRAM, as the former is
always server initiated and the latter doesn't care.

bq. Prevents supporting tokens for multi-interface, multi-A record, or CNAMEs

Not true. Token auth don't care, see above. And the initiate proto is
extensible for all kinds of auth metadata.

bq. Breaks my ability to add IP failover support

Not true. IP failover works with tokens as is and for Kerberos if server
principal is shared among the servers for the same logical server. Can be
extended to support insane cross server principal failover, while maintaining
minimum round-trips in normal cases.

bq. Allows clients to do complete DOS attacks by tying up the socket
indefinitely with initiates

Clients can already do the same by keeping RPC connections indefinitely. DoS is
only significant if it requires client less resource to DoS a server, which is
not the case.

In summary, your patch changes the major flow of the current RPC with a new
negotiate round-trip except for a round-trip reduction hack for Digest-MD5
tokens, since it disallows client to send any new auth metadata in the first
packet. My patch is actually a (conceptually) small change to extend the
capability to send arbitrary auth metadata in the first packet and allows
server to intelligently respond with either challenge or negotiate, which
allows round-trip optimization for all future auths besides Digest-MD5 tokens.

Convert SASL to use ProtoBuf and add lengths for non-blocking processing

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

[
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689490#comment-13689490
]

Daryn Sharp commented on HADOOP-9421:
-

I'm referring to the roundtrip your patch introduces by responding with
negotiate if it's a non-token auth.

The client can't chose the best auth, or even know the supported auths, if it's
already guessed prior to connection. How will the client know whether the
server does DIGEST-MD5 or SCRAM for tokens? It won't work in a mixed
environment.

Eliminating use_ip is not related to the mech. A server hint is for the token
selection itself instead of the fragile way tokens are currently selected.
Tokens are completely sensitive to multi-interface hosts, and different
hostnames for the same machine.

IP failover with a shared principal isn't an option, at least for us. A shared
principal prevents direct communication with the HA NNs because the client will
use the actual host's principal, not the shared principal. Which also means
DNs can't heartbeat into both NNs w/o hardcoding in the config, which may be
problematic for federation + HA.

The roundtrip reduction hack is a feature that can be extended to any sasl
mechanism that can initiate.

The point you keep missing is +the client can't guess an auth method+ but you
keep focusing on retaining that behavior. We need to resolve this with the
offline call today.

Convert SASL to use ProtoBuf and add lengths for non-blocking processing

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

[
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689619#comment-13689619
]

Luke Lu commented on HADOOP-9421:
-

bq. I'm referring to the roundtrip your patch introduces by responding with
negotiate if it's a non-token auth.

The roundtrip is mandatory for non-token auth in your patch. It's optional in
mine.

bq. The client can't chose the best auth

Sometimes it's the only choice, eg. delegation token for tasks, which covers
99% of the auth workload in typical clusters.

bq. The roundtrip reduction hack is a feature that can be extended to any
sasl mechanism that can initiate.

No it cannot reduce round-trip for SCRAM or Kerberos, while my patch can.

bq. The point you keep missing is the client can't guess an auth method but you
keep focusing on retaining that behavior.

The point you keep missing is that the client initiate proto is free and
optional from network round-trip point of view, which opens door for future
optimization for SCRAM etc.

Convert SASL to use ProtoBuf and add lengths for non-blocking processing

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

2013-06-20 Thread Sanjay Radia (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689739#comment-13689739
 ] 

Sanjay Radia commented on HADOOP-9421:
--

bq.  The client can't chose the best auth

bq.Sometimes it's the only choice, eg. delegation token for tasks, which covers 
99% of the auth workload in typical clusters.
That is correct. BTW Client can guess the best default  for initial auth since 
it is the default authentication for the Cluster (e.g. Kerberos or LDAP).

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

2013-06-20 Thread Sanjay Radia (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689759#comment-13689759
 ] 

Sanjay Radia commented on HADOOP-9421:
--

Luke, Daryn can both of you please summarize the packet exchange for simple, 
kerberos, and hadoop-tokens for each of your approaches. ALso the exchange when 
server does not support auth-method X initiated by client and the client is 
suppose to use another auth-method from a list. Lets first get an agreement on 
the exchange.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

[
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689763#comment-13689763
]

Daryn Sharp commented on HADOOP-9421:
-

bq. That is correct. BTW Client can guess the best default for initial auth
since it is the default authentication for the Cluster (e.g. Kerberos or LDAP).

Wrong. You can't guess in a heterogeneous security environment for different
services. You can't even guess which mechanism a particular service will use
for tokens.

How will you simultaneously support accessing both LDAP and KERBEROS secured
services? Or something pluggable like a SSO_TOKEN on one cluster, normal
TOKEN on another. And either of them might be using DIGEST-MD5 or SCRAM or
something else.

How will you guess w/o a slew of config options?

Convert SASL to use ProtoBuf and add lengths for non-blocking processing

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

2013-06-20 Thread Larry McCay (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689774#comment-13689774
 ] 

Larry McCay commented on HADOOP-9421:
-

Just a thought...
I have recently found the use of plantuml very useful for these sorts of
discussions. Sequence diagrams may go a long way and they are refreshingly
easy with that tool.



 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

[
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689823#comment-13689823
]

Daryn Sharp commented on HADOOP-9421:
-

+Simple to insecure+
{noformat}
C - S connectHeader(SIMPLE)
C - S connectionContext
C - S RPC request
{noformat}

No change/overhead.

+Simple to secure+
{noformat}
C - S connectHeader(SIMPLE)
C - S connectionContext
C - S RPC request
C - S RPC exception SIMPLE not supported, close connection
{noformat}

No change/overhead.

Note: There's an existing race condition. The server sent the exception in
response to the connect header, but the client already blasted the context and
request.

+SASL to insecure server+
{noformat}
C - S connectHeader(SASL)
C - S SUCCESS
C - S connectionContext
C - S RPC request
{noformat}

Immediate success message replaces switch to simple.

+SASL to secure server+
{noformat}
C - S connectHeader(SASL)
C - S NEGOTIATE {
[TOKEN, DIGEST-MD5, proto, realm, first-challenge-token]
[KERBEROS, GSSAPI, user, host, null]
}
C - S INITIATE [ Auth from NEGOTIATE ] response-token
[... back and forth CHALLENGE/RESPONSE ...]
S - C SUCCESS final-token
C - S connectionContext
C - S RPC request
{noformat}

The INITIATE for a token will use the first-challenge-token provided by the
server to cut out a roundtrip. Since the client cannot initiate, the existing
code wastes a round trip soliciting the first DIGEST-MD5 challenge that I'm now
immediately returning.

The INITIATE for kerberos cannot include a first challenge. The GSSAPI
mechanism cannot generate a challenge w/o an initial response from the client.
This is fine and expected.

With this design, a follow on change can allow the client to pick the first
auth type it supports instead of guessing in advance what it should try. It
will use the fields as provided, of course with safeguards to sanity check for
malicious behavior. Using the advertised fields is how the client will support
IP failover.

Walking the advertised auths will allow completely decoupling the auth type
code from the RPC client. The auth types can be implemented as pluggable
services that will be called on demand if the server requests that auth type.
The pluggable services may use the protocol/serverId fields to decide if they
have the credentials to even attempt the auth.

The client should never attempt an auth unless it knows the server supports the
auth, and the client actually has the credentials to do the auth.

+Bad SASL client to secure+
{noformat}
C - S connectHeader(SASL)
C - S NEGOTIATE {
[TOKEN, DIGEST-MD5, proto, realm, first-challenge-token]
[KERBEROS, GSSAPI, user, host, null]
}
C - S INITIATE [ INVALID ] response-token
C - S RPC Exception INVALID not supported, connection closed
{noformat}

Well, you were supposed to reply with one of the advertised methods, not make a
blind guess... Sorry, game over.

Convert SASL to use ProtoBuf and add lengths for non-blocking processing

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689870#comment-13689870
 ] 

Luke Lu commented on HADOOP-9421:
-

My simple to \* is equivalent Daryn's. Note, consecutive C - S can be merged 
into one TCP packet.

SASL to insecure
{code}
C - S connectionHeader(SASL), INITIATE(optional initial token) 
C - S SUCCESS
C - S connectionContext, RPC request
{code}

SASL to secure
{code}
C - S connectionHeader(SASL), INITIATE(optional initial token, [(TOKEN, 
DIGEST-MD5)])
C - S CHALLENGE(challenge-token) or NEGOTIATE([(TOKEN, DIGEST-MD5), (KERBEROS, 
GSSAPI), ...])
C - S RESPONSE(response-token) or REINITIATE(initial token, [(TOKEN, 
DIGEST-MD5)])
...
C - S SUCCESS(final-token)
C - S connectionContext, RPC request
{code}

Bottom line: my patch is a strict superset of Daryn's patch from protocol POV. 
The keyward is *optional* client initiate.  Daryn's protocol can *not* support 
SCRAM (or any modern auths requiring client nonce) without an extra round-trip.

Most of the credit of my patch goes to Daryn, as adding optional client 
initiate is simple (only a few extra lines).

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

2013-06-20 Thread Arun C Murthy (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689895#comment-13689895
 ] 

Arun C Murthy commented on HADOOP-9421:
---

Guys, how far are we from getting this done? 

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689940#comment-13689940
 ] 

Daryn Sharp commented on HADOOP-9421:
-

I know Luke means well, but an initial initiate will never be used after 
subsequent changes.  The negotiate will always be required for IP failover and 
improved token selection.  The approach is trying to over optimize and weakens 
the design.

I benched both patches, and Luke's token auth times are 1ms slower after 100 
samples but let's say they are the same. However, Luke's patch is 8-11ms slower 
with Kerberos than mine. It also has a subset of the changes to make future 
changes compatible. 

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689943#comment-13689943
 ] 

Luke Lu commented on HADOOP-9421:
-

My patch is ready to review, commit and scale testing. I'd appreciate fellow 
committer reviews and +1s :)

I was hoping to nudge Daryn to add optional client initiate. My patch shows 
that adding optional client initiate is simple and future proof.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689950#comment-13689950
 ] 

Daryn Sharp commented on HADOOP-9421:
-

We also need to consider the issues caused by guessing the auth during rolling 
upgrades that change the security mechanisms.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

2013-06-20 Thread Benoy Antony (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689952#comment-13689952
 ] 

Benoy Antony commented on HADOOP-9421:
--

Luke, Is there a sufficient need to add a new state REINITIATE ? The only 
benefit is that it avoids a isSupportedInitiate() call.

From what I understood, Luke's change is a desirable optimization of Daryn's 
protocol. Such optimization is quite common in protocols. I agree that it 
complicates the design a bit. Tt may never be used with the subsequent changes 
related to IP Failover etc, but why not keep it there as it should not cause 
any impact on performance?  Why would it cause change in performance ? 

 


 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689955#comment-13689955
 ] 

Daryn Sharp commented on HADOOP-9421:
-

If it's not obvious, I'm -1 on Luke's patch.  The biggest issue is we must have 
IP failover support.  At best the client will fail to initiate GSSAPI with the 
wrong principle. You'd have to assume an exception means solicit a negotiate 
message to recreate the SASL client.  At worst you get the wrong service ticket 
and fail the negotiation. 

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689959#comment-13689959
 ] 

Daryn Sharp commented on HADOOP-9421:
-

You can do the feature that will never used by blasting the initiate, ignore 
the negotiate response, and hope for the best. 

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689961#comment-13689961
 ] 

Luke Lu commented on HADOOP-9421:
-

bq. Luke, Is there a sufficient need to add a new state REINITIATE ? The only 
benefit is that it avoids a isSupportedInitiate() call.

Yes. It's the most straightforward way to avoid a initiate/negotiate loop 
besides debuggability :)

Bad SASL to secure example:
{code}
C - S connectionHeader(SASL), INITIATE([(INVALID, ...)])
C - S NEGOTIATE([(TOKEN, DIGEST-MD5), (KERBEROS, GSSAPI), ...]) // due to 
isSupportedInitiate returning false.
C - S REINITIATE([(INVALID, ...)]) // note, there is no transition from 
REINITIATE to NEGOTIATE
C - S S RPC Exception INVALID not supported, connection closed
{code}

The code is simple and hopefully not too subtle (see 
Connection#saslReadAndProcess and ProcessSaslMessage).

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689971#comment-13689971
 ] 

Luke Lu commented on HADOOP-9421:
-

bq. Luke's patch is 8-11ms slower with Kerberos than mine

Ah, that's probably due to my unoptimized patch does initial token twice (one 
in INITIATE and the other in REINITIATE), which causes an extra round trip to 
KDC to get service ticket. But I can fix this with no protocol changes and 
eliminate the difference. OTOH, I can further eliminate the NEGOTIATE round 
trip for Kerberos in normal/stead state cases. 

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689976#comment-13689976
 ] 

Daryn Sharp commented on HADOOP-9421:
-

No, you cannot try to further avoid the negotiate. W/o the negotiate to 
properly initialize the SASL client for the proper service host, IP failover 
will never work!



 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689980#comment-13689980
 ] 

Luke Lu commented on HADOOP-9421:
-

bq. At best the client will fail to initiate GSSAPI with the wrong principle. 
You'd have to assume an exception means solicit a negotiate message to recreate 
the SASL client. At worst you get the wrong service ticket and fail the 
negotiation. 

I think we can do better with my protocol for the insane (separate server 
principals for HA servers for a logical server) fail over case:
{code}
C - S connectionHeader, INITIATE(old-token, [KERBEROS], old-host)
C - S NEGOTIATE([TOKEN, KERBEROS], new-host) // no exception as the server can 
detect that old-host and new-host being different.
C - S REINITIATE(new-token, KERBEROS)
...
{code}

Client can cache the server name and in steady state case:
{code}
C - S connectionHeader, INITIATE(new-token, [KERBEROS], new-host)
C - S CHALLENGE(final-token)
{code}

Look ma, no NEGOTIATE!



 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13690034#comment-13690034
 ] 

Daryn Sharp commented on HADOOP-9421:
-

You seem to be trying to tailor a design that only considers today's 
implementation of tokens and kerberos.  It seems easy when you assume there 
are only two choices.  The optimization becomes more and more complicated, and 
in many cases impossible, instead of simply doing something the server tells 
you to do.

When pluggable auth support allows a world of heterogenous security, such as 
Knox or Rhino, requiring REINITIATE penalties becomes very expensive.

Sorry for the very long read, but these are topics I intended to address on the 
call that unfortunately didn't happen today.

+IP failover+
Distinct service principals with IP failover isn't insane.  With a shared 
principal services can't be accessed directly because the host doesn't match 
the shared principal.  So a different config with a hardcoded shared principal 
is needed.  Similarly, DNs won't be able to heartbeat directly into HA NNs.  
I'm sure there are more problems than we've already discovered investigating 
that route.

The root issue is the client must only use the hostname that appears in the 
kerberos service principal.  Which means you can't access the service via all 
its interface, hostnames, or even pretty CNAMEs.

If server advertises this is who I am via the NEGOTIATE, then the problem is 
solved.

+Token selection issues+
Selecting tokens pre-connection based on the service as a host or ip port tuple 
is a problem.  Let's take a few examples:

Using the IP precludes multi-interface host support, for instance if you want 
to have a fast/private intra-cluster network and a separate public network.  
Tokens will contain the public IP, but clients using the private interface 
(different IP) can't find them.  This isn't contrived, it's something Cloudera 
has wanted to do.

You also can't use the IP because changing a service's IP will break clients 
using tokens with the old IP.  In comes the bane of my creation, use_ip=false, 
to use the given hostname.  But you can't allow non-fully qualified names 
because it will resolve differently on depending on the dns search path.  
There's a raft of reasons why the canonicalization isn't as straightforward as 
you'd think, which led to a custom NetUtils resolver and complicated path 
normalization.

Likewise, any sort of public proxy or NAT-ing between an external client and a 
cluster service creates an unusable token service within the grid.

HA token logic is unnecessarily convoluted to clone tokens from a logical uri 
into multiple tokens with each failover's service.

_Solution_
A clean solution to all these problems is tokens contain a server generated 
opaque id.  The server NEGOTIATE reports this id.  The client looks for a token 
with that id.  Now no matter what interface/IP/hostname/proxy/NAT is used, the 
client will always find the token.

If you cut out the use of the NEGOTIATE, this ability is gone.

+Supporting new auth methods+
Other new auths in the future may need the protocol/serverId hints from the 
NEGOTIATE to locate the required credentials.  Guessing may not be an option.

The RPC client shouldn't have to be modified to make a pre-connection guess for 
all the auth methods it supports.  Because...

Why should the client attempt an auth method before it _even knows if the 
server can do it_?  Let's look at some hairy examples:

The client tries to do kerberos, so it needs to generate the initial response 
to take advantage of your optimization.  But the server isn't kerberized.  So 
either the client fails because it has no TGT, which it doesn't even need!  Or 
fails to get a non-existent service principal.

What if the client decides to use an SSO service, but the server doesn't do 
SSO?  Take a REINITIATE penalty every time?

+Supporting new mechanisms+
Let's say we add support for a new mechanism like SCRAM.  Just because the 
client can do it doesn't mean all services across all clusters can do it.  The 
server's NEGOTIATE will tell the client if it can do DIGEST-MD5, SCRAM, etc.

Inter-cluster compatibility and rolling upgrades will introduce scenarios where 
the required mechanism differs, and penalizing the client to REINITIATE is not 
a valid option.

---

In all of these scenarios, there aren't complex issues if the NEGOTIATE is used 
to chose an appropriate auth type.  In a world of multiple auths and multiple 
mechanisms for an auth, requiring REINITIATE penalties is too expensive.

Ignoring all the issues I've cited, your optimization doesn't appear to have a 
positive impact on performance.  Even if it did shave a few milliseconds or 
even 100ms, will it have a measurable real-world impact?  Considering how many 
RPC requests are performed over a single connection, will the negligible 
penalty from

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

[
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13690051#comment-13690051
]

Luke Lu commented on HADOOP-9421:
-

bq. In a world of multiple auths and multiple mechanisms for an auth, requiring
REINITIATE penalties is too expensive.

If a client can't pick a mechanism, it could skip the initial token and send an
empty INITIATE, REINITIATE is then not expensive, i.e. exactly equivalent to
yours.

Hadoop RPC foremost should serve its most common workload: delegation tokens. A
performance regression for the most common workload in the name of integration
is not acceptable. The specific optimization for Digest-MD5 (cramming a
speculative challenge with a negotiate) doesn't work with modern client
initiated auths like SCRAM. If we have to replace Digest-MD5 for security
reasons, we'll be SOL.

bq. Ignoring all the issues I've cited, your optimization doesn't appear to
have a positive impact on performance.

There is no optimization my patch, which merely leave the door for future
optimization. In fact, there is performance bug in my impl for Kerberos. It's
you who added a speculative optimization for Digest-MD5 that doesn't work with
its future replacement SCRAM.

bq. I feel like we've spent weeks haggling over an ill-suited pre-mature
optimization that could been spent building upon this implementation.

I merely want to leave the optional client initiate proto *in the protocol* for
future optimizations. I feel like being forced to implement the optimization to
show that it's straight forward and incremental.

Convert SASL to use ProtoBuf and add lengths for non-blocking processing

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687645#comment-13687645
 ] 

Sanjay Radia commented on HADOOP-9421:
--

bq. Client today does send/(send/read)+, and now it's send/read/(send/read)+.
Daryn, I assume the first read in the new version is to read the server's list 
of auth methods.
Generally the preferred approach is: client sends preferred auth with a list of 
alternatives client can do and server either accepts it or throws back a 
counter with one of the alternates that the client proposed. This would avoid 
an extra trip in the normal case.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13688005#comment-13688005
 ] 

Daryn Sharp commented on HADOOP-9421:
-

Maybe I'm misunderstanding, but won't the server's response to accept/counter 
the client's proposal introduce the same delay?  It also introduces a 
complication whereby the client is still guessing if it has the required 
credentials for the auth methods, even though it may need information from the 
server to determine if it has the credentials.  For instance, to remove use_ip 
and support multiple interfaces, the server will need to provide a uniqueId to 
the client to locate a token.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13688055#comment-13688055
 ] 

Daryn Sharp commented on HADOOP-9421:
-

I think I may be able to make everyone happy, by actually reducing the current 
message exchange for tokens.

Current flow:
{noformat}
1: C - S connectionHeader
2: C - S TOKEN sasl:null
3: C - S TOKEN sasl:token-challenge
4: C - S TOKEN sasl:token-response
...
{noformat}

Patch flow:
{noformat}
1  : C - S connectionHeader
1.1: C - S [TOKEN, KERBEROS] sasl:null
2  : C - S TOKEN sasl:null
3  : C - S TOKEN sasl:token-challenge
4  : C - S TOKEN sasl:token-response
...
{noformat}

How about:
{noformat}
1  : C - S connectionHeader
1.1: C - S [TOKEN, KERBEROS] sasl:token-challenge
4  : C - S TOKEN sasl:token-response
...
{noformat}

I'm testing to see if this will work.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

2013-06-19 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13688438#comment-13688438
 ] 

Hadoop QA commented on HADOOP-9421:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12588682/HADOOP-9421.patch
  against trunk revision .

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/2678//console

This message is automatically generated.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13688441#comment-13688441
 ] 

Luke Lu commented on HADOOP-9421:
-

New patch flow:

packet 1: C-S connectionHeader + blind-initiate
packet 2: C-S (token-challenge|success-for-simple|negotiate)
...

Note, I refactored general purpose protobuf RpcWrapper into ProtobufHelper as 
they don't belong in ProtobufEngine.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13688449#comment-13688449
 ] 

Luke Lu commented on HADOOP-9421:
-

Looks like I need to merge with atm's fall-back-to-simple option commit 
(without a JIRA). 

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13688459#comment-13688459
 ] 

Sanjay Radia commented on HADOOP-9421:
--

Daryn, it appears that this proposal is not wrapping sasl in frames but 
replacing sasl with an alternate protocol. 

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13688464#comment-13688464
 ] 

Sanjay Radia commented on HADOOP-9421:
--

Daryn, to fully understand you need to describe what happens for simple-auth, 
token, kerberos. Do we want to retain the current protocol's property where 
simpl-auth is not done within SASL but outside SASL.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13688470#comment-13688470
 ] 

Luke Lu commented on HADOOP-9421:
-

bq. it appears that this proposal is not wrapping sasl in frames but replacing 
sasl with an alternate protocol. 

It does, but it's further extended to allow negotiation of sasl mechanisms. 
Simply wrapping sasl in protobuf headers would be sufficient to address our 
near future needs. I believe that my latest patch addresses both performance 
for token auth and flexibility for mech negotiation.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13688472#comment-13688472
 ] 

Luke Lu commented on HADOOP-9421:
-

bq.  Simply wrapping sasl in protobuf headers would be sufficient to address 
our near future needs

I meant would not

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13688514#comment-13688514
 ] 

Sanjay Radia commented on HADOOP-9421:
--

bq. packet 2: C-S (token-challenge|success-for-simple|negotiate)
Since SASL does not have a negotiate response are you proposing that we 
encode this in a -ve length of the token (just as SWITCH_TO_SIMPLE is encoded 
as -88)?

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13688556#comment-13688556
 ] 

Luke Lu commented on HADOOP-9421:
-

SASL itself doesn't negotiate. Currently we have hack via -88 length to switch 
to SIMPLE. The proposal got rid of the hack and replace it with negotiate proto 
which contains a list of mechs that the server supports.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13688558#comment-13688558
 ] 

Sanjay Radia commented on HADOOP-9421:
--

We do not have a way to test such a major change this late in the game for 2.0. 
Let's leave the SASL part  unchanged for V9 (ie Hadoop 2.x) and do a backwards 
compatible change in trunk; a backwards compatible change is possible because 
of  the version number in the header.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

[
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13688564#comment-13688564
]

Daryn Sharp commented on HADOOP-9421:
-

SASL just defines the format of the bytes used by mechanisms. It's left as an
exercise to the reader as to how to negotiate and wire encode the
challenge/response sequence. Per the SASL RFC
http://tools.ietf.org/html/rfc4422#section-4

{noformat}
4. Protocol Requirements

In order for a protocol to offer SASL services, its specification
MUST supply the following information:

1) A service name, to be selected from registry of service elements
for the Generic Security Service Application Program Interface
(GSSAPI) host-based service name form, as described in Section 4.1
of [RFC2743]. Note that this registry is shared by all GSSAPI and
SASL mechanisms.

2) Detail any mechanism negotiation facility that the protocol
provides (see Section 3.2).

A protocol SHOULD specify a facility through which the client may
discover, both before initiation of the SASL exchange and after
installing security layers negotiated by the exchange, the names
of the SASL mechanisms that the server makes available to the
client. The latter is important to allow the client to detect
downgrade attacks. This facility is typically provided through
the protocol's extensions or capabilities discovery facility.

3) Definition of the messages necessary for authentication exchange,
including the following:

a) A message to initiate the authentication exchange (see Section
3.3).

This message MUST contain a field for carrying the name of the
mechanism selected by the client.

This message SHOULD contain an optional field for carrying an
initial response. If the message is defined with this field,
the specification MUST describe how messages with an empty
initial response are distinguished from messages with no
initial response. This field MUST be capable of carrying
arbitrary sequences of octets (including zero-length sequences
and sequences containing zero-valued octets).

b) Messages to transfer server challenges and client responses
(see Section 3.4).

Each of these messages MUST be capable of carrying arbitrary
sequences of octets (including zero-length sequences and
sequences containing zero-valued octets).

c) A message to indicate the outcome of the authentication
exchange (see Section 3.6).

This message SHOULD contain an optional field for carrying
additional data with a successful outcome. If the message is
defined with this field, the specification MUST describe how
messages with an empty additional data are distinguished from
messages with no additional data. This field MUST be capable
of carrying arbitrary sequences of octets (including zero-
length sequences and sequences containing zero-valued octets).
{noformat}

Convert SASL to use ProtoBuf and add lengths for non-blocking processing

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13688576#comment-13688576
 ] 

Daryn Sharp commented on HADOOP-9421:
-

bq. We do not have a way to test such a major change this late in the game for 
2.0.

As with a number of things security related, Y! and by extension me are the 
test. :)

bq. Let's leave the SASL part unchanged for V9 (ie Hadoop 2.x) and do a 
backwards compatible change in trunk; a backwards compatible change is possible 
because of the version number in the header.

We really need to discard the current limited implementation before 
compatibility is a must.  Supporting two implementations will be a huge burden.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13688579#comment-13688579
 ] 

Luke Lu commented on HADOOP-9421:
-

bq. We do not have a way to test such a major change this late in the game for 
2.0.

Well, Daryn will be doing such tests on Y clusters. I believe that we should 
try to make 2.0 friendly for backward compatibility.

bq. a backward compatible change is possible because of the version number in 
the header.

Switching on version number means that the community will be supporting some 
old cruft for a long time both in client and server. Not sure it's a worthy 
trade off for a week or two schedule. 

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

2013-06-19 Thread Arun C Murthy (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13688622#comment-13688622
 ] 

Arun C Murthy commented on HADOOP-9421:
---

How much more work are we talking about here? Is the patch ready to go?

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13688657#comment-13688657
 ] 

Luke Lu commented on HADOOP-9421:
-

I think it's close. It needs to be rebased against trunk for atm's security 
fix. I'm also adding two unit tests to make sure fallback prevention actually 
works.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

2013-06-19 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13688801#comment-13688801
 ] 

Hadoop QA commented on HADOOP-9421:
---

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12588738/HADOOP-9421.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-common-project/hadoop-common.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/2681//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/2681//console

This message is automatically generated.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

2013-06-18 Thread Daryn Sharp (JIRA)

[
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687233#comment-13687233
]

Daryn Sharp commented on HADOOP-9421:
-

Per Sanjay's request, here's a quick summary:

This patch does not direct address deficiencies in negotiation process. It
merely provides the flexibility to fix them in the near future. The basic
changes are:
* SASL protobufs
* SASL protobufs wrapped with RPC header
* Server advertised SASL auth methods - currently ignored by client

Wrapping SASL negotiation in a RPC header avoids decoding errors. Ex. A secure
client currently tries to decode a RPC exception from an incompatible
connection header as a SASL message. The client also can't handle a RPC
exception mid-stream during SASL negotiation. The RPC header allows the server
and client to correctly decode the payload.

Using the RPC headers, with the addition of a streamId, will also allow the
_future potential_ to multiplex multiple UGIs over the same connection.
Services like the NN can be overwhelmed by a stampede of connections. The NM
may be an ideal candidate for aggregation of connections, or even a per-node
multiplexor for tasks.

Server advertisement of auth methods aims to address the current limitation of
allowing 1 pre-determined auth or simple. Additional consideration is needed
for a client-side whitelist to avoid server unexpectedly requesting weak
authentication. In the current patch, the client ignores the server and
blindly attempts the negotiation as it does today.

+Other future features unlocked by this design+
* Brings us much closer to pluggable auth methods w/o changing server and
client code
* Client may select a server advertised auth method to:
*# Support services and/or clusters with heterogenous auth methods (ex.
thinking of knox, rhino, etc)
*# *Critical to Y!*: IP failover, especially for HA with kerberos. The client
needs the active NN's host to acquire a service ticket
*# Simplify token selection by using an opaque identifier supplied by the
server - will eliminate use_ip, and even the complex HA token logic
*# Support accessing multi-interface hosts on all interfaces
*# Support accessing services via any of their hostnames, ips, or CNAMEs
* Ability to greatly reduce complexity of client/server auth code, and cleanly
decouple SASL logic
* As mentioned before, multiplexing of different UGIs over a shared connection

Convert SASL to use ProtoBuf and add lengths for non-blocking processing

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

2013-06-18 Thread Arun C Murthy (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687375#comment-13687375
 ] 

Arun C Murthy commented on HADOOP-9421:
---

Is there any reason to not commit this and possibly fix performance issues 
later? Will it change the protocols incompatibly to improve perf?

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

2013-06-18 Thread Sanjay Radia (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687643#comment-13687643
 ] 

Sanjay Radia commented on HADOOP-9421:
--

bq. Client today does send/(send/read)+, and now it's send/read/(send/read)+.


 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

2013-06-17 Thread Daryn Sharp (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13685656#comment-13685656
 ] 

Daryn Sharp commented on HADOOP-9421:
-

Depending on how you define round trip, it's just a 1/2 trip for a new 
response. Client today does send/(send/read)\+, and now it's 
send/*read*/(send/read)\+.  First file status RPC time increased from 340ms to 
348ms.  The overall times seem a bit high, so I'm going to test on a third 
cluster.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

2013-06-14 Thread Daryn Sharp (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13683488#comment-13683488
 ] 

Daryn Sharp commented on HADOOP-9421:
-

As a followup, I'm trying to get some performance numbers on authentication 
time.  I'm testing a second cluster to ensure the results are valid.

I also have measured that the NEGOTIATE header is ~61 bytes + length of 
hostname.  On my test cluster, it's 92 bytes.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

2013-06-14 Thread Daryn Sharp (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13683896#comment-13683896
 ] 

Daryn Sharp commented on HADOOP-9421:
-

Given today's network speeds, I'm not going to fret over a handful of bytes 
that greatly enhance the design of the authentication layer.

I did extensive testing with a more advanced follow on patch that instantiates 
the SASL client upon receipt of the NEGOTIATE.  That patch resulted in an 
average 1% penalty and a median 2% penalty to the first RPC call using kerberos 
authentication - which translates to less than 10ms.  That held true while 
issuing concurrent calls up until the number of socket readers.  Once the 
number of socket readers is exceeded, performance drops off so sharply that any 
penalty is lost in the statistical noise.

Assuming the penalty is entirely due to the additional NEGOTIATE response, I 
can likely erase it by removing the unnecessary connection context.  We're 
passing null for the SASL authz user when it could be the effective user.  
SIMPLE is the only auth that actually requires the context context, but that 
could be a special case (as it already is in many ways) or could be replaced 
with SASL PLAIN which simply sends the effective and real user separated by 
null bytes.

Note this patch does have a minor issue with kerberos where the client is 
confused about the state of negotiation after it completes.


 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

2013-06-14 Thread Luke Lu (JIRA)

[
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13683963#comment-13683963
]

Luke Lu commented on HADOOP-9421:
-

bq. Given today's network speeds, I'm not going to fret over a handful of bytes
that greatly enhance the design of the authentication layer.

It's not the bytes but a whole new network/rpc round trip that can be easily
avoided. In WAN and/or when the network is busy the round trip latency can
easily be in hundreds of ms.

bq. That patch resulted in an average 1% penalty and a median 2% penalty to the
first RPC call using kerberos authentication - which translates to less than
10ms.

10ms / 1% means the total latency is 1 second! Something is not right here.

bq. I can likely erase it by removing the unnecessary connection context

I doubt it, the unnecessary connection context is in the noise compared with a
network/rpc round trip.

Convert SASL to use ProtoBuf and add lengths for non-blocking processing

Key: HADOOP-9421
URL: https://issues.apache.org/jira/browse/HADOOP-9421
Project: Hadoop Common
Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch,
HADOOP-9421.patch, HADOOP-9421-v2-demo.patch

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

2013-06-13 Thread Daryn Sharp (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13682366#comment-13682366
 ] 

Daryn Sharp commented on HADOOP-9421:
-

The changes in this jira fundamentally boils down to:
* SASL protobufs
* RPC header wrapped SASL protobufs
* Server advertised SASL auth methods

There's not much additional complexity involved.  Using this model opens the 
door to a lot of features such as easier support of the alternate auth methods 
being proposed, IP failover, token support for multi-interface and 
multi-hostnames for servers, multi-plexing UGIs over a single connection, etc.

Sanjay, I'll contact you offline to bring you up to speed on this lengthy jira.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

2013-06-12 Thread Daryn Sharp (JIRA)

[
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13681360#comment-13681360
]

Daryn Sharp commented on HADOOP-9421:
-

bq. I don't think ID_TOKEN or SSO_TOKEN being appropriate in
SaslRpcServer.AuthMethod.
No worries, this will eventually become dynamic and not live in the rpc server
class since we can't afford to change the code every time we add a new
AuthMethod. At the moment, I'm only trying to make the minimal changes to the
base wire protocol to allow future extensibility.

bq. Challenge response is the main performance use case that's worth
optimizing for.
All SASL mechanisms are challenge/response, not just tokens...

We can continue on another jira, but I think you are predicating the
optimization on the assumption the client can always pre-determine the
AuthMethod to use. That's one of the fundamental flaws with the current
implementation. When there are more supported auths and more supported tokens,
the client's initial connection has no way to know which to chose until the
server tells it what it needs. We might be able to optimize a re-connect but
it's going to further complicate and delay this jira.

To put this all in perspective, we're holding up the base SASL RPCv9 changes to
discuss a pre-mature optimization to avoid sending 100 bytes, compared to
token negotiation requiring at least 5X more bytes, and kerberos requiring at
least 20X more bytes. Both of these are dwarfed by the real activity over the
connection which why I feel we should defer to another jira.

Convert SASL to use ProtoBuf and add lengths for non-blocking processing

Key: HADOOP-9421
URL: https://issues.apache.org/jira/browse/HADOOP-9421
Project: Hadoop Common
Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch,
HADOOP-9421.patch, HADOOP-9421-v2-demo.patch

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

2013-06-12 Thread Luke Lu (JIRA)

[
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13681564#comment-13681564
]

Luke Lu commented on HADOOP-9421:
-

bq. At the moment, I'm only trying to make the minimal changes to the base wire
protocol to allow future extensibility.

The current mechanism is a major change that introduce an extra round-trip of
server processing (more chances of timeouts) that can potentially break a busy
cluster. I'm only suggest that we simply send a length + RpcSaslProto after the
connection header (analogous to the current authmethod byte but more
extensible), which open up many opportunities to reduce auth round-trips in the
future without future client baggage.

bq. All SASL mechanisms are challenge/response, not just tokens...

PLAIN is not...

bq. I think you are predicating the optimization on the assumption the client
can always pre-determine the AuthMethod to use.

Not always but 99% of the time for most common workload, where container/tasks
access HDFS with delegation tokens, which is not gonna change no matter how
many mechs we'll support in the future, which are mostly to bootstrap a
delegation token or equivalent.

bq. we're holding up the base SASL RPCv9 changes to discuss a pre-mature
optimization to avoid sending 100 bytes, compared to token negotiation
requiring at least 5X more bytes, and kerberos requiring at least 20X more
bytes.

The mandatory negotiate response is way more than 100 bytes and will grow as
the number of mechs/protocols grows. The most important thing here is not bytes
(as long as it fits into one packet) but an extra round trip for the most
common challenge response mechanism that currently requires only one extra
round trip (over simple).

I guess that I don't understand what's so hard to include the extra (length +
RpcSaslProto) in the protocol. You don't even need to change your existing
logic: always send a negotiate for RPC v9. But that little extra proto makes
future evolution of auth so much easier.

Convert SASL to use ProtoBuf and add lengths for non-blocking processing

Key: HADOOP-9421
URL: https://issues.apache.org/jira/browse/HADOOP-9421
Project: Hadoop Common
Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch,
HADOOP-9421.patch, HADOOP-9421-v2-demo.patch

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

2013-06-12 Thread Sanjay Radia (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13681607#comment-13681607
 ] 

Sanjay Radia commented on HADOOP-9421:
--

bq. I'm only suggest that we simply send a length + RpcSaslProto after the 
connection header (analogous to the current authmethod byte but more 
extensible), which open up many opportunities to reduce auth round-trips in the 
future without future client baggage.
To echo this. This jira started out as adding just the length to the reply, 
then Protobuf and now a lot more. We can only afford a simple change for hadoop 
2.x GA. Anything more adds too many changes to go into 2.x since it will 
require extensive testing including performance testing.

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

2013-06-11 Thread Daryn Sharp (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13680770#comment-13680770
 ] 

Daryn Sharp commented on HADOOP-9421:
-

bq.  As long as the mech is TOKEN (or better CHALLENGE_RESPONSE),
The authMethods are mapping an abstract name (ie. TOKEN) to the SASL mechanism. 
 With the other security designs being discussed, we might have an ID_TOKEN or 
SSO_TOKEN.  The abstract name appears in the stringified UGI, so it's not 
applicable to call it CHALLENGE_RESPONSE.

bq. why can't you instantiate the sasl client then (after receiving server 
challenge) with the info from server challenge? Why would an additional 
roundtrip be necessary unless the mech is not supported by server?

That's precisely the convoluted logic I outlined in steps 1-4.  It's a lot of 
additional complexity to pre-maturely optimize just a reconnect, and introduces 
much more complexity.  Ex. If the client's INITIATE is invalid, now the server 
can't return an fatal error and close the connection.  It has to return 
NEGOTIATE.  However a second bad INITIATE should return a fatal error.  The 
client has to know to create a SASL client on the first CHALLENGE, but not 
subsequent CHALLENGES.  Other details rapidly add up.

As opposed to INITIATE means the client and server both instantiate their SASL 
objects at that time.

bq. Trying to see why you're not seeing what I'm seeing: perhaps it's not 
obvious that SaslClient#hasInitialResponse is always false for new connection 
with token (Digest-MD5 at least, cf. rfc-2831)?

Initial response is true for other mechanisms like GSSAPI and PLAIN, which 
means kerberos has just been penalized.  If we upgrade the digest mechanism for 
tokens, that new mechanism may also have an initial response.  We can't design 
this around an internal detail of one particular mechanism (DIGEST-MD5).

Backing up, I thought we agreed earlier to defer reconnect optimizations to a 
future jira?

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

2013-06-11 Thread Luke Lu (JIRA)

[
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13680842#comment-13680842
]

Luke Lu commented on HADOOP-9421:
-

bq. The abstract name appears in the stringified UGI, so it's not applicable to
call it CHALLENGE_RESPONSE.

It's still really a mechanism name (vs KERBEROS, PLAIN etc.) and I don't think
ID_TOKEN or SSO_TOKEN being appropriate in SaslRpcServer.AuthMethod. But I'm
not insisting.

bq. If we upgrade the digest mechanism for tokens, that new mechanism may also
have an initial response

Challenge response is the main performance use case that's worth optimizing
for. Digest-MD5 doesn't have initial response. The main (only?) potential
replacement SCRAM doesn't use server name at all (ie, no digest-uri issue in HA
cases). The code can be generic sasl exchange, we just take advantage of the
fact the token based auth is automatically optimized if you send the client
initiation with connection header. The server can simply return a negotiate
response with server name if the server name from the initial response doesn't
match for non-token mechanisms.

bq. Initial response is true for other mechanisms like GSSAPI and PLAIN, which
means kerberos has just been penalized

No. For normal situation (where the client's assumed server name matches
server's), I save a negotiation round trip. For fail over situation, server can
simply return a negotiation with server name, so the client can reinitialize
the saslclient with the correct servername and send the correct initial
response, which is the same number of round trip as your normal case.

bq. Backing up, I thought we agreed earlier to defer reconnect optimizations to
a future jira?

I can definitely compromise for clear trade-offs. But I'd like to make sure we
both fully understand the implications/alternatives before moving on.

Convert SASL to use ProtoBuf and add lengths for non-blocking processing

Key: HADOOP-9421
URL: https://issues.apache.org/jira/browse/HADOOP-9421
Project: Hadoop Common
Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch,
HADOOP-9421.patch, HADOOP-9421-v2-demo.patch

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

2013-06-10 Thread Daryn Sharp (JIRA)

[
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679663#comment-13679663
]

Daryn Sharp commented on HADOOP-9421:
-

Shorting out the NEGOTIATE for a re-connect becomes a bit complicated. The
connection header doesn't contain a length like other RPC packets, so about the
only way to signal the server is another authProtocol to not send a NEGOTIATE,
but expect a subsequent INITIATE. As discussed earlier, I'd prefer to defer
the reconnect optimization to a followup jira.

BTW, I'm currently working on IP failover. We're blocked on HA deployment
because we can't manage the logistics of updating confs and restarting every
cluster services like oozie and hdfsproxy, etc when one cluster is HA enabled
or its HA config changes. IP failover is the answer but that's blocked too
because the service principal changes when failover occurs. The client needs
the server's NEGOTIATE to correctly instantiate its SASL client. This negates
the ability for the client to cache values for an immediate INITIATE. The IP
failover work is based upon this jira.

bq. Though DIGEST is not exactly a precise word here, TOKEN is, IMO, even more
nebulous. How about CHALLENGE_RESPONSE or simply CR? I'd prefer the rename to
be in separate JIRA as well, as it doesn't really affect the wire protocol.

Actually, it does affect the wire because that is a value passed over the wire.
Ie. TOKEN via DIGEST-MD5, or TOKEN via SCRAM, etc.

Convert SASL to use ProtoBuf and add lengths for non-blocking processing

Key: HADOOP-9421
URL: https://issues.apache.org/jira/browse/HADOOP-9421
Project: Hadoop Common
Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch,
HADOOP-9421.patch, HADOOP-9421-v2-demo.patch

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

[
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679738#comment-13679738
]

Luke Lu commented on HADOOP-9421:
-

bq. Shorting out the NEGOTIATE for a re-connect becomes a bit complicated. The
connection header doesn't contain a length like other RPC packets

You can always send the length + RpcSaslProto after the fixed connection
header. Server can than send the appropriate challenge or negotiate
accordingly. It seems more straight forward than the alternatives here.

bq. the only way to signal the server is another authProtocol to not send a
NEGOTIATE,

It's not the only way (see above). But it could work, which is the saving grace
of AuthProtocol :) though SASL2 or HSASL (pronounced as hassle) and its variant
is kinda ugly.

bq. The client needs the server's NEGOTIATE to correctly instantiate its SASL
client. This negates the ability for the client to cache values for an
immediate INITIATE.

That's why it's called cached initiation, server can always send NEGOTIATE as
it see fit after fail over. This is also why I prefer always sending
RpcSaslProto first, so server can decide what to respond in a straight forward
way. Fail over handling is not a common workload, the goal of client cached
initiation is to reduce server side processing in common cases like
container/task launching, when NN/RM are not failing over left and right.

Anyway, though I'm not too happy with the evolution via AuthProtocol approach.
I think I can live with it.

Convert SASL to use ProtoBuf and add lengths for non-blocking processing

Key: HADOOP-9421
URL: https://issues.apache.org/jira/browse/HADOOP-9421
Project: Hadoop Common
Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch,
HADOOP-9421.patch, HADOOP-9421-v2-demo.patch

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

2013-06-10 Thread Daryn Sharp (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679842#comment-13679842
 ] 

Daryn Sharp commented on HADOOP-9421:
-

I believe you are proposing the client always sends two messages upon connect: 
the connect header and a SASL INITIATE.  This optimizes a reconnect, but the 
common case of initial connect will now take an additional round trip penalty 
by sending a second message to request the NEGOTIATE.  To allow for IP 
failover, I guess that means the server response to an invalid INITIATE is a 
NEGOTIATE instead of returning authentication failed and closing the 
connection?  Presumably the second bad INITIATE will continue to return auth 
failed?

I'm unclear what the next step is for this patch.  I'm not too happy about 
evolution via AuthProtocol either but it's the only way I can think of to avoid 
penalizing the initial connect with another roundtrip.  By saying you can live 
with it, is that a tacit +1 or do you want further changes to this patch?

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679870#comment-13679870
 ] 

Luke Lu commented on HADOOP-9421:
-

bq.  This optimizes a reconnect, but the common case of initial connect will 
now take an additional round trip penalty by sending a second message to 
request the NEGOTIATE.

No, for common case, server can detect that client selected the correct 
mechanism (note RpcSaslProto is extensible to contain the appropriate metadata 
needed for server to verify) and respond with a normal sasl challenge (again 
extensible to contain server principal) instead of negotiate. So my proposal 
actually works with majority of failover case (the mechanism doesn't change 
besides server name etc.) with no additional round trip as well. 
  

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679908#comment-13679908
 ] 

Luke Lu commented on HADOOP-9421:
-

bq.  By saying you can live with it, is that a tacit +1 or do you want further 
changes to this patch?

Based on your response, I'm not completely sure that you fully considered my 
proposal (which is more evolution friendly and only requires a small change to 
your patch). But I'm fine with a scapegoat for RPC v9 :)

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing

2013-06-10 Thread Daryn Sharp (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679966#comment-13679966
 ] 

Daryn Sharp commented on HADOOP-9421:
-

I'm open to improvements, but I'm having a hard reconciling how to add this 
capability.  The SASL client and server must be instantiated with the exact 
same mechanism/protocol/serverId or the SASL server throws an exception.

The initial connect will require a way for the client to solicit the NEGOTIATE 
otherwise it has no idea what to do.  This is what incurs a roundtrip.

Here's what it would take for a reconnect:
# The client sends an INITIATE using the cached SaslAuth, but w/o instantiating 
it's SASL client until it receives the CHALLENGE response
# The client can't generate an initial SASL response token since it hasn't 
instantiated the SASL client - leading to an additional roundtrip for SASL
# The server's first CHALLENGE response must set the SaslAuth protobuf field 
which it currently doesn't, but not a big deal
# Client now instantiates SASL client based on SaslAuth in CHALLENGE and 
processes the token


 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing


[ 
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679985#comment-13679985
 ] 

Luke Lu commented on HADOOP-9421:
-

bq. The client can't generate an initial SASL response token since it hasn't 
instantiated the SASL client - leading to an additional roundtrip for SASL

As long as the mech is TOKEN (or better CHALLENGE_RESPONSE), why can't you 
instantiate the sasl client then (after receiving server challenge) with the 
info from server challenge? Why would an additional roundtrip be necessary 
unless the mech is not supported by server?

 Convert SASL to use ProtoBuf and add lengths for non-blocking processing
 

 Key: HADOOP-9421
 URL: https://issues.apache.org/jira/browse/HADOOP-9421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
 Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, 
 HADOOP-9421.patch, HADOOP-9421-v2-demo.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing