[jira] [Created] (HADOOP-9679) KerberosName.rules are not initialized during adding kerberos support to a web servlet using hadoop authentications
fang fang chen created HADOOP-9679: -- Summary: KerberosName.rules are not initialized during adding kerberos support to a web servlet using hadoop authentications Key: HADOOP-9679 URL: https://issues.apache.org/jira/browse/HADOOP-9679 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.0.4-alpha, 1.1.2 Reporter: fang fang chen I am using hadoop-1.1.1 to add kerberos authentication to a web service. But found rules are not initialized, that makes following error happened: java.lang.NullPointerException at org.apache.hadoop.security.KerberosName.getShortName(KerberosName.java:384) at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:328) at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:302) at java.security.AccessController.doPrivileged(AccessController.java:310) at javax.security.auth.Subject.doAs(Subject.java:573) at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:302) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:340) Seems in hadoop-2.0.3, this issue still is not fixed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Build failed in Jenkins: Hadoop-Common-trunk #816
See https://builds.apache.org/job/Hadoop-Common-trunk/816/changes Changes: [szetszwo] HDFS-4797. BlockScanInfo does not override equals(..) and hashCode() consistently. -- [...truncated 52021 lines...] Adding reference: maven.compile.classpath Adding reference: maven.runtime.classpath Adding reference: maven.test.classpath Adding reference: maven.plugin.classpath Adding reference: maven.project Adding reference: maven.project.helper Adding reference: maven.local.repository [DEBUG] Initialize Maven Ant Tasks parsing buildfile jar:file:/home/jenkins/.m2/repository/org/apache/maven/plugins/maven-antrun-plugin/1.6/maven-antrun-plugin-1.6.jar!/org/apache/maven/ant/tasks/antlib.xml with URI = jar:file:/home/jenkins/.m2/repository/org/apache/maven/plugins/maven-antrun-plugin/1.6/maven-antrun-plugin-1.6.jar!/org/apache/maven/ant/tasks/antlib.xml from a zip file parsing buildfile jar:file:/home/jenkins/.m2/repository/org/apache/ant/ant/1.8.1/ant-1.8.1.jar!/org/apache/tools/ant/antlib.xml with URI = jar:file:/home/jenkins/.m2/repository/org/apache/ant/ant/1.8.1/ant-1.8.1.jar!/org/apache/tools/ant/antlib.xml from a zip file Class org.apache.maven.ant.tasks.AttachArtifactTask loaded from parent loader (parentFirst) +Datatype attachartifact org.apache.maven.ant.tasks.AttachArtifactTask Class org.apache.maven.ant.tasks.DependencyFilesetsTask loaded from parent loader (parentFirst) +Datatype dependencyfilesets org.apache.maven.ant.tasks.DependencyFilesetsTask Setting project property: test.build.dir - https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/target/test-dir Setting project property: test.exclude.pattern - _ Setting project property: hadoop.assemblies.version - 3.0.0-SNAPSHOT Setting project property: test.exclude - _ Setting project property: distMgmtSnapshotsId - apache.snapshots.https Setting project property: project.build.sourceEncoding - UTF-8 Setting project property: distMgmtSnapshotsUrl - https://repository.apache.org/content/repositories/snapshots Setting project property: distMgmtStagingUrl - https://repository.apache.org/service/local/staging/deploy/maven2 Setting project property: test.build.data - https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/target/test-dir Setting project property: commons-daemon.version - 1.0.13 Setting project property: hadoop.common.build.dir - https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/../../hadoop-common-project/hadoop-common/target Setting project property: testsThreadCount - 4 Setting project property: maven.test.redirectTestOutputToFile - true Setting project property: jdiff.version - 1.0.9 Setting project property: distMgmtStagingName - Apache Release Distribution Repository Setting project property: project.reporting.outputEncoding - UTF-8 Setting project property: build.platform - Linux-i386-32 Setting project property: failIfNoTests - false Setting project property: distMgmtStagingId - apache.staging.https Setting project property: distMgmtSnapshotsName - Apache Development Snapshot Repository Setting project property: ant.file - https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/pom.xml [DEBUG] Setting properties with prefix: Setting project property: project.groupId - org.apache.hadoop Setting project property: project.artifactId - hadoop-common-project Setting project property: project.name - Apache Hadoop Common Project Setting project property: project.description - Apache Hadoop Common Project Setting project property: project.version - 3.0.0-SNAPSHOT Setting project property: project.packaging - pom Setting project property: project.build.directory - https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/target Setting project property: project.build.outputDirectory - https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/target/classes Setting project property: project.build.testOutputDirectory - https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/target/test-classes Setting project property: project.build.sourceDirectory - https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/src/main/java Setting project property: project.build.testSourceDirectory - https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/src/test/java Setting project property: localRepository -id: local url: file:///home/jenkins/.m2/repository/ layout: none Setting project property: settings.localRepository - /home/jenkins/.m2/repository Setting project property: maven.project.dependencies.versions - [INFO] Executing tasks Build sequence for target(s) `main' is [main] Complete build sequence is [main, ] main: [mkdir] Created dir: https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/target/test-dir [mkdir]
[jira] [Created] (HADOOP-9680) Extend S3FS and S3NativeFS to work with AWS IAM Temporary Security Credentials
Robert Gibbon created HADOOP-9680: - Summary: Extend S3FS and S3NativeFS to work with AWS IAM Temporary Security Credentials Key: HADOOP-9680 URL: https://issues.apache.org/jira/browse/HADOOP-9680 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Reporter: Robert Gibbon Priority: Minor Attachments: s3fs-temp-iam-creds.diff.patch Here is a patch in unified diff format to enable Amazon Web Services IAM Temporary Security Credentials secured interactions with S3 from Hadoop. It bumps the JetS3t release version up to 0.9.0. To use a temporary security credential set, you need to provide the following properties, depending on the implementation (s3 or s3native): fs.s3.awsAccessKeyId or fs.s3n.awsAccessKeyId - the temporary access key id issued by AWS IAM fs.s3.awsSecretAccessKey or fs.s3n.awsSecretAccessKey - the temporary secret access key issued by AWS IAM fs.s3.awsSessionToken or fs.s3n.awsSessionToken - the session ticket issued by AWS IAM along with the temporary key fs.s3.awsTokenFriendlyName or fs.s3n.awsTokenFriendlyName - any string -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
KerberosName.rules are null during KerberosName.getShortName() in KerberosAuthenticationHandler
Hi All,
I am trying to add kerberos support to a web servlet via hadoop authentication
classes. This is to make this web servlet server to authenticate its client via
kerberos. I assume this should work. Right?
The whole design is to add AuthFilter at server side and
AuthenticatedURL.injectToken(conn, currentToken) during create connection at
client side. But the process failed at KerberosName.rules, I made a fix based
on 2.0.4-alpha branch. Could you please help to review it and give some
suggestions? I think with this fix, we can add kerberos support to any web
servlet via hadoop authentication classes. I have opened HADOOP-9679 to trace
this issue and applied the patch.
Error:
The process failed during AuthenticationFilter.doFilter, with following error:
java.lang.NullPointerException
at
org.apache.hadoop.security.KerberosName.getShortName(KerberosName.java:384)
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:328)
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:302)
at
java.security.AccessController.doPrivileged(AccessController.java:310)
at javax.security.auth.Subject.doAs(Subject.java:573)
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:302)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:340)
Root cause:
this error happened because KerberosName.rules are not initialized. I found
that this parameter only be initialized during initialize UserGroupInformation
which is used for manager hadoop user and group. Then this parameter will be
initialized during hadoop client(like oozie) access hadoop. But the servlet I
am testing is not hadoop client, then current there is no place for
initializing it. But I think we should make it work via value
KerberosName.rules with default value DEFAULT.
FIX:
Following is my draft fix based on hadoop-2.0.4-alpha branch, with this fix, my
test web servlet can support kerberos now.
---
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/KerberosAuthenticationHandler.java
+++
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/KerberosAuthenticationHandler.java
@@ -308,6 +308,10 @@ public AuthenticationToken run() throws Exception {
} else {
String clientPrincipal = gssContext.getSrcName().toString();
KerberosName kerberosName = new KerberosName(clientPrincipal);
+if( !KerberosName.hasRulesBeenSet()){
+LOG.warn(No rules applied to + kerberosName.toString()
+ . Using DEFAULT rules.);
+KerberosName.setRules(DEFAULT);
+}
String userName = kerberosName.getShortName();
token = new AuthenticationToken(userName, clientPrincipal,
getType());
response.setStatus(HttpServletResponse.SC_OK);
[VOTE] Release Apache Hadoop 0.23.9
I've created a release candidate (RC0) for hadoop-0.23.9 that I would like to release. The RC is available at: http://people.apache.org/~tgraves/hadoop-0.23.9-candidate-0/ The RC tag in svn is here: http://svn.apache.org/viewvc/hadoop/common/tags/release-0.23.9-rc0/ The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days til July 8th. I am +1 (binding). thanks, Tom Graves
Re: git line endings trouble since recent trunk
On Sat, Jun 29, 2013 at 5:18 PM, Luke Lu l...@vicaya.com wrote: The problem is due to relnotes.py generating the html containing some CRLF (from JIRA) and the release manager not using git-svn, which caused the html with mixed eol getting checked in. The problem would then manifest for git users due to text=auto in .gitattributes (see HADOOP-8912) that auto converts CRLF to LF, hence the persistent modified status of the releasenotes.html for a fresh checkout. Adding svn:eol-style=native would only fix half the problem. We need to fix relnotes.py to avoid generating html with mixed eols (by normalizing everything to LF or native). While I agree that it would be nice to fix relnotes.py, it seems to me that setting svn:eol-style=native should fix the problem completely. Files with this attribute set are stored internally by subversion with all newlines as LF, and converted to CRLF as needed. After all, eol-style=native would not be very useful if it only applied on checkout. Windows users would be constantly checking in CRLF in that case. I'm not an svn expert, though, and I haven't tested the above. Colin On Fri, Jun 28, 2013 at 1:03 PM, Colin McCabe cmcc...@alumni.cmu.eduwrote: Clarification: svn:eol-style = native causes the files to contain whatever the native platform used to check out the code uses. I think just setting this property on all the HTML files should resolve this and future problems. patch posted. C. On Fri, Jun 28, 2013 at 12:56 PM, Colin McCabe cmcc...@alumni.cmu.edu wrote: I think the fix for this is to set svn:eol-style to native on this file. It's set on many other files, just not on this one: cmccabe@keter:~/hadoopST/trunk svn propget svn:eol-style ./hadoop-project-dist/README.txt native cmccabe@keter:~/hadoopST/trunk svn propget svn:eol-style ./hadoop-hdfs-project/hadoop-hdfs/src/main/docs/releasenotes.html cmccabe@keter:~/hadoopST/trunk It might also be a good idea to run dos2unix on it. I thought that in general we wanted to have 'LF' everywhere, so perhaps we should add this to the patch.sh script to prevent this from re-occurring. Colin On Fri, Jun 28, 2013 at 12:27 PM, Sandy Ryza sandy.r...@cloudera.com wrote: I haven't been able to find a solution. Just filed https://issues.apache.org/jira/browse/HADOOP-9675. On Fri, Jun 28, 2013 at 12:19 PM, Omkar Joshi ojo...@hortonworks.com wrote: Sandy... did you fix this? any jira to track? me too facing same problem.. Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Fri, Jun 28, 2013 at 11:54 AM, Zhijie Shen zs...@hortonworks.com wrote: Have the same problem here, have to edit the patch manually to exclude the changes in releasenotes.html On Fri, Jun 28, 2013 at 11:01 AM, Sandy Ryza sandy.r...@cloudera.com wrote: Has anybody else been having trouble with line endings since pulling trunk recently? hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html shows up as modified even though I haven't touched it, and I can't check it out or reset to a previous version to make that go away. The only thing I can do to neutralize it is to put it in a dummy commit, but I have to do this every time I switch branches or rebase. This appears to have began after the release notes commit (8c5676830bb176157b2dc28c48cd3dd0a9712741), and must be due to a line endings change. -Sandy -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/
Re: git line endings trouble since recent trunk
I added a couple of links that discusses 'line endings' when I added .gitattributes in this JIRA. HADOOP-8912https://issues.apache.org/jira/browse/HADOOP-8912 I am just reproducing them here. 1. http://git-scm.com/docs/gitattributes#_checking_out_and_checking_in 2. http://stackoverflow.com/questions/170961/whats-the-best-crlf-handling-strategy-with-git -- Raja On Mon, Jul 1, 2013 at 10:42 AM, Colin McCabe cmcc...@alumni.cmu.eduwrote: On Sat, Jun 29, 2013 at 5:18 PM, Luke Lu l...@vicaya.com wrote: The problem is due to relnotes.py generating the html containing some CRLF (from JIRA) and the release manager not using git-svn, which caused the html with mixed eol getting checked in. The problem would then manifest for git users due to text=auto in .gitattributes (see HADOOP-8912) that auto converts CRLF to LF, hence the persistent modified status of the releasenotes.html for a fresh checkout. Adding svn:eol-style=native would only fix half the problem. We need to fix relnotes.py to avoid generating html with mixed eols (by normalizing everything to LF or native). While I agree that it would be nice to fix relnotes.py, it seems to me that setting svn:eol-style=native should fix the problem completely. Files with this attribute set are stored internally by subversion with all newlines as LF, and converted to CRLF as needed. After all, eol-style=native would not be very useful if it only applied on checkout. Windows users would be constantly checking in CRLF in that case. I'm not an svn expert, though, and I haven't tested the above. Colin On Fri, Jun 28, 2013 at 1:03 PM, Colin McCabe cmcc...@alumni.cmu.edu wrote: Clarification: svn:eol-style = native causes the files to contain whatever the native platform used to check out the code uses. I think just setting this property on all the HTML files should resolve this and future problems. patch posted. C. On Fri, Jun 28, 2013 at 12:56 PM, Colin McCabe cmcc...@alumni.cmu.edu wrote: I think the fix for this is to set svn:eol-style to native on this file. It's set on many other files, just not on this one: cmccabe@keter:~/hadoopST/trunk svn propget svn:eol-style ./hadoop-project-dist/README.txt native cmccabe@keter:~/hadoopST/trunk svn propget svn:eol-style ./hadoop-hdfs-project/hadoop-hdfs/src/main/docs/releasenotes.html cmccabe@keter:~/hadoopST/trunk It might also be a good idea to run dos2unix on it. I thought that in general we wanted to have 'LF' everywhere, so perhaps we should add this to the patch.sh script to prevent this from re-occurring. Colin On Fri, Jun 28, 2013 at 12:27 PM, Sandy Ryza sandy.r...@cloudera.com wrote: I haven't been able to find a solution. Just filed https://issues.apache.org/jira/browse/HADOOP-9675. On Fri, Jun 28, 2013 at 12:19 PM, Omkar Joshi ojo...@hortonworks.com wrote: Sandy... did you fix this? any jira to track? me too facing same problem.. Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Fri, Jun 28, 2013 at 11:54 AM, Zhijie Shen zs...@hortonworks.com wrote: Have the same problem here, have to edit the patch manually to exclude the changes in releasenotes.html On Fri, Jun 28, 2013 at 11:01 AM, Sandy Ryza sandy.r...@cloudera.com wrote: Has anybody else been having trouble with line endings since pulling trunk recently? hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html shows up as modified even though I haven't touched it, and I can't check it out or reset to a previous version to make that go away. The only thing I can do to neutralize it is to put it in a dummy commit, but I have to do this every time I switch branches or rebase. This appears to have began after the release notes commit (8c5676830bb176157b2dc28c48cd3dd0a9712741), and must be due to a line endings change. -Sandy -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/
Re: git line endings trouble since recent trunk
why not just add a precommit hook in svn to reject commits with CRLF? On Mon, Jul 1, 2013 at 10:51 AM, Raja Aluri r...@cmbasics.com wrote: I added a couple of links that discusses 'line endings' when I added .gitattributes in this JIRA. HADOOP-8912https://issues.apache.org/jira/browse/HADOOP-8912 I am just reproducing them here. 1. http://git-scm.com/docs/gitattributes#_checking_out_and_checking_in 2. http://stackoverflow.com/questions/170961/whats-the-best-crlf-handling-strategy-with-git -- Raja On Mon, Jul 1, 2013 at 10:42 AM, Colin McCabe cmcc...@alumni.cmu.edu wrote: On Sat, Jun 29, 2013 at 5:18 PM, Luke Lu l...@vicaya.com wrote: The problem is due to relnotes.py generating the html containing some CRLF (from JIRA) and the release manager not using git-svn, which caused the html with mixed eol getting checked in. The problem would then manifest for git users due to text=auto in .gitattributes (see HADOOP-8912) that auto converts CRLF to LF, hence the persistent modified status of the releasenotes.html for a fresh checkout. Adding svn:eol-style=native would only fix half the problem. We need to fix relnotes.py to avoid generating html with mixed eols (by normalizing everything to LF or native). While I agree that it would be nice to fix relnotes.py, it seems to me that setting svn:eol-style=native should fix the problem completely. Files with this attribute set are stored internally by subversion with all newlines as LF, and converted to CRLF as needed. After all, eol-style=native would not be very useful if it only applied on checkout. Windows users would be constantly checking in CRLF in that case. I'm not an svn expert, though, and I haven't tested the above. Colin On Fri, Jun 28, 2013 at 1:03 PM, Colin McCabe cmcc...@alumni.cmu.edu wrote: Clarification: svn:eol-style = native causes the files to contain whatever the native platform used to check out the code uses. I think just setting this property on all the HTML files should resolve this and future problems. patch posted. C. On Fri, Jun 28, 2013 at 12:56 PM, Colin McCabe cmcc...@alumni.cmu.edu wrote: I think the fix for this is to set svn:eol-style to native on this file. It's set on many other files, just not on this one: cmccabe@keter:~/hadoopST/trunk svn propget svn:eol-style ./hadoop-project-dist/README.txt native cmccabe@keter:~/hadoopST/trunk svn propget svn:eol-style ./hadoop-hdfs-project/hadoop-hdfs/src/main/docs/releasenotes.html cmccabe@keter:~/hadoopST/trunk It might also be a good idea to run dos2unix on it. I thought that in general we wanted to have 'LF' everywhere, so perhaps we should add this to the patch.sh script to prevent this from re-occurring. Colin On Fri, Jun 28, 2013 at 12:27 PM, Sandy Ryza sandy.r...@cloudera.com wrote: I haven't been able to find a solution. Just filed https://issues.apache.org/jira/browse/HADOOP-9675. On Fri, Jun 28, 2013 at 12:19 PM, Omkar Joshi ojo...@hortonworks.com wrote: Sandy... did you fix this? any jira to track? me too facing same problem.. Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Fri, Jun 28, 2013 at 11:54 AM, Zhijie Shen zs...@hortonworks.com wrote: Have the same problem here, have to edit the patch manually to exclude the changes in releasenotes.html On Fri, Jun 28, 2013 at 11:01 AM, Sandy Ryza sandy.r...@cloudera.com wrote: Has anybody else been having trouble with line endings since pulling trunk recently? hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html shows up as modified even though I haven't touched it, and I can't check it out or reset to a previous version to make that go away. The only thing I can do to neutralize it is to put it in a dummy commit, but I have to do this every time I switch branches or rebase. This appears to have began after the release notes commit (8c5676830bb176157b2dc28c48cd3dd0a9712741), and must be due to a line endings change. -Sandy -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ -- Alejandro
Re: git line endings trouble since recent trunk
They are not that many that I can think of unless you are using notepad for editing, but some of the windows related files might require CRLF. This can be handled in .gitattributes I think it's a very good idea to add this check to test-patch script and reject the patch based on the CRLF check. -Raja On Mon, Jul 1, 2013 at 11:00 AM, Alejandro Abdelnur t...@cloudera.comwrote: why not just add a precommit hook in svn to reject commits with CRLF? On Mon, Jul 1, 2013 at 10:51 AM, Raja Aluri r...@cmbasics.com wrote: I added a couple of links that discusses 'line endings' when I added .gitattributes in this JIRA. HADOOP-8912https://issues.apache.org/jira/browse/HADOOP-8912 I am just reproducing them here. 1. http://git-scm.com/docs/gitattributes#_checking_out_and_checking_in 2. http://stackoverflow.com/questions/170961/whats-the-best-crlf-handling-strategy-with-git -- Raja On Mon, Jul 1, 2013 at 10:42 AM, Colin McCabe cmcc...@alumni.cmu.edu wrote: On Sat, Jun 29, 2013 at 5:18 PM, Luke Lu l...@vicaya.com wrote: The problem is due to relnotes.py generating the html containing some CRLF (from JIRA) and the release manager not using git-svn, which caused the html with mixed eol getting checked in. The problem would then manifest for git users due to text=auto in .gitattributes (see HADOOP-8912) that auto converts CRLF to LF, hence the persistent modified status of the releasenotes.html for a fresh checkout. Adding svn:eol-style=native would only fix half the problem. We need to fix relnotes.py to avoid generating html with mixed eols (by normalizing everything to LF or native). While I agree that it would be nice to fix relnotes.py, it seems to me that setting svn:eol-style=native should fix the problem completely. Files with this attribute set are stored internally by subversion with all newlines as LF, and converted to CRLF as needed. After all, eol-style=native would not be very useful if it only applied on checkout. Windows users would be constantly checking in CRLF in that case. I'm not an svn expert, though, and I haven't tested the above. Colin On Fri, Jun 28, 2013 at 1:03 PM, Colin McCabe cmcc...@alumni.cmu.edu wrote: Clarification: svn:eol-style = native causes the files to contain whatever the native platform used to check out the code uses. I think just setting this property on all the HTML files should resolve this and future problems. patch posted. C. On Fri, Jun 28, 2013 at 12:56 PM, Colin McCabe cmcc...@alumni.cmu.edu wrote: I think the fix for this is to set svn:eol-style to native on this file. It's set on many other files, just not on this one: cmccabe@keter:~/hadoopST/trunk svn propget svn:eol-style ./hadoop-project-dist/README.txt native cmccabe@keter:~/hadoopST/trunk svn propget svn:eol-style ./hadoop-hdfs-project/hadoop-hdfs/src/main/docs/releasenotes.html cmccabe@keter:~/hadoopST/trunk It might also be a good idea to run dos2unix on it. I thought that in general we wanted to have 'LF' everywhere, so perhaps we should add this to the patch.sh script to prevent this from re-occurring. Colin On Fri, Jun 28, 2013 at 12:27 PM, Sandy Ryza sandy.r...@cloudera.com wrote: I haven't been able to find a solution. Just filed https://issues.apache.org/jira/browse/HADOOP-9675. On Fri, Jun 28, 2013 at 12:19 PM, Omkar Joshi ojo...@hortonworks.com wrote: Sandy... did you fix this? any jira to track? me too facing same problem.. Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Fri, Jun 28, 2013 at 11:54 AM, Zhijie Shen zs...@hortonworks.com wrote: Have the same problem here, have to edit the patch manually to exclude the changes in releasenotes.html On Fri, Jun 28, 2013 at 11:01 AM, Sandy Ryza sandy.r...@cloudera.com wrote: Has anybody else been having trouble with line endings since pulling trunk recently? hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html shows up as modified even though I haven't touched it, and I can't check it out or reset to a previous version to make that go away. The only thing I can do to neutralize it is to put it in a dummy commit, but I have to do this every time I switch branches or rebase. This appears to have began after the release notes commit (8c5676830bb176157b2dc28c48cd3dd0a9712741), and must be due to a line endings change. -Sandy -- Zhijie Shen
write access to hbase wiki
Hi, I was planning to add our current project, BigSecret, to the powered-by list of HBase. https://wiki.apache.org/hadoop/PoweredBy I believe I need to get write permissions to do so. My username is ermanpattuk. Thanks, Erman Pattuk
[jira] [Created] (HADOOP-9681) FileUtil.unTarUsingJava() should close the InputStream upon finishing
Chuan Liu created HADOOP-9681:
-
Summary: FileUtil.unTarUsingJava() should close the InputStream
upon finishing
Key: HADOOP-9681
URL: https://issues.apache.org/jira/browse/HADOOP-9681
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor
In {{FileUtil.unTarUsingJava()}} method, we did not close input steams
explicitly upon finish. This could lead to a file handle leak on Windows.
I discovered this when investigating the unit test case failure of
{{TestFSDownload.testDownloadArchive()}}. FSDownload class will use
{{FileUtil.unTarUsingJava()}} to unpack some temporary archive file. Later, the
temporary file should be deleted. Because of the file handle leak, the
{{File.delete()}} method fails. The test case then fails because it assert the
temporary file should not exist.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9682) Deadlock between RenewalTimerTask methods cancel() and run()
Karthik Kambatla created HADOOP-9682:
Summary: Deadlock between RenewalTimerTask methods cancel() and
run()
Key: HADOOP-9682
URL: https://issues.apache.org/jira/browse/HADOOP-9682
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 1.2.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
MAPREDUCE-4860 introduced a local variable {{cancelled}} in
{{RenewalTimerTask}} to fix the race where {{DelegationTokenRenewal}} attempts
to renew a token even after the job is removed. However, the patch also makes
{{run()}} and {{cancel()}} synchronized methods leading to a potential deadlock
against {{run()}}'s catch-block (error-path).
The deadlock stacks below:
{noformat}
-
org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask.cancel()
@bci=0, line=240 (Interpreted frame)
-
org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal.removeDelegationTokenRenewalForJob(org.apache.hadoop.mapreduce.JobID)
@bci=109, line=319 (Interpreted frame)
{noformat}
{noformat}
-
org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal.removeFailedDelegationToken(org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$DelegationTokenToRenew)
@bci=62, line=297 (Interpreted frame)
-
org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal.access$300(org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$DelegationTokenToRenew)
@bci=1, line=47 (Interpreted frame)
-
org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask.run()
@bci=148, line=234 (Interpreted frame)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Hadoop Summit: Security Design Lounge Session
All - Last week at Hadoop Summit there was a room dedicated as the summit Design Lounge. This was a place where like folks could get together and talk about design issues with other contributors with a simple flip board and some beanbag chairs. We used this as an opportunity to bootstrap some discussions within common-dev for security related topics. I'd like to summarize the security session and takeaways here for everyone. This summary and set of takeaways are largely from memory. Please - anyone that attended - feel free to correct anything that is inaccurate or omitted. Pretty well attended - companies represented: * Yahoo! * Microsoft * Hortonworks * Cloudera * Intel * eBay * Voltage Security * Flying Penguins * EMC * others... Most folks were pretty engaged throughout the session. We set expectations as a meet and greet/project kickoff - project being the emerging security development community. In order to keep the scope of conversations manageable we tried to keep focused on authentication and the ideas around SSO and tokens. We discussed kerberos as: 1. major pain point and barrier to entry for some 2. seemingly perfect for others a. obviously requiring backward compatibility It seemed to be consensus that: 1. user authentication should be easily integrated with alternative enterprise identity solutions 2. that service identity issues should not require thousands of service identities added to enterprise user repositories 3. that customers should not be forced to install/deploy and manage a KDC for services - this implies a couple options: a. alternatives to kerberos for service identities b. hadoop KDC implementation - ie. ApacheDS? There was active discussion around: 1. Hadoop SSO server a. acknowledgement of Hadoop SSO tokens as something that can be standardized for representing both the identity and authentication event data as well and access tokens representing a verifiable means for the authenticated identity to access resources or services b. a general understanding of Hadoop SSO as being an analogue and alternative for the kerberos KDC and the related tokens being analogous to TGTs and service tickets c. an agreement that there are interesting attributes about the authentication event that may be useful in cross cluster trust for SSO - such as a rating of authentication strength and number of factors, etc d. that existing Hadoop tokens - ie. delegation, job, block access - will all continue to work and that we are initially looking at alternatives to the KDC, TGTs and service tickets 2. authentication mechanism discovery by clients - Daryn Sharp has done a bunch of work around this and our SSO solution may want to consider a similar mechanism for discovering trusted IDPs and service endpoints 3. backward compatibility - kerberos shops need to just continue to work 4. some insight into where/how folks believe that token based authentication can be accomplished within existing contracts - SASL/GSSAPI, REST, web ui 5. what the establishment of a cross cutting concern community around security and what that means in terms of the Apache way - email lists, wiki, Jiras across projects, etc 6. dependencies, rolling updates, patching and how it related to hadoop projects versus packaging 7. collaboration road ahead A number of breakout discussions were had outside of the designated design lounge session as well. Takeaways for the immediate road ahead: 1. common-dev may be sufficient to discuss security related topics a. many developers are already subscribed to it b. there is not that much traffic there anyway c. we can discuss a more security focused list if we like 2. we will discuss the establishment of a wiki space for a holistic view of security model, patterns, approaches, etc 3. we will begin discussion on common-dev in near-term for the following: a. discuss and agree on the high level moving parts required for our goals for authentication: SSO service, tokens, token validation handlers, credential management tools, etc b. discuss and agree on the natural seams across these moving parts and agree on collaboration by tackling various pieces in a divide and conquer approach c. more than likely - the first piece that will need some immediate discussion will be the shape and form of the tokens d. we will follow up or supplement discussions with POC code patches and/or specs attached to jiras Overall, design lounge was rather effective for what we wanted to do - which was to bootstrap discussions and collaboration within the community at large. As always, no specific decisions have been made during this session and we can discuss any or all of this within common-dev and on related jiras. Jiras related to the security development group and these discussions: Centralized SSO/Token Server
Re: [VOTE] Release Apache Hadoop 2.1.0-beta
Hi Arun! On Fri, Jun 28, 2013 at 2:05 PM, Arun C Murthy a...@hortonworks.com wrote: Thanks Hitesh Roman, I'll roll RC1 once these are fixed. Here's the latest update. I've incorporated Collin's patch for HADOOP-9676 and re-ran the tests. A few subtests of TestCLI still fail, but quite a few pass and on top of that the failing ones no longer make the NN go OOM. Now, we still need to get to the bottom of why the subset of tests keep failing, but at this point I'd feel comfortable committing a change for HADOOP-9676 to branch-2.1 and going from there. Let me know if I can help some more. Thanks, Roman.
