[jira] [Commented] (HADOOP-13598) Add eol=lf for unix format files in .gitattributes
[
https://issues.apache.org/jira/browse/HADOOP-13598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15486817#comment-15486817
]
Hadoop QA commented on HADOOP-13598:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 0m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13598 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12828196/HADOOP-13598.001.patch
|
| Optional Tests | asflicense |
| uname | Linux b50f02106a63 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / f0876b8 |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10491/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add eol=lf for unix format files in .gitattributes
> --
>
> Key: HADOOP-13598
> URL: https://issues.apache.org/jira/browse/HADOOP-13598
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Akira Ajisaka
>Assignee: Yiqun Lin
> Labels: newbie
> Attachments: HADOOP-13598.001.patch
>
>
> In .gitattributes, eol=crlf is set for .cmd and .bat, but eol=lf is not set
> for .java, .html, and so on. Setting eol=lf would be great for developers
> using Windows.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13598) Add eol=lf for unix format files in .gitattributes
[ https://issues.apache.org/jira/browse/HADOOP-13598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HADOOP-13598: --- Status: Patch Available (was: Open) Attach a initial patch to address the comment that [~ajisakaa] has mentioned in HDFS-10856. > Add eol=lf for unix format files in .gitattributes > -- > > Key: HADOOP-13598 > URL: https://issues.apache.org/jira/browse/HADOOP-13598 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Akira Ajisaka >Assignee: Yiqun Lin > Labels: newbie > Attachments: HADOOP-13598.001.patch > > > In .gitattributes, eol=crlf is set for .cmd and .bat, but eol=lf is not set > for .java, .html, and so on. Setting eol=lf would be great for developers > using Windows. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13598) Add eol=lf for unix format files in .gitattributes
[ https://issues.apache.org/jira/browse/HADOOP-13598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HADOOP-13598: --- Attachment: HADOOP-13598.001.patch > Add eol=lf for unix format files in .gitattributes > -- > > Key: HADOOP-13598 > URL: https://issues.apache.org/jira/browse/HADOOP-13598 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Akira Ajisaka >Assignee: Yiqun Lin > Labels: newbie > Attachments: HADOOP-13598.001.patch > > > In .gitattributes, eol=crlf is set for .cmd and .bat, but eol=lf is not set > for .java, .html, and so on. Setting eol=lf would be great for developers > using Windows. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13591) Unit test failed in 'TestOSSContractGetFileStatus' and 'TestOSSContractRootDir'
[
https://issues.apache.org/jira/browse/HADOOP-13591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15486775#comment-15486775
]
Steve Loughran commented on HADOOP-13591:
-
I'm =0 on making "/" a constant, as I don't see anyone ever changing it. What
probably makes more sense is to factor out the repeated operation. From S3aFS
{code}
/**
* Turns a path (relative or otherwise) into an S3 key, adding a trailing
* "/" if the path is not the root and does not already have a "/"
* at the end.
*
* @param key s3 key or ""
* @return the with a trailing "/", or, if it is the root key, "",
*/
private String maybeAddTrailingSlash(String key) {
if (!key.isEmpty() && !key.endsWith("/")) {
return key + '/';
} else {
return key;
}
}
{code}
+can you stop your IDE automatically converting imports to .* ? It complicates
merging and may create conflicts now that there's a {{java.io.FileSystem}} in
Java7+.
> Unit test failed in 'TestOSSContractGetFileStatus' and
> 'TestOSSContractRootDir'
> ---
>
> Key: HADOOP-13591
> URL: https://issues.apache.org/jira/browse/HADOOP-13591
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: HADOOP-12756
>Reporter: Genmao Yu
>Assignee: Genmao Yu
> Fix For: HADOOP-12756
>
> Attachments: HADOOP-13591-HADOOP-12756.001.patch,
> HADOOP-13591-HADOOP-12756.002.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13599) s3a close() to be non-synchronized, so avoid risk of deadlock on shutdown
Steve Loughran created HADOOP-13599: --- Summary: s3a close() to be non-synchronized, so avoid risk of deadlock on shutdown Key: HADOOP-13599 URL: https://issues.apache.org/jira/browse/HADOOP-13599 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 2.7.3 Reporter: Steve Loughran Assignee: Steve Loughran We've had a report of hive deadlocking on teardown, as a synchronous FS close was blocking shutdown threads, similar to HADOOP-3139 S3a close needs to be made non-synchronized. All we need is some code to prevent re-entrancy at the start; easily done -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12981) Remove/deprecate s3native properties from S3NativeFileSystemConfigKeys and core-default.xml
[
https://issues.apache.org/jira/browse/HADOOP-12981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15486699#comment-15486699
]
Steve Loughran commented on HADOOP-12981:
-
OK, I've misread this.
s3n is the FS we still ship. s3:// is the obsolete one. s3n must not be touched.
-1
also, all patches against hadoop-aws have to go through the process of
submitted declaring which s3 endpoint they ran against. Please get into the
habit of this, as its how we deal with jenkins's lack of testing, and identify
issues with different endpoints (Frankfurt, Seoul).
> Remove/deprecate s3native properties from S3NativeFileSystemConfigKeys and
> core-default.xml
> ---
>
> Key: HADOOP-12981
> URL: https://issues.apache.org/jira/browse/HADOOP-12981
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation, tools
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Minor
> Labels: aws
> Attachments: HADOOP-12981.001.patch, HADOOP-12981.002.patch
>
>
> It seems all properties defined in {{S3NativeFileSystemConfigKeys}} are not
> used. Those properties are prefixed by {{s3native}}, and the current s3native
> properties are all prefixed by {{fs.s3n}}, so this is likely not used
> currently. Additionally, core-default.xml has the description of these unused
> properties:
> {noformat}
>
>
> s3native.stream-buffer-size
> 4096
> The size of buffer to stream files.
> The size of this buffer should probably be a multiple of hardware
> page size (4096 on Intel x86), and it determines how much data is
> buffered during read and write operations.
>
>
> s3native.bytes-per-checksum
> 512
> The number of bytes per checksum. Must not be larger than
> s3native.stream-buffer-size
>
>
> s3native.client-write-packet-size
> 65536
> Packet size for clients to write
>
>
> s3native.blocksize
> 67108864
> Block size
>
>
> s3native.replication
> 3
> Replication factor
>
> {noformat}
> I think they should be removed (or deprecated) to avoid confusion if these
> properties are defunct.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10075) Update jetty dependency to version 9
[ https://issues.apache.org/jira/browse/HADOOP-10075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15486689#comment-15486689 ] Steve Loughran commented on HADOOP-10075: - Jersey has been great to work with: the first embedded servlet engine. It did have problems, ones we had got familiar with (DNs used to have to probe on startup to see if jersey was actually serving requests, if it hadn't started properly they'd terminate). Things like that kept us on an old jetty version for a long time. > Update jetty dependency to version 9 > > > Key: HADOOP-10075 > URL: https://issues.apache.org/jira/browse/HADOOP-10075 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.2.0, 2.6.0 >Reporter: Robert Rati >Assignee: Robert Kanter > Attachments: HADOOP-10075-002-wip.patch, HADOOP-10075.patch > > > Jetty6 is no longer maintained. Update the dependency to jetty9. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS to use Jetty
[
https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15486679#comment-15486679
]
Steve Loughran commented on HADOOP-13597:
-
I'd prefer making the leap to jersey/grizzy
> Switch KMS to use Jetty
> ---
>
> Key: HADOOP-13597
> URL: https://issues.apache.org/jira/browse/HADOOP-13597
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>
> The Tomcat 6 we are using will reach EOL at the end of 2017. While there are
> other good options, I would propose switching to {{Jetty 9}} for the
> following reasons:
> * Easier migration. Both Tomcat and Jetty are based on {{Servlet
> Containers}}, so we don't have change client code that much. It would require
> more work to switch to {{JAX-RS}}.
> * Well established.
> * Good performance and scalability.
> Other alternatives:
> * Jersey + Grizzly
> * Tomcat 8
> Your opinions will be greatly appreciated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13590) Retry until TGT expires even if the UGI renewal thread encountered exception
[
https://issues.apache.org/jira/browse/HADOOP-13590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-13590:
---
Attachment: HADOOP-13590.01.patch
Attaching a patch for the idea.
Current behavior is to just print a exception message on the first failure.
Feels to me we should retry here, since the UGI had a successful login. This
would help in scenarios where the renew failure happen to be intermittent.
Patch 1 just to have the retry to be simply every
{{kerberosMinSecondsBeforeRelogin}} interval, until it succeeds or the tgt
expires. We could add a more sophisticated retry logic (e.g. max # of retries
with exponential backoff) if people like this direction.
Also added the exception stack trace to the log message, instead of just the
exception msg alone.
> Retry until TGT expires even if the UGI renewal thread encountered exception
>
>
> Key: HADOOP-13590
> URL: https://issues.apache.org/jira/browse/HADOOP-13590
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.8.0, 2.7.3, 2.6.4
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-13590.01.patch
>
>
> The UGI has a background thread to renew the tgt. On exception, it
> [terminates
> itself|https://github.com/apache/hadoop/blob/bee9f57f5ca9f037ade932c6fd01b0dad47a1296/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java#L1013-L1014]
> If something temporarily goes wrong that results in an IOE, even if it
> recovered no renewal will be done and client will eventually fail to
> authenticate. We should retry with our best effort, until tgt expires, in the
> hope that the error recovers before that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13590) Retry until TGT expires even if the UGI renewal thread encountered exception
[ https://issues.apache.org/jira/browse/HADOOP-13590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-13590: --- Status: Patch Available (was: Open) > Retry until TGT expires even if the UGI renewal thread encountered exception > > > Key: HADOOP-13590 > URL: https://issues.apache.org/jira/browse/HADOOP-13590 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.6.4, 2.7.3, 2.8.0 >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HADOOP-13590.01.patch > > > The UGI has a background thread to renew the tgt. On exception, it > [terminates > itself|https://github.com/apache/hadoop/blob/bee9f57f5ca9f037ade932c6fd01b0dad47a1296/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java#L1013-L1014] > If something temporarily goes wrong that results in an IOE, even if it > recovered no renewal will be done and client will eventually fail to > authenticate. We should retry with our best effort, until tgt expires, in the > hope that the error recovers before that. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13582) Implement logExpireToken in ZKDelegationTokenSecretManager
[
https://issues.apache.org/jira/browse/HADOOP-13582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-13582:
---
Resolution: Invalid
Status: Resolved (was: Patch Available)
Oops... on a closer look we do log the details in the next call,
{{removeStoredToken}}. Sorry for the churn... closing this as invalid.
SequenceNumber should be enough to ID a token, and the other info can be found
in the logs when searching for that ID, so I think it's fine. Thanks for the
suggestion Wei-Chiu!
> Implement logExpireToken in ZKDelegationTokenSecretManager
> --
>
> Key: HADOOP-13582
> URL: https://issues.apache.org/jira/browse/HADOOP-13582
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Affects Versions: 2.8.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Minor
> Labels: supportability
> Attachments: HADOOP-13582.01.patch
>
>
> During my investigation on HADOOP-13487, I found it pretty difficult to track
> KMS DTs in ZK, even in debug level.
> Adding a simple debug log to make future triages easier.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13598) Add eol=lf for unix format files in .gitattributes
[ https://issues.apache.org/jira/browse/HADOOP-13598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin reassigned HADOOP-13598: -- Assignee: Yiqun Lin > Add eol=lf for unix format files in .gitattributes > -- > > Key: HADOOP-13598 > URL: https://issues.apache.org/jira/browse/HADOOP-13598 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Akira Ajisaka >Assignee: Yiqun Lin > Labels: newbie > > In .gitattributes, eol=crlf is set for .cmd and .bat, but eol=lf is not set > for .java, .html, and so on. Setting eol=lf would be great for developers > using Windows. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS to use Jetty
[
https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15486401#comment-15486401
]
Xiao Chen commented on HADOOP-13597:
Thanks for creating this John.
+1 to the idea, based on discussions from HADOOP-10075 so far.
> Switch KMS to use Jetty
> ---
>
> Key: HADOOP-13597
> URL: https://issues.apache.org/jira/browse/HADOOP-13597
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>
> The Tomcat 6 we are using will reach EOL at the end of 2017. While there are
> other good options, I would propose switching to {{Jetty 9}} for the
> following reasons:
> * Easier migration. Both Tomcat and Jetty are based on {{Servlet
> Containers}}, so we don't have change client code that much. It would require
> more work to switch to {{JAX-RS}}.
> * Well established.
> * Good performance and scalability.
> Other alternatives:
> * Jersey + Grizzly
> * Tomcat 8
> Your opinions will be greatly appreciated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13598) Add eol=lf for unix format files in .gitattributes
Akira Ajisaka created HADOOP-13598: -- Summary: Add eol=lf for unix format files in .gitattributes Key: HADOOP-13598 URL: https://issues.apache.org/jira/browse/HADOOP-13598 Project: Hadoop Common Issue Type: Improvement Reporter: Akira Ajisaka In .gitattributes, eol=crlf is set for .cmd and .bat, but eol=lf is not set for .java, .html, and so on. Setting eol=lf would be great for developers using Windows. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
