[jira] [Created] (HDFS-5889) When rolling upgrade is in progress, standby NN should create checkpoint for downgrade.
Tsz Wo (Nicholas), SZE created HDFS-5889: Summary: When rolling upgrade is in progress, standby NN should create checkpoint for downgrade. Key: HDFS-5889 URL: https://issues.apache.org/jira/browse/HDFS-5889 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE After rolling upgrade is started and checkpoint is disabled, the edit log may grow to a huge size. It is not a problem if rolling upgrade is finalized normally since NN keeps the current state in memory and it writes a new checkpoint during finalize. However, it is a problem if admin decides to downgrade. It could take a long time to apply edit log. Rollback does not have such problem. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
Re: Re-swizzle 2.3
IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very odd ways (to the point it seems un-deterministic). I'd say eiher YARN-1577 is fixed or we revert YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean reverts) from Hadoop 2.3 branch before doing the release. I've verified that after reverting those JIRAs things work fine with unmanaged AMs. Thanks. On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy wrote: > I punted YARN-1444 to 2.4 since it's a long-standing issue. > > Jian is away and I don't see YARN-1577 & YARN-1206 making much progress > till he is back; so I'm inclined to push both to 2.4 too. Any objections? > > Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered. > > Overall, I'll try get this out in next couple of days if we can clear the > list. > > thanks, > Arun > > On Feb 3, 2014, at 12:14 PM, Arun C Murthy wrote: > > > An update. Per https://s.apache.org/hadoop-2.3.0-blockers we are now > down to 5 blockers: 1 Common, 1 HDFS, 3 YARN. > > > > Daryn (thanks!) has both the non-YARN covered. Vinod is helping out with > the YARN ones. > > > > thanks, > > Arun > > > > > > > > -- > Arun C. Murthy > Hortonworks Inc. > http://hortonworks.com/ > > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. > -- Alejandro
[jira] [Created] (HDFS-5888) Cannot chmod / with new Globber.
Andrew Wang created HDFS-5888: - Summary: Cannot chmod / with new Globber. Key: HDFS-5888 URL: https://issues.apache.org/jira/browse/HDFS-5888 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.3.0 Reporter: Andrew Wang Assignee: Andrew Wang Due to some changes in the new Globber code, we can no longer chmod "/". We should support this. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (HDFS-5872) Validate configuration of dfs.http.policy
[ https://issues.apache.org/jira/browse/HDFS-5872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai resolved HDFS-5872. -- Resolution: Duplicate HDFS-5873 includes the fix of this bug thus closing this one as a duplicate. > Validate configuration of dfs.http.policy > - > > Key: HDFS-5872 > URL: https://issues.apache.org/jira/browse/HDFS-5872 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haohui Mai >Assignee: Haohui Mai > > The current implementation does not complain on invalid values of > dfs.http.policy. The implementation should bail out to alert the user that he > / she has misconfigured the system. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (HDFS-5884) LoadDelegator should use IOUtils.readFully() to read the magic header
[
https://issues.apache.org/jira/browse/HDFS-5884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao resolved HDFS-5884.
-
Resolution: Fixed
Fix Version/s: HDFS-5698 (FSImage in protobuf)
Hadoop Flags: Reviewed
+1. I've committed this.
> LoadDelegator should use IOUtils.readFully() to read the magic header
> -
>
> Key: HDFS-5884
> URL: https://issues.apache.org/jira/browse/HDFS-5884
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-5698 (FSImage in protobuf)
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Fix For: HDFS-5698 (FSImage in protobuf)
>
> Attachments: HDFS-5884.000.patch
>
>
> Currently FSImageFormat.LoadDelegator reads the magic header using
> {{FileInputStream.read()}}. It does not guarantee that the magic header is
> fully read. It should use {{IOUtils.readFully()}} instead.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (HDFS-5887) Add suffix to generated protobuf class
Haohui Mai created HDFS-5887: Summary: Add suffix to generated protobuf class Key: HDFS-5887 URL: https://issues.apache.org/jira/browse/HDFS-5887 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: HDFS-5698 (FSImage in protobuf) Reporter: Haohui Mai Assignee: Haohui Mai Priority: Minor As suggested by [~tlipcon], the code is more readable if we give each class generated by the protobuf the suffix "Proto". This jira proposes to rename the classes and to introduce no functionality changes. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HDFS-5886) Potential null pointer deference in RpcProgramNfs3#readlink()
Ted Yu created HDFS-5886:
Summary: Potential null pointer deference in
RpcProgramNfs3#readlink()
Key: HDFS-5886
URL: https://issues.apache.org/jira/browse/HDFS-5886
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Ted Yu
Here is related code:
{code}
if (MAX_READ_TRANSFER_SIZE < target.getBytes().length) {
return new READLINK3Response(Nfs3Status.NFS3ERR_IO, postOpAttr, null);
}
{code}
READLINK3Response constructor would dereference the third parameter:
{code}
this.path = new byte[path.length];
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (HDFS-5885) Add annotation for repeated fields in the protobuf definition
Haohui Mai created HDFS-5885: Summary: Add annotation for repeated fields in the protobuf definition Key: HDFS-5885 URL: https://issues.apache.org/jira/browse/HDFS-5885 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: HDFS-5698 (FSImage in protobuf) Reporter: Haohui Mai Assignee: Haohui Mai As suggested by the documentation of Protocol Buffers, the protobuf specification of the fsimage should specify [packed=true] for all repeated fields. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HDFS-5884) LoadDelegator should use IOUtils.readFully() to read the magic header
Haohui Mai created HDFS-5884:
Summary: LoadDelegator should use IOUtils.readFully() to read the
magic header
Key: HDFS-5884
URL: https://issues.apache.org/jira/browse/HDFS-5884
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
Currently FSImageFormat.LoadDelegator reads the magic header using
{FileInputStream.read()}. It does not guarantee that the magic header is fully
read. It should use IOUtils.readFully() instead.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (HDFS-5883) TestZKPermissionsWatcher.testPermissionsWatcher fails sometimes
Jimmy Xiang created HDFS-5883: - Summary: TestZKPermissionsWatcher.testPermissionsWatcher fails sometimes Key: HDFS-5883 URL: https://issues.apache.org/jira/browse/HDFS-5883 Project: Hadoop HDFS Issue Type: Test Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Trivial It looks like sleeping 100 ms is not enough for the permission change to propagate to other watchers. Will increase the sleeping time a little. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HDFS-5882) TestAuditLogs is flaky
Jimmy Xiang created HDFS-5882:
-
Summary: TestAuditLogs is flaky
Key: HDFS-5882
URL: https://issues.apache.org/jira/browse/HDFS-5882
Project: Hadoop HDFS
Issue Type: Test
Reporter: Jimmy Xiang
Priority: Minor
TestAuditLogs fails sometimes:
{noformat}
Running org.apache.hadoop.hdfs.server.namenode.TestAuditLogs
Tests run: 14, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 37.913 sec <<<
FAILURE! - in org.apache.hadoop.hdfs.server.namenode.TestAuditLogs
testAuditAllowedStat[1](org.apache.hadoop.hdfs.server.namenode.TestAuditLogs)
Time elapsed: 2.085 sec <<< FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNotNull(Assert.java:526)
at org.junit.Assert.assertNotNull(Assert.java:537)
at
org.apache.hadoop.hdfs.server.namenode.TestAuditLogs.verifyAuditLogsRepeat(TestAuditLogs.java:312)
at
org.apache.hadoop.hdfs.server.namenode.TestAuditLogs.verifyAuditLogs(TestAuditLogs.java:295)
at
org.apache.hadoop.hdfs.server.namenode.TestAuditLogs.testAuditAllowedStat(TestAuditLogs.java:163)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
Re: Re-swizzle 2.3
I punted YARN-1444 to 2.4 since it's a long-standing issue. Jian is away and I don't see YARN-1577 & YARN-1206 making much progress till he is back; so I'm inclined to push both to 2.4 too. Any objections? Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered. Overall, I'll try get this out in next couple of days if we can clear the list. thanks, Arun On Feb 3, 2014, at 12:14 PM, Arun C Murthy wrote: > An update. Per https://s.apache.org/hadoop-2.3.0-blockers we are now down to > 5 blockers: 1 Common, 1 HDFS, 3 YARN. > > Daryn (thanks!) has both the non-YARN covered. Vinod is helping out with the > YARN ones. > > thanks, > Arun > > > -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Resolved] (HDFS-5871) Use PBHelper to serialize CacheDirectiveInfoExpirationProto
[ https://issues.apache.org/jira/browse/HDFS-5871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao resolved HDFS-5871. - Resolution: Fixed Fix Version/s: HDFS-5698 (FSImage in protobuf) Hadoop Flags: Reviewed I've committed this. > Use PBHelper to serialize CacheDirectiveInfoExpirationProto > --- > > Key: HDFS-5871 > URL: https://issues.apache.org/jira/browse/HDFS-5871 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-5698 (FSImage in protobuf) >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: HDFS-5698 (FSImage in protobuf) > > Attachments: HDFS-5871.000.patch, HDFS-5871.001.patch > > > The current implementation of CacheManager serializes the > CacheDirectiveInfoExpirationProto in-house. It should call PBHelper.convert() >to avoid code duplication. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
Re: Is it me or is the new bootstrap HDFS UI HWX green? (See tip of hadoop-2.3 branch)
On Mon, Feb 3, 2014 at 9:26 PM, Chris Douglas wrote: > > ... > Please take this offline. -C > > No problem. St.Ack
[jira] [Created] (HDFS-5881) Back-port the skip() fix in the short-circuit local reader to 0.23.
Kihwal Lee created HDFS-5881: Summary: Back-port the skip() fix in the short-circuit local reader to 0.23. Key: HDFS-5881 URL: https://issues.apache.org/jira/browse/HDFS-5881 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.23.10 Reporter: Kihwal Lee Priority: Critical It looks like a bug in skip() was introduced by HDFS:2356 and got fixed as a part of HDFS:2834, which is an API change JIRA. This bug causes to skip more (as much as the new offsetFromChunkBoundary) data in certain cases. It is only for branch-0.23. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HDFS-5880) Fix a typo at the title of HDFS Snapshots document
Akira AJISAKA created HDFS-5880: --- Summary: Fix a typo at the title of HDFS Snapshots document Key: HDFS-5880 URL: https://issues.apache.org/jira/browse/HDFS-5880 Project: Hadoop HDFS Issue Type: Bug Components: documentation, snapshots Affects Versions: 2.2.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor The title of the HDFS Snapshots document is "HFDS Snapshots". We should fix it. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HDFS-5879) TestHftpFileSystem: testFileNameEncoding and testSeek leak open fsdis
Gera Shegalov created HDFS-5879: --- Summary: TestHftpFileSystem: testFileNameEncoding and testSeek leak open fsdis Key: HDFS-5879 URL: https://issues.apache.org/jira/browse/HDFS-5879 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: Gera Shegalov FSDataInputStream should be closed once no longer needed for reading in testFileNameEncoding and testSeek. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
