Re: [VOTE] Move Submarine source code, documentation, etc. to a separate Apache Git repo

+1 given the scope of the future of Submarine, I think that makes sense. I am also happy to see Hadoop contributors being able to commit to Submarine repo. I think that'll help grow the Submarine community in the long run. On Mon, Aug 26, 2019 at 9:26 AM Ayush Saxena wrote: > +1 > > -Ayush > >

Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

This is starting to diverge from the original topic :) But I would like to say (1) If we are treating branch-2/2.0 as the final, last 2.x release, we should be proactive in updating dependencies. I did a quick search and found a lot of dependencies in branch-2 are very old, and some are even

[jira] [Created] (HDFS-14774) Improve RouterWebhdfsMethods#chooseDatanode() error handling

Wei-Chiu Chuang created HDFS-14774: -- Summary: Improve RouterWebhdfsMethods#chooseDatanode() error handling Key: HDFS-14774 URL: https://issues.apache.org/jira/browse/HDFS-14774 Project: Hadoop HDFS

Re: Hadoop storage community online sync

to fix or revert before developing the namespace partitioning feature. On Mon, Aug 19, 2019 at 2:55 PM Wei-Chiu Chuang wrote: > For this week, > We will have Konstantin and the LinkedIn folks to discuss a recent project > that's been baking for quite a while. This is an exciting project as

Revert HDFS-14476

I am reverting HDFS-14476 from trunk, branch-3.2, branch-3.1, branch-2, branch-2.9, branch-2.8. This commit caused test failures as reported in HDFS-14751 by Lisheng Sun. Sorry for the inconvenience.

[jira] [Reopened] (HDFS-14476) lock too long when fix inconsistent blocks between disk and in-memory

[ https://issues.apache.org/jira/browse/HDFS-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reopened HDFS-14476: Reopen. I'll revert from the branches first to unblock folks > lock too long when

[jira] [Reopened] (HDFS-14725) Backport HDFS-12914 to branch-2 (Block report leases cause missing blocks until next report)

[ https://issues.apache.org/jira/browse/HDFS-14725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reopened HDFS-14725: > Backport HDFS-12914 to branch-2 (Block report leases cause missing blocks > unti

Re: Make EOL branches to public?

+1 I feel like one year of inactivity is a good sign that the community is not interested in the branch any more. On Fri, Aug 16, 2019 at 3:14 AM Wangda Tan wrote: > Hi folks, > > Want to hear your thoughts about what we should do to make some branches > EOL. We discussed a couple of times

[jira] [Created] (HDFS-14757) TestBalancerRPCDelay.testBalancerRPCDelay failed

Wei-Chiu Chuang created HDFS-14757: -- Summary: TestBalancerRPCDelay.testBalancerRPCDelay failed Key: HDFS-14757 URL: https://issues.apache.org/jira/browse/HDFS-14757 Project: Hadoop HDFS

Re: [DISCUSS] move storage community online sync schedule

> +1 for 10 pm. > > > > BR, > > - Sid > > > > On Mon, Aug 19, 2019, 8:36 AM Wei-Chiu Chuang > > > wrote: > > > >> I received some feedbacks that the bi-weekly storage online sync that > >> happens Wednesday 9AM US pacific time (GMT-8) is t

Re: [DISCUSS] Hadoop-3.2.1 release proposal

Hi Rohith, Thanks for initiating this. I want to bring up one blocker issue: HDFS-13596 (NN restart fails after RollingUpgrade from 2.x to 3.x) This should be a blocker for all active Hadoop 3.x releases: 3.3.0, 3.2.1, 3.1.3. Hopefully we can get

Re: Hadoop Community Sync Up Schedule

+1 On Mon, Aug 19, 2019 at 8:32 PM Wangda Tan wrote: > Hi folks, > > We have run community sync up for 1.5 months. I spoke to folks offline and > got some feedback. Here's a summary of what I've observed from sync ups and > talked to organizers. > > Following sync ups have very good

Re: Thoughts about moving submarine to a separate git repo?

> > > > Submarine dev community has a total of 8 developers and submits an average > of 4 to 5 PR per day. > But there are a limited number of Hadoop committer actively help review and > merge patches, which causes development progress delays. > > I just want to point this out that this is

Re: Hadoop storage community online sync

zone (probably more specifically, California) So GMT-7 it is. On Mon, Aug 19, 2019 at 11:16 PM Akira Ajisaka wrote: > Thank you for the information. > > Now US pacific time is GMT-7, isn't it? > > -Akira > > On Tue, Aug 20, 2019 at 6:56 AM Wei-Chiu Chuang > wrote: >

[jira] [Resolved] (HDFS-14208) A large number missingblocks happend after failover to active.

[ https://issues.apache.org/jira/browse/HDFS-14208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-14208. Resolution: Duplicate Thanks [~xuzq_zander] and [~hexiaoqiao] > A large num

Hadoop storage community online sync

For this week, We will have Konstantin and the LinkedIn folks to discuss a recent project that's been baking for quite a while. This is an exciting project as it has the potential to improve NameNode's throughput by 40%. HDFS-14703 NameNode

[jira] [Reopened] (HDFS-14665) HttpFS: LISTSTATUS response is missing HDFS-specific fields

[ https://issues.apache.org/jira/browse/HDFS-14665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reopened HDFS-14665: Reopen for branch-3.1 > HttpFS: LISTSTATUS response is missing HDFS-specific fie

[DISCUSS] move storage community online sync schedule

I received some feedbacks that the bi-weekly storage online sync that happens Wednesday 9AM US pacific time (GMT-8) is too early for west coast folks, and the fact is that the majority of Hadoop developers are in the US. Would it make sense to move it to a later time to allow more US west coast

[jira] [Created] (HDFS-14746) Trivial test code update after HDFS-14687

Wei-Chiu Chuang created HDFS-14746: -- Summary: Trivial test code update after HDFS-14687 Key: HDFS-14746 URL: https://issues.apache.org/jira/browse/HDFS-14746 Project: Hadoop HDFS Issue Type

[jira] [Reopened] (HDFS-13101) Yet another fsimage corruption related to snapshot

[ https://issues.apache.org/jira/browse/HDFS-13101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reopened HDFS-13101: > Yet another fsimage corruption related to snaps

[jira] [Reopened] (HDFS-14723) Add helper method FSNamesystem#setBlockManagerForTesting() in branch-2

[ https://issues.apache.org/jira/browse/HDFS-14723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reopened HDFS-14723: Reopen, in order to get HDFS-14725 into branch-2.9 and even 2.8, I need this for branch-2.9

[DISCUSS] Seeking specific area of experts within HDFS

Hi! There are currently a few areas where there is a lack of expertise, and therefore bug reports and patches are left unreviewed. These areas include: (1) NFS gateway (2) libhdfs (3) Misc. file systems, like SFTPFileSystem Are there other areas where you have a patch that's never reviewed

Re: Unassigned Hadoop jiras with patch available

Interestingly enough, I think I can create new wiki, but I don't see a button to edit existing wiki. On Thu, Aug 15, 2019 at 6:24 AM Masatake Iwasaki < iwasak...@oss.nttdata.co.jp> wrote: > Hi Wei-Chiu Chuang, > > Thanks for doing this and sorry for late reply. > > Ho

[jira] [Resolved] (HDFS-14665) HttpFS: LISTSTATUS response is missing HDFS-specific fields

[ https://issues.apache.org/jira/browse/HDFS-14665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-14665. Resolution: Fixed Fix Version/s: 3.2.1 3.3.0 Pushed the PR

[jira] [Resolved] (HDFS-14148) HDFS OIV ReverseXML SnapshotSection parser throws exception when there are more than one snapshottable directory

[ https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-14148. Resolution: Fixed Fix Version/s: 3.3.0 PR was merged. Thanks [~smeng] for reporting

[jira] [Resolved] (HDFS-12914) Block report leases cause missing blocks until next report

[ https://issues.apache.org/jira/browse/HDFS-12914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-12914. Resolution: Fixed I've muddled the water too much in this jira. Let's use HDFS-14725

[jira] [Created] (HDFS-14725) Backport HDFS-12914 to branch-2 (Block report leases cause missing blocks until next report)

Wei-Chiu Chuang created HDFS-14725: -- Summary: Backport HDFS-12914 to branch-2 (Block report leases cause missing blocks until next report) Key: HDFS-14725 URL: https://issues.apache.org/jira/browse/HDFS-14725

[jira] [Created] (HDFS-14724) Fix JDK7 compatibility in branch-2

Wei-Chiu Chuang created HDFS-14724: -- Summary: Fix JDK7 compatibility in branch-2 Key: HDFS-14724 URL: https://issues.apache.org/jira/browse/HDFS-14724 Project: Hadoop HDFS Issue Type: Bug

[jira] [Created] (HDFS-14723) Add helper method BlockManager#setBlockManagerForTesting() in branch-2

Wei-Chiu Chuang created HDFS-14723: -- Summary: Add helper method BlockManager#setBlockManagerForTesting() in branch-2 Key: HDFS-14723 URL: https://issues.apache.org/jira/browse/HDFS-14723 Project

Revert of HDFS-12914 breaks the branch-2

Heads up, My bad. This is becoming a mess. For the context, HDFS-12914 in branch-2 has a test failure, so I reverted it. But HDFS-13898 used a helper method (BlockManager#setBlockManagerForTesting()) added by HDFS-12914, so the revert breaks the build. Here's what I propose: (1) File a new

[jira] [Reopened] (HDFS-12914) Block report leases cause missing blocks until next report

[ https://issues.apache.org/jira/browse/HDFS-12914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reopened HDFS-12914: Reopen. Thanks for reporting the issue. I'll start with reverts in branch-2/branch-2.9

[jira] [Resolved] (HDFS-14408) HttpFS handles paths with special charactes different than WebHdfs

[ https://issues.apache.org/jira/browse/HDFS-14408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-14408. Resolution: Duplicate > HttpFS handles paths with special charactes different than WebH

[jira] [Resolved] (HDFS-14696) Backport HDFS-11273 to branch-2 (Move TransferFsImage#doGetUrl function to a Util class)

[ https://issues.apache.org/jira/browse/HDFS-14696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-14696. Resolution: Fixed Fix Version/s: 2.10.0 Merge the PR, resolve this jira. Thanks

Please file a JIRA for your PR

The Hadoop community welcome your patch contribution, and like increasingly patches are submitted via GitHub Pull Requests. That is great, as it reduces the friction to review code & commit code. However, please make sure to file a jira for your PR, as described in the How to Contribute

[jira] [Created] (HDFS-14709) Add encryption zone related REST APIs to WebHDFS

Wei-Chiu Chuang created HDFS-14709: -- Summary: Add encryption zone related REST APIs to WebHDFS Key: HDFS-14709 URL: https://issues.apache.org/jira/browse/HDFS-14709 Project: Hadoop HDFS

Hadoop Storage Online Sync Notes 8/5/2019

Very happy to have CR from Uber leading today's discussion. Here's today's sync meeting notes. https://docs.google.com/document/d/1jXM5Ujvf-zhcyw_5kiQVx6g-HeKe-YGnFS_1-qFXomI/edit 8/5/2019 CR Hota (Uber) gave an update on Router Based Federation Attendee: Cloudera (Weichiu, Adam), Uber (CR Hota)

[jira] [Resolved] (HDFS-11273) Move TransferFsImage#doGetUrl function to a Util class

[ https://issues.apache.org/jira/browse/HDFS-11273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-11273. Resolution: Fixed > Move TransferFsImage#doGetUrl function to a Util cl

HDFS At Rest Encryption (Encryption Zone, KMS) Improvements

There are a bunch of stuff that can use some attention to improve HDFS At Rest Encryption (aka Data Transparent Encryption). Here's a spreadsheet of features, bug fixes, improvements https://docs.google.com/spreadsheets/d/13oeUy0Mvmq6Ngvw9IjkY9KXiQyoRIcRlijKXG1YqGug/edit?usp=sharing I am

[jira] [Reopened] (HDFS-14674) [SBN read] Got an unexpected txid when tail editlog

[ https://issues.apache.org/jira/browse/HDFS-14674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reopened HDFS-14674: [~wangzhaohui] this patch is not yet committed. Reopen this jira for now. > [SBN read]

[jira] [Resolved] (HDFS-13483) Crypto command should give proper exception when user is trying to create an EZ with the same key with which it is already created

[ https://issues.apache.org/jira/browse/HDFS-13483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-13483. Resolution: Duplicate > Crypto command should give proper exception when user is try

[jira] [Resolved] (HDFS-14605) Note missing on expunge command description for encrypted zones

[ https://issues.apache.org/jira/browse/HDFS-14605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-14605. Resolution: Duplicate > Note missing on expunge command description for encrypted zo

Re: Unassigned Hadoop jiras with patch available

I assigned all jiras with patch available that are created since 2019. If you have jiras that you are actively working on that you can't assign to yourself, please let me know. On Wed, Jul 31, 2019 at 3:11 PM Wei-Chiu Chuang wrote: > I was told the filter is private. I am sorry. > >

Re: Unassigned Hadoop jiras with patch available

%20updated%20DESC On Wed, Jul 31, 2019 at 3:02 PM Wei-Chiu Chuang wrote: > I am using this jira filter to find jiras with patch available but > unassigned. > > https://issues.apache.org/jira/issues/?filter=12346814=project%20in%20(HADOOP%2C%20HDFS%2CYARN%2CMAPREDUCE%2CHDDS%2CSUBM

Unassigned Hadoop jiras with patch available

I am using this jira filter to find jiras with patch available but unassigned.

[jira] [Resolved] (HDFS-14690) java.io.IOException: Unable to close file because the last block does not have enough number of replicas.

[ https://issues.apache.org/jira/browse/HDFS-14690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-14690. Resolution: Not A Bug > java.io.IOException: Unable to close file because the last bl

[jira] [Resolved] (HDFS-14496) Backport HDFS-13056 into branch-2.8/2.9/2

[ https://issues.apache.org/jira/browse/HDFS-14496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-14496. Resolution: Duplicate > Backport HDFS-13056 into branch-2.8/2.

[jira] [Resolved] (HDFS-14641) Possible Intermediate Block Report Race Condition

[ https://issues.apache.org/jira/browse/HDFS-14641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-14641. Resolution: Invalid [~belugabehr] I think this is the expected behavior. COMITTED means

[jira] [Resolved] (HDFS-10927) Lease Recovery: File not getting closed on HDFS when block write operation fails

[ https://issues.apache.org/jira/browse/HDFS-10927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-10927. Resolution: Duplicate Resolve as adup. Thanks everyone for chiming in! > Lease Recov

[jira] [Created] (HDFS-14682) TestStateStoreFileSystem failed because /tmp/hadoop/dfs/name/current can't be removed

Wei-Chiu Chuang created HDFS-14682: -- Summary: TestStateStoreFileSystem failed because /tmp/hadoop/dfs/name/current can't be removed Key: HDFS-14682 URL: https://issues.apache.org/jira/browse/HDFS-14682

[jira] [Created] (HDFS-14681) TestDisableRouterQuota failed because port 8888 was occupied

Wei-Chiu Chuang created HDFS-14681: -- Summary: TestDisableRouterQuota failed because port was occupied Key: HDFS-14681 URL: https://issues.apache.org/jira/browse/HDFS-14681 Project: Hadoop HDFS

[jira] [Resolved] (HDFS-14425) Native build fails on macos due to jlong in hdfs.c

[ https://issues.apache.org/jira/browse/HDFS-14425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-14425. Resolution: Fixed Fix Version/s: 3.3.0 Merged into trunk. Thanks [~hunhun

Hadoop Storage Sync up Meeting Minutes 07/24

https://docs.google.com/document/d/1jXM5Ujvf-zhcyw_5kiQVx6g-HeKe-YGnFS_1-qFXomI/edit?pli=1# Attendees: Wei-Chiu (Cloudera), Ekanth (Uber), Craig (Target), Fabbri Fabric: mostly mentoring folks. Ad-hoc, self initiating. Google's Summer of Code is too much overhead. meeting every 2-4 weeks.

Re: [DISCUSS] EOL 2.8 or another 2.8.x release?

My bad -- Didn't realize I was looking at the old Hadoop page. Here's the correct list of releases. https://hadoop.apache.org/releases.html On Thu, Jul 25, 2019 at 12:49 AM 张铎(Duo Zhang) wrote: > IIRC we have a 2.8.5 release? > > On the download page: > > 2.8.5 2018 Sep 15 >

[DISCUSS] EOL 2.8 or another 2.8.x release?

The last 2.8 release (2.8.4) was made in the last May, more than a year ago. https://hadoop.apache.org/old/releases.html How do folks feel about the fate of branch-2.8? During the last community meetup in June, it sounds like most users are still on 2.8 or even 2.7, so I don't think we want to

[DISCUSS] Prefer Github PR Integration over patch in JIRA

Historically, Hadoop developers create patches and attache them to JIRA, andthen the Yetus bot runs precommit against the patch in the JIRA. The Github PR is more convenient for code review and less hassle for committers to merge a commit. I am proposing for the community to prefer Github PR over

Re: Branch-2 and Tomcat

in other projects (Oozie, for example) On Fri, Jun 28, 2019 at 11:47 PM Wei-Chiu Chuang wrote: > During the Wednesday's meetup, we discussed the plan for the "bridge > release" branch-2.10. > > A smooth upgrade path (that is, rolling upgrade) from 2.10 to Hadoop 3 was > called

HDFS/Cloud connector Bi-weekly online sync

Hi, We will be having the next HDFS/Cloud connector online meetup later this week (US Pacific time Wednesday 10pm or GMT+8 1PM Thursay) We had another online sync two weeks ago, with developers from a few Cloudera, Didi, and JD.com, and it was a good meetup. We kept a meeting minutes in this doc

Branch-2 and Tomcat

During the Wednesday's meetup, we discussed the plan for the "bridge release" branch-2.10. A smooth upgrade path (that is, rolling upgrade) from 2.10 to Hadoop 3 was called out as the most critical release criteria. I am wondering if Tomcat (as well as other dependency upgrade) upgrade should be

[jira] [Created] (HDFS-14615) NPE writing edit logs

Wei-Chiu Chuang created HDFS-14615: -- Summary: NPE writing edit logs Key: HDFS-14615 URL: https://issues.apache.org/jira/browse/HDFS-14615 Project: Hadoop HDFS Issue Type: Bug

[jira] [Created] (HDFS-14614) Add Secure Flag for DataNode Web UI Cookies

Wei-Chiu Chuang created HDFS-14614: -- Summary: Add Secure Flag for DataNode Web UI Cookies Key: HDFS-14614 URL: https://issues.apache.org/jira/browse/HDFS-14614 Project: Hadoop HDFS Issue

No Hadoop Storage community sync-up today

Hi fellow HDFS/cloud connector developers, As of now it looks like I am the only volunteer to host the Hadoop Storage community sync-up calls. There is a calendar invite for today 9am PDT (3 hours from now) but since it conflicts with the Hadoop Contributors Meetup, let's skip this time. Please

Re: HDFS Performance Improvement JIRAs: reviewers wanted

le, etc, in different categories. > > Spark community maintains a dashboard https://spark-prs.appspot.com/ > which > > seems very useful. > > > > Chao > > > > On Sun, Jun 23, 2019 at 8:24 PM Wei-Chiu Chuang > > wrote: > > > > > Thank you

Re: HDFS Performance Improvement JIRAs: reviewers wanted

Like I see a few folks from Xiaomi making really good work there, and I'd be interested to learn more. b. Beyond that, I think we should also scan some BUG JIRAs (for instance > HDFS-12862) reported but not fixed up to now. > Thanks Wei-Chiu again. > > Best Regards, > Hexiaoqiao > &

[jira] [Created] (HDFS-14599) HDFS-12487 breaks test TestDiskBalancer.testDiskBalancerWithFedClusterWithOneNameServiceEmpty

Wei-Chiu Chuang created HDFS-14599: -- Summary: HDFS-12487 breaks test TestDiskBalancer.testDiskBalancerWithFedClusterWithOneNameServiceEmpty Key: HDFS-14599 URL: https://issues.apache.org/jira/browse/HDFS-14599

[jira] [Created] (HDFS-14598) Findbugs warning caused by HDFS-12487

Wei-Chiu Chuang created HDFS-14598: -- Summary: Findbugs warning caused by HDFS-12487 Key: HDFS-14598 URL: https://issues.apache.org/jira/browse/HDFS-14598 Project: Hadoop HDFS Issue Type

HDFS Performance Improvement JIRAs: reviewers wanted

I spent the past week going through most of the jiras with a patch attached in the past, and turned up some really good stuff to helps improve HDFS performance. The list of jiras are listed in the following spreadsheet. If you are interested in reviewing those jiras, please update the following

[jira] [Created] (HDFS-14595) HDFS-11848 breaks API compatibility

Wei-Chiu Chuang created HDFS-14595: -- Summary: HDFS-11848 breaks API compatibility Key: HDFS-14595 URL: https://issues.apache.org/jira/browse/HDFS-14595 Project: Hadoop HDFS Issue Type: Bug

HDFS-13287 broke the build; reverted

Sorry for the cross-post. If you find your build breaks, it's on me. I committed HDFS-13287 which resulted in compilation error in trunk/branch-3.2/branch-3.1. The commit has since been reverted. Thanks Akira for identifying the issue!

[jira] [Reopened] (HDFS-13287) TestINodeFile#testGetBlockType results in NPE when run alone

[ https://issues.apache.org/jira/browse/HDFS-13287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reopened HDFS-13287: [~aajisaka] thanks! I've reverted the commit from trunk. Reopen the jira to correct the fix

Re: [DISCUSS] A unified and open Hadoop community sync up schedule?

Thanks Wangda, I just like to make a correction -- the .ics calendar file says the first Wednesday for HDFS/cloud connector is in Mandarin whereas on the gdoc is to host it on the third Wednesday. On Tue, Jun 18, 2019 at 5:29 PM Wangda Tan wrote: > Hi Folks, > > I just updated doc: > >

[jira] [Created] (HDFS-14578) AvailableSpaceBlockPlacementPolicy always prefers local node

Wei-Chiu Chuang created HDFS-14578: -- Summary: AvailableSpaceBlockPlacementPolicy always prefers local node Key: HDFS-14578 URL: https://issues.apache.org/jira/browse/HDFS-14578 Project: Hadoop HDFS

[jira] [Resolved] (HDFS-14535) The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read

[ https://issues.apache.org/jira/browse/HDFS-14535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-14535. Resolution: Fixed Fix Version/s: 3.1.3 2.9.3

Re: HDFS Scalability Limit?

at the per-datanode limit is. > > Thanks and 73 > Kihwal > > On Thu, Jun 13, 2019 at 1:57 PM Wei-Chiu Chuang > wrote: > >> Hi community, >> >> I am currently drafting a HDFS scalability guideline doc, and I'd like to >> understand any data

[jira] [Created] (HDFS-14574) [distcp] Add ability to increase the replication factor for fileList.seq

Wei-Chiu Chuang created HDFS-14574: -- Summary: [distcp] Add ability to increase the replication factor for fileList.seq Key: HDFS-14574 URL: https://issues.apache.org/jira/browse/HDFS-14574 Project

HDFS Scalability Limit?

Hi community, I am currently drafting a HDFS scalability guideline doc, and I'd like to understand any data points regarding HDFS scalability limit. I'd like to share it publicly eventually. As an example, through my workplace, and through community chatters, I am aware that HDFS is capable of

Re: [DISCUSS] HDFS roadmap/wish list

m > dfs.datanode.balance.bandwidthPerSecbut only for re-replication. > I am pretty sure I've got people asking about this before a few times. > > Thanks and regards > JL > > Le lun. 10 juin 2019 à 19:08, Wei-Chiu Chuang > a écrit : > >> Hi! >> >> I am solic

Re: [DISCUSS] HDFS roadmap/wish list

, for that matter) On Mon, Jun 10, 2019 at 12:11 PM Sudeep Singh Thakur < sudeepthaku...@gmail.com> wrote: > Hi , > > Examples are most helpful for developer. Please add examples as much as we > can. > > Thanks > Sudeep Thakur > > On Mon, Jun 10, 2019, 10:38 PM Wei-Chiu

[jira] [Created] (HDFS-14557) JournalNode error: Can't scan a pre-transactional edit log

Wei-Chiu Chuang created HDFS-14557: -- Summary: JournalNode error: Can't scan a pre-transactional edit log Key: HDFS-14557 URL: https://issues.apache.org/jira/browse/HDFS-14557 Project: Hadoop HDFS

[DISCUSS] HDFS roadmap/wish list

Hi! I am soliciting feedbacks for HDFS roadmap items and wish list in the future Hadoop releases. A community meetup is happening soon, and perhaps

[jira] [Created] (HDFS-14510) Backport HDFS-13087 to branch-2 (Snapshotted encryption zone information should be immutable)

Wei-Chiu Chuang created HDFS-14510: -- Summary: Backport HDFS-13087 to branch-2 (Snapshotted encryption zone information should be immutable) Key: HDFS-14510 URL: https://issues.apache.org/jira/browse/HDFS-14510

[jira] [Created] (HDFS-14496) Backport HDFS-13056 into branch-2.8/2.9/2

Wei-Chiu Chuang created HDFS-14496: -- Summary: Backport HDFS-13056 into branch-2.8/2.9/2 Key: HDFS-14496 URL: https://issues.apache.org/jira/browse/HDFS-14496 Project: Hadoop HDFS Issue Type

[jira] [Created] (HDFS-14492) Snapshot memory leak

Wei-Chiu Chuang created HDFS-14492: -- Summary: Snapshot memory leak Key: HDFS-14492 URL: https://issues.apache.org/jira/browse/HDFS-14492 Project: Hadoop HDFS Issue Type: Bug

[jira] [Resolved] (HDFS-14485) Optionally disable LazyPersistFileScrubber thread

[ https://issues.apache.org/jira/browse/HDFS-14485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-14485. Resolution: Invalid > Optionally disable LazyPersistFileScrubber thr

[jira] [Created] (HDFS-14485) Optionally disable LazyPersistFileScrubber thread

Wei-Chiu Chuang created HDFS-14485: -- Summary: Optionally disable LazyPersistFileScrubber thread Key: HDFS-14485 URL: https://issues.apache.org/jira/browse/HDFS-14485 Project: Hadoop HDFS

[jira] [Reopened] (HDFS-10477) Stop decommission a rack of DataNodes caused NameNode fail over to standby

[ https://issues.apache.org/jira/browse/HDFS-10477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reopened HDFS-10477: Reopen to run precommit for the branch-2.8 patch. Thanks [~starphin] > Stop decommiss

[jira] [Resolved] (HDFS-10585) Incorrect offset/length calculation in pipeline recovery causes block corruption

[ https://issues.apache.org/jira/browse/HDFS-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-10585. Resolution: Duplicate > Incorrect offset/length calculation in pipeline recovery cau

[jira] [Created] (HDFS-14414) Clean up findbugs warning in branch-2

Wei-Chiu Chuang created HDFS-14414: -- Summary: Clean up findbugs warning in branch-2 Key: HDFS-14414 URL: https://issues.apache.org/jira/browse/HDFS-14414 Project: Hadoop HDFS Issue Type

Re: Update guava to 27.0-jre in hadoop-project

+1 I watched Gabor working on this and this is a very comprehensive, which also includes testing in downstreamers (HBase and Hive). So very good work. Thanks! On Wed, Apr 3, 2019 at 3:41 AM Steve Loughran wrote: > I am taking silence as happiness here. > > +1 to the patch > > On Tue, Apr 2,

[jira] [Reopened] (HDFS-14261) Kerberize JournalNodeSyncer unit test

[ https://issues.apache.org/jira/browse/HDFS-14261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reopened HDFS-14261: Reopened. The test is failing almost consistently. [~smeng] could you please look

[jira] [Created] (HDFS-14312) Scale test KMS using kms audit log

Wei-Chiu Chuang created HDFS-14312: -- Summary: Scale test KMS using kms audit log Key: HDFS-14312 URL: https://issues.apache.org/jira/browse/HDFS-14312 Project: Hadoop HDFS Issue Type: Sub

Re: [DISCUSS] Hadoop RPC encryption performance improvements

. On Mon, Feb 18, 2019 at 1:37 PM Jonathan Eagles wrote: > Filed https://issues.apache.org/jira/browse/HADOOP-16119 to > implemented KMS on Hadoop RPC. It seemed there were interested parties > involved in the development of this feature. > > On Fri, Nov 2, 2018 at 4:35 PM Wei-Chi

[jira] [Created] (HDFS-14283) DFSInputStream to prefer cached replica

Wei-Chiu Chuang created HDFS-14283: -- Summary: DFSInputStream to prefer cached replica Key: HDFS-14283 URL: https://issues.apache.org/jira/browse/HDFS-14283 Project: Hadoop HDFS Issue Type

[jira] [Created] (HDFS-14277) [SBN read] Observer benchmark results

Wei-Chiu Chuang created HDFS-14277: -- Summary: [SBN read] Observer benchmark results Key: HDFS-14277 URL: https://issues.apache.org/jira/browse/HDFS-14277 Project: Hadoop HDFS Issue Type

[jira] [Created] (HDFS-14276) [SBN read] Reduce tailing overhead

Wei-Chiu Chuang created HDFS-14276: -- Summary: [SBN read] Reduce tailing overhead Key: HDFS-14276 URL: https://issues.apache.org/jira/browse/HDFS-14276 Project: Hadoop HDFS Issue Type

[jira] [Resolved] (HDFS-14269) [SBN read] Observer node switches back to Standby after restart

[ https://issues.apache.org/jira/browse/HDFS-14269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-14269. Resolution: Won't Fix > [SBN read] Observer node switches back to Standby after rest

[jira] [Created] (HDFS-14272) [SBN read] HDFS command line tools does not guarantee consistency

Wei-Chiu Chuang created HDFS-14272: -- Summary: [SBN read] HDFS command line tools does not guarantee consistency Key: HDFS-14272 URL: https://issues.apache.org/jira/browse/HDFS-14272 Project: Hadoop

[jira] [Created] (HDFS-14271) [SBN read] StandbyException is logged if Observer is the first NameNode

Wei-Chiu Chuang created HDFS-14271: -- Summary: [SBN read] StandbyException is logged if Observer is the first NameNode Key: HDFS-14271 URL: https://issues.apache.org/jira/browse/HDFS-14271 Project

[jira] [Created] (HDFS-14269) [SBN read] Observer node switches back to Standby after restart

Wei-Chiu Chuang created HDFS-14269: -- Summary: [SBN read] Observer node switches back to Standby after restart Key: HDFS-14269 URL: https://issues.apache.org/jira/browse/HDFS-14269 Project: Hadoop

Re: DFSClient/DistriburedFileSystem fault injection?

For what is worth, At Cloudera we have an internal fault injection tool that is based on Chaos Monkey. We use it to kill disks or kill nodes, for example. So it doesn't instrument HDFS directly. On Mon, Feb 11, 2019 at 3:42 PM Sergey Shelukhin wrote: > Hi. > I've been looking for a client-side

[jira] [Resolved] (HDFS-14140) JournalNodeSyncer authentication is failing in secure cluster

[ https://issues.apache.org/jira/browse/HDFS-14140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-14140. Resolution: Fixed Fix Version/s: 3.1.3 3.2.1

<    1   2   3   4   5   6   7   8   >