Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

Hi folks, apologies for being late to the release conversation, but I think we need to get HDFS-16923 into 3.3.5. HDFS-16732 , which also went into 3.3.5, introduced an issue whereby Observer NameNo

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

Thanks Steve. I see now that the branch cut was way back in October so I definitely understand your frustration here! This made me realize that HDFS-16832 , which resolves a very similar issue as the aforementioned HDFS-16923, is also missing from

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

ct were able to get completed > > within timeout). > > > > > > On Sat, Mar 4, 2023 at 4:29 AM Steve Loughran > > > > > wrote: > > > > > On Sat, 4 Mar 2023 at 01:47, Erik Krogen wrote: > > > > > > > Thanks Steve. I see no

Re: [VOTE] Release Apache Hadoop 2.7.6 (RC0)

Thanks putting this together, Konstantin. +1 (non-binding) * Verified signatures, MD5, and SHA-* checksums for bin and src tarball * Started 2-node HDFS and 1-node YARN clusters * Tested basic file operations, copy(To/From)Local, mkdir, ls, etc. via hdfs and webhdfs protocol * Verified Web UIs *

Re: [DISCUSS] Hadoop RPC encryption performance improvements

sults from RPCCallBenchmark. Clearly the performance of the approach is crucial in making the decision and we should ensure that any comparisons made are apples-to-apples with the same test setup. Thanks, Erik Krogen LinkedIn From: Wei-Chiu Chuang Sent: Wednesday, Octobe

Re: [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

+1 (non-binding) from me I've been a part of the development of this feature throughout its existence, and have reviewed most of the code committed to the branch. The implementation has been conducted carefully and thoughtfully, and tested at a variety of sites. This feature has no potential n

Hadoop Contributor Meetup Hosted by LinkedIn

Hello fellow Hadoopers! LinkedIn is pleased to be hosting an all-day meetup event for Hadoop contributors and community members to gather together to share information and insights on the present, past and future of the Hadoop software we all know and love. The day will begin with a series of

Re: [DISCUSS] Release 3.0.4 or branch-3.0 EOL

I'd like to revive this discussion. Today I see committers variously skipping backports to branch-3.0 and/or branch-3.1 (when backporting to branch-2, for example) and I'm concerned that we will have divergence between what commits are present in which branches. Wei-Chiu, is Cloudera still reba

Re: [DISCUSS] Release 3.0.4 or branch-3.0 EOL

I'd like to revive this discussion. Today I see committers variously skipping backports to branch-3.0 and/or branch-3.1 (when backporting to branch-2, for example) and I'm concerned that we will have divergence between what commits are present in which branches. Wei-Chiu, is Cloudera still rebasin

Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

+1 thanks for initiating an official vote Wangda! From: Jonathan Eagles Sent: Wednesday, August 21, 2019 8:22 AM To: Sunil Govindan Cc: Wangda Tan ; yarn-dev ; Hdfs-dev ; mapreduce-dev ; Hadoop Common Subject: Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL +1

Re: [VOTE] Release Apache Hadoop 2.10.0 (RC1)

Thanks for putting this together, Jonathan! I noticed that the RELEASENOTES.md makes no mention of any of the major features you mentioned in your email about the RC. Is this expected? I guess it is caused by the lack of a release note on the JIRAs for those features. I also noticed that buildi

Re: [VOTE] Release Apache Hadoop 2.10.0 (RC1)

Sounds good to me, thank you for addressing the concerns Jonathan. +1 from me. From: Masatake Iwasaki Sent: Monday, October 28, 2019 7:52 AM To: Jonathan Hung ; Hdfs-dev ; mapreduce-dev ; yarn-dev ; Hadoop Common Subject: Re: [VOTE] Release Apache Hadoop 2.10.

Re: [DISCUSS] Making 2.10 the last minor 2.x release

I'm in support of this. The current scheme is confusing, and as you mentioned, is making the backport strategy less clear. It reminds me of the branch-2.8 vs. branch-2 (destined for 2.9) days when various fixes would make it into one or the other. One other action item would be to do a quick verif

Re: About 2.7.4 Release

List LGTM Konstantin! Let's say that we will only create a new tracking JIRA for patches which do not backport cleanly, to avoid having too many lying around. Otherwise we can directly attach to old ticket. If a clean backport does happen to break a test the nightly build will help us catch it. E

LinkedIn Dynamometer Tool (was About 2.7.4 Release)

>>>>>> HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311> >>> >>>>>>>> Seems like a new feature. It helps failover to standby node when >>> >>>>>>>> primary >>> >>>&

Re: LinkedIn Dynamometer Tool (was About 2.7.4 Release)

ow the “fake out DNs into thinking they are storing > data” — works? Or I can be patient and read your blog post too. > > Thanks > Anu > > > > > > > On 7/20/17, 10:42 AM, "Erik Krogen" wrote: > > >forking off of the 2.7.4 release thread to answer

Re: LinkedIn Dynamometer Tool (was About 2.7.4 Release)

thanks sanjay > On Jul 20, 2017, at 10:42 AM, Erik Krogen wrote: > > forking off of the 2.7.4 release thread to answer this question about > Dynamometer > > Dynamometer is a tool developed at LinkedIn for scale testing HDFS, > specifically the

Re: Should ToolRunner call UserGroupInformation.setConfiguration?

I have found this troublesome as well, +1 to this unless there is a good reason not to. Erik On 7/26/17, 10:05 PM, "John Zhuge" wrote: The static UGI.conf is set to a new Configuration object in UGI.ensureInitialized if setConfiguration is not already called. ToolRunner.c

Re: [VOTE] Release Apache Hadoop 2.7.4 (RC0)

+1 (non-binding) - Verified checksum and signature of source and binary tarballs - Built from source, macOS Sierra 10.12.3 and RHEL 6.6, Oracle JDK 1.8.0_72 - Ran basic HDFS commands and some MR example jobs (using 2.7.4 client) against a 2.6.1 cluster - Started MiniCluster, ran basic HDFS comman

Re: [VOTE] Release Apache Hadoop 2.7.4 (RC0)

not see any noticeable performance regressions as compared to previous releases. Erik On 8/4/17, 11:38 AM, "Erik Krogen" wrote: +1 (non-binding) - Verified checksum and signature of source and binary tarballs - Built from source, macOS Sierra 10.12.3 and RHEL 6.6,

Re: [DISCUSS] Can we make our precommit test robust to dependency changes while staying usable?

I am +1 for a way to specify additional modules that should be run. Always running all module dependencies would be prohibitively expensive (especially for hadoop-common patches) but developers should have a good understanding of which patches are more high-risk for downstream consumers and can

Re: VOTE] Release Apache Hadoop 2.7.5 (RC0)

+1 (non-binding) - Verified signatures, MD5, RMD160, SHA* for bin and src tarballs - Built from source on macOS 10.12.6 and RHEL 6.6 - Ran local HDFS cluster, ran basic commands, verified read and write capability. - Ran 3000 node cluster via Dynamometer and do not see significant performance va

Re: [VOTE] Release Apache Hadoop 2.7.5 (RC1)

+1 (non-binding) • Verified that the missed JIRAs now show up in the release notes • Clicked around the documentation included in the bin tarball • Verified hashes and signatures for the bin and src tarball • Built from source on RHEL 6.6 • Ran small HDFS cluster, executed basic commands, poked aro

Re: Missing some trunk commit history

Eric, if you prefer to view commits in order of author date, you can use this one-liner courtesy of this site[1]: git log --pretty="format:%at commit %C(yellow)%H%Creset\nAuthor: %an <%ae>\nDate: %aD\n\n %s\n" | sort -r | cut -d" " -f2- | sed -e "s/\\\n/\\`echo -e '\n\r'`/g" | tr -d '\15\32' |

Re: [ANNOUNCE] New Hadoop Committer - Simbarashe Dzinamarira

Congratulations Simba! Thanks for the great work you've done on making HDFS more scalable! On Mon, Oct 2, 2023 at 4:31 PM Iñigo Goiri wrote: > I am pleased to announce that Simbarashe Dzinamarira has been elected as a > committer on the Apache Hadoop project. > We appreciate all of Simbarashe's

[jira] [Resolved] (HADOOP-18446) Add a re-queue metric to RpcMetrics.java to quantify the number of re-queue RPCs

[ https://issues.apache.org/jira/browse/HADOOP-18446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen resolved HADOOP-18446. -- Fix Version/s: 3.4.0 Resolution: Fixed > Add a re-queue metric to RpcMetrics.java

[jira] [Created] (HADOOP-15311) HttpServer2 needs a way to configure the acceptor/selector count

Erik Krogen created HADOOP-15311: Summary: HttpServer2 needs a way to configure the acceptor/selector count Key: HADOOP-15311 URL: https://issues.apache.org/jira/browse/HADOOP-15311 Project: Hadoop

[jira] [Created] (HADOOP-15481) Emit FairCallQueue stats as metrics

Erik Krogen created HADOOP-15481: Summary: Emit FairCallQueue stats as metrics Key: HADOOP-15481 URL: https://issues.apache.org/jira/browse/HADOOP-15481 Project: Hadoop Common Issue Type

[jira] [Created] (HADOOP-15637) LocalFs#listLocatedStatus does not filter out hidden .crc files

Erik Krogen created HADOOP-15637: Summary: LocalFs#listLocatedStatus does not filter out hidden .crc files Key: HADOOP-15637 URL: https://issues.apache.org/jira/browse/HADOOP-15637 Project: Hadoop

[jira] [Created] (HADOOP-15726) Create utility to limit frequency of log statements

Erik Krogen created HADOOP-15726: Summary: Create utility to limit frequency of log statements Key: HADOOP-15726 URL: https://issues.apache.org/jira/browse/HADOOP-15726 Project: Hadoop Common

[jira] [Created] (HADOOP-16013) DecayRpcScheduler decay thread should run as a daemon

Erik Krogen created HADOOP-16013: Summary: DecayRpcScheduler decay thread should run as a daemon Key: HADOOP-16013 URL: https://issues.apache.org/jira/browse/HADOOP-16013 Project: Hadoop Common

[jira] [Created] (HADOOP-16097) Provide proper documentation for FairCallQueue

Erik Krogen created HADOOP-16097: Summary: Provide proper documentation for FairCallQueue Key: HADOOP-16097 URL: https://issues.apache.org/jira/browse/HADOOP-16097 Project: Hadoop Common

[jira] [Resolved] (HADOOP-12382) Add Documentation for FairCallQueue

[ https://issues.apache.org/jira/browse/HADOOP-12382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen resolved HADOOP-12382. -- Resolution: Fixed > Add Documentation for FairCallQu

[jira] [Reopened] (HADOOP-12382) Add Documentation for FairCallQueue

[ https://issues.apache.org/jira/browse/HADOOP-12382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen reopened HADOOP-12382: -- Assignee: (was: Chris Li) > Add Documentation for FairCallQu

[jira] [Resolved] (HADOOP-12382) Add Documentation for FairCallQueue

[ https://issues.apache.org/jira/browse/HADOOP-12382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen resolved HADOOP-12382. -- Resolution: Duplicate > Add Documentation for FairCallQu

[jira] [Resolved] (HADOOP-9640) RPC Congestion Control with FairCallQueue

[ https://issues.apache.org/jira/browse/HADOOP-9640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen resolved HADOOP-9640. - Resolution: Fixed Target Version/s: (was: ) > RPC Congestion Control w

[jira] [Created] (HADOOP-16192) CallQueue doesn't perform backoff when add() is used

Erik Krogen created HADOOP-16192: Summary: CallQueue doesn't perform backoff when add() is used Key: HADOOP-16192 URL: https://issues.apache.org/jira/browse/HADOOP-16192 Project: Hadoop C

[jira] [Created] (HADOOP-16224) RPC FairCallQueue Reservation Support

Erik Krogen created HADOOP-16224: Summary: RPC FairCallQueue Reservation Support Key: HADOOP-16224 URL: https://issues.apache.org/jira/browse/HADOOP-16224 Project: Hadoop Common Issue Type

[jira] [Created] (HADOOP-16245) Enabling SSL within LdapGroupsMapping can break system SSL configs

Erik Krogen created HADOOP-16245: Summary: Enabling SSL within LdapGroupsMapping can break system SSL configs Key: HADOOP-16245 URL: https://issues.apache.org/jira/browse/HADOOP-16245 Project: Hadoop

[jira] [Created] (HADOOP-16290) Enable RpcMetrics units to be configurable

Erik Krogen created HADOOP-16290: Summary: Enable RpcMetrics units to be configurable Key: HADOOP-16290 URL: https://issues.apache.org/jira/browse/HADOOP-16290 Project: Hadoop Common Issue

[jira] [Created] (HADOOP-16345) Potential NPE when instantiating FairCallQueue metrics

Erik Krogen created HADOOP-16345: Summary: Potential NPE when instantiating FairCallQueue metrics Key: HADOOP-16345 URL: https://issues.apache.org/jira/browse/HADOOP-16345 Project: Hadoop Common

[jira] [Resolved] (HADOOP-16266) Add more fine-grained processing time metrics to the RPC layer

[ https://issues.apache.org/jira/browse/HADOOP-16266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen resolved HADOOP-16266. -- Resolution: Fixed Just noticed this was never re-resolved after the follow-on patch, doing

[jira] [Created] (HADOOP-16459) Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the RPC layer" to branch-2

Erik Krogen created HADOOP-16459: Summary: Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the RPC layer" to branch-2 Key: HADOOP-16459 URL: https://issues.apache.org/jira/bro

[jira] [Created] (HADOOP-13782) Make MutableRates metrics thread-local write, aggregate-on-read

Erik Krogen created HADOOP-13782: Summary: Make MutableRates metrics thread-local write, aggregate-on-read Key: HADOOP-13782 URL: https://issues.apache.org/jira/browse/HADOOP-13782 Project: Hadoop

[jira] [Created] (HADOOP-13804) MutableStat mean loses accuracy if add(long, long) is used

Erik Krogen created HADOOP-13804: Summary: MutableStat mean loses accuracy if add(long, long) is used Key: HADOOP-13804 URL: https://issues.apache.org/jira/browse/HADOOP-13804 Project: Hadoop Common

[jira] [Created] (HADOOP-14136) Default FS For ViewFS

Erik Krogen created HADOOP-14136: Summary: Default FS For ViewFS Key: HADOOP-14136 URL: https://issues.apache.org/jira/browse/HADOOP-14136 Project: Hadoop Common Issue Type: Improvement

[jira] [Created] (HADOOP-14169) Implement listStatusIterator, listLocatedStatus for ViewFs

Erik Krogen created HADOOP-14169: Summary: Implement listStatusIterator, listLocatedStatus for ViewFs Key: HADOOP-14169 URL: https://issues.apache.org/jira/browse/HADOOP-14169 Project: Hadoop Common

[jira] [Created] (HADOOP-14211) ChRootedFs is too aggressive about enforcing "needsAuthority"

Erik Krogen created HADOOP-14211: Summary: ChRootedFs is too aggressive about enforcing "needsAuthority" Key: HADOOP-14211 URL: https://issues.apache.org/jira/browse/HADOOP-14211 Proje

[jira] [Created] (HADOOP-14276) Add a nanosecond API to Time/Timer/FakeTimer

Erik Krogen created HADOOP-14276: Summary: Add a nanosecond API to Time/Timer/FakeTimer Key: HADOOP-14276 URL: https://issues.apache.org/jira/browse/HADOOP-14276 Project: Hadoop Common Issue

[jira] [Created] (HADOOP-14408) Backport to 2.7: HADOOP-12975 Add jitter to du thread

Erik Krogen created HADOOP-14408: Summary: Backport to 2.7: HADOOP-12975 Add jitter to du thread Key: HADOOP-14408 URL: https://issues.apache.org/jira/browse/HADOOP-14408 Project: Hadoop Common

[jira] [Created] (HADOOP-14413) Add Javadoc comment for jitter parameter on CachingGetSpaceUsed

Erik Krogen created HADOOP-14413: Summary: Add Javadoc comment for jitter parameter on CachingGetSpaceUsed Key: HADOOP-14413 URL: https://issues.apache.org/jira/browse/HADOOP-14413 Project: Hadoop

[jira] [Created] (HADOOP-14502) Confusion/name conflict between NameNodeActivity#BlockReportNumOps and RpcDetailedActivity#BlockReportNumOps

Erik Krogen created HADOOP-14502: Summary: Confusion/name conflict between NameNodeActivity#BlockReportNumOps and RpcDetailedActivity#BlockReportNumOps Key: HADOOP-14502 URL: https://issues.apache.org/jira/browse

[jira] [Resolved] (HADOOP-14492) RpcDetailedMetrics and NameNodeMetrics use different rate metrics abstraction cause the Xavgtime confused

[ https://issues.apache.org/jira/browse/HADOOP-14492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen resolved HADOOP-14492. -- Resolution: Invalid > RpcDetailedMetrics and NameNodeMetrics use different rate metr

[jira] [Created] (HADOOP-14827) Allow StopWatch to accept a Timer parameter for tests

Erik Krogen created HADOOP-14827: Summary: Allow StopWatch to accept a Timer parameter for tests Key: HADOOP-14827 URL: https://issues.apache.org/jira/browse/HADOOP-14827 Project: Hadoop Common

[jira] [Created] (HADOOP-14970) MiniHadoopClusterManager doesn't respect lack of format option

Erik Krogen created HADOOP-14970: Summary: MiniHadoopClusterManager doesn't respect lack of format option Key: HADOOP-14970 URL: https://issues.apache.org/jira/browse/HADOOP-14970 Project: H

[jira] [Created] (HADOOP-14989) Multiple metrics2 sinks (incl JMX) result in inconsistent Mutable(Stat|Rate) values

Erik Krogen created HADOOP-14989: Summary: Multiple metrics2 sinks (incl JMX) result in inconsistent Mutable(Stat|Rate) values Key: HADOOP-14989 URL: https://issues.apache.org/jira/browse/HADOOP-14989

[jira] [Created] (HADOOP-15008) Metrics sinks may emit too frequently if multiple sink periods are configured

Erik Krogen created HADOOP-15008: Summary: Metrics sinks may emit too frequently if multiple sink periods are configured Key: HADOOP-15008 URL: https://issues.apache.org/jira/browse/HADOOP-15008