Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

[Chris Douglas]

Yeah, consistently failing nightly builds are just burning resources.
Until someone starts working to fix it, we shouldn't keep submitting
the job. Too bad; I thought build times and resource usage were
becoming more manageable on branch-2.

If anyone has cycles to work on this, the job is here [1]. -C

[1]: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/

On Tue, May 15, 2018 at 10:27 AM, Allen Wittenauer
<a...@effectivemachines.com> wrote:
>
>> On May 15, 2018, at 10:16 AM, Chris Douglas <cdoug...@apache.org> wrote:
>>
>> They've been failing for a long time. It can't install bats, and
>> that's fatal? -C
>
>
> The bats error is new and causes the build to fail enough that it 
> produces the email output.  For the past few months, it hasn’t been producing 
> email output at all because the builds have been timing out.  (The last 
> ‘good’ report was Feb 26.)  Since no one [*] is paying attention to them 
> enough to notice, I figured it was better to free up the cycles for the rest 
> of the ASF.
>
> * - I noticed a while back, but for various reasons I’ve mostly moved to only 
> working on Hadoop things where I’m getting paid.

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

[Chris Douglas]

They've been failing for a long time. It can't install bats, and
that's fatal? -C

On Tue, May 15, 2018 at 9:43 AM, Allen Wittenauer
 wrote:
>
>
> FYI:
>
> I’m going to disable the branch-2 nightly jobs.
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [RESULT][VOTE] Merge HDDS (HDFS-7240) *code* into trunk

[Chris Douglas]

Thank you for doing this, Owen. If the result is the same, then the
rebased version is much easier to work with. If not this, then we can
squash the branch into a single merge commit. The current state would
be a distant, third choice.

+1 -C

On Thu, Apr 26, 2018 at 5:49 AM, Owen O'Malley <owen.omal...@gmail.com> wrote:
> Ok, it was bad enough that I spent yesterday afternoon hacking a fix for it.
>
> I basically did a git rebase with always picking the current master version
> and then put a new fixup commit to bring the branches into sync.
>
> The original - https://github.com/omalley/hadoop/tree/hdfs-7240
> The rebase hack - https://github.com/omalley/hadoop/tree/hdfs-7240-rebase
>
> Compare the git log --graph difference for one of the recent commits:
>
> | * | | | | | | | | | | | | | | | | | | | | | | | commit
> 919ae746c3a4a424a77e6359af0abe3d7d2c1541
> | | | | | | | | | | | | | | | | | | | | | | | | | Author: Weiwei Yang <
> w...@apache.org>
> | | | | | | | | | | | | | | | | | | | | | | | | | Date:   Wed Nov 1
> 17:17:03 2017 +0800
> | | | | | | | | | | | | | | | | | | | | | | | | |
> | | | | | | | | | | | | | | | | | | | | | | | | | HDFS-12750. Ozone:
> Fix TestStorageContainerManager#testBlockDeletionTransactions. Contributed
> by Xiaoyu Yao.
> | | | | | | | | | | | | | | | | | | | | | | | | |
>
> versus
>
> * commit 05592658078acef10c3995940510ff09792e
> | Author: Weiwei Yang <w...@apache.org>
> | Date:   Wed Nov 1 17:17:03 2017 +0800
> |
> | HDFS-12750. Ozone: Fix
> TestStorageContainerManager#testBlockDeletionTransactions. Contributed by
> Xiaoyu Yao.
> |
>
> I'd like permission to edit the hadoop git history to make the change.
>
> Thoughts?
>
> ... Owen
>
> On Wed, Apr 25, 2018 at 11:47 AM, Xiaoyu Yao <x...@hortonworks.com> wrote:
>
>> We followed the process laid out in a mail thread from 2015 that had two
>> workflows proposed.
>>
>> “Git rebase” workflows and “Git merge” workflows.
>> Ozone had followed git merge workflow, which merged trunk at a regular
>> frequency into ozone, and then ozone was merged back.
>>
>> Here is the mail that we followed for the merge process.
>> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>>
>> Thanks
>> Xiaoyu
>>
>>
>>
>>
>>
>> *From: *Owen O'Malley <owen.omal...@gmail.com>
>> *Date: *Wednesday, April 25, 2018 at 11:15 AM
>> *To: *Chris Douglas <cdoug...@apache.org>
>> *Cc: *Xiaoyu Yao <x...@hortonworks.com>, Jitendra Pandey <
>> jiten...@hortonworks.com>, Hdfs-dev <hdfs-dev@hadoop.apache.org>
>> *Subject: *Re: [RESULT][VOTE] Merge HDDS (HDFS-7240) *code* into trunk
>>
>>
>>
>> *sigh*
>>
>>
>>
>> I asked you guys to rebase this before merge. It is tempting to squash it
>> into a single commit.
>>
>>
>>
>> .. Owen
>>
>>
>>
>> On Wed, Apr 25, 2018 at 11:06 AM, Chris Douglas <cdoug...@apache.org>
>> wrote:
>>
>> This really made a mess of trunk. Periodically merging trunk into the
>> HDFS-7240 branch, then merging the whole thing back, created a tangle
>> of references that's very difficult to work with (look at the output
>> of git log --graph).
>>
>> I'm not sure it's even possible to fix this, but this is why feature
>> branches should rebase and create a merge commit with --no-ff. -C
>>
>>
>>
>> On Tue, Apr 24, 2018 at 1:20 PM, Xiaoyu Yao <x...@hortonworks.com> wrote:
>> > I just merged the branch to trunk (834 commits in total)
>> > Again, thanks for all who contributed to HDDS/Ozone!
>> >
>> > Cheers
>> > -Xiaoyu
>> >
>> > On 4/23/18, 7:26 PM, "Jitendra Pandey" <jiten...@hortonworks.com> wrote:
>> >
>> > The vote passes with many +1s (12 committers + 5 contributors) and
>> no -1.
>> >
>> > Thanks everyone for voting.
>> >
>> > On 4/17/18, 5:19 AM, "Jitendra Pandey" <jiten...@hortonworks.com>
>> wrote:
>> >
>> > Hi All,
>> >
>> >The community unanimously voted (https://s.apache.org/
>> HDDSMergeResult) to adopt
>> > HDDS/Ozone as a sub-project of Hadoop, here is the formal
>> vote for code merge.
>> >
>> > Here is a quick summary of the code changes:
>> >
>> > - As decided in the JIRA HDFS-10419, the project has been
>> renamed to Hadoop Distributed Data Store or HDDS.
>> > 

Re: [RESULT][VOTE] Merge HDDS (HDFS-7240) *code* into trunk

[Chris Douglas]

This really made a mess of trunk. Periodically merging trunk into the
HDFS-7240 branch, then merging the whole thing back, created a tangle
of references that's very difficult to work with (look at the output
of git log --graph).

I'm not sure it's even possible to fix this, but this is why feature
branches should rebase and create a merge commit with --no-ff. -C


On Tue, Apr 24, 2018 at 1:20 PM, Xiaoyu Yao  wrote:
> I just merged the branch to trunk (834 commits in total)
> Again, thanks for all who contributed to HDDS/Ozone!
>
> Cheers
> -Xiaoyu
>
> On 4/23/18, 7:26 PM, "Jitendra Pandey"  wrote:
>
> The vote passes with many +1s (12 committers + 5 contributors) and no -1.
>
> Thanks everyone for voting.
>
> On 4/17/18, 5:19 AM, "Jitendra Pandey"  
> wrote:
>
> Hi All,
>
>The community unanimously voted 
> (https://s.apache.org/HDDSMergeResult) to adopt
> HDDS/Ozone as a sub-project of Hadoop, here is the formal vote 
> for code merge.
>
> Here is a quick summary of the code changes:
>
> - As decided in the JIRA HDFS-10419, the project has been renamed 
> to Hadoop Distributed Data Store or HDDS.
> - HDDS becomes a sub-project of Hadoop.
> - Added a Maven profile that disables HDDS compilation by default.
> - The releases of HDDS will be independent of Hadoop and will 
> have no impact to current Hadoop release process.
> - We have made HDDS a loadable module.
> - Cleaned up changes in HDFS/Hadoop Common to make sure HDDS does 
> not impact current users of HDFS.
>
> The vote will run for 7 days, I will start this vote with my +1.
>
> Thanks
> Jitendra
>
>
> 
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>
>
>
>
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>
>

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Adopt HDSL as a new Hadoop subproject

[Chris Douglas]

On Thu, Mar 22, 2018 at 10:23 AM, Andrew Wang <andrew.w...@cloudera.com> wrote:
> We want the git hash to match the contents of the tarball and tag, which is
> beyond what create release does right now. It doesn't do any git stuff.

...and it can't? Even if this remains a manual step, it's not a
significant concession.

> If this vote to adopt a new project also includes merging to trunk (it
> sounds like it?), I feel like we should settle these questions first.

No, this is just measuring consensus so effort arranging the merge is
well-spent. The merge vote will come later. -C

> Best,
> Andrew
>
> On Mar 22, 2018 9:51 AM, "Chris Douglas" <cdoug...@apache.org> wrote:
>
> +1 (binding)
>
> This compromise seems to address most of the concerns raised during
> the discussion. Thanks for proposing and driving this, Owen.
>
> On Thu, Mar 22, 2018 at 9:30 AM, Andrew Wang <andrew.w...@cloudera.com>
> wrote:
>> In Owen's proposal, it says to delete the module from the release branch.
>> We need to do this since the source tarball is our official Apache release
>> artifact, the rest are convenience binaries. So the Maven profile is
>> insufficient for this.
>
> Eliminating manual steps to create a release is desirable, but
> privileging it above all the development efficiencies gained by
> merging to the same repo... we don't cut releases that often.
>
> Moreover, the steps to remove the module don't need to be manual. Once
> we work out the steps, would you be willing to update the
> create-release script? -C
>
>
> On Tue, Mar 20, 2018 at 11:20 AM, Owen O'Malley <owen.omal...@gmail.com>
> wrote:
>> All,
>>
>> Following our discussions on the previous thread (Merging branch HDFS-7240
>> to trunk), I'd like to propose the following:
>>
>> * HDSL become a subproject of Hadoop.
>> * HDSL will release separately from Hadoop. Hadoop releases will not
>> contain HDSL and vice versa.
>> * HDSL will get its own jira instance so that the release tags stay
>> separate.
>> * On trunk (as opposed to release branches) HDSL will be a separate module
>> in Hadoop's source tree. This will enable the HDSL to work on their trunk
>> and the Hadoop trunk without making releases for every change.
>> * Hadoop's trunk will only build HDSL if a non-default profile is enabled.
>> * When Hadoop creates a release branch, the RM will delete the HDSL module
>> from the branch.
>> * HDSL will have their own Yetus checks and won't cause failures in the
>> Hadoop patch check.
>>
>> I think this accomplishes most of the goals of encouraging HDSL
>> development
>> while minimizing the potential for disruption of HDFS development.
>>
>> The vote will run the standard 7 days and requires a lazy 2/3 vote. PMC
>> votes are binding, but everyone is encouraged to vote.
>>
>> +1 (binding)
>>
>> .. Owen
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Adopt HDSL as a new Hadoop subproject

[Chris Douglas]

+1 (binding)

This compromise seems to address most of the concerns raised during
the discussion. Thanks for proposing and driving this, Owen.

On Thu, Mar 22, 2018 at 9:30 AM, Andrew Wang  wrote:
> In Owen's proposal, it says to delete the module from the release branch.
> We need to do this since the source tarball is our official Apache release
> artifact, the rest are convenience binaries. So the Maven profile is
> insufficient for this.

Eliminating manual steps to create a release is desirable, but
privileging it above all the development efficiencies gained by
merging to the same repo... we don't cut releases that often.

Moreover, the steps to remove the module don't need to be manual. Once
we work out the steps, would you be willing to update the
create-release script? -C


On Tue, Mar 20, 2018 at 11:20 AM, Owen O'Malley  wrote:
> All,
>
> Following our discussions on the previous thread (Merging branch HDFS-7240
> to trunk), I'd like to propose the following:
>
> * HDSL become a subproject of Hadoop.
> * HDSL will release separately from Hadoop. Hadoop releases will not
> contain HDSL and vice versa.
> * HDSL will get its own jira instance so that the release tags stay
> separate.
> * On trunk (as opposed to release branches) HDSL will be a separate module
> in Hadoop's source tree. This will enable the HDSL to work on their trunk
> and the Hadoop trunk without making releases for every change.
> * Hadoop's trunk will only build HDSL if a non-default profile is enabled.
> * When Hadoop creates a release branch, the RM will delete the HDSL module
> from the branch.
> * HDSL will have their own Yetus checks and won't cause failures in the
> Hadoop patch check.
>
> I think this accomplishes most of the goals of encouraging HDSL development
> while minimizing the potential for disruption of HDFS development.
>
> The vote will run the standard 7 days and requires a lazy 2/3 vote. PMC
> votes are binding, but everyone is encouraged to vote.
>
> +1 (binding)
>
> .. Owen

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Apache Hadoop qbt Report: trunk+JDK8 on Windows/x64

[Chris Douglas]

Thanks, Allen.

On Thu, Mar 15, 2018 at 10:20 AM, Iñigo Goiri  wrote:
>> * It ALWAYS applies HADOOP-14667.05.patch prior to running.  As a result,
>> this is only set up for trunk with no parameterization to run other
>> branches.

I tried to get this running in my environment a few weeks ago, but I
suspect my problems are with my environment, not the patch.

Let's at least get this committed to trunk. We can look at other
branches later. -C

>> * The URI handling for file paths in hadoop-common and elsewhere is pretty
>> broken on Windows, so many many many unit tests are failing and I wouldn't
>> be surprised if Windows hadoop installs are horked as a result.
>>
>> * Runtime is about 12-13 hours with many tests taking significantly longer
>> than their UNIX counterparts.  My guess is that this caused by winutils.
>> Changing from winutils to Java 7 API calls would get this more in line and
>> be a significant performance boost for Windows clients/servers as well.
>>
>> Have fun.
>>
>> =
>>
>> For more details, see https://builds.apache.org/job/hadoop-trunk-win/406/
>> 
>>
>> [Mar 14, 2018 6:26:58 PM] (xyao) HDFS-13251. Avoid using hard coded
>> datanode data dirs in unit tests.
>> [Mar 14, 2018 8:05:24 PM] (jlowe) MAPREDUCE-7064. Flaky test
>> [Mar 14, 2018 8:14:36 PM] (inigoiri) HDFS-13198. RBF:
>> RouterHeartbeatService throws out CachedStateStore
>> [Mar 14, 2018 8:36:53 PM] (wangda) Revert "HADOOP-13707. If kerberos is
>> enabled while HTTP SPNEGO is not
>> [Mar 14, 2018 10:47:56 PM] (fabbri) HADOOP-15278 log s3a at info.
>> Contributed by Steve Loughran.
>>
>>
>>
>>
>> -1 overall
>>
>>
>> The following subsystems voted -1:
>>unit
>>
>>
>> The following subsystems are considered long running:
>> (runtime bigger than 1h 00m 00s)
>>unit
>>
>>
>> Specific tests:
>>
>>Failed CTEST tests :
>>
>>   test_test_libhdfs_threaded_hdfs_static
>>
>>Failed junit tests :
>>
>>   hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec
>>   hadoop.fs.contract.rawlocal.TestRawlocalContractAppend
>>   hadoop.fs.TestFsShellCopy
>>   hadoop.fs.TestFsShellList
>>   hadoop.fs.TestLocalFileSystem
>>   hadoop.http.TestHttpServer
>>   hadoop.http.TestHttpServerLogs
>>   hadoop.io.compress.TestCodec
>>   hadoop.io.nativeio.TestNativeIO
>>   hadoop.ipc.TestSocketFactory
>>   hadoop.metrics2.impl.TestStatsDMetrics
>>   hadoop.metrics2.sink.TestRollingFileSystemSinkWithLocal
>>   hadoop.security.TestSecurityUtil
>>   hadoop.security.TestShellBasedUnixGroupsMapping
>>   hadoop.security.token.TestDtUtilShell
>>   hadoop.util.TestNativeCodeLoader
>>   hadoop.fs.TestWebHdfsFileContextMainOperations
>>   hadoop.hdfs.client.impl.TestBlockReaderLocalLegacy
>>   hadoop.hdfs.crypto.TestHdfsCryptoStreams
>>   hadoop.hdfs.qjournal.client.TestQuorumJournalManager
>>   hadoop.hdfs.qjournal.server.TestJournalNode
>>   hadoop.hdfs.qjournal.server.TestJournalNodeSync
>>   hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
>>   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
>>   hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages
>>   hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
>>   hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
>>   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
>>   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles
>>   hadoop.hdfs.server.datanode.fsdataset.impl.
>> TestLazyPersistLockedMemory
>>   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy
>>   hadoop.hdfs.server.datanode.fsdataset.impl.
>> TestLazyPersistReplicaPlacement
>>   hadoop.hdfs.server.datanode.fsdataset.impl.
>> TestLazyPersistReplicaRecovery
>>   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter
>>   hadoop.hdfs.server.datanode.fsdataset.impl.TestProvidedImpl
>>   hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation
>>   hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica
>>   hadoop.hdfs.server.datanode.TestBlockPoolSliceStorage
>>   hadoop.hdfs.server.datanode.TestBlockRecovery
>>   hadoop.hdfs.server.datanode.TestBlockScanner
>>   hadoop.hdfs.server.datanode.TestDataNodeFaultInjector
>>   hadoop.hdfs.server.datanode.TestDataNodeMetrics
>>   hadoop.hdfs.server.datanode.TestDataNodeUUID
>>   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
>>   hadoop.hdfs.server.datanode.TestDirectoryScanner
>>   hadoop.hdfs.server.datanode.TestHSync
>>   hadoop.hdfs.server.datanode.web.TestDatanodeHttpXFrame
>>   hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand
>>   hadoop.hdfs.server.diskbalancer.TestDiskBalancerRPC
>>   hadoop.hdfs.server.federation.router.TestRouterAdminCLI
>>   

Re: [EVENT] HDFS Bug Bash: March 12

[Chris Douglas]

Thanks, Vinod. -C

On Mon, Mar 12, 2018 at 2:38 PM, Vinod Kumar Vavilapalli
<vino...@apache.org> wrote:
> Was out of country and away from email for weeks.
>
> We have been using this in the past:
> https://www.meetup.com/Hadoop-Contributors/. I believe this doesn't have the
> per-meetup-charge issue, but will double check. Let me know if there are
> more meetups planned in the near future and we can use this.
>
> Thanks
> +Vinod
>
> On Mar 6, 2018, at 7:48 PM, Chris Douglas <cdoug...@apache.org> wrote:
>
> Found a meetup alternative (thanks Subru):
> https://meetingstar.io/event/fk13172f1d75KN
>
> So we can get a rough headcount, please add (local) if you plan to
> attend in-person. -C
>
>
> On Mon, Mar 5, 2018 at 4:03 PM, Chris Douglas <cdoug...@apache.org> wrote:
>
> [Cross-posting, as this affects the rest of the project]
>
> Hey folks-
>
> As discussed last month [1], the HDFS build hasn't been healthy
> recently. We're dedicating a bug bash to stabilize the build and
> address some longstanding issues with our unit tests. We rely on our
> CI infrastructure to keep the project releasable, and in its current
> state, it's not protecting us from regressions. While we probably
> won't achieve all our goals in this session, we can develop the
> conditions for reestablishing a firm foundation.
>
> If you're new to the project, please consider attending and
> contributing. Committers often prioritize large or complicated
> patches, and the issues that make the project livable don't get enough
> attention. A bug bash is a great opportunity to pull reviewers'
> collars, and fix the annoyances that slow us all down.
>
> If you're a committer, please join us! While some of the proposed
> repairs are rote, many unit tests rely on implementation details and
> non-obvious invariants. We need domain experts to help untangle
> complex dependencies and to prevent breakage of deliberate, but
> counter-intuitive code.
>
> We're collecting tasks in wiki [2] and will include a dial-in option
> for folks who aren't local.
>
> Meetup has started charging for creating new events, so we'll have to
> find another way to get an approximate headcount and publish the
> address. Please ping me if you have a preferred alternative. -C
>
> [1]: https://s.apache.org/nEoQ
> [2]:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75965105
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [EVENT] HDFS Bug Bash: March 12

[Chris Douglas]

We're using a shared doc to track work in progress, PA/review ready,
and committed [1]. -C

[1]: https://s.apache.org/RLlx

On Mon, Mar 12, 2018 at 9:17 AM, Chris Douglas <cdoug...@apache.org> wrote:
> On Fri, Mar 9, 2018 at 7:52 PM, 俊平堵 <junping...@apache.org> wrote:
>> Thanks for organizing this, Chris! Please let me know if anything else I
>> can help here.
>
> The plan is to work through items on the wiki [1], including build
> reliability and other "quality of life" issues in the project. -C
>
> [1]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75965105
>
>> 2018-03-05 16:03 GMT-08:00 Chris Douglas <cdoug...@apache.org>:
>>
>>> [Cross-posting, as this affects the rest of the project]
>>>
>>> Hey folks-
>>>
>>> As discussed last month [1], the HDFS build hasn't been healthy
>>> recently. We're dedicating a bug bash to stabilize the build and
>>> address some longstanding issues with our unit tests. We rely on our
>>> CI infrastructure to keep the project releasable, and in its current
>>> state, it's not protecting us from regressions. While we probably
>>> won't achieve all our goals in this session, we can develop the
>>> conditions for reestablishing a firm foundation.
>>>
>>> If you're new to the project, please consider attending and
>>> contributing. Committers often prioritize large or complicated
>>> patches, and the issues that make the project livable don't get enough
>>> attention. A bug bash is a great opportunity to pull reviewers'
>>> collars, and fix the annoyances that slow us all down.
>>>
>>> If you're a committer, please join us! While some of the proposed
>>> repairs are rote, many unit tests rely on implementation details and
>>> non-obvious invariants. We need domain experts to help untangle
>>> complex dependencies and to prevent breakage of deliberate, but
>>> counter-intuitive code.
>>>
>>> We're collecting tasks in wiki [2] and will include a dial-in option
>>> for folks who aren't local.
>>>
>>> Meetup has started charging for creating new events, so we'll have to
>>> find another way to get an approximate headcount and publish the
>>> address. Please ping me if you have a preferred alternative. -C
>>>
>>> [1]: https://s.apache.org/nEoQ
>>> [2]: https://cwiki.apache.org/confluence/pages/viewpage.
>>> action?pageId=75965105
>>>
>>> -
>>> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
>>> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>>>
>>>

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [EVENT] HDFS Bug Bash: March 12

[Chris Douglas]

On Fri, Mar 9, 2018 at 7:52 PM, 俊平堵 <junping...@apache.org> wrote:
> Thanks for organizing this, Chris! Please let me know if anything else I
> can help here.

The plan is to work through items on the wiki [1], including build
reliability and other "quality of life" issues in the project. -C

[1]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75965105

> 2018-03-05 16:03 GMT-08:00 Chris Douglas <cdoug...@apache.org>:
>
>> [Cross-posting, as this affects the rest of the project]
>>
>> Hey folks-
>>
>> As discussed last month [1], the HDFS build hasn't been healthy
>> recently. We're dedicating a bug bash to stabilize the build and
>> address some longstanding issues with our unit tests. We rely on our
>> CI infrastructure to keep the project releasable, and in its current
>> state, it's not protecting us from regressions. While we probably
>> won't achieve all our goals in this session, we can develop the
>> conditions for reestablishing a firm foundation.
>>
>> If you're new to the project, please consider attending and
>> contributing. Committers often prioritize large or complicated
>> patches, and the issues that make the project livable don't get enough
>> attention. A bug bash is a great opportunity to pull reviewers'
>> collars, and fix the annoyances that slow us all down.
>>
>> If you're a committer, please join us! While some of the proposed
>> repairs are rote, many unit tests rely on implementation details and
>> non-obvious invariants. We need domain experts to help untangle
>> complex dependencies and to prevent breakage of deliberate, but
>> counter-intuitive code.
>>
>> We're collecting tasks in wiki [2] and will include a dial-in option
>> for folks who aren't local.
>>
>> Meetup has started charging for creating new events, so we'll have to
>> find another way to get an approximate headcount and publish the
>> address. Please ping me if you have a preferred alternative. -C
>>
>> [1]: https://s.apache.org/nEoQ
>> [2]: https://cwiki.apache.org/confluence/pages/viewpage.
>> action?pageId=75965105
>>
>> -
>> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>>
>>

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [EVENT] HDFS Bug Bash: March 12

[Chris Douglas]

If you want to join remotely, we have a bridge set up:

Skype: https://meet.lync.com/microsoft/cdoug/DM56LHTJ
Web app: https://meet.lync.com/microsoft/cdoug/DM56LHTJ?sl=1
Phone: +1 (425) 616-0754 Conference ID: 65908231
Find a local number:
https://dialin.lync.com/8551f4c1-bea3-441a-8738-69aa517a91c5?id=65908231

Please let me know if this doesn't work for you. -C


On Fri, Mar 9, 2018 at 7:52 PM, 俊平堵 <junping...@apache.org> wrote:
> I cannot join this in person as I am currently oversea. but +1 on this
> event.
> Thanks for organizing this, Chris! Please let me know if anything else I
> can help here.
>
> Thanks,
>
> Junping
>
> 2018-03-05 16:03 GMT-08:00 Chris Douglas <cdoug...@apache.org>:
>
>> [Cross-posting, as this affects the rest of the project]
>>
>> Hey folks-
>>
>> As discussed last month [1], the HDFS build hasn't been healthy
>> recently. We're dedicating a bug bash to stabilize the build and
>> address some longstanding issues with our unit tests. We rely on our
>> CI infrastructure to keep the project releasable, and in its current
>> state, it's not protecting us from regressions. While we probably
>> won't achieve all our goals in this session, we can develop the
>> conditions for reestablishing a firm foundation.
>>
>> If you're new to the project, please consider attending and
>> contributing. Committers often prioritize large or complicated
>> patches, and the issues that make the project livable don't get enough
>> attention. A bug bash is a great opportunity to pull reviewers'
>> collars, and fix the annoyances that slow us all down.
>>
>> If you're a committer, please join us! While some of the proposed
>> repairs are rote, many unit tests rely on implementation details and
>> non-obvious invariants. We need domain experts to help untangle
>> complex dependencies and to prevent breakage of deliberate, but
>> counter-intuitive code.
>>
>> We're collecting tasks in wiki [2] and will include a dial-in option
>> for folks who aren't local.
>>
>> Meetup has started charging for creating new events, so we'll have to
>> find another way to get an approximate headcount and publish the
>> address. Please ping me if you have a preferred alternative. -C
>>
>> [1]: https://s.apache.org/nEoQ
>> [2]: https://cwiki.apache.org/confluence/pages/viewpage.
>> action?pageId=75965105
>>
>> -
>> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>>
>>

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Merging branch HDFS-8707 (native HDFS client) to trunk

[Chris Douglas]

+1 (binding) -C

On Thu, Mar 8, 2018 at 9:31 AM, Jim Clampffer  wrote:
> Hi Everyone,
>
> The feedback was generally positive on the discussion thread [1] so I'd
> like to start a formal vote for merging HDFS-8707 (libhdfs++) into trunk.
> The vote will be open for 7 days and end 6PM EST on 3/15/18.
>
> This branch includes a C++ implementation of an HDFS client for use in
> applications that don't run an in-process JVM.  Right now the branch only
> supports reads and metadata calls.
>
> Features (paraphrasing the list from the discussion thread):
> -Avoiding the JVM means applications that use libhdfs++ can explicitly
> control resources (memory, FDs, threads).  The driving goal for this
> project was to let C/C++ applications access HDFS while maintaining a
> single heap.
> -Includes support for Kerberos authentication.
> -Includes a libhdfs/libhdfs3 compatible C API as well as a C++ API that
> supports asynchronous operations.  Applications that only do reads may be
> able to use this as a drop in replacement for libhdfs.
> -Asynchronous IO is built on top of boost::asio which in turn uses
> select/epoll so many sockets can be monitored from a single thread (or
> thread pool) rather than spawning a thread to sleep on a blocked socket.
> -Includes a set of utilities written in C++ that mirror the CLI tools (e.g.
> ./hdfs dfs -ls).  These have a 3 order of magnitude lower startup time than
> java client which is useful for scripts that need to work with many files.
> -Support for cancelable reads that release associated resources
> immediately.  Useful for applications that need to be responsive to
> interactive users.
>
> Other points:
> -This is almost all new code in a new subdirectory.  No Java source for the
> rest of hadoop was changed so there's no risk of regressions there.  The
> only changes outside of that subdirectory were integrating the build in
> some of the pom files and adding a couple dependencies to the DockerFile.
> -The library has had plenty of burn-in time.  It's been used in production
> for well over a year and is indirectly being distributed as part of the
> Apache ORC project (in the form of a third party dependency).
> -There isn't much in the way of well formatted documentation right now.
> The documentation for the libhdfs API is applicable to the libhdfs++ C API.
> Header files describe various component including details about threading
> and lifecycle expectations for important objects.  Good places to start are
> hdfspp.h, filesystem.h, filehandle.h, rpc_connection.h and rpc_enginel.h.
>
> I'll start with my +1 (binding).
>
> [1]
> http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201803.mbox/browser
> (second message in thread, can't figure out how to link directly to mine)
>
> Thanks!

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [EVENT] HDFS Bug Bash: March 12

[Chris Douglas]

Hey folks-

Remember we're getting together on Monday to fix some "quality of
life" issues in the build. Please RSVP [1] so we can preprint badges,
that sort of thing. The meetup site only allows for 6 hour events, but
we have the room through 6:00PM PST and can keep the bridge/channel
open for other timezones that want to keep going.

We'll post remote access information when we get it. Please update the
wiki [2] as you think of issues we should address. -C

[1] https://meetingstar.io/event/fk13172f1d75KN
[2]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75965105

On Tue, Mar 6, 2018 at 7:48 PM, Chris Douglas <cdoug...@apache.org> wrote:
> Found a meetup alternative (thanks Subru):
> https://meetingstar.io/event/fk13172f1d75KN
>
> So we can get a rough headcount, please add (local) if you plan to
> attend in-person. -C
>
>
> On Mon, Mar 5, 2018 at 4:03 PM, Chris Douglas <cdoug...@apache.org> wrote:
>> [Cross-posting, as this affects the rest of the project]
>>
>> Hey folks-
>>
>> As discussed last month [1], the HDFS build hasn't been healthy
>> recently. We're dedicating a bug bash to stabilize the build and
>> address some longstanding issues with our unit tests. We rely on our
>> CI infrastructure to keep the project releasable, and in its current
>> state, it's not protecting us from regressions. While we probably
>> won't achieve all our goals in this session, we can develop the
>> conditions for reestablishing a firm foundation.
>>
>> If you're new to the project, please consider attending and
>> contributing. Committers often prioritize large or complicated
>> patches, and the issues that make the project livable don't get enough
>> attention. A bug bash is a great opportunity to pull reviewers'
>> collars, and fix the annoyances that slow us all down.
>>
>> If you're a committer, please join us! While some of the proposed
>> repairs are rote, many unit tests rely on implementation details and
>> non-obvious invariants. We need domain experts to help untangle
>> complex dependencies and to prevent breakage of deliberate, but
>> counter-intuitive code.
>>
>> We're collecting tasks in wiki [2] and will include a dial-in option
>> for folks who aren't local.
>>
>> Meetup has started charging for creating new events, so we'll have to
>> find another way to get an approximate headcount and publish the
>> address. Please ping me if you have a preferred alternative. -C
>>
>> [1]: https://s.apache.org/nEoQ
>> [2]: 
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75965105

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [EVENT] HDFS Bug Bash: March 12

[Chris Douglas]

Found a meetup alternative (thanks Subru):
https://meetingstar.io/event/fk13172f1d75KN

So we can get a rough headcount, please add (local) if you plan to
attend in-person. -C


On Mon, Mar 5, 2018 at 4:03 PM, Chris Douglas <cdoug...@apache.org> wrote:
> [Cross-posting, as this affects the rest of the project]
>
> Hey folks-
>
> As discussed last month [1], the HDFS build hasn't been healthy
> recently. We're dedicating a bug bash to stabilize the build and
> address some longstanding issues with our unit tests. We rely on our
> CI infrastructure to keep the project releasable, and in its current
> state, it's not protecting us from regressions. While we probably
> won't achieve all our goals in this session, we can develop the
> conditions for reestablishing a firm foundation.
>
> If you're new to the project, please consider attending and
> contributing. Committers often prioritize large or complicated
> patches, and the issues that make the project livable don't get enough
> attention. A bug bash is a great opportunity to pull reviewers'
> collars, and fix the annoyances that slow us all down.
>
> If you're a committer, please join us! While some of the proposed
> repairs are rote, many unit tests rely on implementation details and
> non-obvious invariants. We need domain experts to help untangle
> complex dependencies and to prevent breakage of deliberate, but
> counter-intuitive code.
>
> We're collecting tasks in wiki [2] and will include a dial-in option
> for folks who aren't local.
>
> Meetup has started charging for creating new events, so we'll have to
> find another way to get an approximate headcount and publish the
> address. Please ping me if you have a preferred alternative. -C
>
> [1]: https://s.apache.org/nEoQ
> [2]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75965105

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[EVENT] HDFS Bug Bash: March 12

[Chris Douglas]

[Cross-posting, as this affects the rest of the project]

Hey folks-

As discussed last month [1], the HDFS build hasn't been healthy
recently. We're dedicating a bug bash to stabilize the build and
address some longstanding issues with our unit tests. We rely on our
CI infrastructure to keep the project releasable, and in its current
state, it's not protecting us from regressions. While we probably
won't achieve all our goals in this session, we can develop the
conditions for reestablishing a firm foundation.

If you're new to the project, please consider attending and
contributing. Committers often prioritize large or complicated
patches, and the issues that make the project livable don't get enough
attention. A bug bash is a great opportunity to pull reviewers'
collars, and fix the annoyances that slow us all down.

If you're a committer, please join us! While some of the proposed
repairs are rote, many unit tests rely on implementation details and
non-obvious invariants. We need domain experts to help untangle
complex dependencies and to prevent breakage of deliberate, but
counter-intuitive code.

We're collecting tasks in wiki [2] and will include a dial-in option
for folks who aren't local.

Meetup has started charging for creating new events, so we'll have to
find another way to get an approximate headcount and publish the
address. Please ping me if you have a preferred alternative. -C

[1]: https://s.apache.org/nEoQ
[2]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75965105

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Reopened] (HDFS-13216) HDFS

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-13216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas reopened HDFS-13216:
--

> HDFS
> 
>
> Key: HDFS-13216
> URL: https://issues.apache.org/jira/browse/HDFS-13216
> Project: Hadoop HDFS
>  Issue Type: Task
>Reporter: mster
>Priority: Trivial
>  Labels: INodesInPath
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-13216) HDFS

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-13216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas resolved HDFS-13216.
--
Resolution: Invalid

> HDFS
> 
>
> Key: HDFS-13216
> URL: https://issues.apache.org/jira/browse/HDFS-13216
> Project: Hadoop HDFS
>  Issue Type: Task
>Reporter: mster
>Priority: Trivial
>  Labels: INodesInPath
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DISCUSS] Merging HDFS-8707 (C++ HDFS client) to trunk

[Chris Douglas]

On Thu, Mar 1, 2018 at 10:04 AM, Jim Clampffer
<james.clampf...@gmail.com> wrote:
> Chris, do you mean potentially landing this in its current state and
> handling some of the rough edges after?  I could see this working just
> because there's no impact on any existing code.

Yes. Better to get this committed and released than to polish it in
the branch. -C

> With regards to your questions Kai:
> There isn't a good doc for the internal architecture yet; I just reassigned
> HDFS-9115 to myself to handle that.  Are there any specific areas you'd like
> to know about so I can prioritize those?
> Here's some header files that include a lot of comments that should help out
> for now:
> -hdfspp.h - main header for the C++ API
> -filesystem.h and filehandle.h - describes some rules about object lifetimes
> and threading from the API point of view (most classes have comments
> describing any restrictions on threading, locking, and lifecycle).
> -rpc_engine.h and rpc_connection.h begin getting into the async RPC
> implementation.
>
>
> 1) Yes, it's a reimplementation of the entire client in C++.  Using libhdfs3
> as a reference helps a lot here but it's still a lot of work.
> 2) EC isn't supported now, though that'd be great to have, and I agree that
> it's going to be take a lot of effort to implement.  Right now if you tried
> to read an EC file I think you'd get some unhelpful error out of the block
> reader but I don't have an EC enabled cluster set up to test.  Adding an
> explicit not supported message would be straightforward.
> 3) libhdfs++ reuses all of the minidfscluster tests that libhdfs already had
> so we get consistency checks on the C API.  There's a few new tests that
> also get run on both libhdfs and libhdfs++ and make sure the expected output
> is the same too.
> 4) I agree, I just haven't had a chance to look into the distribution build
> to see how to do it.  HDFS-9465 is tracking this.
> 5) Not yet (HDFS-8765).
>
> Regards,
> James
>
>
>
>
> On Thu, Mar 1, 2018 at 4:28 AM, 郑锴(铁杰) <zhengkai...@alibaba-inc.com> wrote:
>>
>> The work sounds solid and great! + to have this.
>>
>> Is there any quick doc to take a glance at? Some quick questions to be
>> familiar with:
>> 1. Seems the client is all implemented in c++ without any Java codes (so
>> no JVM overhead), which means lots of work, rewriting HDFS client. Right?
>> 2.  Guess erasure coding feature isn't supported, as it'd involve
>> significant development, right? If yes, what will it say when read erasure
>> coded file?
>> 3. Is there any building/testing mechanism to enforce the consistency
>> between the c++ part and Java part?
>> 4. I thought the public header and lib should be exported when building
>> the distribution package, otherwise hard to use the new C api.
>> 5. Is the short-circuit read supported?
>>
>> Thanks.
>>
>>
>> Regards,
>> Kai
>>
>> --
>> 发件人：Chris Douglas <cdoug...@apache.org>
>> 发送时间：2018年3月1日(星期四) 05:08
>> 收件人：Jim Clampffer <james.clampf...@gmail.com>
>> 抄　送：Hdfs-dev <hdfs-dev@hadoop.apache.org>
>> 主　题：Re: [DISCUSS] Merging HDFS-8707 (C++ HDFS client) to trunk
>>
>> +1
>>
>> Let's get this done. We've had many false starts on a native HDFS
>> client. This is a good base to build on. -C
>>
>> On Wed, Feb 28, 2018 at 9:55 AM, Jim Clampffer
>> <james.clampf...@gmail.com> wrote:
>> > Hi everyone,
>> >
>> > I'd like to start a thread to discuss merging the HDFS-8707 aka
>> > libhdfs++
>> > into trunk.  I sent originally sent a similar email out last October but
>> > it
>> > sounds like it was buried by discussions about other feature merges that
>> > were going on at the time.
>> >
>> > libhdfs++ is an HDFS client written in C++ designed to be used in
>> > applications that are written in non-JVM based languages.  In its
>> > current
>> > state it supports kerberos authenticated reads from HDFS and has been
>> > used
>> > in production clusters for over a year so it has had a significant
>> > amount
>> > of burn-in time.  The HDFS-8707 branch has been around for about 2 years
>> > now so I'd like to know people's thoughts on what it would take to merge
>> > current branch and handling writes and encrypted reads in a new one.
>> >
>> > Current notable features:
>> >   -A libhdfs/libhdfs3 compatible C API that allows libhdfs++ to serve as
>> > a
>> > drop-in replacement for clien

Re: [DISCUSS] Merging HDFS-8707 (C++ HDFS client) to trunk

[Chris Douglas]

+1

Let's get this done. We've had many false starts on a native HDFS
client. This is a good base to build on. -C

On Wed, Feb 28, 2018 at 9:55 AM, Jim Clampffer
 wrote:
> Hi everyone,
>
> I'd like to start a thread to discuss merging the HDFS-8707 aka libhdfs++
> into trunk.  I sent originally sent a similar email out last October but it
> sounds like it was buried by discussions about other feature merges that
> were going on at the time.
>
> libhdfs++ is an HDFS client written in C++ designed to be used in
> applications that are written in non-JVM based languages.  In its current
> state it supports kerberos authenticated reads from HDFS and has been used
> in production clusters for over a year so it has had a significant amount
> of burn-in time.  The HDFS-8707 branch has been around for about 2 years
> now so I'd like to know people's thoughts on what it would take to merge
> current branch and handling writes and encrypted reads in a new one.
>
> Current notable features:
>   -A libhdfs/libhdfs3 compatible C API that allows libhdfs++ to serve as a
> drop-in replacement for clients that only need read support (until
> libhdfs++ also supports writes).
>   -An asynchronous C++ API with synchronous shims on top if the client
> application wants to do blocking operations.  Internally a single thread
> (optionally more) uses select/epoll by way of boost::asio to watch
> thousands of sockets without the overhead of spawning threads to emulate
> async operation.
>   -Kerberos/SASL authentication support
>   -HA namenode support
>   -A set of utility programs that mirror the HDFS CLI utilities e.g.
> "./hdfs dfs -chmod".  The major benefit of these is the tool startup time
> is ~3 orders of magnitude faster (<1ms vs hundreds of ms) and occupies a
> lot less memory since it isn't dealing with the JVM.  This makes it
> possible to do things like write a simple bash script that stats a file,
> applies some rules to the result, and decides if it should move it in a way
> that scales to thousands of files without being penalized with O(N) JVM
> startups.
>   -Cancelable reads.  This has proven to be very useful in multiuser
> applications that (pre)fetch large blocks of data but need to remain
> responsive for interactive users.  Rather than waiting for a large and/or
> slow read to finish it will return immediately and the associated resources
> (buffer, file descriptor) become available for the rest of the application
> to use.
>
> There are a couple known issues: the doc build isn't integrated with the
> rest of hadoop and the public API headers aren't being exported when
> building a distribution.  A short term solution for missing docs is to go
> through the libhdfs(3) compatible API and use the libhdfs docs.  Other than
> a few modifications to the pom files to integrate the build and the changes
> are isolated to a new directory so the chance of causing any regressions in
> the rest of the code is minimal.
>
> Please share your thoughts, thanks!

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DISCUSS] Meetup for HDFS tests and build infra

[Chris Douglas]

Correction: 3/12 or 3/13, as in March 12 or 13th. -C

On Tue, Feb 20, 2018 at 3:59 PM, Chris Douglas <cdoug...@apache.org> wrote:
> Looking at the poll [1], it looks like 12/12 or 12/13 would be the
> best day to do this. If you haven't posted your availability and would
> like to participate, please fill this out soon.
>
> We'll work on some common space in the bay area, but we'll also open a
> channel for folks who can't attend in-person. -C
>
> [1]: https://doodle.com/poll/r22znitzae9apfbf
>
>
> On Fri, Feb 9, 2018 at 1:45 PM, Chris Douglas <cdoug...@apache.org> wrote:
>> On Thu, Feb 8, 2018 at 5:47 PM, 郑锴(铁杰) <zhengkai...@alibaba-inc.com> wrote:
>>>>>I'm looking at you, TestDFSStripedOutputStreamWithFailure ...
>>> AFAIK and IMO, it's pretty hard to get all the test cases stably running
>>> given the limitation of MiniDFSCluster, and if we'd agree on that, we could
>>> remove these cases as unit tests and cover them in integration tests instead
>>> using a true cluster, like based on k8s infra.
>>
>> Since its inception, the Mini*Clusters have made it much easier to
>> write unit tests (good!), but these tests take more time and resources
>> than fine-grained tests (unfortunate) and they're often unreliable
>> (bad). As we see in issues like HDFS-12711, the resource problem seems
>> to harm reliability across the entire suite of tests (dire).
>>
>> Running integration tests with our CI is helpful, but perhaps we
>> should separate the Mini*Cluster tests and/or press contributors to
>> write lighter unit tests for new functionality. -C
>>
>>> We're lacking basic facility
>>> infra env and tools to get most of the complicated functionalities well
>>> tested and covered, so let's avoid too much complicated tests. Fixing of
>>> such tests should definitely help and be appreciated.
>>>
>>> Regards,
>>> Kai
>>>
>>> --
>>> 发件人：Chris Douglas <cdoug...@apache.org>
>>> 发送时间：2018年2月8日(星期四) 08:39
>>> 收件人：Hdfs-dev <hdfs-dev@hadoop.apache.org>
>>> 主　题：Re: [DISCUSS] Meetup for HDFS tests and build infra
>>>
>>> Created a poll [1] to inform scheduling. -C
>>>
>>> [1]: https://doodle.com/poll/r22znitzae9apfbf
>>>
>>> On Tue, Feb 6, 2018 at 3:09 PM, Chris Douglas <cdoug...@apache.org> wrote:
>>>> The HDFS build is not healthy. Many of the unit tests aren't actually
>>>> run in Jenkins due to resource exhaustion, haven't been updated since
>>>> build/test/data was the test temp dir, or are chronically unstable
>>>> (I'm looking at you, TestDFSStripedOutputStreamWithFailure). The
>>>> situation has deteriorated slowly, but we can't confidently merge
>>>> patches, let alone significant features, when our CI infra is in this
>>>> state.
>>>>
>>>> How would folks feel about a half to full-day meetup to work through
>>>> patches improving this, specifically? We can improve tests,
>>>> troubleshoot the build, and rev/commit existing patches. It would
>>>> require some preparation, so the simultaneous attention is productive
>>>> and not a coordination bottleneck. I started a wiki page for this [1],
>>>> please add to it.
>>>>
>>>> If enough people can make time for this, say in 2-3 weeks, the project
>>>> would certainly benefit. -C
>>>>
>>>> [1]: https://s.apache.org/ng3C
>>>
>>> -
>>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DISCUSS] Meetup for HDFS tests and build infra

[Chris Douglas]

Looking at the poll [1], it looks like 12/12 or 12/13 would be the
best day to do this. If you haven't posted your availability and would
like to participate, please fill this out soon.

We'll work on some common space in the bay area, but we'll also open a
channel for folks who can't attend in-person. -C

[1]: https://doodle.com/poll/r22znitzae9apfbf


On Fri, Feb 9, 2018 at 1:45 PM, Chris Douglas <cdoug...@apache.org> wrote:
> On Thu, Feb 8, 2018 at 5:47 PM, 郑锴(铁杰) <zhengkai...@alibaba-inc.com> wrote:
>>>>I'm looking at you, TestDFSStripedOutputStreamWithFailure ...
>> AFAIK and IMO, it's pretty hard to get all the test cases stably running
>> given the limitation of MiniDFSCluster, and if we'd agree on that, we could
>> remove these cases as unit tests and cover them in integration tests instead
>> using a true cluster, like based on k8s infra.
>
> Since its inception, the Mini*Clusters have made it much easier to
> write unit tests (good!), but these tests take more time and resources
> than fine-grained tests (unfortunate) and they're often unreliable
> (bad). As we see in issues like HDFS-12711, the resource problem seems
> to harm reliability across the entire suite of tests (dire).
>
> Running integration tests with our CI is helpful, but perhaps we
> should separate the Mini*Cluster tests and/or press contributors to
> write lighter unit tests for new functionality. -C
>
>> We're lacking basic facility
>> infra env and tools to get most of the complicated functionalities well
>> tested and covered, so let's avoid too much complicated tests. Fixing of
>> such tests should definitely help and be appreciated.
>>
>> Regards,
>> Kai
>>
>> --
>> 发件人：Chris Douglas <cdoug...@apache.org>
>> 发送时间：2018年2月8日(星期四) 08:39
>> 收件人：Hdfs-dev <hdfs-dev@hadoop.apache.org>
>> 主　题：Re: [DISCUSS] Meetup for HDFS tests and build infra
>>
>> Created a poll [1] to inform scheduling. -C
>>
>> [1]: https://doodle.com/poll/r22znitzae9apfbf
>>
>> On Tue, Feb 6, 2018 at 3:09 PM, Chris Douglas <cdoug...@apache.org> wrote:
>>> The HDFS build is not healthy. Many of the unit tests aren't actually
>>> run in Jenkins due to resource exhaustion, haven't been updated since
>>> build/test/data was the test temp dir, or are chronically unstable
>>> (I'm looking at you, TestDFSStripedOutputStreamWithFailure). The
>>> situation has deteriorated slowly, but we can't confidently merge
>>> patches, let alone significant features, when our CI infra is in this
>>> state.
>>>
>>> How would folks feel about a half to full-day meetup to work through
>>> patches improving this, specifically? We can improve tests,
>>> troubleshoot the build, and rev/commit existing patches. It would
>>> require some preparation, so the simultaneous attention is productive
>>> and not a coordination bottleneck. I started a wiki page for this [1],
>>> please add to it.
>>>
>>> If enough people can make time for this, say in 2-3 weeks, the project
>>> would certainly benefit. -C
>>>
>>> [1]: https://s.apache.org/ng3C
>>
>> -
>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DISCUSS] Meetup for HDFS tests and build infra

[Chris Douglas]

On Thu, Feb 8, 2018 at 5:47 PM, 郑锴(铁杰) <zhengkai...@alibaba-inc.com> wrote:
>>>I'm looking at you, TestDFSStripedOutputStreamWithFailure ...
> AFAIK and IMO, it's pretty hard to get all the test cases stably running
> given the limitation of MiniDFSCluster, and if we'd agree on that, we could
> remove these cases as unit tests and cover them in integration tests instead
> using a true cluster, like based on k8s infra.

Since its inception, the Mini*Clusters have made it much easier to
write unit tests (good!), but these tests take more time and resources
than fine-grained tests (unfortunate) and they're often unreliable
(bad). As we see in issues like HDFS-12711, the resource problem seems
to harm reliability across the entire suite of tests (dire).

Running integration tests with our CI is helpful, but perhaps we
should separate the Mini*Cluster tests and/or press contributors to
write lighter unit tests for new functionality. -C

> We're lacking basic facility
> infra env and tools to get most of the complicated functionalities well
> tested and covered, so let's avoid too much complicated tests. Fixing of
> such tests should definitely help and be appreciated.
>
> Regards,
> Kai
>
> ----------
> 发件人：Chris Douglas <cdoug...@apache.org>
> 发送时间：2018年2月8日(星期四) 08:39
> 收件人：Hdfs-dev <hdfs-dev@hadoop.apache.org>
> 主　题：Re: [DISCUSS] Meetup for HDFS tests and build infra
>
> Created a poll [1] to inform scheduling. -C
>
> [1]: https://doodle.com/poll/r22znitzae9apfbf
>
> On Tue, Feb 6, 2018 at 3:09 PM, Chris Douglas <cdoug...@apache.org> wrote:
>> The HDFS build is not healthy. Many of the unit tests aren't actually
>> run in Jenkins due to resource exhaustion, haven't been updated since
>> build/test/data was the test temp dir, or are chronically unstable
>> (I'm looking at you, TestDFSStripedOutputStreamWithFailure). The
>> situation has deteriorated slowly, but we can't confidently merge
>> patches, let alone significant features, when our CI infra is in this
>> state.
>>
>> How would folks feel about a half to full-day meetup to work through
>> patches improving this, specifically? We can improve tests,
>> troubleshoot the build, and rev/commit existing patches. It would
>> require some preparation, so the simultaneous attention is productive
>> and not a coordination bottleneck. I started a wiki page for this [1],
>> please add to it.
>>
>> If enough people can make time for this, say in 2-3 weeks, the project
>> would certainly benefit. -C
>>
>> [1]: https://s.apache.org/ng3C
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DISCUSS] Meetup for HDFS tests and build infra

[Chris Douglas]

Created a poll [1] to inform scheduling. -C

[1]: https://doodle.com/poll/r22znitzae9apfbf

On Tue, Feb 6, 2018 at 3:09 PM, Chris Douglas <cdoug...@apache.org> wrote:
> The HDFS build is not healthy. Many of the unit tests aren't actually
> run in Jenkins due to resource exhaustion, haven't been updated since
> build/test/data was the test temp dir, or are chronically unstable
> (I'm looking at you, TestDFSStripedOutputStreamWithFailure). The
> situation has deteriorated slowly, but we can't confidently merge
> patches, let alone significant features, when our CI infra is in this
> state.
>
> How would folks feel about a half to full-day meetup to work through
> patches improving this, specifically? We can improve tests,
> troubleshoot the build, and rev/commit existing patches. It would
> require some preparation, so the simultaneous attention is productive
> and not a coordination bottleneck. I started a wiki page for this [1],
> please add to it.
>
> If enough people can make time for this, say in 2-3 weeks, the project
> would certainly benefit. -C
>
> [1]: https://s.apache.org/ng3C

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[DISCUSS] Meetup for HDFS tests and build infra

[Chris Douglas]

The HDFS build is not healthy. Many of the unit tests aren't actually
run in Jenkins due to resource exhaustion, haven't been updated since
build/test/data was the test temp dir, or are chronically unstable
(I'm looking at you, TestDFSStripedOutputStreamWithFailure). The
situation has deteriorated slowly, but we can't confidently merge
patches, let alone significant features, when our CI infra is in this
state.

How would folks feel about a half to full-day meetup to work through
patches improving this, specifically? We can improve tests,
troubleshoot the build, and rev/commit existing patches. It would
require some preparation, so the simultaneous attention is productive
and not a coordination bottleneck. I started a wiki page for this [1],
please add to it.

If enough people can make time for this, say in 2-3 weeks, the project
would certainly benefit. -C

[1]: https://s.apache.org/ng3C

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Apache Hadoop 3.0.1 Release plan

[Chris Douglas]

On Fri, Feb 2, 2018 at 10:22 AM, Arpit Agarwal  wrote:
> Do you plan to roll an RC with an uncommitted fix? That isn't the right 
> approach.

The fix will be committed to the release branch. We'll vote on the
release, and if it receives a majority of +1 votes then it becomes
3.0.1. That's how the PMC decides how to move forward. In this case,
that will also resolve whether or not it can be committed to trunk.

If this logic is unpersuasive, then we can require a 2/3 majority to
replace the codebase. Either way, the PMC will vote to define the
consensus view when it is not emergent.

> This issue has good visibility and enough discussion.

Yes, it has. We always prefer consensus to voting, but when discussion
reveals that complete consensus is impossible, we still need a way
forward. This is rare, and usually reserved for significant changes
(like merging YARN). Frankly, it's embarrassing to resort to it here,
but here we are.

> If there is a binding veto in effect then the change must be abandoned. Else 
> you should be able to proceed with committing. However, 3.0.0 must be called 
> out as an abandoned release if we commit it.

This is not accurate. A binding veto from any committer halts
progress, but the PMC sets the direction of the project. That includes
making decisions that are not universally accepted. -C

> On 2/1/18, 3:01 PM, "Lei Xu"  wrote:
>
> Sounds good to me, ATM.
>
> On Thu, Feb 1, 2018 at 2:34 PM, Aaron T. Myers  wrote:
> > Hey Anu,
> >
> > My feeling on HDFS-12990 is that we've discussed it quite a bit already 
> and
> > it doesn't seem at this point like either side is going to budge. I'm
> > certainly happy to have a phone call about it, but I don't expect that 
> we'd
> > make much progress.
> >
> > My suggestion is that we simply include the patch posted to HDFS-12990 
> in
> > the 3.0.1 RC and call this issue out clearly in the subsequent VOTE 
> thread
> > for the 3.0.1 release. Eddy, are you up for that?
> >
> > Best,
> > Aaron
> >
> > On Thu, Feb 1, 2018 at 1:13 PM, Lei Xu  wrote:
> >>
> >> +Xiao
> >>
> >> My understanding is that we will have this for 3.0.1.   Xiao, could
> >> you give your inputs here?
> >>
> >> On Thu, Feb 1, 2018 at 11:55 AM, Anu Engineer 
> 
> >> wrote:
> >> > Hi Eddy,
> >> >
> >> > Thanks for driving this release. Just a quick question, do we have 
> time
> >> > to close this issue?
> >> > https://issues.apache.org/jira/browse/HDFS-12990
> >> >
> >> > or are we abandoning it? I believe that this is the last window for 
> us
> >> > to fix this issue.
> >> >
> >> > Should we have a call and get this resolved one way or another?
> >> >
> >> > Thanks
> >> > Anu
> >> >
> >> > On 2/1/18, 10:51 AM, "Lei Xu"  wrote:
> >> >
> >> > Hi, All
> >> >
> >> > I just cut branch-3.0.1 from branch-3.0.  Please make sure all
> >> > patches
> >> > targeted to 3.0.1 being checked in both branch-3.0 and 
> branch-3.0.1.
> >> >
> >> > Thanks!
> >> > Eddy
> >> >
> >> > On Tue, Jan 9, 2018 at 11:17 AM, Lei Xu  
> wrote:
> >> > > Hi, All
> >> > >
> >> > > We have released Apache Hadoop 3.0.0 in December [1]. To 
> further
> >> > > improve the quality of release, we plan to cut branch-3.0.1 
> branch
> >> > > tomorrow for the preparation of Apache Hadoop 3.0.1 release. 
> The
> >> > focus
> >> > > of 3.0.1 will be fixing blockers (3), critical bugs (1) and bug
> >> > fixes
> >> > > [2].  No new features and improvement should be included.
> >> > >
> >> > > We plan to cut branch-3.0.1 tomorrow (Jan 10th) and vote for 
> RC on
> >> > Feb
> >> > > 1st, targeting for Feb 9th release.
> >> > >
> >> > > Please feel free to share your insights.
> >> > >
> >> > > [1]
> >> > https://www.mail-archive.com/general@hadoop.apache.org/msg07757.html
> >> > > [2] https://issues.apache.org/jira/issues/?filter=12342842
> >> > >
> >> > > Best,
> >> > > --
> >> > > Lei (Eddy) Xu
> >> > > Software Engineer, Cloudera
> >> >
> >> >
> >> >
> >> > --
> >> > Lei (Eddy) Xu
> >> > Software Engineer, Cloudera
> >> >
> >> >
> >> > -
> >> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> >> > For additional commands, e-mail: 
> common-dev-h...@hadoop.apache.org
> >> >
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Lei (Eddy) Xu
> >> Software Engineer, Cloudera
> >>
> >> 

Re: [VOTE] Merge YARN-6592 feature branch to trunk

[Chris Douglas]

+1 Looking forward to this. -C

On Fri, Jan 26, 2018 at 7:28 AM, Arun Suresh  wrote:
> Hello yarn-dev@
>
> Based on the positive feedback from the DISCUSS thread [1], I'd like to
> start a formal vote to merge YARN-6592 [2] to trunk. The vote will run for 5
> days, and will end Jan 31 7:30AM PDT.
>
> This feature adds support for placing containers in YARN using rich
> placement constraints. For example, this can be used by applications to
> co-locate containers on a node or rack (affinity constraint), spread
> containers across nodes or racks (anti-affinity constraint), or even specify
> the maximum number of containers on a node/rack (cardinality constraint).
>
> We have integrated this feature into the Distributed-Shell application for
> feature testing. We have performed end-to-end testing on moderately-sized
> clusters to verify that constraints work fine. Performance tests have been
> done via both SLS tests and Capacity Scheduler performance unit tests, and
> no regression was found. We have opened a JIRA to track Jenkins acceptance
> of the aggregated patch [3]. Documentation is in the process of being
> completed [4]. You can also check our design document for more details [5].
>
> Config flags are needed to enable this feature and it should not have any
> effect on YARN when turned off. Once merged, we plan to work on further
> improvements, which can be tracked in the umbrella YARN-7812 [6].
>
> Kindly do take a look at the branch and raise issues/concerns that need to
> be addressed before the merge.
>
> Many thanks to Konstantinos, Wangda, Panagiotis, Weiwei, and Sunil for their
> contributions to this effort, as well as Subru, Chris, Carlo, and Vinod for
> their inputs and discussions.
>
>
> Cheers
> -Arun
>
> [1]
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201801.mbox/%3CCAMreUaz%3DGnsjOLZ%3Dem2x%3DQS7qh27euCWNw6Bo_4Cu%2BfXnXhyNA%40mail.gmail.com%3E
> [2] https://issues.apache.org/jira/browse/YARN-6592
> [3] https://issues.apache.org/jira/browse/YARN-7792
> [4] https://issues.apache.org/jira/browse/YARN-7780
> [5]
> https://issues.apache.org/jira/secure/attachment/12867869/YARN-6592-Rich-Placement-Constraints-Design-V1.pdf
> [6] https://issues.apache.org/jira/browse/YARN-7812

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-12970) HdfsFileStatus#getPath returning null.

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas resolved HDFS-12970.
--
Resolution: Invalid

> HdfsFileStatus#getPath returning null.
> --
>
> Key: HDFS-12970
> URL: https://issues.apache.org/jira/browse/HDFS-12970
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Affects Versions: 3.1.0
>Reporter: Rushabh S Shah
>Priority: Critical
>
> After HDFS-12681, HdfsFileStatus#getPath() returns null.
> I don't think this is expected.
> Both the implementation of {{HdfsFileStatus}} sets the path to null.
> {code:title=HdfsNamedFileStatus.java|borderStyle=solid}
>   HdfsNamedFileStatus(long length, boolean isdir, int replication,
>   long blocksize, long mtime, long atime,
>   FsPermission permission, Set flags,
>   String owner, String group,
>   byte[] symlink, byte[] path, long fileId,
>   int childrenNum, FileEncryptionInfo feInfo,
>   byte storagePolicy, ErasureCodingPolicy ecPolicy) {
> super(length, isdir, replication, blocksize, mtime, atime,
> HdfsFileStatus.convert(isdir, symlink != null, permission, flags),
> owner, group, null, null,   -- The last null is for path.
> HdfsFileStatus.convert(flags));
> {code}
> {code:title=HdfsLocatedFileStatus.java|borderStyle=solid}
>   HdfsLocatedFileStatus(long length, boolean isdir, int replication,
> long blocksize, long mtime, long atime,
> FsPermission permission, EnumSet flags,
> String owner, String group,
> byte[] symlink, byte[] path, long fileId,
> int childrenNum, FileEncryptionInfo feInfo,
> byte storagePolicy, ErasureCodingPolicy ecPolicy,
> LocatedBlocks hdfsloc) {
> super(length, isdir, replication, blocksize, mtime, atime,
> HdfsFileStatus.convert(isdir, symlink != null, permission, flags),
> owner, group, null, null, HdfsFileStatus.convert(flags),  -- The last 
> null on this line is for path.
> null);
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12944) Update NOTICE for AssertJ dependency

[Chris Douglas (JIRA)]

Chris Douglas created HDFS-12944:


 Summary: Update NOTICE for AssertJ dependency
 Key: HDFS-12944
 URL: https://issues.apache.org/jira/browse/HDFS-12944
 Project: Hadoop HDFS
  Issue Type: Task
Reporter: Chris Douglas


HDFS-12665 added a dependency on the ALv2 
[AssertJ|https://github.com/joel-costigliola/assertj-core] library. We should 
update the notice.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Merge HDFS-9806 to trunk

[Chris Douglas]

+1 to close it out.

With 6 +1 votes (5 binding) this passes.

Thanks everyone, particularly Sean and Inigo for trying it out. We'll
merge this soon. -C

On Fri, Dec 8, 2017 at 3:44 PM, Chris Douglas <cdoug...@apache.org> wrote:
> Discussion thread: https://s.apache.org/kxT1
>
> We're down to the last few issues and are preparing the branch to
> merge to trunk. We'll post merge patches to HDFS-9806 [1]. Minor,
> "cleanup" tasks (checkstyle, findbugs, naming, etc.) will be tracked
> in HDFS-12712 [2].
>
> We've tried to ensure that when this feature is disabled, HDFS is
> unaffected. For those reviewing this, please look for places where
> this might add overheads and we'll address them before the merge. The
> site documentation [3] and design doc [4] should be up to date and
> sufficient to try this out. Again, please point out where it is
> unclear and we can address it.
>
> This has been a long effort and we're grateful for the support we've
> received from the community. In particular, thanks to Íñigo Goiri,
> Andrew Wang, Anu Engineer, Steve Loughran, Sean Mackrory, Lukas
> Majercak, Uma Gunuganti, Kai Zheng, Rakesh Radhakrishnan, Sriram Rao,
> Lei Xu, Zhe Zhang, Jing Zhao, Bharat Viswanadham, ATM, Chris Nauroth,
> Sanjay Radia, Atul Sikaria, and Peng Li for all your input into the
> design, testing, and review of this feature.
>
> The vote will close no earlier than one week from today, 12/15. -C
>
> [1]: https://issues.apache.org/jira/browse/HDFS-9806
> [2]: https://issues.apache.org/jira/browse/HDFS-12712
> [3]: 
> https://github.com/apache/hadoop/blob/HDFS-9806/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsProvidedStorage.md
> [4]: 
> https://issues.apache.org/jira/secure/attachment/12875791/HDFS-9806-design.002.pdf

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-12903) [READ] Fix closing streams in ImageWriter

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas resolved HDFS-12903.
--
Resolution: Fixed

> [READ] Fix closing streams in ImageWriter
> -
>
> Key: HDFS-12903
> URL: https://issues.apache.org/jira/browse/HDFS-12903
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Virajith Jalaparti
> Attachments: HDFS-12903-HDFS-9806.001.patch, 
> HDFS-12903-HDFS-9806.002.patch
>
>
> HDFS-12894 showed a FindBug in HDFS-9806. This seems related to HDFS-12881 
> when using {{IOUtils.cleanupWithLogger()}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Reopened] (HDFS-12903) [READ] Fix closing streams in ImageWriter

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas reopened HDFS-12903:
--

> [READ] Fix closing streams in ImageWriter
> -
>
> Key: HDFS-12903
> URL: https://issues.apache.org/jira/browse/HDFS-12903
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Virajith Jalaparti
> Attachments: HDFS-12903-HDFS-9806.001.patch
>
>
> HDFS-12894 showed a FindBug in HDFS-9806. This seems related to HDFS-12881 
> when using {{IOUtils.cleanupWithLogger()}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12926) open(PathHandle) contract test should be exhaustive for default options

[Chris Douglas (JIRA)]

Chris Douglas created HDFS-12926:


 Summary: open(PathHandle) contract test should be exhaustive for 
default options
 Key: HDFS-12926
 URL: https://issues.apache.org/jira/browse/HDFS-12926
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Chris Douglas


The current {{AbstractContractOpenTest}} covers many, but not all of the 
permutations of the default {{HandleOpt}}. It could also be refactored to be 
clearer as documentation



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[VOTE] Merge HDFS-9806 to trunk

[Chris Douglas]

Discussion thread: https://s.apache.org/kxT1

We're down to the last few issues and are preparing the branch to
merge to trunk. We'll post merge patches to HDFS-9806 [1]. Minor,
"cleanup" tasks (checkstyle, findbugs, naming, etc.) will be tracked
in HDFS-12712 [2].

We've tried to ensure that when this feature is disabled, HDFS is
unaffected. For those reviewing this, please look for places where
this might add overheads and we'll address them before the merge. The
site documentation [3] and design doc [4] should be up to date and
sufficient to try this out. Again, please point out where it is
unclear and we can address it.

This has been a long effort and we're grateful for the support we've
received from the community. In particular, thanks to Íñigo Goiri,
Andrew Wang, Anu Engineer, Steve Loughran, Sean Mackrory, Lukas
Majercak, Uma Gunuganti, Kai Zheng, Rakesh Radhakrishnan, Sriram Rao,
Lei Xu, Zhe Zhang, Jing Zhao, Bharat Viswanadham, ATM, Chris Nauroth,
Sanjay Radia, Atul Sikaria, and Peng Li for all your input into the
design, testing, and review of this feature.

The vote will close no earlier than one week from today, 12/15. -C

[1]: https://issues.apache.org/jira/browse/HDFS-9806
[2]: https://issues.apache.org/jira/browse/HDFS-12712
[3]: 
https://github.com/apache/hadoop/blob/HDFS-9806/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsProvidedStorage.md
[4]: 
https://issues.apache.org/jira/secure/attachment/12875791/HDFS-9806-design.002.pdf

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-10262) Change HdfsFileStatus::fileId to an opaque identifier

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas resolved HDFS-10262.
--
Resolution: Duplicate

Fixed in HDFS-7878

> Change HdfsFileStatus::fileId to an opaque identifier
> -
>
> Key: HDFS-10262
> URL: https://issues.apache.org/jira/browse/HDFS-10262
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client, webhdfs
>    Reporter: Chris Douglas
>
> HDFS exposes the INode ID as a long via HdfsFileStatus::getFileId. Since 
> equality is the only valid client operation (sequential/monotonically 
> increasing ids are not guaranteed in any spec; leases do not rely on any 
> other property), this identifier can be opaque instead of assigning it a 
> primitive type in HdfsFileStatus.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12885) Add visibility/stability annotations

[Chris Douglas (JIRA)]

Chris Douglas created HDFS-12885:


 Summary: Add visibility/stability annotations
 Key: HDFS-12885
 URL: https://issues.apache.org/jira/browse/HDFS-12885
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Chris Douglas
Priority: Trivial


Classes added in HDFS-9806 should include stability/visibility annotations 
(HADOOP-5073)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[DISCUSS] Merge HDFS-9806 to trunk

[Chris Douglas]

Hey hdfs-dev@-

The HDFS-9806 dev branch is getting ready to merge to trunk. This branch
adds a new storage type (PROVIDED) to support reading remote data as HDFS
blocks. Design documentation and discussion are available on JIRA [1].

Over the next week or so, we'll work through the remaining issues, write
the site documentation, and generally clean up the branch. The current code
has been stressed in experimental settings, including a large production
benchmark. When this feature is not enabled and active, it should have no
effect on HDFS. The merge will checkpoint progress on reading remote data
through HDFS, setting up work on the write path [2] and improving usability.

Please look over the branch and raise issues that should be addressed
before the merge. -C

[1]: https://issues.apache.org/jira/browse/HDFS-9806
[2]: https://issues.apache.org/jira/browse/HDFS-12090

[jira] [Created] (HDFS-12882) Support full open(PathHandle) contract in HDFS

[Chris Douglas (JIRA)]

Chris Douglas created HDFS-12882:


 Summary: Support full open(PathHandle) contract in HDFS
 Key: HDFS-12882
 URL: https://issues.apache.org/jira/browse/HDFS-12882
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client
Reporter: Chris Douglas
Assignee: Chris Douglas


HDFS-7878 added support for {{open(PathHandle)}}, but it only partially 
implemented the semantics specified in the contract (i.e., open-by-inodeID). 
HDFS should implement all permutations of the default options for 
{{PathHandle}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Reopened] (HDFS-11576) Block recovery will fail indefinitely if recovery time > heartbeat interval

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas reopened HDFS-11576:
--

Reopening for branch-2

> Block recovery will fail indefinitely if recovery time > heartbeat interval
> ---
>
> Key: HDFS-11576
> URL: https://issues.apache.org/jira/browse/HDFS-11576
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode, hdfs, namenode
>Affects Versions: 2.7.1, 2.7.2, 2.7.3, 3.0.0-alpha1, 3.0.0-alpha2
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HDFS-11576-branch-2.00.patch, HDFS-11576.001.patch, 
> HDFS-11576.002.patch, HDFS-11576.003.patch, HDFS-11576.004.patch, 
> HDFS-11576.005.patch, HDFS-11576.006.patch, HDFS-11576.007.patch, 
> HDFS-11576.008.patch, HDFS-11576.009.patch, HDFS-11576.010.patch, 
> HDFS-11576.011.patch, HDFS-11576.012.patch, HDFS-11576.013.patch, 
> HDFS-11576.014.patch, HDFS-11576.015.patch, HDFS-11576.repro.patch
>
>
> Block recovery will fail indefinitely if the time to recover a block is 
> always longer than the heartbeat interval. Scenario:
> 1. DN sends heartbeat 
> 2. NN sends a recovery command to DN, recoveryID=X
> 3. DN starts recovery
> 4. DN sends another heartbeat
> 5. NN sends a recovery command to DN, recoveryID=X+1
> 6. DN calls commitBlockSyncronization after succeeding with first recovery to 
> NN, which fails because X < X+1
> ... 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Reopened] (HDFS-11576) Block recovery will fail indefinitely if recovery time > heartbeat interval

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas reopened HDFS-11576:
--

Reopening because {{TestPipelinesFailover}} isn't spurious, this time.

> Block recovery will fail indefinitely if recovery time > heartbeat interval
> ---
>
> Key: HDFS-11576
> URL: https://issues.apache.org/jira/browse/HDFS-11576
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode, hdfs, namenode
>Affects Versions: 2.7.1, 2.7.2, 2.7.3, 3.0.0-alpha1, 3.0.0-alpha2
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HDFS-11576.001.patch, HDFS-11576.002.patch, 
> HDFS-11576.003.patch, HDFS-11576.004.patch, HDFS-11576.005.patch, 
> HDFS-11576.006.patch, HDFS-11576.007.patch, HDFS-11576.008.patch, 
> HDFS-11576.009.patch, HDFS-11576.010.patch, HDFS-11576.011.patch, 
> HDFS-11576.012.patch, HDFS-11576.013.patch, HDFS-11576.014.patch, 
> HDFS-11576.repro.patch
>
>
> Block recovery will fail indefinitely if the time to recover a block is 
> always longer than the heartbeat interval. Scenario:
> 1. DN sends heartbeat 
> 2. NN sends a recovery command to DN, recoveryID=X
> 3. DN starts recovery
> 4. DN sends another heartbeat
> 5. NN sends a recovery command to DN, recoveryID=X+1
> 6. DN calls commitBlockSyncronization after succeeding with first recovery to 
> NN, which fails because X < X+1
> ... 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12877) Add open(PathHandle) with default buffersize

[Chris Douglas (JIRA)]

Chris Douglas created HDFS-12877:


 Summary: Add open(PathHandle) with default buffersize
 Key: HDFS-12877
 URL: https://issues.apache.org/jira/browse/HDFS-12877
 Project: Hadoop HDFS
  Issue Type: Improvement
Affects Versions: 3.1.0
Reporter: Chris Douglas
Assignee: Chris Douglas
Priority: Trivial


HDFS-7878 added an overload for {{FileSystem::open}} that requires the user to 
provide a buffer size when opening by {{PathHandle}}. Similar to 
{{open(Path)}}, it'd be convenient to have another overload that takes the 
default from the config.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12874) [READ] Documentation for provided storage

[Chris Douglas (JIRA)]

Chris Douglas created HDFS-12874:


 Summary: [READ] Documentation for provided storage
 Key: HDFS-12874
 URL: https://issues.apache.org/jira/browse/HDFS-12874
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Chris Douglas


The configuration and deployment of provided storage should be documented for 
end-users.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12844) TestClientDistributedCacheManager::testDetermineCacheVisibilities assumes all parent dirs set other exec

[Chris Douglas (JIRA)]

Chris Douglas created HDFS-12844:


 Summary: 
TestClientDistributedCacheManager::testDetermineCacheVisibilities assumes all 
parent dirs set other exec
 Key: HDFS-12844
 URL: https://issues.apache.org/jira/browse/HDFS-12844
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Chris Douglas
Priority: Minor


{{TestClientDistributedCacheManager}} sets up some local directories to check 
the visibility set for dependencies, given their filesystem permissions. 
However, if it is run in an environment where the scratch directory is not 
itself PUBLIC ({{ClientDistributedCacheManager::isPublic}}), then it will fail.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Reopened] (HDFS-12681) Fold HdfsLocatedFileStatus into HdfsFileStatus

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas reopened HDFS-12681:
--

Reopening to revert this, and try another approach

> Fold HdfsLocatedFileStatus into HdfsFileStatus
> --
>
> Key: HDFS-12681
> URL: https://issues.apache.org/jira/browse/HDFS-12681
> Project: Hadoop HDFS
>  Issue Type: Bug
>        Reporter: Chris Douglas
>    Assignee: Chris Douglas
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HDFS-12681.00.patch, HDFS-12681.01.patch, 
> HDFS-12681.02.patch, HDFS-12681.03.patch, HDFS-12681.04.patch, 
> HDFS-12681.05.patch, HDFS-12681.06.patch, HDFS-12681.07.patch, 
> HDFS-12681.08.patch, HDFS-12681.09.patch, HDFS-12681.10.patch
>
>
> {{HdfsLocatedFileStatus}} is a subtype of {{HdfsFileStatus}}, but not of 
> {{LocatedFileStatus}}. Conversion requires copying common fields and shedding 
> unknown data. It would be cleaner and sufficient for {{HdfsFileStatus}} to 
> extend {{LocatedFileStatus}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC3)

[Chris Douglas]

+1 (binding)

Verified source tarball. Checksum and signature match, built from
source, ran some unit tests. Skimmed NOTICE/LICENSE. -C

On Mon, Nov 13, 2017 at 4:10 PM, Arun Suresh  wrote:
> Hi Folks,
>
> Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will be the
> starting release for Apache Hadoop 2.9.x line - it includes 30 New Features
> with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues since
> 2.8.2.
>
> More information about the 2.9.0 release plan can be found here:
> *https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
> *
>
> New RC is available at: *https://home.apache.org/~asuresh/hadoop-2.9.0-RC3/
> *
>
> The RC tag in git is: release-2.9.0-RC3, and the latest commit id is:
> 756ebc8394e473ac25feac05fa493f6d612e6c50.
>
> The maven artifacts are available via repository.apache.org at:
> *https://repository.apache.org/content/repositories/orgapachehadoop-1068/
> *
>
> We are carrying over the votes from the previous RC given that the delta is
> the license fix.
>
> Given the above - we are also going to stick with the original deadline for
> the vote : ending on Friday 17th November 2017 2pm PT time.
>
> Thanks,
> -Arun/Subru

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC0)

[Chris Douglas]

The labor required for these release formalisms is exceeding their
value. Our minor releases have more bugs than our patch releases (we
hope), but every consumer should understand how software versioning
works. Every device I own has bugs on major OS updates. That doesn't
imply that every minor release is strictly less stable than a patch
release, and users need to be warned off it.

In contrast, we should warn users about features that compromise
invariants like security or durability, either by design or due to
their early stage of development. We can't reasonably expect them to
understand those tradeoffs, since they depend on internal details of
Hadoop.

On Wed, Nov 8, 2017 at 5:34 PM, Vinod Kumar Vavilapalli
 wrote:
> When we tried option (b), we used to make .0 as a GA release, but downstream 
> projects like Tez, Hive, Spark would come back and find an incompatible 
> change - and now we were forced into a conundrum - is fixing this 
> incompatible change itself an incompatibility?

Every project takes these case-by-case. Most of the time we'll
accommodate the old semantics- and we try to be explicit where we
promise compatibility- but this isn't a logic problem, it's a
practical one. If it's an easy fix to an obscure API, we probably
won't even hear about it.

> Long story short, I'd just add to your voting thread and release notes that 
> 2.9.0 still needs to be tested downstream and so users may want to wait for 
> subsequent point releases.

It's uncomfortable to have four active release branches, with 3.1
coming in early 2018. We all benefit from the shared deployment
experiences that harden these releases, and fragmentation creates
incentives to compete for that attention. Rather than tacitly
scuffling over waning interest in the 2.x series, I'd endorse your
other thread encouraging consolidation around 3.x.

To that end, there is no policy or precedent that requires that new
minor releases be labeled as "alpha". If there is cause to believe
that 2.9.0 is not ready to release in the stable line, then we
shouldn't release it. -C

>> On Nov 8, 2017, at 12:43 AM, Subru Krishnan  wrote:
>>
>> We are canceling the RC due to the issue that Rohith/Sunil identified. The
>> issue was difficult to track down as it only happens when you use IP for ZK
>> (works fine with host names) and moreover if ZK and RM are co-located on
>> same machine. We are hopeful to get the fix in tomorrow and roll out RC1.
>>
>> Thanks to everyone for the extensive testing/validation. Hopefully cost to
>> replicate with RC1 is much lower.
>>
>> -Subru/Arun.
>>
>> On Tue, Nov 7, 2017 at 5:27 PM, Konstantinos Karanasos >> wrote:
>>
>>> +1 from me too.
>>>
>>> Did the following:
>>> 1) set up a 9-node cluster;
>>> 2) ran some Gridmix jobs;
>>> 3) ran (2) after enabling opportunistic containers (used a mix of
>>> guaranteed and opportunistic containers for each job);
>>> 4) ran (3) but this time enabling distributed scheduling of opportunistic
>>> containers.
>>>
>>> All the above worked with no issues.
>>>
>>> Thanks for all the effort guys!
>>>
>>> Konstantinos
>>>
>>>
>>>
>>> Konstantinos
>>>
>>> On Tue, Nov 7, 2017 at 2:56 PM, Eric Badger 
>>> wrote:
>>>
 +1 (non-binding) pending the issue that Sunil/Rohith pointed out

 - Verified all hashes and checksums
 - Built from source on macOS 10.12.6, Java 1.8.0u65
 - Deployed a pseudo cluster
 - Ran some example jobs

 Thanks,

 Eric

 On Tue, Nov 7, 2017 at 4:03 PM, Wangda Tan  wrote:

> Sunil / Rohith,
>
> Could you check if your configs are same as Jonathan posted configs?
> https://issues.apache.org/jira/browse/YARN-7453?
 focusedCommentId=16242693&
> page=com.atlassian.jira.plugin.system.issuetabpanels:
> comment-tabpanel#comment-16242693
>
> And could you try if using Jonathan's configs can still reproduce the
> issue?
>
> Thanks,
> Wangda
>
>
> On Tue, Nov 7, 2017 at 1:52 PM, Arun Suresh 
>>> wrote:
>
>> Thanks for testing Rohith and Sunil
>>
>> Can you please confirm if it is not a config issue at your end ?
>> We (both Jonathan and myself) just tried testing this on a fresh
 cluster
>> (both automatic and manual) and we are not able to reproduce this.
>>> I've
>> updated the YARN-7453 >> jira/browse/YARN-7453
>
>> JIRA
>> with details of testing.
>>
>> Cheers
>> -Arun/Subru
>>
>> On Tue, Nov 7, 2017 at 3:17 AM, Rohith Sharma K S <
>> rohithsharm...@apache.org
>>> wrote:
>>
>>> Thanks Sunil for confirmation. Btw, I have raised YARN-7453
>>>  JIRA to track
>>> this
>>> issue.
>>>
>>> - Rohith Sharma K S
>>>
>>> On 7 November 2017 at 16:44, Sunil G 

[jira] [Created] (HDFS-12729) Document special paths in HDFS

[Chris Douglas (JIRA)]

Chris Douglas created HDFS-12729:


 Summary: Document special paths in HDFS
 Key: HDFS-12729
 URL: https://issues.apache.org/jira/browse/HDFS-12729
 Project: Hadoop HDFS
  Issue Type: Task
  Components: documentation
Reporter: Chris Douglas


Paths under {{/.reserved}} are handled specially by HDFS. The behavior of all 
of these should be documented.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

[Chris Douglas]

Sean/Junping-

Ignoring the epistemology, it's a problem. Let's figure out what's
causing memory to balloon and then we can work out the appropriate
remedy.

Is this reproducible outside the CI environment? To Junping's point,
would YETUS-561 provide more detailed information to aid debugging? -C

On Tue, Oct 24, 2017 at 2:50 PM, Junping Du  wrote:
> In general, the "solid evidence" of memory leak comes from analysis of 
> heapdump, jastack, gc log, etc. In many cases, we can locate/conclude which 
> piece of code are leaking memory from the analysis.
>
> Unfortunately, I cannot find any conclusion from previous comments and it 
> even cannot tell which daemons/components of HDFS consumes unexpected high 
> memory. Don't sounds like a solid bug report to me.
>
>
>
> Thanks,?
>
>
> Junping
>
>
> 
> From: Sean Busbey 
> Sent: Tuesday, October 24, 2017 2:20 PM
> To: Junping Du
> Cc: Allen Wittenauer; Hadoop Common; Hdfs-dev; 
> mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
> Subject: Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86
>
> Just curious, Junping what would "solid evidence" look like? Is the 
> supposition here that the memory leak is within HDFS test code rather than 
> library runtime code? How would such a distinction be shown?
>
> On Tue, Oct 24, 2017 at 4:06 PM, Junping Du 
> > wrote:
> Allen,
>  Do we have any solid evidence to show the HDFS unit tests going through 
> the roof are due to serious memory leak by HDFS? Normally, I don't expect 
> memory leak are identified in our UTs - mostly, it (test jvm gone) is just 
> because of test or deployment issues.
>  Unless there is concrete evidence, my concern on seriously memory leak 
> for HDFS on 2.8 is relatively low given some companies (Yahoo, Alibaba, etc.) 
> have deployed 2.8 on large production environment for months. Non-serious 
> memory leak (like forgetting to close stream in non-critical path, etc.) and 
> other non-critical bugs always happens here and there that we have to live 
> with.
>
> Thanks,
>
> Junping
>
> 
> From: Allen Wittenauer 
> >
> Sent: Tuesday, October 24, 2017 8:27 AM
> To: Hadoop Common
> Cc: Hdfs-dev; 
> mapreduce-...@hadoop.apache.org; 
> yarn-...@hadoop.apache.org
> Subject: Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86
>
>> On Oct 23, 2017, at 12:50 PM, Allen Wittenauer 
>> > wrote:
>>
>>
>>
>> With no other information or access to go on, my current hunch is that one 
>> of the HDFS unit tests is ballooning in memory size.  The easiest way to 
>> kill a Linux machine is to eat all of the RAM, thanks to overcommit and 
>> that's what this "feels" like.
>>
>> Someone should verify if 2.8.2 has the same issues before a release goes out 
>> ...
>
>
> FWIW, I ran 2.8.2 last night and it has the same problems.
>
> Also: the node didn't die!  Looking through the workspace (so the 
> next run will destroy them), two sets of logs stand out:
>
> https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/ws/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
>
> and
>
> https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/ws/sourcedir/hadoop-hdfs-project/hadoop-hdfs/
>
> It looks like my hunch is correct:  RAM in the HDFS unit tests are 
> going through the roof.  It's also interesting how MANY log files there are.  
> Is surefire not picking up that jobs are dying?  Maybe not if memory is 
> getting tight.
>
> Anyway, at the point, branch-2.8 and higher are probably fubar'd. 
> Additionally, I've filed YETUS-561 so that Yetus-controlled Docker containers 
> can have their RAM limits set in order to prevent more nodes going catatonic.
>
>
>
> -
> To unsubscribe, e-mail: 
> yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: 
> yarn-dev-h...@hadoop.apache.org
>
>
>
> -
> To unsubscribe, e-mail: 
> common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: 
> common-dev-h...@hadoop.apache.org
>
>
>
>
> --
> busbey

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC1)

[Chris Douglas]

+1 (binding)

Looked through the src distribution. Checksum, signatures match, ran
some of the unit tests. Also checked the site docs; thanks for
updating the Docker container security docs.

Thanks, Junping. -C

On Thu, Oct 19, 2017 at 5:42 PM, Junping Du  wrote:
> Hi folks,
>  I've created our new release candidate (RC1) for Apache Hadoop 2.8.2.
>
>  Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line and 
> will be the latest stable/production release for Apache Hadoop - it includes 
> 315 new fixed issues since 2.8.1 and 69 fixes are marked as blocker/critical 
> issues.
>
>   More information about the 2.8.2 release plan can be found here: 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>
>   New RC is available at: 
> http://home.apache.org/~junping_du/hadoop-2.8.2-RC1
>
>   The RC tag in git is: release-2.8.2-RC1, and the latest commit id is: 
> 66c47f2a01ad9637879e95f80c41f798373828fb
>
>   The maven artifacts are available via 
> repository.apache.org at: 
> https://repository.apache.org/content/repositories/orgapachehadoop-1064
>
>   Please try the release and vote; the vote will run for the usual 5 
> days, ending on 10/24/2017 6pm PST time.
>
> Thanks,
>
> Junping
>

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12681) Fold HdfsLocatedFileStatus into HdfsFileStatus

[Chris Douglas (JIRA)]

Chris Douglas created HDFS-12681:


 Summary: Fold HdfsLocatedFileStatus into HdfsFileStatus
 Key: HDFS-12681
 URL: https://issues.apache.org/jira/browse/HDFS-12681
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Chris Douglas
Priority: Minor


{{HdfsLocatedFileStatus}} is a subtype of {{HdfsFileStatus}}, but not of 
{{LocatedFileStatus}}. Conversion requires copying common fields and shedding 
unknown data. It would be cleaner and sufficient for {{HdfsFileStatus}} to 
extend {{LocatedFileStatus}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Merge Router-Based Federation (HDFS-10467) branch into trunk/branch-3

[Chris Douglas]

+1 (binding)

Inigo has done an excellent job driving this work, both hardening it
in production and merging it into Apache.

The implementation, documentation, and tooling are mature enough to
ship with the 3.0.0 GA release. -C


On Mon, Oct 2, 2017 at 3:39 PM, Subru Krishnan <su...@apache.org> wrote:
> Thanks Inigo for driving this.
>
> +1 (binding).
>
> I have been involved during the design and have reviewed a couple of
> patches. I feel this is a good addition and is well aligned with YARN
> Federation (YARN-2915).
>
> We deployed the HDFS Router in our test cluster and mapped it against 2
> clusters - one running 3.0.0-beta1 and another 2.7.4 (to demonstrate
> isolation) and it works fine.
>
> -Subru
>
> On Fri, Sep 29, 2017 at 11:58 AM, Iñigo Goiri <elgo...@gmail.com> wrote:
>>
>> Hi all,
>>
>> Given that 3.0-beta1 is already cut, I’d like to call a vote for merging
>> Router-Based Federation (HDFS-10467) to trunk and branch-3.
>>
>> The vote will run for 7 days as usual.
>>
>>
>>
>> We started the discussion about merging HDFS-10467 a few weeks ago [1] and
>> got good feedback which we’ve incorporated already [2, 3, 4].
>>
>> There are a couple tasks left:
>>
>> HDFS-12273 for the UI. This should be completed in the next couple days.
>> HDFS-12284 for adding security. We can move this for v2 if not completed.
>>
>> We have deployed this in production for 2.7 and we did a few tests with
>> trunk a few months ago.
>>
>> This week, I’m rebasing to trunk (last one was a couple weeks ago) and
>> test trunk in one of our test clusters.
>>
>>
>> Finally, note that all the functionality is in the Router (a new
>> component) so everything is isolated.
>>
>> In addition, no new APIs have been added and we rely fully in
>> ClientProtocol.
>>
>>
>>
>> I’d like to thank the people at Microsoft (specially, Jason, Ricardo,
>> Chris, Subru, Jakob, Carlo and Giovanni), Twitter (Ming and Gera), LinkedIn
>> (Zhe, Erik and Konstantin), and Cloudera (Andrew and Manoj) for the
>> discussion and the ideas.
>>
>> Special thanks to Chris Douglas for the thorough reviews!
>>
>>
>>
>> Regards,
>> Inigo
>>
>>
>>
>> [1]
>> http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201708.mbox/%3CCAB1dGgogTu6kHtkkYeUycmNv-H3RupfPF4Cd7rpuFi6vHGdBLg%40mail.gmail.com%3E
>>
>> [2] https://issues.apache.org/jira/browse/HDFS-12381
>>
>> [3] https://issues.apache.org/jira/browse/HDFS-12430
>>
>> [4] https://issues.apache.org/jira/browse/HDFS-12450
>
>

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

[Chris Douglas]

On Mon, Sep 11, 2017 at 2:52 PM, Junping Du <j...@hortonworks.com> wrote:
> I don't think this -1 is reasonable, because:
> - If you look at YARN-6622 closely, it targets to fix a problematic 
> documentation work on YARN-5258 which get checked into 2.9 and 3.0 branch 
> only. It means it targets to fix a problem that 2.8.2 never exists.

...we're not going to document security implications- which include
escalations to root- because we don't have _any_ documentation? Why
don't we backport the documentation?

> - New docker container support (replace of old DockerContainerExectutor) is 
> still an alpha feature now which doesn't highlight in 2.8 major 
> features/improvement (http://hadoop.apache.org/docs/r2.8.0/index.html). So 
> adding documentation here is also not a blocker.

YARN-6622 is *documenting* the fact that this is an alpha feature and
that it shouldn't be enabled in secure environments. How are users
supposed to make this determination without it?

> Vote still continue until a real blocker comes.

Soright. I remain -1. -C

> ________
> From: Chris Douglas <cdoug...@apache.org>
> Sent: Monday, September 11, 2017 12:00 PM
> To: Junping Du
> Cc: Miklos Szegedi; Mingliang Liu; Hadoop Common; Hdfs-dev; 
> mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org; junping_du
> Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)
>
> -1 (binding)
>
> I don't think we should release this without YARN-6622.
>
> Since this doesn't happen often: a -1 in this case is NOT a veto.
> Releases are approved by majority vote of the PMC. -C
>
> On Mon, Sep 11, 2017 at 11:45 AM, Junping Du <j...@hortonworks.com> wrote:
>> Thanks Mikols for notifying on this. I think docker support is general known 
>> as alpha feature so document it as experimental is nice to have but not a 
>> blocker for 2.8.2. I also noticed that our 2.7.x document 
>> (https://hadoop.apache.org/docs/r2.7.4/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html)
>>  without mentioning docker support is experimental. We may need to fix that 
>> as well in following releases.
>>
>> I can also add it (mentioning docker container support feature is 
>> experimental) to release message in public website just like previous 
>> release we call 2.7.0/2.8.0 as non-production release.
>>
>> I think vote should continue until we could find a real blocker.
>>
>>
>> Thanks,
>>
>>
>> Junping
>>
>>
>> 
>> From: Miklos Szegedi <miklos.szeg...@cloudera.com>
>> Sent: Monday, September 11, 2017 10:07 AM
>> To: Mingliang Liu
>> Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org; 
>> yarn-...@hadoop.apache.org; junping_du; Junping Du
>> Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)
>>
>> Hello Junping,
>>
>> Thank you for working on this. Should not YARN-6622 be addressed first? 
>> "Summary: Document Docker work as experimental".
>>
>> Thank you,
>> Miklos
>>
>>
>> On Sun, Sep 10, 2017 at 6:39 PM, Mingliang Liu 
>> <lium...@gmail.com<mailto:lium...@gmail.com>> wrote:
>> Thanks Junping for doing this!
>>
>> +1 (non-binding)
>>
>> - Download the hadoop-2.8.2-src.tar.gz file and checked the md5 value
>> - Build package using maven (skipping tests) with Java 8
>> - Spin up a test cluster in Docker containers having 1 master node (NN/RM) 
>> and 3 slave nodes (DN/NM)
>> - Operate the basic HDFS/YARN operations from command line, both client and 
>> admin
>> - Check NN/RM Web UI
>> - Run distcp to copy files from/to local and HDFS
>> - Run hadoop mapreduce examples: grep and wordcount
>> - Check the HDFS service logs
>>
>> All looked good to me.
>>
>> Mingliang
>>
>>> On Sep 10, 2017, at 5:00 PM, Junping Du 
>>> <j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
>>>
>>> Hi folks,
>>> With fix of HADOOP-14842 get in, I've created our first release 
>>> candidate (RC0) for Apache Hadoop 2.8.2.
>>>
>>> Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line and 
>>> will be the latest stable/production release for Apache Hadoop - it 
>>> includes 305 new fixed issues since 2.8.1 and 63 fixes are marked as 
>>> blocker/critical issues.
>>>
>>>  More information about the 2.8.2 release plan can be found here: 
>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>>>
>>>  New RC is available at: 
>>> ht

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

[Chris Douglas]

-1 (binding)

I don't think we should release this without YARN-6622.

Since this doesn't happen often: a -1 in this case is NOT a veto.
Releases are approved by majority vote of the PMC. -C

On Mon, Sep 11, 2017 at 11:45 AM, Junping Du  wrote:
> Thanks Mikols for notifying on this. I think docker support is general known 
> as alpha feature so document it as experimental is nice to have but not a 
> blocker for 2.8.2. I also noticed that our 2.7.x document 
> (https://hadoop.apache.org/docs/r2.7.4/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html)
>  without mentioning docker support is experimental. We may need to fix that 
> as well in following releases.
>
> I can also add it (mentioning docker container support feature is 
> experimental) to release message in public website just like previous release 
> we call 2.7.0/2.8.0 as non-production release.
>
> I think vote should continue until we could find a real blocker.
>
>
> Thanks,
>
>
> Junping
>
>
> 
> From: Miklos Szegedi 
> Sent: Monday, September 11, 2017 10:07 AM
> To: Mingliang Liu
> Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org; 
> yarn-...@hadoop.apache.org; junping_du; Junping Du
> Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)
>
> Hello Junping,
>
> Thank you for working on this. Should not YARN-6622 be addressed first? 
> "Summary: Document Docker work as experimental".
>
> Thank you,
> Miklos
>
>
> On Sun, Sep 10, 2017 at 6:39 PM, Mingliang Liu 
> > wrote:
> Thanks Junping for doing this!
>
> +1 (non-binding)
>
> - Download the hadoop-2.8.2-src.tar.gz file and checked the md5 value
> - Build package using maven (skipping tests) with Java 8
> - Spin up a test cluster in Docker containers having 1 master node (NN/RM) 
> and 3 slave nodes (DN/NM)
> - Operate the basic HDFS/YARN operations from command line, both client and 
> admin
> - Check NN/RM Web UI
> - Run distcp to copy files from/to local and HDFS
> - Run hadoop mapreduce examples: grep and wordcount
> - Check the HDFS service logs
>
> All looked good to me.
>
> Mingliang
>
>> On Sep 10, 2017, at 5:00 PM, Junping Du 
>> > wrote:
>>
>> Hi folks,
>> With fix of HADOOP-14842 get in, I've created our first release 
>> candidate (RC0) for Apache Hadoop 2.8.2.
>>
>> Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line and 
>> will be the latest stable/production release for Apache Hadoop - it includes 
>> 305 new fixed issues since 2.8.1 and 63 fixes are marked as blocker/critical 
>> issues.
>>
>>  More information about the 2.8.2 release plan can be found here: 
>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>>
>>  New RC is available at: 
>> http://home.apache.org/~junping_du/hadoop-2.8.2-RC0
>>
>>  The RC tag in git is: release-2.8.2-RC0, and the latest commit id is: 
>> e6597fe3000b06847d2bf55f2bab81770f4b2505
>>
>>  The maven artifacts are available via 
>> repository.apache.org at: 
>> https://repository.apache.org/content/repositories/orgapachehadoop-1062
>>
>>  Please try the release and vote; the vote will run for the usual 5 
>> days, ending on 09/15/2017 5pm PST time.
>>
>> Thanks,
>>
>> Junping
>>
>
>
> -
> To unsubscribe, e-mail: 
> mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: 
> mapreduce-dev-h...@hadoop.apache.org
>
>

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12417) Disable flaky TestDFSStripedOutputStreamWithFailure

[Chris Douglas (JIRA)]

Chris Douglas created HDFS-12417:


 Summary: Disable flaky TestDFSStripedOutputStreamWithFailure
 Key: HDFS-12417
 URL: https://issues.apache.org/jira/browse/HDFS-12417
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: test
Reporter: Chris Douglas


Some subset of TestDFSStripedOutputStreamWithFailure tests almost always fail 
in test-patch runs. Since its failure is no longer seen as a blocker for 
commit, it should be disabled until it is more reliable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Reopened] (HDFS-12349) Improve log message when it could not alloc enough blocks for EC

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas reopened HDFS-12349:
--

> Improve log message when it could not alloc enough blocks for EC 
> -
>
> Key: HDFS-12349
> URL: https://issues.apache.org/jira/browse/HDFS-12349
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Minor
> Fix For: 3.0.0-beta1
>
> Attachments: HDFS-12349.00.patch, HDFS-12349.01.patch, 
> HDFS-12349.02.patch
>
>
> When an EC output stream could not alloc enough blocks for parity blocks, it 
> sets the warning.
> {code}
> if (blocks[i] == null) {
> LOG.warn("Failed to get block location for parity block, index=" + i);
> {code}
> We should clarify the cause of this warning message.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12410) Ignore unknown StorageTypes

[Chris Douglas (JIRA)]

Chris Douglas created HDFS-12410:


 Summary: Ignore unknown StorageTypes
 Key: HDFS-12410
 URL: https://issues.apache.org/jira/browse/HDFS-12410
 Project: Hadoop HDFS
  Issue Type: Task
  Components: datanode, fs
Reporter: Chris Douglas
Priority: Minor


A storage configured with an unknown type will cause runtime exceptions. 
Instead, these storages can be ignored/skipped.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DISCUSS] Merge HDFS-10467 to (Router-based federation) trunk

[Chris Douglas]

I'd definitely support merging this to trunk. The implementation is
almost entirely outside of HDFS and, as Inigo detailed, has been
tested at scale. The branch is in a functional state with
documentation and tests. -C

On Mon, Aug 21, 2017 at 6:11 PM, Iñigo Goiri <elgo...@gmail.com> wrote:
> Hi all,
>
>
>
> We would like to open a discussion on merging the Router-based Federation
> feature to trunk.
>
> Last week, there was a thread about which branches would go into 3.0 and
> given that YARN federation is going, this might be a good time for this to
> be merged too.
>
>
> We have been running "Router-based federation" in production for a year.
>
> Meanwhile, we have been releasing it in a feature branch (HDFS-10467 [1])
> for a while.
>
> We are reasonably confident that the state of the branch is about to meet
> the criteria to be merged onto trunk.
>
>
> *Feature*:
>
> This feature aggregates multiple namespaces into a single one transparently
> to the user.
>
> It has a similar architecture to YARN federation (YARN-2915).
>
> It consists on Routers that handle requests from the clients and forwards
> them to the right subcluster and exposes the same API as the Namenode.
>
> Currently we use a mount table (similar to ViewFs) but can be replaced by
> other approaches.
>
> The Routers share their state in a State Store.
>
>
>
> The main advantage is that clients interact with the Routers as they were
> Namenode so there is no changes in the client required other than poiting
> to the right address.
>
> In addition, all the management is moved to the server side so changes to
> the Mount Table can be done without having to sync the clients (pull/push).
>
>
>
> *Status*:
>
> The branch already contains all the features required to work end-to-end.
>
> There are a couple open JIRAs that would be required for the merged (i.e.,
> Web UI) but they should be finished soon.
>
> We have been running it in production for the last year and we have a paper
> with some of the details of our production deployment [2].
>
> We have 4 production deployments with the largest one spanning more than
> 20k servers across 6 subclusters.
>
> In addition, the guys at LinkedIn had started testing Router-based
> federation and they will be adding security to the branch.
>
>
>
> The modifications to the rest of HDFS are minimal:
>
>- Changed visibility for some methods (e.g., MiniDFSCluster)
>- Added some utilities to extract addresses
>- Modified hdfs and hdfs.cmd to start the Router and manager the
>federation
>- Modified hdfs-default.xml
>
> Everything else is self-contained in a federation package.
>
> In addition, all the functionality is in the Router so it’s disabled by
> default.
>
> Even when enabled, there is no impact for regular HDFS and it would only
> require to configure the trust between the Namenode and the Router once
> security is enabled.
>
>
>
> I have been continuously rebasing the feature branch (updated up to 1 week
> ago) so the merge should be a straightforward cherry-pick.
>
>
>
> *Problems*:
>
> The problems I’m aware of are the following:
>
>- We implement ClientProtocol so anytime a new method is added there, we
>would need to add it to the Router. However, it’s straightforward to add
>unimplemented methods.
>- There is some argument about naming the feature as “Router-based
>    federation” but I’m open for better names.
>
>
>
> *Credits*:
>
> I’d like to thank the people at Microsoft (specially, Jason, Ricardo,
> Chris, Subru, Jakob, Carlo and Giovanni), Twitter (Ming and Gera), and
> LinkedIn (Zhe, Erik and Konstantin) for the discussion and the ideas.
>
> Special thanks to Chris Douglas for the thorough reviews!
>
>
>
> Please look through the branch; feedback is welcome. Thanks!
>
>
> Cheers,
>
> Inigo
>
>
>
>
> [1] https://issues.apache.org/jira/browse/HDFS-10467
>
> [2] https://www.usenix.org/conference/atc17/technical-
> sessions/presentation/misra

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Chris Douglas]

+1 (binding)

Looked through the src tarball. Checksum and signature match, skimmed
NOTICE/LICENSE, ran some unit tests. -C

On Sat, Jul 29, 2017 at 4:29 PM, Konstantin Shvachko
 wrote:
> Hi everybody,
>
> Here is the next release of Apache Hadoop 2.7 line. The previous stable
> release 2.7.3 was available since 25 August, 2016.
> Release 2.7.4 includes 264 issues fixed after release 2.7.3, which are
> critical bug fixes and major optimizations. See more details in Release
> Note:
> http://home.apache.org/~shv/hadoop-2.7.4-RC0/releasenotes.html
>
> The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.4-RC0/
>
> Please give it a try and vote on this thread. The vote will run for 5 days
> ending 08/04/2017.
>
> Please note that my up to date public key are available from:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> Please don't forget to refresh the page if you've been there recently.
> There are other place on Apache sites, which may contain my outdated key.
>
> Thanks,
> --Konstantin

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12250) Reduce usage of FsPermissionExtension in HDFS unit tests

[Chris Douglas (JIRA)]

Chris Douglas created HDFS-12250:


 Summary: Reduce usage of FsPermissionExtension in HDFS unit tests
 Key: HDFS-12250
 URL: https://issues.apache.org/jira/browse/HDFS-12250
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: test
Affects Versions: 3.0.0-alpha4
Reporter: Chris Douglas
Priority: Minor


HDFS-6984 deprecated FsPermissionExtension, moving the flags to FileStatus. 
This generated a large number of deprecation warnings, particularly in unit 
tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Chris Douglas]

On Wed, Aug 2, 2017 at 12:02 AM, Zhe Zhang <z...@apache.org> wrote:
> Quick question for Chris: was your +1 for the RC, or to support
> Konstantin's statement regarding packaging?

For Konstantin's statement. I haven't had a chance to look through the
RC yet, but it's on my list. -C

> On Mon, Jul 31, 2017 at 3:56 PM Chris Douglas <cdoug...@apache.org> wrote:
>
>> On Mon, Jul 31, 2017 at 3:02 PM, Konstantin Shvachko
>> <shv.had...@gmail.com> wrote:
>> > For the packaging, here is the exact phrasing from the sited
>> release-policy
>> > document relevant to binaries:
>> > "As a convenience to users that might not have the appropriate tools to
>> > build a compiled version of the source, binary/bytecode packages MAY be
>> > distributed alongside official Apache releases. In all such cases, the
>> > binary/bytecode package MUST have the same version number as the source
>> > release and MUST only add binary/bytecode files that are the result of
>> > compiling that version of the source code release and its dependencies."
>> > I don't think my binary package violates any of these.
>>
>> +1 The PMC VOTE applies to source code, only. If someone wants to
>> rebuild the binary tarball with native libs and replace this one,
>> that's fine.
>>
>> My reading of the above is that source code must be distributed with
>> binaries, not that we omit the source code from binary releases... -C
>>
>> > But I'll upload an additional tar.gz with native bits and no src, as you
>> > guys requested.
>> > Will keep it as RC0 as there is no source code change and it comes from
>> the
>> > same build.
>> > Hope this is satisfactory.
>> >
>> > Thanks,
>> > --Konstantin
>> >
>> > On Mon, Jul 31, 2017 at 1:53 PM, Andrew Wang <andrew.w...@cloudera.com>
>> > wrote:
>> >
>> >> I agree with Brahma on the two issues flagged (having src in the binary
>> >> tarball, missing native libs). These are regressions from prior
>> releases.
>> >>
>> >> As an aside, "we release binaries as a convenience" doesn't relax the
>> >> quality bar. The binaries are linked on our website and distributed
>> through
>> >> official Apache channels. They have to adhere to Apache release
>> >> requirements. And, most users consume our work via Maven dependencies,
>> >> which are binary artifacts.
>> >>
>> >> http://www.apache.org/legal/release-policy.html goes into this in more
>> >> detail. A release must minimally include source packages, and can also
>> >> include binary artifacts.
>> >>
>> >> Best,
>> >> Andrew
>> >>
>> >> On Mon, Jul 31, 2017 at 12:30 PM, Konstantin Shvachko <
>> >> shv.had...@gmail.com> wrote:
>> >>
>> >>> To avoid any confusion in this regard. I built RC0 manually in
>> compliance
>> >>> with Apache release policy
>> >>> http://www.apache.org/legal/release-policy.html
>> >>> I edited the HowToReleasePreDSBCR page to make sure people don't use
>> >>> Jenkins option for building.
>> >>>
>> >>> A side note. This particular build is broken anyways, so no worries
>> there.
>> >>> I think though it would be useful to have it working for testing and
>> as a
>> >>> packaging standard.
>> >>>
>> >>> Thanks,
>> >>> --Konstantin
>> >>>
>> >>> On Mon, Jul 31, 2017 at 11:40 AM, Allen Wittenauer <
>> >>> a...@effectivemachines.com
>> >>> > wrote:
>> >>>
>> >>> >
>> >>> > > On Jul 31, 2017, at 11:20 AM, Konstantin Shvachko <
>> >>> shv.had...@gmail.com>
>> >>> > wrote:
>> >>> > >
>> >>> > > https://wiki.apache.org/hadoop/HowToReleasePreDSBCR
>> >>> >
>> >>> > FYI:
>> >>> >
>> >>> > If you are using ASF Jenkins to create an ASF release
>> >>> > artifact, it's pretty much an automatic vote failure as any such
>> >>> release is
>> >>> > in violation of ASF policy.
>> >>> >
>> >>> >
>> >>>
>> >>
>> >>
>>
>> -
>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>
>> --
> Zhe Zhang
> Apache Hadoop Committer
> http://zhe-thoughts.github.io/about/ | @oldcap

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Chris Douglas]

On Mon, Jul 31, 2017 at 3:02 PM, Konstantin Shvachko
 wrote:
> For the packaging, here is the exact phrasing from the sited release-policy
> document relevant to binaries:
> "As a convenience to users that might not have the appropriate tools to
> build a compiled version of the source, binary/bytecode packages MAY be
> distributed alongside official Apache releases. In all such cases, the
> binary/bytecode package MUST have the same version number as the source
> release and MUST only add binary/bytecode files that are the result of
> compiling that version of the source code release and its dependencies."
> I don't think my binary package violates any of these.

+1 The PMC VOTE applies to source code, only. If someone wants to
rebuild the binary tarball with native libs and replace this one,
that's fine.

My reading of the above is that source code must be distributed with
binaries, not that we omit the source code from binary releases... -C

> But I'll upload an additional tar.gz with native bits and no src, as you
> guys requested.
> Will keep it as RC0 as there is no source code change and it comes from the
> same build.
> Hope this is satisfactory.
>
> Thanks,
> --Konstantin
>
> On Mon, Jul 31, 2017 at 1:53 PM, Andrew Wang 
> wrote:
>
>> I agree with Brahma on the two issues flagged (having src in the binary
>> tarball, missing native libs). These are regressions from prior releases.
>>
>> As an aside, "we release binaries as a convenience" doesn't relax the
>> quality bar. The binaries are linked on our website and distributed through
>> official Apache channels. They have to adhere to Apache release
>> requirements. And, most users consume our work via Maven dependencies,
>> which are binary artifacts.
>>
>> http://www.apache.org/legal/release-policy.html goes into this in more
>> detail. A release must minimally include source packages, and can also
>> include binary artifacts.
>>
>> Best,
>> Andrew
>>
>> On Mon, Jul 31, 2017 at 12:30 PM, Konstantin Shvachko <
>> shv.had...@gmail.com> wrote:
>>
>>> To avoid any confusion in this regard. I built RC0 manually in compliance
>>> with Apache release policy
>>> http://www.apache.org/legal/release-policy.html
>>> I edited the HowToReleasePreDSBCR page to make sure people don't use
>>> Jenkins option for building.
>>>
>>> A side note. This particular build is broken anyways, so no worries there.
>>> I think though it would be useful to have it working for testing and as a
>>> packaging standard.
>>>
>>> Thanks,
>>> --Konstantin
>>>
>>> On Mon, Jul 31, 2017 at 11:40 AM, Allen Wittenauer <
>>> a...@effectivemachines.com
>>> > wrote:
>>>
>>> >
>>> > > On Jul 31, 2017, at 11:20 AM, Konstantin Shvachko <
>>> shv.had...@gmail.com>
>>> > wrote:
>>> > >
>>> > > https://wiki.apache.org/hadoop/HowToReleasePreDSBCR
>>> >
>>> > FYI:
>>> >
>>> > If you are using ASF Jenkins to create an ASF release
>>> > artifact, it's pretty much an automatic vote failure as any such
>>> release is
>>> > in violation of ASF policy.
>>> >
>>> >
>>>
>>
>>

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Can we update protobuf's version on trunk?

[Chris Douglas]

On Wed, Mar 29, 2017 at 4:59 PM, Stack  wrote:
>> The former; an intermediate handler decoding, [modifying,] and
>> encoding the record without losing unknown fields.
>>
>
> I did not try this. Did you? Otherwise I can.

Yeah, I did. Same format. -C

>> This looks fine. -C
>>
>> > Thanks,
>> > St.Ack
>> >
>> >
>> > # Using the protoc v3.0.2 tool
>> > $ protoc --version
>> > libprotoc 3.0.2
>> >
>> > # I have a simple proto definition with two fields in it
>> > $ more pb.proto
>> > message Test {
>> >   optional string one = 1;
>> >   optional string two = 2;
>> > }
>> >
>> > # This is a text-encoded instance of a 'Test' proto message:
>> > $ more pb.txt
>> > one: "one"
>> > two: "two"
>> >
>> > # Now I encode the above as a pb binary
>> > $ protoc --encode=Test pb.proto < pb.txt > pb.bin
>> > [libprotobuf WARNING google/protobuf/compiler/parser.cc:546] No syntax
>> > specified for the proto file: pb.proto. Please use 'syntax = "proto2";'
>> > or
>> > 'syntax = "proto3";' to specify a syntax version. (Defaulted to proto2
>> > syntax.)
>> >
>> > # Here is a dump of the binary
>> > $ od -xc pb.bin
>> > 000  030a6e6f126574036f77
>> >   \n 003   o   n   e 022 003   t   w   o
>> > 012
>> >
>> > # Here is a proto definition file that has a Test Message minus the
>> > 'two'
>> > field.
>> > $ more pb_drops_two.proto
>> > message Test {
>> >   optional string one = 1;
>> > }
>> >
>> > # Use it to decode the bin file:
>> > $ protoc --decode=Test pb_drops_two.proto < pb.bin
>> > [libprotobuf WARNING google/protobuf/compiler/parser.cc:546] No syntax
>> > specified for the proto file: pb_drops_two.proto. Please use 'syntax =
>> > "proto2";' or 'syntax = "proto3";' to specify a syntax version.
>> > (Defaulted
>> > to proto2 syntax.)
>> > one: "one"
>> > 2: "two"
>> >
>> > Note how the second field is preserved (absent a field name). It is not
>> > dropped.
>> >
>> > If I change the syntax of pb_drops_two.proto to be proto3, the field IS
>> > dropped.
>> >
>> > # Here proto file with proto3 syntax specified (had to drop the
>> > 'optional'
>> > qualifier -- not allowed in proto3):
>> > $ more pb_drops_two.proto
>> > syntax = "proto3";
>> > message Test {
>> >   string one = 1;
>> > }
>> >
>> > $ protoc --decode=Test pb_drops_two.proto < pb.bin  > pb_drops_two.txt
>> > $ more pb_drops_two.txt
>> > one: "one"
>> >
>> >
>> > I cannot reencode the text output using pb_drops_two.proto. It
>> > complains:
>> >
>> > $ protoc --encode=Test pb_drops_two.proto < pb_drops_two.txt >
>> > pb_drops_two.bin
>> > [libprotobuf WARNING google/protobuf/compiler/parser.cc:546] No syntax
>> > specified for the proto file: pb_drops_two.proto. Please use 'syntax =
>> > "proto2";' or 'syntax = "proto3";' to specify a syntax version.
>> > (Defaulted
>> > to proto2 syntax.)
>> > input:2:1: Expected identifier, got: 2
>> >
>> > Proto 2.5 does same:
>> >
>> > $ ~/bin/protobuf-2.5.0/src/protoc --encode=Test pb_drops_two.proto <
>> > pb_drops_two.txt > pb_drops_two.bin
>> > input:2:1: Expected identifier.
>> > Failed to parse input.
>> >
>> > St.Ack
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Wed, Mar 29, 2017 at 10:14 AM, Stack  wrote:
>> >>
>> >> On Tue, Mar 28, 2017 at 4:18 PM, Andrew Wang 
>> >> wrote:
>> >>>
>> >>> >
>> >>> > > If unknown fields are dropped, then applications proxying tokens
>> >>> > > and
>> >>> > other
>> >>> > >> data between servers will effectively corrupt those messages,
>> >>> > >> unless
>> >>> > >> we
>> >>> > >> make everything opaque bytes, which- absent the convenient,
>> >>> > >> prenominate
>> >>> > >> semantics managing the conversion- obviate the compatibility
>> >>> > >> machinery
>> >>> > that
>> >>> > >> is the whole point of PB. Google is removing the features that
>> >>> > >> justified
>> >>> > >> choosing PB over its alternatives. Since we can't require that
>> >>> > >> our
>> >>> > >> applications compile (or link) against our updated schema, this
>> >>> > >> creates
>> >>> > a
>> >>> > >> problem that PB was supposed to solve.
>> >>> > >
>> >>> > >
>> >>> > > This is scary, and it potentially affects services outside of the
>> >>> > > Hadoop
>> >>> > > codebase. This makes it difficult to assess the impact.
>> >>> >
>> >>> > Stack mentioned a compatibility mode that uses the proto2 semantics.
>> >>> > If that carries unknown fields through intermediate handlers, then
>> >>> > this objection goes away. -C
>> >>>
>> >>>
>> >>> Did some more googling, found this:
>> >>>
>> >>> https://groups.google.com/d/msg/protobuf/Z6pNo81FiEQ/fHkdcNtdAwAJ
>> >>>
>> >>> Feng Xiao appears to be a Google engineer, and suggests workarounds
>> >>> like
>> >>> packing the fields into a byte type. No mention of a PB2 compatibility
>> >>> mode. Also here:
>> >>>
>> >>> https://groups.google.com/d/msg/protobuf/bO2L6-_t91Q/-zIaJAR9AAAJ
>> >>>
>> >>> Participants say that unknown fields were dropped for automatic JSON
>> >>> encoding, since 

Re: Can we update protobuf's version on trunk?

[Chris Douglas]

On Wed, Mar 29, 2017 at 1:13 PM, Stack  wrote:
> Is the below evidence enough that pb3 in proto2 syntax mode does not drop
> 'unknown' fields? (Maybe you want evidence that java tooling behaves the
> same?)

I reproduced your example with the Java tooling, including changing
some of the fields in the intermediate representation. As long as the
syntax is "proto2", it seems to have compatible semantics.

> To be clear, when we say proxy above, are we expecting that a pb message
> deserialized by a process down-the-line that happens to have a crimped proto
> definition that is absent a couple of fields somehow can re-serialize and at
> the end of the line, all fields are present? Or are we talking pass-through
> of the message without rewrite?

The former; an intermediate handler decoding, [modifying,] and
encoding the record without losing unknown fields.

This looks fine. -C

> Thanks,
> St.Ack
>
>
> # Using the protoc v3.0.2 tool
> $ protoc --version
> libprotoc 3.0.2
>
> # I have a simple proto definition with two fields in it
> $ more pb.proto
> message Test {
>   optional string one = 1;
>   optional string two = 2;
> }
>
> # This is a text-encoded instance of a 'Test' proto message:
> $ more pb.txt
> one: "one"
> two: "two"
>
> # Now I encode the above as a pb binary
> $ protoc --encode=Test pb.proto < pb.txt > pb.bin
> [libprotobuf WARNING google/protobuf/compiler/parser.cc:546] No syntax
> specified for the proto file: pb.proto. Please use 'syntax = "proto2";' or
> 'syntax = "proto3";' to specify a syntax version. (Defaulted to proto2
> syntax.)
>
> # Here is a dump of the binary
> $ od -xc pb.bin
> 000  030a6e6f126574036f77
>   \n 003   o   n   e 022 003   t   w   o
> 012
>
> # Here is a proto definition file that has a Test Message minus the 'two'
> field.
> $ more pb_drops_two.proto
> message Test {
>   optional string one = 1;
> }
>
> # Use it to decode the bin file:
> $ protoc --decode=Test pb_drops_two.proto < pb.bin
> [libprotobuf WARNING google/protobuf/compiler/parser.cc:546] No syntax
> specified for the proto file: pb_drops_two.proto. Please use 'syntax =
> "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted
> to proto2 syntax.)
> one: "one"
> 2: "two"
>
> Note how the second field is preserved (absent a field name). It is not
> dropped.
>
> If I change the syntax of pb_drops_two.proto to be proto3, the field IS
> dropped.
>
> # Here proto file with proto3 syntax specified (had to drop the 'optional'
> qualifier -- not allowed in proto3):
> $ more pb_drops_two.proto
> syntax = "proto3";
> message Test {
>   string one = 1;
> }
>
> $ protoc --decode=Test pb_drops_two.proto < pb.bin  > pb_drops_two.txt
> $ more pb_drops_two.txt
> one: "one"
>
>
> I cannot reencode the text output using pb_drops_two.proto. It complains:
>
> $ protoc --encode=Test pb_drops_two.proto < pb_drops_two.txt >
> pb_drops_two.bin
> [libprotobuf WARNING google/protobuf/compiler/parser.cc:546] No syntax
> specified for the proto file: pb_drops_two.proto. Please use 'syntax =
> "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted
> to proto2 syntax.)
> input:2:1: Expected identifier, got: 2
>
> Proto 2.5 does same:
>
> $ ~/bin/protobuf-2.5.0/src/protoc --encode=Test pb_drops_two.proto <
> pb_drops_two.txt > pb_drops_two.bin
> input:2:1: Expected identifier.
> Failed to parse input.
>
> St.Ack
>
>
>
>
>
>
> On Wed, Mar 29, 2017 at 10:14 AM, Stack  wrote:
>>
>> On Tue, Mar 28, 2017 at 4:18 PM, Andrew Wang 
>> wrote:
>>>
>>> >
>>> > > If unknown fields are dropped, then applications proxying tokens and
>>> > other
>>> > >> data between servers will effectively corrupt those messages, unless
>>> > >> we
>>> > >> make everything opaque bytes, which- absent the convenient,
>>> > >> prenominate
>>> > >> semantics managing the conversion- obviate the compatibility
>>> > >> machinery
>>> > that
>>> > >> is the whole point of PB. Google is removing the features that
>>> > >> justified
>>> > >> choosing PB over its alternatives. Since we can't require that our
>>> > >> applications compile (or link) against our updated schema, this
>>> > >> creates
>>> > a
>>> > >> problem that PB was supposed to solve.
>>> > >
>>> > >
>>> > > This is scary, and it potentially affects services outside of the
>>> > > Hadoop
>>> > > codebase. This makes it difficult to assess the impact.
>>> >
>>> > Stack mentioned a compatibility mode that uses the proto2 semantics.
>>> > If that carries unknown fields through intermediate handlers, then
>>> > this objection goes away. -C
>>>
>>>
>>> Did some more googling, found this:
>>>
>>> https://groups.google.com/d/msg/protobuf/Z6pNo81FiEQ/fHkdcNtdAwAJ
>>>
>>> Feng Xiao appears to be a Google engineer, and suggests workarounds like
>>> packing the fields into a byte type. No mention of a PB2 compatibility
>>> mode. Also here:
>>>
>>> 

[jira] [Resolved] (HDFS-10640) Modify LOG statements to use sl4j templates.

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas resolved HDFS-10640.
--
Resolution: Later

Closing this, given attention to solving this across the project in HADOOP-12956

> Modify LOG statements to use  sl4j templates.
> -
>
> Key: HDFS-10640
> URL: https://issues.apache.org/jira/browse/HDFS-10640
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, fs
>Reporter: Virajith Jalaparti
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Can we update protobuf's version on trunk?

[Chris Douglas]

On Tue, Mar 28, 2017 at 4:18 PM, Andrew Wang  wrote:
> Unfortunately, it sounds like these are intrinsic differences with PB3.

That's too bad... but possibly not fatal: most of the data we proxy
through client code is, if not opaque, it's at least immutable
(particularly tokens). If PB3 does support reading valid PB fields as
bytes, then we could proxy the payload through application code as an
opaque blob. That opacity has a drawback: if clients could use that
information (e.g., StorageType), we'd need to include it in a
redundant field.

Ewan Higgs used a technique in HDFS-11026 [1] to handle a transition
from Writable to protobuf. This probably could be used for most of our
token types. It's not a general solution, but it would be sufficient
for existing applications to continue working, with some accommodation
for proxy versioning and rolling upgrades.

I haven't seen data identifying PB as a bottleneck, but the
non-x86/non-Linux and dev setup arguments may make this worthwhile. -C

[1] https://issues.apache.org/jira/browse/HDFS-11026

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Can we update protobuf's version on trunk?

[Chris Douglas]

On Tue, Mar 28, 2017 at 3:04 PM, Andrew Wang  wrote:
> There's no mention of the convenient "Embedded messages are compatible with
>> bytes if the bytes contain an encoded version of the message" semantics in
>> proto3.
>
>
> I checked the proto3 guide, and I think this is supported:
> https://developers.google.com/protocol-buffers/docs/proto3#updating

You're right, it looks like this is supported.

> If unknown fields are dropped, then applications proxying tokens and other
>> data between servers will effectively corrupt those messages, unless we
>> make everything opaque bytes, which- absent the convenient, prenominate
>> semantics managing the conversion- obviate the compatibility machinery that
>> is the whole point of PB. Google is removing the features that justified
>> choosing PB over its alternatives. Since we can't require that our
>> applications compile (or link) against our updated schema, this creates a
>> problem that PB was supposed to solve.
>
>
> This is scary, and it potentially affects services outside of the Hadoop
> codebase. This makes it difficult to assess the impact.

Stack mentioned a compatibility mode that uses the proto2 semantics.
If that carries unknown fields through intermediate handlers, then
this objection goes away. -C

> Paraphrasing, the issues with PB2.5 are:
>
>1. poor support for non-x86, non-Linux platforms
>2. not as available, so harder to setup a dev environment
>3. missing zero-copy support, which helped performance in HBase
>
> #1 and #2 can be addressed if we rehosted PB (with cross-OS compilation
> patches) elsewhere.
> #3 I don't think we benefit from, since we don't pass around big PB byte
> arrays (at least in HDFS).
>
> So the way I see it, upgrading to PB3 has risk from the behavior change wrt
> unknown fields, while there are other ways of addressing the stated issues
> with PB2.5.
>
> Best,
> Andrew

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha2 RC0

[Chris Douglas]

On Wed, Jan 25, 2017 at 11:42 AM, Andrew Wang  wrote:
> Chris and Karthik, could you clarify the contingency of your votes? Is
> fixing just the release notes sufficient?

My +1 was not contingent on any changes.

The release is fine as-is. Fixing any subset of the release notes,
minicluster jar, and the organization of LICENSE files within jars
should not reset the clock on the VOTE. -C

> On Wed, Jan 25, 2017 at 11:14 AM, Karthik Kambatla 
> wrote:
>
>> Thanks for driving the alphas, Andrew. I don't see the need to restart the
>> vote and I feel it is okay to fix the minor issues before releasing.
>>
>> +1 (binding). Downloaded source, stood up a pseudo-distributed cluster
>> with FairScheduler, ran example jobs, and played around with the UI.
>>
>> Thanks
>> Karthik
>>
>>
>> On Fri, Jan 20, 2017 at 2:36 PM, Andrew Wang 
>> wrote:
>>
>>> Hi all,
>>>
>>> With heartfelt thanks to many contributors, the RC0 for 3.0.0-alpha2 is
>>> ready.
>>>
>>> 3.0.0-alpha2 is the second alpha in the planned 3.0.0 release line leading
>>> up to a 3.0.0 GA. It comprises 857 fixes, improvements, and new features
>>> since alpha1 was released on September 3rd, 2016.
>>>
>>> More information about the 3.0.0 release plan can be found here:
>>>
>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release
>>>
>>> The artifacts can be found here:
>>>
>>> http://home.apache.org/~wang/3.0.0-alpha2-RC0/
>>>
>>> This vote will run 5 days, ending on 01/25/2017 at 2PM pacific.
>>>
>>> I ran basic validation with a local pseudo cluster and a Pi job. RAT
>>> output
>>> was clean.
>>>
>>> My +1 to start.
>>>
>>> Thanks,
>>> Andrew
>>>
>>
>>

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha2 RC0

[Chris Douglas]

On Mon, Jan 23, 2017 at 9:32 PM, Allen Wittenauer
 wrote:
> The problem here is that there is a 'license' directory and a file called 
> 'LICENSE'.  If this gets extracted by jar via jar xf, it will fail.  unzip 
> can be made to extract it via an option like -o.  To make matters worse, none 
> of these license files match the one in the generated tarball. :(

Ah, got it. I didn't strip the trailing slash on directories. With
that, it looks like the "license" directory and "LICENSE" file are the
only conflict?

I've not followed the development of packaging LICENSE/NOTICE in the
jar files. AFAIK, it's sufficient that we have the top-level
LICENSE/NOTICE in the tarball. Unless there's a LEGAL thread to the
contrary, it's OK as-is.

Again, I don't think we need to restart the clock on the RC vote if
the release notes and LICENSE/NOTICE were fixed, but it's Andrew's
time and I don't think any of these are blockers for the release. -C

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha2 RC0

[Chris Douglas]

Thanks for all your work on this, Andrew. It's great to see the 3.x
series moving forward.

If you were willing to modify the release notes and add the LICENSE to
the jar, we don't need to reset the clock on the VOTE, IMO.

What's the issue with the minicluster jar [1]? I tried to reproduce,
but had no issues with 1.8.0_92-b14.

+1 Verified checksums, signature, built src tarball. -C

[1] 
hadoop-3.0.0-alpha2/share/hadoop/client/hadoop-client-minicluster-3.0.0-alpha2.jar


On Mon, Jan 23, 2017 at 10:51 AM, Andrew Wang  wrote:
> There are 5 JIRAs by my count that are missing release notes, which isn't
> that bad but could of course be improved. Four of those I had already
> checked earlier and forgot to follow up, and they were very minorly
> incompatible (affecting private APIs) or mistakenly marked incompatible.
>
> I'm not too concerned about the shaded minicluster since it's a new
> feature, this is an alpha, and we have an IT test against the shaded
> minicluster. Multiple files with the same name are actually also allowed by
> the zip standard, so it's not clear if there is a functionality bug.
>
> Could I get some additional PMC input on this vote? The most critical issue
> in my mind is the missing LICENSE on that one new artifact. If we end up
> spinning a new RC, I'll also handle the missing release notes that Allen
> mentioned.
>
> Thanks,
> Andrew
>
> On Sun, Jan 22, 2017 at 10:45 PM, Allen Wittenauer > wrote:
>
>>
>> > On Jan 22, 2017, at 9:05 PM, Allen Wittenauer 
>> wrote:
>> >
>> >
>> >
>> >
>> >
>> >> On Jan 20, 2017, at 2:36 PM, Andrew Wang 
>> wrote:
>> >>
>> >> http://home.apache.org/~wang/3.0.0-alpha2-RC0/
>> >
>> >   There are quite a few JIRA issues that need release notes.
>> >
>>
>>
>> One other thing, before I forget... I'm not sure the
>> hadoop-client-minicluster jar is getting built properly.  If you look
>> inside, you'll find a real mishmash of things, including files and
>> directories with the same names but different cases.  This means it won't
>> extract properly on OS X.  (jar xf on that jar file literally stack traces
>> on my El Capitan machine. Neat!)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release cadence and EOL

[Chris Douglas]

On Fri, Jan 20, 2017 at 2:50 PM, Sangjin Lee  wrote:
> The security patch for the 2.6.x line is a case in point. Without any
> guideline, we would start with "What should we do for 2.6.x? Should we
> continue to patch it?" With this guideline, the baseline is already "it's
> been 2 years since 2.6.0 is released and we should consider stopping
> releasing from 2.6.x and encourage users to upgrade to 2.7.x."

Unless it falls under the "good reason" clause. To invent an example,
if 3.6.x were within the two year/minor release window, but 3.5.x was
more widely deployed/stable, then we'd use this escape hatch to patch
3.5.x and likely just violate our policy on 3.6.x (when the
implementation cost is high). How do we "force" a fix to 3.6.x?

We can't actually compel work from people. Even when we can point to a
prior consensus, someone needs to be motivated to actually complete
that task. That's the rub: this proposal doesn't only allow us to stop
working on old code, it also promises that we'll work on code we want
to abandon.

Pointing to a bylaw, and telling a contributor they "must" support a
branch/release isn't going to result in shorter discussions, either.
In the preceding hypothetical, if someone wants a fix in the 3.6 line,
they either need to convince others that it's important or they need
to do the work themselves.

> Actually the decision on security issues is a pretty strong indicator of our
> desire for EOL. If we say we do not want to patch that line for security
> vulnerability, then there would be even *less* rationale for fixing any
> other issue on that line. So the decision to stop backporting security
> patches is a sufficient indication of EOL in my mind.

Agreed. If we're not backporting security patches to a branch, then we
need to release a CVE, file a JIRA, and move on. If someone wants to
fix it and roll an RC for that release line, it lives. Otherwise it
dies as people move to later versions (unpatched security flaws are
motivating). A branch is EOL when we stop releasing from it. Two years
or two minor releases is a good heuristic based on recent history, but
overfitting policy to experience doesn't seem to buy us anything.

I'm all for spending less time discussing release criteria, but if
it's sufficient to observe which release lines are getting updates and
label them accordingly, that's cheaper to implement than a curated set
of constraints. -C

>> We can still embargo security flaws if someone asks (to give them time
>> time to implement a fix and call a vote). If there's nothing else in
>> the release, then we're effectively announcing it. In those cases, we
>> call a vote on private@ (cc: security@). -C
>>
>>
>> On Thu, Jan 19, 2017 at 1:30 PM, Andrew Wang 
>> wrote:
>> > I don't think the motivation here is vendor play or taking away power
>> > from
>> > committers. Having a regular release cadence helps our users understand
>> > when a feature will ship so they can plan their upgrades. Having an EOL
>> > policy and a minimum support period helps users choose a release line,
>> > and
>> > understand when they will need to upgrade.
>> >
>> > In the earlier thread, we discussed how these are not rules, but
>> > guidelines. There's a lot of flexibility if someone wants to keep
>> > maintaining a release line (particularly if they are willing to do the
>> > backporting work). More power to them; more releases are a good thing
>> > for
>> > the project.
>> >
>> > My main concern (which I raised on the earlier thread) is that without
>> > significant improvements to the release process and upstream integration
>> > testing, it's unlikely we'll actually ship more releases. Too often,
>> > branches are simply not in a releaseable state, or they have latent
>> > blocker
>> > bugs due to a lack of testing. This is what we've been struggling with
>> > on
>> > both the 2.8.x and 3.0.0-x release lines.
>> >
>> > So, in the abstract, I'm +1 on having a published policy on release
>> > cadence
>> > and EOL. This information helps users.
>> >
>> > However, I don't think we're ready to actually execute on this policy
>> > for
>> > the above reasons. This leaves me ambivalent overall, perhaps -0 since
>> > publishing a policy we don't follow is more confusing to users.
>> >
>> > My 2c,
>> > Andrew
>> >
>> >
>> >
>> > On Thu, Jan 19, 2017 at 12:28 PM, Arpit Agarwal
>> > 
>> > wrote:
>> >
>> >> The ASF release policy says releases may not be vetoed [1] so the EOL
>> >> policy sounds unenforceable. Not sure a release cadence is enforceable
>> >> either since Release Managers are volunteers.
>> >>
>> >> 1. https://www.apache.org/dev/release.html#approving-a-release
>> >>
>> >>
>> >>
>> >> On 1/18/17, 7:06 PM, "Junping Du"  wrote:
>> >>
>> >> +1 on Sangjin's proposal -
>> >> "A minor release line is end-of-lifed 2 years after it is released
>> >> or
>> >> there
>> >> are 2 newer minor 

Re: [VOTE] Release cadence and EOL

[Chris Douglas]

Sorry, I'd missed the end of the EOL discussion thread.

As several people have pointed out, this is unenforceable. The release
dates on the front page are a decent signal for liveness... do we need
something more formal? All these hypothetical situations would be
decided with more context. The "good reason" clause allows revive a
release line if two "live" branches straddle a dud, so this proposal
only commits us to maintain our mistakes. For two years? As Andrew
points out, while this heuristic usually holds, we're not set up to
enforce it.

We may want to have an informal policy for security issues, since
there are known holes in 2.6.x and earlier that aren't going to be
patched. We need to issue CVEs for those. A policy would simplify
tracking (e.g., announce vulnerabilities no more than a month after a
fix is available in a later release), so we don't wait indefinitely to
announce. Additionally, creating a JIRA and flagging the release on
the download page would be ample warning.

We can still embargo security flaws if someone asks (to give them time
time to implement a fix and call a vote). If there's nothing else in
the release, then we're effectively announcing it. In those cases, we
call a vote on private@ (cc: security@). -C


On Thu, Jan 19, 2017 at 1:30 PM, Andrew Wang  wrote:
> I don't think the motivation here is vendor play or taking away power from
> committers. Having a regular release cadence helps our users understand
> when a feature will ship so they can plan their upgrades. Having an EOL
> policy and a minimum support period helps users choose a release line, and
> understand when they will need to upgrade.
>
> In the earlier thread, we discussed how these are not rules, but
> guidelines. There's a lot of flexibility if someone wants to keep
> maintaining a release line (particularly if they are willing to do the
> backporting work). More power to them; more releases are a good thing for
> the project.
>
> My main concern (which I raised on the earlier thread) is that without
> significant improvements to the release process and upstream integration
> testing, it's unlikely we'll actually ship more releases. Too often,
> branches are simply not in a releaseable state, or they have latent blocker
> bugs due to a lack of testing. This is what we've been struggling with on
> both the 2.8.x and 3.0.0-x release lines.
>
> So, in the abstract, I'm +1 on having a published policy on release cadence
> and EOL. This information helps users.
>
> However, I don't think we're ready to actually execute on this policy for
> the above reasons. This leaves me ambivalent overall, perhaps -0 since
> publishing a policy we don't follow is more confusing to users.
>
> My 2c,
> Andrew
>
>
>
> On Thu, Jan 19, 2017 at 12:28 PM, Arpit Agarwal 
> wrote:
>
>> The ASF release policy says releases may not be vetoed [1] so the EOL
>> policy sounds unenforceable. Not sure a release cadence is enforceable
>> either since Release Managers are volunteers.
>>
>> 1. https://www.apache.org/dev/release.html#approving-a-release
>>
>>
>>
>> On 1/18/17, 7:06 PM, "Junping Du"  wrote:
>>
>> +1 on Sangjin's proposal -
>> "A minor release line is end-of-lifed 2 years after it is released or
>> there
>> are 2 newer minor releases, whichever is sooner. The community
>> reserves the
>> right to extend or shorten the life of a release line if there is a
>> good
>> reason to do so."
>>
>> I also noticed Karthik bring up some new proposals - some of them
>> looks interesting to me and I have some ideas as well. Karthik, can you
>> bring it out in a separated discussion threads so that we can discuss from
>> there?
>>
>> About Chris Trezzo's question about definition of EOL of hadoop
>> release, I think potentially changes could be:
>> 1. For users of Apache hadoop, they would expect to upgrade to a new
>> minor/major releases after EOL of their current release because there is no
>> guarantee of new maintenance release.
>>
>> 2. For release effort, apache law claim that committer can volunteer
>> RM for any release. With this release EOL proposal passes and written into
>> hadoop bylaw, anyone want to call for a release which is EOL then she/he
>> have to provide a good reason to community and get voted before to start
>> release effort. We don't want to waste community time/resource to
>> verify/vote a narrow interested release.
>>
>> 3. About committer's responsibility, I think the bottom line is
>> committer should commit patch contributor's target release and her/his own
>> interest release which I conservatively agree with Allen's point that this
>> vote doesn't change anything. But if a committer want to take care more
>> interest from the whole community like most committers are doing today,
>> he/she should understand which branches can benefit more people and could
>> skip some EOL release branches for backport 

Re: [VOTE] Merge HADOOP-13341

[Chris Douglas]

+1 (also on JIRA) -C

On Wed, Sep 7, 2016 at 6:44 AM, Allen Wittenauer
 wrote:
>
> I’d like to call for a vote to run for 5 days (ending  Mon 12, 2016 
> at 7AM PT) to merge the HADOOP-13341 feature branch into trunk. This branch 
> was developed exclusively by me.  As usual with large shell script changes, 
> it's been broken up into several smaller commits to make it easier to read.  
> The core of the functionality is almost entirely in hadoop-functions.sh with 
> the majority of the rest of the new additions either being documentation or 
> test code. In addition, large swaths of code is removed from the hadoop, 
> hdfs, mapred, and yarn executables.
>
> Here's a quick summary:
>
> * makes the rules around _OPTS consistent across all the projects
> * makes it possible to provide custom _OPTS for every hadoop, hdfs, mapred, 
> and yarn subcommand
> * with the exception of deprecations, removes all of the custom daemon _OPTS 
> handling sprinkled around the hadoop, hdfs, mapred, and yarn subcommands
> * removes the custom handling handling of HADOOP_CLIENT_OPTS and makes it 
> consistent for non-daemon subcommands
> * makes the _USER blocker consistent with _OPTS as well as providing better 
> documentation around this feature's existence.  Note that this is an 
> incompatible change against -alpha1.
> * by consolidating all of this code, makes it possible to finally fix a good 
> chunk of the "directory name containing spaces blows up the bash code" 
> problems that's been around since the beginning of the project
>
> Thanks!
>
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DISCUSS] Release numbering semantics with concurrent (>2) releases [Was Setting JIRA fix versions for 3.0.0 releases]

[Chris Douglas]

> I'm certainly open to alternate proposals for versioning and fix versions,
> but to reiterate, I like this versioning since it imitates other enterprise
> software. RHEL has versions like 6.2 Beta 2 and 7.0 Beta, so versions like
> 3.0.0-alpha1 will be immediately familiar to end users. Conversely, I've
> met people who were confused by the 2.0/2.1/2.2 progression. As far as I
> know, we were unique in using this style of versioning.

Yes, but even after a stable release, we haven't blocked new (or
incompatible) features in minor releases. Some minor releases- 0.16,
0.21, 2.6.0- included complicated features that took awhile to
stabilize. Most of our x.y.0 releases are not stable, because
downstream reports come back only after we cut a release. Signaling
stability is useful, but this identifier is not reliable. At least,
it's less reliable than a section on the website recommending the
latest stable release beside alpha/beta versions.

Anyway- if we do use this, then we can use Maven's schema:
..--

Though I hope we won't need it, starting with 3.0.0-alpha-01 will
avoid -alpha10 as preceding -alpha2. We've discussed it enough; I'll
let it go.

> I also think what we're doing now *is* considered releasing from trunk.
> Maybe I'm missing something, but we can't literally release from trunk
> without a release branch (e.g. branch-3.0.0-alpha1) unless we hold commits
> until the release or change fix versions afterwards.

We're not constrained on where we cut releases. If subsequent releases
will be off of trunk- not the -alpha branch- and we aren't
removing/holding any change until stabilizing at -beta, then we don't
need to maintain a separate release branch concurrent with development
on trunk. Bumping the release version, etc. might be useful, but a
living branch just mixes up the lineage and adds steps to commit
(trunk > 3.0.0-alpha > branch-2 > branch-2.8 > ...). If it's easier
for you to create the RC then OK. -C

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DISCUSS] Release numbering semantics with concurrent (>2) releases [Was Setting JIRA fix versions for 3.0.0 releases]

[Chris Douglas]

I agree with Konst. The virtues of branching (instead of releasing
from trunk) and using the version suffix for the 3.x releases are lost
on me. Both introduce opportunities for error, in commits, in
consistent JIRA tagging, in packaging...

We can mark stability on the website. If someone builds a cluster
without doing this basic research, marking stability in the version
number saves them from the least of the problems they'll have. This
adds complexity for clarity that's redundant, at best. -C


On Thu, Aug 4, 2016 at 11:20 AM, Andrew Wang  wrote:
> Hi Konst, thanks for commenting,
>
> On Wed, Aug 3, 2016 at 11:29 PM, Konstantin Shvachko 
> wrote:
>
>> 1. I probably missed something but I didn't get it how "alpha"s made their
>> way into release numbers again. This was discussed on several occasions and
>> I thought the common perception was to use just three level numbers for
>> release versioning and avoid branding them.
>> It is particularly confusing to have 3.0.0-alpha1 and 3.0.0-alpha2. What
>> is alphaX - fourth level? I think releasing 3.0.0 and setting trunk to
>> 3.1.0 would be perfectly in line with our current release practices.
>>
>
> We discussed release numbering a while ago when discussing the release plan
> for 3.0.0, and agreed on this scheme. "-alphaX" is essentially a fourth
> level as you say, but the intent is to only use it (and "-betaX") in the
> leadup to 3.0.0.
>
> The goal here is clarity for end users, since most other enterprise
> software uses a a.0.0 version to denote the GA of a new major version. Same
> for a.b.0 for a new minor version, though we haven't talked about that yet.
> The alphaX and betaX scheme also shares similarity to release versioning of
> other enterprise software.
>
>>
>> 2. I do not see any confusions with releasing 2.8.0 after 3.0.0.
>> The release number is not intended to reflect historical release sequence,
>> but rather the point in the source tree, which it was branched off. So one
>> can release 2.8, 2.9, etc. after or before 3.0.
>>
>
> As described earlier in this thread, the issue here is setting the fix
> versions such that the changelog is a useful diff from a previous version,
> and also clear about what changes are present in each branch. If we do not
> order a specific 2.x before 3.0, then we don't know what 2.x to diff from.
>
>>
>> 3. I agree that current 3.0.0 branch can be dropped and re-cut. We may
>> think of another rule that if a release branch is not released in 3 month
>> it should be abandoned. Which is applicable to branch 2.8.0 and it is too
>> much work syncing it with branch-2.
>>
>> Time-based rules are tough here. I'd prefer we continue to leave this up
> to release managers. If you think we should recut branch-2.8, recommend
> pinging Vinod and discussing on a new thread.
>
> Best,
> Andrew

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-10706) Add tool generating FSImage from external store

[Chris Douglas (JIRA)]

Chris Douglas created HDFS-10706:


 Summary: Add tool generating FSImage from external store
 Key: HDFS-10706
 URL: https://issues.apache.org/jira/browse/HDFS-10706
 Project: Hadoop HDFS
  Issue Type: Task
  Components: namenode, tools
Reporter: Chris Douglas


To experiment with provided storage, this provides a tool to map an external 
namespace to an FSImage/NN storage. By loading it in a NN, one can access the 
remote FS using HDFS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Why there are so many revert operations on trunk?

[Chris Douglas]

Reading through HDFS-9924, a request for a design doc- and a -1 on
committing to trunk- was raised in mid-May, but commits to trunk
continued. Why is that? Shouldn't this have paused while the details
were discussed? Branching is neutral to the pace of feature
development, but consensus on the result is required. Working through
possibilities in a branch- or in multiple branches- seems like a
reasonable way to determine which approach has support and code to
back it.

Reverting code is not "illegal"; the feature will be in/out of any
release by appealing to bylaws. Our rules exist to facilitate
consensus, not declare it a fiat accompli.

An RM only exists by creating an RC. Someone can declare themselves
Grand Marshall of trunk and stomp around in a fancy hat, but it
doesn't affect anything. -C


On Mon, Jun 6, 2016 at 9:36 AM, Junping Du  wrote:
> Thanks Aaron for pointing it out. I didn't see any consensus on HDFS-9924 so 
> I think we should bring it here with broader audiences for more discussions.
>
> I saw several very bad practices here:
>
> 1. committer (no need to say who) revert all commits from trunk without 
> making consensus with all related contributors/committers.
>
> 2. Someone's comments on feature branch are very misleading... If I didn't 
> remember wrong, feature development doesn't have to go through feature branch 
> which is just an optional process. This creative process of feature branch 
> and branch committer - I believe the intention is trying to accelerate 
> features development but not to slow them down.
>
> 3. Someone (again, no need to say who) seems to claim himself as RM for 
> trunk. I don't think we need any RM for trunk. Even for RM of 3.0.0-alpha, I 
> think we need someone else who demonstrates he/she is more responsible, work 
> hardly and carefully and open communication with all community. Only through 
> this, the success of Hadoop in age of 3.0 are guranteed.
>
>
> Thanks,
>
>
> Junping
>
>
> 
> From: Aaron T. Myers 
> Sent: Monday, June 06, 2016 4:46 PM
> To: Junping Du
> Cc: Andrew Wang; common-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org; 
> mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
> Subject: Re: Why there are so many revert operations on trunk?
>
> Junping,
>
> All of this is being discussed on HDFS-9924. Suggest you follow the 
> conversation there.
>
> --
> Aaron T. Myers
> Software Engineer, Cloudera
>
> On Mon, Jun 6, 2016 at 7:20 AM, Junping Du 
> > wrote:
> Hi Andrew,
>
>  I just noticed you revert 8 commits on trunk last Friday:
>
> HADOOP-13226
>
> HDFS-10430
>
> HDFS-10431
>
> HDFS-10390
>
> HADOOP-13168
>
> HDFS-10390
>
> HADOOP-13168
>
> HDFS-10346
>
> HADOOP-12957
>
> HDFS-10224
>
>And I didn't see you have any comments on JIRA or email discussion before 
> you did this. I don't think we are legally allowed to do this even as 
> committer/PMC member. Can you explain what's your intention to do this?
>
>BTW, thanks Nicolas to revert all these "illegal" revert operations.
>
>
>
> Thanks,
>
>
> Junping
>

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Looking to a Hadoop 3 release

[Chris Douglas]

If we're not starting branch-3/trunk, what would distinguish it from
trunk/trunk-incompat? Is it the same mechanism with different labels?

That may be a reasonable strategy when we create branch-3, as a
release branch for beta. Releasing 3.x from trunk will help us figure
out which incompatibilities can be called out in an upgrade guide
(e.g., "new feature X is incompatible with uncommon configuration Y")
and which require code changes (e.g., "data loss upgrading a cluster
with feature X"). Given how long trunk has been unreleased, we need
more data from deployments to triage. How to manage transitions
between major versions will always be case-by-case; consensus on how
we'll address generic incompatible changes is not saving any work.

Once created, removing functionality from branch-3 (leaving it in
trunk) _because_ nobody volunteers cycles to address urgent
compatibility issues is fair. It's also more workable than asking that
features be committed to a branch that we have no plan to release,
even as alpha. -C

On Fri, Apr 22, 2016 at 6:50 PM, Vinod Kumar Vavilapalli
 wrote:
> Tx for your replies, Andrew.
>
>>> For exit criteria, how about we time box it? My plan was to do monthly
>> alphas through the summer, leading up to beta in late August / early Sep.
>> At that point we freeze and stabilize for GA in Nov/Dec.
>
>
> Time-boxing is a reasonable exit-criterion.
>
>
>> In this case, does trunk-incompat essentially become the new trunk? Or are
>> we treating trunk-incompat as a feature branch, which periodically merges
>> changes from trunk?
>
>
> It’s the later. Essentially
>  - trunk-incompat = trunk + only incompatible changes, periodically kept 
> up-to-date to trunk
>  - trunk is always ready to ship
>  - and no compatible code gets left behind
>
> The reason for my proposal like this is to address the tension between “there 
> is lot of compatible code in trunk that we are not shipping” and “don’t ship 
> trunk, it has incompatibilities”. With this, we will not have (compatible) 
> code not getting shipped to users.
>
> Obviously, we can forget about all of my proposal completely if everyone puts 
> in all compatible code into branch-2 / branch-3 or whatever the main 
> releasable branch is. This didn’t work in practice, have seen this not 
> happening prominently during 0.21, and now 3.x.
>
> There is another related issue - "my feature is nearly ready, so I’ll just 
> merge it into trunk as we don’t release that anyways, but not the current 
> releasable branch - I’m lazy to fix the last few stability related issues”. 
> With this, we will (should) get more disciplined, take feature stability on a 
> branch seriously and merge a feature branch only when it is truly ready!
>
>> For 3.x, my strawman was to release off trunk for the alphas, then branch a
>> branch-3 for the beta and onwards.
>
>
> Repeating above, I’m proposing continuing to make GA 3.x releases also off of 
> trunk! This way only incompatible changes don’t get shipped to users - by 
> design! Eventually, trunk-incompat will be latest 3.x GA + enough 
> incompatible code to warrant a 4.x, 5.x etc.
>
> +Vinod

[jira] [Created] (HDFS-10262) Change HdfsFileStatus::fileId to an opaque identifier

[Chris Douglas (JIRA)]

Chris Douglas created HDFS-10262:


 Summary: Change HdfsFileStatus::fileId to an opaque identifier
 Key: HDFS-10262
 URL: https://issues.apache.org/jira/browse/HDFS-10262
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client, webhdfs
Reporter: Chris Douglas


HDFS exposes the INode ID as a long via HdfsFileStatus::getFileId. Since 
equality is the only valid client operation (sequential/monotonically 
increasing ids are not guaranteed in any spec; leases do not rely on any other 
property), this identifier can be opaque instead of assigning it a primitive 
type in HdfsFileStatus.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-9938) Support extensions to WebHdfsFileSystem

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas resolved HDFS-9938.
-
Resolution: Won't Fix

Yes, thanks for opening this [~vishwajeet.dusane]. Resolving as WONTFIX.

[~fabbri], you had also raised this in HADOOP-12666. Please reopen if you want 
to discuss another approach.

> Support extensions to WebHdfsFileSystem
> ---
>
> Key: HDFS-9938
> URL: https://issues.apache.org/jira/browse/HDFS-9938
> Project: Hadoop HDFS
>  Issue Type: Wish
>Reporter: Vishwajeet Dusane
> Attachments: HDFS-9938-001.patch
>
>
> This JIRA is to request opinion from the community, Can new file system use 
> an extension of WebHDFS file system. 
> Considering all the known limitation WebHDFS has over implementing a new 
> client by extending FileSystem class.
> Option we have is 
> 1. Use the namespace org.apache.hadoop.hdfs.web in new file system 
> implementation to access protected functionality from WebHdfsFileSystem.
> 2. Change the WebHdfs to support extensions
> 3. Suggestion on different approach like new client by sub classing 
> FileSystem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9806) Allow HDFS block replicas to be provided by an external storage system

[Chris Douglas (JIRA)]

Chris Douglas created HDFS-9806:
---

 Summary: Allow HDFS block replicas to be provided by an external 
storage system
 Key: HDFS-9806
 URL: https://issues.apache.org/jira/browse/HDFS-9806
 Project: Hadoop HDFS
  Issue Type: New Feature
Reporter: Chris Douglas


In addition to heterogeneous media, many applications work with heterogeneous 
storage systems. The guarantees and semantics provided by these systems are 
often similar, but not identical to those of 
[HDFS|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/filesystem/index.html].
 Any client accessing multiple storage systems is responsible for reasoning 
about each system independently, and must propagate/and renew credentials for 
each store.

Remote stores could be mounted under HDFS. Block locations could be mapped to 
immutable file regions, opaque IDs, or other tokens that represent a consistent 
view of the data. While correctness for arbitrary operations requires careful 
coordination between stores, in practice we can provide workable semantics with 
weaker guarantees.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9807) Add an optional StorageID to writes

[Chris Douglas (JIRA)]

Chris Douglas created HDFS-9807:
---

 Summary: Add an optional StorageID to writes
 Key: HDFS-9807
 URL: https://issues.apache.org/jira/browse/HDFS-9807
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Chris Douglas


The {{BlockPlacementPolicy}} considers specific storages, but when the replica 
is written the DN {{VolumeChoosingPolicy}} is unaware of any preference or 
constraints from other policies affecting placement. This limits heterogeneity 
to the declared storage types, which are treated as fungible within the target 
DN. It should be possible to influence or constrain the DN policy to select a 
particular storage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9808) Combine READ_ONLY_SHARED DatanodeStorages with the same ID

[Chris Douglas (JIRA)]

Chris Douglas created HDFS-9808:
---

 Summary: Combine READ_ONLY_SHARED DatanodeStorages with the same ID
 Key: HDFS-9808
 URL: https://issues.apache.org/jira/browse/HDFS-9808
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Chris Douglas


In HDFS-5318, each datanode that can reach a (read only) block reports itself 
as a valid location for the block. While accurate, this increases (redundant) 
block report traffic and- without partitioning on the backend- may return an 
overwhelming number of replica locations for each block.

Instead, a DN could report only that the shared storage is reachable. The 
contents of the storage could be reported separately/synthetically to the block 
manager, which can collapse all instances into a single storage. A subset of 
locations- closest to the client, etc.- can be returned, rather than all 
possible locations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Hadoop encryption module as Apache Chimera incubator project

[Chris Douglas]

On Thu, Feb 4, 2016 at 12:06 PM, Gangumalla, Uma
<uma.ganguma...@intel.com> wrote:

> [UMA] Ok. Great. You are right. I have cc¹ed to hadoop common. (You mean
> to cc Apache commons as well?)

I meant, if you start a discussion with Apache Commons, please CC
common-dev@hadoop to coordinate.

> [UMA] Right now we plan to have encryption libraries are the only one¹s we
> planned and as we see lot of interest from other projects like spark to
> use them. I see some challenges when we bring lot of code(other common
> codes) into this project is that, they all would have different
> requirements and may be different expected timelines for release etc. Some
> projects may just wanted to use encryption interfaces alone but not all.
> As they are completely independent codes, may be better to scope out
> clearly.

Yes, but even if the artifact is widely consumed, as a TLP it would
need to sustain a community. If the scope is too narrow, then it will
quickly fall into maintenance mode, its contributors will move on, and
it will retire to the attic. Alone, I doubt its viability as a TLP. So
as a first option, donating only this code to Apache Commons would
accomplish some immediate goals in a sustainable forum.

APR has a similar scope. As a second option, that may also be a
reasonable home, particularly if some of the native bits could
integrate with APR.

If the scope is broader, the effort could sustain prolonged
development. The current code is developing a strategy for packing
native libraries on multiple platforms, a capability that, say, the
native compression codecs (AFAIK) still lack. While java.nio is
improving, many projects would benefit from a better, native interface
to the filesystem (e.g., NativeIO). We could avoid duplicating effort
and collaborate on a common library.

As a third option, Hadoop already implements some useful native
libraries, which is why a subproject might be a sound course. That
would enable the subproject to coordinate with Hadoop on migrating its
native functionality to a separable, reusable component, then move to
a TLP when we can rely on it exclusively (if it has a well-defined,
independent community). It could control its release cadence and limit
its dependencies.

Finally, this is beside the point if nobody is interested in doing the
work on such a project. It's rude to pull code out of Hadoop and
donate it to another project so Spark can avoid a dependency, but this
instance seems reasonable to me. -C

[1] https://apr.apache.org/

> On 2/3/16, 6:46 PM, "Chen, Haifeng" <haifeng.c...@intel.com> wrote:
>
>>Thanks Chris.
>>
>>>> I went through the repository, and now understand the reasoning that
>>>>would locate this code in Apache Commons. This isn't proposing to
>>>>extract much of the implementation and it takes none of the
>>>>integration. It's limited to interfaces to crypto libraries and
>>>>streams/configuration.
>>Exactly.
>>
>>>> Chimera would be a boutique TLP, unless we wanted to draw out more of
>>>>the integration and tooling. Is that a goal you're interested in
>>>>pursuing? There's a tension between keeping this focused and including
>>>>enough functionality to make it viable as an independent component.
>>The Chimera goal was for providing useful, common and optimized
>>cryptographic functionalities. I would prefer that it is still focused in
>>this clear scope. Multiple domain requirements will put more challenges
>>and uncertainties in where and how it should go, thus more risk in
>>stalling.
>>
>>>> If the encryption libraries are the only ones you're interested in
>>>>pulling out, then Apache Commons does seem like a better target than a
>>>>separate project.
>>Yes. Just mentioned above, the library will be positioned in
>>cryptographic.
>>
>>
>>Thanks,
>>
>>-Original Message-
>>From: Chris Douglas [mailto:cdoug...@apache.org]
>>Sent: Thursday, February 4, 2016 7:26 AM
>>To: hdfs-dev@hadoop.apache.org
>>Subject: Re: Hadoop encryption module as Apache Chimera incubator project
>>
>>I went through the repository, and now understand the reasoning that
>>would locate this code in Apache Commons. This isn't proposing to extract
>>much of the implementation and it takes none of the integration. It's
>>limited to interfaces to crypto libraries and streams/configuration. It
>>might be a reasonable fit for commons-codec, but that's a pretty sparse
>>library and driving the release cadence might be more complicated. It'd
>>be worth discussing on their lists (please also CC common-dev@).
>>
>>Chimera would be a boutique TLP, unless we wanted to draw out more of the
>>

Re: Hadoop encryption module as Apache Chimera incubator project

[Chris Douglas]

On Wed, Feb 3, 2016 at 12:48 AM, Gangumalla, Uma
<uma.ganguma...@intel.com> wrote:
>>Standing in the point of shared fundamental piece of code like this, I do
>>think Apache Commons might be the best direction which we can try as the
>>first effort. In this direction, we still need to work with Apache Common
>>community for buying in and accepting the proposal.
> Make sense.

Makes sense how?

> For this we should define the independent release cycles for this project
> and it would just place under Hadoop tree if we all conclude with this
> option at the end.

Yes.

> [Chris]
>>If Chimera is not successful as an independent project or stalls,
>>Hadoop and/or Spark and/or $project will have to reabsorb it as
>>maintainers.
>>
> I am not so strong on this point. If we assume project would be
> unsuccessful, it can be unsuccessful(less maintained) even under hadoop.
> But if other projects depending on this piece then they would get less
> support. Of course right now we feel this piece of code is very important
> and we feel(expect) it can be successful as independent project,
> irrespective of whether it as separate project outside hadoop or inside.
> So, I feel this point would not really influence to judge the discussion.

Sure; code can idle anywhere, but that wasn't the point I was after.
You propose to extract code from Hadoop, but if Chimera fails then
what recourse do we have among the other projects taking a dependency
on it? Splitting off another project is feasible, but Chimera should
be sustainable before this PMC can divest itself of responsibility for
security libraries. That's a pretty low bar.

Bundling the library with the jar is helpful; I've used that before.
It should prefer (updated) libraries from the environment, if
configured. Otherwise it's a pain (or impossible) for ops to patch
security bugs. -C

>>-Original Message-
>>From: Colin P. McCabe [mailto:cmcc...@apache.org]
>>Sent: Wednesday, February 3, 2016 4:56 AM
>>To: hdfs-dev@hadoop.apache.org
>>Subject: Re: Hadoop encryption module as Apache Chimera incubator project
>>
>>It's great to see interest in improving this functionality.  I think
>>Chimera could be successful as an Apache project.  I don't have a strong
>>opinion one way or the other as to whether it belongs as part of Hadoop
>>or separate.
>>
>>I do think there will be some challenges splitting this functionality out
>>into a separate jar, because of the way our CLASSPATH works right now.
>>For example, let's say that Hadoop depends on Chimera 1.2 and Spark
>>depends on Chimera 1.1.  Now Spark jobs have two different versions
>>fighting it out on the classpath, similar to the situation with Guava and
>>other libraries.  Perhaps if Chimera adopts a policy of strong backwards
>>compatibility, we can just always use the latest jar, but it still seems
>>likely that there will be problems.  There are various classpath
>>isolation ideas that could help here, but they are big projects in their
>>own right and we don't have a clear timeline for them.  If this does end
>>up being a separate jar, we may need to shade it to avoid all these
>>issues.
>>
>>Bundling the JNI glue code in the jar itself is an interesting idea,
>>which we have talked about before for libhadoop.so.  It doesn't really
>>have anything to do with the question of TLP vs. non-TLP, of course.
>>We could do that refactoring in Hadoop itself.  The really complicated
>>part of bundling JNI code in a jar is that you need to create jars for
>>every cross product of (JVM version, openssl version, operating system).
>>For example, you have the RHEL6 build for openJDK7 using openssl 1.0.1e.
>>If you change any one thing-- say, change openJDK7 to Oracle JDK8, then
>>you might need to rebuild.  And certainly using Ubuntu would be a
>>rebuild.  And so forth.  This kind of clashes with Maven's philosophy of
>>pulling prebuilt jars from the internet.
>>
>>Kai Zheng's question about whether we would bundle openSSL's libraries is
>>a good one.  Given the high rate of new vulnerabilities discovered in
>>that library, it seems like bundling would require Hadoop users and
>>vendors to update very frequently, much more frequently than Hadoop is
>>traditionally updated.  So probably we would not choose to bundle openssl.
>>
>>best,
>>Colin
>>
>>On Tue, Feb 2, 2016 at 12:29 AM, Chris Douglas <cdoug...@apache.org>
>>wrote:
>>> As a subproject of Hadoop, Chimera could maintain its own cadence.
>>> There's also no reason why it should maintain dependencies on other
>>> parts of Hadoop, if those are separable. How is this sol

Re: Hadoop encryption module as Apache Chimera incubator project

[Chris Douglas]

I went through the repository, and now understand the reasoning that
would locate this code in Apache Commons. This isn't proposing to
extract much of the implementation and it takes none of the
integration. It's limited to interfaces to crypto libraries and
streams/configuration. It might be a reasonable fit for commons-codec,
but that's a pretty sparse library and driving the release cadence
might be more complicated. It'd be worth discussing on their lists
(please also CC common-dev@).

Chimera would be a boutique TLP, unless we wanted to draw out more of
the integration and tooling. Is that a goal you're interested in
pursuing? There's a tension between keeping this focused and including
enough functionality to make it viable as an independent component. By
way of example, Hadoop's common project requires too many dependencies
and carries too much historical baggage for other projects to rely on.
I agree with Colin/Steve: we don't want this to grow into another
guava-like dependency that creates more work in conflicts than it
saves in implementation...

Would it make sense to also package some of the compression libraries,
and maybe some of the text processing from MapReduce? Evolving some of
this code to a common library with few/no dependencies would be
generally useful. As a subproject, it could have a broader scope that
could evolve into a viable TLP. If the encryption libraries are the
only ones you're interested in pulling out, then Apache Commons does
seem like a better target than a separate project. -C


On Wed, Feb 3, 2016 at 1:49 AM, Chris Douglas <cdoug...@apache.org> wrote:
> On Wed, Feb 3, 2016 at 12:48 AM, Gangumalla, Uma
> <uma.ganguma...@intel.com> wrote:
>>>Standing in the point of shared fundamental piece of code like this, I do
>>>think Apache Commons might be the best direction which we can try as the
>>>first effort. In this direction, we still need to work with Apache Common
>>>community for buying in and accepting the proposal.
>> Make sense.
>
> Makes sense how?
>
>> For this we should define the independent release cycles for this project
>> and it would just place under Hadoop tree if we all conclude with this
>> option at the end.
>
> Yes.
>
>> [Chris]
>>>If Chimera is not successful as an independent project or stalls,
>>>Hadoop and/or Spark and/or $project will have to reabsorb it as
>>>maintainers.
>>>
>> I am not so strong on this point. If we assume project would be
>> unsuccessful, it can be unsuccessful(less maintained) even under hadoop.
>> But if other projects depending on this piece then they would get less
>> support. Of course right now we feel this piece of code is very important
>> and we feel(expect) it can be successful as independent project,
>> irrespective of whether it as separate project outside hadoop or inside.
>> So, I feel this point would not really influence to judge the discussion.
>
> Sure; code can idle anywhere, but that wasn't the point I was after.
> You propose to extract code from Hadoop, but if Chimera fails then
> what recourse do we have among the other projects taking a dependency
> on it? Splitting off another project is feasible, but Chimera should
> be sustainable before this PMC can divest itself of responsibility for
> security libraries. That's a pretty low bar.
>
> Bundling the library with the jar is helpful; I've used that before.
> It should prefer (updated) libraries from the environment, if
> configured. Otherwise it's a pain (or impossible) for ops to patch
> security bugs. -C
>
>>>-Original Message-
>>>From: Colin P. McCabe [mailto:cmcc...@apache.org]
>>>Sent: Wednesday, February 3, 2016 4:56 AM
>>>To: hdfs-dev@hadoop.apache.org
>>>Subject: Re: Hadoop encryption module as Apache Chimera incubator project
>>>
>>>It's great to see interest in improving this functionality.  I think
>>>Chimera could be successful as an Apache project.  I don't have a strong
>>>opinion one way or the other as to whether it belongs as part of Hadoop
>>>or separate.
>>>
>>>I do think there will be some challenges splitting this functionality out
>>>into a separate jar, because of the way our CLASSPATH works right now.
>>>For example, let's say that Hadoop depends on Chimera 1.2 and Spark
>>>depends on Chimera 1.1.  Now Spark jobs have two different versions
>>>fighting it out on the classpath, similar to the situation with Guava and
>>>other libraries.  Perhaps if Chimera adopts a policy of strong backwards
>>>compatibility, we can just always use the latest jar, but it still seems
>>>likely that there will be problems.  There are various classpath
&

Re: Hadoop encryption module as Apache Chimera incubator project

[Chris Douglas]

As a subproject of Hadoop, Chimera could maintain its own cadence.
There's also no reason why it should maintain dependencies on other
parts of Hadoop, if those are separable. How is this solution
inadequate?

If Chimera is not successful as an independent project or stalls,
Hadoop and/or Spark and/or $project will have to reabsorb it as
maintainers. Projects have high mortality in early life, and a fight
over inheritance/maintenance is something we'd like to avoid. If, on
the other hand, it develops enough of a community where it is
obviously viable, then we can (and should) break it out as a TLP (as
we have before). If other Apache projects take a dependency on
Chimera, we're open to adding them to security@hadoop.

Unlike Yetus, which was largely rewritten right before it was made
into a TLP, security in Hadoop has a complicated pedigree. If Chimera
eventually becomes a TLP, it seems fair to include those who work on
it while it is a subproject. Declared upfront, that criterion is
fairer than any post hoc justification, and will lead to a more
accurate account of its community than a subset of the Hadoop
PMC/committers that volunteer. -C


On Mon, Feb 1, 2016 at 9:29 PM, Chen, Haifeng  wrote:
> Thanks to all folks providing feedbacks and participating the discussions.
>
> @Owen, do you still have any concerns on going forward in the direction of 
> Apache Commons (or other options, TLP)?
>
> Thanks,
> Haifeng
>
> -Original Message-
> From: Chen, Haifeng [mailto:haifeng.c...@intel.com]
> Sent: Saturday, January 30, 2016 10:52 AM
> To: hdfs-dev@hadoop.apache.org
> Subject: RE: Hadoop encryption module as Apache Chimera incubator project
>
>>> I believe encryption is becoming a core part of Hadoop. I think that
>>> moving core components out of Hadoop is bad from a project management 
>>> perspective.
>
>> Although it's certainly true that encryption capabilities (in HDFS, YARN, 
>> etc.) are becoming core to Hadoop, I don't think that should really 
>> influence whether or not the non-Hadoop-specific encryption routines should 
>> be part of the Hadoop code base, or part of the code base of another project 
>> that Hadoop depends on. If Chimera had existed as a library hosted at ASF 
>> when HDFS encryption was first developed, HDFS probably would have just 
>> added that as a dependency and been done with it. I don't think we would've 
>> copy/pasted the code for Chimera into the Hadoop code base.
>
> Agree with ATM. I want to also make an additional clarification. I agree that 
> the encryption capabilities are becoming core to Hadoop. While this effort is 
> to put common and shared encryption routines such as crypto stream 
> implementations into a scope which can be widely shared across the Apache 
> ecosystem. This doesn't move Hadoop encryption out of Hadoop (that is not 
> possible).
>
> Agree if we make it a separate and independent releases project in Hadoop 
> takes a step further than the existing approach and solve some issues (such 
> as libhadoop.so problem). Frankly speaking, I think it is not the best option 
> we can try. I also expect that an independent release project within Hadoop 
> core will also complicate the existing release ideology of Hadoop release.
>
> Thanks,
> Haifeng
>
> -Original Message-
> From: Aaron T. Myers [mailto:a...@cloudera.com]
> Sent: Friday, January 29, 2016 9:51 AM
> To: hdfs-dev@hadoop.apache.org
> Subject: Re: Hadoop encryption module as Apache Chimera incubator project
>
> On Wed, Jan 27, 2016 at 11:31 AM, Owen O'Malley  wrote:
>
>> I believe encryption is becoming a core part of Hadoop. I think that
>> moving core components out of Hadoop is bad from a project management 
>> perspective.
>>
>
> Although it's certainly true that encryption capabilities (in HDFS, YARN,
> etc.) are becoming core to Hadoop, I don't think that should really influence 
> whether or not the non-Hadoop-specific encryption routines should be part of 
> the Hadoop code base, or part of the code base of another project that Hadoop 
> depends on. If Chimera had existed as a library hosted at ASF when HDFS 
> encryption was first developed, HDFS probably would have just added that as a 
> dependency and been done with it. I don't think we would've copy/pasted the 
> code for Chimera into the Hadoop code base.
>
>
>> To put it another way, a bug in the encryption routines will likely
>> become a security problem that security@hadoop needs to hear about.
>>
> I don't think
>> adding a separate project in the middle of that communication chain is
>> a good idea. The same applies to data corruption problems, and so on...
>>
>
> Isn't the same true of all the libraries that Hadoop currently depends upon? 
> If the commons-httpclient library (or commons-codec, or commons-io, or guava, 
> or...) has a security vulnerability, we need to know about it so that we can 
> update our dependency to a fixed version. This case doesn't seem 

Re: [DISCUSS] Looking to a 2.8.0 release

[Chris Douglas]

With two active sustaining branches (2.6, 2.7), what would you think
of releasing trunk as 3.x instead of pushing 2.8? There are many new
features (EC, Y1197, etc.), and trunk could be the source of several
alpha/beta releases before we fork the 3.x line. -C

On Sat, Sep 26, 2015 at 12:49 PM, Vinod Vavilapalli
 wrote:
> As you may have noted, 2.8.0 got completely derailed what with 2.7.x and the 
> unusually long 2.6.1 release.
>
> With 2.6.1 out of the way, and two parallel threads in progress for 2.6.2 and 
> 2.7.2, it’s time for us to look back at where we are with Hadoop 2.8.
>
> I’ll do a quick survey of where the individual features are and the amount of 
> content already present in 2.8 and kick-start 2.8.0 process again.
>
> +Vinod
>
>
>> On Apr 21, 2015, at 2:39 PM, vino...@apache.org wrote:
>>
>> With 2.7.0 out of the way, and with more maintenance releases to stabilize 
>> it, I propose we start thinking about 2.8.0.
>>
>> Here's my first cut of the proposal, will update the Roadmap wiki.
>>  - Support *both* JDK7 and JDK8 runtimes: HADOOP-11090
>>  - Compatibility tools to catch backwards, forwards compatibility issues at 
>> patch submission, release times. Some of it is captured at YARN-3292. This 
>> also involves resurrecting jdiff (HADOOP-11776/YARN-3426/MAPREDUCE-6310) 
>> and/or investing in new tools.
>>  - HADOOP-11656 Classpath isolation for downstream clients
>>  - Support for Erasure Codes in HDFS HDFS-7285
>>  - Early work for disk and network isolation in YARN: YARN-2139, YARN-2140
>>  - YARN Timeline Service Next generation: YARN-2928. At least branch-merge + 
>> early peek.
>>  - Supporting non-exclusive node-labels: YARN-3214
>>
>> I'm experimenting with more agile 2.7.x releases and would like to continue 
>> the same by volunteering as the RM for 2.8.x too.
>>
>> Given the long time we took with 2.7.0, the timeline I am looking at is 8-12 
>> weeks. We can pick as many features as they finish along and make a more 
>> predictable releases instead of holding up releases for ever.
>>
>> Thoughts?
>>
>> Thanks
>> +Vinod
>

[jira] [Created] (HDFS-9020) Support hflush/hsync in WebHDFS

[Chris Douglas (JIRA)]

Chris Douglas created HDFS-9020:
---

 Summary: Support hflush/hsync in WebHDFS
 Key: HDFS-9020
 URL: https://issues.apache.org/jira/browse/HDFS-9020
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: webhdfs
Reporter: Chris Douglas


In the current implementation, hflush/hsync have no effect on WebHDFS streams, 
particularly w.r.t. visibility to other clients. This proposes to extend the 
protocol and implementation to enable this functionality.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Looking to a Hadoop 3 release

[Chris Douglas]

On Fri, Mar 6, 2015 at 4:32 PM, Vinod Kumar Vavilapalli
vino...@hortonworks.com wrote:
 I'd encourage everyone to post their wish list on the Roadmap wiki that 
 *warrants* making incompatible changes forcing us to go 3.x.

This is a useful exercise, but not a prerequisite to releasing 3.0.0
as an alpha off of trunk, right? Andrew summarized the operating
assumptions for anyone working on it: rolling upgrades still work,
wire compat is preserved, breaking changes may get rolled back when
branch-3 is in beta (so be very conservative, notify others loudly).
This applies to branches merged to trunk, also.

 +1 to Jason's comments on general. We can keep rolling alphas that downstream 
 can pick up, but I'd also like us to clarify the exit criterion for a GA 
 release of 3.0 and its relation to the life of 2.x if we are going this 
 route. This brings us back to the roadmap discussion, and a collective 
 agreement about a logical step at a future point in time where we say we have 
 enough incompatible features in 3.x that we can stop putting more of them and 
 start stabilizing it.

We'll have this discussion again. We don't need to reach consensus on
the roadmap, just that each artifact reflects the output of the
project.

 Irrespective of that, here is my proposal in the interim:
  - Run JDK7 + JDK8 first in a compatible manner like I mentioned before for 
 atleast two releases in branch-2: say 2.8 and 2.9 before we consider taking 
 up the gauntlet on 3.0.
  - Continue working on the classpath isolation effort and try making it as 
 compatible as is possible for users to opt in and migrate easily.

+1 for 2.x, but again I don't understand the sequencing. -C

 On Mar 5, 2015, at 1:44 PM, Jason Lowe jl...@yahoo-inc.com.INVALID wrote:

 I'm OK with a 3.0.0 release as long as we are minimizing the pain of 
 maintaining yet another release line and conscious of the incompatibilities 
 going into that release line.
 For the former, I would really rather not see a branch-3 cut so soon.  It's 
 yet another line onto which to cherry-pick, and I don't see why we need to 
 add this overhead at such an early phase.  We should only create branch-3 
 when there's an incompatible change that the community wants and it should 
 _not_ go into the next major release (i.e.: it's for Hadoop 4.0).  We can 
 develop 3.0 alphas and betas on trunk and release from trunk in the interim. 
  IMHO we need to stop treating trunk as a place to exile patches.

 For the latter, I think as a community we need to evaluate the benefits of 
 breaking compatibility against the costs of migrating.  Each time we break 
 compatibility we create a hurdle for people to jump when they move to the 
 new release, and we should make those hurdles worth their time.  For 
 example, wire-compatibility has been mentioned as part of this.  Any feature 
 that breaks wire compatibility better be absolutely amazing, as it creates a 
 huge hurdle for people to jump.
 To summarize:+1 for a community-discussed roadmap of what we're breaking in 
 Hadoop 3 and why it's worth it for users
 -1 for creating branch-3 now, we can release from trunk until the next 
 incompatibility for Hadoop 4 arrives
 +1 for baking classpath isolation as opt-in on 2.x and eventually default on 
 in 3.0
 Jason
  From: Andrew Wang andrew.w...@cloudera.com
 To: hdfs-dev@hadoop.apache.org hdfs-dev@hadoop.apache.org
 Cc: common-...@hadoop.apache.org common-...@hadoop.apache.org; 
 mapreduce-...@hadoop.apache.org mapreduce-...@hadoop.apache.org; 
 yarn-...@hadoop.apache.org yarn-...@hadoop.apache.org
 Sent: Wednesday, March 4, 2015 12:15 PM
 Subject: Re: Looking to a Hadoop 3 release

 Let's not dismiss this quite so handily.

 Sean, Jason, and Stack replied on HADOOP-11656 pointing out that while we
 could make classpath isolation opt-in via configuration, what we really
 want longer term is to have it on by default (or just always on). Stack in
 particular points out the practical difficulties in using an opt-in method
 in 2.x from a downstream project perspective. It's not pretty.

 The plan that both Sean and Jason propose (which I support) is to have an
 opt-in solution in 2.x, bake it there, then turn it on by default
 (incompatible) in a new major release. I think this lines up well with my
 proposal of some alphas and betas leading up to a GA 3.x. I'm also willing
 to help with 2.x release management if that would help with testing this
 feature.

 Even setting aside classpath isolation, a new major release is still
 justified by JDK8. Somehow this is being ignored in the discussion. Allen,
 historically the voice of the user in our community, just highlighted it as
 a major compatibility issue, and myself and Tucu have also expressed our
 very strong concerns about bumping this in a minor release. 2.7's bump is a
 unique exception, but this is not something to be cited as precedent or
 policy.

 Where does this resistance to a new major release stem from? As I've
 described 

Re: [VOTE] Migration from subversion to git for version control

[Chris Douglas]

+1 -C

On Fri, Aug 8, 2014 at 7:57 PM, Karthik Kambatla ka...@cloudera.com wrote:
 I have put together this proposal based on recent discussion on this topic.

 Please vote on the proposal. The vote runs for 7 days.

1. Migrate from subversion to git for version control.
2. Force-push to be disabled on trunk and branch-* branches. Applying
changes from any of trunk/branch-* to any of branch-* should be through
git cherry-pick -x.
3. Force-push on feature-branches is allowed. Before pulling in a
feature, the feature-branch should be rebased on latest trunk and the
changes applied to trunk through git rebase --onto or git cherry-pick
commit-range.
4. Every time a feature branch is rebased on trunk, a tag that
identifies the state before the rebase needs to be created (e.g.
tag_feature_JIRA-2454_2014-08-07_rebase). These tags can be deleted once
the feature is pulled into trunk and the tags are no longer useful.
5. The relevance/use of tags stay the same after the migration.

 Thanks
 Karthik

 PS: Per Andrew Wang, this should be a Adoption of New Codebase kind of
 vote and will be Lazy 2/3 majority of PMC members.

Re: [VOTE] Change by-laws on release votes: 5 days instead of 7

[Chris Douglas]

+1 -C

On Tue, Jun 24, 2014 at 1:53 AM, Arun C Murthy a...@hortonworks.com wrote:
 Folks,

  As discussed, I'd like to call a vote on changing our by-laws to change 
 release votes from 7 days to 5.

  I've attached the change to by-laws I'm proposing.

  Please vote, the vote will the usual period of 7 days.

 thanks,
 Arun

 

 [main]$ svn diff
 Index: author/src/documentation/content/xdocs/bylaws.xml
 ===
 --- author/src/documentation/content/xdocs/bylaws.xml   (revision 1605015)
 +++ author/src/documentation/content/xdocs/bylaws.xml   (working copy)
 @@ -344,7 +344,16 @@
  pVotes are open for a period of 7 days to allow all active
  voters time to consider the vote. Votes relating to code
  changes are not subject to a strict timetable but should be
 -made as timely as possible./p/li
 +made as timely as possible./p
 +
 + ul
 + li strongProduct Release - Vote Timeframe/strong
 +   pRelease votes, alone, run for a period of 5 days. All other
 + votes are subject to the above timeframe of 7 days./p
 + /li
 +   /ul
 +   /li
 +
 /ul
 /section
  /body
 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.

Re: [VOTE] Release Apache Hadoop 2.3.0

[Chris Douglas]

+1 (binding)

Verified checksum, signature. Built from src, poked at single-node
cluster, ran some unit tests. -C

On Tue, Feb 11, 2014 at 6:49 AM, Arun C Murthy a...@hortonworks.com wrote:
 Folks,

 I've created a release candidate (rc0) for hadoop-2.3.0 that I would like to 
 get released.

 The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.3.0-rc0
 The RC tag in svn is here: 
 https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.3.0-rc0

 The maven artifacts are available via repository.apache.org.

 Please try the release and vote; the vote will run for the usual 7 days.

 thanks,
 Arun

 PS: Thanks to Andrew, Vinod  Alejandro for all their help in various release 
 activities.
 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.

Re: Is it me or is the new bootstrap HDFS UI HWX green? (See tip of hadoop-2.3 branch)

[Chris Douglas]

Stack-

It's impossible to distinguish uncertainty from concern-trolling on
public lists. I'm concerned that the right honorable gentleman's rash
may be syphilis. What says the senate? is not innocent, even if it's
sincere. Speak discreetly to your colleagues, who happen to share your
aversion to corporate billboards in Apache projects.

The solutions to both your concerns are obvious. Build on ASF
infrastructure (and volunteer to audit other ecosystem projects, so
others don't feel vindictive). Make the color configurable if it
bothers you. This was completely avoidable.

Correspondingly, shifting the debate from the facts at issue to the
tone in which they're raised is a transparent dodge. The codebase is
more inviting when it's scrubbed of corporate leavings. Maybe the
particular rule is novel, but we agree on this, right? If the
objection is correct, determining whether it was raised _too_strongly_
is not profitably debated here.

Please take this offline. -C

On Sun, Feb 2, 2014 at 10:15 PM, Stack st...@duboce.net wrote:
 Sorry for the delay.


 On Wed, Jan 29, 2014 at 10:05 PM, Vinod Kumar Vavilapalli vinodkv@
 hortonworks.com wrote:


 My response was to your direct association of the green color to HWX green
 as if it were deliberately done. Nobody here intentionally left a vendor's
 signature like you claimed.



 I did not do what you accuse me of above.  Please gp back to the original.
  All is couched in 'is it me' and 'I think'.  No accusations of deliberate
 vendor insert.  That is your addition.



 And to your other comment Does the apache binary have to be compiled by
 'hortonmu'?   Could it be compiled by 'arun', or 'apachemu'? to the
 message in the build. As if somebody said it has to be.



 The Apache Hadoop version string should be pure, free of vendor pollution.
  Seems obvious to me.  I could call a vote and get it written into the
 bylaws but seems a bit of a useless exercise?

 (This is now a non-issue anyways having been 'fixed'.  While some chose to
 do histrionics's, another committer spent a few minutes and committed a
 patch so builds no longer have to be done on dev machines and can instead
 come off Apache Infra and now version string has apache infra in it
 instead... nice).



 You know how I'd have raised this? I'd say Hey guys, seems like the build
 messages have hortonmu and that seems like an issue with our branding. Can
 we fix this?. Then I or somebody could have replied Oh, that seems
 totally by mistake. Agreed, let's fix this.


 Ain't this what I did give or take a bit on the wording?



 Instead, you post it in another orthogonal thread (which in itself is
 making claims of causing deliberate confusion of brand), make it look like
 an innocuous question asking if apache binary has to be compiled by the
 specific user-name.


 Sorry. Seemed related to me at the time at least.  I was trying out tip of
 the branch and the color made me 'sensitive' and then I tripped over the
 version string (Its hard to miss being up top in our UI).


 I said 'unbelievable'. Sorry, I should have used 'disappointing'. This is
 not the way I'd post 'concerns'.


 You should make up your mind.  When you waffle on your dramatic lead-in,
 the 'unbelievable' becoming 'disappointing', it reads like a 'device'.
  Your reaction comes across as false, artificial, not genuine.  Just
 saying...



 There is a reason why brand issues are gently discussed on private lists.
 And to think this thread is posted out in the open like this, it was me who
 was taken aback by your oh-not-so-explicit insinuations.


 I do not apologize for thinking us as a community mature enough to answer a
 basic it looks like X to me, what do you lot think? even if X might come
 close to the bone for some of us involved here.  A simple no, you are way
 off or you may have a point... and variants thereof was what I was
 expecting (You did this up in the related issue, thanks for doing that, but
 IMO it would have been more effective if you'd done it in this thread...).

 Thanks Vinod,
 St.Ack

Re: [VOTE] Release Apache Hadoop 2.0.5-alpha (rc2)

[Chris Douglas]

+1

Checksum and signature match, ran some unit tests, checked diff
against 2.0.4-alpha.

Thanks for seeing this through, Cos. -C

On Mon, Jun 3, 2013 at 1:13 PM, Alejandro Abdelnur t...@cloudera.com wrote:
 +1 RC2. Verified MD5  signature, checked CHANGES.txt files, built,
 configured pseudo cluster, run a couple of sample jobs, tested HTTPFS.


 On Mon, Jun 3, 2013 at 12:51 PM, Konstantin Boudnik c...@apache.org wrote:

 I have rolled out release candidate (rc2) for hadoop-2.0.5-alpha.

 The difference between rc1 and rc2 is the optimistic release date is set
 for
 06/06/2013 in the CHANGES.txt files.

 The binary artifact is the same - there's no need to rebuild it. The maven
 artifacts are the same.

 The difference between the two RCs:

 svn diff \

 https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.5-alpha-rc1/\

 https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.5-alpha-rc2/

 New RC builds are uploaded to the web.
 The RC is available at:
 http://people.apache.org/~cos/hadoop-2.0.5-alpha-rc2/
 The RC tag in svn is here:
 http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.5-alpha-rc2

 I would like to extend the vote for another three days for it is such a
 minor
 change that doesn't affect anything but the recorded release date. Please
 cast your vote before 06/06/2013 5pm PDT.

 Thanks for your patience!
   Cos

 On Fri, May 31, 2013 at 09:27PM, Konstantin Boudnik wrote:
  All,
 
  I have created a release candidate (rc1) for hadoop-2.0.5-alpha that I
 would
  like to release.
 
  This is a stabilization release that includes fixed for a couple a of
 issues
  discovered in the testing with BigTop 0.6.0 release candidate.
 
  The RC is available at:
 http://people.apache.org/~cos/hadoop-2.0.5-alpha-rc1/
  The RC tag in svn is here:
 http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.5-alpha-rc1
 
  The maven artifacts will be available via repository.apache.org on Sat,
 June
  1st, 2013 at 2 pm PDT as outlined here
  http://s.apache.org/WKD
 
  Please try the release bits and vote; the vote will run for the 3 days,
  because this is just a version name change. The bits are identical to
 the ones
  voted on before in
  http://s.apache.org/2041move
 
  Thanks for your voting
Cos
 




 --
 Alejandro

Re: [VOTE] Release Apache Hadoop 0.23.8

[Chris Douglas]

+1

Checksum and signature match, ran some tests. -C

On Mon, Jun 3, 2013 at 1:19 PM, Sandy Ryza sandy.r...@cloudera.com wrote:
 +1 (non-binding).  Did a full build from source and ran a few sample jobs
 on a pseudo-distributed cluster.

 -Sandy


 On Mon, Jun 3, 2013 at 11:29 AM, Kihwal Lee kih...@yahoo-inc.com wrote:

 +1 I've downloaded  built the RC and ran several tests on a single node
 cluster.

 Kihwal

 On 5/28/13 11:00 AM, Thomas Graves tgra...@yahoo-inc.com wrote:

 
 I've created a release candidate (RC0) for hadoop-0.23.8 that I would like
 to release.
 
 This release is a sustaining release with several important bug fixes in
 it.  The most critical one is MAPREDUCE-5211.
 
 The RC is available at:
 http://people.apache.org/~tgraves/hadoop-0.23.8-candidate-0/
 The RC tag in svn is here:
 http://svn.apache.org/viewvc/hadoop/common/tags/release-0.23.8-rc0/
 
 The maven artifacts are available via repository.apache.org.
 
 Please try the release and vote; the vote will run for the usual 7 days.
 
 I am +1 (binding).
 
 thanks,
 Tom Graves
 

Re: [VOTE] Release Apache Hadoop 2.0.4.1-alpha

[Chris Douglas]

On Thu, May 30, 2013 at 10:57 AM, Arun C Murthy a...@hortonworks.com wrote:
 Why not include MAPREDUCE-4211 as well rather than create one release per 
 patch?

From Cos's description, it sounded like these were backports of fixes
to help Sqoop2 and fix some build issues. If it's not just to fixup
leftover bugs in 2.0.4 *once* so downstream projects can integrate
against 2.0.4.1, and this a release series, then I've completely
misunderstood the purpose.

Cos, are you planning 2.0.4.2?

 Also, this is the first time we are seeing a four-numbered scheme in Hadoop. 
 Why not call this 2.0.5-alpha?

Good point. Since it contains only backports from branch-2, it would
make sense for it to be an intermediate release.

I shouldn't have to say this, but I'm changing my vote to -1 while we
work this out. -C

 On May 24, 2013, at 8:48 PM, Konstantin Boudnik wrote:

 All,

 I have created a release candidate (rc0) for hadoop-2.0.4.1-alpha that I 
 would
 like to release.

 This is a stabilization release that includes fixed for a couple a of issues
 discovered in the testing with BigTop 0.6.0 release candidate.

 The RC is available at: 
 http://people.apache.org/~cos/hadoop-2.0.4.1-alpha-rc0/
 The RC tag in svn is here: 
 http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4.1-alpha-rc0

 The maven artifacts are available via repository.apache.org.

 Please try the release bits and vote; the vote will run for the usual 7 days.

 Thanks for your voting
  Cos

Re: [VOTE] Release Apache Hadoop 2.0.4.1-alpha

[Chris Douglas]

On Thu, May 30, 2013 at 2:39 PM, Konstantin Boudnik c...@apache.org wrote:
 There's no misunderstanding Chris - this release is to unblock downstream.

 As for your question: I don't have a crystal ball; I wish though. I think the
 answer depends on will be there more blocking bugs found in the later releases
 of Bigtop or other downstream components.
 This is bugfix release and, I guess, if there are more bugs found in the
 future - more releases would have to be cut. Isn't this is why the software is
 being released?

Sure, but they're all backports from the release currently marked for
2.0.5. Either (a) these are really blocker bugs and we should roll a
patch release or (b) some bleeding-edge work needs to work around this
while branch-2 is released in the next few weeks. If it's not severe
enough to justify disrupting the versioning of snapshot maven
artifacts in branch-2, then we're clearly not in case (a).

I thought this was the result of extensive testing, and 2.0.4.1 was a
release to enable additional integration before 2.0.5. If we plan to
roll more releases as a subset of the bug fixes committed to branch-2
then just call it 2.0.5. Please make sure it- and any future,
intermediate release- is worth the disruption.

 Now, the -1: I am not clear about the justification. What exactly we expect to
 work out?

It's become fashionable to close threads and count votes in the middle
of the discussion. I changed my vote instead of trusting you. -C

 On Thu, May 30, 2013 at 01:48PM, Chris Douglas wrote:
 On Thu, May 30, 2013 at 10:57 AM, Arun C Murthy a...@hortonworks.com wrote:
  Why not include MAPREDUCE-4211 as well rather than create one release per 
  patch?

 From Cos's description, it sounded like these were backports of fixes
 to help Sqoop2 and fix some build issues. If it's not just to fixup
 leftover bugs in 2.0.4 *once* so downstream projects can integrate
 against 2.0.4.1, and this a release series, then I've completely
 misunderstood the purpose.

 Cos, are you planning 2.0.4.2?

  Also, this is the first time we are seeing a four-numbered scheme in 
  Hadoop. Why not call this 2.0.5-alpha?

 Good point. Since it contains only backports from branch-2, it would
 make sense for it to be an intermediate release.

 I shouldn't have to say this, but I'm changing my vote to -1 while we
 work this out. -C

  On May 24, 2013, at 8:48 PM, Konstantin Boudnik wrote:
 
  All,
 
  I have created a release candidate (rc0) for hadoop-2.0.4.1-alpha that I 
  would
  like to release.
 
  This is a stabilization release that includes fixed for a couple a of 
  issues
  discovered in the testing with BigTop 0.6.0 release candidate.
 
  The RC is available at: 
  http://people.apache.org/~cos/hadoop-2.0.4.1-alpha-rc0/
  The RC tag in svn is here: 
  http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4.1-alpha-rc0
 
  The maven artifacts are available via repository.apache.org.
 
  Please try the release bits and vote; the vote will run for the usual 7 
  days.
 
  Thanks for your voting
   Cos
 
 
 

Re: [VOTE] Release Apache Hadoop 2.0.4.1-alpha

[Chris Douglas]

On Thu, May 30, 2013 at 3:25 PM, Konstantin Boudnik c...@apache.org wrote:
 There's no plans to release anything else at this point - this is a bug-fix
 release, as I pointed out on a numerous occasions. There's no new features -
 just 2 fixes.

If you're worried about extending the voting by a week, I don't think
anyone can reasonably object to a truncated schedule if the only
change is the version number. Downstream fixes discovered in Bigtop
are a sufficient reason for a patch release and I think we'd all like
them to become routine. Not just in 2.0.x, but in later release
branches.

 2.0.5 matter became and still is too controversial at some point. The vote
 started by Arun to override the results of the Konstantin's vote never been
 closed.

For the nth time, neither release plan vote was binding. We recently
amended the bylaws to eliminate this confusion. There is no ambiguity
between votes when neither is in scope.

 The downstream projects are handing in the middle of the air because
 of that confusion.

Can we please ground our discussion when discussing compatibility and
bugs? Breathless alarm is not proportional to the severity, here.

 Have I missed something or you just called me a cheater and a lair right to 
 my face?

I called you neither. The prenominate votes were closed, counted, and
declared to be binding over objections. I'm annoyed that I have to
toggle my vote to prevent that tactic, but based on recent experience
I don't expect you to forgo it. I'd be happy to learn my caution is
unnecessary. -C

  On Thu, May 30, 2013 at 01:48PM, Chris Douglas wrote:
  On Thu, May 30, 2013 at 10:57 AM, Arun C Murthy a...@hortonworks.com 
  wrote:
   Why not include MAPREDUCE-4211 as well rather than create one release 
   per patch?
 
  From Cos's description, it sounded like these were backports of fixes
  to help Sqoop2 and fix some build issues. If it's not just to fixup
  leftover bugs in 2.0.4 *once* so downstream projects can integrate
  against 2.0.4.1, and this a release series, then I've completely
  misunderstood the purpose.
 
  Cos, are you planning 2.0.4.2?
 
   Also, this is the first time we are seeing a four-numbered scheme in 
   Hadoop. Why not call this 2.0.5-alpha?
 
  Good point. Since it contains only backports from branch-2, it would
  make sense for it to be an intermediate release.
 
  I shouldn't have to say this, but I'm changing my vote to -1 while we
  work this out. -C
 
   On May 24, 2013, at 8:48 PM, Konstantin Boudnik wrote:
  
   All,
  
   I have created a release candidate (rc0) for hadoop-2.0.4.1-alpha that 
   I would
   like to release.
  
   This is a stabilization release that includes fixed for a couple a of 
   issues
   discovered in the testing with BigTop 0.6.0 release candidate.
  
   The RC is available at: 
   http://people.apache.org/~cos/hadoop-2.0.4.1-alpha-rc0/
   The RC tag in svn is here: 
   http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4.1-alpha-rc0
  
   The maven artifacts are available via repository.apache.org.
  
   Please try the release bits and vote; the vote will run for the usual 
   7 days.
  
   Thanks for your voting
Cos
  
  
  

Re: [VOTE] Release Apache Hadoop 2.0.4.1-alpha

[Chris Douglas]

On Thu, May 30, 2013 at 5:51 PM, Konstantin Boudnik c...@apache.org wrote:
 I have no issues of changing the version to 2.0.5-alpha and restarting to vote
 for the release content, e.g. 2 bug fixes. Shall I call 3 days re-vote because
 of the number change?

+1 Sounds great.

 Does the result of bylaw vote nullifies the unfinished vote started by Arun?
 Sorry, I am dense, apparently.

Yes, nobody should feel bound by either vote. The bylaw change
clarifies that release plans are for RMs to solicit feedback and gauge
PMC support for an artifact, not pre-approvals for doing work.

 Can we limit the vote thread to the merits of the release then?

Happily.

 That sound like adding an insult to injury, if my forth-language skills do not
 mislead me.

They do mislead you, or I've expressed the point imprecisely. We can
take this offline. -C

   On Thu, May 30, 2013 at 01:48PM, Chris Douglas wrote:
   On Thu, May 30, 2013 at 10:57 AM, Arun C Murthy a...@hortonworks.com 
   wrote:
Why not include MAPREDUCE-4211 as well rather than create one 
release per patch?
  
   From Cos's description, it sounded like these were backports of fixes
   to help Sqoop2 and fix some build issues. If it's not just to fixup
   leftover bugs in 2.0.4 *once* so downstream projects can integrate
   against 2.0.4.1, and this a release series, then I've completely
   misunderstood the purpose.
  
   Cos, are you planning 2.0.4.2?
  
Also, this is the first time we are seeing a four-numbered scheme in 
Hadoop. Why not call this 2.0.5-alpha?
  
   Good point. Since it contains only backports from branch-2, it would
   make sense for it to be an intermediate release.
  
   I shouldn't have to say this, but I'm changing my vote to -1 while we
   work this out. -C
  
On May 24, 2013, at 8:48 PM, Konstantin Boudnik wrote:
   
All,
   
I have created a release candidate (rc0) for hadoop-2.0.4.1-alpha 
that I would
like to release.
   
This is a stabilization release that includes fixed for a couple a 
of issues
discovered in the testing with BigTop 0.6.0 release candidate.
   
The RC is available at: 
http://people.apache.org/~cos/hadoop-2.0.4.1-alpha-rc0/
The RC tag in svn is here: 
http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4.1-alpha-rc0
   
The maven artifacts are available via repository.apache.org.
   
Please try the release bits and vote; the vote will run for the 
usual 7 days.
   
Thanks for your voting
 Cos
   
   
   

Re: [VOTE] Release Apache Hadoop 2.0.4.1-alpha

[Chris Douglas]

+1

Checksum and signature match, ran some unit tests, verified w/ a diff
of release-2.0.4-alpha that the release contains MAPREDUCE-5240 and
HADOOP-9407, plus some fixups to the release notes. -C

On Fri, May 24, 2013 at 8:48 PM, Konstantin Boudnik c...@apache.org wrote:
 All,

 I have created a release candidate (rc0) for hadoop-2.0.4.1-alpha that I would
 like to release.

 This is a stabilization release that includes fixed for a couple a of issues
 discovered in the testing with BigTop 0.6.0 release candidate.

 The RC is available at: 
 http://people.apache.org/~cos/hadoop-2.0.4.1-alpha-rc0/
 The RC tag in svn is here: 
 http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4.1-alpha-rc0

 The maven artifacts are available via repository.apache.org.

 Please try the release bits and vote; the vote will run for the usual 7 days.

 Thanks for your voting
   Cos

Re: Heads up - 2.0.5-beta

[Chris Douglas]

On Thu, May 2, 2013 at 2:11 AM, Konstantin Shvachko
shv.had...@gmail.com wrote:
 On Thu, May 2, 2013 at 12:07 AM, Chris Douglas cdoug...@apache.org wrote:
 Can anyone remember why we vote on release plans? -C

 To vote on features to include in the release.

Since most features are developed in branches (requiring a merge
vote), each change is RTC, and the release itself requires a vote... a
vote on the executive summary for a release is a poor time to engage
development. It doesn't seem to accomplish anything when it's not a
formality, so maybe we're better without it. Thoughts?

 I am arguing against invasive and destructive features proposed for the
 release.

Heh; do we need new tags in JIRA?

Setting aside the choice of words, we don't assign work by voting.
Stability is a shared goal, but conflating it with inertia after our
experiences with the 0.20 forks, 0.21, and 0.22 takes exactly the
wrong lessons from those episodes.

If you want to create a 2.x branch, pull out the features you view as
high-risk, and invite others to join your effort: you don't need
anyone's permission. If the bylaws contradict this, then that's a bug.
But one can't vote a set of priorities into preeminence, he can only
convince others to share them *and work on them.* It's cheap to
reassign versions to accommodate whatever shape the community takes,
but voting first and expecting everyone to follow the result has never
worked. Cos's chart gives the lie to the attempt: every time we've
tried, we end up reassigning the versions according to reality,
anyway. -C