Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)

[Shashikant Banerjee]

Looks like the link with the filter seems to be private. I can't see the 
blocker list. 
https://issues.apache.org/jira/issues/?filter=12343997

Meanwhile , I will be working on testing the release.

Thanks
Shashi
On 6/1/18, 11:18 AM, "Yongjun Zhang"  wrote:

Greetings all,

I've created the first release candidate (RC0) for Apache Hadoop
3.0.3. This is our next maintenance release to follow up 3.0.2. It includes
about 249
important fixes and improvements, among which there are 8 blockers. See
https://issues.apache.org/jira/issues/?filter=12343997

The RC artifacts are available at:
https://dist.apache.org/repos/dist/dev/hadoop/3.0.3-RC0/

The maven artifacts are available via
https://repository.apache.org/content/repositories/orgapachehadoop-1126

Please try the release and vote; the vote will run for the usual 5 working
days, ending on 06/07/2018 PST time. Would really appreciate your
participation here.

I bumped into quite some issues along the way, many thanks to quite a few
people who helped, especially Sammi Chen, Andrew Wang, Junping Du, Eddy Xu.

Thanks,

--Yongjun

[VOTE] Release Apache Hadoop 3.0.3 (RC0)

[Yongjun Zhang]

Greetings all,

I've created the first release candidate (RC0) for Apache Hadoop
3.0.3. This is our next maintenance release to follow up 3.0.2. It includes
about 249
important fixes and improvements, among which there are 8 blockers. See
https://issues.apache.org/jira/issues/?filter=12343997

The RC artifacts are available at:
https://dist.apache.org/repos/dist/dev/hadoop/3.0.3-RC0/

The maven artifacts are available via
https://repository.apache.org/content/repositories/orgapachehadoop-1126

Please try the release and vote; the vote will run for the usual 5 working
days, ending on 06/07/2018 PST time. Would really appreciate your
participation here.

I bumped into quite some issues along the way, many thanks to quite a few
people who helped, especially Sammi Chen, Andrew Wang, Junping Du, Eddy Xu.

Thanks,

--Yongjun

Apache Hadoop qbt Report: trunk+JDK8 on Windows/x64

[Apache Jenkins Server]

For more details, see https://builds.apache.org/job/hadoop-trunk-win/484/

[May 31, 2018 7:53:34 AM] (aajisaka) YETUS-615. Fix a typo in smart-apply-patch
[May 30, 2018 3:52:07 PM] (aengineer) HDDS-88. Create separate message 
structure to represent ports in
[May 30, 2018 4:50:19 PM] (ericp) YARN-8377: Javadoc build failed in 
hadoop-yarn-server-nodemanager.
[May 30, 2018 5:13:52 PM] (inigoiri) HDFS-13632. Randomize baseDir for 
MiniJournalCluster in MiniQJMHACluster
[May 30, 2018 5:22:04 PM] (inigoiri) HDFS-13629. Some tests in 
TestDiskBalancerCommand fail on Windows due to
[May 30, 2018 7:37:43 PM] (billie) YARN-8368. yarn app start cli should print 
applicationId. Contributed by
[May 30, 2018 8:19:13 PM] (billie) YARN-8350. NPE in service AM related to 
placement policy. Contributed by
[May 30, 2018 9:21:13 PM] (aengineer) HDDS-128. Support for DN to SCM 
signaling. Contributed by Nanda Kumar.
[May 31, 2018 9:31:10 AM] (aajisaka) MAPREDUCE-7098. Upgrade common-langs 
version to 3.7 in




-1 overall


The following subsystems voted -1:
compile mvninstall pathlen unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc javac


The following subsystems are considered long running:
(runtime bigger than 1h 00m 00s)
unit


Specific tests:

Failed junit tests :

   hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec 
   hadoop.fs.contract.rawlocal.TestRawlocalContractAppend 
   hadoop.fs.TestFsShellCopy 
   hadoop.fs.TestFsShellList 
   hadoop.fs.TestLocalFileSystem 
   hadoop.http.TestHttpServer 
   hadoop.http.TestHttpServerLogs 
   hadoop.io.compress.TestCodec 
   hadoop.io.nativeio.TestNativeIO 
   hadoop.ipc.TestSocketFactory 
   hadoop.metrics2.impl.TestStatsDMetrics 
   hadoop.security.TestSecurityUtil 
   hadoop.security.TestShellBasedUnixGroupsMapping 
   hadoop.security.token.TestDtUtilShell 
   hadoop.util.TestDiskCheckerWithDiskIo 
   hadoop.util.TestNativeCodeLoader 
   hadoop.util.TestNodeHealthScriptRunner 
   hadoop.hdfs.qjournal.server.TestJournalNode 
   hadoop.hdfs.qjournal.server.TestJournalNodeSync 
   hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestProvidedImpl 
   hadoop.hdfs.server.datanode.TestBlockPoolSliceStorage 
   hadoop.hdfs.server.datanode.TestBlockScanner 
   hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics 
   hadoop.hdfs.server.datanode.TestDataNodeFaultInjector 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure 
   hadoop.hdfs.server.datanode.TestDirectoryScanner 
   hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand 
   hadoop.hdfs.server.diskbalancer.TestDiskBalancerRPC 
   hadoop.hdfs.server.mover.TestMover 
   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA 
   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA 
   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics 
   hadoop.hdfs.server.namenode.TestDecommissioningStatus 
   hadoop.hdfs.server.namenode.TestEditLogRace 
   hadoop.hdfs.server.namenode.TestReencryptionHandler 
   hadoop.hdfs.server.namenode.TestStartup 
   hadoop.hdfs.TestBlockStoragePolicy 
   hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs 
   hadoop.hdfs.TestDFSShell 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy 
   hadoop.hdfs.TestDFSUpgradeFromImage 
   hadoop.hdfs.TestFetchImage 
   hadoop.hdfs.TestHDFSFileSystemContract 
   hadoop.hdfs.TestPread 
   hadoop.hdfs.TestReadStripedFileWithDecodingCorruptData 
   hadoop.hdfs.TestReconstructStripedFile 
   hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy 
   hadoop.hdfs.TestSecureEncryptionZoneWithKMS 
   hadoop.hdfs.TestTrashWithSecureEncryptionZones 
   hadoop.hdfs.tools.TestDFSAdmin 
   hadoop.hdfs.tools.TestDFSAdminWithHA 
   hadoop.hdfs.tools.TestGetConf 
   hadoop.hdfs.web.TestWebHDFS 
   hadoop.hdfs.web.TestWebHdfsUrl 
   hadoop.fs.http.server.TestHttpFSServerWebServer 
   
hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch 
   
hadoop.yarn.server.nodemanager.containermanager.linux.privileged.TestPrivilegedOperationExecutor
 
   
hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestCGroupElasticMemoryController
 
   
hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestCGroupsHandlerImpl
 
   
hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime
 
   
hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestJavaSandboxLinuxContainerRuntime
 
   
hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestAppLogAggregatorImpl
 
   

[jira] [Created] (HADOOP-15507) Add MapReduce counters about EC bytes read

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-15507:
--

 Summary: Add MapReduce counters about EC bytes read
 Key: HADOOP-15507
 URL: https://issues.apache.org/jira/browse/HADOOP-15507
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Xiao Chen
Assignee: Xiao Chen


HDFS has added Erasure Coding support in HDFS-7285. There are HDFS level 
[ReadStatistics|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ReadStatistics.java]
 so from DFSClient we can know how much reads are EC/replication.

In order for users to have a better view of how much of their workload is 
impacted by EC, we can expose EC read bytes to File System Counters, and to 
MapReduce's job counters. This way, end users can tell from MR jobs directly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15506) Upgrading Azure Storage Sdk version and updated corresponding code blocks

[Esfandiar Manii (JIRA)]

Esfandiar Manii created HADOOP-15506:


 Summary: Upgrading Azure Storage Sdk version and updated 
corresponding code blocks
 Key: HADOOP-15506
 URL: https://issues.apache.org/jira/browse/HADOOP-15506
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs/azure
Affects Versions: 3.2.0
Reporter: Esfandiar Manii
Assignee: Esfandiar Manii


- Upgraded Azure Storage Sdk to 7.0.0
- Fixed code issues and couple of tests




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

REMINDER: Apache EU Roadshow 2018 in Berlin is less than 2 weeks away!

[sharan]

Hello Apache Supporters and Enthusiasts

This is a reminder that our Apache EU Roadshow in Berlin is less than 
two weeks away and we need your help to spread the word. Please let your 
work colleagues, friends and anyone interested in any attending know 
about our Apache EU Roadshow event.


We have a great schedule including tracks on Apache Tomcat, Apache Http 
Server, Microservices, Internet of Things (IoT) and Cloud Technologies. 
You can find more details at the link below:


https://s.apache.org/0hnG

Ticket prices will be going up on 8^th June 2018, so please make sure 
that you register soon if you want to beat the price increase. 
https://foss-backstage.de/tickets


Remember that registering for the Apache EU Roadshow also gives you 
access to FOSS Backstage so you can attend any talks and workshops from 
both conferences. And don’t forget that our Apache Lounge will be open 
throughout the whole conference as a place to meet up, hack and relax.


We look forward to seeing you in Berlin!

Thanks
Sharan Foga,  VP Apache Community Development

http://apachecon.com/
@apachecon

PLEASE NOTE: You are receiving this message because you are subscribed 
to a user@ or dev@ list of one or more Apache Software Foundation projects.

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/797/

[May 30, 2018 2:20:22 AM] (fabbri) HADOOP-15480 
AbstractS3GuardToolTestBase.testDiffCommand fails when
[May 30, 2018 8:52:21 AM] (yqlin) HDFS-13626. Fix incorrect username when deny 
the setOwner operation.
[May 30, 2018 3:52:07 PM] (aengineer) HDDS-88. Create separate message 
structure to represent ports in
[May 30, 2018 4:50:19 PM] (ericp) YARN-8377: Javadoc build failed in 
hadoop-yarn-server-nodemanager.
[May 30, 2018 5:13:52 PM] (inigoiri) HDFS-13632. Randomize baseDir for 
MiniJournalCluster in MiniQJMHACluster
[May 30, 2018 5:22:04 PM] (inigoiri) HDFS-13629. Some tests in 
TestDiskBalancerCommand fail on Windows due to
[May 30, 2018 7:37:43 PM] (billie) YARN-8368. yarn app start cli should print 
applicationId. Contributed by
[May 30, 2018 8:19:13 PM] (billie) YARN-8350. NPE in service AM related to 
placement policy. Contributed by
[May 30, 2018 9:21:13 PM] (aengineer) HDDS-128. Support for DN to SCM 
signaling. Contributed by Nanda Kumar.




-1 overall


The following subsystems voted -1:
asflicense findbugs pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-hdds/common 
   Found reliance on default encoding in 
org.apache.hadoop.utils.MetadataKeyFilters$KeyPrefixFilter.filterKey(byte[], 
byte[], byte[]):in 
org.apache.hadoop.utils.MetadataKeyFilters$KeyPrefixFilter.filterKey(byte[], 
byte[], byte[]): String.getBytes() At MetadataKeyFilters.java:[line 97] 

Failed junit tests :

   hadoop.hdfs.client.impl.TestBlockReaderLocal 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart 
   
hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities 
   
hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
 
   
hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageDomain 
   
hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction
 
   
hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageSchema 
   hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun 
   hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps 
   
hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity 
   hadoop.yarn.applications.distributedshell.TestDistributedShell 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/797/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/797/artifact/out/diff-compile-javac-root.txt
  [332K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/797/artifact/out/diff-checkstyle-root.txt
  [17M]

   pathlen:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/797/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/797/artifact/out/diff-patch-pylint.txt
  [24K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/797/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/797/artifact/out/diff-patch-shelldocs.txt
  [16K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/797/artifact/out/whitespace-eol.txt
  [9.4M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/797/artifact/out/whitespace-tabs.txt
  [1.1M]

   xml:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/797/artifact/out/xml.txt
  [4.0K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/797/artifact/out/branch-findbugs-hadoop-hdds_client.txt
  [56K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/797/artifact/out/branch-findbugs-hadoop-hdds_common-warnings.html
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/797/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt
  [52K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/797/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt
  [56K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/797/artifact/out/branch-findbugs-hadoop-hdds_tools.txt
  [16K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/797/artifact/out/branch-findbugs-hadoop-ozone_client.txt
  [8.0K]
   

Re: Apache Hadoop 3.0.3 Release plan

[Yongjun Zhang]

Unfortunately still hit another issue, step 4 in
https://wiki.apache.org/hadoop/HowToRelease doesn't deploy artifacts to
Nexus. According to Andrew, it should and we should skip step 10.

Though I have the release tarballs, because of this deployment issue, I
still have to investigate further. Sorry for the inconvenience.

--Yongjun

On Wed, May 30, 2018 at 11:49 AM, Yongjun Zhang 
wrote:

> Hi,
>
> The build issues are all solved, and I have cut the 3.0.3 branch and close
> to get a build out. Since it's taking me a bit more time (I expect to send
> vote invitation email by today). I would like to send a heads-up notice now.
>
> Thank you all for feedback, and many thanks to Sammi Chen, Andrew Wang,
> Eddy Xu who helped when I tried to solve the build issues.
>
> At this point, please be aware of the existence of branch-3.0,
> branch-3.0.3.
>
> Best,
>
> --Yongjun
>
>
>
> On Sat, May 26, 2018 at 11:52 PM, Yongjun Zhang 
> wrote:
>
>> HI,
>>
>> I did build before cut branch and hit some issues, have not got to the
>> bottom, will cut branch after the build issues are resolved.
>>
>> Thanks.
>>
>> --Yongjun
>>
>> On Sat, May 26, 2018 at 1:46 PM, Yongjun Zhang 
>> wrote:
>>
>>> Hi All,
>>>
>>> I will be working on cutting the 3.0.3 branch and trying a build today.
>>>
>>> Thanks.
>>>
>>> --Yongjun
>>>
>>>
>>>
>>> On Wed, May 23, 2018 at 3:31 PM, Yongjun Zhang 
>>> wrote:
>>>
 Thanks Eric. Sounds good. I may try to see if I can do the branching/RC
 sooner.

 --Yongjun


 On Wed, May 23, 2018 at 2:18 PM, Eric Badger  wrote:

> My thinking is to cut the branch in next couple of days and create RC
> for
> vote at the end of month.
>   >  We plan to cut branch-3.0.3 by the coming Wednesday (May 9th)
> and vote for RC on May 30th
>   I much prefer to wait to cut the branch until just before the
> production of the release and the vote. With so many branches, we 
> sometimes
> miss putting critical bug fixes in unreleased branches if thebranch is
> cut too early.
>
> Echoing Eric Payne, I think we should wait to cut the branch until we
> are actually creating the RC to vote on (i.e. on May 29 or 30 if the vote
> is to be on May 30).
>
> Eric
>
>
>
> On Wed, May 23, 2018 at 4:11 PM, Yongjun Zhang 
> wrote:
>
>> Hi,
>>
>> I have gardened the jiras for 3.0.3, and have the following open
>> issues:
>>
>> https://issues.apache.org/jira/issues/?filter=12343970
>>
>> Two of them are blockers, one of them (YARN-8346) has already got +1
>> for
>> patch, the other (YARN-8108) will take longer time to resolve and it
>> seems
>> we can possibly push it to next release given 3.0.2 also has the
>> issue.
>>
>> My thinking is to cut the branch in next couple of days and create RC
>> for
>> vote at the end of month.
>>
>> Comments are welcome.
>>
>> Thanks,
>>
>> --Yongjun
>>
>> On Tue, May 8, 2018 at 11:40 AM, Vrushali C 
>> wrote:
>>
>> > +1 for including the YARN-7190 patch in 3.0.3 release. This is a
>> fix that
>> > will enable HBase to use Hadoop 3.0.x in the production line.
>> >
>> > thanks
>> > Vrushali
>> >
>> >
>> > On Tue, May 8, 2018 at 10:24 AM, Yongjun Zhang > >
>> > wrote:
>> >
>> >> Thanks Wei-Chiu and Haibo for the feedback!
>> >>
>> >> Good thing is that I have made the following note couple of days
>> ago when
>> >> I
>> >> looked the at branch diff, so we are on the same page:
>> >>
>> >>  496dc57 Revert "YARN-7190. Ensure only NM classpath in 2.x
>> gets TSv2
>> >> related hbase jars, not the user classpath. Contributed by Varun
>> Saxena."
>> >>
>> >> *YARN-7190 is not in 3.0.2,  I will include it in 3.0.3 per* the
>> comment
>> >> below:
>> >> https://issues.apache.org/jira/browse/YARN-7190?focusedComme
>> >> ntId=16457649&
>> >> page=com.atlassian.jira.plugin.system.issuetabpanels
>> >> > entId=16457649=com.atlassian.jira.plugin.system.issuetabpanels>
>>
>> >> :
>> >> comment-tabpanel#comment-16457649
>> >>
>> >>
>> >> In addition, I will revert   https://issues.apache.org/
>> >> jira/browse/HADOOP-13055 from 3.0.3 since it's a feature.
>> >>
>> >>
>> >> Best,
>> >>
>> >> --Yongjun
>> >>
>> >> On Tue, May 8, 2018 at 8:57 AM, Haibo Chen > >
>> >> wrote:
>> >>
>> >> > +1 on adding YARN-7190 to Hadoop 3.0.x despite the fact that it
>> is
>> >> > technically incompatible.
>> >> > It is critical enough to justify being an exception, IMO.
>> >> >
>> >> > Added Rohith and Vrushali
>> >> >
>> >> > On Tue, May 8, 2018 at 6:20 AM, Wei-Chiu Chuang <
>> weic...@apache.org>
>> >> > wrote: