[VOTE] Release Apache Hadoop 2.9.2 (RC0)

[Akira Ajisaka]

Hi folks,

I have put together a release candidate (RC0) for Hadoop 2.9.2. It
includes 204 bug fixes and improvements since 2.9.1. [1]

The RC is available at http://home.apache.org/~aajisaka/hadoop-2.9.2-RC0/
Git signed tag is release-2.9.2-RC0 and the checksum is
826afbeae31ca687bc2f8471dc841b66ed2c6704
The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1166/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Please try the release and vote. The vote will run for 5 days.

[1] https://s.apache.org/2.9.2-fixed-jiras

Thanks,
Akira

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: branch-2.9.2 is almost closed for commit

[Akira Ajisaka]

Hi all,

I wanted to push release-2.9.2-RC0 tag and wrongly pushed all the
local tags by 'git push --tags'. I deleted some tags but I couldn't
delete 'rel/release-' tag because all the tags start with 'rel/' are
protected. I'll ask infra to delete this tag.
Sorry for pushing the wrong tags.

-Akira

2018年11月13日(火) 16:07 Akira Ajisaka :
>
> Recently I hit the gpg-agent cache ttl issue while creating 2.9.2 RC0,
> and the issue was fixed by HADOOP-15923.
> I'll create RC0 and start the vote by the end of this week. Sorry for the 
> delay.
>
> Thanks,
> Akira
> 2018年11月6日(火) 13:51 Akira Ajisaka :
> >
> > Hi folks,
> >
> > Now there is only 1 critical issues targeted for 2.9.2 (YARN-8233), so
> > I'd like to close branch-2.9.2 except YARN-8233.
> > I create RC0 as soon as YARN-8233 is committed to branch-2.9.2.
> >
> > Thanks,
> > Akira

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-9019) Ratio of ResourceCaculator implementations could return NaN

[Szilard Nemeth (JIRA)]

Szilard Nemeth created YARN-9019:


 Summary: Ratio of ResourceCaculator implementations could return 
NaN
 Key: YARN-9019
 URL: https://issues.apache.org/jira/browse/YARN-9019
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Szilard Nemeth
Assignee: Szilard Nemeth






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-9018) Add functionality to AuxiliaryLocalPathHandler to return all locations to read for a given path

[Kuhu Shukla (JIRA)]

Kuhu Shukla created YARN-9018:
-

 Summary: Add functionality to AuxiliaryLocalPathHandler to return 
all locations to read for a given path
 Key: YARN-9018
 URL: https://issues.apache.org/jira/browse/YARN-9018
 Project: Hadoop YARN
  Issue Type: Improvement
Affects Versions: 2.8.5, 3.0.3
Reporter: Kuhu Shukla


Analogous to LocalDirAllocator#getAllLocalPathsToRead, this will allow aux 
services(and other components) to use this function that they rely on when 
using the former class objects.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-9017) PlacementRule order is not maintained in CS

[Bibin A Chundatt (JIRA)]

Bibin A Chundatt created YARN-9017:
--

 Summary: PlacementRule order is not maintained in CS
 Key: YARN-9017
 URL: https://issues.apache.org/jira/browse/YARN-9017
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Bibin A Chundatt


{{yarn.scheduler.queue-placement-rules}} doesn't work as expected in Capacity 
Scheduler
{quote}
* **Queue Mapping Interface based on Default or User Defined Placement Rules** 
- This feature allows users to map a job to a specific queue based on some 
default placement rule. For instance based on user & group, or application 
name. User can also define their own placement rule.
{quote}
As per current UserGroupMapping is always added in placementRule. 
{{CapacityScheduler#updatePlacementRules}}
{code}
// Initialize placement rules
Collection placementRuleStrs = conf.getStringCollection(
YarnConfiguration.QUEUE_PLACEMENT_RULES);
List placementRules = new ArrayList<>();
...

// add UserGroupMappingPlacementRule if absent
distingushRuleSet.add(YarnConfiguration.USER_GROUP_PLACEMENT_RULE);
{code}
PlacementRule configuration order is not maintained 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-9016) DocumentStore as a backend for ATSv2

[Sushil Ks (JIRA)]

Sushil Ks created YARN-9016:
---

 Summary: DocumentStore as a backend for ATSv2
 Key: YARN-9016
 URL: https://issues.apache.org/jira/browse/YARN-9016
 Project: Hadoop YARN
  Issue Type: New Feature
  Components: ATSv2
Reporter: Sushil Ks
Assignee: Sushil Ks


h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h4.  

*How to enable?*

Add the flowing properties under *yarn-site.xml*

 
{code:java}


 yarn.timeline-service.writer.class
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.documentstore.db-name   
   YOUR_DATABASE_NAME  

 {code}
h5.  
h3. *Creating DB and Collections for storing documents*

           *           *This is similar to HBase *TimelineSchemaCreator* the 
following command needs to be executed once for setting up the database and 
collections for storing documents.
{code:java}
hadoop 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
h3.  

*Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..

 

{{
 
 yarn.timeline-service.store-type
 COSMOS_DB
 }}
 {{
 yarn.timeline-service.cosmos-db.endpoint
 
[http://YOUR_AZURE_COSMOS_DB_URL:443/|http://your_azure_cosmos_db_url:443/]
 }}
 {{
 yarn.timeline-service.cosmos-db.masterkey
 YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL
 }}

 
   *Testing locally*
               In order to test the Azure CosmosDB as a DocumentStore locally, 
install the emulator from 
[here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
start it locally. Set the endpoint and master key under *yarn-site.xml* as 
mentioned above and run any example job like DistributedShell etc. Later you 
can check the data explorer UI of Azure CosmosDB locally to query the documents 
or even launch the *TimelineReader* locally and fetch/query the data from REST 
API's.   
                   

      



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Hadoop 3.2 Release Plan proposal

[Sunil G]

Hi Folks,

All blockers are closed by last weekend and corrected jiras as well.
Preparing RC0 now. Facing some issue with shaded jars file size. Hence
respinning again.
Planning to complete this by end of this week.

Thanks
Sunil

On Fri, Oct 26, 2018 at 2:34 AM Konstantin Shvachko 
wrote:

> Another thing is that I see a bunch of jiras under HDFS-8707, which don't
> have the Fix Version field listing 3.2, some have it just empty.
> This means they will not be populated into release notes.
>
> Thanks,
> --Konstantin
>
> On Thu, Oct 25, 2018 at 7:59 AM Sunil G  wrote:
>
> > Thanks Konstantin for pointing out.
> > As 3.2 is pretty much on RC level, its better we try to find a good
> > solution to this issue.
> >
> > I ll follow up on this in the jira.
> >
> > - Sunil
> >
> > On Thu, Oct 25, 2018 at 11:35 AM Konstantin Shvachko <
> shv.had...@gmail.com>
> > wrote:
> >
> >> I've tried to attract attention to an incompatibility issue through the
> >> jira, but it didn't work. So pitching in in this thread.
> >> https://issues.apache.org/jira/browse/HDFS-12026
> >> It introduced binary incompatibility, which will prevent people from
> >> upgrading from 3.1 to 3.2.
> >> I think it can get messy if we release anything with this feature.
> >>
> >> Thanks,
> >> --Konstantin
> >>
> >> On Mon, Oct 22, 2018 at 5:01 AM Steve Loughran 
> >> wrote:
> >>
> >>> its in.
> >>>
> >>> good catch!
> >>>
> >>> On 20 Oct 2018, at 01:35, Wei-Chiu Chuang   >>> weic...@cloudera.com>> wrote:
> >>>
> >>> Thanks Sunil G for driving the release,
> >>> I filed HADOOP-15866<
> https://issues.apache.org/jira/browse/HADOOP-15866>
> >>> for a compat fix. If any one has cycle please review it, as I think it
> is
> >>> needed for 3.2.0.
> >>>
> >>> On Thu, Oct 18, 2018 at 4:43 AM Sunil G  >>> sun...@apache.org>> wrote:
> >>> Hi Folks,
> >>>
> >>> As we previously communicated for 3.2.0 release, we have delayed due to
> >>> few
> >>> blockers in our gate.
> >>>
> >>> I just cut branch-3.2.0 for release purpose. branch-3.2 will be open
> for
> >>> all bug fixes.
> >>>
> >>> - Sunil
> >>>
> >>>
> >>> On Tue, Oct 16, 2018 at 8:59 AM Sunil G  >>> sun...@apache.org>> wrote:
> >>>
> >>> > Hi Folks,
> >>> >
> >>> > We are now close to RC as other blocker issues are now merged to
> trunk
> >>> and
> >>> > branch-3.2. Last 2 critical issues are closer to merge and will be
> >>> > committed in few hours.
> >>> > With this, I will be creating 3.2.0 branch today and will go ahead
> >>> with RC
> >>> > related process.
> >>> >
> >>> > - Sunil
> >>> >
> >>> > On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <
> jonben...@stripe.com
> >>> >
> >>> > wrote:
> >>> >
> >>> >> Hello, were there any updates around the 3.2.0 RC timing? All I see
> in
> >>> >> the current blockers are related to the new Submarine subproject,
> >>> wasn't
> >>> >> sure if that is what is holding things up.
> >>> >>
> >>> >> Cheers,
> >>> >> Jon
> >>> >>
> >>> >> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G  >>> sun...@apache.org>> wrote:
> >>> >>
> >>> >>> Thanks Robert and Haibo for quickly correcting same.
> >>> >>> Sigh, I somehow missed one file while committing the change. Sorry
> >>> for
> >>> >>> the
> >>> >>> trouble.
> >>> >>>
> >>> >>> - Sunil
> >>> >>>
> >>> >>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter  >>> >
> >>> >>> wrote:
> >>> >>>
> >>> >>> > Looks like there's two that weren't updated:
> >>> >>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" .
> >>> -r
> >>> >>> > --include=pom.xml
> >>> >>> > ./hadoop-project/pom.xml:
> >>> >>> >
> >>> 3.2.0-SNAPSHOT
> >>> >>> > ./pom.xml:3.2.0-SNAPSHOT
> >>> >>> >
> >>> >>> > I've just pushed in an addendum commit to fix those.
> >>> >>> > In the future, please make sure to do a sanity compile when
> >>> updating
> >>> >>> poms.
> >>> >>> >
> >>> >>> > thanks
> >>> >>> > - Robert
> >>> >>> >
> >>> >>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
> >>> >>> mailto:fab...@cloudera.com.invalid>>
> >>> >>> > wrote:
> >>> >>> >
> >>> >>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in
> >>> the
> >>> >>> >> top-level pom.xml?
> >>> >>> >>
> >>> >>> >>
> >>> >>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G  >>> > wrote:
> >>> >>> >>
> >>> >>> >> > Hi All
> >>> >>> >> >
> >>> >>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset
> >>> trunk
> >>> >>> to
> >>> >>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
> >>> >>> necessary
> >>> >>> >> > patches are pulled into branch-3.2.
> >>> >>> >> >
> >>> >>> >> > Thank You
> >>> >>> >> > - Sunil
> >>> >>> >> >
> >>> >>> >> >
> >>> >>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G  >>> > wrote:
> >>> >>> >> >
> >>> >>> >> > > Hi All
> >>> >>> >> > >
> >>> >>> >> > > We are now down to the last Blocker and HADOOP-15407 is
> >>> merged to
> >>> >>> >> trunk.
> >>> >>> >> > > Thanks for the support.
> >>> >>> >> > >
> 

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/956/

[Nov 12, 2018 6:27:02 AM] (sunilg) YARN-8902. [CSI] Add volume manager that 
manages CSI volume lifecycle.
[Nov 12, 2018 9:54:41 AM] (elek) HDDS-767. OM should not search for STDOUT root 
logger for audit logging.
[Nov 12, 2018 10:18:23 AM] (wwei) YARN-8987. Usability improvements 
node-attributes CLI. Contributed by 
[Nov 12, 2018 12:58:05 PM] (stevel) HADOOP-15110. Gauges are getting logged in 
exceptions from
[Nov 12, 2018 3:39:30 PM] (sunilg) YARN-8877. [CSI] Extend service spec to 
allow setting resource
[Nov 12, 2018 6:03:11 PM] (nanda) HDDS-576. Move ContainerWithPipeline creation 
to RPC endpoint.
[Nov 12, 2018 6:42:30 PM] (billie) YARN-8776. Implement Container Exec feature 
in LinuxContainerExecutor.
[Nov 12, 2018 10:08:39 PM] (jitendra) HDDS-709. Modify Close Container handling 
sequence on datanodes.
[Nov 12, 2018 11:06:43 PM] (gifuma) YARN-8997. [Submarine] Small refactors of 
modifier, condition check and
[Nov 12, 2018 11:31:42 PM] (arp) HDFS-14065. Failed Storage Locations shows 
nothing in the Datanode
[Nov 13, 2018 12:53:10 AM] (eyang) YARN-8761. Service AM support for 
decommissioning component instances.  




-1 overall


The following subsystems voted -1:
asflicense findbugs hadolint pathlen unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.util.TestDiskCheckerWithDiskIo 
   hadoop.util.TestReadWriteDiskValidator 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.yarn.server.resourcemanager.TestCapacitySchedulerMetrics 
   
hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
 
   hadoop.yarn.sls.TestSLSRunner 
   hadoop.yarn.sls.TestSLSStreamAMSynth 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/956/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/956/artifact/out/diff-compile-javac-root.txt
  [324K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/956/artifact/out/diff-checkstyle-root.txt
  [17M]

   hadolint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/956/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   pathlen:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/956/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/956/artifact/out/diff-patch-pylint.txt
  [40K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/956/artifact/out/diff-patch-shellcheck.txt
  [68K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/956/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/956/artifact/out/whitespace-eol.txt
  [9.3M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/956/artifact/out/whitespace-tabs.txt
  [1.1M]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/956/artifact/out/branch-findbugs-hadoop-hdds_client.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/956/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/956/artifact/out/branch-findbugs-hadoop-hdds_framework.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/956/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/956/artifact/out/branch-findbugs-hadoop-hdds_tools.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/956/artifact/out/branch-findbugs-hadoop-ozone_client.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/956/artifact/out/branch-findbugs-hadoop-ozone_common.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/956/artifact/out/branch-findbugs-hadoop-ozone_objectstore-service.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/956/artifact/out/branch-findbugs-hadoop-ozone_ozone-manager.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/956/artifact/out/branch-findbugs-hadoop-ozone_ozonefs.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/956/artifact/out/branch-findbugs-hadoop-ozone_s3gateway.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/956/artifact/out/branch-findbu

[VOTE] Release Apache Hadoop Ozone 0.3.0-alpha (RC0)

[Elek, Marton]

Hi all,

I've created the first release candidate (RC0) for Apache Hadoop Ozone
0.3.0-alpha according to the plans shared here previously.

This is the second release of Apache Hadoop Ozone. Notable changes since
the first release:

* A new S3 compatible rest server is added. Ozone can be used from any
S3 compatible tools (HDDS-434)
* Ozone Hadoop file system URL prefix is renamed from o3:// to o3fs://
(HDDS-651)
* Extensive testing and stability improvements of OzoneFs.
* Spark, YARN and Hive support and stability improvements.
* Improved Pipeline handling and recovery.
* Separated/dedicated classpath definitions for all the Ozone
components. (HDDS-447)

The RC artifacts are available from:
https://home.apache.org/~elek/ozone-0.3.0-alpha-rc0/

The RC tag in git is: ozone-0.3.0-alpha-RC0 (dc661083683)

Please try it out, vote, or just give us feedback.

The vote will run for 5 days, ending on November 18, 2018 13:00 UTC.


Thank you very much,
Marton

PS:

The easiest way to try it out is:

1. Download the binary artifact
2. Read the docs from ./docs/index.html
3. TLDR; cd compose/ozone && docker-compose up -d
4. open localhost:9874 or localhost:9876



The easiest way to try it out from the source:

1. mvn  install -DskipTests -Pdist -Dmaven.javadoc.skip=true -Phdds
-DskipShade -am -pl :hadoop-ozone-dist
2. cd hadoop-ozone/dist/target/ozone-0.3.0-alpha && docker-compose up -d



The easiest way to test basic functionality (with acceptance tests):

1. mvn  install -DskipTests -Pdist -Dmaven.javadoc.skip=true -Phdds
-DskipShade -am -pl :hadoop-ozone-dist
2. cd hadoop-ozone/dist/target/ozone-0.3.0-alpha/smoketest
3. ./test.sh

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-9015) Add an interface for device plugin to provide customized scheduler

[Zhankun Tang (JIRA)]

Zhankun Tang created YARN-9015:
--

 Summary: Add an interface for device plugin to provide customized 
scheduler
 Key: YARN-9015
 URL: https://issues.apache.org/jira/browse/YARN-9015
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhankun Tang
Assignee: Zhankun Tang


A vendor might need a customized scheduling policy for their devices. It could 
be scheduled based on topology, resource utilization, virtualization, device 
attribute and so on.

We'll provide another optional interface "DevicePluginScheduler" for the vendor 
device plugin to implement. Once it's implemented, the framework will prefer it 
to the default scheduler.

This would bring more flexibility to the framework's scheduling mechanism.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org