[jira] [Resolved] (HADOOP-17970) unguava: remove Preconditions from hdfs-projects module

[Ahmed Hussein (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-17970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Hussein resolved HADOOP-17970.

Fix Version/s: 3.4.0
   Resolution: Fixed

> unguava: remove Preconditions from hdfs-projects module
> ---
>
> Key: HADOOP-17970
> URL: https://issues.apache.org/jira/browse/HADOOP-17970
> Project: Hadoop Common
>  Issue Type: Sub-task
>        Reporter: Ahmed Hussein
>    Assignee: Ahmed Hussein
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Replace guava Preconditions by internal implementations that rely on java8+ 
> APIs in the hadoop.util for all modules in hadoop-hdfs-project



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17970) unguava: remove Preconditions from hdfs-project module

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17970:
--

 Summary: unguava: remove Preconditions from hdfs-project module
 Key: HADOOP-17970
 URL: https://issues.apache.org/jira/browse/HADOOP-17970
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


Replace guava Preconditions by internal implementations that rely on java8+ 
APIs in the hadoop.util for all modules in hadoop-hdfs-project





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-17960) hadoop-auth module cannot import non-guava implementation in hatoop util

[Ahmed Hussein (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-17960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Hussein resolved HADOOP-17960.

Resolution: Won't Fix

> hadoop-auth module cannot import non-guava implementation in hatoop util
> 
>
> Key: HADOOP-17960
> URL: https://issues.apache.org/jira/browse/HADOOP-17960
> Project: Hadoop Common
>  Issue Type: Sub-task
>        Reporter: Ahmed Hussein
>Priority: Major
>
> hadoop-common  provides several util implementations in 
> {{org.apache.hadoop.util.*}}. Since hadoop-common depends on hadoop-auth, all 
> the utility implementations cannot be used within hadoop-auth.
> There are several options:
> * similar to {{hadoop-annotations}} generic and utility implementations such 
> as maps, Strings, Preconditions, ..etc could be moved to a new common-util 
> module that has no dependency on other modules.
> * easier fix is to manually replace the guava calls in hadoop-auth module 
> without importing {{hadoop.util.*}}. Only few calls need to be manually 
> replaced: {{Splitter}}, {{Preconditions.checkNotNull}}, and 
> {{Preconditions.checkArgument}}
> CC: [~vjasani] , [~ste...@apache.org], [~tasanuma]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17960) hadoop-auth module cannot import non-guava implementation in hatoop util

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17960:
--

 Summary: hadoop-auth module cannot import non-guava implementation 
in hatoop util
 Key: HADOOP-17960
 URL: https://issues.apache.org/jira/browse/HADOOP-17960
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Ahmed Hussein


hadoop-common  provides several util implementations in 
{{org.apache.hadoop.util.*}}. Since hadoop-common depends on hadoop-auth, all 
the utility implementations cannot be used within hadoop-auth.

There are several options:
* similar to {{hadoop-annotations}} generic and utility implementations such as 
maps, Strings, Preconditions, ..etc could be moved to a new common-util module 
that has no dependency on other modules.
* easier fix is to manually replace the guava calls in hadoop-auth module 
without importing {{hadoop.util.*}}. Only few calls need to be manually 
replaced: {{Splitter}}, {{Preconditions.checkNotNull}}, and 
{{Preconditions.checkArgument}}

CC: [~vjasani] , [~ste...@apache.org], [~tasanuma]





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17930) implement non-guava Precondition checkState

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17930:
--

 Summary: implement non-guava Precondition checkState
 Key: HADOOP-17930
 URL: https://issues.apache.org/jira/browse/HADOOP-17930
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 3.4.0, 3.2.3, 3.3.2
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


As part In order to replace Guava Preconditions, we need to implement our own 
versions of the API.
 This Jira is to add the implementation {{checkState}} to the existing class 
{{org.apache.hadoop.util.Preconditions}}

 +The plan is as follows+
 * implement  {{org.apache.hadoop.util.Preconditions.checkState}} with the 
minimum set of interface used in the current hadoop repo.
 * we can replace {{guava.Preconditions}} by 
{{org.apache.hadoop.util.Preconditions}} once all the interfaces have been 
implemented (both this jira and HADOOP-17929 are complete).
 * We need the change to be easily to be backported in 3.x.


previous jiras:
* HADOOP-17126 was created to replace CheckNotNull.
* HADOOP-17929 replacing checkArgument.

CC: [~ste...@apache.org], [~vjasani]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17929) implement non-guava Precondition checkArgument

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17929:
--

 Summary: implement non-guava Precondition checkArgument
 Key: HADOOP-17929
 URL: https://issues.apache.org/jira/browse/HADOOP-17929
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 3.4.0, 3.2.3, 3.3.2
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


As part In order to replace Guava Preconditions, we need to implement our own 
versions of the API.
 This Jira is to add the implementation {{checkArgument}} to the existing class 
{{org.apache.hadoop.util.Preconditions}}

 +The plan is as follows+
 * implement  {{org.apache.hadoop.util.Preconditions.checkArgument}} with the 
minimum set of interface used in the current hadoop repo.
 * we can replace {{guava.Preconditions}} by 
{{org.apache.hadoop.util.Preconditions}} once all the interfaces have been 
implemented.
 * We need the change to be easily to be backported in 3.x.


A previous jira HADOOP-17126 was created to replace CheckNotNull. Another will 
be created to implement checkState.

CC: [~ste...@apache.org], [~vjasani]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17903) javadoc broken in branch-2.10 root

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17903:
--

 Summary: javadoc broken in branch-2.10 root
 Key: HADOOP-17903
 URL: https://issues.apache.org/jira/browse/HADOOP-17903
 Project: Hadoop Common
  Issue Type: Bug
  Components: build
Affects Versions: 2.10.2
Reporter: Ahmed Hussein


I went through some of the qbt-reports email reports.
I noticed that javadoc root was failing for quite sometime.


{code:bash}
[INFO] --- maven-javadoc-plugin:3.3.0:javadoc (default-cli) @ hadoop-main ---
[WARNING] Error injecting: org.apache.maven.plugins.javadoc.JavadocReport
java.lang.TypeNotPresentException: Type 
org.apache.maven.plugins.javadoc.JavadocReport not present
at 
org.eclipse.sisu.space.URLClassSpace.loadClass(URLClassSpace.java:147)
at org.eclipse.sisu.space.NamedClass.load(NamedClass.java:46)
at 
org.eclipse.sisu.space.AbstractDeferredClass.get(AbstractDeferredClass.java:48)
at 
com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
at 
com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53)
at 
com.google.inject.internal.ProviderInternalFactory$1.call(ProviderInternalFactory.java:65)
at 
com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:115)
at 
org.eclipse.sisu.bean.BeanScheduler$Activator.onProvision(BeanScheduler.java:176)
at 
com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:126)
at 
com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:68)
at 
com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:63)
at 
com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:45)
at 
com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016)
at 
com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012)
at org.eclipse.sisu.inject.Guice4$1.get(Guice4.java:162)
at org.eclipse.sisu.inject.LazyBeanEntry.getValue(LazyBeanEntry.java:81)
at 
org.eclipse.sisu.plexus.LazyPlexusBean.getValue(LazyPlexusBean.java:51)
at 
org.codehaus.plexus.DefaultPlexusContainer.lookup(DefaultPlexusContainer.java:263)
at 
org.codehaus.plexus.DefaultPlexusContainer.lookup(DefaultPlexusContainer.java:255)
at 
org.apache.maven.plugin.internal.DefaultMavenPluginManager.getConfiguredMojo(DefaultMavenPluginManager.java:517)
at 
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:121)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at 
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:607)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: java.lang.UnsupportedClassVersionError: 
org/apache/maven/plugins/javadoc/JavadocReport : Unsupported major.minor 
version 52.0

[jira] [Created] (HADOOP-17898) Upgrade BouncyCastle to 1.69

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17898:
--

 Summary: Upgrade BouncyCastle to 1.69
 Key: HADOOP-17898
 URL: https://issues.apache.org/jira/browse/HADOOP-17898
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.4.0, 3.2.3, 3.3.2
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


Vulnerabilities reported in BouncyCastle:
[CVE-2020-26939|https://nvd.nist.gov/vuln/detail/CVE-2020-26939] moderate 
severity
[CVE-2020-15522|https://nvd.nist.gov/vuln/detail/CVE-2020-15522] moderate 
severity

Affecting releases before 1.66.
 
Upgrade to latest 1.69.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17886) Upgrade ant to 1.10.11

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17886:
--

 Summary: Upgrade ant to 1.10.11
 Key: HADOOP-17886
 URL: https://issues.apache.org/jira/browse/HADOOP-17886
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.2.2, 3.3.0, 3.4.0, 2.10.2
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


Vulnerabilities reported in org.apache.ant:ant:1.10.9
 * [CVE-2021-36374|https://nvd.nist.gov/vuln/detail/CVE-2021-36374] moderate 
severity
 * [CVE-2021-36373|https://nvd.nist.gov/vuln/detail/CVE-2021-36373] moderate 
severity

suggested: org.apache.ant:ant ~> 1.10.11



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17885) Upgrade JSON smart to 1.3.3 on branch-2.10

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17885:
--

 Summary: Upgrade JSON smart to 1.3.3 on branch-2.10
 Key: HADOOP-17885
 URL: https://issues.apache.org/jira/browse/HADOOP-17885
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.10.1, 2.10.0
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


Currently branch-2.10 is using JSON Smart 1.3.1 version which is vulnerable to 
[link CVE-2021-27568|https://nvd.nist.gov/vuln/detail/CVE-2021-27568].

We can upgrade the version to 1.3.1.

+Description of the vulnerability:+

{quote}An issue was discovered in netplex json-smart-v1 through 2015-10-23 and 
json-smart-v2 through 2.4. An exception is thrown from a function, but it is 
not caught, as demonstrated by NumberFormatException. When it is not caught, it 
may cause programs using the library to crash or expose sensitive 
information.{quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] Tips for improving productivity, workflow in the Hadoop project?

[Ahmed Hussein]

Do you consider migrating Jira issues to Github issues?

I am a little bit concerned that there are some committers who still prefer
Jira-precommits over GitHub PR
(P.S. I am not a committer).

Their point is that Github-PR confuses them with discussions/comments being
in two places rather than one.

Personally, I found several Github-PRs comments discussing the validity of
the feature/bug.
As a result:
- recently, JIRA became some sort of a "number generator" with insufficient
description/details as the
  developers and the reviewers spending more time discussing in the PR.
- the relation between a single Jira and Github-PR is 1-to-M. In order to
find related discussions, the user may
  need to visit every PR (that may include closed ones)



On Wed, Jul 14, 2021 at 8:46 AM Steve Loughran 
wrote:

> not sure about stale PR closing; when you've a patch which is still pending
> review it's not that fun to have it closed.
>
> maybe better to have review sessions. I recall many, many years ago
> attempts to try and catch up with all outstanding patch reviews.
>
>
>
>
> On Wed, 14 Jul 2021 at 03:00, Akira Ajisaka  wrote:
>
> > Thank you Wei-Chiu for starting the discussion,
> >
> > > 3. JIRA security
> > I'm +1 to use private JIRA issues to handle vulnerabilities.
> >
> > > 5. Doc update
> > +1, I build the document daily and it helps me fixing documents:
> > https://aajisaka.github.io/hadoop-document/ It's great if the latest
> > document is built and published by the Apache Hadoop community.
> >
> > My idea related to GitHub PR:
> > 1. Disable the precommit jobs for JIRA, always use GitHub PR. It saves
> > costs to configure and debug the precommit jobs.
> > https://issues.apache.org/jira/browse/HADOOP-17798
> > 2. Improve the pull request template for the contributors
> > https://issues.apache.org/jira/browse/HADOOP-17799
> >
> > Regards,
> > Akira
> >
> > On Tue, Jul 13, 2021 at 12:35 PM Wei-Chiu Chuang 
> > wrote:
> > >
> > > I work on multiple projects and learned a bunch from those
> projects.There
> > > are nice add-ons that help with productivity. There are things we can
> do
> > to
> > > help us manage the project better.
> > >
> > > 1. Add new issue types.
> > > We can add "Epic" jira type to organize a set of related jiras. This
> > could
> > > be easier to manage than using a regular JIRA and call it "umbrella".
> > >
> > > 2. GitHub Actions
> > > I am seeing more projects moving to GitHub Actions for precommits. We
> > don't
> > > necessarily need to migrate off Jenkins, but there are nice add-ons
> that
> > > can perform static analysis, catching potential issues. For example,
> > Ozone
> > > adds SonarQube to post-commit, and exports the report to SonarCloud.
> > Other
> > > add-ons are available to scan for docker images, vulnerabilities scans.
> > >
> > > 3. JIRA security
> > > It is possible to set up security level (public/private) in JIRA. This
> > can
> > > be used to track vulnerability issues and be made only visible to
> > > committers. Example: INFRA-15258
> > > <https://issues.apache.org/jira/browse/INFRA-15258>
> > >
> > > 4. New JIRA fields
> > > It's possible to add new fields. For example, we can add a "Reviewer"
> > > field, which could help improve the attention to issues.
> > >
> > > 5. Doc update
> > > It is possible to set up automation such that the doc on the Hadoop
> > website
> > > is refreshed for every commit, providing the latest doc to the public.
> > >
> > > 6. Webhook
> > > It's possible to set up webhook such that every commit in GitHub sends
> a
> > > notification to the ASF slack. It can be used for other kinds of
> > > automation. Sky's the limit.
> > >
> > > Thoughts? What else can do we?
> >
> > -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
> >
>


-- 
Best Regards,

*Ahmed Hussein, PhD*

Re: [DISCUSS] Tips for improving productivity, workflow in the Hadoop project?

[Ahmed Hussein]

Hi Wei-Chu,

Thanks for sharing your experience working on other projects.
The Jira related suggestions sound very useful. The extra fields work very
good for us in our internal Jira system at Yahoo.
In addition to your suggestions, is it possible revisit the
labels/components in Jira. If we can set a quick guide on how to label the
most
common types of Jiras, then this will be of a great help to browse the
system.

Regarding the Webhook and GitHub related suggestions, I am a little bit
concerned about the extra work it will add compared
to their benefits.

- Credit goes to all the contributors who work on maintaining Yetus and the
CI/CD for the current branches.
  Do we have any bandwidth to support the new GitHub webhooks?
- Just for discussion: is Integrating SonarQube worth the effort of
supporting the GitHub webhooks. SonarQube
   is a double edge weapon and it needs dedicated man hours to weed out the
way through the reports.

On Mon, Jul 12, 2021 at 11:18 PM Wei-Chiu Chuang  wrote:

> Here's another one:
>
> We can update the JIRA workflow and add more state. For example, Cassandra
> has "Review in Progress" and "Need Reviewer" states. INFRA-22049
> <https://issues.apache.org/jira/browse/INFRA-22049>
>
> On Tue, Jul 13, 2021 at 11:34 AM Wei-Chiu Chuang 
> wrote:
>
> > I work on multiple projects and learned a bunch from those projects.There
> > are nice add-ons that help with productivity. There are things we can do
> to
> > help us manage the project better.
> >
> > 1. Add new issue types.
> > We can add "Epic" jira type to organize a set of related jiras. This
> could
> > be easier to manage than using a regular JIRA and call it "umbrella".
> >
> > 2. GitHub Actions
> > I am seeing more projects moving to GitHub Actions for precommits. We
> > don't necessarily need to migrate off Jenkins, but there are nice add-ons
> > that can perform static analysis, catching potential issues. For example,
> > Ozone adds SonarQube to post-commit, and exports the report to
> SonarCloud.
> > Other add-ons are available to scan for docker images, vulnerabilities
> > scans.
> >
> > 3. JIRA security
> > It is possible to set up security level (public/private) in JIRA. This
> can
> > be used to track vulnerability issues and be made only visible to
> > committers. Example: INFRA-15258
> > <https://issues.apache.org/jira/browse/INFRA-15258>
> >
> > 4. New JIRA fields
> > It's possible to add new fields. For example, we can add a "Reviewer"
> > field, which could help improve the attention to issues.
> >
> > 5. Doc update
> > It is possible to set up automation such that the doc on the Hadoop
> > website is refreshed for every commit, providing the latest doc to the
> > public.
> >
> > 6. Webhook
> > It's possible to set up webhook such that every commit in GitHub sends a
> > notification to the ASF slack. It can be used for other kinds of
> > automation. Sky's the limit.
> >
> > Thoughts? What else can do we?
> >
>


-- 
Best Regards,

*Ahmed Hussein, PhD*

[jira] [Created] (HADOOP-17769) Upgrade JUnit to 4.13.2

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17769:
--

 Summary: Upgrade JUnit to 4.13.2
 Key: HADOOP-17769
 URL: https://issues.apache.org/jira/browse/HADOOP-17769
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


JUnit 4.13.1 has a bug that is reported in Junit 
[issue-1652|https://github.com/junit-team/junit4/issues/1652] _Timeout 
ThreadGroups should not be destroyed_

After upgrading Junit to 4.13.1 in HADOOP-17602, {{TestBlockRecovery}}  started 
to fail regularly in branch-3.x and branch-2.10.
While investigating the failure in branch-2.10 HDFS-16072, I found out that the 
bug is the main reason {{TestBlockRecovery}}  started to fail because the 
timeout of the Junit would try to close a ThreadGroup that has been already 
closed which throws the {{java.lang.IllegalThreadStateException}}.

The bug has been fixed in Junit-4.13.2

For branch-3.x, HDFS-15940 did not address the root cause of the problem. 
Eventually, Splitting the {{TestBlockRecovery}} hid the bug, but the upgrade 
needs to be done so that the problem does not show up in another unit test.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17762) branch-2.10 daily build fails to pull latest changes

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17762:
--

 Summary: branch-2.10 daily build fails to pull latest changes
 Key: HADOOP-17762
 URL: https://issues.apache.org/jira/browse/HADOOP-17762
 Project: Hadoop Common
  Issue Type: Bug
  Components: build, yetus
Affects Versions: 2.10.1
Reporter: Ahmed Hussein


I noticed that the build for branch-2.10 failed to pull the latest changes for 
the last few days.

CC: [~aajisaka], [~tasanuma], [~Jim_Brennan]

https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/329/console

{code:bash}
Started by timer
Running as SYSTEM
Building remotely on H20 (Hadoop) in workspace 
/home/jenkins/jenkins-home/workspace/hadoop-qbt-branch-2.10-java7-linux-x86_64
The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
Using shallow clone with depth 10
Avoid fetching tags
Cloning repository https://github.com/apache/hadoop
ERROR: Failed to clean the workspace
jenkins.util.io.CompositeIOException: Unable to delete 
'/home/jenkins/jenkins-home/workspace/hadoop-qbt-branch-2.10-java7-linux-x86_64/sourcedir'.
 Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 
1 additional exceptions)
at 
jenkins.util.io.PathRemover.forceRemoveDirectoryContents(PathRemover.java:90)
at hudson.Util.deleteContentsRecursive(Util.java:262)
at hudson.Util.deleteContentsRecursive(Util.java:251)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:743)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:375)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Suppressed: java.nio.file.AccessDeniedException: 
/home/jenkins/jenkins-home/workspace/hadoop-qbt-branch-2.10-java7-linux-x86_64/sourcedir/hadoop-hdfs-project/hadoop-hdfs/target/test/data/3/dfs/data/data1/current
at 
sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at 
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at 
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at 
sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427)
at java.nio.file.Files.newDirectoryStream(Files.java:457)
at 
jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:224)
at 
jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at 
jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at 
jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at 
jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at 
jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at 
jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at 
jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at 
jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at 
jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at 
jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at 
jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at 
jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at 
jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at 
jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at 
jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at 
jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
at 
jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
at 
jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226

[jira] [Resolved] (HADOOP-17463) Replace currentTimeMillis with monotonicNow in elapsed time

[Ahmed Hussein (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Hussein resolved HADOOP-17463.

Release Note: see discussion in HADOOP-15901
  Resolution: Won't Fix

> Replace currentTimeMillis with monotonicNow in elapsed time
> ---
>
> Key: HADOOP-17463
> URL: https://issues.apache.org/jira/browse/HADOOP-17463
> Project: Hadoop Common
>  Issue Type: Bug
>        Reporter: Ahmed Hussein
>    Assignee: Ahmed Hussein
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> I noticed that there is a widespread incorrect usage of 
> {{System.currentTimeMillis()}}  throughout the hadoop code.
> For example:
> {code:java}
> // Some comments here
> long start = System.currentTimeMillis();
> while (System.currentTimeMillis() - start < timeout) {
>   // Do something
> }
> {code}
> Elapsed time should be measured using `monotonicNow()`.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Java 8 Lambdas

[Ahmed Hussein]

Thanks Eric for raising this issue!

The debate about lambda is very complicated and won't be resolved any time
soon.

 I don't personally know a lot about the
> performance of lambdas and welcome arguments on behalf of why lambdas

No one probably knows :)
- Lambda performance would depend on the JVM implementation. This changes
between
releases.
- Java8+ features forces lambda. For example,
ConcurrentHashMap.computeIfAbsent()

I believe that we can transform this discussion into specific action items
for future commits:
For instance, a couple of those specifications would be:
- No refactor just for the sake of using Lambda, unless there is a strong
technical justification.
- Usage of lambda in Unit-tests should be fine. If lambda makes the test
more readable, and
  allows passing method references, then this should make the unit-tests.
- We put sample code in the "how-to-contribute" to elaborate "capturing Vs
non-capturing"
  lambda expressions and the implications of each type on the performance.
- Without getting into much detail, IMHO, streams should be committed into
the code
  in exceptional cases. The possibility of executing code in parallel makes
debugging
  a nightmare. i.e., Usage of ForEach needs to be justified, what does it
bring to the table?

On Tue, Apr 27, 2021 at 3:07 PM Eric Badger
 wrote:

> Hello all,
>
> I'd like to gauge the community on the usage of lambdas within Hadoop code.
> I've been reviewing a lot of patches recently that either add or modify
> lambdas and I'm beginning to think that sometimes we, as a community, are
> writing lambdas because we can rather than because we should. To me, it
> seems that lambdas often decrease the readability of the code, making it
> more difficult to understand. I don't personally know a lot about the
> performance of lambdas and welcome arguments on behalf of why lambdas
> should be used. An additional argument is that lambdas aren't available in
> Java 7, and branch-2.10 currently supports Java 7. So any code going back
> to branch-2.10 has to be redone upon backporting. Anyway, my main point
> here is to encourage us to rethink whether we should be using lambdas in
> any given circumstance just because we can.
>
> Eric
>
> p.s. I'm also happy to accept this as my personal "old man yells at cloud"
> issue if everyone else thinks lambdas are the greatest
>


-- 
Best Regards,

*Ahmed Hussein, PhD*

[jira] [Created] (HADOOP-17603) Upgrade tomcat-embed-core to 7.0.108

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17603:
--

 Summary: Upgrade tomcat-embed-core to 7.0.108
 Key: HADOOP-17603
 URL: https://issues.apache.org/jira/browse/HADOOP-17603
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


[CVE-2021-25329|https://nvd.nist.gov/vuln/detail/CVE-2021-25329] critical 
severity.
Impact: [CVE-2020-9494|https://nvd.nist.gov/vuln/detail/CVE-2020-9494]
7.0.0-7.0.107 are all affected by the vulnerability.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17602) Upgrade JUnit to 4.13.1

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17602:
--

 Summary: Upgrade JUnit to 4.13.1
 Key: HADOOP-17602
 URL: https://issues.apache.org/jira/browse/HADOOP-17602
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


A reported vulnerability reported in JUnit4.7-4.13.
The JUnit4 test rule [TemporaryFolder on unix-like systems does not limit 
access to created 
files|https://github.com/junit-team/junit4/security/advisories/GHSA-269g-pwp5-87pp]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17601) Upgrade Jackson databind in branch-2.10 to 2.9.10.6

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17601:
--

 Summary: Upgrade Jackson databind in branch-2.10 to 2.9.10.6
 Key: HADOOP-17601
 URL: https://issues.apache.org/jira/browse/HADOOP-17601
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


Two known vulnerabilities found in Jackson-databind

[CVE-2021-20190|https://nvd.nist.gov/vuln/detail/CVE-2021-20190] high severity
[CVE-2020-25649|https://nvd.nist.gov/vuln/detail/CVE-2020-25649] high severity



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17557) skip-dirs option is not processed by Yetus

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17557:
--

 Summary: skip-dirs option is not processed by Yetus
 Key: HADOOP-17557
 URL: https://issues.apache.org/jira/browse/HADOOP-17557
 Project: Hadoop Common
  Issue Type: Bug
  Components: build, precommit, yetus
Reporter: Ahmed Hussein


Running test patch locally does not work anymore after the Yetus upgrade


{code:bash}
dev-support/bin/test-patch --plugins="maven,checkstyle" --test-parallel=true 
patch-file.patch
{code}

Error is 

{code:bash}
Testing  patch on trunk.
ERROR: Unprocessed flag(s): --skip-dir

environment {
SOURCEDIR = 'src'
// will also need to change notification section below
PATCHDIR = 'out'
DOCKERFILE = "${SOURCEDIR}/dev-support/docker/Dockerfile"
YETUS='yetus'
// Branch or tag name.  Yetus release tags are 'rel/X.Y.Z'
YETUS_VERSION='rel/0.13.0'
/skip-
# URL for user-side presentation in reports and such to 
our artifacts

 _ _ __
|  ___|_ _(_) |_   _ _ __ ___| |
| |_ / _` | | | | | | '__/ _ \ |
|  _| (_| | | | |_| | | |  __/_|
|_|  \__,_|_|_|\__,_|_|  \___(_)



| Vote |Subsystem |  Runtime   | Comment

|  -1  |   yetus  |   0m 05s   | Unprocessed flag(s): --skip-dir
{code}

It seems that the "{{--skip-dir}}" option was never supported by any Yetus 
release.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Seeing YETUS issue in jenkins runs

[Ahmed Hussein]

I noticed that the qbt-reports were not generated for the last 2 days.
It is probably related to 16870. I filed a jira
https://issues.apache.org/jira/browse/HADOOP-17541


On Mon, Feb 22, 2021 at 1:26 PM Steve Loughran 
wrote:

> Akira,
>
> I see you've started fixing upstream OSS projects.
>
> take care: first it's spotbugs, next it'll be maven, before long you'll be
> doing linux device drivers...
>
>
>
> On Fri, 19 Feb 2021 at 05:53, Akira Ajisaka  wrote:
>
> > Probably this issue is due to
> > https://issues.apache.org/jira/browse/HADOOP-16870. Reverted.
> >
> > -Akira
> >
> > On Fri, Feb 19, 2021 at 11:27 AM Sunil Govindan 
> wrote:
> > >
> > > Hi All,
> > >
> > > We are getting the below error in recent runs.
> > >
> > > -1.  yetus.   0m 8s.   Unprocessed flag(s): --findbugs-strict-precheck
> > >
> > > https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/636/console
> > >
> > > Are there any knows issues or some workarounds?
> > > Please advise.
> > >
> > > Thanks
> > > Sunil
> > >
> >
> > ---------
> > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> >
> >
>


-- 
Best Regards,

*Ahmed Hussein, PhD*

[jira] [Created] (HADOOP-17541) Yetus does not run qbt-trunk

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17541:
--

 Summary: Yetus does not run qbt-trunk
 Key: HADOOP-17541
 URL: https://issues.apache.org/jira/browse/HADOOP-17541
 Project: Hadoop Common
  Issue Type: Bug
  Components: bin, build, yetus
Reporter: Ahmed Hussein


On Feb20th, qbt-reports started to generate empty reports

{code:bash}
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/424/
ERROR: File 'out/email-report.txt' does not exist
{code}

On Jenkins, the job fails with the following error:
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/425/console
{code:bash}
ERROR: 
/home/jenkins/jenkins-home/workspace/hadoop-qbt-trunk-java8-linux-x86_64//dev-support/bin/hadoop.sh
 does not exist.
Build step 'Execute shell' marked build as failure
Archiving artifacts
[Fast Archiver] No prior successful build to compare, so performing full copy 
of artifacts
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?
{code}

[~aajisaka], I think this would be caused by HADOOP-16748 . I noticed that the 
PR of that HADOOP-16748 ceased from showing any reports, but for some reason I 
forgot about that while reviewing.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17513) Checkstyle IllegalImport does not catch guava imports

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17513:
--

 Summary: Checkstyle IllegalImport does not catch guava imports
 Key: HADOOP-17513
 URL: https://issues.apache.org/jira/browse/HADOOP-17513
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


Although YARN-10352 introduces {{guava iterator import}}, it was committed to 
trunk without checkstyle errors.

According to [IllegalImportCheck#setIllegalPkgs 
|https://github.com/checkstyle/checkstyle/blob/master/src/main/java/com/puppycrawl/tools/checkstyle/checks/imports/IllegalImportCheck.java],
 the packages regex should be the prefix of the package. The code automatically 
append {{\.*}} to the regex.

CC: [~aajisaka]
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17488) Race condition refreshing NetgroupCache

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17488:
--

 Summary: Race condition refreshing NetgroupCache 
 Key: HADOOP-17488
 URL: https://issues.apache.org/jira/browse/HADOOP-17488
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


There is potential concurrency bug in the {{NetgroupCache}} implementation.

{{NetgroupCache}} is static. When ACL is built, its groups will be added to the 
{{NetgroupCache}}.

A {{-refreshUserToGroupsMappings}} forces the cache to reload the users for 
each group.
 This is done by first getting the keys, clearing the cache, then finally 
reloading the users for each group.
 The problem that the three steps are not atomic.
 Adding ACLs concurrently may take place between L80-L81 
([JniBasedUnixGroupsNetgroupMapping#L79|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.java#L79]).
 This results in the loss of the most recently added group.
 Since group names are used in the JNI level, the users of that group won't be 
retrieved.
{code:java}
78 @Override
79  public void cacheGroupsRefresh() throws IOException {
80List groups = NetgroupCache.getNetgroupNames();
81 NetgroupCache.clear();
82cacheGroupsAdd(groups);
83  }
{code}
+Solution:+

Refreshing {{NetgroupCache}} should not clear the cache keys.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17487) Replace deprecated getGroups with getGroupsSet

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17487:
--

 Summary: Replace deprecated getGroups with getGroupsSet
 Key: HADOOP-17487
 URL: https://issues.apache.org/jira/browse/HADOOP-17487
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.10.1, 3.4.0
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


This ticket is a follow up to HADOOP-17079 which added an optimization to use 
sets instead of lists retrieving user groups.
However, the deprecated getGroups is still being called throughout the code.
This is an attempt to clean up the code.

CC: [~xyao], [~Jim_Brennan]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17485) port UGI#getGroupsSet optimizations into 2.10

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17485:
--

 Summary: port UGI#getGroupsSet optimizations into 2.10
 Key: HADOOP-17485
 URL: https://issues.apache.org/jira/browse/HADOOP-17485
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


HADOOP-17079 introduced an optimization adding a UGI#getGroupsSet and use 
Set#contains() instead of List#contains() to speed up large group look up while 
minimize List->Set conversions in Groups#getGroups() call.

This ticket is to port the changes into branch-2.10.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17481) Groups user mapping based on Netgroup do not auto-refresh

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17481:
--

 Summary: Groups user mapping based on Netgroup do not auto-refresh 
 Key: HADOOP-17481
 URL: https://issues.apache.org/jira/browse/HADOOP-17481
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


It was noticed that a manual refresh ({{yarn rmadmin 
-refreshUserToGroupsMappings}}) is required for services to pick a user-group 
membership.
The auto-refresh of the {{Groups.caching}} only works for the upper layer and 
won't fetch the groups from the system.

There is a need to automatically reload all the groups (currently in 
{{NetgroupCache}}). The interval can be a new configuration, with a default 
value of 6 hours.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17467) netgroup-user mapping breaks by a race in NetgroupCache

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17467:
--

 Summary: netgroup-user mapping breaks by a race in NetgroupCache 
 Key: HADOOP-17467
 URL: https://issues.apache.org/jira/browse/HADOOP-17467
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


There is potential concurrency bug in the {{NetgroupCache}} implementation.

{{NetgroupCache}} is static. When ACL is built, its groups will be added to the 
{{NetgroupCache}}.

A {{-refreshUserToGroupsMappings}} forces the cache to reload the users for 
each group.
This is done by first getting the keys, clearing the cache, then finally 
reloading the users for each group.
The problem that the three steps are not atomic.
Adding ACLs concurrently may take place between L80-L81 
([JniBasedUnixGroupsNetgroupMapping#L79|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.java#L79]).
 This results in the loss of the most recently added group.
Since group names are used in the JNI level, the users of that group won't be 
retrieved.

{code:java}
78 @Override
79  public void cacheGroupsRefresh() throws IOException {
80List groups = NetgroupCache.getNetgroupNames();
81 NetgroupCache.clear();
82cacheGroupsAdd(groups);
83  }
{code}

+Solution:+

Refreshing {{NetgroupCache}} should not clear the cache keys. 




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17463) Replace currentTimeMillis with monotonicNow in elapsed time

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17463:
--

 Summary: Replace currentTimeMillis with monotonicNow in elapsed 
time
 Key: HADOOP-17463
 URL: https://issues.apache.org/jira/browse/HADOOP-17463
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


I noticed that there is a widespread incorrect usage of 
{{System.currentTimeMillis()}}  throughout the hadoop code.

For example:

{code:java}
// Some comments here
long start = System.currentTimeMillis();
while (System.currentTimeMillis() - start < timeout) {
  // Do something
}
{code}

Elapsed time should be measured using `monotonicNow()`.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17449) Jetty 9.4.20 can't generate resourceBase with NPE

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17449:
--

 Summary: Jetty 9.4.20 can't generate resourceBase with NPE
 Key: HADOOP-17449
 URL: https://issues.apache.org/jira/browse/HADOOP-17449
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ahmed Hussein


While I was looking into TestDistributedShell logs, I noticed the following 
{{Warning}}


{code:bash}
2020-12-29 16:22:26,379 INFO  [Time-limited test] handler.ContextHandler 
(ContextHandler.java:doStart(824)) - Started 
o.e.j.s.ServletContextHandler@75389179{logs,/logs,file:///hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/target/log,AVAILABLE}
2020-12-29 16:22:26,380 INFO  [Time-limited test] handler.ContextHandler 
(ContextHandler.java:doStart(824)) - Started 
o.e.j.s.ServletContextHandler@116ed75c{static,/static,jar:file:~/.m2/repository/org/apache/hadoop/hadoop-yarn-common/3.4.0-SNAPSHOT/hadoop-yarn-common-3.4.0-SNAPSHOT.jar!/webapps/static,AVAILABLE}
2020-12-29 16:22:26,390 WARN  [Time-limited test] webapp.WebInfConfiguration 
(WebInfConfiguration.java:getCanonicalNameForWebAppTmpDir(794)) - Can't 
generate resourceBase as part of webapp tmp dir name: 
java.lang.NullPointerException
2020-12-29 16:22:26,469 INFO  [Time-limited test] util.TypeUtil 
(TypeUtil.java:(201)) - JVM Runtime does not support Modules
{code}

For OS X, it looks like {{webAppContext.setBaseResource}} and accessing the 
sources from a jar file will cause {{file.resource.toURI().getPath()}} to 
return {{null}} for {{jar:-urls}}

I checked that changing the jetty-version  from {{9.4.20.v20190813}} to 
something above {{9.4.21}} (aka., 9.4.23.v20191118) fixes the warning.

[~inigoiri], [~aajisaka], [~weichiu], [~ayushtkn]
Do you guys think we should consider upgrading Jetty to the [latest versions of 
9.4.x|https://mvnrepository.com/artifact/org.eclipse.jetty/jetty-webapp] like 
9.4.35?





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [E] Yetus is failing with Java unable to create threads

[Ahmed Hussein]

> In most cases, the OOM occurs when closing MiniDFSCluster.

@Akira Ajisaka  , I see some usage of parallelStream()
in RouterRPCServer.
If we consider pool executors in the code and parallelStream(), could it be
possible that there
is a large number of threads created in the MiniCluster and
MiniRouterCluster that causes the JVM to crash?

On Thu, Dec 17, 2020 at 9:13 PM Akira Ajisaka  wrote:

> Thank you for your reply.
>
> > Will reducing the number of threads not increase the build time?
> Yes, but the difference is 30 ~ 60 mins. Not so much, I think.
>
> > Can we not ask for more resources?
> Now the machines in https://ci-hadoop.apache.org/ are physical, and
> the memory size is fixed.
> (They are donated from Y!.
>
> https://cwiki.apache.org/confluence/display/INFRA/Build+nodes+-+node+name+to+hostname+mappings
> )
>
> I'll ask the infrastructure team how much memory we can use. If the
> size is not 20GB, we can update in
> https://github.com/apache/hadoop/pull/2560
>
> > I think RBF builds are quite stable
> Actually not:
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/357/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
> I attached the log because it will be deleted in 2 weeks.
>
> -Akira
>
> On Thu, Dec 17, 2020 at 7:19 PM Ayush Saxena  wrote:
> >
> > Hi Akira,
> > Will reducing the number of threads not increase the build time? I guess
> it takes in general 2.5-3.5 hrs in the present scenario. Moreover the
> thread count hasn’t been increased recently, Would that be the root of all
> evil?
> >
> > Can we not ask for more resources?
> >
> > Anyway, I think RBF builds are quite stable, I don’t remember seeing OOM
> there, so, in case we decide to reduce the thread count, may be we can keep
> RBF as is?
> >
> > -Ayush
> >
> > > On 17-Dec-2020, at 2:15 PM, Akira Ajisaka  wrote:
> > >
> > > Sorry, now I think the above comment is wrong. Please ignore.
> > > In hadoop-common, hadoop-hdfs, and hadoop-hdfs-rbf, the unit tests are
> > > executed in parallel. I'd like to reduce the number of tests running
> > > at the same time to avoid OOM. Filed
> > > https://issues.apache.org/jira/browse/HDFS-15731
> > >
> > >> On Thu, Dec 17, 2020 at 4:17 PM Akira Ajisaka 
> wrote:
> > >>
> > >> In most cases, the OOM occurs when closing MiniDFSCluster.
> > >> Added a detailed comment in
> > >> https://issues.apache.org/jira/browse/HDFS-13579 and created a PR:
> > >> https://github.com/apache/hadoop/pull/2555
> > >>
> > >> -Akira
> > >>
> > >>> On Fri, Dec 4, 2020 at 12:43 AM Ahmed Hussein  wrote:
> > >>>
> > >>> I remember this error was there for more than 6 months. It
> significantly
> > >>> slows down the progress of collaboration.
> > >>> Then, eventually, the community will develop another habit of
> ignoring the
> > >>> prebuilds (out of despair).
> > >>>
> > >>> I am willing to help to get this fixed.
> > >>> Anyone knows who owns and has experience with Yetus environment?
> > >>>
> > >>> On Wed, Dec 2, 2020 at 4:43 PM Jim Brennan <
> james.bren...@verizonmedia.com>
> > >>> wrote:
> > >>>
> > >>>> This is still happening.
> > >>>> Latest build:
> > >>>>
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/343/#showFailuresLink
> > >>>>
> > >>>> Looks like we are running out of threads in the containers where
> the unit
> > >>>> tests run.  Anyone know where this is setup?
> > >>>>
> > >>>> On Wed, Oct 21, 2020 at 5:51 PM Ahmed Hussein 
> wrote:
> > >>>>
> > >>>>> Hey folks,
> > >>>>>
> > >>>>> Yetus has been failing miserably over the last couple of days.
> > >>>>> In the Lastest qbt-report
> > >>>>> <
> > >>>>>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__ci-2Dhadoop.apache.org_job_hadoop-2Dqbt-2Dtrunk-2Djava8-2Dlinux-2Dx86-5F64_301_artifact_out_patch-2Dunit-2Dhadoop-2Dhdfs-2Dproject-5Fhadoop-2Dhdfs.txt=DwIBaQ=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY=7Imi06B91L3gbxmt5ChzH4cwlA2_f2tmXh3OXmV9MLw=HChJ3ymJ0kOlFyiTBsyRZLs9qcTOQD864ZFb8g7y2CA=N-PB427UiouJCuX_U3UbUXvIh2HQTt7VdM2Bs_4XILI=
> > >>>>>> ,
> > >>>>> hundreds of J

[jira] [Created] (HADOOP-17438) Increase docker memory limit in Jenkins

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17438:
--

 Summary: Increase docker memory limit in Jenkins
 Key: HADOOP-17438
 URL: https://issues.apache.org/jira/browse/HADOOP-17438
 Project: Hadoop Common
  Issue Type: Bug
  Components: build, scripts, test, yetus
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


Yetus keeps failing with OOM.

 
{code:bash}
unable to create new native thread

java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at 
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
at 
java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1603)
at 
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:334)
at 
java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
at 
org.apache.maven.surefire.booter.ForkedBooter.launchLastDitchDaemonShutdownThread(ForkedBooter.java:369)
at 
org.apache.maven.surefire.booter.ForkedBooter.acknowledgedExit(ForkedBooter.java:333)
at 
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:145)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
{code}
 

This jira to increase the memory limit from 20g to 22g.

*Note: This is only a workaround to get things more productive. If this change 
reduces the frequency of the OOM failure, there must be a follow-up profile the 
runtime to figure out which components are causing the docker to run out of 
memory.*

CC: [~aajisaka], [~elgoiri], [~weichiu], [~ebadger], [~tasanuma], [~iwasakims], 
[~ayushtkn], [~inigoiri]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17417) Reduce UGI overhead in token ops

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17417:
--

 Summary: Reduce UGI overhead in token ops 
 Key: HADOOP-17417
 URL: https://issues.apache.org/jira/browse/HADOOP-17417
 Project: Hadoop Common
  Issue Type: Improvement
  Components: common, kms, performance, rpc-server, security
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


{{DelegationTokenIdentifier}} has a {{ugiCache}} but  
AbstractDelegationTokenManager calls a static method {{getRemoteUser()}} which 
would bypass the cache.

Performance analysis of the KMS revealed the RPC server layer is creating many 
and redundant UGI instances.. UGIs are not cheap to instantiate, require 
synchronization, and waste memory. Reducing instantiations will improve the 
performance of the ipc readers.



 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17416) Reduce synchronization in the token secret manager

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17416:
--

 Summary: Reduce synchronization in the token secret manager
 Key: HADOOP-17416
 URL: https://issues.apache.org/jira/browse/HADOOP-17416
 Project: Hadoop Common
  Issue Type: Improvement
  Components: common, performance, security
Reporter: Ahmed Hussein


Reducing synchronization in the ZK secret manager is complicated by excessive 
and unnecessary global synchronization in the 
AbstractDelegationTokenSecretManager.  All RPC services, not just the KMS, will 
benefit from the reduced synchronization.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [E] Yetus is failing with Java unable to create threads

[Ahmed Hussein]

I remember this error was there for more than 6 months. It significantly
slows down the progress of collaboration.
Then, eventually, the community will develop another habit of ignoring the
prebuilds (out of despair).

I am willing to help to get this fixed.
Anyone knows who owns and has experience with Yetus environment?

On Wed, Dec 2, 2020 at 4:43 PM Jim Brennan 
wrote:

> This is still happening.
> Latest build:
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/343/#showFailuresLink
>
> Looks like we are running out of threads in the containers where the unit
> tests run.  Anyone know where this is setup?
>
> On Wed, Oct 21, 2020 at 5:51 PM Ahmed Hussein  wrote:
>
>> Hey folks,
>>
>> Yetus has been failing miserably over the last couple of days.
>> In the Lastest qbt-report
>> <
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__ci-2Dhadoop.apache.org_job_hadoop-2Dqbt-2Dtrunk-2Djava8-2Dlinux-2Dx86-5F64_301_artifact_out_patch-2Dunit-2Dhadoop-2Dhdfs-2Dproject-5Fhadoop-2Dhdfs.txt=DwIBaQ=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY=7Imi06B91L3gbxmt5ChzH4cwlA2_f2tmXh3OXmV9MLw=HChJ3ymJ0kOlFyiTBsyRZLs9qcTOQD864ZFb8g7y2CA=N-PB427UiouJCuX_U3UbUXvIh2HQTt7VdM2Bs_4XILI=
>> >,
>> hundreds of Junits fail after java failed to acquire resources
>> to create new threads.
>>
>> [ERROR]
>> >
>> testRecoverAllDataBlocks1(org.apache.hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy)
>> >  Time elapsed: 8.509 s  <<< ERROR!
>> > java.lang.OutOfMemoryError: unable to create new native thread
>>
>>
>> Any thoughts on what could trigger that in the last few days? Do we need
>> more resources for the image?
>>
>> --
>> Best Regards,
>>
>> *Ahmed Hussein, PhD*
>>
>

-- 
Best Regards,

*Ahmed Hussein, PhD*

[jira] [Created] (HADOOP-17408) Optimize NetworkTopology while sorting of block locations

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17408:
--

 Summary: Optimize NetworkTopology while sorting of block locations
 Key: HADOOP-17408
 URL: https://issues.apache.org/jira/browse/HADOOP-17408
 Project: Hadoop Common
  Issue Type: Improvement
  Components: common, net
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


In {{NetworkTopology}}, I noticed that there are some hanging fruits to improve 
the performance.

Inside {{sortByDistance}}, collections.shuffle is performed on the list before 
calling {{secondarySort}}.

{code:java}
Collections.shuffle(list, r);
if (secondarySort != null) {
  secondarySort.accept(list);
}
{code}

However, in different call sites, {{collections.shuffle}} is passed as the 
secondarySort to {{sortByDistance}}. This means that the shuffle is executed 
twice on each list.
Also, logic wise, it is useless to shuffle before applying a tie breaker which 
might make the shuffle work obsolete.

In addition, [~daryn] reported that:
* topology is unnecessarily locking/unlocking to calculate the distance for 
every node
* shuffling uses a seeded Random, instead of ThreadLocalRandom, which is 
heavily synchronized




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17393) Avoid calling UpdateHeartBeatState inside DataNodeDescriptor

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17393:
--

 Summary: Avoid calling UpdateHeartBeatState inside 
DataNodeDescriptor
 Key: HADOOP-17393
 URL: https://issues.apache.org/jira/browse/HADOOP-17393
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


[~kshukla] reported that {{DataNodeDescriptor}} constructor calls 
{{updateHeartBeat}} which is spamming the NN logs. This call does not update 
much since all the fields for the call are null or 0.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17392) Remote exception messages should not include the exception class

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17392:
--

 Summary: Remote exception messages should not include the 
exception class
 Key: HADOOP-17392
 URL: https://issues.apache.org/jira/browse/HADOOP-17392
 Project: Hadoop Common
  Issue Type: Improvement
  Components: ipc
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


HADOOP-9844 an unnecessary change that accidentally causes some remote SASL 
exceptions to redundantly include the exception class causing the client to see 
"`Class: Class: message`" from an unwrapped RemoteException.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17389) KMS should log full UGI principal

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17389:
--

 Summary: KMS should log full UGI principal
 Key: HADOOP-17389
 URL: https://issues.apache.org/jira/browse/HADOOP-17389
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


[~daryn] reported that the kms-audit log only logs the short username:
{{OK[op=GENERATE_EEK, key=key1, user=hdfs, accessCount=4206, interval=10427ms]}}

In this example, it's impossible to tell which NN(s) requested EDEKs when they 
are all lumped together.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17368) Zookeeper secret manager attempts to reuse token sequence numbers

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17368:
--

 Summary: Zookeeper secret manager attempts to reuse token sequence 
numbers
 Key: HADOOP-17368
 URL: https://issues.apache.org/jira/browse/HADOOP-17368
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


[~daryn] reported that the ZK delegation token secret manager uses a 
{{SharedCounter}} to synchronize increments of a monotonically increasing 
sequence number for new tokens. Yet the KMS logs occasionally, depending on 
load, contains an odd error indicating collisions: 


{code:bash}
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = 
NodeExists for /zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_137547444
{code}


ZKDTSM does a CAS get and set of the sequence number. Rather than return the 
value it set, it returns the current value which may have already been 
incremented by another KMS.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17367) Improve TLS/SSL default settings for security and performance

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17367:
--

 Summary: Improve TLS/SSL default settings for security and 
performance
 Key: HADOOP-17367
 URL: https://issues.apache.org/jira/browse/HADOOP-17367
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


[~kihwal] reported that {{HttpServer2}} is still accepting TLS 1.1 or 1.0. 
These are only rejected when the java security setting excludes them. The 
expensive algorithms arte still being used.


{code:bash}
main, WRITE: TLSv1.2 Handshake, length = 239
main, READ: TLSv1.2 Handshake, length = 1508
*** ServerHello, TLSv1.2
...
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
{code}

SSLFactory calls {{sslEngine.setEnabledCipherSuites()}} to set enabled ciphers.
Apparently this does not disable unincluded ciphers, so SSLFactory's cipher 
disabling feature does not work. Or it could be jetty's undoing.

Jetty9 introduced SSLContextFactory. Following methods can be used.

{code:java}
setExcludeCipherSuites()
setExcludeProtocols()
setIncludeCipherSuites()
setIncludeProtocols()
{code}

SSLFactory is not used by HttpServer2. It is only used by 
{{DatanodeHttpServer}} and {{ShuffleHandler}}. The reloading feature is also 
broken for the same reason.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Intervention. Stabilizing Yetus (Attn. Azure)

[Ahmed Hussein]

NativeAzureFileSystemMocked/testRedoRenameFolder/>
   -
   
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testStoreDeleteFolder
   
<https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testStoreDeleteFolder/>
   - org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRename
   
<https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemMocked/testRename/>
   -
   
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testListStatus
   
<https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemOperationsMocked/testListStatus/>
   -
   
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testRenameDirectoryAsEmptyDirectory
   
<https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemOperationsMocked/testRenameDirectoryAsEmptyDirectory/>
   -
   
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testListStatusFilterWithSomeMatches
   
<https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemOperationsMocked/testListStatusFilterWithSomeMatches/>
   -
   
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testRenameDirectoryAsNonExistentDirectory
   
<https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemOperationsMocked/testRenameDirectoryAsNonExistentDirectory/>
   -
   
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testGlobStatusSomeMatchesInDirectories
   
<https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemOperationsMocked/testGlobStatusSomeMatchesInDirectories/>
   -
   
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testGlobStatusWithMultipleWildCardMatches
   
<https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemOperationsMocked/testGlobStatusWithMultipleWildCardMatches/>
   -
   
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testDeleteRecursively
   
<https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemOperationsMocked/testDeleteRecursively/>
   -
   
org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testImplicitFolderDeleted
   
<https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestOutOfBandAzureBlobOperations/testImplicitFolderDeleted/>
   -
   
org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testFileAndImplicitFolderSameName
   
<https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestOutOfBandAzureBlobOperations/testFileAndImplicitFolderSameName/>
   -
   
org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testSetOwnerOnImplicitFolder
   
<https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestOutOfBandAzureBlobOperations/testSetOwnerOnImplicitFolder/>
   -
   
org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testFileInImplicitFolderDeleted
   
<https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestOutOfBandAzureBlobOperations/testFileInImplicitFolderDeleted/>
   -
   
org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testImplicitFolderListed
   
<https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestOutOfBandAzureBlobOperations/testImplicitFolderListed/>
   -
   
org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testCreatingDeepFileCreatesExplicitFolder
   
<https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestOutOfBandAzureBlobOperations/testCreatingDeepFileCreatesExplicitFolder/>
   -
   
org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testSetPermissionOnImplicitFolder
   
<https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestOutOfBandAzureBlobOperations/testSetPermissionOnImplicitFolder/>
   - org.apache.hadoop.fs.azure.TestWasbFsck.testDelete
   
<https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x

Intervention. Stabilizing Yetus (Attn. Azure)

[Ahmed Hussein]

Hello folks,

Over the last month, there has been concern about the stability of Hadoop.
Looking at the latest QBT report (Nov 8th, 2020 1:39 AM)
<https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/#showFailuresLink>,
there were 50 failing tests, 41 of which are in "hadoop-azure" module.
Thanks to the effort of the community, the yetus qbt report looks better by
miles. However, it will be highly appreciated if some developers volunteer
some time to take a look at the hadoop-azure.

If tests in fs.azure are irrelevant to active contributors, then please
consider disabling those tests to save resources and avoid side effects of
those failures on the other modules (busy CPUs, memory release, ports
listening..etc).

Thank you.

-- 
Best Regards,

*Ahmed Hussein, PhD*

[jira] [Resolved] (HADOOP-17361) TestLdapGroupsMapping is failing in trunk

[Ahmed Hussein (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Hussein resolved HADOOP-17361.

Release Note: duplicate of HADOOP-17340
  Resolution: Duplicate

> TestLdapGroupsMapping is failing in trunk
> -
>
> Key: HADOOP-17361
> URL: https://issues.apache.org/jira/browse/HADOOP-17361
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Ayush Saxena
>Priority: Major
>
>  
> [https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/317/testReport/junit/org.apache.hadoop.security/TestLdapGroupsMapping/testLdapReadTimeout/]
>  
> [https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/317/testReport/junit/org.apache.hadoop.security/TestLdapGroupsMapping/testLdapConnectionTimeout/]
>  
> The tests are failing in open-jdk due to change in exception message(One 
> space character). The tests passes in oracle java(for me)
>  
> {noformat}
>  Expected to find 'LDAP response read timed out, timeout used:3000ms' but got 
> unexpected exception: javax.naming.NamingException: LDAP response read timed 
> out, timeout used: 3000 ms.{noformat}
>  
>  
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17362) Doing hadoop ls on Har file triggers too many RPC calls

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17362:
--

 Summary: Doing hadoop ls on Har file triggers too many RPC calls
 Key: HADOOP-17362
 URL: https://issues.apache.org/jira/browse/HADOOP-17362
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


[~daryn] has noticed that Invoking hadoop ls on HAR is taking too much of time.

The har system has multiple deficiencies that significantly impacted 
performance:

# Parsing the master index references ranges within the archive index. Each 
range required re-opening the hdfs input stream and seeking to the same 
location where it previously stopped.
# Listing a har stats the archive index for every "directory". The per-call 
cache used a unique key for each stat, rendering the cache useless and 
significantly increasing memory pressure.
# Determining the children of a directory scans the entire archive contents and 
filters out children. The cached metadata already stores the exact child list.
# Globbing a har's contents resulted in unnecessary stats for every leaf path.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17360) Log the remote address for authentication success

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17360:
--

 Summary: Log the remote address for authentication success
 Key: HADOOP-17360
 URL: https://issues.apache.org/jira/browse/HADOOP-17360
 Project: Hadoop Common
  Issue Type: Bug
  Components: ipc
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


IPC server logs "Authentication successful for USER". 
Unlike the hdfs audit log that includes the address with every request, the kms 
audit log is 10s aggregate per-user op counts.  Including the remote address 
would make debugging much easier.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-17224) Install Intel ISA-L library in Dockerfile

[Ahmed Hussein (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-17224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Hussein reopened HADOOP-17224:


> Install Intel ISA-L library in Dockerfile
> -
>
> Key: HADOOP-17224
> URL: https://issues.apache.org/jira/browse/HADOOP-17224
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Fix For: 3.4.0
>
>
> Currently, there is not isa-l library in the docker container, and jenkins 
> skips the natvie tests, TestNativeRSRawCoder and TestNativeXORRawCoder.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17358) Improve excessive reloading of Configurations

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17358:
--

 Summary: Improve excessive reloading of Configurations
 Key: HADOOP-17358
 URL: https://issues.apache.org/jira/browse/HADOOP-17358
 Project: Hadoop Common
  Issue Type: Bug
  Components: conf
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


[~daryn] reported that adding a new resource to a conf forces a complete reload 
of the conf instead of just loading the new resource. Instantiating a 
{{SSLFactory}} adds a new resource for the ssl client/server file. Formerly 
only the KMS client used the SSLFactory but now TLS/RPC uses it too.

The reload is so costly that RM token cancellation falls behind by hours or 
days. The accumulation of uncancelled tokens in the KMS rose from a few 
thousand to hundreds of thousands which risks ZK scalability issues causing a 
KMS outage.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17346) Fair call queue is defeated by abusive service principals

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17346:
--

 Summary: Fair call queue is defeated by abusive service principals
 Key: HADOOP-17346
 URL: https://issues.apache.org/jira/browse/HADOOP-17346
 Project: Hadoop Common
  Issue Type: Bug
  Components: common, ipc
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


[~daryn] reported  that the FCQ prioritizes based on the full kerberos 
principal (ie. "user/host@realm") rather than short name (ie. "user") to 
prevent service principals like the DNs and NMs being de-prioritized since 
service principals are expected to be well behaved.  Notably the DNs contribute 
a significant but important load so the intent is not to de-prioritize all DNs 
because their sum total load is high relative to users.

This has the unfortunate side effect of allowing misbehaving & non-critical 
service principals to abuse the FCQ. The gstorm/* principals are a prime 
example.   Each server is spamming opens as fast as possible which ensures that 
none of the gstorm servers can be de-prioritized because each principal is a 
fraction of the total load from all principals.

The secondary and more devasting problem is other abusive non-service 
principals cannot be effectively de-prioritized.  The sum total of all gstorm 
load prevents other principals from surpassing the priority thresholds.  
Principals stay in the highest priority queues which allows the abusive 
principals to overflow the entire call queue for extended periods of time.  
Notably it prevents the FCQ from moderating the heavy create loads from p_gup @ 
DB which cause significant performance degradation.

Prioritization should be based on short name with configurable exemptions for 
services like the DN/NM.

[~daryn] suggested a solution that we applied on our clusters.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Attach logs/stack trace on Jira

[Ahmed Hussein]

Hey,

I found that there is a significant number of opened jiras that use "links"
to yetus builds, or jenkins console output.
Keep in mind that those links expire in a few days and the information
becomes
unavailable to anyone.
Please, make sure that you put the stack trace and a log file along with
the description.

Thank you!

-- 
Best Regards,

*Ahmed Hussein, PhD*

Re: Fixing flaky tests in Apache Hadoop

[Ahmed Hussein]

>
> 2. Dedicate a community "Bug Bash Day" / "Fix it Day". We had a bug bash
> day two years ago, and maybe it's time to repeat it again:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75965105
>  this
> is going to be tricky as we are in a pandemic and most of the community are
> working from home, unlike the last time when we can lock ourselves in a
> conference room and force everybody to work :)


How about the following idea:

We set a monthly window during which only Unit test fixes can be merged.
Any other commit that is not directly
linked to Junit test failures would be blocked until the end of this
"Bug-Window".
For example, we set "Bug-days" to be from 25th to 31st of each month. All
commits during those days are meant to
fix and improve the testing environment.

Any thoughts?

On Thu, Oct 22, 2020 at 11:53 PM Ahmed Hussein  wrote:

> Thank you Akira and We-Chiu.
> IMHO, the citation is more than just flaky tests. It has more depth:
> - Every developer stays committed to keep the code healthy.
> - Those flaky tests are actually "*bugs*" that need to be fixed. It is
> evident that there is a major problem in handling
>   the resources as I will explain below.
>
> 1. Other projects such as HBase have a tool to exclude flaky tests from
>> being executed. They track flaky tests and display them in a dashboard.
>> This will allow good tests to pass while leaving time for folks to fix
>> them. Or we could manually exclude tests (this is what we used to do at
>> Cloudera)
>>
>
> I like the idea of having a tool that gives a view of broken tests.
>
>  I spent a long time converting HDFS flaky tests into sub-tasks under
> HDFS-15646 <https://issues.apache.org/jira/browse/HDFS-15646>. I
> believe there are still tons
> on the loose.
> I remember I explored a tool called DeFlaker
> <https://www.jonbell.net/icse18-deflaker.pdf> which detects flaky tests.
> Then it reruns the tests to verify that they still
> pass.
>
> I do not think we necessarily want to exclude the flaky tests, but at
> least they should be enumerated and addressed
> regularly because they are after all "bugs". Having few flaky tests that
> cause everything to blow up indicates
> that there is a major problem with handling resources.
> I pointed out this issue in YARN-10334
> <https://issues.apache.org/jira/browse/YARN-10334> where I found that
> TestDistributedShell is nothing but a black hole that sucks
> all the resources memory/cpu/port of resources.
> Another example, I ran some few Unit tests on my local machine. In less
> than an hour, I found that there are 6 java
> processes still listening to ports.
>
> The point is flaky tests should not be *undermined* for such a long time
> as they could be indicators of a serious bug.
> In this current situation, we should find what is eating all those
> resources.
>
> 2. Dedicate a community "Bug Bash Day" / "Fix it Day". We had a bug bash
>> day two years ago, and maybe it's time to repeat it again:
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75965105
>>  this
>> is going to be tricky as we are in a pandemic and most of the community
>> are
>> working from home, unlike the last time when we can lock ourselves in a
>> conference room and force everybody to work :)
>>
> This sounds fun and I like it actually but I doubt it is feasible to apply
> :)
>
> I also wondered if the hardware was too stressed since all Hadoop related
>> projects all use the same set of Jenkins servers.
>> However, HBase just recently moved to their own dedicated machines, so I'm
>> actually surprised to see a lot of resource related failures even now.
>>
> As I mentioned in my response on the first point, a black-hole is created
> once the tests are triggered.
> I could not even run TestDistibutedShell on my local machine. The tests
> run out of everything  after the first 11 unit tests.
> It takes only 1 unit to fail to break the rest.
>
> On Thu, Oct 22, 2020 at 5:28 PM Wei-Chiu Chuang 
> wrote:
>
>> I also wondered if the hardware was too stressed since all Hadoop related
>> projects all use the same set of Jenkins servers.
>> However, HBase just recently moved to their own dedicated machines, so I'm
>> actually surprised to see a lot of resource related failures even now.
>>
>> On Thu, Oct 22, 2020 at 2:03 PM Wei-Chiu Chuang 
>> wrote:
>>
>> > Thanks for raising the issue, Akira and Ahmed,
>> >
>> > Fixing flaky tests is a thankless job so I want to take this opportunity
>> > to recognize the time and effort.
&

Re: Fixing flaky tests in Apache Hadoop

[Ahmed Hussein]

t; >>
> >> Now there are a lot of failing unit tests and there is an issue to
> >> tackle this bad situation.
> >> https://issues.apache.org/jira/browse/HDFS-15646
> >>
> >> Although this issue is in HDFS project, this issue is related to all
> >> the Hadoop developers. Please check the above URL, read the
> >> description, and volunteer to dedicate more time to fix flaky tests.
> >> Your contribution to fixing the flaky tests will be really
> >> appreciated!
> >>
> >> Thank you Ahmed Hussein for your report.
> >>
> >> Regards,
> >> Akira
> >>
> >> -
> >> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> >> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> >>
> >>
>


-- 
Best Regards,

*Ahmed Hussein, PhD*

[jira] [Resolved] (HADOOP-16780) Track unstable tests according to aarch CI due to OOM

[Ahmed Hussein (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Hussein resolved HADOOP-16780.

Resolution: Resolved

> Track unstable tests according to aarch CI due to OOM
> -
>
> Key: HADOOP-16780
> URL: https://issues.apache.org/jira/browse/HADOOP-16780
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: liusheng
>Priority: Major
>
> |!https://builds.apache.org/static/8ad18952/images/16x16/document_add.png!  
> [org.apache.hadoop.hdfs.TestDFSClientRetries.testLeaseRenewSocketTimeout|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/testReport/org.apache.hadoop.hdfs/TestDFSClientRetries/testLeaseRenewSocketTimeout/]|1.9
>  sec|[1|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/]|
> |!https://builds.apache.org/static/8ad18952/images/16x16/document_add.png!  
> [org.apache.hadoop.hdfs.TestFileChecksum.testStripedFileChecksum1|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/testReport/org.apache.hadoop.hdfs/TestFileChecksum/testStripedFileChecksum1/]|2
>  min 53 sec|[1|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/]|
> |!https://builds.apache.org/static/8ad18952/images/16x16/document_add.png!  
> [org.apache.hadoop.hdfs.TestFileChecksum.testStripedFileChecksum3|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/testReport/org.apache.hadoop.hdfs/TestFileChecksum/testStripedFileChecksum3/]|2
>  min 13 sec|[1|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/]|
> |!https://builds.apache.org/static/8ad18952/images/16x16/document_add.png!  
> [org.apache.hadoop.hdfs.TestFileChecksum.testStripedFileChecksumWithMissedDataBlocksRangeQuery4|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/testReport/org.apache.hadoop.hdfs/TestFileChecksum/testStripedFileChecksumWithMissedDataBlocksRangeQuery4/]|1
>  min 32 sec|[1|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/]|
> |!https://builds.apache.org/static/8ad18952/images/16x16/document_add.png!  
> [org.apache.hadoop.hdfs.TestFileChecksum.testStripedFileChecksumWithMissedDataBlocksRangeQuery5|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/testReport/org.apache.hadoop.hdfs/TestFileChecksum/testStripedFileChecksumWithMissedDataBlocksRangeQuery5/]|11
>  sec|[1|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/]|
> |!https://builds.apache.org/static/8ad18952/images/16x16/document_add.png!  
> [org.apache.hadoop.hdfs.TestFileChecksum.testStripedFileChecksumWithMissedDataBlocksRangeQuery6|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/testReport/org.apache.hadoop.hdfs/TestFileChecksum/testStripedFileChecksumWithMissedDataBlocksRangeQuery6/]|3.8
>  sec|[1|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/]|
> |!https://builds.apache.org/static/8ad18952/images/16x16/document_add.png!  
> [org.apache.hadoop.hdfs.TestFileChecksum.testStripedFileChecksumWithMissedDataBlocksRangeQuery7|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/testReport/org.apache.hadoop.hdfs/TestFileChecksum/testStripedFileChecksumWithMissedDataBlocksRangeQuery7/]|7.7
>  sec|[1|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/]|
> |!https://builds.apache.org/static/8ad18952/images/16x16/document_add.png!  
> [org.apache.hadoop.hdfs.TestFileChecksum.testStripedFileChecksumWithMissedDataBlocksRangeQuery8|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/testReport/org.apache.hadoop.hdfs/TestFileChecksum/testStripedFileChecksumWithMissedDataBlocksRangeQuery8/]|3.7
>  sec|[1|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/]|
> |!https://builds.apache.org/static/8ad18952/images/16x16/document_add.png!  
> [org.apache.hadoop.hdfs.TestFileChecksum.testStripedFileChecksumWithMissedDataBlocksRangeQuery9|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/testReport/org.apache.hadoop.hdfs/TestFileChecksum/testStripedFileChecksumWithMissedDataBlocksRangeQuery9/]|4
>  sec|[1|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/]|
> |!https://builds.apache.org/static/8ad18952/images/16x16/document_add.png!  
> [org.apache.hadoop.hdfs.TestFileChecksumCompositeCrc.testStripedFileChecksumWithMissedDataBlocksRangeQuery6|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/testReport/org.apache.hadoop.hdfs/TestFileChecksumCompositeCrc/testStripedFileChecksumWithMissedDataBlocksRangeQuery6/]|10
>  sec|[1|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/]|
> |!https://builds.apache.org/static/8ad18952/images/16x16/document_add.png!  
> [org.apache.hadoop.hdfs.TestFileChecksumCompositeCrc.testStripedFileChecksumWithMissedDataBlocksRangeQue

[jira] [Reopened] (HADOOP-16780) Track unstable tests according to aarch CI due to OOM

[Ahmed Hussein (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Hussein reopened HADOOP-16780:


> Track unstable tests according to aarch CI due to OOM
> -
>
> Key: HADOOP-16780
> URL: https://issues.apache.org/jira/browse/HADOOP-16780
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: liusheng
>Priority: Major
>
> |!https://builds.apache.org/static/8ad18952/images/16x16/document_add.png!  
> [org.apache.hadoop.hdfs.TestDFSClientRetries.testLeaseRenewSocketTimeout|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/testReport/org.apache.hadoop.hdfs/TestDFSClientRetries/testLeaseRenewSocketTimeout/]|1.9
>  sec|[1|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/]|
> |!https://builds.apache.org/static/8ad18952/images/16x16/document_add.png!  
> [org.apache.hadoop.hdfs.TestFileChecksum.testStripedFileChecksum1|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/testReport/org.apache.hadoop.hdfs/TestFileChecksum/testStripedFileChecksum1/]|2
>  min 53 sec|[1|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/]|
> |!https://builds.apache.org/static/8ad18952/images/16x16/document_add.png!  
> [org.apache.hadoop.hdfs.TestFileChecksum.testStripedFileChecksum3|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/testReport/org.apache.hadoop.hdfs/TestFileChecksum/testStripedFileChecksum3/]|2
>  min 13 sec|[1|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/]|
> |!https://builds.apache.org/static/8ad18952/images/16x16/document_add.png!  
> [org.apache.hadoop.hdfs.TestFileChecksum.testStripedFileChecksumWithMissedDataBlocksRangeQuery4|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/testReport/org.apache.hadoop.hdfs/TestFileChecksum/testStripedFileChecksumWithMissedDataBlocksRangeQuery4/]|1
>  min 32 sec|[1|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/]|
> |!https://builds.apache.org/static/8ad18952/images/16x16/document_add.png!  
> [org.apache.hadoop.hdfs.TestFileChecksum.testStripedFileChecksumWithMissedDataBlocksRangeQuery5|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/testReport/org.apache.hadoop.hdfs/TestFileChecksum/testStripedFileChecksumWithMissedDataBlocksRangeQuery5/]|11
>  sec|[1|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/]|
> |!https://builds.apache.org/static/8ad18952/images/16x16/document_add.png!  
> [org.apache.hadoop.hdfs.TestFileChecksum.testStripedFileChecksumWithMissedDataBlocksRangeQuery6|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/testReport/org.apache.hadoop.hdfs/TestFileChecksum/testStripedFileChecksumWithMissedDataBlocksRangeQuery6/]|3.8
>  sec|[1|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/]|
> |!https://builds.apache.org/static/8ad18952/images/16x16/document_add.png!  
> [org.apache.hadoop.hdfs.TestFileChecksum.testStripedFileChecksumWithMissedDataBlocksRangeQuery7|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/testReport/org.apache.hadoop.hdfs/TestFileChecksum/testStripedFileChecksumWithMissedDataBlocksRangeQuery7/]|7.7
>  sec|[1|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/]|
> |!https://builds.apache.org/static/8ad18952/images/16x16/document_add.png!  
> [org.apache.hadoop.hdfs.TestFileChecksum.testStripedFileChecksumWithMissedDataBlocksRangeQuery8|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/testReport/org.apache.hadoop.hdfs/TestFileChecksum/testStripedFileChecksumWithMissedDataBlocksRangeQuery8/]|3.7
>  sec|[1|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/]|
> |!https://builds.apache.org/static/8ad18952/images/16x16/document_add.png!  
> [org.apache.hadoop.hdfs.TestFileChecksum.testStripedFileChecksumWithMissedDataBlocksRangeQuery9|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/testReport/org.apache.hadoop.hdfs/TestFileChecksum/testStripedFileChecksumWithMissedDataBlocksRangeQuery9/]|4
>  sec|[1|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/]|
> |!https://builds.apache.org/static/8ad18952/images/16x16/document_add.png!  
> [org.apache.hadoop.hdfs.TestFileChecksumCompositeCrc.testStripedFileChecksumWithMissedDataBlocksRangeQuery6|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/testReport/org.apache.hadoop.hdfs/TestFileChecksumCompositeCrc/testStripedFileChecksumWithMissedDataBlocksRangeQuery6/]|10
>  sec|[1|https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/55/]|
> |!https://builds.apache.org/static/8ad18952/images/16x16/document_add.png!  
> [org.apache.hadoop.hdfs.TestFileChecksumCompositeCrc.testStripedFileChecksumWithMissedDataBlocksRangeQue

Yetus is failing with Java unable to create threads

[Ahmed Hussein]

Hey folks,

Yetus has been failing miserably over the last couple of days.
In the Lastest qbt-report
<https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/301/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt>,
hundreds of Junits fail after java failed to acquire resources
to create new threads.

[ERROR]
> testRecoverAllDataBlocks1(org.apache.hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy)
>  Time elapsed: 8.509 s  <<< ERROR!
> java.lang.OutOfMemoryError: unable to create new native thread


Any thoughts on what could trigger that in the last few days? Do we need
more resources for the image?

-- 
Best Regards,

*Ahmed Hussein, PhD*

[jira] [Created] (HADOOP-17321) Improve job submitter framework path handling

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17321:
--

 Summary: Improve job submitter framework path handling
 Key: HADOOP-17321
 URL: https://issues.apache.org/jira/browse/HADOOP-17321
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


[~daryn] pointed out that {{HttpServer2#bindForPortRange(}}) handles IOE then 
check whether the exception is instance of {{BindException}}.

This Jira improves the handling of the Exception



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17314) Reduce port collisions in JUnits

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17314:
--

 Summary: Reduce port collisions in JUnits
 Key: HADOOP-17314
 URL: https://issues.apache.org/jira/browse/HADOOP-17314
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


Several Junit tests passes a non-zero port number to 
{{ServerSocketUtil.getPort()}}.
I noticed that those non-zero values are shared between test methods. Based, on 
the implementation of {{ServerSocketUtil.getPort()}}, the port is checked then, 
the socket is closed immediately.
Two concurrent tests can have the same port number, but one of them fails with 
binding exceptions.

Passing a zero port number reduces the probability of returning the same port 
numbers in two different tests, because the {{ServerSocketUtil.getPort()}} will 
call a random port number.

An example of that is as described in HDFS-15618. {{testRead()}} between 
{{TestBlockTokenWithDFSStriped}} and {{TestBlockTokenWithDFS}} where port 19870 
is used by the two test cases.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17152) Implement wrapper for guava newArrayList and newLinkedList

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17152:
--

 Summary: Implement wrapper for guava newArrayList and newLinkedList
 Key: HADOOP-17152
 URL: https://issues.apache.org/jira/browse/HADOOP-17152
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: common
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


guava Lists class provide some wrappers to java ArrayList and LinkedList.

Replacing the method calls throughout the code can be invasive because guava 
offers some APIs that do not exist in java util. This Jira is the task of 
implementing those missing APIs in hadoop common in a step toward getting rid 
of guava.
 * create a wrapper class org.apache.hadoop.util.unguava.Lists 
 * implement the following interfaces in Lists:
 ** public static  ArrayList newArrayList()
 ** public static  ArrayList newArrayList(E... elements)
 ** public static  ArrayList newArrayList(Iterable elements)
 ** public static  ArrayList newArrayList(Iterator elements)
 ** public static  ArrayList newArrayListWithCapacity(int 
initialArraySize)
 ** public static  LinkedList newLinkedList()
 ** public static  LinkedList newLinkedList(Iterable 
elements)
 ** public static  List asList(@Nullable E first, E[] rest)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17146) Replace Guava.Splitter with common.util.StringUtils

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17146:
--

 Summary: Replace Guava.Splitter with common.util.StringUtils
 Key: HADOOP-17146
 URL: https://issues.apache.org/jira/browse/HADOOP-17146
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: common
Reporter: Ahmed Hussein


Hadoop source code uses {{com.google.common.base.Splitter}} . We need to 
analyze the performance overhead of splitter and consider different 
implementations such as apache-commons.

Hadoop has an implementation {{org.apache.hadoop.util.StringUtils.split()}}. 
Therefore, we should all split() calls use the wrapper in common. This will 
make the utility calls less invasive and confusing as the behavior of 
apache-commons.stringUtils is not the same as guava.

Once we have the wrapper, and all calls are using that wrapper, we can decide 
to use apache-commons or do specific optimizations without changing the entire 
source code.

 
{code:bash}
Targets
Occurrences of 'import com.google.common.base.Splitter;' in project with 
mask '*.java'
Found Occurrences  (18 usages found)
org.apache.hadoop.crypto  (1 usage found)
CryptoCodec.java  (1 usage found)
34 import com.google.common.base.Splitter;
org.apache.hadoop.mapred.nativetask.kvtest  (1 usage found)
KVTest.java  (1 usage found)
44 import com.google.common.base.Splitter;
org.apache.hadoop.mapreduce.v2.util  (1 usage found)
MRWebAppUtil.java  (1 usage found)
20 import com.google.common.base.Splitter;
org.apache.hadoop.metrics2.impl  (1 usage found)
MetricsConfig.java  (1 usage found)
32 import com.google.common.base.Splitter;
org.apache.hadoop.registry.client.impl.zk  (1 usage found)
RegistrySecurity.java  (1 usage found)
22 import com.google.common.base.Splitter;
org.apache.hadoop.security.authentication.server  (1 usage found)
MultiSchemeAuthenticationHandler.java  (1 usage found)
34 import com.google.common.base.Splitter;
org.apache.hadoop.security.token.delegation.web  (1 usage found)
MultiSchemeDelegationTokenAuthenticationHandler.java  (1 usage found)
40 import com.google.common.base.Splitter;
org.apache.hadoop.tools.dynamometer  (1 usage found)
Client.java  (1 usage found)
22 import com.google.common.base.Splitter;
org.apache.hadoop.tools.dynamometer.workloadgenerator.audit  (2 usages 
found)
AuditLogDirectParser.java  (1 usage found)
20 import com.google.common.base.Splitter;
AuditReplayThread.java  (1 usage found)
20 import com.google.common.base.Splitter;
org.apache.hadoop.util  (2 usages found)
TestApplicationClassLoader.java  (1 usage found)
44 import com.google.common.base.Splitter;
ZKUtil.java  (1 usage found)
31 import com.google.common.base.Splitter;
org.apache.hadoop.yarn.api.records.timeline  (1 usage found)
TimelineEntityGroupId.java  (1 usage found)
27 import com.google.common.base.Splitter;
org.apache.hadoop.yarn.server.resourcemanager.scheduler  (1 usage found)
QueueMetrics.java  (1 usage found)
55 import com.google.common.base.Splitter;
org.apache.hadoop.yarn.util  (1 usage found)
StringHelper.java  (1 usage found)
21 import com.google.common.base.Splitter;
org.apache.hadoop.yarn.webapp  (1 usage found)
WebApp.java  (1 usage found)
37 import com.google.common.base.Splitter;
org.apache.hadoop.yarn.webapp.hamlet  (1 usage found)
HamletImpl.java  (1 usage found)
22 import com.google.common.base.Splitter;
org.apache.hadoop.yarn.webapp.hamlet2  (1 usage found)
HamletImpl.java  (1 usage found)
22 import com.google.common.base.Splitter;

{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-17104) Replace Guava Supplier with Java8+ Supplier in hdfs

[Ahmed Hussein (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-17104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Hussein resolved HADOOP-17104.

Resolution: Implemented

After HADOOP-17090, It was possible to combine the change into a single patch.

HADOOP-17100 already include changes required for the HDFS module.

> Replace Guava Supplier with Java8+ Supplier in hdfs
> ---
>
> Key: HADOOP-17104
> URL: https://issues.apache.org/jira/browse/HADOOP-17104
> Project: Hadoop Common
>  Issue Type: Sub-task
>        Reporter: Ahmed Hussein
>    Assignee: Ahmed Hussein
>Priority: Major
>
> Replacing Usage of Guava supplier are in Unit tests 
> {{GenereicTestUtils.waitFor()}} in hadoop-hdfs-project subdirectory.
> {code:java}
> Targets
> Occurrences of 'com.google.common.base.Supplier' in directory 
> hadoop-hdfs-project with mask '*.java'
> Found Occurrences  (99 usages found)
> org.apache.hadoop.fs  (1 usage found)
> TestEnhancedByteBufferAccess.java  (1 usage found)
> 75 import com.google.common.base.Supplier;
> org.apache.hadoop.fs.viewfs  (1 usage found)
> TestViewFileSystemWithTruncate.java  (1 usage found)
> 23 import com.google.common.base.Supplier;
> org.apache.hadoop.hdfs  (20 usages found)
> DFSTestUtil.java  (1 usage found)
> 79 import com.google.common.base.Supplier;
> MiniDFSCluster.java  (1 usage found)
> 78 import com.google.common.base.Supplier;
> TestBalancerBandwidth.java  (1 usage found)
> 29 import com.google.common.base.Supplier;
> TestClientProtocolForPipelineRecovery.java  (1 usage found)
> 30 import com.google.common.base.Supplier;
> TestDatanodeRegistration.java  (1 usage found)
> 44 import com.google.common.base.Supplier;
> TestDataTransferKeepalive.java  (1 usage found)
> 47 import com.google.common.base.Supplier;
> TestDeadNodeDetection.java  (1 usage found)
> 20 import com.google.common.base.Supplier;
> TestDecommission.java  (1 usage found)
> 41 import com.google.common.base.Supplier;
> TestDFSShell.java  (1 usage found)
> 37 import com.google.common.base.Supplier;
> TestEncryptedTransfer.java  (1 usage found)
> 35 import com.google.common.base.Supplier;
> TestEncryptionZonesWithKMS.java  (1 usage found)
> 22 import com.google.common.base.Supplier;
> TestFileCorruption.java  (1 usage found)
> 21 import com.google.common.base.Supplier;
> TestLeaseRecovery2.java  (1 usage found)
> 32 import com.google.common.base.Supplier;
> TestLeaseRecoveryStriped.java  (1 usage found)
> 21 import com.google.common.base.Supplier;
> TestMaintenanceState.java  (1 usage found)
> 63 import com.google.common.base.Supplier;
> TestPread.java  (1 usage found)
> 61 import com.google.common.base.Supplier;
> TestQuota.java  (1 usage found)
> 39 import com.google.common.base.Supplier;
> TestReplaceDatanodeOnFailure.java  (1 usage found)
> 20 import com.google.common.base.Supplier;
> TestReplication.java  (1 usage found)
> 27 import com.google.common.base.Supplier;
> TestSafeMode.java  (1 usage found)
> 62 import com.google.common.base.Supplier;
> org.apache.hadoop.hdfs.client.impl  (2 usages found)
> TestBlockReaderLocalMetrics.java  (1 usage found)
> 20 import com.google.common.base.Supplier;
> TestLeaseRenewer.java  (1 usage found)
> 20 import com.google.common.base.Supplier;
> org.apache.hadoop.hdfs.qjournal  (1 usage found)
> MiniJournalCluster.java  (1 usage found)
> 31 import com.google.common.base.Supplier;
> org.apache.hadoop.hdfs.qjournal.client  (1 usage found)
> TestIPCLoggerChannel.java  (1 usage found)
> 43 import com.google.common.base.Supplier;
> org.apache.hadoop.hdfs.qjournal.server  (1 usage found)
> TestJournalNodeSync.java  (1 usage found)
> 20 import com.google.common.base.Supplier;
> org.apache.hadoop.hdfs.server.blockmanagement  (7 usages found)
> TestBlockManagerSafeMode.java  (1 usage found)
> 20 import com.google.common.base.Supplier;
> TestBlockReportRateLimiting.java  (1 usage found)
> 25 import com.google.common.

[jira] [Resolved] (HADOOP-17103) Replace Guava Supplier with Java8+ Supplier in MAPREDUCE

[Ahmed Hussein (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-17103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Hussein resolved HADOOP-17103.

Resolution: Implemented

After HADOOP-17090, It was possible to combine the change into a single patch.

HADOOP-17100 already include changes required for the Mapreduce module.

> Replace Guava Supplier with Java8+ Supplier in MAPREDUCE
> 
>
> Key: HADOOP-17103
> URL: https://issues.apache.org/jira/browse/HADOOP-17103
> Project: Hadoop Common
>  Issue Type: Sub-task
>        Reporter: Ahmed Hussein
>    Assignee: Ahmed Hussein
>Priority: Major
>
> Replacing Usage of Guava supplier are in Unit tests 
> {{GenereicTestUtils.waitFor()}} in hadoop-mapreduce-project subdirectory.
> {code:java}
> Targets
> hadoop-mapreduce-project with mask '*.java'
> Found Occurrences  (8 usages found)
> org.apache.hadoop.mapred  (2 usages found)
> TestTaskAttemptListenerImpl.java  (1 usage found)
> 20 import com.google.common.base.Supplier;
> UtilsForTests.java  (1 usage found)
> 64 import com.google.common.base.Supplier;
> org.apache.hadoop.mapreduce.v2.app  (4 usages found)
> TestFetchFailure.java  (1 usage found)
> 29 import com.google.common.base.Supplier;
> TestMRApp.java  (1 usage found)
> 31 import com.google.common.base.Supplier;
> TestRecovery.java  (1 usage found)
> 31 import com.google.common.base.Supplier;
> TestTaskHeartbeatHandler.java  (1 usage found)
> 28 import com.google.common.base.Supplier;
> org.apache.hadoop.mapreduce.v2.app.rm  (1 usage found)
> TestRMContainerAllocator.java  (1 usage found)
> 156 import com.google.common.base.Supplier;
> org.apache.hadoop.mapreduce.v2.hs  (1 usage found)
> TestJHSDelegationTokenSecretManager.java  (1 usage found)
> 30 import com.google.common.base.Supplier;
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[unguava] Heads up: patch affecting 147 files in HADOOP-17100

[Ahmed Hussein]

Hi folks,

In the process of replacing Guava APIs, I uploaded a patch to
[YHADOOP-1700.HADOOP-17100.006.patch
<https://issues.apache.org/jira/secure/attachment/13007798/13007798_HADOOP-17100.006.patch>]
that
replaces guava.Supplier
with java.Supplier.

The change is straightforward changing an import line, but it touches 147
files (mostly unit tests).

I appreciate if someone takes a look at the patch and merge it into
trunk, branch-3.3, branch-3.2, and branch-3.1

--
Best Regards,

*Ahmed Hussein, PhD*

[jira] [Created] (HADOOP-17126) implement non-guava Precondition checkNotNull

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17126:
--

 Summary: implement non-guava Precondition checkNotNull
 Key: HADOOP-17126
 URL: https://issues.apache.org/jira/browse/HADOOP-17126
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Ahmed Hussein


As part In order to replace Guava Preconditions, we need to implement our own 
versions of the API.
This Jira is to create {{checkNotNull}} in a new package dubbed {{unguava}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17123) remove guava Preconditions from Hadoop-common module

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17123:
--

 Summary: remove guava Preconditions from Hadoop-common module
 Key: HADOOP-17123
 URL: https://issues.apache.org/jira/browse/HADOOP-17123
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: common
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


Replace guava Preconditions by internal implementations that rely on java8+ 
APIs in the hadoop



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: A more inclusive elephant...

[Ahmed Hussein]

 and US
> > >> culture/history might have more or less strong feelings about your
> role
> > in
> > >> the problem and potential solution.
> > >>
> > >> *What can we do about it?* I think a simple action we can take is to
> > work
> > >> on our code/comments/documentation/websites and remove racist
> > terminology.
> > >> Here is a IETF draft to fix up some of the most egregious examples
> > >> (master/slave, whitelist/backlist) with proposed alternatives.
> > >>
> > >>
> >
> https://tools.ietf.org/id/draft-knodel-terminology-00.html#rfc.section.1.1.1
> > >> Also as we go about this effort, we should also consider other
> > >> "non-inclusive" terminology issues around gender (e.g., binary
> gendered
> > >> examples, "Alice" doing the wrong security thing systematically), and
> > >> ableism (e.g., referring to misbehaving hardware as "lame" or
> "limping",
> > >> etc.).
> > >> The easiest action item is to avoid this going forward (ideally adding
> > it
> > >> to the checkstyles if possible), a more costly one is to start going
> > back
> > >> and refactor away existing instances.
> > >>
> > >> I know this requires a bunch of work as refactorings might break dev
> > >> branches and non-committed patches, possibly scripts, etc. but I think
> > >> this
> > >> is something important and relatively simple we can do. The effect
> goes
> > >> well beyond some text in github, it signals what we believe in, and
> > forces
> > >> hundreds of users and contributors to notice and think about it. Our
> > >> force-multiplier is huge and it matches our responsibility.
> > >>
> > >> What do you folks think?
> > >>
> > >> Thanks,
> > >> Carlo
> > >>
> > >
> >
>


-- 
--
Best Regards,

*Ahmed Hussein, PhD*

[jira] [Created] (HADOOP-17118) TestFileCreation#testServerDefaultsWithMinimalCaching fails intermittently

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17118:
--

 Summary: TestFileCreation#testServerDefaultsWithMinimalCaching 
fails intermittently
 Key: HADOOP-17118
 URL: https://issues.apache.org/jira/browse/HADOOP-17118
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ahmed Hussein


{{TestFileCreation.testServerDefaultsWithMinimalCaching}} fails intermittently 
on trunk


{code:bash}
[ERROR] Tests run: 25, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 
103.413 s <<< FAILURE! - in org.apache.hadoop.hdfs.TestFileCreation
[ERROR] 
testServerDefaultsWithMinimalCaching(org.apache.hadoop.hdfs.TestFileCreation)  
Time elapsed: 2.435 s  <<< FAILURE!
java.lang.AssertionError: expected:<402653184> but was:<268435456>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:631)
at 
org.apache.hadoop.hdfs.TestFileCreation.testServerDefaultsWithMinimalCaching(TestFileCreation.java:279)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at 
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17115) Replace Guava initialization of Sets.newHashSet

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17115:
--

 Summary: Replace Guava initialization of Sets.newHashSet
 Key: HADOOP-17115
 URL: https://issues.apache.org/jira/browse/HADOOP-17115
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Ahmed Hussein


Unjustified usage of Guava API to initialize a {{HashSet}}. This should be 
replaced by Java APIs.


{code:java}
Targets
Occurrences of 'Sets.newHashSet' in project
Found Occurrences  (223 usages found)
org.apache.hadoop.crypto.key  (2 usages found)
TestValueQueue.java  (2 usages found)
testWarmUp()  (2 usages found)
106 Assert.assertEquals(Sets.newHashSet("k1", "k2", "k3"),
107 Sets.newHashSet(fillInfos[0].key,
org.apache.hadoop.crypto.key.kms  (6 usages found)
TestLoadBalancingKMSClientProvider.java  (6 usages found)
testCreation()  (6 usages found)
86 
assertEquals(Sets.newHashSet("http://host1:9600/kms/foo/v1/;),
87 Sets.newHashSet(providers[0].getKMSUrl()));
95 assertEquals(Sets.newHashSet("http://host1:9600/kms/foo/v1/;,
98 Sets.newHashSet(providers[0].getKMSUrl(),
108 
assertEquals(Sets.newHashSet("http://host1:9600/kms/foo/v1/;,
111 Sets.newHashSet(providers[0].getKMSUrl(),
org.apache.hadoop.crypto.key.kms.server  (1 usage found)
KMSAudit.java  (1 usage found)
59 static final Set AGGREGATE_OPS_WHITELIST = 
Sets.newHashSet(
org.apache.hadoop.fs.s3a  (1 usage found)
TestS3AAWSCredentialsProvider.java  (1 usage found)
testFallbackToDefaults()  (1 usage found)
183 Sets.newHashSet());
org.apache.hadoop.fs.s3a.auth  (1 usage found)
AssumedRoleCredentialProvider.java  (1 usage found)
AssumedRoleCredentialProvider(URI, Configuration)  (1 usage found)
113 Sets.newHashSet(this.getClass()));
org.apache.hadoop.fs.s3a.commit.integration  (1 usage found)
ITestS3ACommitterMRJob.java  (1 usage found)
test_200_execute()  (1 usage found)
232 Set expectedKeys = Sets.newHashSet();
org.apache.hadoop.fs.s3a.commit.staging  (5 usages found)
TestStagingCommitter.java  (3 usages found)
testSingleTaskMultiFileCommit()  (1 usage found)
341 Set keys = Sets.newHashSet();
runTasks(JobContext, int, int)  (1 usage found)
603 Set uploads = Sets.newHashSet();
commitTask(StagingCommitter, TaskAttemptContext, int)  (1 usage 
found)
640 Set files = Sets.newHashSet();
TestStagingPartitionedTaskCommit.java  (2 usages found)
verifyFilesCreated(PartitionedStagingCommitter)  (1 usage found)
148 Set files = Sets.newHashSet();
buildExpectedList(StagingCommitter)  (1 usage found)
188 Set expected = Sets.newHashSet();
org.apache.hadoop.hdfs  (5 usages found)
DFSUtil.java  (2 usages found)
getNNServiceRpcAddressesForCluster(Configuration)  (1 usage found)
615 Set availableNameServices = Sets.newHashSet(conf
getNNLifelineRpcAddressesForCluster(Configuration)  (1 usage found)
660 Set availableNameServices = Sets.newHashSet(conf
MiniDFSCluster.java  (1 usage found)
597 private Set fileSystems = Sets.newHashSet();
TestDFSUtil.java  (2 usages found)
testGetNNServiceRpcAddressesForNsIds()  (2 usages found)
1046 assertEquals(Sets.newHashSet("nn1"), internal);
1049 assertEquals(Sets.newHashSet("nn1", "nn2"), all);
org.apache.hadoop.hdfs.net  (5 usages found)
TestDFSNetworkTopology.java  (5 usages found)
testChooseRandomWithStorageType()  (4 usages found)
277 Sets.newHashSet("host2", "host4", "host5", "host6");
278 Set archiveUnderL1 = Sets.newHashSet("host1", 
"host3");
279 Set ramdiskUnderL1 = Sets.newHashSet("host7");
280 Set ssdUnderL1 = Sets.newHashSet("host8");
testChooseRandomWithStorageTypeWithExcluded()  (1 usage found)
363 Set expectedSet = Sets.newHashSet("host4", "host5");
org.apache.hadoop.hdfs.qjournal.server  (2 usages found)
JournalNodeSyncer.java  (2 usages found)
getOtherJournalNodeAddrs()  (1 usage found)
276 HashSet sharedEditsUri = Sets.newHashSet();
getJournalAddrList(String)  (1 usage found)
318 Sets.newHashSet(jn.getBoundIpcAddress()));
org.apache.hadoop.hdfs.server.datanode  (5 usages fo

[jira] [Created] (HADOOP-17114) Replace Guava initialization of Lists.newArrayList

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17114:
--

 Summary: Replace Guava initialization of Lists.newArrayList
 Key: HADOOP-17114
 URL: https://issues.apache.org/jira/browse/HADOOP-17114
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Ahmed Hussein


There are unjustified use of Guava APIs to initialize Lists. This could be 
simply replaced by Java API.


{code:java}
Targets
Occurrences of 'Lists.newArrayList' in project
Found Occurrences  (787 usages found)
org.apache.hadoop.conf  (2 usages found)
TestReconfiguration.java  (2 usages found)
testAsyncReconfigure()  (1 usage found)
391 List changes = Lists.newArrayList();
testStartReconfigurationFailureDueToExistingRunningTask()  (1 usage 
found)
435 List changes = Lists.newArrayList(
org.apache.hadoop.crypto  (1 usage found)
CryptoCodec.java  (1 usage found)
getCodecClasses(Configuration, CipherSuite)  (1 usage found)
107 List> result = 
Lists.newArrayList();
org.apache.hadoop.fs.azurebfs  (84 usages found)
ITestAbfsIdentityTransformer.java  (7 usages found)
transformAclEntriesForSetRequest()  (3 usages found)
240 List aclEntriesToBeTransformed = 
Lists.newArrayList(
253 List aclEntries = 
Lists.newArrayList(aclEntriesToBeTransformed);
271 List expectedAclEntries = Lists.newArrayList(
transformAclEntriesForGetRequest()  (4 usages found)
291 List aclEntriesToBeTransformed = 
Lists.newArrayList(
302 List aclEntries = 
Lists.newArrayList(aclEntriesToBeTransformed);
318 aclEntries = Lists.newArrayList(aclEntriesToBeTransformed);
322 List expectedAclEntries = Lists.newArrayList(
ITestAzureBlobFilesystemAcl.java  (76 usages found)
testModifyAclEntries()  (2 usages found)
95 List aclSpec = Lists.newArrayList(
103 aclSpec = Lists.newArrayList(
testModifyAclEntriesOnlyAccess()  (2 usages found)
128 List aclSpec = Lists.newArrayList(
134 aclSpec = Lists.newArrayList(
testModifyAclEntriesOnlyDefault()  (2 usages found)
151 List aclSpec = Lists.newArrayList(
154 aclSpec = Lists.newArrayList(
testModifyAclEntriesMinimal()  (1 usage found)
175 List aclSpec = Lists.newArrayList(
testModifyAclEntriesMinimalDefault()  (1 usage found)
192 List aclSpec = Lists.newArrayList(
testModifyAclEntriesCustomMask()  (1 usage found)
213 List aclSpec = Lists.newArrayList(
testModifyAclEntriesStickyBit()  (2 usages found)
231 List aclSpec = Lists.newArrayList(
238 aclSpec = Lists.newArrayList(
testModifyAclEntriesPathNotFound()  (1 usage found)
261 List aclSpec = Lists.newArrayList(
testModifyAclEntriesDefaultOnFile()  (1 usage found)
276 List aclSpec = Lists.newArrayList(
testModifyAclEntriesWithDefaultMask()  (2 usages found)
287 List aclSpec = Lists.newArrayList(
291 List modifyAclSpec = Lists.newArrayList(
testModifyAclEntriesWithAccessMask()  (2 usages found)
311 List aclSpec = Lists.newArrayList(
315 List modifyAclSpec = Lists.newArrayList(
testModifyAclEntriesWithDuplicateEntries()  (2 usages found)
332 List aclSpec = Lists.newArrayList(
336 List modifyAclSpec = Lists.newArrayList(
testRemoveAclEntries()  (2 usages found)
348 List aclSpec = Lists.newArrayList(
355 aclSpec = Lists.newArrayList(
testRemoveAclEntriesOnlyAccess()  (2 usages found)
377 List aclSpec = Lists.newArrayList(
384 aclSpec = Lists.newArrayList(
testRemoveAclEntriesOnlyDefault()  (2 usages found)
401 List aclSpec = Lists.newArrayList(
408 aclSpec = Lists.newArrayList(
testRemoveAclEntriesMinimal()  (2 usages found)
429 List aclSpec = Lists.newArrayList(
435 aclSpec = Lists.newArrayList(
testRemoveAclEntriesMinimalDefault()  (2 usages found)
451 List aclSpec = Lists.newArrayList(
458 aclSpec = Lists.newArrayList(
testRemoveAclEntriesStickyBit()  (2 usages found)
479 List aclSpec = Lists.newArrayList(
486 aclSpec = Lists.newArrayList(
testRemoveAclEntriesPathNotFound()  (1 usage found)
507 List aclSpec = Lists.newArrayList(
testRemoveAclEntriesAccessMask()  (2 usages found)
518 L

Add Guava common packages/classes to illegal imports

[Ahmed Hussein]

Hi folks,

I believe we need prevent future commits from re-introducing Guava classes
This can be achieved by two options:

   1. We add rules on the go. Each time we add an illegal import to the
   class that has been replaced. This won't prevent developers from committing
   code that uses Guava lasses. The disadvantage, that it makes the task a
   moving target somehow. Another disadvantage in that approach is an
   increased conflict of code changes because each patch has to append to the
   illegal-imports rules.
   2. Add rules to warn against Guava packages. This will prevent
   introducing Guava imports.


Both options are not bullet-proof to prevent future usage of Guava through
the code because the checkstyle illegal import does not cover API usage.

Does anyone have concerns adding those rules to checkstyle configurations?
And which option should we go for?
--
Best Regards,

*Ahmed Hussein, PhD*

[jira] [Reopened] (HADOOP-17102) Add checkstyle rule to prevent further usage of Guava classes

[Ahmed Hussein (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-17102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Hussein reopened HADOOP-17102:


Lets see if we can add checkstyle so that no one would import any further Guava 
base classes

> Add checkstyle rule to prevent further usage of Guava classes
> -
>
> Key: HADOOP-17102
> URL: https://issues.apache.org/jira/browse/HADOOP-17102
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: build, precommit
>    Reporter: Ahmed Hussein
>    Assignee: Ahmed Hussein
>Priority: Major
>
> We should have precommit rules to prevent further usage of Guava classes that 
> are available in Java8+
> A list replacing Guava APIs with java8 features:
> {code:java}
> com.google.common.io.BaseEncoding#base64()java.util.Base64
> com.google.common.io.BaseEncoding#base64Url() java.util.Base64
> com.google.common.base.Joiner.on()
> java.lang.String#join() or 
>   
>java.util.stream.Collectors#joining()
> com.google.common.base.Optional#of()  java.util.Optional#of()
> com.google.common.base.Optional#absent()  
> java.util.Optional#empty()
> com.google.common.base.Optional#fromNullable()
> java.util.Optional#ofNullable()
> com.google.common.base.Optional   
> java.util.Optional
> com.google.common.base.Predicate  
> java.util.function.Predicate
> com.google.common.base.Function   
> java.util.function.Function
> com.google.common.base.Supplier   
> java.util.function.Supplier
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17111) Replace Guava Optional with Java8+ Optional

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17111:
--

 Summary: Replace Guava Optional with Java8+ Optional
 Key: HADOOP-17111
 URL: https://issues.apache.org/jira/browse/HADOOP-17111
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein



{code:java}
Targets
Occurrences of 'com.google.common.base.Optional' in project with mask 
'*.java'
Found Occurrences  (3 usages found)
org.apache.hadoop.yarn.server.nodemanager  (2 usages found)
DefaultContainerExecutor.java  (1 usage found)
71 import com.google.common.base.Optional;
LinuxContainerExecutor.java  (1 usage found)
22 import com.google.common.base.Optional;
org.apache.hadoop.yarn.server.resourcemanager.recovery  (1 usage found)
TestZKRMStateStorePerf.java  (1 usage found)
21 import com.google.common.base.Optional;

{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17110) Replace Guava Preconditions to avoid Guava dependency

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17110:
--

 Summary: Replace Guava Preconditions to avoid Guava dependency
 Key: HADOOP-17110
 URL: https://issues.apache.org/jira/browse/HADOOP-17110
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Ahmed Hussein


By far, one of the most painful replacement in hadoop. There are two options:
# Using Apache commons
# Using Java wrapper without dependency on third party.

{code:java}
Targets
Occurrences of 'com.google.common.base.Preconditions' in project with mask 
'*.java'
Found Occurrences  (577 usages found)
org.apache.hadoop.conf  (2 usages found)
Configuration.java  (1 usage found)
108 import com.google.common.base.Preconditions;
ReconfigurableBase.java  (1 usage found)
22 import com.google.common.base.Preconditions;
org.apache.hadoop.crypto  (7 usages found)
AesCtrCryptoCodec.java  (1 usage found)
23 import com.google.common.base.Preconditions;
CryptoInputStream.java  (1 usage found)
33 import com.google.common.base.Preconditions;
CryptoOutputStream.java  (1 usage found)
32 import com.google.common.base.Preconditions;
CryptoStreamUtils.java  (1 usage found)
32 import com.google.common.base.Preconditions;
JceAesCtrCryptoCodec.java  (1 usage found)
32 import com.google.common.base.Preconditions;
OpensslAesCtrCryptoCodec.java  (1 usage found)
32 import com.google.common.base.Preconditions;
OpensslCipher.java  (1 usage found)
32 import com.google.common.base.Preconditions;
org.apache.hadoop.crypto.key  (2 usages found)
JavaKeyStoreProvider.java  (1 usage found)
21 import com.google.common.base.Preconditions;
KeyProviderCryptoExtension.java  (1 usage found)
32 import com.google.common.base.Preconditions;
org.apache.hadoop.crypto.key.kms  (3 usages found)
KMSClientProvider.java  (1 usage found)
83 import com.google.common.base.Preconditions;
LoadBalancingKMSClientProvider.java  (1 usage found)
54 import com.google.common.base.Preconditions;
ValueQueue.java  (1 usage found)
36 import com.google.common.base.Preconditions;
org.apache.hadoop.crypto.key.kms.server  (5 usages found)
KeyAuthorizationKeyProvider.java  (1 usage found)
35 import com.google.common.base.Preconditions;
KMS.java  (1 usage found)
20 import com.google.common.base.Preconditions;
KMSAudit.java  (1 usage found)
24 import com.google.common.base.Preconditions;
KMSWebApp.java  (1 usage found)
29 import com.google.common.base.Preconditions;
MiniKMS.java  (1 usage found)
29 import com.google.common.base.Preconditions;
org.apache.hadoop.crypto.random  (1 usage found)
OpensslSecureRandom.java  (1 usage found)
25 import com.google.common.base.Preconditions;
org.apache.hadoop.fs  (19 usages found)
ByteBufferUtil.java  (1 usage found)
29 import com.google.common.base.Preconditions;
ChecksumFileSystem.java  (1 usage found)
32 import com.google.common.base.Preconditions;
FileContext.java  (1 usage found)
68 import com.google.common.base.Preconditions;
FileEncryptionInfo.java  (2 usages found)
27 import static com.google.common.base.Preconditions.checkArgument;
28 import static com.google.common.base.Preconditions.checkNotNull;
FileSystem.java  (2 usages found)
86 import com.google.common.base.Preconditions;
91 import static com.google.common.base.Preconditions.checkArgument;
FileSystemStorageStatistics.java  (1 usage found)
23 import com.google.common.base.Preconditions;
FSDataOutputStreamBuilder.java  (1 usage found)
31 import static com.google.common.base.Preconditions.checkNotNull;
FSInputStream.java  (1 usage found)
24 import com.google.common.base.Preconditions;
FsUrlConnection.java  (1 usage found)
27 import com.google.common.base.Preconditions;
GlobalStorageStatistics.java  (1 usage found)
26 import com.google.common.base.Preconditions;
Globber.java  (1 usage found)
35 import static com.google.common.base.Preconditions.checkNotNull;
MultipartUploader.java  (1 usage found)
31 import static com.google.common.base.Preconditions.checkArgument;
PartialListing.java  (1 usage found)
20 import com.google.common.base.Preconditions;
TestEnhancedByteBufferAccess.java  (1 usage found)
74 import com.google.common.base.Preconditions;
TestLocalFileSystem.java  (1 usage found

[jira] [Created] (HADOOP-17109) Replace Guava base64Url and base64 with Java8+ base64

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17109:
--

 Summary: Replace Guava base64Url and base64 with Java8+ base64
 Key: HADOOP-17109
 URL: https://issues.apache.org/jira/browse/HADOOP-17109
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Ahmed Hussein


One important thing to not here as pointed out by [~jeagles] in [his comment on 
the parent 
task|https://issues.apache.org/jira/browse/HADOOP-17098?focusedCommentId=17147935=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17147935]

{quote}One note to be careful about is that base64 translation is not a 
standard, so the two implementations could produce different results. This 
might matter in the case of serialization, persistence, or client server 
different versions.{quote}


*Base64Url:*

{code:java}
Targets
Occurrences of 'base64Url' in project with mask '*.java'
Found Occurrences  (6 usages found)
org.apache.hadoop.mapreduce  (3 usages found)
CryptoUtils.java  (3 usages found)
wrapIfNecessary(Configuration, FSDataOutputStream, boolean)  (1 
usage found)
138 + Base64.encodeBase64URLSafeString(iv) + "]");
wrapIfNecessary(Configuration, InputStream, long)  (1 usage found)
183 + Base64.encodeBase64URLSafeString(iv) + "]");
wrapIfNecessary(Configuration, FSDataInputStream)  (1 usage found)
218 + Base64.encodeBase64URLSafeString(iv) + "]");
org.apache.hadoop.util  (2 usages found)
KMSUtil.java  (2 usages found)
toJSON(KeyVersion)  (1 usage found)
104 Base64.encodeBase64URLSafeString(
toJSON(EncryptedKeyVersion)  (1 usage found)
117 
.encodeBase64URLSafeString(encryptedKeyVersion.getEncryptedKeyIv()));
org.apache.hadoop.yarn.server.resourcemanager.webapp  (1 usage found)
TestRMWebServicesAppsModification.java  (1 usage found)
testAppSubmit(String, String)  (1 usage found)
837 .put("test", 
Base64.encodeBase64URLSafeString("value12".getBytes("UTF8")));

{code}

*Base64:*

{code:java}
Targets
Occurrences of 'base64;' in project with mask '*.java'
Found Occurrences  (51 usages found)
org.apache.hadoop.crypto.key.kms  (1 usage found)
KMSClientProvider.java  (1 usage found)
20 import org.apache.commons.codec.binary.Base64;
org.apache.hadoop.crypto.key.kms.server  (1 usage found)
KMS.java  (1 usage found)
22 import org.apache.commons.codec.binary.Base64;
org.apache.hadoop.fs  (2 usages found)
XAttrCodec.java  (2 usages found)
23 import org.apache.commons.codec.binary.Base64;
56 BASE64;
org.apache.hadoop.fs.azure  (3 usages found)
AzureBlobStorageTestAccount.java  (1 usage found)
23 import com.microsoft.azure.storage.core.Base64;
BlockBlobAppendStream.java  (1 usage found)
50 import org.apache.commons.codec.binary.Base64;
ITestBlobDataValidation.java  (1 usage found)
50 import com.microsoft.azure.storage.core.Base64;
org.apache.hadoop.fs.azurebfs  (2 usages found)
AzureBlobFileSystemStore.java  (1 usage found)
99 import org.apache.hadoop.fs.azurebfs.utils.Base64;
TestAbfsConfigurationFieldsValidation.java  (1 usage found)
34 import org.apache.hadoop.fs.azurebfs.utils.Base64;
org.apache.hadoop.fs.azurebfs.diagnostics  (2 usages found)
Base64StringConfigurationBasicValidator.java  (1 usage found)
26 import org.apache.hadoop.fs.azurebfs.utils.Base64;
TestConfigurationValidators.java  (1 usage found)
25 import org.apache.hadoop.fs.azurebfs.utils.Base64;
org.apache.hadoop.fs.azurebfs.extensions  (2 usages found)
MockDelegationSASTokenProvider.java  (1 usage found)
37 import org.apache.hadoop.fs.azurebfs.utils.Base64;
MockSASTokenProvider.java  (1 usage found)
27 import org.apache.hadoop.fs.azurebfs.utils.Base64;
org.apache.hadoop.fs.azurebfs.services  (1 usage found)
SharedKeyCredentials.java  (1 usage found)
47 import org.apache.hadoop.fs.azurebfs.utils.Base64;
org.apache.hadoop.fs.cosn  (1 usage found)
CosNativeFileSystemStore.java  (1 usage found)
61 import com.qcloud.cos.utils.Base64;
org.apache.hadoop.fs.s3a  (1 usage found)
EncryptionTestUtils.java  (1 usage found)
26 import org.apache.commons.net.util.Base64;
org.apache.hadoop.hdfs.protocol.datatransfer.sasl  (3 usages found)
DataTransferSaslUtil.java  (1 usage found)
39 import org.apache.commons.codec.binary.Base64;
SaslDataTransferClient.java  (1 usage found)
47 import org.apache.commons.codec.binary.Base64;
   

[jira] [Created] (HADOOP-17108) Create Classes to wrap Guava code replacement

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17108:
--

 Summary: Create Classes to wrap Guava code replacement
 Key: HADOOP-17108
 URL: https://issues.apache.org/jira/browse/HADOOP-17108
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Ahmed Hussein


Usage of Guava APIs in hadoop may not have one line replacement in Java8+. We 
need to create some classes to wrap those common functionalities instead of 
reinventing the wheel everywhere.
For example, we should have new package {{package 
org.apache.hadoop.util.collections}}.
Then we create classes like {{MultiMap}} which may have the entire 
implementation from scratch or we can use Apache Commons Collections 4.4 API.
The Pros of using wrapper is to avoid adding more dependencies in POM if we 
vote to use a third party jar.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17106) Replace Guava Joiner with Java8 String Join

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17106:
--

 Summary: Replace Guava Joiner with Java8 String Join
 Key: HADOOP-17106
 URL: https://issues.apache.org/jira/browse/HADOOP-17106
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


Replace \{{com.google.common.base.Joiner}} with String.join.

 
{code:java}
Targets Occurrences of 'com.google.common.base.Joiner' in project with mask 
'*.java' Found Occurrences (103 usages found) 
org.apache.hadoop.crypto.key.kms.server (1 usage found) 
SimpleKMSAuditLogger.java (1 usage found) 26 import 
com.google.common.base.Joiner; org.apache.hadoop.fs (1 usage found) 
TestPath.java (1 usage found) 37 import com.google.common.base.Joiner; 
org.apache.hadoop.fs.s3a (1 usage found) StorageStatisticsTracker.java (1 usage 
found) 25 import com.google.common.base.Joiner; org.apache.hadoop.ha (1 usage 
found) TestHAAdmin.java (1 usage found) 34 import 
com.google.common.base.Joiner; org.apache.hadoop.hdfs (8 usages found) 
DFSClient.java (1 usage found) 196 import com.google.common.base.Joiner; 
DFSTestUtil.java (1 usage found) 76 import com.google.common.base.Joiner; 
DFSUtil.java (1 usage found) 108 import com.google.common.base.Joiner; 
DFSUtilClient.java (1 usage found) 20 import com.google.common.base.Joiner; 
HAUtil.java (1 usage found) 59 import com.google.common.base.Joiner; 
MiniDFSCluster.java (1 usage found) 145 import com.google.common.base.Joiner; 
StripedFileTestUtil.java (1 usage found) 20 import 
com.google.common.base.Joiner; TestDFSUpgrade.java (1 usage found) 53 import 
com.google.common.base.Joiner; org.apache.hadoop.hdfs.protocol (1 usage found) 
LayoutFlags.java (1 usage found) 26 import com.google.common.base.Joiner; 
org.apache.hadoop.hdfs.protocolPB (1 usage found) TestPBHelper.java (1 usage 
found) 118 import com.google.common.base.Joiner; 
org.apache.hadoop.hdfs.qjournal (1 usage found) MiniJournalCluster.java (1 
usage found) 43 import com.google.common.base.Joiner; 
org.apache.hadoop.hdfs.qjournal.client (5 usages found) AsyncLoggerSet.java (1 
usage found) 38 import com.google.common.base.Joiner; QuorumCall.java (1 usage 
found) 32 import com.google.common.base.Joiner; QuorumException.java (1 usage 
found) 25 import com.google.common.base.Joiner; QuorumJournalManager.java (1 
usage found) 62 import com.google.common.base.Joiner; TestQuorumCall.java (1 
usage found) 29 import com.google.common.base.Joiner; 
org.apache.hadoop.hdfs.server.blockmanagement (4 usages found) HostSet.java (1 
usage found) 21 import com.google.common.base.Joiner; TestBlockManager.java (1 
usage found) 20 import com.google.common.base.Joiner; 
TestBlockReportRateLimiting.java (1 usage found) 24 import 
com.google.common.base.Joiner; TestPendingDataNodeMessages.java (1 usage found) 
41 import com.google.common.base.Joiner; org.apache.hadoop.hdfs.server.common 
(1 usage found) StorageInfo.java (1 usage found) 37 import 
com.google.common.base.Joiner; org.apache.hadoop.hdfs.server.datanode (7 usages 
found) BlockPoolManager.java (1 usage found) 32 import 
com.google.common.base.Joiner; BlockRecoveryWorker.java (1 usage found) 21 
import com.google.common.base.Joiner; BPServiceActor.java (1 usage found) 75 
import com.google.common.base.Joiner; DataNode.java (1 usage found) 226 import 
com.google.common.base.Joiner; ShortCircuitRegistry.java (1 usage found) 49 
import com.google.common.base.Joiner; TestDataNodeHotSwapVolumes.java (1 usage 
found) 21 import com.google.common.base.Joiner; TestRefreshNamenodes.java (1 
usage found) 35 import com.google.common.base.Joiner; 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl (1 usage found) 
FsVolumeImpl.java (1 usage found) 90 import com.google.common.base.Joiner; 
org.apache.hadoop.hdfs.server.namenode (13 usages found) 
FileJournalManager.java (1 usage found) 49 import 
com.google.common.base.Joiner; FSDirectory.java (1 usage found) 24 import 
com.google.common.base.Joiner; FSEditLogLoader.java (1 usage found) 120 import 
com.google.common.base.Joiner; FSEditLogOp.java (1 usage found) 141 import 
com.google.common.base.Joiner; FSImage.java (1 usage found) 78 import 
com.google.common.base.Joiner; FSImageTestUtil.java (1 usage found) 66 import 
com.google.common.base.Joiner; NameNode.java (1 usage found) 21 import 
com.google.common.base.Joiner; TestAuditLogAtDebug.java (1 usage found) 21 
import com.google.common.base.Joiner; TestCheckpoint.java (1 usage found) 97 
import com.google.common.base.Joiner; TestFileJournalManager.java (1 usage 
found) 52 import com.google.common.base.Joiner; 
TestNNStorageRetentionFunctional.java (1 usage found) 39 import 
com.google.common.base.Joiner; TestNNStorageRetentionManager.java (1 usage 
found) 53 import com.google.common.base.Joiner; TestProtectedDirectories.java 
(1 usage found) 21 import com.google.common.base.Joiner

[jira] [Created] (HADOOP-17104) Replace Guava Supplier with Java8+ Supplier in hdfs

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17104:
--

 Summary: Replace Guava Supplier with Java8+ Supplier in hdfs
 Key: HADOOP-17104
 URL: https://issues.apache.org/jira/browse/HADOOP-17104
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


Replacing Usage of Guava supplier are in Unit tests 
{{GenereicTestUtils.waitFor()}} in hadoop-hdfs-project subdirectory.
{code:java}
Targets
Occurrences of 'com.google.common.base.Supplier' in directory 
hadoop-hdfs-project with mask '*.java'
Found Occurrences  (99 usages found)
org.apache.hadoop.fs  (1 usage found)
TestEnhancedByteBufferAccess.java  (1 usage found)
75 import com.google.common.base.Supplier;
org.apache.hadoop.fs.viewfs  (1 usage found)
TestViewFileSystemWithTruncate.java  (1 usage found)
23 import com.google.common.base.Supplier;
org.apache.hadoop.hdfs  (20 usages found)
DFSTestUtil.java  (1 usage found)
79 import com.google.common.base.Supplier;
MiniDFSCluster.java  (1 usage found)
78 import com.google.common.base.Supplier;
TestBalancerBandwidth.java  (1 usage found)
29 import com.google.common.base.Supplier;
TestClientProtocolForPipelineRecovery.java  (1 usage found)
30 import com.google.common.base.Supplier;
TestDatanodeRegistration.java  (1 usage found)
44 import com.google.common.base.Supplier;
TestDataTransferKeepalive.java  (1 usage found)
47 import com.google.common.base.Supplier;
TestDeadNodeDetection.java  (1 usage found)
20 import com.google.common.base.Supplier;
TestDecommission.java  (1 usage found)
41 import com.google.common.base.Supplier;
TestDFSShell.java  (1 usage found)
37 import com.google.common.base.Supplier;
TestEncryptedTransfer.java  (1 usage found)
35 import com.google.common.base.Supplier;
TestEncryptionZonesWithKMS.java  (1 usage found)
22 import com.google.common.base.Supplier;
TestFileCorruption.java  (1 usage found)
21 import com.google.common.base.Supplier;
TestLeaseRecovery2.java  (1 usage found)
32 import com.google.common.base.Supplier;
TestLeaseRecoveryStriped.java  (1 usage found)
21 import com.google.common.base.Supplier;
TestMaintenanceState.java  (1 usage found)
63 import com.google.common.base.Supplier;
TestPread.java  (1 usage found)
61 import com.google.common.base.Supplier;
TestQuota.java  (1 usage found)
39 import com.google.common.base.Supplier;
TestReplaceDatanodeOnFailure.java  (1 usage found)
20 import com.google.common.base.Supplier;
TestReplication.java  (1 usage found)
27 import com.google.common.base.Supplier;
TestSafeMode.java  (1 usage found)
62 import com.google.common.base.Supplier;
org.apache.hadoop.hdfs.client.impl  (2 usages found)
TestBlockReaderLocalMetrics.java  (1 usage found)
20 import com.google.common.base.Supplier;
TestLeaseRenewer.java  (1 usage found)
20 import com.google.common.base.Supplier;
org.apache.hadoop.hdfs.qjournal  (1 usage found)
MiniJournalCluster.java  (1 usage found)
31 import com.google.common.base.Supplier;
org.apache.hadoop.hdfs.qjournal.client  (1 usage found)
TestIPCLoggerChannel.java  (1 usage found)
43 import com.google.common.base.Supplier;
org.apache.hadoop.hdfs.qjournal.server  (1 usage found)
TestJournalNodeSync.java  (1 usage found)
20 import com.google.common.base.Supplier;
org.apache.hadoop.hdfs.server.blockmanagement  (7 usages found)
TestBlockManagerSafeMode.java  (1 usage found)
20 import com.google.common.base.Supplier;
TestBlockReportRateLimiting.java  (1 usage found)
25 import com.google.common.base.Supplier;
TestNameNodePrunesMissingStorages.java  (1 usage found)
21 import com.google.common.base.Supplier;
TestPendingInvalidateBlock.java  (1 usage found)
43 import com.google.common.base.Supplier;
TestPendingReconstruction.java  (1 usage found)
34 import com.google.common.base.Supplier;
TestRBWBlockInvalidation.java  (1 usage found)
49 import com.google.common.base.Supplier;
TestSlowDiskTracker.java  (1 usage found)
48 import com.google.common.base.Supplier;
org.apache.hadoop.hdfs.server.datanode  (13 usages found)
DataNodeTestUtils.java  (1 usage found)
40 import com.google.common.base.Supplier;
TestBlockRecovery.java  (1 usage found)
120 import

[jira] [Created] (HADOOP-17103) Replace Guava Supplier with Java8+ Supplier in MAPREDUCE

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17103:
--

 Summary: Replace Guava Supplier with Java8+ Supplier in MAPREDUCE
 Key: HADOOP-17103
 URL: https://issues.apache.org/jira/browse/HADOOP-17103
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


Replacing Usage of Guava supplier are in Unit tests 
{{GenereicTestUtils.waitFor()}} in hadoop-mapreduce-project subdirectory.
{code:java}
Targets
hadoop-mapreduce-project with mask '*.java'
Found Occurrences  (8 usages found)
org.apache.hadoop.mapred  (2 usages found)
TestTaskAttemptListenerImpl.java  (1 usage found)
20 import com.google.common.base.Supplier;
UtilsForTests.java  (1 usage found)
64 import com.google.common.base.Supplier;
org.apache.hadoop.mapreduce.v2.app  (4 usages found)
TestFetchFailure.java  (1 usage found)
29 import com.google.common.base.Supplier;
TestMRApp.java  (1 usage found)
31 import com.google.common.base.Supplier;
TestRecovery.java  (1 usage found)
31 import com.google.common.base.Supplier;
TestTaskHeartbeatHandler.java  (1 usage found)
28 import com.google.common.base.Supplier;
org.apache.hadoop.mapreduce.v2.app.rm  (1 usage found)
TestRMContainerAllocator.java  (1 usage found)
156 import com.google.common.base.Supplier;
org.apache.hadoop.mapreduce.v2.hs  (1 usage found)
TestJHSDelegationTokenSecretManager.java  (1 usage found)
30 import com.google.common.base.Supplier;

{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-17102) Add checkstyle rule to prevent further usage of Guava classes

[Ahmed Hussein (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-17102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Hussein resolved HADOOP-17102.

Resolution: Abandoned

This is a moving target. It is better we this precommit rule merged with its 
relevant subtak.

> Add checkstyle rule to prevent further usage of Guava classes
> -
>
> Key: HADOOP-17102
> URL: https://issues.apache.org/jira/browse/HADOOP-17102
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: build, precommit
>    Reporter: Ahmed Hussein
>    Assignee: Ahmed Hussein
>Priority: Major
>
> We should have precommit rules to prevent further usage of Guava classes that 
> are available in Java8+
> A list replacing Guava APIs with java8 features:
> {code:java}
> com.google.common.io.BaseEncoding#base64()java.util.Base64
> com.google.common.io.BaseEncoding#base64Url() java.util.Base64
> com.google.common.base.Joiner.on()
> java.lang.String#join() or 
>   
>java.util.stream.Collectors#joining()
> com.google.common.base.Optional#of()  java.util.Optional#of()
> com.google.common.base.Optional#absent()  
> java.util.Optional#empty()
> com.google.common.base.Optional#fromNullable()
> java.util.Optional#ofNullable()
> com.google.common.base.Optional   
> java.util.Optional
> com.google.common.base.Predicate  
> java.util.function.Predicate
> com.google.common.base.Function   
> java.util.function.Function
> com.google.common.base.Supplier   
> java.util.function.Supplier
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17102) Add checkstyle rule to prevent further usage of Guava classes

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17102:
--

 Summary: Add checkstyle rule to prevent further usage of Guava 
classes
 Key: HADOOP-17102
 URL: https://issues.apache.org/jira/browse/HADOOP-17102
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: build, precommit
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


We should have precommit rules to prevent further usage of Guava classes that 
are available in Java8+


A list replacing Guava APIs with java8 features:
{code:java}
com.google.common.io.BaseEncoding#base64()  java.util.Base64
com.google.common.io.BaseEncoding#base64Url()   java.util.Base64
com.google.common.base.Joiner.on()  
java.lang.String#join() or 

 java.util.stream.Collectors#joining()
com.google.common.base.Optional#of()java.util.Optional#of()
com.google.common.base.Optional#absent()
java.util.Optional#empty()
com.google.common.base.Optional#fromNullable()  java.util.Optional#ofNullable()
com.google.common.base.Optional java.util.Optional
com.google.common.base.Predicate
java.util.function.Predicate
com.google.common.base.Function 
java.util.function.Function
com.google.common.base.Supplier 
java.util.function.Supplier
{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17101) Replace Guava Function with Java8+ Function

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17101:
--

 Summary: Replace Guava Function with Java8+ Function
 Key: HADOOP-17101
 URL: https://issues.apache.org/jira/browse/HADOOP-17101
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein



{code:java}
Targets
Occurrences of 'com.google.common.base.Function' in directory 
/Users/ahussein/workspace/repos/community/guava-dependency/amahadoop-17100
Found Occurrences  (7 usages found)
hadoop-hdfs-project/hadoop-hdfs/dev-support/jdiff  (1 usage found)
Apache_Hadoop_HDFS_2.6.0.xml  (1 usage found)
13603 

[jira] [Created] (HADOOP-17100) Replace Guava Supplier with Java8+ Supplier

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17100:
--

 Summary: Replace Guava Supplier with Java8+ Supplier
 Key: HADOOP-17100
 URL: https://issues.apache.org/jira/browse/HADOOP-17100
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


Usage of Guava supplier are in Unit tests.

 
{code:java}
Targets
Occurrences of 'com.google.common.base.Supplier' in project with mask 
'*.java'
Found Occurrences  (146 usages found)
org.apache.hadoop.conf  (1 usage found)
TestReconfiguration.java  (1 usage found)
21 import com.google.common.base.Supplier;
org.apache.hadoop.crypto.key.kms.server  (1 usage found)
TestKMS.java  (1 usage found)
20 import com.google.common.base.Supplier;
org.apache.hadoop.fs  (2 usages found)
FCStatisticsBaseTest.java  (1 usage found)
40 import com.google.common.base.Supplier;
TestEnhancedByteBufferAccess.java  (1 usage found)
75 import com.google.common.base.Supplier;
org.apache.hadoop.fs.viewfs  (1 usage found)
TestViewFileSystemWithTruncate.java  (1 usage found)
23 import com.google.common.base.Supplier;
org.apache.hadoop.ha  (1 usage found)
TestZKFailoverController.java  (1 usage found)
25 import com.google.common.base.Supplier;
org.apache.hadoop.hdfs  (20 usages found)
DFSTestUtil.java  (1 usage found)
79 import com.google.common.base.Supplier;
MiniDFSCluster.java  (1 usage found)
78 import com.google.common.base.Supplier;
TestBalancerBandwidth.java  (1 usage found)
29 import com.google.common.base.Supplier;
TestClientProtocolForPipelineRecovery.java  (1 usage found)
30 import com.google.common.base.Supplier;
TestDatanodeRegistration.java  (1 usage found)
44 import com.google.common.base.Supplier;
TestDataTransferKeepalive.java  (1 usage found)
47 import com.google.common.base.Supplier;
TestDeadNodeDetection.java  (1 usage found)
20 import com.google.common.base.Supplier;
TestDecommission.java  (1 usage found)
41 import com.google.common.base.Supplier;
TestDFSShell.java  (1 usage found)
37 import com.google.common.base.Supplier;
TestEncryptedTransfer.java  (1 usage found)
35 import com.google.common.base.Supplier;
TestEncryptionZonesWithKMS.java  (1 usage found)
22 import com.google.common.base.Supplier;
TestFileCorruption.java  (1 usage found)
21 import com.google.common.base.Supplier;
TestLeaseRecovery2.java  (1 usage found)
32 import com.google.common.base.Supplier;
TestLeaseRecoveryStriped.java  (1 usage found)
21 import com.google.common.base.Supplier;
TestMaintenanceState.java  (1 usage found)
63 import com.google.common.base.Supplier;
TestPread.java  (1 usage found)
61 import com.google.common.base.Supplier;
TestQuota.java  (1 usage found)
39 import com.google.common.base.Supplier;
TestReplaceDatanodeOnFailure.java  (1 usage found)
20 import com.google.common.base.Supplier;
TestReplication.java  (1 usage found)
27 import com.google.common.base.Supplier;
TestSafeMode.java  (1 usage found)
62 import com.google.common.base.Supplier;
org.apache.hadoop.hdfs.client.impl  (2 usages found)
TestBlockReaderLocalMetrics.java  (1 usage found)
20 import com.google.common.base.Supplier;
TestLeaseRenewer.java  (1 usage found)
20 import com.google.common.base.Supplier;
org.apache.hadoop.hdfs.qjournal  (1 usage found)
MiniJournalCluster.java  (1 usage found)
31 import com.google.common.base.Supplier;
org.apache.hadoop.hdfs.qjournal.client  (1 usage found)
TestIPCLoggerChannel.java  (1 usage found)
43 import com.google.common.base.Supplier;
org.apache.hadoop.hdfs.qjournal.server  (1 usage found)
TestJournalNodeSync.java  (1 usage found)
20 import com.google.common.base.Supplier;
org.apache.hadoop.hdfs.server.blockmanagement  (7 usages found)
TestBlockManagerSafeMode.java  (1 usage found)
20 import com.google.common.base.Supplier;
TestBlockReportRateLimiting.java  (1 usage found)
25 import com.google.common.base.Supplier;
TestNameNodePrunesMissingStorages.java  (1 usage found)
21 import com.google.common.base.Supplier;
TestPendingInvalidateBlock.java  (1 usage found)
43 import com.google.common.base.Supplier;
TestPendingReconstruction.java  (1 usage found)
34 import

[jira] [Created] (HADOOP-17099) Replace Guava Predicate with Java8+ Predicate

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17099:
--

 Summary: Replace Guava Predicate with Java8+ Predicate
 Key: HADOOP-17099
 URL: https://issues.apache.org/jira/browse/HADOOP-17099
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


{{com.google.common.base.Predicate}} can be replaced with 
{{java.util.function.Predicate}}. 
The change involving 9 occurrences is straightforward:


{code:java}
Targets
Occurrences of 'com.google.common.base.Predicate' in project with mask 
'*.java'
Found Occurrences  (9 usages found)
org.apache.hadoop.hdfs.server.blockmanagement  (1 usage found)
CombinedHostFileManager.java  (1 usage found)
43 import com.google.common.base.Predicate;
org.apache.hadoop.hdfs.server.namenode  (1 usage found)
NameNodeResourceChecker.java  (1 usage found)
38 import com.google.common.base.Predicate;
org.apache.hadoop.hdfs.server.namenode.snapshot  (1 usage found)
Snapshot.java  (1 usage found)
41 import com.google.common.base.Predicate;
org.apache.hadoop.metrics2.impl  (2 usages found)
MetricsRecords.java  (1 usage found)
21 import com.google.common.base.Predicate;
TestMetricsSystemImpl.java  (1 usage found)
41 import com.google.common.base.Predicate;
org.apache.hadoop.yarn.logaggregation  (1 usage found)
AggregatedLogFormat.java  (1 usage found)
77 import com.google.common.base.Predicate;
org.apache.hadoop.yarn.logaggregation.filecontroller  (1 usage found)
LogAggregationFileController.java  (1 usage found)
22 import com.google.common.base.Predicate;
org.apache.hadoop.yarn.logaggregation.filecontroller.ifile  (1 usage found)
LogAggregationIndexedFileController.java  (1 usage found)
22 import com.google.common.base.Predicate;
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation  
(1 usage found)
AppLogAggregatorImpl.java  (1 usage found)
75 import com.google.common.base.Predicate;

{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17098) Reduce Guava dependency in Hadoop source code

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17098:
--

 Summary: Reduce Guava dependency in Hadoop source code
 Key: HADOOP-17098
 URL: https://issues.apache.org/jira/browse/HADOOP-17098
 Project: Hadoop Common
  Issue Type: Task
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


Relying on Guava implementation in Hadoop has been painful due to compatibility 
and vulnerability issues.
 Guava updates tend to break/deprecate APIs. This made It hard to maintain 
backward compatibility within hadoop versions and clients/downstreams.

With 3.x uses java8+, the java 8 features should preferred to Guava, reducing 
the footprint, and giving stability to source code.

This jira should serve as an umbrella toward an incremental effort to reduce 
the usage of Guava in the source code and to create subtasks to replace Guava 
classes with Java features.

Furthermore, it will be good to add a rule in the pre-commit build to warn 
against introducing a new Guava usage in certain modules.

Any one willing to take part in this code refactoring has to:
 # Focus on one module at a time in order to reduce the conflicts and the size 
of the patch. This will significantly help the reviewers.
 # Run all the unit tests related to the module being affected by the change. 
It is critical to verify that any change will not break the unit tests, or 
cause a stable test case to become flaky.

 

A list of sub tasks replacing Guava APIs with java8 features:
{code:java}
com.google.common.io.BaseEncoding#base64()  java.util.Base64
com.google.common.io.BaseEncoding#base64Url()   java.util.Base64
com.google.common.base.Joiner.on()  
java.lang.String#join() or 

 java.util.stream.Collectors#joining()
com.google.common.base.Optional#of()java.util.Optional#of()
com.google.common.base.Optional#absent()
java.util.Optional#empty()
com.google.common.base.Optional#fromNullable()  java.util.Optional#ofNullable()
com.google.common.base.Optional java.util.Optional
com.google.common.base.Predicate
java.util.function.Predicate
com.google.common.base.Function 
java.util.function.Function
com.google.common.base.Supplier 
java.util.function.Supplier
{code}
 

I also vote for the replacement of {{Precondition}} with either a wrapper, or 
Apache commons lang.

I believe you guys have dealt with Guava compatibilities in the past and 
probably have better insights. Any thoughts? [~weichiu], [~gabor.bota], 
[~ste...@apache.org], [~ayushtkn], [~busbey], [~jeagles], [~kihwal]

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17095) findbugs warnings building branch-2.10

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17095:
--

 Summary: findbugs warnings building branch-2.10
 Key: HADOOP-17095
 URL: https://issues.apache.org/jira/browse/HADOOP-17095
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.10.0, 2.10.1
Reporter: Ahmed Hussein


precommit build for branch-2.10 generates finbugs warnings in several 
components.
 This is an umbrella to analyze those warnings and fix/ignore them as necessary.

 
|{color:#FF}-1{color}|{color:#FF}findbugs{color}|{color:#FF}2m 
1s{color}|{color:#FF}hadoop-common-project/hadoop-common in branch-2.10 has 
14 extant findbugs warnings.{color}|
|{color:#FF}-1{color}|{color:#FF}findbugs{color}|{color:#FF}2m 
54s{color}|{color:#FF}hadoop-hdfs-project/hadoop-hdfs in branch-2.10 has 10 
extant findbugs warnings.{color}|
|{color:#FF}-1{color}|{color:#FF}findbugs{color}|{color:#FF}2m 
12s{color}|{color:#FF}hadoop-hdfs-project/hadoop-hdfs-client in branch-2.10 
has 1 extant findbugs warnings.{color}|
|{color:#FF}-1{color}|{color:#FF}findbugs{color}|{color:#FF}1m 
35s{color}|{color:#FF}hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
 in branch-2.10 has 3 extant findbugs warnings.{color}|
|{color:#FF}-1{color}|{color:#FF}findbugs{color}|{color:#FF}1m 
50s{color}|{color:#FF}hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in 
branch-2.10 has 1 extant findbugs warnings.{color}|



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17094) vulnerabilities reported in jackson and jackson-databind in branch-2.10

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17094:
--

 Summary: vulnerabilities reported in jackson and jackson-databind 
in branch-2.10
 Key: HADOOP-17094
 URL: https://issues.apache.org/jira/browse/HADOOP-17094
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.10.0, 2.10.1
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


There are known vulnerabilities in the 
com.fasterxml.jackson.core:jackson-databind package [,2.9.10.5).

[List of 
vulnerabilities|https://snyk.io/vuln/maven:com.fasterxml.jackson.core%3Ajackson-databind].

Upgrading jackson and jackson-databind to 2.10 should get rid of those 
vulnerabilities.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17083) Update guava to 27.0-jre in hadoop branch-2.10

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17083:
--

 Summary: Update guava to 27.0-jre in hadoop branch-2.10
 Key: HADOOP-17083
 URL: https://issues.apache.org/jira/browse/HADOOP-17083
 Project: Hadoop Common
  Issue Type: Bug
  Components: common, security
Affects Versions: 2.10.0
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


com.google.guava:guava should be upgraded to 27.0-jre due to new CVE's found 
[CVE-2018-10237|https://nvd.nist.gov/vuln/detail/CVE-2018-10237].

 

The upgrade should not affect the version of java used. branch-2.10 still 
sticks to JDK7



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Update guava to 27.0-jre in hadoop branch-2.10

[Ahmed Hussein]

Hi folks,

I was looking into upgrading guava to  27.0-jre on branch-2.10 in order to
address the vulnerabilities reported as CVE-2018-10237
<https://nvd.nist.gov/vuln/detail/CVE-2018-10237>.
Since there are concerns using Java8, the plan is to stick to JDK7.

Obviously, it is expected that the upgrade will break downstream projects.

I opened this for discussion to get feedback and make sure that we have
common ground to address the security of vulnerabilities.

Let me know WDYT.

--
Best Regards,

*Ahmed Hussein, PhD*

mvnsite-hadoop-common fails on Yetus

[Ahmed Hussein]

Hi,

Yetus fails in hadoop-common's pre-commit build for quite some time. It
fails for the trunk branch as well.
Any idea how to fix this?

https://builds.apache.org/job/PreCommit-HADOOP-Build/16957/console

https://builds.apache.org/job/PreCommit-HADOOP-Build/16957/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt


[INFO] --- exec-maven-plugin:1.3.1:exec (shelldocs) @ hadoop-common ---
> ERROR: yetus-dl: gpg unable to import
> /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/sourcedir/patchprocess/KEYS_YETUS
> [INFO]
> 
> [INFO] BUILD FAILURE
> [INFO]
> 
> [INFO] Total time:  9.377 s
> [INFO] Finished at: 2020-05-28T17:37:41Z
> [INFO]
> 
> [ERROR] Failed to execute goal
> org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (shelldocs) on project
> hadoop-common: Command execution failed. Process exited with an error: 1
> (Exit value: 1) -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the
> -e switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions,
> please read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException


--
Best Regards,

*Ahmed Hussein, PhD*

[jira] [Created] (HADOOP-17003) No Log compression and retention at KMS

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-17003:
--

 Summary: No Log compression and retention at KMS
 Key: HADOOP-17003
 URL: https://issues.apache.org/jira/browse/HADOOP-17003
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein



{code:bash}
-rw-r--r-- 1 hkms users 704M Mar 20 23:59 kms.log.2020-03-20
-rw-r--r-- 1 hkms users 731M Mar 21 23:59 kms.log.2020-03-21
-rw-r--r-- 1 hkms users 750M Mar 22 23:59 kms.log.2020-03-22
-rw-r--r-- 1 hkms users 757M Mar 23 23:59 kms.log.2020-03-23
-rw-r--r-- 1 hkms users 805M Mar 24 23:59 kms.log.2020-03-24
-rw-r--r-- 1 hkms users 858M Mar 25 23:59 kms.log.2020-03-25
-rw-r--r-- 1 hkms users 875M Mar 26 23:59 kms.log.2020-03-26
-rw-r--r-- 1 hkms users 754M Mar 27 23:59 kms.log.2020-03-27
{code}

KMS logs have no retention or compression.
They are eating up space generating disk space alerts



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: This week's Hadoop storage community online meetup (APAC)

[Ahmed Hussein]

Hi Wei-Chiu,

I joined the meeting on Feb 19th on the zoom link sent in the original
message but no one was there.
Is the schedule up to date? (
https://calendar.google.com/calendar/b/3?cid=aGFkb29wLmNvbW11bml0eS5zeW5jLnVwQGdtYWlsLmNvbQ
)

On Thu, Feb 13, 2020 at 11:47 AM Wei-Chiu Chuang
 wrote:

> Thanks for joining the call last night/yesterday.
>
> Please find the video recording here:
>
> https://cloudera.zoom.us/rec/play/tMYpdeyp_Ts3EoaR5ASDVPV7W429Kays03Id-PMPzxq9WyZQZlGkbrtBM-Hk48mM3YD3z9xi2zWexHZz?continueMode=true
>
> The presentation slides is available in my personal Google Drive:
> https://drive.google.com/open?id=1IUPtknaPUeKIL74TpNt-R6CK5ICz5veW
>
> On Wed, Feb 12, 2020 at 9:50 PM Wei-Chiu Chuang 
> wrote:
>
> > Gentle reminder for this event.
> > Siyao will lead the session first, followed by a demo.
> >
> > Zoom link: https://cloudera.zoom.us/j/880548968
> >
> > Past meeting minutes:
> >
> >
> https://docs.google.com/document/d/1jXM5Ujvf-zhcyw_5kiQVx6g-HeKe-YGnFS_1-qFXomI/edit?usp=sharing
> >
> >
> > On Mon, Feb 10, 2020 at 1:42 PM Wei-Chiu Chuang 
> > wrote:
> >
> >> Hi!
> >>
> >> I would like to spend this week's session to discuss Distributed Tracing
> >> in Hadoop.
> >>
> >> We had a session at the last Bay Area Hadoop meetup back in June
> >> discussing the Distributed Tracing work we have been doing. I'd like to
> >> share our latest update with the APAC community.
> >>
> >> Zoom link: https://cloudera.zoom.us/j/880548968
> >>
> >> Time/Date:
> >> Feb 12 10PM (US west coast PST) / Feb 13 2pm (Beijing) / Feb 13 3pm
> >> (Tokyo) / Feb 13 11:30am (New Delhi)
> >>
> >
>

[jira] [Resolved] (HADOOP-16861) Trunk is broken on OS X

[Ahmed Hussein (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Hussein resolved HADOOP-16861.

Resolution: Invalid

The problem was in the local repository settings on my local Mac.

> Trunk is broken on OS X
> ---
>
> Key: HADOOP-16861
> URL: https://issues.apache.org/jira/browse/HADOOP-16861
> Project: Hadoop Common
>  Issue Type: Bug
>        Reporter: Ahmed Hussein
>Priority: Major
>
> Trunk does not compiler on Mac OS X with Java "1.8.0_242".
> {code:bash}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) 
> on project hadoop-maven-plugins: Compilation failure
> [ERROR] 
> /home/ahussein/hadoop/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/protoc/ProtocRunner.java:[16,39]
>  package com.fasterxml.jackson.core.type does not exist
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-16861) Trunk is broken on OS X

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-16861:
--

 Summary: Trunk is broken on OS X
 Key: HADOOP-16861
 URL: https://issues.apache.org/jira/browse/HADOOP-16861
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ahmed Hussein


Trunk does not compiler on Mac OS X with Java "1.8.0_242".


{code:bash}
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on 
project hadoop-maven-plugins: Compilation failure
[ERROR] 
/home/ahussein/hadoop/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/protoc/ProtocRunner.java:[16,39]
 package com.fasterxml.jackson.core.type does not exist
{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Needs support to add more entropy to improve cryptographic randomness on Linux VMs

[Ahmed Hussein]

It turned out that INFRA was not the right place to file.
I created another one HADOOP-16810
<https://issues.apache.org/jira/browse/HADOOP-16810> suggesting a quick fix
for docker.
Basically, it requires passing "-v /dev/urandom:/dev/random" to the docker
that runs the pre-commit tests. Any idea how to do that?


On Wed, Jan 15, 2020 at 11:17 AM Ahmed Hussein  wrote:

> I filed a new Jira INFRA-19730
> <https://issues.apache.org/jira/browse/INFRA-19730>
>
> On Wed, Jan 15, 2020 at 10:58 AM Ahmed Hussein  wrote:
>
>>
>> For testing, it is important to be able to use Random instead of
>>> SecureRandom. We should never be using the OS entropy for unit tests.
>>>
>>
>> Is this feasible to do? I assume that going that road implies scanning
>> through all objects and classes and override the methods to consume
>> "java.util.Random" instead of "SecureRandom". For example, the JVM may get
>> stuck initializing "KeyGenerator", and changing that to Random seems a
>> painful task.
>>
>> urandom is a perfectly acceptable option. I am not sure who maintains the
>>> pre-commit machines these days.
>>>
>>
>> Another thing to put into consideration is that changing the Java config
>> will only fix the problem for Java, but the Docker image may get stuck with
>> a similar issue in native or SSL.
>> This makes installing the rng service the reliable way to make the docker
>> image stable.
>>
>>
>>
>>> On Tue, Jan 14, 2020 at 1:05 PM Ahmed Hussein  wrote:
>>>
>>>> I tried setting JVM option  "-Djava.security.egd=file:///dev/./urandom"
>>>> from the command line but it did not work for me.
>>>> In my case, I assume the JVM ignores  "java.security.egd" because it is
>>>> considered a security thing.
>>>>
>>>> On Tue, Jan 14, 2020 at 1:27 PM István Fajth  wrote:
>>>>
>>>> > Hi,
>>>> >
>>>> > based on this article, we might not need infra to do this, but we can
>>>> > specify the /dev/urandom as a random source directly to the JVM for
>>>> test
>>>> > runs with the option: -Djava.security.egd=file:///dev/urandom.
>>>> >
>>>> >
>>>> https://security.stackexchange.com/questions/14386/what-do-i-need-to-configure-to-make-sure-my-software-uses-dev-urandom
>>>> >
>>>> > Pifta
>>>> >
>>>> > Sean Busbey  ezt írta (időpont: 2020.
>>>> jan.
>>>> > 14., K, 20:14):
>>>> >
>>>> >> You should file an INFRA jira asking about this. They can get in
>>>> touch
>>>> >> with the folks who maintain the Hadoop labeled nodes.
>>>> >>
>>>> >> On Tue, Jan 14, 2020 at 12:42 PM Ahmed Hussein 
>>>> wrote:
>>>> >> >
>>>> >> > Hi,
>>>> >> >
>>>> >> > I was investigating a JUnit test
>>>> >> > (MAPREDUCE-7079:TestMRIntermediateDataEncryption
>>>> >> > is failing in precommit builds
>>>> >> > <https://issues.apache.org/jira/browse/MAPREDUCE-7079>) that was
>>>> >> > consistently hanging on Linux VMs and failing Mapreduce pre-builds.
>>>> >> > I found that the test hangs slows or hangs indefinitely whenever
>>>> Java
>>>> >> reads
>>>> >> > the random file.
>>>> >> >
>>>> >> > I explored two different ways to get that test case to work
>>>> properly on
>>>> >> my
>>>> >> > local Linux VM running rel7:
>>>> >> >
>>>> >> >1. To install "haveged" and "rng-tools" on the virtual machine
>>>> >> running
>>>> >> >Rel7. Then, start rngd service {{sudo service rngd start}} .
>>>> >> >2. Change java configuration to load urandom
>>>> >> >{{sudo vim $JAVA_HOME/jre/lib/security/java.security}}
>>>> >> >## Change the line “securerandom.source=file:/dev/random” to
>>>> read:
>>>> >> >securerandom.source=file:/dev/./urandom
>>>> >> >
>>>> >> >
>>>> >> > Is it possible to apply any of the above solutions to the VM that
>>>> runs
>>>> >> the
>>>> >> > precommit builds?
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> busbey
>>>> >>
>>>> >> -
>>>> >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>>>> >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>>> >>
>>>> >>
>>>> >
>>>> > --
>>>> > Pifta
>>>> >
>>>>
>>>

[jira] [Created] (HADOOP-16810) Increase entropy to improve cryptographic randomness on precommit Linux VMs

[Ahmed Hussein (Jira)]

Ahmed Hussein created HADOOP-16810:
--

 Summary: Increase entropy to improve cryptographic randomness on 
precommit Linux VMs
 Key: HADOOP-16810
 URL: https://issues.apache.org/jira/browse/HADOOP-16810
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein


I was investigating a JUnit test (MAPREDUCE-7079 
:TestMRIntermediateDataEncryption is failing in precommit builds) that was 
consistently hanging on Linux VMs and failing Mapreduce pre-builds.
I found that the test hangs slows or hangs indefinitely whenever Java reads the 
random file.

I explored two different ways to get that test case to work properly on my 
local Linux VM running rel7:
# To install "haveged" and "rng-tools" on the virtual machine running Rel7. 
Then, start rngd service {{sudo service rngd start}} . This will fix the 
problem for all the components on the image including java, native and any 
other component.
# Change java configuration to load urandom
{code:bash}
sudo vim $JAVA_HOME/jre/lib/security/java.security
## Change the line “securerandom.source=file:/dev/random” to read: 
securerandom.source=file:/dev/./urandom
{code}

The first solution is better because this will fix the problem for everything 
that requires SSL/TLS or other services that depend upon encryption.

Since the precommit build runs on Docker, then it would be best to mount 
{{/dev/urandom}} from the host as {{/dev/random}} into the container:

{code:java}
docker run -v /dev/urandom:/dev/random
{code}

For Yetus, we need to add the mount to the {{DOCKER_EXTRAARGS}} as follows:

{code:java}
DOCKER_EXTRAARGS+=("-v" "/dev/urandom:/dev/random")
{code}

 ...



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Needs support to add more entropy to improve cryptographic randomness on Linux VMs

[Ahmed Hussein]

I filed a new Jira INFRA-19730
<https://issues.apache.org/jira/browse/INFRA-19730>

On Wed, Jan 15, 2020 at 10:58 AM Ahmed Hussein  wrote:

>
> For testing, it is important to be able to use Random instead of
>> SecureRandom. We should never be using the OS entropy for unit tests.
>>
>
> Is this feasible to do? I assume that going that road implies scanning
> through all objects and classes and override the methods to consume
> "java.util.Random" instead of "SecureRandom". For example, the JVM may get
> stuck initializing "KeyGenerator", and changing that to Random seems a
> painful task.
>
> urandom is a perfectly acceptable option. I am not sure who maintains the
>> pre-commit machines these days.
>>
>
> Another thing to put into consideration is that changing the Java config
> will only fix the problem for Java, but the Docker image may get stuck with
> a similar issue in native or SSL.
> This makes installing the rng service the reliable way to make the docker
> image stable.
>
>
>
>> On Tue, Jan 14, 2020 at 1:05 PM Ahmed Hussein  wrote:
>>
>>> I tried setting JVM option  "-Djava.security.egd=file:///dev/./urandom"
>>> from the command line but it did not work for me.
>>> In my case, I assume the JVM ignores  "java.security.egd" because it is
>>> considered a security thing.
>>>
>>> On Tue, Jan 14, 2020 at 1:27 PM István Fajth  wrote:
>>>
>>> > Hi,
>>> >
>>> > based on this article, we might not need infra to do this, but we can
>>> > specify the /dev/urandom as a random source directly to the JVM for
>>> test
>>> > runs with the option: -Djava.security.egd=file:///dev/urandom.
>>> >
>>> >
>>> https://security.stackexchange.com/questions/14386/what-do-i-need-to-configure-to-make-sure-my-software-uses-dev-urandom
>>> >
>>> > Pifta
>>> >
>>> > Sean Busbey  ezt írta (időpont: 2020.
>>> jan.
>>> > 14., K, 20:14):
>>> >
>>> >> You should file an INFRA jira asking about this. They can get in touch
>>> >> with the folks who maintain the Hadoop labeled nodes.
>>> >>
>>> >> On Tue, Jan 14, 2020 at 12:42 PM Ahmed Hussein  wrote:
>>> >> >
>>> >> > Hi,
>>> >> >
>>> >> > I was investigating a JUnit test
>>> >> > (MAPREDUCE-7079:TestMRIntermediateDataEncryption
>>> >> > is failing in precommit builds
>>> >> > <https://issues.apache.org/jira/browse/MAPREDUCE-7079>) that was
>>> >> > consistently hanging on Linux VMs and failing Mapreduce pre-builds.
>>> >> > I found that the test hangs slows or hangs indefinitely whenever
>>> Java
>>> >> reads
>>> >> > the random file.
>>> >> >
>>> >> > I explored two different ways to get that test case to work
>>> properly on
>>> >> my
>>> >> > local Linux VM running rel7:
>>> >> >
>>> >> >1. To install "haveged" and "rng-tools" on the virtual machine
>>> >> running
>>> >> >Rel7. Then, start rngd service {{sudo service rngd start}} .
>>> >> >2. Change java configuration to load urandom
>>> >> >{{sudo vim $JAVA_HOME/jre/lib/security/java.security}}
>>> >> >## Change the line “securerandom.source=file:/dev/random” to
>>> read:
>>> >> >securerandom.source=file:/dev/./urandom
>>> >> >
>>> >> >
>>> >> > Is it possible to apply any of the above solutions to the VM that
>>> runs
>>> >> the
>>> >> > precommit builds?
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> busbey
>>> >>
>>> >> -
>>> >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>>> >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>> >>
>>> >>
>>> >
>>> > --
>>> > Pifta
>>> >
>>>
>>

Re: Needs support to add more entropy to improve cryptographic randomness on Linux VMs

[Ahmed Hussein]

> For testing, it is important to be able to use Random instead of
> SecureRandom. We should never be using the OS entropy for unit tests.
>

Is this feasible to do? I assume that going that road implies scanning
through all objects and classes and override the methods to consume
"java.util.Random" instead of "SecureRandom". For example, the JVM may get
stuck initializing "KeyGenerator", and changing that to Random seems a
painful task.

urandom is a perfectly acceptable option. I am not sure who maintains the
> pre-commit machines these days.
>

Another thing to put into consideration is that changing the Java config
will only fix the problem for Java, but the Docker image may get stuck with
a similar issue in native or SSL.
This makes installing the rng service the reliable way to make the docker
image stable.



> On Tue, Jan 14, 2020 at 1:05 PM Ahmed Hussein  wrote:
>
>> I tried setting JVM option  "-Djava.security.egd=file:///dev/./urandom"
>> from the command line but it did not work for me.
>> In my case, I assume the JVM ignores  "java.security.egd" because it is
>> considered a security thing.
>>
>> On Tue, Jan 14, 2020 at 1:27 PM István Fajth  wrote:
>>
>> > Hi,
>> >
>> > based on this article, we might not need infra to do this, but we can
>> > specify the /dev/urandom as a random source directly to the JVM for test
>> > runs with the option: -Djava.security.egd=file:///dev/urandom.
>> >
>> >
>> https://security.stackexchange.com/questions/14386/what-do-i-need-to-configure-to-make-sure-my-software-uses-dev-urandom
>> >
>> > Pifta
>> >
>> > Sean Busbey  ezt írta (időpont: 2020. jan.
>> > 14., K, 20:14):
>> >
>> >> You should file an INFRA jira asking about this. They can get in touch
>> >> with the folks who maintain the Hadoop labeled nodes.
>> >>
>> >> On Tue, Jan 14, 2020 at 12:42 PM Ahmed Hussein  wrote:
>> >> >
>> >> > Hi,
>> >> >
>> >> > I was investigating a JUnit test
>> >> > (MAPREDUCE-7079:TestMRIntermediateDataEncryption
>> >> > is failing in precommit builds
>> >> > <https://issues.apache.org/jira/browse/MAPREDUCE-7079>) that was
>> >> > consistently hanging on Linux VMs and failing Mapreduce pre-builds.
>> >> > I found that the test hangs slows or hangs indefinitely whenever Java
>> >> reads
>> >> > the random file.
>> >> >
>> >> > I explored two different ways to get that test case to work properly
>> on
>> >> my
>> >> > local Linux VM running rel7:
>> >> >
>> >> >1. To install "haveged" and "rng-tools" on the virtual machine
>> >> running
>> >> >Rel7. Then, start rngd service {{sudo service rngd start}} .
>> >> >2. Change java configuration to load urandom
>> >> >{{sudo vim $JAVA_HOME/jre/lib/security/java.security}}
>> >> >## Change the line “securerandom.source=file:/dev/random” to read:
>> >> >securerandom.source=file:/dev/./urandom
>> >> >
>> >> >
>> >> > Is it possible to apply any of the above solutions to the VM that
>> runs
>> >> the
>> >> > precommit builds?
>> >>
>> >>
>> >>
>> >> --
>> >> busbey
>> >>
>> >> -
>> >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>> >>
>> >>
>> >
>> > --
>> > Pifta
>> >
>>
>

Re: Needs support to add more entropy to improve cryptographic randomness on Linux VMs

[Ahmed Hussein]

I tried setting JVM option  "-Djava.security.egd=file:///dev/./urandom"
from the command line but it did not work for me.
In my case, I assume the JVM ignores  "java.security.egd" because it is
considered a security thing.

On Tue, Jan 14, 2020 at 1:27 PM István Fajth  wrote:

> Hi,
>
> based on this article, we might not need infra to do this, but we can
> specify the /dev/urandom as a random source directly to the JVM for test
> runs with the option: -Djava.security.egd=file:///dev/urandom.
>
> https://security.stackexchange.com/questions/14386/what-do-i-need-to-configure-to-make-sure-my-software-uses-dev-urandom
>
> Pifta
>
> Sean Busbey  ezt írta (időpont: 2020. jan.
> 14., K, 20:14):
>
>> You should file an INFRA jira asking about this. They can get in touch
>> with the folks who maintain the Hadoop labeled nodes.
>>
>> On Tue, Jan 14, 2020 at 12:42 PM Ahmed Hussein  wrote:
>> >
>> > Hi,
>> >
>> > I was investigating a JUnit test
>> > (MAPREDUCE-7079:TestMRIntermediateDataEncryption
>> > is failing in precommit builds
>> > <https://issues.apache.org/jira/browse/MAPREDUCE-7079>) that was
>> > consistently hanging on Linux VMs and failing Mapreduce pre-builds.
>> > I found that the test hangs slows or hangs indefinitely whenever Java
>> reads
>> > the random file.
>> >
>> > I explored two different ways to get that test case to work properly on
>> my
>> > local Linux VM running rel7:
>> >
>> >1. To install "haveged" and "rng-tools" on the virtual machine
>> running
>> >Rel7. Then, start rngd service {{sudo service rngd start}} .
>> >2. Change java configuration to load urandom
>> >{{sudo vim $JAVA_HOME/jre/lib/security/java.security}}
>> >## Change the line “securerandom.source=file:/dev/random” to read:
>> >securerandom.source=file:/dev/./urandom
>> >
>> >
>> > Is it possible to apply any of the above solutions to the VM that runs
>> the
>> > precommit builds?
>>
>>
>>
>> --
>> busbey
>>
>> -
>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>
>>
>
> --
> Pifta
>

Needs support to add more entropy to improve cryptographic randomness on Linux VMs

[Ahmed Hussein]

Hi,

I was investigating a JUnit test
(MAPREDUCE-7079:TestMRIntermediateDataEncryption
is failing in precommit builds
) that was
consistently hanging on Linux VMs and failing Mapreduce pre-builds.
I found that the test hangs slows or hangs indefinitely whenever Java reads
the random file.

I explored two different ways to get that test case to work properly on my
local Linux VM running rel7:

   1. To install "haveged" and "rng-tools" on the virtual machine running
   Rel7. Then, start rngd service {{sudo service rngd start}} .
   2. Change java configuration to load urandom
   {{sudo vim $JAVA_HOME/jre/lib/security/java.security}}
   ## Change the line “securerandom.source=file:/dev/random” to read:
   securerandom.source=file:/dev/./urandom


Is it possible to apply any of the above solutions to the VM that runs the
precommit builds?

Re: [DISCUSS] Guidelines for Code cleanup JIRAs

[Ahmed Hussein]

+1
Can we also make sure to add a label for the code cleanup Jiras? At least,
this will make it easy to search and filter jiras.

On Mon, Jan 13, 2020 at 7:24 AM Wei-Chiu Chuang  wrote:

> +1
>
> On Thu, Jan 9, 2020 at 9:33 AM epa...@apache.org 
> wrote:
>
> > There was some discussion on
> > https://issues.apache.org/jira/browse/YARN-9052
> > about concerns surrounding the costs/benefits of code cleanup JIRAs. This
> > email
> > is to get the discussion going within a wider audience.
> >
> > The positive points for code cleanup JIRAs:
> > - Clean up tech debt
> > - Make code more readable
> > - Make code more maintainable
> > - Make code more performant
> >
> > The concerns regarding code cleanup JIRAs are as follows:
> > - If the changes only go into trunk, then contributors and committers
> > trying to
> >  backport to prior releases will have to create and test multiple patch
> > versions.
> > - Some have voiced concerns that code cleanup JIRAs may not be tested as
> >   thoroughly as features and bug fixes because functionality is not
> > supposed to
> >   change.
> > - Any patches awaiting review that are touching the same code will have
> to
> > be
> >   redone, re-tested, and re-reviewed.
> > - JIRAs that are opened for code cleanup and not worked on right away
> tend
> > to
> >   clutter up the JIRA space.
> >
> > Here are my opinions:
> > - Code changes of any kind force a non-trivial amount of overhead for
> other
> >   developers. For code cleanup JIRAs, sometimes the usability,
> > maintainability,
> >   and performance is worth the overhead (as in the case of YARN-9052).
> > - Before opening any JIRA, please always consider whether or not the
> added
> >   usability will outweigh the added pain you are causing other
> developers.
> > - If you believe the benefits outweigh the costs, please backport the
> > changes
> >   yourself to all active lines. My preference is to port all the way back
> > to 2.10.
> > - Please don't run code analysis tools and then open many JIRAs that
> > document
> >   those findings. That activity does not put any thought into this
> > cost-benefit
> >   analysis.
> >
> > Thanks everyone. I'm looking forward to your thoughts. I appreciate all
> > you do
> > for the open source community and it is always a pleasure to work with
> you.
> > -Eric Payne
> >
> > -
> > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> >
> >
>