Re: [DISCUSS] Secure Hadoop without Kerberos

not really Kerberos issues. > > Yes, we do understand that making Kerberos work *in a right way* is > always > > an uphill task (I'm a long time Kerberos+Hadoop Support Engineer) but > that > > shouldn't be the reason to replace it. > > > > Hint: CVE-2020-9492

Re: [DISCUSS] Secure Hadoop without Kerberos

. regards, Eric On Thu, May 21, 2020 at 9:22 AM Steve Loughran wrote: > On Wed, 6 May 2020 at 23:32, Eric Yang wrote: > > > Hi all, > > > > > > 4. Passing different form of tokens does not work well with cloud > provider > > security mechanism. For examp

Re: [DISCUSS] Secure Hadoop without Kerberos

See my comments inline: On Wed, May 20, 2020 at 4:50 PM Rajive Chittajallu wrote: > On Wed, May 20, 2020 at 1:47 PM Eric Yang wrote: > > > >> > Kerberos was developed decade before web development becomes popular. > >> > There are some Kerberos limitations whi

Re: [DISCUSS] Secure Hadoop without Kerberos

DC > and that way operating costs will be reduced. > > Regards, > Akira > > On Thu, May 7, 2020 at 7:32 AM Eric Yang wrote: > >> Hi all, >> >> Kerberos was developed decade before web development becomes popular. >> There are some Kerberos limitations which does

Re: [NOTICE] Removal of protobuf classes from Hadoop Token's public APIs' signature

ProtobufHelper should not be a public API. Hadoop uses protobuf serialization to expertise RPC performance with many drawbacks. The generalized object usually require another indirection to map to usable Java object, this is making Hadoop code messy, and that is topic for another day. The main

Re: ZStandard compression crashes

If I recall this problem correctly, the root cause is the default zstd compression block size is 256kb, and Hadoop Zstd compression will attempt to use the OS platform default compression size, if it is available. The recommended output size is slightly bigger than input size to account for

[DISCUSS] Secure Hadoop without Kerberos

Hi all, Kerberos was developed decade before web development becomes popular. There are some Kerberos limitations which does not work well in Hadoop. A few examples of corner cases: 1. Kerberos principal doesn't encode port number, it is difficult to know if the principal is coming from an

[jira] [Resolved] (HADOOP-16590) IBM Java has deprecated OS login module classes and OS principal classes.

[ https://issues.apache.org/jira/browse/HADOOP-16590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang resolved HADOOP-16590. Fix Version/s: 3.3.0 Resolution: Fixed [~nmarion] Thanks for the patch. I merged pull

[jira] [Resolved] (HADOOP-16614) Missing leveldbjni package of aarch64 platform

[ https://issues.apache.org/jira/browse/HADOOP-16614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang resolved HADOOP-16614. Fix Version/s: 3.3.0 Resolution: Fixed Thank you [~seanlau] for the patch. +1 merged

[jira] [Created] (HADOOP-16463) Migrate away from jsr305 jar

Eric Yang created HADOOP-16463: -- Summary: Migrate away from jsr305 jar Key: HADOOP-16463 URL: https://issues.apache.org/jira/browse/HADOOP-16463 Project: Hadoop Common Issue Type: Bug

[jira] [Created] (HADOOP-16457) Hadoop does not work without Kerberos for simple security

Eric Yang created HADOOP-16457: -- Summary: Hadoop does not work without Kerberos for simple security Key: HADOOP-16457 URL: https://issues.apache.org/jira/browse/HADOOP-16457 Project: Hadoop Common

[jira] [Resolved] (HADOOP-16095) Support impersonation for AuthenticationFilter

[ https://issues.apache.org/jira/browse/HADOOP-16095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang resolved HADOOP-16095. Resolution: Fixed All related tasks have been closed, mark this as resolved.  Thank you

[jira] [Resolved] (HADOOP-16356) Distcp with webhdfs is not working with ProxyUserAuthenticationFilter or AuthenticationFilter

[ https://issues.apache.org/jira/browse/HADOOP-16356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang resolved HADOOP-16356. Resolution: Duplicate > Distcp with webhdfs is not working with ProxyUserAuthenticationFil

[jira] [Reopened] (HADOOP-16095) Support impersonation for AuthenticationFilter

[ https://issues.apache.org/jira/browse/HADOOP-16095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang reopened HADOOP-16095: Found an issue with distcp backward compatibility, opened HADOOP-16356 to track required changes

[jira] [Created] (HADOOP-16356) Distcp with webhdfs is not working with ProxyUserAuthenticationFilter or AuthenticationFilter

Eric Yang created HADOOP-16356: -- Summary: Distcp with webhdfs is not working with ProxyUserAuthenticationFilter or AuthenticationFilter Key: HADOOP-16356 URL: https://issues.apache.org/jira/browse/HADOOP-16356

[jira] [Resolved] (HADOOP-16095) Support impersonation for AuthenticationFilter

[ https://issues.apache.org/jira/browse/HADOOP-16095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang resolved HADOOP-16095. Resolution: Fixed Fix Version/s: 3.3.0 The current implementation is based on option 1

[jira] [Created] (HADOOP-16325) Add ability to run pytthon test and build docker in docker in start-build-env.sh

Eric Yang created HADOOP-16325: -- Summary: Add ability to run pytthon test and build docker in docker in start-build-env.sh Key: HADOOP-16325 URL: https://issues.apache.org/jira/browse/HADOOP-16325

[jira] [Created] (HADOOP-16314) Make sure all end point URL is covered by the same AuthenticationFilter

Eric Yang created HADOOP-16314: -- Summary: Make sure all end point URL is covered by the same AuthenticationFilter Key: HADOOP-16314 URL: https://issues.apache.org/jira/browse/HADOOP-16314 Project

Re: [DISCUSS] Docker build process

ockerfile would be > adjusted). > > Marton > > ps: for the development (non published images) I am convinced that the > optional docker profile can be an easier way to create images. Will > create a similar plugin execution for this Dockerfile: > > https://github.com/apache/hadoop/t

Re: VOTE: Hadoop Ozone 0.4.0-alpha RC2

+1 On 4/29/19, 9:05 PM, "Ajay Kumar" wrote: Hi All, We have created the third release candidate (RC2) for Apache Hadoop Ozone 0.4.0-alpha. This release contains security payload for Ozone. Below are some important features in it:

[jira] [Created] (HADOOP-16236) Update C code to run with c99 standard

Eric Yang created HADOOP-16236: -- Summary: Update C code to run with c99 standard Key: HADOOP-16236 URL: https://issues.apache.org/jira/browse/HADOOP-16236 Project: Hadoop Common Issue Type

Re: [DISCUSS] Docker build process

The flexibility of date appended release number is equivalent to maven snapshot or Docker latest image convention, machine can apply timestamp better than human. By using the Jenkins release process, this can be done with little effort. For official release, it is best to use Docker image

Re: [DISCUSS] Kerberos credential cache location in UGI

IBM JDK is one of the Java that default ticket cache to different than /tmp/krb5* path. If I recall correctly, most logic had been implemented in UGI by passing ticketCachePath parameter in 2012-13 time frame. The new addition will follow the MIT Kerberos lookup order, this is a good

Re: [DISCUSS] Docker build process

, Eric From: Jonathan Eagles Date: Tuesday, March 19, 2019 at 11:48 AM To: Eric Yang Cc: "Elek, Marton" , Hadoop Common , "yarn-...@hadoop.apache.org" , Hdfs-dev , Eric Badger , Eric Payne , Jim Brennan Subject: Re: [DISCUSS] Docker build process This email discussion

Re: [DISCUSS] Docker build process

terms or share a link to the description? > I will make adjustment accordingly unless 7 more people comes > out and say otherwise. What adjustment is this? Thanks, Arpit > On Mar 19, 2019, at 10:19 AM, Eric Yang wrote: > > Hi Ma

Re: [DISCUSS] Docker build process

jdk8/jdk11). In Hadoop, profiles are used to introduce optional steps. I think it's fine as the maven lifecycle/phase model is very static (compare it with the tree based approach in Gradle). Marton [1]: https://issues.apache.org/jira/browse/HADOOP-16091 On 3/13/19 11:

Re: [DISCUSS] Docker build process

Loughran Date: Monday, March 18, 2019 at 3:36 AM To: Eric Yang Cc: Hadoop Common , "yarn-...@hadoop.apache.org" , Hdfs-dev , Eric Badger , Eric Payne , Jonathan Eagles , Jim Brennan , "Elek, Marton" Subject: Re: [DISCUSS] Docker build process I'm not enthusiastic about mak

[DISCUSS] Docker build process

Hi Hadoop developers, In the recent months, there were various discussions on creating docker build process for Hadoop. There was convergence to make docker build process inline in the mailing list last month when Ozone team is planning new repository for Hadoop/ozone docker images. New

[jira] [Resolved] (HADOOP-16106) hadoop-aws project javadoc does not compile

[ https://issues.apache.org/jira/browse/HADOOP-16106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang resolved HADOOP-16106. Resolution: Duplicate This is a duplicate of HADOOP-16098. > hadoop-aws project javadoc d

[jira] [Created] (HADOOP-16106) hadoop-aws project javadoc does not compile

Eric Yang created HADOOP-16106: -- Summary: hadoop-aws project javadoc does not compile Key: HADOOP-16106 URL: https://issues.apache.org/jira/browse/HADOOP-16106 Project: Hadoop Common Issue Type

[jira] [Created] (HADOOP-16094) AuthenticationFilter can trigger NullPointerException in KerberosName class

Eric Yang created HADOOP-16094: -- Summary: AuthenticationFilter can trigger NullPointerException in KerberosName class Key: HADOOP-16094 URL: https://issues.apache.org/jira/browse/HADOOP-16094 Project

Re: [DISCUSS] Making submarine to different release model like Ozone

AM To: Eric Yang Cc: Weiwei Yang , Xun Liu , Hadoop Common , "yarn-...@hadoop.apache.org" , Hdfs-dev , "mapreduce-...@hadoop.apache.org" Subject: Re: [DISCUSS] Making submarine to different release model like Ozone Thanks everyone for sharing thoughts! Eric, appreci

Re: [DISCUSS] Making submarine to different release model like Ozone

Submarine is an application built for YARN framework, but it does not have strong dependency on YARN development. For this kind of projects, it would be best to enter Apache Incubator cycles to create a new community. Apache commons is the only project other than Incubator that has

Re: proposed new repository for hadoop/ozone docker images (+update on docker works)

Thanks a lot, Marton On 1/30/19 6:50 PM, Eric Yang wrote: > Hi Marton, > > Hi Marton, > > Flagging automated build on dockerhub seems conflicts with Apache release policy. The vote and release process are manual processes of Apache Way. T

Re: proposed new repository for hadoop/ozone docker images (+update on docker works)

a Apache account? > > > Thanks > Anu > > > On 1/29/19, 4:54 PM, "Eric Yang" wrote: > > By separating Hadoop docker related build into a separate git repository have some slippery slope. It is harder to synchronize the change

Re: proposed new repository for hadoop/ozone docker images (+update on docker works)

By separating Hadoop docker related build into a separate git repository have some slippery slope. It is harder to synchronize the changes between two separate source trees. There is multi-steps process to build jar, tarball, and docker images. This might be problematic to reproduce. It

[jira] [Resolved] (HADOOP-15959) revert HADOOP-12751

[ https://issues.apache.org/jira/browse/HADOOP-15959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang resolved HADOOP-15959. Resolution: Fixed The failed registry DNS unit test has been addressed in HADOOP-16031. Hence

[jira] [Created] (HADOOP-15996) Plugin interface to support more complex usernames in Hadoop

Eric Yang created HADOOP-15996: -- Summary: Plugin interface to support more complex usernames in Hadoop Key: HADOOP-15996 URL: https://issues.apache.org/jira/browse/HADOOP-15996 Project: Hadoop Common

[jira] [Reopened] (HADOOP-15922) DelegationTokenAuthenticationFilter get wrong doAsUser since it does not decode URL

[ https://issues.apache.org/jira/browse/HADOOP-15922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang reopened HADOOP-15922: > DelegationTokenAuthenticationFilter get wrong doAsUser since it does not > deco

[jira] [Created] (HADOOP-15896) Refine Kerberos based AuthenticationHandler to check proxyuser ACL

Eric Yang created HADOOP-15896: -- Summary: Refine Kerberos based AuthenticationHandler to check proxyuser ACL Key: HADOOP-15896 URL: https://issues.apache.org/jira/browse/HADOOP-15896 Project: Hadoop

[jira] [Reopened] (HADOOP-15821) Move Hadoop YARN Registry to Hadoop Registry

[ https://issues.apache.org/jira/browse/HADOOP-15821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang reopened HADOOP-15821: > Move Hadoop YARN Registry to Hadoop Regis

[jira] [Resolved] (HADOOP-15670) UserGroupInformation TGT renewer thread doesn't use monotonically increasing time for calculating interval to sleep

[ https://issues.apache.org/jira/browse/HADOOP-15670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang resolved HADOOP-15670. Resolution: Not A Problem > UserGroupInformation TGT renewer thread doesn't use monotonica

Re: [DISCUSS] Tracing in the Hadoop ecosystem

Most of code coverage tools can instrument java classes without make any source code changes, but tracing distributed system is more involved because code execution via network interactions are not easy to match up. All interactions between sender and receiver have some form of session id or

Re: Hadoop 3.2 Release Plan proposal

Hi Sunil, For YARN service and docker related work, I like to propose the following features to be merged: YARN-7129 Application Catalog for YARN applications YARN-7512 Support service upgrade via YARN Service API and CLI YARN-8220 Running Tensorflow on YARN with GPU and Docker - Examples

[jira] [Created] (HADOOP-15601) Change yarn.admin.acl setting to be more restricted

Eric Yang created HADOOP-15601: -- Summary: Change yarn.admin.acl setting to be more restricted Key: HADOOP-15601 URL: https://issues.apache.org/jira/browse/HADOOP-15601 Project: Hadoop Common

[jira] [Created] (HADOOP-15600) Set default proxy user settings to non-routable IP addresses and default users group

Eric Yang created HADOOP-15600: -- Summary: Set default proxy user settings to non-routable IP addresses and default users group Key: HADOOP-15600 URL: https://issues.apache.org/jira/browse/HADOOP-15600

[jira] [Resolved] (HADOOP-15597) UserGroupInformation class throws NPE when Kerberos TGT expired

[ https://issues.apache.org/jira/browse/HADOOP-15597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang resolved HADOOP-15597. Resolution: Duplicate > UserGroupInformation class throws NPE when Kerberos TGT expi

[jira] [Created] (HADOOP-15597) UserGroupInformation class throws NPE when Kerberos TGT expired

Eric Yang created HADOOP-15597: -- Summary: UserGroupInformation class throws NPE when Kerberos TGT expired Key: HADOOP-15597 URL: https://issues.apache.org/jira/browse/HADOOP-15597 Project: Hadoop Common

[jira] [Created] (HADOOP-15588) Add proxy acl check for AuthenticationFilter

Eric Yang created HADOOP-15588: -- Summary: Add proxy acl check for AuthenticationFilter Key: HADOOP-15588 URL: https://issues.apache.org/jira/browse/HADOOP-15588 Project: Hadoop Common Issue

[jira] [Created] (HADOOP-15587) Securing ASF Hadoop releases out of the box

Eric Yang created HADOOP-15587: -- Summary: Securing ASF Hadoop releases out of the box Key: HADOOP-15587 URL: https://issues.apache.org/jira/browse/HADOOP-15587 Project: Hadoop Common Issue Type

Re: [DISCUSS]: securing ASF Hadoop releases out of the box

+1 on Non-routable IP idea. My preference is to start in Hadoop-common to minimize the scope and incrementally improve. However, this will be incompatible change for initial user experience on public cloud. What would be the right release vehicle for this work (3.2+ or 4.x)? Regards, Eric

Re: [VOTE] reset/force push to clean up inadvertent merge commit pushed to trunk

+1 On 7/5/18, 2:44 PM, "Giovanni Matteo Fumarola" wrote: +1 On Thu, Jul 5, 2018 at 2:41 PM, Wangda Tan wrote: > +1 > > On Thu, Jul 5, 2018 at 2:37 PM Subru Krishnan wrote: > > > Folks, > > > > There was a merge commit accidentally pushed to

Re: [DISCUSS]: securing ASF Hadoop releases out of the box

Hadoop default configuration aimed for user friendliness to increase adoption, and security can be enabled one by one. This approach is most problematic to security because system can be compromised before all security features are turned on. Larry's proposal will add some safety to remind

[jira] [Created] (HADOOP-15284) Could not determine real path of mount

Eric Yang created HADOOP-15284: -- Summary: Could not determine real path of mount Key: HADOOP-15284 URL: https://issues.apache.org/jira/browse/HADOOP-15284 Project: Hadoop Common Issue Type: Bug

[jira] [Reopened] (HADOOP-14077) Improve the patch of HADOOP-13119

[ https://issues.apache.org/jira/browse/HADOOP-14077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang reopened HADOOP-14077: > Improve the patch of HADOOP-13119 > - > >

[jira] [Created] (HADOOP-15222) Refine proxy user authorization to support multiple ACL list

Eric Yang created HADOOP-15222: -- Summary: Refine proxy user authorization to support multiple ACL list Key: HADOOP-15222 URL: https://issues.apache.org/jira/browse/HADOOP-15222 Project: Hadoop Common

Re: When are incompatible changes acceptable (HDFS-12990)

gt; - What if someone is already using 3.0.0 and has changed >> all the scripts to 9820? Just let them fail? >> - Compared to 2.x, 3.0.0 has many incompatible changes. Are >> we going to have other incompatible changes in the future minor and dot &

Re: When are incompatible changes acceptable (HDFS-12990)

, 2018 at 7:36 PM To: Eric Yang <ey...@hortonworks.com> Cc: "Aaron T. Myers" <a...@apache.org>, Daryn Sharp <da...@oath.com>, Hadoop Common <common-dev@hadoop.apache.org>, larry mccay <lmc...@apache.org> Subject: Re: When are incompatible changes acceptable

Re: When are incompatible changes acceptable (HDFS-12990)

ary 10, 2018 at 10:53 AM To: Daryn Sharp <da...@oath.com> Cc: "Aaron T. Myers" <a...@apache.org>, Eric Yang <ey...@hortonworks.com>, Chris Douglas <cdoug...@apache.org>, Hadoop Common <common-dev@hadoop.apache.org> Subject: Re: When are incompatible changes

[jira] [Resolved] (HADOOP-15162) UserGroupInformation.createRemoteUser hardcode authentication method to SIMPLE

[ https://issues.apache.org/jira/browse/HADOOP-15162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang resolved HADOOP-15162. Resolution: Not A Problem Close this as not a problem. Bad assumption for SIMPLE security mode

Re: When are incompatible changes acceptable (HDFS-12990)

See comments inline. Regards, Eric From: <a...@cloudera.com> on behalf of "Aaron T. Myers" <a...@apache.org> Date: Wednesday, January 10, 2018 at 9:21 AM To: Eric Yang <ey...@hortonworks.com> Cc: Chris Douglas <cdoug...@apache.org>, larry mccay <lmc...@ap

Re: When are incompatible changes acceptable (HDFS-12990)

..@apache.org> Date: Tuesday, January 9, 2018 at 9:22 PM To: Eric Yang <ey...@hortonworks.com> Cc: Chris Douglas <cdoug...@apache.org>, larry mccay <lmc...@apache.org>, Hadoop Common <common-dev@hadoop.apache.org> Subject: Re: When are incompatible changes acceptable (HDF

Re: When are incompatible changes acceptable (HDFS-12990)

While I agree the original port change was unnecessary, I don’t think Hadoop NN port change is a bad thing. I worked for a Hadoop distro that NN RPC port was default to port 9000. When we migrate from BigInsights to IOP and now to HDP, we have to move customer Hive metadata to new NN RPC

[jira] [Created] (HADOOP-15162) UserGroupInformation.createRmoteUser hardcode authentication method to SIMPLE

Eric Yang created HADOOP-15162: -- Summary: UserGroupInformation.createRmoteUser hardcode authentication method to SIMPLE Key: HADOOP-15162 URL: https://issues.apache.org/jira/browse/HADOOP-15162 Project

[jira] [Reopened] (HADOOP-10054) ViewFsFileStatus.toString() is broken

[ https://issues.apache.org/jira/browse/HADOOP-10054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang reopened HADOOP-10054: This patch broke trunk development. Please run unit test: {code} mvn clean test -Dtest

Re: Missing some trunk commit history

Friday, December 15, 2017 at 11:06 AM To: Eric Yang <ey...@hortonworks.com> Cc: Chris Douglas <cdoug...@apache.org>, Eric Yang <ey...@apache.org>, Arun Suresh <asur...@apache.org>, Sunil G <sun...@apache.org>, Hadoop Common <common-dev@hadoop.apache.org> Subj

Re: Missing some trunk commit history

+1 for merge –no-ff for feature merge. Do we all agree on this optimization for going forward? Regards, Eric On 12/15/17, 10:34 AM, "Chris Douglas" <cdoug...@apache.org> wrote: On Thu, Dec 14, 2017 at 9:40 PM, Eric Yang <ey...@apache.org> wrote: > I am looking

Re: Missing some trunk commit history

On Thu, Dec 14, 2017 at 6:52 PM, Chris Douglas <cdoug...@apache.org> wrote: > Eric- > > What problem are you trying to solve? Most of us understand how git works, > you can omit that. -C > > On Thu, Dec 14, 2017 at 6:31 PM Eric Yang <ey...@hortonworks.com> wrote: >

Re: Missing some trunk commit history

On Thu, Dec 14, 2017 at 2:31 PM Eric Yang <ey...@hortonworks.com> wrote: > When details are rebased, the number of entries to test through the linear > history is much more than a merge point to isolate where the error might > have occurred. It is similar to trav

Re: Missing some trunk commit history

it so it's clear the branch was developed outside the mainline. -C On Thu, Dec 14, 2017 at 1:18 PM, Eric Yang <ey...@hortonworks.com> wrote: > +1 on squash merge to keep history compressed. The rebase + merge contains good deals, but it is easy to get confused for people

Re: Missing some trunk commit history

he.org> wrote: Another option - atleast for feature branches is to maybe squash merge - this way we see it as a single commit ? Although we will loose the feature branch history (I am ok with that though) Cheers -Arun On Thu, Dec 14, 2017 at 11:32 AM, Eric Yang <ey

Re: Missing some trunk commit history

the root cause of trunk failure was due to merge or some recent commits. Regards, Eric From: Sunil G <sun...@apache.org> Date: Thursday, December 14, 2017 at 11:11 AM To: Eric Yang <ey...@hortonworks.com> Cc: Hadoop Common <common-dev@hadoop.apache.org> Subject: Re: Missing some tr

Missing some trunk commit history

Hi all, While troubleshooting a trunk build failure, I notice the commit history for trunk between Nov 30th to Dec 6th are squashed or disappeared for no reason. This seems to have taken place in the last 24 hours. I can see the commit logs from github UI. When doing a new clone from Apache

Re: [VOTE] Merge yarn-native-services branch into trunk

- YARN-6419[4]. UI support for native-services on the new YARN UI > > All these new services are optional and are sitting outside of the > > existing system, and have no impact on existing system if disabled. > > > > Special thanks to a team of folks who worked

Re: [DISCUSS] Feature Branch Merge and Security Audits

Looks good and +1 for markdown documentations to provide per release specific information. On Sat, Oct 21, 2017 at 8:47 AM, larry mccay wrote: > New Revision... > > This revision acknowledges the reality that we often have multiple phases > of feature lifecycle and that we

Re: [DISCUSS] Feature Branch Merge and Security Audits

The check list looks good. Some more items to add: Kerberos TGT renewal SPNEGO support Delegation token Proxy User ACL CVE tracking list We might want to start a security section for Hadoop wiki for each of the services and components. This helps to track what has been completed. How

[jira] [Created] (HADOOP-14967) Use jetty CORS filter for web interface

Eric Yang created HADOOP-14967: -- Summary: Use jetty CORS filter for web interface Key: HADOOP-14967 URL: https://issues.apache.org/jira/browse/HADOOP-14967 Project: Hadoop Common Issue Type

Re: 2017-10-06 Hadoop 3 release status update

Hi Allen, What if the commands are: yarn application -deploy –f spec.json yarn application -stop yarn application -restart yarn application -remove and yarn application –list will display both application list from RM as well as docker services? I think the development team was concerned

Re: [DISCUSS] HADOOP-9122 Add power mock library for writing better unit tests

On Mon, Oct 2, 2017 at 2:12 PM, Eric Yang <ey...@hortonworks.com> wrote: > Mock provides tool chains to run simulation for a piece of code. It helps to prevent null pointer exception, and reduce unexpected runtime exceptions. When a piece of code is finished with

Re: [DISCUSS] HADOOP-9122 Add power mock library for writing better unit tests

n <ste...@hortonworks.com> Date: Sunday, October 1, 2017 at 12:36 PM To: Eric Yang <ey...@hortonworks.com> Cc: Andrew Wang <andrew.w...@cloudera.com>, Chris Douglas <cdoug...@apache.org>, "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org> Subjec

Re: [DISCUSS] HADOOP-9122 Add power mock library for writing better unit tests

to rethink the improvement that Powermock can bring to the table. Thank you for your considation. Regards, Eric From: Andrew Wang <andrew.w...@cloudera.com> Date: Friday, September 29, 2017 at 1:55 PM To: Chris Douglas <cdoug...@apache.org> Cc: Eric Yang <ey...@hortonworks.co

[DISCUSS] HADOOP-9122 Add power mock library for writing better unit tests

Hi Hadoop-dev, Long time ago, Hadoop community decided to put Powermock on hold for unit tests. Both mockito and powermock has evolved a lot in the past 5 years. There are mature versions of both software, and there are compatibility charts to indicate which versions can work together.

[jira] [Reopened] (HADOOP-9122) Add powermock library for writing better unit tests

[ https://issues.apache.org/jira/browse/HADOOP-9122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang reopened HADOOP-9122: --- Assignee: Eric Yang (was: Radim Kolar) Reopening this old issue. Powermockito is great

[jira] [Reopened] (HADOOP-13119) Web UI authorization error accessing /logs/ when Kerberos

[ https://issues.apache.org/jira/browse/HADOOP-13119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang reopened HADOOP-13119: > Web UI authorization error accessing /logs/ when Kerbe

Re: Possible path bug

Hi Gregorio, ${BASH_SOURCE-$0} is not a bug. That is a short hand notation for locating the script's directory. There are some legacy reasons for this arrangement. $bin in hadoop-daemon[s].sh are designed to run in both development and binary deployment environment. Where $bin is hadoop

Re: Github integration for Hadoop

Hi Andrew, Many Apache projects already have Github integration. Chukwa also has Github integration. Pull request can be integrated into JIRA, i.e. https://issues.apache.org/jira/browse/CHUKWA-783 This allows code review process to happen at per line level on Github, or comment on JIRA for

[jira] [Resolved] (HADOOP-12110) Consolidate usage of JSON libraries

[ https://issues.apache.org/jira/browse/HADOOP-12110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang resolved HADOOP-12110. Resolution: Invalid Opened for the wrong project. Sorry, close as invalid. Consolidate usage

[jira] [Created] (HADOOP-12110) Consolidate usage of JSON libraries

Eric Yang created HADOOP-12110: -- Summary: Consolidate usage of JSON libraries Key: HADOOP-12110 URL: https://issues.apache.org/jira/browse/HADOOP-12110 Project: Hadoop Common Issue Type: Bug

Re: Looking to a Hadoop 3 release

Some lesson learn during 2.x. WebHDFS, HDFS ACL, QJM HA, rolling upgrade are great features. Mapreduce 1.x uses resources more efficiently, containers have rigid constraint, and applications get killed prematurely. When a node has a lot of containers, YARN takes significant amount of system

[jira] [Resolved] (HADOOP-10759) Remove hardcoded JAVA_HEAP_MAX in hadoop-config.sh

[ https://issues.apache.org/jira/browse/HADOOP-10759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang resolved HADOOP-10759. Resolution: Fixed Allen, this JIRA is filed for Hadoop Common, YARN problems can be addressed

Re: [DISCUSS] Stop support for non-packaged i.e. developer build single-node installs

Single node developer build may be supported independently from package installed builds. There are multiple ways to achieve this. The current method is using shell script to support different deployment conditions. This is a awkward and inefficient way to maintain compatibility to support

[jira] [Resolved] (HADOOP-7751) rpm version is not being picked from the -Dversion option in 205

[ https://issues.apache.org/jira/browse/HADOOP-7751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang resolved HADOOP-7751. --- Resolution: Won't Fix For maven, the number number includes -SNAPSHOT for on going builds

[jira] [Created] (HADOOP-7765) Debian package contain both system and tar ball layout

Components: build Affects Versions: 0.20.205.0 Environment: Java, ant Reporter: Eric Yang Assignee: Eric Yang Fix For: 0.20.205.1 When packaging is invoked as ant clean tar deb. The system creates both system layout and tarball layout in the same build

[jira] [Created] (HADOOP-7756) Ability to specify kinit location for setup scripts

Components: scripts Affects Versions: 0.20.205.0 Environment: Linux RHEL6 Reporter: Eric Yang Assignee: Eric Yang Priority: Minor kinit location is in /usr/kerberos/bin in RHEL5, and switched to /usr/bin in RHEL6. Hence, it may be better to support

Re: [VOTE] 0.20.205.0 Release Candidate 2

+1 (non-binding) Exercised Hadoop 0.20.205 with HBase 0.90.4+, Pig 0.9.1+, Chukwa trunk on 20 nodes clusters, works fine. regards, Eric On Oct 13, 2011, at 5:46 PM, Aaron T. Myers wrote: +1 (non-binding) I downloaded the tar ball and deployed on a 6 node cluster with security disabled. I

[jira] [Resolved] (HADOOP-7731) Hadoop 0.20.2-4 Deb Install hangs on Ubuntu 11.04

[ https://issues.apache.org/jira/browse/HADOOP-7731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang resolved HADOOP-7731. --- Resolution: Duplicate Fix Version/s: 0.20.205.0 Assignee: Eric Yang This is fixed

[jira] [Created] (HADOOP-7684) jobhistory server and secondarynamenode should have init.d script

: Bug Components: scripts Affects Versions: 0.20.205.0, 0.23.0 Environment: Java, Linux Reporter: Eric Yang Assignee: Eric Yang Fix For: 0.20.205.0, 0.23.0 The current set of init.d scripts can start/stop process for: namenode datanode

[jira] [Created] (HADOOP-7637) Fair scheduler configuration file is not bundled in RPM

Components: build Affects Versions: 0.20.205.0 Reporter: Eric Yang Assignee: Eric Yang Fix For: 0.20.205.0 205 build of tar is fine, but rpm failed with: {noformat} [rpm] Processing files: hadoop-0.20.205.0-1 [rpm] warning: File listed twice: /usr

[jira] [Created] (HADOOP-7626) Allow overwrite of HADOOP_CLASSPATH and HADOOP_OPTS

Components: scripts Affects Versions: 0.20.205.0 Environment: Java, Linux Reporter: Eric Yang Assignee: Eric Yang Fix For: 0.20.205.0 Quote email from Ashutosh Chauhan: bq. There is a bug in hadoop-env.sh which prevents hcatalog server to start

Re: Hadoop Tools Layout (was Re: DistCpV2 in 0.23)

increase/reduce the risk of dead code and how they make more-difficult/easier the packagedeployment. Would you please explain this? Thanks. Alejandro On Tue, Sep 6, 2011 at 6:38 PM, Eric Yang eric...@gmail.com wrote: Option #2 proposed by Amareshwari, seems like a better proposal.  We don't

  1   2   >