Re: [Discuss] - HDDS-4 Branch merge

[Jitendra Pandey]

+1, This is a great effort and another milestone in making HDDS and Ozone ready 
for enterprises.

On 1/7/19, 5:10 PM, "Anu Engineer"  wrote:

Hi All,

I would like to propose a merge of HDDS-4 branch to the Hadoop trunk.
HDDS-4 branch implements the security work for HDDS and Ozone.

HDDS-4 branch contains the following features:
- Hadoop Kerberos and Tokens support
- A Certificate infrastructure used by Ozone and HDDS.
- Audit Logging and parsing support (Spread across trunk and HDDS-4)
- S3 Security Support - AWS Signature Support.
- Apache Ranger Support for Ozone

I will follow up with a formal vote later this week if I hear no
objections. AFAIK, the changes are isolated to HDDS/Ozone and should not
impact any other Hadoop project.

Thanks
Anu

[jira] [Created] (HADOOP-16038) Upgrade Spotbugs to the latest version

[Akira Ajisaka (JIRA)]

Akira Ajisaka created HADOOP-16038:
--

 Summary: Upgrade Spotbugs to the latest version
 Key: HADOOP-16038
 URL: https://issues.apache.org/jira/browse/HADOOP-16038
 Project: Hadoop Common
  Issue Type: Bug
  Components: build
Reporter: Akira Ajisaka


The latest version of Spotbugs is 3.1.9 and it can be available via 
spotbugs-maven-plugin.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-16037) DistCp: Document usage of -diff option in detail

[Siyao Meng (JIRA)]

Siyao Meng created HADOOP-16037:
---

 Summary: DistCp: Document usage of -diff option in detail
 Key: HADOOP-16037
 URL: https://issues.apache.org/jira/browse/HADOOP-16037
 Project: Hadoop Common
  Issue Type: Task
  Components: documentation, tools/distcp
Reporter: Siyao Meng
Assignee: Siyao Meng


Create a new doc section similar to "Update and Overwrite" for -diff option. 
Provide step by step guidance.

Current doc link: 
https://hadoop.apache.org/docs/current/hadoop-distcp/DistCp.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Hadoop Contributor Meetup Hosted by LinkedIn

[Erik Krogen]

Hello fellow Hadoopers!


LinkedIn is pleased to be hosting an all-day meetup event for Hadoop 
contributors and community members to gather together to share information and 
insights on the present, past and future of the Hadoop software we all know and 
love. The day will begin with a series of presentations on important new 
features and improvements, followed by breakout discussion sessions on a 
variety of attendee-driven topics. The target audience is Hadoop contributors, 
committers, PMC members, and invested members of the community. Topics are 
generally focused around the future of Hadoop with special focus on scalability 
and integration with the cloud. Lunch and reception refreshments will be 
provided.

If you are interested in joining us, please RSVP at the Meetup page[1] for the 
event.

Looking forward to seeing some familiar faces and meeting many new ones,

Erik
LinkedIn Hadoop Team

[1] https://www.meetup.com/Hadoop-Contributors/events/257793743/

[jira] [Created] (HADOOP-16036) WASB: Disable jetty logging configuration announcement

[Da Zhou (JIRA)]

Da Zhou created HADOOP-16036:


 Summary: WASB: Disable jetty logging configuration announcement
 Key: HADOOP-16036
 URL: https://issues.apache.org/jira/browse/HADOOP-16036
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/azure
Reporter: Da Zhou
Assignee: Da Zhou


Remove the unnecessary logging configuration announcement from WASB console 
output.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-16033) hamcrest-library declaration in hadoop-aws to be scoped test

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HADOOP-16033.
-

latest HADOOP-14556 patch incorporates this

> hamcrest-library declaration in hadoop-aws to be scoped test
> 
>
> Key: HADOOP-16033
> URL: https://issues.apache.org/jira/browse/HADOOP-16033
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: build, hadoop-aws
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
>
> *Good*: HADOOP-14556 uses hamcrest assertThat() for declaring assertions
> *Bad*: this new hamcrest-library declaration is scoped for compile, not test, 
> so becomes a transitive dependency. Needs fixing before a release to keep the 
> cruft out of downstream apps.
> I broke it, I get to fix it



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-16035) test yetus master

[Allen Wittenauer (JIRA)]

Allen Wittenauer created HADOOP-16035:
-

 Summary: test yetus master
 Key: HADOOP-16035
 URL: https://issues.apache.org/jira/browse/HADOOP-16035
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Allen Wittenauer


Just a JIRA for me to test yetus github PR support against hadoop



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-7050) proxyuser host/group config properties don't work if user name as DOT in it

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-7050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar resolved HADOOP-7050.
-
Resolution: Duplicate

> proxyuser host/group config properties don't work if user name as DOT in it
> ---
>
> Key: HADOOP-7050
> URL: https://issues.apache.org/jira/browse/HADOOP-7050
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Reporter: Alejandro Abdelnur
>Priority: Major
>
> If the user contains a DOT, "foo.bar", proxy user configuration fails to be 
> read properly and it does not kick in.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-16034) Yarn registry to create Curator binding on demand

[Steve Loughran (JIRA)]

Steve Loughran created HADOOP-16034:
---

 Summary: Yarn registry to create Curator binding on demand
 Key: HADOOP-16034
 URL: https://issues.apache.org/jira/browse/HADOOP-16034
 Project: Hadoop Common
  Issue Type: Improvement
  Components: registry
Affects Versions: 3.1.1
Reporter: Steve Loughran
Assignee: Steve Loughran


If enough slider worker nodes are created, and then each create a Registry 
client, then ZK can overload even when they aren't making calls of the 
registry. Why so? Because curator gets started up on serviceStart(), even when 
not used.

Proposed fix: make it on-demand on use of a ZK read/write operation



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[VOTE] Release Apache Hadoop 3.2.0 - RC1

[Sunil G]

Hi folks,


Thanks to all of you who helped in this release [1] and for helping to vote
for RC0. I have created second release candidate (RC1) for Apache Hadoop
3.2.0.


Artifacts for this RC are available here:

http://home.apache.org/~sunilg/hadoop-3.2.0-RC1/


RC tag in git is release-3.2.0-RC1.



The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1178/


This vote will run 7 days (5 weekdays), ending on 14th Jan at 11:59 pm PST.



3.2.0 contains 1092 [2] fixed JIRA issues since 3.1.0. Below feature
additions

are the highlights of this release.

1. Node Attributes Support in YARN

2. Hadoop Submarine project for running Deep Learning workloads on YARN

3. Support service upgrade via YARN Service API and CLI

4. HDFS Storage Policy Satisfier

5. Support Windows Azure Storage - Blob file system in Hadoop

6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a

7. Improvements in Router-based HDFS federation



Thanks to Wangda, Vinod, Marton for helping me in preparing the release.

I have done few testing with my pseudo cluster. My +1 to start.



Regards,

Sunil



[1]

https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E

[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
ORDER BY fixVersion ASC

[jira] [Created] (HADOOP-16033) hamcrest-library declaration in hadoop-aws to be scoped test

[Steve Loughran (JIRA)]

Steve Loughran created HADOOP-16033:
---

 Summary: hamcrest-library declaration in hadoop-aws to be scoped 
test
 Key: HADOOP-16033
 URL: https://issues.apache.org/jira/browse/HADOOP-16033
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: hadoop-aws, build
Affects Versions: 3.3.0
Reporter: Steve Loughran
Assignee: Steve Loughran


*Good*: HADOOP-14556 uses hamcrest assertThat() for declaring assertions

*Bad*: this new hamcrest-library declaration is scoped for compile, not test, 
so becomes a transitive dependency. Needs fixing before a release to keep the 
cruft out of downstream apps.

I broke it, I get to fix it



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1010/

[Jan 7, 2019 12:58:09 AM] (wwei) Revert "YARN-9178. TestRMAdminCli#testHelp is 
failing in trunk.
[Jan 7, 2019 7:56:49 AM] (wwei) HADOOP-16030. AliyunOSS: bring fixes back from 
HADOOP-15671. Contributed
[Jan 7, 2019 7:57:31 AM] (wwei) YARN-9173. FairShare calculation broken for 
large values after
[Jan 7, 2019 8:32:28 AM] (aajisaka) YARN-9162. Fix TestRMAdminCLI#testHelp. 
Contributed by Ayush Saxena.
[Jan 7, 2019 10:17:23 AM] (aajisaka) HADOOP-15937. [JDK 11] Update 
maven-shade-plugin.version to 3.2.1.
[Jan 7, 2019 10:27:36 AM] (elek) HDDS-916. MultipartUpload: Complete Multipart 
upload request.
[Jan 7, 2019 11:06:22 AM] (elek) HDDS-901. MultipartUpload: S3 API for Initiate 
multipart upload.
[Jan 7, 2019 11:39:46 AM] (elek) HDDS-902. MultipartUpload: S3 API for 
uploading a part file. Contributed
[Jan 7, 2019 1:18:03 PM] (stevel) HADOOP-14556. S3A to support Delegation 
Tokens.
[Jan 7, 2019 6:21:14 PM] (eyang) HADOOP-16031.  Fixed TestSecureLogins unit 
test.  Contributed by Akira
[Jan 7, 2019 7:09:14 PM] (bharat) HDDS-930. Multipart Upload: Abort multiupload 
request. Contributed by
[Jan 7, 2019 7:19:42 PM] (aengineer) HDDS-946. AuditParser - insert audit to 
database in batches. contributed
[Jan 7, 2019 8:07:26 PM] (wangda) YARN-8822. Nvidia-docker v2 support for YARN 
GPU feature. (Charo Zhang
[Jan 7, 2019 9:06:45 PM] (aengineer) HDDS-931. Add documentation for ozone 
shell command providing ozone
[Jan 7, 2019 11:16:55 PM] (gifuma) YARN-9169. Add metrics for queued 
opportunistic and guaranteed


[Error replacing 'FILE' - Workspace is not accessible]

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [NOTICE] Move to gitbox

[Elek, Marton]

Yes. git.apache.org/hadoop is deceased, deleted by INFRA.

But both the gitbox and github urls are working. If you don't configure
credentials for one of these (or don't link your apache account with
github account) you can use that one as read only repo url.

Marton

((I don't now the history of the git.apache.org. It seems to be a mirror
page only for the old-style+svn repositories))

On 1/7/19 3:18 PM, Steve Loughran wrote:
> OK. so the original git://git.apache.org/hadoop.git is in fact deceased?
> 
> 
> On 7 Jan 2019, at 14:15, Elek, Marton 
> mailto:e...@apache.org>> wrote:
> 
> Please use the gitbox url instead of git.a.o:
> 
> https://gitbox.apache.org/repos/asf/hadoop.git
> 
> Or the github url:
> 
> g...@github.com:apache/hadoop.git
> 
> 
> Note: To use the github url for _push_, you may need to link your github
> account to your apache account: https://gitbox.apache.org/setup/
> 
> Marton
> 
> ps: git.apache.org/hadoop was synced to the old repository and removed
> by INFRA
> 
> 
> On 1/7/19 3:11 PM, Steve Loughran wrote:
> doesn't like me doing a git pull on my R/O view of the repo
> 
>  git remote -v
> apachegit://git.apache.org/hadoop.git (fetch)
> apachegit://git.apache.org/hadoop.git (push)
> 
>  git checkout trunk; and git pull
> Checking out files: 100% (5084/5084), done.
> Switched to branch 'trunk'
> Your branch is up to date with 'apache/trunk'.
> fatal: remote error: access denied or repository not exported: /hadoop.git
> 
> 
> 
> 
> On 7 Jan 2019, at 06:55, Akira Ajisaka  > wrote:
> 
> Thanks Ayush for the report and thanks Elek for the fix!
> 
> -Akira
> 
> 2019年1月3日(木) 0:54 Elek, Marton  >:
> 
> Thanks the report Ayush,
> 
> The bogus repository is removed by the INFRA:
> 
> https://issues.apache.org/jira/browse/INFRA-17526
> 
> And the cwiki page[1] is updated to use the gitbox url instead of
> git.apache.org
> 
> Marton
> 
> [1] https://cwiki.apache.org/confluence/display/HADOOP/Git+And+Hadoop
> 
> On 1/2/19 8:56 AM, Ayush Saxena wrote:
> Hi Akira
> 
> I guess the mirror at git.apache.org/hadoop.git hasn’t been updated
> with the new location.
> 
> It is still pointing to
> https://git-wip-us.apache.org/repos/asf/hadoop.git
> 
> Checking out the source still has its mention
> As
> git clone git://git.apache.org/hadoop.git
> 
> Or this link needs to be updated?
> 
> Can you give a check.
> 
> -Ayush
> 
> On 31-Dec-2018, at 3:33 AM, Akira Ajisaka  wrote:
> 
> Hi all,
> 
> The migration has been finished.
> All the jenkins jobs under the hadoop view and ozone view were also
> updated except the beam_PerformanceTests_* jobs.
> 
> Thank you Elek and ASF infra team for your help!
> 
> Regards,
> Akira
> 
> 2018年12月25日(火) 13:27 Akira Ajisaka :
> 
> Hi all,
> 
> The Apache Hadoop git repository will be migrated to gitbox at 9PM UTC
> in 30th December.
> After the migration, the old repository cannot be accessed. Please use
> the new repository for committing. The migration is pretty much atomic
> and it will take up to a few minutes.
> 
> Old repository: https://git-wip-us.apache.org/repos/asf?p=hadoop.git
> New repository: https://gitbox.apache.org/repos/asf?p=hadoop.git
> 
> The GitHub repository (https://github.com/apache/hadoop) is not
> affected.
> 
> Elek will update the jenkins jobs and I'll update the source code and
> documentation as soon as the migration is finished.
> 
> Discussion:
> https://lists.apache.org/thread.html/8b37cd69191648f1163ee23e3498f33da1c44ac876c6225b429dc835@%3Ccommon-dev.hadoop.apache.org%3E
> 
> JIRA:
> - https://issues.apache.org/jira/browse/HADOOP-16003
> - https://issues.apache.org/jira/browse/INFRA-17448
> 
> Happy Holidays!
> 
> -Akira
> 
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> 
> 
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> 
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> 
> 
> 
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> 
> 

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org