[
https://issues.apache.org/jira/browse/HDDS-1299?focusedWorklogId=218121&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-218121
]
ASF GitHub Bot logged work on HDDS-1299:
----------------------------------------
Author: ASF GitHub Bot
Created on: 25/Mar/19 16:17
Start Date: 25/Mar/19 16:17
Worklog Time Spent: 10m
Work Description: xiaoyuyao commented on issue #627: HDDS-1299. Support
TokenIssuer interface for running jobs with OzoneFileSystem.
URL: https://github.com/apache/hadoop/pull/627#issuecomment-476270295
Thanks Steve for the heads up. Ozone DT via HCFS requires Kerberos setup for
various component to test e2e. There is no easy way to test inside unit test as
OM/SCM/DN all login with different principles.
We have follow up JIRA to add system test.
Before committing the changes, I spend a few days to get the manual steps
with various testing documented here:
%your_hadoop_rep%/hadoop-ozone/dist/src/main/compose/ozonesecure-mr/README.md
This will be the basis of system tests being added, where we may use
different Hadoop-versions like 3.2 and 2.7 to test the combabilities.
The test will be very similar to the one below just slightly different
security setup.
%your_hadoop_rep%/hadoop/hadoop-ozone/dist/src/main/smoketest/security/ozone-secure.robot
Bq. Without tests, when people like myself make an incompatible change which
breaks Ozone's DT support, we're just going to shrug and say "well, you should
have written the tests"
Agree, test will be added. I have a WIP patch for DT system tests but there
is a refactor of Ozone smoke test in
HDDS-1255<https://github.com/apache/hadoop/pull/632>. I will post my test
after that.
HDDS-1299 based on HADOOP-14445 are all Hadoop-3 only changes, it brings
Ozone DT support. Before that, Ozone does not support DT. There is a pending
HADOOP-14445 patch for Hadoop-2.
We could build support for Hadoop-2 after that in a separate ozone branch.
Thanks,
Xiaoyu
From: Steve Loughran <[email protected]>
Reply-To: apache/hadoop <[email protected]>
Date: Monday, March 25, 2019 at 8:58 AM
To: apache/hadoop <[email protected]>
Cc: Xiaoyu Yao <[email protected]>, State change
<[email protected]>
Subject: Re: [apache/hadoop] HDDS-1299. Support TokenIssuer interface for
running jobs with OzoneFileSystem. (#627)
Given this has broken things, can I ask a process question here,
specifically regarding Yetus's v-1
The patch doesn't appear to include any new or modified tests. Please
justify why no new tests are needed for this patch. Also please list what
manual steps were performed to verify this patch.
There are lots of ways to test for DTs being ussued: The hdfs fs command,
the hadoop dtuils, the actualyh M
* Marshalling/unmarshalling of tokens:
org.apache.hadoop.fs.s3a.auth.delegation.TestS3ADelegationTokenSupport
* org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationTokens
* org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationInFileystem
* org.apache.hadoop.fs.azurebfs.extensions.ITestAbfsDelegationTokens
You can even verify that MR cluster launch collects the tokens:
org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob
Please. write tests. Apart from the MR job one its straightforward, and I've
just given you all the clases to start with.
Without tests, when people like myself make an incompatible change which
breaks Ozone's DT support, we're just going to shrug and say "well, you should
have written the tests"
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on
GitHub<https://github.com/apache/hadoop/pull/627#issuecomment-476256841>, or
mute the
thread<https://github.com/notifications/unsubscribe-auth/AGto0K4e1tFGFPgU-8npgy5vjk8phZnUks5vaO-3gaJpZM4b9Vx2>.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 218121)
Time Spent: 6.5h (was: 6h 20m)
> Support TokenIssuer interface for running jobs with OzoneFileSystem
> -------------------------------------------------------------------
>
> Key: HDDS-1299
> URL: https://issues.apache.org/jira/browse/HDDS-1299
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Reporter: Xiaoyu Yao
> Assignee: Xiaoyu Yao
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 0.4.0
>
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
> This ticket is opened to add TokenIssuer interface support to OzoneFileSystem
> so that MR and Spark jobs can run with OzoneFileSystem in secure mode.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
