[jira] [Created] (HDFS-15724) Hive should use DFSClient#isHDFSEncryptionEnabled

[Xiaoyu Yao (Jira)]

Xiaoyu Yao created HDFS-15724:
-

 Summary: Hive should use DFSClient#isHDFSEncryptionEnabled
 Key: HDFS-15724
 URL: https://issues.apache.org/jira/browse/HDFS-15724
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.3.0
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


HIVE-23212 and HIVE-23261 adding check for hdfs encryption to determine if copy 
is needed. However, the API used DFSUtilClient.isHDFSEncryptionEnabled is 
solely based on local hadoop.security.key.provider.path configuration. This has 
been evolved into a HDFS internal configuration that may not exist in client 
side. The right API to be called here should be   
DFSClient.isHDFSEncryptionEnabled which will get the keyprovider info from NN 
directly after HDFS-11687 and HDFS-12396. This ticket is opened to fix it. 




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Apache Hadoop Ozone 1.0.0 RC1

[Xiaoyu Yao]

+1 (binding)

* Verify the checksums and signatures.
* Verify basic Ozone file system and S3 operations via CLI in secure docker
compose environment
* Run MR examples and teragen/terasort with ozone secure enabled.
* Verify EN/CN document rendering with hugo serve

Thanks Sammi for driving the release.

Regards,
Xiaoyu

On Mon, Aug 31, 2020 at 8:55 AM Shashikant Banerjee
 wrote:

> +1(binding)
>
> 1.Verified checksums
> 2.Verified signatures
> 3.Verified the output of `ozone version
> 4.Tried creating volume and bucket, write and read key, by Ozone shell
> 5.Verified basic Ozone Filesystem operations
>
> Thank you very much Sammi for putting up the release together.
>
> Thanks
> Shashi
>
> On Mon, Aug 31, 2020 at 4:35 PM Elek, Marton  wrote:
>
> > +1 (binding)
> >
> >
> > 1. verified signatures
> >
> > 2. verified checksums
> >
> > 3. verified the output of `ozone version` (includes the good git
> revision)
> >
> > 4. verified that the source package matches the git tag
> >
> > 5. verified source can be used to build Ozone without previous state
> > (docker run -v ... -it maven ... --> built from the source with zero
> > local maven cache during 16 minutes --> did on a sever at this time)
> >
> > 6. Verified Ozone can be used from binary package (cd compose/ozone &&
> > test.sh --> all tests were passed)
> >
> > 7. Verified documentation is included in SCM UI
> >
> > 8. Deployed to Kubernetes and executed Teragen on Yarn [1]
> >
> > 9. Deployed to Kubernetes and executed Spark (3.0) Word count (local
> > executor) [2]
> >
> > 10. Deployed to Kubernetes and executed Flink Word count [3]
> >
> > 11. Deployed to Kubernetes and executed Nifi
> >
> > Thanks very much Sammi, to drive this release...
> > Marton
> >
> > ps:  NiFi setup requires some more testing. Counters were not updated on
> > the UI and at some cases, I saw DirNotFound exceptions when I used
> > master. But during the last test with -rc1 it worked well.
> >
> > [1]: https://github.com/elek/ozone-perf-env/tree/master/teragen-ozone
> >
> > [2]: https://github.com/elek/ozone-perf-env/tree/master/spark-ozone
> >
> > [3]: https://github.com/elek/ozone-perf-env/tree/master/flink-ozone
> >
> >
> > On 8/25/20 4:01 PM, Sammi Chen wrote:
> > > RC1 artifacts are at:
> > > https://home.apache.org/~sammichen/ozone-1.0.0-rc1/
> > > 
> > >
> > > Maven artifacts are staged at:
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1278
> > > <
> https://repository.apache.org/content/repositories/orgapachehadoop-1277
> > >
> > >
> > > The public key used for signing the artifacts can be found at:
> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >
> > > The RC1 tag in github is at:
> > > https://github.com/apache/hadoop-ozone/releases/tag/ozone-1.0.0-RC1
> > > 
> > >
> > > Change log of RC1, add
> > > 1. HDDS-4063. Fix InstallSnapshot in OM HA
> > > 2. HDDS-4139. Update version number in upgrade tests.
> > > 3. HDDS-4144, Update version info in hadoop client dependency readme
> > >
> > > *The vote will run for 7 days, ending on Aug 31th 2020 at 11:59 pm
> PST.*
> > >
> > > Thanks,
> > > Sammi Chen
> > >
> >
> > -
> > To unsubscribe, e-mail: ozone-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: ozone-dev-h...@hadoop.apache.org
> >
> >
>

[jira] [Resolved] (HDFS-15344) DataNode#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442

[Xiaoyu Yao (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-15344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoyu Yao resolved HDFS-15344.
---
Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

Thanks all for the reviews. The PR has been merged to trunk. 

> DataNode#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442
> 
>
> Key: HDFS-15344
> URL: https://issues.apache.org/jira/browse/HDFS-15344
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Affects Versions: 2.7.5
>        Reporter: Xiaoyu Yao
>    Assignee: Xiaoyu Yao
>Priority: Major
> Fix For: 3.4.0
>
>
> HADOOP-13442 added UGI#getGroups to avoid list->array->list conversions. This 
> ticket is opened to change DataNode#checkSuperuserPrivilege to use 
> UGI#getGroups. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-15345) RouterPermissionChecker#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442

[Xiaoyu Yao (Jira)]

Xiaoyu Yao created HDFS-15345:
-

 Summary:  RouterPermissionChecker#checkSuperuserPrivilege should 
use UGI#getGroups after HADOOP-13442
 Key: HDFS-15345
 URL: https://issues.apache.org/jira/browse/HDFS-15345
 Project: Hadoop HDFS
  Issue Type: Improvement
Affects Versions: 2.7.5
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


HADOOP-13442 added UGI#getGroups to avoid list->array->list conversions. This 
ticket is opened to change DataNode#checkSuperuserPrivilege to use 
UGI#getGroups. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-15344) DataNode#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442

[Xiaoyu Yao (Jira)]

Xiaoyu Yao created HDFS-15344:
-

 Summary: DataNode#checkSuperuserPrivilege should use UGI#getGroups 
after HADOOP-13442
 Key: HDFS-15344
 URL: https://issues.apache.org/jira/browse/HDFS-15344
 Project: Hadoop HDFS
  Issue Type: Improvement
Affects Versions: 2.7.5
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


HADOOP-13442 added UGI#getGroups to avoid list->array->list conversions. This 
ticket is opened to change DataNode#checkSuperuserPrivilege to use 
UGI#getGroups. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Apache Hadoop Ozone 0.5.0-beta RC2

[Xiaoyu Yao]

+1 binding
Download source and verify signature.
Verify build and documents.
Deployed an 11 node cluster (3 om with ha, 6 datanodes, 1 scm and 1 s3g)
Verify multiple RATIS-3 pipelines are created as expected.
Tried ozone shell commands via o3 and o3fs, focus on security and HA
related.
 Only find a few minor issues that we can fix in followup JIRAs.
1) ozone getconf -ozonemanagers does not return all the om instances
bash-4.2$ ozone getconf -ozonemanagers
0.0.0.0
2) The document on specifying service/ID can be improved. More
specifically, the URI should give examples for the Service ID in HA.
Currently, it only mentions host/port.

ozone sh vol create /vol1
Service ID or host name must not be omitted when ozone.om.service.ids is
defined.
bash-4.2$ ozone sh vol create --help
Usage: ozone sh volume create [-hV] [--root] [-q=] [-u=]

Creates a volume for the specified user
 URI of the volume.
  Ozone URI could start with o3:// or without
prefix. URI
may contain the host and port of the OM server.
Both are
optional. If they are not specified it will be
identified from the config files.
3). ozone scmcli container list seems report incorrect numberOfKeys and
usedBytes
Also, container owner is set as the current leader om(om3), should we use
the om service id here instead?
bash-4.2$ ozone scmcli container list
{
  "state" : "OPEN",
  "replicationFactor" : "THREE",
  "replicationType" : "RATIS",
  "usedBytes" : 3813,
  "numberOfKeys" : 1,
...
bash-4.2$ ozone sh key list o3://id1/vol1/bucket1/
{
  "volumeName" : "vol1",
  "bucketName" : "bucket1",
  "name" : "k1",
  "dataSize" : 3813,
  "creationTime" : "2020-03-23T03:23:30.670Z",
  "modificationTime" : "2020-03-23T03:23:33.207Z",
  "replicationType" : "RATIS",
  "replicationFactor" : 3
}
{
  "volumeName" : "vol1",
  "bucketName" : "bucket1",
  "name" : "k2",
  "dataSize" : 3813,
  "creationTime" : "2020-03-23T03:18:46.735Z",
  "modificationTime" : "2020-03-23T03:20:15.005Z",
  "replicationType" : "RATIS",
  "replicationFactor" : 3
}


Run freon with random key generation.

Thanks Dinesh for driving the the release of Beta RC2.

Xiaoyu

On Sun, Mar 22, 2020 at 2:51 PM Aravindan Vijayan
 wrote:

> +1
> Deployed a 3 node cluster
> Tried ozone shell and filesystem commands
> Ran freon load generator
>
> Thanks Dinesh for working on the RC2.
>
> On Sun, Mar 15, 2020 at 7:27 PM Dinesh Chitlangia 
> wrote:
>
> > Hi Folks,
> >
> > We have put together RC2 for Apache Hadoop Ozone 0.5.0-beta.
> >
> > The RC artifacts are at:
> > https://home.apache.org/~dineshc/ozone-0.5.0-rc2/
> >
> > The public key used for signing the artifacts can be found at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > The maven artifacts are staged at:
> > https://repository.apache.org/content/repositories/orgapachehadoop-1262
> >
> > The RC tag in git is at:
> > https://github.com/apache/hadoop-ozone/tree/ozone-0.5.0-beta-RC2
> >
> > This release contains 800+ fixes/improvements [1].
> > Thanks to everyone who put in the effort to make this happen.
> >
> > *The vote will run for 7 days, ending on March 22nd 2020 at 11:59 pm
> PST.*
> >
> > Note: This release is beta quality, it’s not recommended to use in
> > production but we believe that it’s stable enough to try out the feature
> > set and collect feedback.
> >
> >
> > [1] https://s.apache.org/ozone-0.5.0-fixed-issues
> >
> > Thanks,
> > Dinesh Chitlangia
> >
>

[ANNOUNCE] New Apache Hadoop Committer - Siyao Meng

[Xiaoyu Yao]

It's my pleasure to announce that Siyao Meng has been elected as committer
on the Apache Hadoop project recognizing his continued contributions to the
project.

Please join me in congratulating him.

Congratulations & Welcome aboard Siyao!

Xiaoyu Yao
(On behalf of the Hadoop PMC)

[jira] [Created] (HDFS-15192) Leaking stream when access encrypted files hit exception via webhdfs

[Xiaoyu Yao (Jira)]

Xiaoyu Yao created HDFS-15192:
-

 Summary: Leaking stream when access encrypted files hit exception 
via webhdfs
 Key: HDFS-15192
 URL: https://issues.apache.org/jira/browse/HDFS-15192
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.3.0
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


This was found during fixing for HADOOP-16885. Given the code is in HDFS, the 
ticket is opened to address the similar issue, i.e., inner stream is not closed 
when wrapper stream create failed. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-15061) KMSClientProvider does not work with client using ticket logged in externally

[Xiaoyu Yao (Jira)]

Xiaoyu Yao created HDFS-15061:
-

 Summary: KMSClientProvider does not work with client using ticket 
logged in externally 
 Key: HDFS-15061
 URL: https://issues.apache.org/jira/browse/HDFS-15061
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


This is a regression from HDFS-13682 that checks not only the kerberos 
credential but also enforce the login is non-external. This breaks client 
applications that need to access HDFS encrypted file using kerberos ticket that 
logged in external in ticket cache. 

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-2375) Refactor BlockOutputStream to allow flexible buffering

[Xiaoyu Yao (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDDS-2375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoyu Yao resolved HDDS-2375.
--
Resolution: Fixed

Thanks [~szetszwo] for the contribution and all for the reviews. I've merged 
the PR to master. 

> Refactor BlockOutputStream to allow flexible buffering
> --
>
> Key: HDDS-2375
> URL: https://issues.apache.org/jira/browse/HDDS-2375
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: Ozone Client
>Reporter: Tsz-wo Sze
>Assignee: Tsz-wo Sze
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> In HDDS-2331, we found that Ozone client allocates a ByteBuffer with chunk 
> size (e.g. 16MB ) to store data, unregarded the actual data size.  The 
> ByteBuffer will create a  byte[] with chunk size.  When the ByteBuffer is 
> wrapped to a ByteString the byte[] remains in the ByteString.
> As a result, when the actual data size is small (e.g. 1MB), a lot of memory 
> spaces (15MB) are wasted.
> In this JIRA, we refactor BlockOutputStream so that the buffering becomes 
> more flexible.  In a later JIRA (HDDS-2386), we implement chunk buffer using 
> a list of smaller buffers which are allocated only if needed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2518) Ensure RATIS leader info is properly updated with pipeline report.

[Xiaoyu Yao (Jira)]

Xiaoyu Yao created HDDS-2518:


 Summary: Ensure RATIS leader info is properly updated with 
pipeline report. 
 Key: HDDS-2518
 URL: https://issues.apache.org/jira/browse/HDDS-2518
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


HDDS-2034 added async pipeline creation and report handling to SCM. The leader 
information is not properly populated as manifested in the test failures from 
TestSCMPipelineManager#testPipelineReport. This ticket is opened to fix it. cc: 
[~sammichen]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2497) SafeMode check should allow key creation on single node pipeline when replication factor is 1

[Xiaoyu Yao (Jira)]

Xiaoyu Yao created HDDS-2497:


 Summary: SafeMode check should allow key creation on single node 
pipeline when replication factor is 1
 Key: HDDS-2497
 URL: https://issues.apache.org/jira/browse/HDDS-2497
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Xiaoyu Yao


Start a single datanode ozone docker-compose with replication factor of 1.
{code:java}
OZONE-SITE.XML_ozone.replication=1{code}
The key creation failed with Safemode exception below. 
{code:java}
>$ docker-compose exec om bash
bash-4.2$ ozone sh vol create /vol1
bash-4.2$ ozone sh bucket create /vol1/bucket1
ozone sh kbash-4.2$ ozone sh key put /vol1/bucket1/key1 README.md
SCM_IN_SAFE_MODE SafeModePrecheck failed for allocateBlock{code}
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2496) Delegate Ozone volume create/list ACL check to authorizer plugin

[Xiaoyu Yao (Jira)]

Xiaoyu Yao created HDDS-2496:


 Summary: Delegate Ozone volume create/list ACL check to authorizer 
plugin
 Key: HDDS-2496
 URL: https://issues.apache.org/jira/browse/HDDS-2496
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Affects Versions: 0.4.1
Reporter: Vivek Ratnavel Subramanian
Assignee: Vivek Ratnavel Subramanian


Today Ozone volume create/list ACL check are not sent to authorization plugins. 
This cause problem when authorization plugin is enabled. Admin still need to 
modify ozone-site.xml to change ozone.administrators to configure admin to 
create volume

 

This ticket is opened to have a consistent ACL check for all Ozone resources 
requests including admin request like volume create. This way, the admin 
defined by the authorization plugin can be honored during volume provision 
without restart ozone services. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2492) Fix test clean up issue in TestSCMPipelineManager

[Xiaoyu Yao (Jira)]

Xiaoyu Yao created HDDS-2492:


 Summary: Fix test clean up issue in TestSCMPipelineManager
 Key: HDDS-2492
 URL: https://issues.apache.org/jira/browse/HDDS-2492
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Sammi Chen
Assignee: Li Cheng


This was opened based on [~sammichen]'s investigation on HDDS-2034.

 
{quote}Failure is caused by newly introduced function 
TestSCMPipelineManager#testPipelineOpenOnlyWhenLeaderReported which doesn't 
close pipelineManager at the end. It's better to fix it in a new JIRA.
{quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2491) Fix TestSCMSafeModeWithPipelineRules

[Xiaoyu Yao (Jira)]

Xiaoyu Yao created HDDS-2491:


 Summary: Fix TestSCMSafeModeWithPipelineRules
 Key: HDDS-2491
 URL: https://issues.apache.org/jira/browse/HDDS-2491
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Affects Versions: 0.4.1
Reporter: Xiaoyu Yao


This was based on [~sammichen]'s investigation on HDDS-2034.

 
{quote}The root cause is failing to exit the safemode. Current pipeline open 
condition(HDDS-1868) is got 3 datanode reports and one datanode marked itself 
as leader. In this failure case, the leader election succeeds while 
XceiverServerRatis#handleLeaderChangedNotification is not called in the next 3 
minutes. So cluster.waitForClusterToBeReady() timeout.
The question is is this Leader change notification reliable? What's the 
expected latency between leader election succeed and notification send?
{quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2463) Remove unnecessary getServiceInfo calls

[Xiaoyu Yao (Jira)]

Xiaoyu Yao created HDDS-2463:


 Summary: Remove unnecessary getServiceInfo calls
 Key: HDDS-2463
 URL: https://issues.apache.org/jira/browse/HDDS-2463
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Affects Versions: 0.4.1
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


OzoneManagerProtocolClientSideTranslatorPB.java Line 766-772 has multiple 
impl.getServiceInfo() which can be reduced by adding a local variable. 
{code:java}
 
resp.addAllServiceInfo(impl.getServiceInfo().getServiceInfoList().stream()
 .map(ServiceInfo::getProtobuf)
 .collect(Collectors.toList()));
if (impl.getServiceInfo().getCaCertificate() != null) {
 resp.setCaCertificate(impl.getServiceInfo().getCaCertificate()); {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2321) Ozone Block Token verify should not apply to all datanode cmd

[Xiaoyu Yao (Jira)]

Xiaoyu Yao created HDDS-2321:


 Summary: Ozone Block Token verify should not apply to all datanode 
cmd
 Key: HDDS-2321
 URL: https://issues.apache.org/jira/browse/HDDS-2321
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Affects Versions: 0.4.1
Reporter: Nilotpal Nandi
Assignee: Xiaoyu Yao


DN container protocol has cmd send from SCM or other DN, which do not bear OM 
block token like OM client. We should restrict the OM Block token check only 
for those issued from OM client. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-2295) Display log of freon on the standard output

[Xiaoyu Yao (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDDS-2295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoyu Yao resolved HDDS-2295.
--
Fix Version/s: 0.5.0
   Resolution: Fixed

Thanks [~elek] for the contribution and all for the reviews. I've merged the 
changes.

> Display log of freon on the standard output
> ---
>
> Key: HDDS-2295
> URL: https://issues.apache.org/jira/browse/HDDS-2295
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>Reporter: Marton Elek
>Assignee: Marton Elek
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.5.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> HDDS-2042 disabled the console logging for all of the ozone command line 
> tools including freon.
> But freon is different, it has a different error handling model. For freon we 
> need all the log on the console.
>  1. To follow all the different errors
>  2. To get information about the used (random) prefix which can be reused 
> during the validation phase.
>  
> I propose to restore the original behavior for Ozone.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-2298) Fix maven warning about duplicated metrics-core jar

[Xiaoyu Yao (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDDS-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoyu Yao resolved HDDS-2298.
--
Fix Version/s: 0.5.0
   Resolution: Fixed

Thanks [~elek] for the contribution and all for the reviews. The change has 
been merged.

> Fix maven warning about duplicated metrics-core jar
> ---
>
> Key: HDDS-2298
> URL: https://issues.apache.org/jira/browse/HDDS-2298
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: build
>Reporter: Marton Elek
>Assignee: Marton Elek
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.5.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Maven build of Ozone is starting with a warning:
> {code:java}
> [WARNING] 
> [WARNING] Some problems were encountered while building the effective model 
> for org.apache.hadoop:hadoop-ozone-tools:jar:0.5.0-SNAPSHOT
> [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must 
> be unique: io.dropwizard.metrics:metrics-core:jar -> version 3.2.4 vs (?) @ 
> line 94, column 17
> [WARNING] 
> [WARNING] It is highly recommended to fix these problems because they 
> threaten the stability of your build.
> [WARNING] 
> [WARNING] For this reason, future Maven versions might no longer support 
> building such malformed projects.
> [WARNING] 
>  {code}
> It's better to avoid it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2304) ozone token cli output can be improved.

[Xiaoyu Yao (Jira)]

Xiaoyu Yao created HDDS-2304:


 Summary: ozone token cli output can be improved.
 Key: HDDS-2304
 URL: https://issues.apache.org/jira/browse/HDDS-2304
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


Some output does not start a new line at the end.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop Ozone 0.4.1-alpha

[Xiaoyu Yao]

+1 binding. Verified
* Verify the signature.
* Build from source.
* Deploy docker compose in secure mode and verify ACL, sample MR jobs

Thanks,
Xiaoyu

On Fri, Oct 11, 2019 at 5:37 PM Hanisha Koneru 
wrote:

> Thank you Nanda for putting up the RC.
>
> +1 binding.
>
> Verified the following:
>   - Built from source
>   - Deployed to 5 node cluster and ran smoke tests.
>   - Ran sanity checks
>
> Thanks
> Hanisha
>
> > On Oct 4, 2019, at 10:42 AM, Nanda kumar  wrote:
> >
> > Hi Folks,
> >
> > I have put together RC0 for Apache Hadoop Ozone 0.4.1-alpha.
> >
> > The artifacts are at:
> > https://home.apache.org/~nanda/ozone/release/0.4.1/RC0/
> >
> > The maven artifacts are staged at:
> > https://repository.apache.org/content/repositories/orgapachehadoop-1238/
> >
> > The RC tag in git is at:
> > https://github.com/apache/hadoop/tree/ozone-0.4.1-alpha-RC0
> >
> > And the public key used for signing the artifacts can be found at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > This release contains 363 fixes/improvements [1].
> > Thanks to everyone who put in the effort to make this happen.
> >
> > *The vote will run for 7 days, ending on October 11th at 11:59 pm IST.*
> > Note: This release is alpha quality, it’s not recommended to use in
> > production but we believe that it’s stable enough to try out the feature
> > set and collect feedback.
> >
> >
> > [1] https://s.apache.org/yfudc
> >
> > Thanks,
> > Team Ozone
>
>

[jira] [Created] (HDDS-2282) scmcli pipeline list command throws NullPointerException

[Xiaoyu Yao (Jira)]

Xiaoyu Yao created HDDS-2282:


 Summary: scmcli pipeline list command throws NullPointerException
 Key: HDDS-2282
 URL: https://issues.apache.org/jira/browse/HDDS-2282
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Nilotpal Nandi
Assignee: Xiaoyu Yao


ozone scmcli pipeline list
{noformat}
java.lang.NullPointerException
at 
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:187)
at 
org.apache.hadoop.hdds.scm.XceiverClientManager.(XceiverClientManager.java:98)
at 
org.apache.hadoop.hdds.scm.XceiverClientManager.(XceiverClientManager.java:83)
at 
org.apache.hadoop.hdds.scm.cli.SCMCLI.createScmClient(SCMCLI.java:139)
at 
org.apache.hadoop.hdds.scm.cli.pipeline.ListPipelinesSubcommand.call(ListPipelinesSubcommand.java:55)
at 
org.apache.hadoop.hdds.scm.cli.pipeline.ListPipelinesSubcommand.call(ListPipelinesSubcommand.java:30)
at picocli.CommandLine.execute(CommandLine.java:1173)
at picocli.CommandLine.access$800(CommandLine.java:141)
at picocli.CommandLine$RunLast.handle(CommandLine.java:1367)
at picocli.CommandLine$RunLast.handle(CommandLine.java:1335)
at 
picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1243)
at picocli.CommandLine.parseWithHandlers(CommandLine.java:1526)
at picocli.CommandLine.parseWithHandler(CommandLine.java:1465)
at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:65)
at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:56)
at org.apache.hadoop.hdds.scm.cli.SCMCLI.main(SCMCLI.java:101){noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2228) Fix NPE in OzoneDelegationTokenManager#addPersistedDelegationToken

[Xiaoyu Yao (Jira)]

Xiaoyu Yao created HDDS-2228:


 Summary: Fix NPE in 
OzoneDelegationTokenManager#addPersistedDelegationToken
 Key: HDDS-2228
 URL: https://issues.apache.org/jira/browse/HDDS-2228
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Affects Versions: 0.4.0
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


The certClient was not initialized in proper order as a result, when OM restart 
with delegation token issued, the ozone delegation token secret manager NPE. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2213) Reduce key provider loading log level in OzoneFileSystem#getAdditionalTokenIssuers

[Xiaoyu Yao (Jira)]

Xiaoyu Yao created HDDS-2213:


 Summary: Reduce key provider loading log level in 
OzoneFileSystem#getAdditionalTokenIssuers
 Key: HDDS-2213
 URL: https://issues.apache.org/jira/browse/HDDS-2213
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Reporter: Vivek Ratnavel Subramanian


OzoneFileSystem#getAdditionalTokenIssuers log an error when secure client tries 
to collect ozone delegation token to run MR/Spark jobs but ozone file system 
does not have a kms provider configured. In this case, we simply return null 
provider here in the code below. This is a benign error and we should reduce 
the log level to debug level.

 \{code}

KeyProvider keyProvider;
try {
 keyProvider = getKeyProvider();
} catch (IOException ioe) {
 LOG.error("Error retrieving KeyProvider.", ioe);
 return null;
}

{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2172) Ozone shell should remove description about REST protocol support

[Xiaoyu Yao (Jira)]

Xiaoyu Yao created HDDS-2172:


 Summary: Ozone shell should remove description about REST protocol 
support
 Key: HDDS-2172
 URL: https://issues.apache.org/jira/browse/HDDS-2172
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Xiaoyu Yao


{code}

Usage: ozone sh key put [-hV] [-r=]  
creates or overwrites an existing key
  URI of the volume/bucket/key.
 Ozone URI could start with o3:// or http(s):// or without prefix.
 REST protocol will be used for http(s), RPC otherwise. URI may
 contain the host and port of the OM server. Both are optional.
 If they are not specified it will be identified from the config
 files.

{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2160) Add acceptance test for ozonesecure-mr compose

[Xiaoyu Yao (Jira)]

Xiaoyu Yao created HDDS-2160:


 Summary: Add acceptance test for ozonesecure-mr compose
 Key: HDDS-2160
 URL: https://issues.apache.org/jira/browse/HDDS-2160
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


This will give us coverage of running basic MR jobs on security enabled OZONE 
cluster against YARN. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-1553) Add metrics in rack aware container placement policy

[Xiaoyu Yao (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoyu Yao resolved HDDS-1553.
--
Fix Version/s: 0.5.0
   Resolution: Fixed

Thanks [~Sammi] for the contribution. I merged the change to trunk.

> Add metrics in rack aware container placement policy
> 
>
> Key: HDDS-1553
> URL: https://issues.apache.org/jira/browse/HDDS-1553
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Sammi Chen
>Assignee: Sammi Chen
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.5.0
>
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> To collect following statistics, 
> 1. total requested datanode count (A)
> 2. success allocated datanode count without constrain compromise (B)
> 3. success allocated datanode count with some comstrain compromise (C)
> B includes C, failed allocation = (A - B)



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2099) Refactor to create pipeline via DN heartbeat response

[Xiaoyu Yao (Jira)]

Xiaoyu Yao created HDDS-2099:


 Summary: Refactor to create pipeline via DN heartbeat response
 Key: HDDS-2099
 URL: https://issues.apache.org/jira/browse/HDDS-2099
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Reporter: Xiaoyu Yao


Currently, SCM directly talk to DN GRPC server to create pipeline in a 
background thread. We should avoid direct communication from SCM to DN for 
better scalability of ozone. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-14828) Add TeraSort to acceptance test

[Xiaoyu Yao (Jira)]

Xiaoyu Yao created HDFS-14828:
-

 Summary: Add TeraSort to acceptance test
 Key: HDFS-14828
 URL: https://issues.apache.org/jira/browse/HDFS-14828
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Xiaoyu Yao


We may begin with 1GB teragen/terasort/teravalidate.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2096) Ozone ACL document missing AddAcl API

[Xiaoyu Yao (Jira)]

Xiaoyu Yao created HDDS-2096:


 Summary: Ozone ACL document missing AddAcl API
 Key: HDDS-2096
 URL: https://issues.apache.org/jira/browse/HDDS-2096
 Project: Hadoop Distributed Data Store
  Issue Type: Test
Reporter: Xiaoyu Yao


Current Ozone Native ACL APIs document looks like below, the AddAcl is missing.

 
h3. Ozone Native ACL APIs

The ACLs can be manipulated by a set of APIs supported by Ozone. The APIs 
supported are:
 # *SetAcl* – This API will take user principal, the name, type of the ozone 
object and a list of ACLs.
 # *GetAcl* – This API will take the name and type of the ozone object and will 
return a list of ACLs.
 # *RemoveAcl* - This API will take the name, type of the ozone object and the 
ACL that has to be removed.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-2047) Datanodes fail to come up after 10 retries in a secure environment

[Xiaoyu Yao (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDDS-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoyu Yao resolved HDDS-2047.
--
Fix Version/s: 0.4.1
   Resolution: Fixed

Thanks all for the reviews and discussions. I've merged the PR to trunk and 
ozone-0.4.1.

> Datanodes fail to come up after 10 retries in a secure environment
> --
>
> Key: HDDS-2047
> URL: https://issues.apache.org/jira/browse/HDDS-2047
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: Ozone Datanode, Security
>Affects Versions: 0.4.1
>Reporter: Vivek Ratnavel Subramanian
>Assignee: Xiaoyu Yao
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.4.1
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> {code:java}
> 10:06:36.585 PMERRORHddsDatanodeService
> Error while storing SCM signed certificate.
> java.net.ConnectException: Call From 
> jmccarthy-ozone-secure-2.vpc.cloudera.com/10.65.50.127 to 
> jmccarthy-ozone-secure-1.vpc.cloudera.com:9961 failed on connection 
> exception: java.net.ConnectException: Connection refused; For more details 
> see:  http://wiki.apache.org/hadoop/ConnectionRefused
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:755)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1515)
> at org.apache.hadoop.ipc.Client.call(Client.java:1457)
> at org.apache.hadoop.ipc.Client.call(Client.java:1367)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy15.getDataNodeCertificate(Unknown Source)
> at 
> org.apache.hadoop.hdds.protocolPB.SCMSecurityProtocolClientSideTranslatorPB.getDataNodeCertificateChain(SCMSecurityProtocolClientSideTranslatorPB.java:156)
> at 
> org.apache.hadoop.ozone.HddsDatanodeService.getSCMSignedCert(HddsDatanodeService.java:278)
> at 
> org.apache.hadoop.ozone.HddsDatanodeService.initializeCertificateClient(HddsDatanodeService.java:248)
> at 
> org.apache.hadoop.ozone.HddsDatanodeService.start(HddsDatanodeService.java:211)
> at 
> org.apache.hadoop.ozone.HddsDatanodeService.start(HddsDatanodeService.java:168)
> at 
> org.apache.hadoop.ozone.HddsDatanodeService.call(HddsDatanodeService.java:143)
> at 
> org.apache.hadoop.ozone.HddsDatanodeService.call(HddsDatanodeService.java:70)
> at picocli.CommandLine.execute(CommandLine.java:1173)
> at picocli.CommandLine.access$800(CommandLine.java:141)
> at picocli.CommandLine$RunLast.handle(CommandLine.java:1367)
> at picocli.CommandLine$RunLast.handle(CommandLine.java:1335)
> at 
> picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1243)
> at picocli.CommandLine.parseWithHandlers(CommandLine.java:1526)
> at picocli.CommandLine.parseWithHandler(CommandLine.java:1465)
> at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:65)
> at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:56)
> at 
> org.apache.hadoop.ozone.HddsDatanodeService.main(HddsDatanodeService.java:126)
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at 
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
> at 
> org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:690)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:794)
> at org.apache.hadoop.ipc.Client$Connection.access$3700(Client.java:411)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1572)
> at org.apache.hadoop.ipc.Client.call(Client.java:1403)
> ... 21 more
> {code}
> Datanodes try to get SCM signed certificate for just 10 times with interval 
> of 1 sec. When SCM takes a little long

[jira] [Created] (HDFS-14791) Fix TestOzoneManagerRatisServer failure

[Xiaoyu Yao (Jira)]

Xiaoyu Yao created HDFS-14791:
-

 Summary: Fix TestOzoneManagerRatisServer failure
 Key: HDFS-14791
 URL: https://issues.apache.org/jira/browse/HDFS-14791
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao






--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2039) Some ozone unit test takes too long to finish.

[Xiaoyu Yao (Jira)]

Xiaoyu Yao created HDDS-2039:


 Summary: Some ozone unit test takes too long to finish.
 Key: HDDS-2039
 URL: https://issues.apache.org/jira/browse/HDDS-2039
 Project: Hadoop Distributed Data Store
  Issue Type: Test
Reporter: Xiaoyu Yao


Here are a few \{code}

[INFO] Running org.apache.hadoop.ozone.om.TestOzoneManagerHA
[INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 436.08 
s - in org.apache.hadoop.ozone.om.TestOzoneManagerHA
[INFO] Running org.apache.hadoop.ozone.om.TestOzoneManager
[INFO] Tests run: 26, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 259.566 
s - in org.apache.hadoop.ozone.om.TestOzoneManager
[INFO] Running org.apache.hadoop.ozone.om.TestScmSafeMode
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 129.653 
s - in org.apache.hadoop.ozone.om.TestScmSafeMode
[INFO] Running org.apache.hadoop.ozone.om.TestOzoneManagerRestart
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 843.129 
s - in org.apache.hadoop.ozone.om.TestOzoneManagerRestart

{code}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2020) Remove mTLS from Ozone GRPC

[Xiaoyu Yao (Jira)]

Xiaoyu Yao created HDDS-2020:


 Summary: Remove mTLS from Ozone GRPC
 Key: HDDS-2020
 URL: https://issues.apache.org/jira/browse/HDDS-2020
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


Generic GRPC support mTLS for mutual authentication. However, Ozone has built 
in block token mechanism for server to authenticate the client. We only need 
TLS for client to authenticate the server and wire encryption. 

Remove the mTLS support also simplify the GRPC server/client configuration.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1946) CertificateClient should not persist keys/certs to ozone.metadata.dir

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1946:


 Summary: CertificateClient should not persist keys/certs to 
ozone.metadata.dir
 Key: HDDS-1946
 URL: https://issues.apache.org/jira/browse/HDDS-1946
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Reporter: Xiaoyu Yao
Assignee: Vivek Ratnavel Subramanian


For example, when OM and SCM are deployed on the same host with 
ozone.metadata.dir defined. SCM can start successfully but OM can not because 
the key/cert from OM will collide with SCM.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1944) Update document for Ozone HTTP SPNEGO authentication

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1944:


 Summary: Update document for Ozone HTTP SPNEGO authentication
 Key: HDDS-1944
 URL: https://issues.apache.org/jira/browse/HDDS-1944
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Affects Versions: 0.4.0
Reporter: Xiaoyu Yao






--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1943) TestKeyManagerImpl.testLookupKeyWithLocation is failing

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1943:


 Summary: TestKeyManagerImpl.testLookupKeyWithLocation is failing
 Key: HDDS-1943
 URL: https://issues.apache.org/jira/browse/HDDS-1943
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Reporter: Xiaoyu Yao


{code}
[ERROR]   TestKeyManagerImpl.testLookupKeyWithLocation:757 
expected:<102ad7e3-4226-4966-af79-2b12a56f83cb{ip: 32.53.16.224, host: 
localhost-32.53.16.224, networkLocation: /default-rack, certSerialId: null}> 
but was:
{code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1930) Test Topology Aware Job scheduling with Ozone Topology

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1930:


 Summary: Test Topology Aware Job scheduling with Ozone Topology
 Key: HDDS-1930
 URL: https://issues.apache.org/jira/browse/HDDS-1930
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Reporter: Xiaoyu Yao


My initial results with Terasort does not seem to report the counter properly. 
Most of the requests are handled by rack locl but no node local. This ticket is 
opened to add more system testing to validate the feature. 

Total Allocated Containers: 3778
Each table cell represents the number of NodeLocal/RackLocal/OffSwitch 
containers satisfied by NodeLocal/RackLocal/OffSwitch resource requests.
Node Local Request  Rack Local Request  Off Switch Request
Num Node Local Containers (satisfied by)0   
Num Rack Local Containers (satisfied by)0   3648
Num Off Switch Containers (satisfied by)0   96  34



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-1865) Use "ozone.network.topology.aware.read" to control both RPC client and server side logic

[Xiaoyu Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoyu Yao resolved HDDS-1865.
--
   Resolution: Fixed
Fix Version/s: 0.5.0

Thanks [~Sammi] for the contribution. I've merged the patch to trunk.

> Use "ozone.network.topology.aware.read" to control both RPC client and server 
> side logic 
> -
>
> Key: HDDS-1865
> URL: https://issues.apache.org/jira/browse/HDDS-1865
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Sammi Chen
>Assignee: Sammi Chen
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.5.0
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1901) Fix Ozone HTTP WebConsole Authentication

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1901:


 Summary: Fix Ozone HTTP WebConsole Authentication
 Key: HDDS-1901
 URL: https://issues.apache.org/jira/browse/HDDS-1901
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Affects Versions: 0.4.0
Reporter: Vivek Ratnavel Subramanian
Assignee: Xiaoyu Yao


This was found during integration testing where the http authentication is 
enabled but anonymous can still access the ozone http web console like scm:9876 
or om:9874. This can be reproed with the following configurations added to the 
ozonesecure docker-compose.

{code}

CORE-SITE.XML_hadoop.http.authentication.simple.anonymous.allowed=false

CORE-SITE.XML_hadoop.http.authentication.signature.secret.file=/etc/security/http_secret

CORE-SITE.XML_hadoop.http.authentication.type=kerberos

CORE-SITE.XML_hadoop.http.authentication.kerberos.principal=HTTP/_h...@example.com

CORE-SITE.XML_hadoop.http.authentication.kerberos.keytab=/etc/security/keytabs/HTTP.keytab

CORE-SITE.XML_hadoop.http.filter.initializers=org.apache.hadoop.security.AuthenticationFilterInitializer

{code}

After debugging into the KerberosAuthenticationFilter, the root cause is the 
name of the keytab does not follow the AuthenticationFilter tradition. The fix 
is to change 

hdds.scm.http.kerberos.keytab.file to hdds.scm.http.kerberos.keytab and
hdds.om.http.kerberos.keytab.file to hdds.om.http.kerberos.keytab

I will also add an integration test for this under ozonesecure docker-compose. 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1896) Suppress WARN log from NetworkTopology#getDistanceCost

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1896:


 Summary: Suppress WARN log from NetworkTopology#getDistanceCost 
 Key: HDDS-1896
 URL: https://issues.apache.org/jira/browse/HDDS-1896
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


When RackAwareness is enabled and client from outside, the distance calculation 
flood SCM log with the following messages. This ticket is opened to suppress 
the WARN log.

{code}

2019-08-01 23:08:05,011 WARN org.apache.hadoop.hdds.scm.net.NetworkTopology: 
One of the nodes is outside of network topology
2019-08-01 23:08:05,011 WARN org.apache.hadoop.hdds.scm.net.NetworkTopology: 
One of the nodes is outside of network topology
2019-08-01 23:08:05,011 WARN org.apache.hadoop.hdds.scm.net.NetworkTopology: 
One of the nodes is outside of network topology

{code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1894) Support listPipelines by filters in scmcli

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1894:


 Summary: Support listPipelines by filters in scmcli
 Key: HDDS-1894
 URL: https://issues.apache.org/jira/browse/HDDS-1894
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Reporter: Xiaoyu Yao
Assignee: Junjie Chen


Today scmcli has a subcmd that allow list all pipelines. This ticket is opened 
to filter the results by switches, e.g., filter by Factor: THREE and State: 
OPEN. This will be useful for trouble shooting in large cluster.

 

{code}

bin/ozone scmcli listPipelines

Pipeline[ Id: a8d1b0c9-e1d4-49ea-8746-3f61dfb5ee3f, Nodes: 
cce44fde-bc8d-4063-97b3-6f557af756e1\{ip: 10.17.112.65, host: 
ia0230.halxg.cloudera.com, networkLocation: /default-rack, certSerialId: null}, 
Type:RATIS, Factor:ONE, State:OPEN]
Pipeline[ Id: c9c453d1-d74c-4414-b87f-1d3585d78a7c, Nodes: 
0b7b0b93-8323-4b82-8cc0-a9a5c10ab827\{ip: 10.17.112.29, host: 
ia0138.halxg.cloudera.com, networkLocation: /default-rack, certSerialId: 
null}c756a0e0-5a1b-4d03-ba5b-cafbcabac877\{ip: 10.17.112.27, host: 
ia0134.halxg.cloudera.com, networkLocation: /default-rack, certSerialId: 
null}bee45bd7-1ee6-4726-b3d1-81476dc1eb49\{ip: 10.17.112.28, host: 
ia0136.halxg.cloudera.com, networkLocation: /default-rack, certSerialId: null}, 
Type:RATIS, Factor:THREE, State:OPEN]

{code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-1857) YARN fails on mapreduce in Kerberos enabled cluster

[Xiaoyu Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoyu Yao resolved HDDS-1857.
--
Resolution: Not A Problem

> YARN fails on mapreduce in Kerberos enabled cluster
> ---
>
> Key: HDDS-1857
> URL: https://issues.apache.org/jira/browse/HDDS-1857
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>Reporter: Eric Yang
>        Assignee: Xiaoyu Yao
>Priority: Blocker
>
> When configured Ozone as secure cluster, running mapreduce job on secure YARN 
> produces this error message:
> {code}
> 2019-07-23 19:33:12,168 INFO retry.RetryInvocationHandler: 
> com.google.protobuf.ServiceException: java.io.IOException: DestHost:destPort 
> eyang-1.openstacklocal:9862 , LocalHost:localPort 
> eyang-1.openstacklocal/172.26.111.17:0. Failed on local exception: 
> java.io.IOException: Couldn't set up IO streams: 
> java.util.ServiceConfigurationError: org.apache.hadoop.security.SecurityInfo: 
> Provider org.apache.hadoop.yarn.server.RMNMSecurityInfoClass not a subtype, 
> while invoking $Proxy13.submitRequest over 
> nodeId=null,nodeAddress=eyang-1.openstacklocal:9862 after 9 failover 
> attempts. Trying to failover immediately.
> 2019-07-23 19:33:12,174 ERROR ha.OMFailoverProxyProvider: Failed to connect 
> to OM. Attempted 10 retries and 10 failovers
> 2019-07-23 19:33:12,176 ERROR client.OzoneClientFactory: Couldn't create 
> protocol class org.apache.hadoop.ozone.client.rpc.RpcClient exception: 
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:291)
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:169)
> at 
> org.apache.hadoop.fs.ozone.BasicOzoneClientAdapterImpl.(BasicOzoneClientAdapterImpl.java:137)
> at 
> org.apache.hadoop.fs.ozone.BasicOzoneClientAdapterImpl.(BasicOzoneClientAdapterImpl.java:101)
> at 
> org.apache.hadoop.fs.ozone.BasicOzoneClientAdapterImpl.(BasicOzoneClientAdapterImpl.java:86)
> at 
> org.apache.hadoop.fs.ozone.OzoneClientAdapterImpl.(OzoneClientAdapterImpl.java:34)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at 
> org.apache.hadoop.fs.ozone.OzoneClientAdapterFactory.lambda$createAdapter$1(OzoneClientAdapterFactory.java:66)
> at 
> org.apache.hadoop.fs.ozone.OzoneClientAdapterFactory.createAdapter(OzoneClientAdapterFactory.java:116)
> at 
> org.apache.hadoop.fs.ozone.OzoneClientAdapterFactory.createAdapter(OzoneClientAdapterFactory.java:62)
> at 
> org.apache.hadoop.fs.ozone.OzoneFileSystem.createAdapter(OzoneFileSystem.java:98)
> at 
> org.apache.hadoop.fs.ozone.BasicOzoneFileSystem.initialize(BasicOzoneFileSystem.java:144)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3338)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:136)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3387)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3355)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:497)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:245)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:481)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
> at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:352)
> at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:250)
> at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:233)
> at 
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:104)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:177)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:327)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.jav

[jira] [Created] (HDDS-1845) OMVolumeSetQuotaRequest#validateAndUpdateCache should not return OMVolumeCreateResponse

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1845:


 Summary: OMVolumeSetQuotaRequest#validateAndUpdateCache should not 
return OMVolumeCreateResponse
 Key: HDDS-1845
 URL: https://issues.apache.org/jira/browse/HDDS-1845
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Xiaoyu Yao


OMVolumeSetQuotaRequest#validateAndUpdateCache Line 115, we should return 

OMVolumeSetQuotaResponse in the failure case.

 

{code}

return new OMVolumeCreateResponse(null, null,
 createErrorOMResponse(omResponse, ex));

{code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1839) Change topology sorting related logs in Pipeline from INFO to DEBUG

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1839:


 Summary: Change topology sorting related logs in Pipeline from 
INFO to DEBUG
 Key: HDDS-1839
 URL: https://issues.apache.org/jira/browse/HDDS-1839
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Affects Versions: 0.4.1
Reporter: Xiaoyu Yao
Assignee: Junjie Chen


This will avoid output like 

{code}

2019-07-19 22:36:40 INFO  Pipeline:342 - Serialize nodesInOrder 
[610d4084-7cce-4691-b43a-f9dd5cdb8809\{ip: 192.168.144.3, host: 
ozonesecure-mr_datanode_1.ozonesecure-mr_default, networkLocation: 
/default-rack, certSerialId: null}] in pipeline 
PipelineID=f9ba269c-aba9-4a42-946c-4048d02cb7d1

2019-07-19 22:36:40 INFO  Pipeline:342 - Deserialize nodesInOrder 
[610d4084-7cce-4691-b43a-f9dd5cdb8809\{ip: 192.168.144.3, host: 
ozonesecure-mr_datanode_1.ozonesecure-mr_default, networkLocation: 
/default-rack, certSerialId: null}] in pipeline 
PipelineID=f9ba269c-aba9-4a42-946c-4048d02cb7d1

{code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-1824) IllegalArgumentException in NetworkTopologyImpl causes SCM to shutdown

[Xiaoyu Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoyu Yao resolved HDDS-1824.
--
Resolution: Duplicate

> IllegalArgumentException in NetworkTopologyImpl causes SCM to shutdown
> --
>
> Key: HDDS-1824
> URL: https://issues.apache.org/jira/browse/HDDS-1824
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: SCM
>Reporter: Lokesh Jain
>Priority: Blocker
>
>  
>  
> {code:java}
> 2019-07-18 02:22:18,005 ERROR 
> org.apache.hadoop.hdds.scm.container.ReplicationManager: Exception in 
> Replication Monitor Thread.
> java.lang.IllegalArgumentException: Affinity node /default-rack/10.17.213.25 
> is not a member of topology
> at 
> org.apache.hadoop.hdds.scm.net.NetworkTopologyImpl.checkAffinityNode(NetworkTopologyImpl.java:780)
> at 
> org.apache.hadoop.hdds.scm.net.NetworkTopologyImpl.chooseRandom(NetworkTopologyImpl.java:408)
> at 
> org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRackAware.chooseNode(SCMContainerPlacementRackAware.java:242)
> at 
> org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRackAware.chooseDatanodes(SCMContainerPlacementRackAware.java:168)
> at 
> org.apache.hadoop.hdds.scm.container.ReplicationManager.handleUnderReplicatedContainer(ReplicationManager.java:487)
> at 
> org.apache.hadoop.hdds.scm.container.ReplicationManager.processContainer(ReplicationManager.java:293)
> at 
> java.util.concurrent.ConcurrentHashMap$KeySetView.forEach(ConcurrentHashMap.java:4649)
> at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)
> at 
> org.apache.hadoop.hdds.scm.container.ReplicationManager.run(ReplicationManager.java:205)
> at java.lang.Thread.run(Thread.java:745)
> 2019-07-18 02:22:18,008 INFO org.apache.hadoop.util.ExitUtil: Exiting with 
> status 1: java.lang.IllegalArgumentException: Affinity node 
> /default-rack/10.17.213.25 is not a member of topology
> 2019-07-18 02:22:18,010 INFO 
> org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter: 
> SHUTDOWN_MSG:
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-1770) SCM crashes when ReplicationManager is trying to re-replicate under replicated containers

[Xiaoyu Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoyu Yao resolved HDDS-1770.
--
Resolution: Duplicate

> SCM crashes when ReplicationManager is trying to re-replicate under 
> replicated containers
> -
>
> Key: HDDS-1770
> URL: https://issues.apache.org/jira/browse/HDDS-1770
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: SCM
>Reporter: Nanda kumar
>Priority: Blocker
>
> SCM crashes with the following exception when ReplicationManager is trying to 
> re-replicate under replicated containers
> {noformat}
> 2019-07-08 12:46:36 ERROR ReplicationManager:215 - Exception in Replication 
> Monitor Thread.
> java.lang.IllegalArgumentException: Affinity node /default-rack/aab15e2d07cc 
> is not a member of topology
> at 
> org.apache.hadoop.hdds.scm.net.NetworkTopologyImpl.checkAffinityNode(NetworkTopologyImpl.java:767)
> at 
> org.apache.hadoop.hdds.scm.net.NetworkTopologyImpl.chooseRandom(NetworkTopologyImpl.java:407)
> at 
> org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRackAware.chooseNode(SCMContainerPlacementRackAware.java:242)
> at 
> org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRackAware.chooseDatanodes(SCMContainerPlacementRackAware.java:168)
> at 
> org.apache.hadoop.hdds.scm.container.ReplicationManager.handleUnderReplicatedContainer(ReplicationManager.java:487)
> at 
> org.apache.hadoop.hdds.scm.container.ReplicationManager.processContainer(ReplicationManager.java:293)
> at 
> java.base/java.util.concurrent.ConcurrentHashMap$KeySetView.forEach(ConcurrentHashMap.java:4698)
> at 
> java.base/java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1083)
> at 
> org.apache.hadoop.hdds.scm.container.ReplicationManager.run(ReplicationManager.java:205)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2019-07-08 12:46:36 INFO  ExitUtil:210 - Exiting with status 1: 
> java.lang.IllegalArgumentException: Affinity node /default-rack/aab15e2d07cc 
> is not a member of topology
> 2019-07-08 12:46:36 INFO  StorageContainerManagerStarter:51 - SHUTDOWN_MSG: 
> /
> SHUTDOWN_MSG: Shutting down StorageContainerManager at 
> 8c763563f672/192.168.112.2
> /
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-1813) Fix false warning from ozones3 acceptance test

[Xiaoyu Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoyu Yao resolved HDDS-1813.
--
Resolution: Fixed

> Fix false warning from ozones3 acceptance test
> --
>
> Key: HDDS-1813
> URL: https://issues.apache.org/jira/browse/HDDS-1813
> Project: Hadoop Distributed Data Store
>  Issue Type: Test
>Affects Versions: 0.4.1
>        Reporter: Xiaoyu Yao
>    Assignee: Xiaoyu Yao
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 0.5.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> All acceptance passed but the results are marked failed due to the following 
> warnings.
> [https://ci.anzix.net/job/ozone/17381/RobotTests/log.html]
> {code}
> [ WARN ] Collapsing consecutive whitespace during parsing is deprecated. Fix 
> ' # Bucket already is created in Test Setup.' in file 
> '/opt/hadoop/smoketest/s3/bucketcreate.robot' on line 31.
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1813) Fix false warning from ozones3 acceptance test

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1813:


 Summary: Fix false warning from ozones3 acceptance test
 Key: HDDS-1813
 URL: https://issues.apache.org/jira/browse/HDDS-1813
 Project: Hadoop Distributed Data Store
  Issue Type: Test
Affects Versions: 0.4.1
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


All acceptance passed but the results are marked failed due to the following 
warnings.

[https://ci.anzix.net/job/ozone/17381/RobotTests/log.html]

{code}

[ WARN ] Collapsing consecutive whitespace during parsing is deprecated. Fix ' 
# Bucket already is created in Test Setup.' in file 
'/opt/hadoop/smoketest/s3/bucketcreate.robot' on line 31.

{code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1801) Make Topology Aware Replication/Read non-default for ozone 0.4.1

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1801:


 Summary: Make Topology Aware Replication/Read non-default for 
ozone 0.4.1   
 Key: HDDS-1801
 URL: https://issues.apache.org/jira/browse/HDDS-1801
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Affects Versions: 0.4.1
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


This helps stablize the ozone-0.4.1 release and fix HDDS-1705, HDDS-1751, 
HDDS-1713 and HDDS-1770 for 0.5. 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-1586) Allow Ozone RPC client to read with topology awareness

[Xiaoyu Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoyu Yao resolved HDDS-1586.
--
   Resolution: Fixed
Fix Version/s: 0.5.0

Thanks [~Sammi] for the contribution. I've committed the patch to trunk. 

> Allow Ozone RPC client to read with topology awareness
> --
>
> Key: HDDS-1586
> URL: https://issues.apache.org/jira/browse/HDDS-1586
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>    Reporter: Xiaoyu Yao
>Assignee: Sammi Chen
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.5.0
>
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> The idea is to leverage the node location from the block locations and perfer 
> read from closer block replicas. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1776) Fix typos in ozone docker-compose files

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1776:


 Summary: Fix typos in ozone docker-compose files
 Key: HDDS-1776
 URL: https://issues.apache.org/jira/browse/HDDS-1776
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


The docker compose file has invalid reference to scm images, which fails the 
docker-compose up with errors like below. This ticket is opened to fix them.

 
{code:java}
ERROR: no such image: apache/ozone-runner::20190617-2: invalid reference format}

or 

ERROR: no such image: apache/ozone-runner:latest:20190617-2: invalid reference 
format{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1713) ReplicationManager fail to find proper node topology based on Datanode details from heartbeat

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1713:


 Summary: ReplicationManager fail to find proper node topology 
based on Datanode details from heartbeat
 Key: HDDS-1713
 URL: https://issues.apache.org/jira/browse/HDDS-1713
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


DN does not have the topology info included in its heartbeat message for 
container report/pipeline report.

SCM is where the topology information is available. During the processing of 
heartbeat, we should not rely on the datanodedetails from report to choose 
datanodes for close container. Otherwise, all the datanode locations of 
existing container replicas will fallback to /default-rack.

 

The fix is to retrieve the corresponding datanode locations from scm 
nodemanager, which has authoritative network topology information. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1695) XceiverClientGrpc#ManagedChannelReference shutdown issue when running TeraGen

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1695:


 Summary: XceiverClientGrpc#ManagedChannelReference shutdown issue 
when running TeraGen
 Key: HDDS-1695
 URL: https://issues.apache.org/jira/browse/HDDS-1695
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Xiaoyu Yao


{code}

Jun 17, 2019 8:45:22 AM 
org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference
 cleanQueue
SEVERE: *~*~*~ Channel ManagedChannelImpl\{logId=1, target=10.17.207.52:9859} 
was not shutdown properly!!! ~*~*~*
 Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() 
returns true.
java.lang.RuntimeException: ManagedChannel allocation site
 at 
org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.(ManagedChannelOrphanWrapper.java:103)
 at 
org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:53)
 at 
org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:44)
 at 
org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:411)
 at 
org.apache.hadoop.hdds.scm.XceiverClientGrpc.connectToDatanode(XceiverClientGrpc.java:172)
 at 
org.apache.hadoop.hdds.scm.XceiverClientGrpc.reconnect(XceiverClientGrpc.java:417)
 at 
org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandAsync(XceiverClientGrpc.java:366)
 at 
org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:279)
 at 
org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:248)
 at 
org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:231)
 at 
org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:118)
 at 
org.apache.hadoop.hdds.scm.storage.BlockInputStream.getChunkInfos(BlockInputStream.java:167)
 at 
org.apache.hadoop.hdds.scm.storage.BlockInputStream.initialize(BlockInputStream.java:118)
 at 
org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:222)
 at 
org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:171)
 at 
org.apache.hadoop.fs.ozone.OzoneFSInputStream.read(OzoneFSInputStream.java:52)
 at java.io.DataInputStream.read(DataInputStream.java:149)
 at 
org.apache.hadoop.examples.terasort.TeraInputFormat$TeraRecordReader.nextKeyValue(TeraInputFormat.java:257)
 at 
org.apache.hadoop.examples.terasort.TeraInputFormat$1.run(TeraInputFormat.java:154)

{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-1671) Multiple unit test fails because of assertion while validating Acls

[Xiaoyu Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoyu Yao resolved HDDS-1671.
--
Resolution: Fixed

This has been fixed in HDDS-1545. 

> Multiple unit test fails because of assertion while validating Acls
> ---
>
> Key: HDDS-1671
> URL: https://issues.apache.org/jira/browse/HDDS-1671
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: Security
>Reporter: Mukul Kumar Singh
>Priority: Blocker
>
> There are multiple unit test failures because of assertion in validateAcls
> https://builds.apache.org/job/hadoop-multibranch/job/PR-846/7/testReport/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-1587) Support dynamically adding delegated class to filteredclass loader

[Xiaoyu Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoyu Yao resolved HDDS-1587.
--
   Resolution: Fixed
Fix Version/s: 0.4.1

Thanks [~junjie] for the contribution. I've merged the change to trunk. 

> Support dynamically adding delegated class to filteredclass loader
> --
>
> Key: HDDS-1587
> URL: https://issues.apache.org/jira/browse/HDDS-1587
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>Affects Versions: 0.4.0
>        Reporter: Xiaoyu Yao
>Assignee: Junjie Chen
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.4.1
>
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> HDDS-922 added a filtered class loader with a list of delegated classes that 
> will be loaded with the app launcher's classloader. With security enabled on 
> ozone-0.4, there are some incompatible changes from Hadoop-common and 
> hadoop-auth module from Hadoop-2.x to Hadoop-3.x. Some examples can be seen 
> HDDS-1080, where the fix has to be made along with a rebuild/release. 
>  
> This ticket is opened to allow dynamically adding delegated classes or class 
> prefix via environment variable. This way, we can easily adjust the setting 
> in different deployment without rebuild/release.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-1663) Add datanode to network topology cluster during node register

[Xiaoyu Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoyu Yao resolved HDDS-1663.
--
   Resolution: Fixed
Fix Version/s: 0.4.1

Thanks [~Sammi] for the contribution. I've commit the patch to trunk. 

> Add datanode to network topology cluster during node register
> -
>
> Key: HDDS-1663
> URL: https://issues.apache.org/jira/browse/HDDS-1663
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Sammi Chen
>Assignee: Sammi Chen
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.4.1
>
>  Time Spent: 7h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-1612) Add "scmcli printTopology" shell command to print datanode topology

[Xiaoyu Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoyu Yao resolved HDDS-1612.
--
   Resolution: Fixed
Fix Version/s: 0.4.1

Thanks [~Sammi] for the contribution. I've committed the patch to trunk. 

> Add "scmcli printTopology" shell command to print datanode topology 
> 
>
> Key: HDDS-1612
> URL: https://issues.apache.org/jira/browse/HDDS-1612
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Sammi Chen
>Assignee: Sammi Chen
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.4.1
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Command "ozone scmcli printTopology".  It will show nodes of all kinds of 
> state(HEALTHY,STALE,DEAD,DECOMMISSIONING,DECOMMISSIONED). 
> Here is an example,
> {noformat}
> State = HEALTHY
> ozone_datanode_1.ozone_default/172.18.0.3/default-rack
> ozone_datanode_2.ozone_default/172.18.0.2/default-rack
> ozone_datanode_3.ozone_default/172.18.0.4/default-rack
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1652) HddsDispatcher should not shutdown volumeSet

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1652:


 Summary: HddsDispatcher should not shutdown volumeSet
 Key: HDDS-1652
 URL: https://issues.apache.org/jira/browse/HDDS-1652
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


Currently both OzoneContainer#stop() and HddsDispatcher#stop() both invoke 
volumeSet.shutdown() explicitly.

 

In addition, OzoneContainer#stop() will invoke HddsDispatcher#stop(). Since the 
volume set object is created by OzoneContainer object, it should be the 
responsibility of OzoneContainer to shutdown. This ticket is opened to remove 
the volumeSet.shutdown() from HddsDispatcher#stop(). 

 

There are benchmark tools relies on HddsDispatcher#stop() to shutdown volumeSet 
object, that we could fix with explict volumeSet#shutdown call. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1650) Fix Ozone tests leaking volume checker thread

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1650:


 Summary: Fix Ozone tests leaking volume checker thread
 Key: HDDS-1650
 URL: https://issues.apache.org/jira/browse/HDDS-1650
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


There are a few test leaking hdds volume checker thread. This ticket is opened 
to fix them. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-1637) Fix random test failure TestSCMContainerPlacementRackAware

[Xiaoyu Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoyu Yao resolved HDDS-1637.
--
   Resolution: Fixed
Fix Version/s: 0.4.1

Thanks [~Sammi] for fixing this. PR has been merged to trunk. 

> Fix random test failure TestSCMContainerPlacementRackAware
> --
>
> Key: HDDS-1637
> URL: https://issues.apache.org/jira/browse/HDDS-1637
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>    Reporter: Xiaoyu Yao
>Assignee: Sammi Chen
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.4.1
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> This has been seen randomly in latest trunk CI, e.g., 
> [https://ci.anzix.net/job/ozone/16980/testReport/org.apache.hadoop.hdds.scm.container.placement.algorithms/TestSCMContainerPlacementRackAware/testFallback/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1637) Fix random test failure TestSCMContainerPlacementRackAware

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1637:


 Summary: Fix random test failure TestSCMContainerPlacementRackAware
 Key: HDDS-1637
 URL: https://issues.apache.org/jira/browse/HDDS-1637
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


This has been seen randomly in latest trunk CI, e.g., 
[https://ci.anzix.net/job/ozone/16980/testReport/org.apache.hadoop.hdds.scm.container.placement.algorithms/TestSCMContainerPlacementRackAware/testFallback/]

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1611) Evaluate ACL and DACL on volume bucket key and perfix to authorize access

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1611:


 Summary: Evaluate ACL and DACL on volume bucket key and perfix to 
authorize access 
 Key: HDDS-1611
 URL: https://issues.apache.org/jira/browse/HDDS-1611
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Reporter: Xiaoyu Yao






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1608) Persist Ozone Prefix ACLs in OM metadata table

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1608:


 Summary: Persist Ozone Prefix ACLs in OM metadata table
 Key: HDDS-1608
 URL: https://issues.apache.org/jira/browse/HDDS-1608
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-1604) ContainerReader#initializeUsedBytes leaks DB reference

[Xiaoyu Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoyu Yao resolved HDDS-1604.
--
   Resolution: Fixed
Fix Version/s: 0.4.1

Thanks all for the reviews. I've committed the patch to trunk. 

> ContainerReader#initializeUsedBytes leaks DB reference
> --
>
> Key: HDDS-1604
> URL: https://issues.apache.org/jira/browse/HDDS-1604
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>    Reporter: Xiaoyu Yao
>        Assignee: Xiaoyu Yao
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.4.1
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> This was caught by the New ContainerCache with reference counting from 
> HDDS-1449. The root cause is an unclosed KeyValueBlockIterator from 
> ContainerReader#initializeUsedBytes.
> I will post a patch shortly, which will fix some UT failures exposed by 
> -HDDS-1449,- such as TestBCSID#testBCSID, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-1536) testSCMSafeModeRestrictedOp is failing consistently

[Xiaoyu Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoyu Yao resolved HDDS-1536.
--
Resolution: Fixed

Thanks [~msingh] for reporting the issue and [~bharatviswa] for the review. 
I've committed the patch to trunk. 

> testSCMSafeModeRestrictedOp is failing consistently
> ---
>
> Key: HDDS-1536
> URL: https://issues.apache.org/jira/browse/HDDS-1536
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: SCM
>Reporter: Mukul Kumar Singh
>    Assignee: Xiaoyu Yao
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.4.1
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The test is failing with the following stack trace.
> {code}
> [ERROR] 
> testSCMSafeModeRestrictedOp(org.apache.hadoop.ozone.om.TestScmSafeMode)  Time 
> elapsed: 9.79 s  <<< FAILURE!
> java.lang.AssertionError
>   at org.junit.Assert.fail(Assert.java:86)
>   at org.junit.Assert.assertTrue(Assert.java:41)
>   at org.junit.Assert.assertFalse(Assert.java:64)
>   at org.junit.Assert.assertFalse(Assert.java:74)
>   at 
> org.apache.hadoop.ozone.om.TestScmSafeMode.testSCMSafeModeRestrictedOp(TestScmSafeMode.java:304)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1604) ContainerReader#initializeUsedBytes leaks DB reference

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1604:


 Summary: ContainerReader#initializeUsedBytes leaks DB reference
 Key: HDDS-1604
 URL: https://issues.apache.org/jira/browse/HDDS-1604
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


This was caught by the New ContainerCache with reference counting from 
HDDS-1449. The root cause is an unclosed KeyValueBlockIterator from 
ContainerReader#initializeUsedBytes.

I will post a patch shortly, which will fix some UT failures exposed by 
HDDS-1449.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1587) Support dynamically adding delegated class to filteredclass loader

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1587:


 Summary: Support dynamically adding delegated class to 
filteredclass loader
 Key: HDDS-1587
 URL: https://issues.apache.org/jira/browse/HDDS-1587
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Affects Versions: 0.4.0
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


HDDS-922 added a filtered class loader with a list of delegated classes that 
will be loaded with the app launcher's classloader. With security enabled on 
ozone-0.4, there are some incompatible changes from Hadoop-common and 
hadoop-auth module from Hadoop-2.x to Hadoop-3.x. Some examples can be seen 
HDDS-1080, where the fix has to be made along with a rebuild/release. 

 

This ticket is opened to allow dynamically adding delegated classes or class 
prefix via environment variable. This way, we can easily adjust the setting in 
different deployment without rebuild/release.

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1586) Allow Ozone RPC client to read with topology awareness

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1586:


 Summary: Allow Ozone RPC client to read with topology awareness
 Key: HDDS-1586
 URL: https://issues.apache.org/jira/browse/HDDS-1586
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Reporter: Xiaoyu Yao


The idea is to leverage the node location from the block locations and perfer 
read from closer block replicas. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-4) Implement security for Hadoop Distributed Storage Layer

[Xiaoyu Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoyu Yao resolved HDDS-4.
---
Resolution: Fixed

Resolved as all subtasks are completed and merged.

> Implement security for Hadoop Distributed Storage Layer 
> 
>
> Key: HDDS-4
> URL: https://issues.apache.org/jira/browse/HDDS-4
> Project: Hadoop Distributed Data Store
>  Issue Type: New Feature
>  Components: Security
>Reporter: Anu Engineer
>    Assignee: Xiaoyu Yao
>Priority: Major
> Attachments: HadoopStorageLayerSecurity.pdf
>
>
> In HDFS-7240, we have created a scalable block layer that facilitates 
> separation of namespace and block layer.  Hadoop Distributed Storage Layer 
> (HDSL) allows us to scale HDFS(HDFS-10419) and as well as create ozone 
> (HDFS-13074).
> This JIRA is an umbrella JIRA that tracks the security-related work items for 
> Hadoop Distributed Storage Layer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: VOTE: Hadoop Ozone 0.4.0-alpha RC2

[Xiaoyu Yao]

+1 Binding. Thanks all who contributed to the release. 

+ Download sources and verify signature.
+ Build from source and ran docker-based ad-hot security tests.
++ From 1 datanode scale to 3 datanodes, verify certificates were correctly 
issued when security enabled
++ Smoke test for both non-secure and secure mode.
++ Put/Get/Delete/Rename Key with 
+++ Kerberos testing
+++ Delegation token testing with DTUtil CLI and MR jobs.
+++ S3 token.

Just have one minor question for the expanded source code which points to 
hadoop-3.3.0-SNAPSHOT-src-with-hdds/hadoop-ozone. But in hadoop-ozone/pom.xml, 
we explicitly declare dependency on Hadoop 3.2.0.
I understand we just take the trunk source code(3.3.0-SNAPSHOT up to the 
ozone-0.4 RC) here, should we fix this by giving the git hash of the trunk or 
clarify it to avoid confusion? 
This might be done by just updating the name of the binaries without reset the 
release itself. 

-Xiaoyu
 

On 5/3/19, 4:07 PM, "Dinesh Chitlangia"  wrote:

+1 (non-binding)

- Built from sources and ran smoke test
- Verified all checksums
- Toggled audit log and verified audit parser tool

Thanks Ajay for organizing the release.

Cheers,
Dinesh



On 5/3/19, 5:42 PM, "Eric Yang"  wrote:

+1

On 4/29/19, 9:05 PM, "Ajay Kumar"  wrote:

Hi All,



We have created the third release candidate (RC2) for Apache Hadoop 
Ozone 0.4.0-alpha.



This release contains security payload for Ozone. Below are some 
important features in it:



  *   Hadoop Delegation Tokens and Block Tokens supported for Ozone.
  *   Transparent Data Encryption (TDE) Support - Allows data 
blocks to be encrypted-at-rest.
  *   Kerberos support for Ozone.
  *   Certificate Infrastructure for Ozone  - Tokens use PKI 
instead of shared secrets.
  *   Datanode to Datanode communication secured via mutual TLS.
  *   Ability secure ozone cluster that works with Yarn, Hive, and 
Spark.
  *   Skaffold support to deploy Ozone clusters on K8s.
  *   Support S3 Authentication Mechanisms like - S3 v4 
Authentication protocol.
  *   S3 Gateway supports Multipart upload.
  *   S3A file system is tested and supported.
  *   Support for Tracing and Profiling for all Ozone components.
  *   Audit Support - including Audit Parser tools.
  *   Apache Ranger Support in Ozone.
  *   Extensive failure testing for Ozone.

The RC artifacts are available at 
https://home.apache.org/~ajay/ozone-0.4.0-alpha-rc2/



The RC tag in git is ozone-0.4.0-alpha-RC2 (git hash 
4ea602c1ee7b5e1a5560c6cbd096de4b140f776b)



Please try 
out,
 vote, or just give us feedback.



The vote will run for 5 days, ending on May 4, 2019, 04:00 UTC.



Thank you very much,

Ajay



-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org




-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: VOTE: Hadoop Ozone 0.4.0-alpha RC1

[Xiaoyu Yao]

+1 (binding)

- Build from source
- Misc security tests with docker compose
- MR and Spark sample jobs with secure ozone cluster

—Xiaoyu

> On Apr 19, 2019, at 3:40 PM, Anu Engineer  
> wrote:
> 
> +1 (Binding)
> 
> -- Verified the checksums.
> -- Built from sources.
> -- Sniff tested the functionality.
> 
> --Anu
> 
> 
> On Mon, Apr 15, 2019 at 4:09 PM Ajay Kumar 
> wrote:
> 
>> Hi all,
>> 
>> We have created the second release candidate (RC1) for Apache Hadoop Ozone
>> 0.4.0-alpha.
>> 
>> This release contains security payload for Ozone. Below are some important
>> features in it:
>> 
>>  *   Hadoop Delegation Tokens and Block Tokens supported for Ozone.
>>  *   Transparent Data Encryption (TDE) Support - Allows data blocks to be
>> encrypted-at-rest.
>>  *   Kerberos support for Ozone.
>>  *   Certificate Infrastructure for Ozone  - Tokens use PKI instead of
>> shared secrets.
>>  *   Datanode to Datanode communication secured via mutual TLS.
>>  *   Ability secure ozone cluster that works with Yarn, Hive, and Spark.
>>  *   Skaffold support to deploy Ozone clusters on K8s.
>>  *   Support S3 Authentication Mechanisms like - S3 v4 Authentication
>> protocol.
>>  *   S3 Gateway supports Multipart upload.
>>  *   S3A file system is tested and supported.
>>  *   Support for Tracing and Profiling for all Ozone components.
>>  *   Audit Support - including Audit Parser tools.
>>  *   Apache Ranger Support in Ozone.
>>  *   Extensive failure testing for Ozone.
>> 
>> The RC artifacts are available at
>> https://home.apache.org/~ajay/ozone-0.4.0-alpha-rc1
>> 
>> The RC tag in git is ozone-0.4.0-alpha-RC1 (git hash
>> d673e16d14bb9377f27c9017e2ffc1bcb03eebfb)
>> 
>> Please try out<
>> https://cwiki.apache.org/confluence/display/HADOOP/Running+via+Apache+Release>,
>> vote, or just give us feedback.
>> 
>> The vote will run for 5 days, ending on April 20, 2019, 19:00 UTC.
>> 
>> Thank you very much,
>> 
>> Ajay
>> 
>> 
>> 

[jira] [Resolved] (HDDS-976) Support YAML format network topology cluster definition

[Xiaoyu Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoyu Yao resolved HDDS-976.
-
   Resolution: Fixed
Fix Version/s: 0.5.0

Thanks [~junjie] for the contribution. I've committed/merged the patch to 
trunk. 

> Support YAML format network topology cluster definition
> ---
>
> Key: HDDS-976
> URL: https://issues.apache.org/jira/browse/HDDS-976
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Sammi Chen
>Assignee: Junjie Chen
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.5.0
>
> Attachments: NetworkTopologyDefault.yaml
>
>  Time Spent: 3h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1377) OM failed to start with incorrect hostname set in CSR

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1377:


 Summary: OM failed to start with incorrect hostname set in CSR
 Key: HDDS-1377
 URL: https://issues.apache.org/jira/browse/HDDS-1377
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Affects Versions: 0.4.0
Reporter: Xiaoyu Yao
Assignee: Ajay Kumar


OM failed to start after HDDS-1355.

{code}

om_1 | 2019-04-03 16:23:50 ERROR OzoneManager:865 - Failed to start the 
OzoneManager.
om_1 | java.lang.IllegalArgumentException: IP Address is invalid
om_1 | at org.bouncycastle.asn1.x509.GeneralName.(Unknown Source)
om_1 | at 
org.apache.hadoop.hdds.security.x509.certificates.utils.CertificateSignRequest$Builder.addAltName(CertificateSignRequest.java:205)
om_1 | at 
org.apache.hadoop.hdds.security.x509.certificates.utils.CertificateSignRequest$Builder.addIpAddress(CertificateSignRequest.java:197)
om_1 | at 
org.apache.hadoop.ozone.om.OzoneManager.getSCMSignedCert(OzoneManager.java:1387)
om_1 | at 
org.apache.hadoop.ozone.om.OzoneManager.initializeSecurity(OzoneManager.java:1018)
om_1 | at org.apache.hadoop.ozone.om.OzoneManager.omInit(OzoneManager.java:971)
om_1 | at 
org.apache.hadoop.ozone.om.OzoneManager.createOm(OzoneManager.java:928)
om_1 | at org.apache.hadoop.ozone.om.OzoneManager.main(OzoneManager.java:859)

{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1350) Fix checkstyle issue in TestDatanodeStateMachine

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1350:


 Summary: Fix checkstyle issue in TestDatanodeStateMachine
 Key: HDDS-1350
 URL: https://issues.apache.org/jira/browse/HDDS-1350
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Reporter: Xiaoyu Yao


The following tests are FAILED:
 
[checkstyle]: checkstyle check is failed 
([https://ci.anzix.net/job/ozone-nightly/44/checkstyle/])



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1346) Remove hard-coded version ozone-0.5 from ReadMe of ozonesecure-mr docker-compose

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1346:


 Summary: Remove hard-coded version ozone-0.5 from ReadMe of 
ozonesecure-mr docker-compose
 Key: HDDS-1346
 URL: https://issues.apache.org/jira/browse/HDDS-1346
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


As we are releasing ozone-0.4, we should not have hard-coded ozone-0.5 for 
trunk. 

The proposal is to use the following to replace it

{{cd}} {{$(git rev-parse 
--show-toplevel)}}{{/hadoop-ozone/dist/target/ozone-}}{{*-SNAPSHOT}}{{/compose/ozone}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-1326) putkey operation failed with java.lang.ArrayIndexOutOfBoundsException

[Xiaoyu Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoyu Yao resolved HDDS-1326.
--
Resolution: Duplicate

> putkey operation failed with java.lang.ArrayIndexOutOfBoundsException
> -
>
> Key: HDDS-1326
> URL: https://issues.apache.org/jira/browse/HDDS-1326
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>Reporter: Nilotpal Nandi
>Assignee: Shashikant Banerjee
>Priority: Blocker
>
> steps taken :
> ---
>  # trying to write key in 40 node cluster.
>  # write failed.
> client output
> ---
>  
> {noformat}
> e530-491c-ab03-3b1c34d1a751:c80390, 
> 974a806d-bf7d-4f1b-adb4-d51d802d368a:c80390, 
> 469bd8c4-5da2-43bb-bc4b-7edd884931e5:c80390]
> 2019-03-22 10:56:19,592 [main] WARN - Encountered exception {}
> java.io.IOException: Unexpected Storage Container Exception: 
> java.util.concurrent.ExecutionException: 
> java.util.concurrent.CompletionException: 
> org.apache.ratis.protocol.StateMachineException: 
> org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException 
> from Server 5d3eb91f-e530-491c-ab03-3b1c34d1a751: Container 1269 in CLOSED 
> state
>  at 
> org.apache.hadoop.hdds.scm.storage.BlockOutputStream.close(BlockOutputStream.java:511)
>  at 
> org.apache.hadoop.ozone.client.io.BlockOutputStreamEntry.close(BlockOutputStreamEntry.java:144)
>  at 
> org.apache.hadoop.ozone.client.io.KeyOutputStream.handleFlushOrClose(KeyOutputStream.java:565)
>  at 
> org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:329)
>  at 
> org.apache.hadoop.ozone.client.io.KeyOutputStream.write(KeyOutputStream.java:273)
>  at 
> org.apache.hadoop.ozone.client.io.OzoneOutputStream.write(OzoneOutputStream.java:49)
>  at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:96)
>  at 
> org.apache.hadoop.ozone.web.ozShell.keys.PutKeyHandler.call(PutKeyHandler.java:111)
>  at 
> org.apache.hadoop.ozone.web.ozShell.keys.PutKeyHandler.call(PutKeyHandler.java:53)
>  at picocli.CommandLine.execute(CommandLine.java:919)
>  at picocli.CommandLine.access$700(CommandLine.java:104)
>  at picocli.CommandLine$RunLast.handle(CommandLine.java:1083)
>  at picocli.CommandLine$RunLast.handle(CommandLine.java:1051)
>  at 
> picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:959)
>  at picocli.CommandLine.parseWithHandlers(CommandLine.java:1242)
>  at picocli.CommandLine.parseWithHandler(CommandLine.java:1181)
>  at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:61)
>  at org.apache.hadoop.ozone.web.ozShell.Shell.execute(Shell.java:82)
>  at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:52)
>  at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:93)
> Caused by: java.util.concurrent.ExecutionException: 
> java.util.concurrent.CompletionException: 
> org.apache.ratis.protocol.StateMachineException: 
> org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException 
> from Server 5d3eb91f-e530-491c-ab03-3b1c34d1a751: Container 1269 in CLOSED 
> state
>  at 
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
>  at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
>  at 
> org.apache.hadoop.hdds.scm.storage.BlockOutputStream.waitOnFlushFutures(BlockOutputStream.java:529)
>  at 
> org.apache.hadoop.hdds.scm.storage.BlockOutputStream.handleFlush(BlockOutputStream.java:481)
>  at 
> org.apache.hadoop.hdds.scm.storage.BlockOutputStream.close(BlockOutputStream.java:496)
>  ... 19 more
> Caused by: java.util.concurrent.CompletionException: 
> org.apache.ratis.protocol.StateMachineException: 
> org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException 
> from Server 5d3eb91f-e530-491c-ab03-3b1c34d1a751: Container 1269 in CLOSED 
> state
>  at 
> org.apache.ratis.client.impl.RaftClientImpl.handleStateMachineException(RaftClientImpl.java:402)
>  at 
> org.apache.ratis.client.impl.RaftClientImpl.lambda$sendAsync$3(RaftClientImpl.java:198)
>  at 
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
>  at 
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
>  at 
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>  at 
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>  at 
> org.apache.ratis.client.impl.RaftClientImpl$PendingAsyncRequest.setReply(RaftClientImpl.java:95)

[jira] [Resolved] (HDDS-1327) ClientProtocol#close failed with EOFException running distcp from o3fs to o3fs

[Xiaoyu Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoyu Yao resolved HDDS-1327.
--
Resolution: Duplicate

This is a dup of HDDS-1317

> ClientProtocol#close failed with EOFException running distcp from o3fs to o3fs
> --
>
> Key: HDDS-1327
> URL: https://issues.apache.org/jira/browse/HDDS-1327
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>    Reporter: Xiaoyu Yao
>Priority: Blocker
>
> {code}
> 2019-03-22 15:47:21 DEBUG OzoneFileSystem:843 - Found file (with /) for path 
> o3fs://bucket1.vol1/user/hadoop/.staging: fake directory
> 2019-03-22 15:47:21 ERROR DistCp:167 - Exception encountered
> java.io.EOFException
>  at java.io.DataInputStream.readFully(DataInputStream.java:197)
>  at 
> org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:70)
>  at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:120)
>  at org.apache.hadoop.io.SequenceFile$Reader.nextRaw(SequenceFile.java:2522)
>  at 
> org.apache.hadoop.io.SequenceFile$Sorter$SortPass.run(SequenceFile.java:2994)
>  at org.apache.hadoop.io.SequenceFile$Sorter.sortPass(SequenceFile.java:2932)
>  at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2880)
>  at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2921)
>  at org.apache.hadoop.tools.util.DistCpUtils.sortListing(DistCpUtils.java:474)
>  at org.apache.hadoop.tools.util.DistCpUtils.sortListing(DistCpUtils.java:448)
>  at 
> org.apache.hadoop.tools.CopyListing.validateFinalListing(CopyListing.java:155)
>  at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:93)
>  at 
> org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:89)
>  at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)
>  at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:368)
>  at org.apache.hadoop.tools.DistCp.prepareFileListing(DistCp.java:96)
>  at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:205)
>  at org.apache.hadoop.tools.DistCp.execute(DistCp.java:182)
>  at org.apache.hadoop.tools.DistCp.run(DistCp.java:153)
>  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
>  at org.apache.hadoop.tools.DistCp.main(DistCp.java:432)
> 2019-03-22 15:47:21 DEBUG Client:97 - stopping client from cache: 
> org.apache.hadoop.ipc.Client@4944252c
> 2019-03-22 15:47:21 DEBUG OzoneClient:55 - Call: public abstract void 
> org.apache.hadoop.ozone.client.protocol.ClientProtocol.close() throws 
> java.io.IOException took 25 ms
> 2019-03-22 15:47:21 DEBUG ShutdownHookManager:97 - Completed shutdown in 
> 0.026 seconds; Timeouts: 0
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1327) ClientProtocol#close failed with EOFException running distcp from o3fs to o3fs

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1327:


 Summary: ClientProtocol#close failed with EOFException running 
distcp from o3fs to o3fs
 Key: HDDS-1327
 URL: https://issues.apache.org/jira/browse/HDDS-1327
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Xiaoyu Yao


{code}

2019-03-22 15:47:21 DEBUG OzoneFileSystem:843 - Found file (with /) for path 
o3fs://bucket1.vol1/user/hadoop/.staging: fake directory
2019-03-22 15:47:21 ERROR DistCp:167 - Exception encountered
java.io.EOFException
 at java.io.DataInputStream.readFully(DataInputStream.java:197)
 at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:70)
 at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:120)
 at org.apache.hadoop.io.SequenceFile$Reader.nextRaw(SequenceFile.java:2522)
 at 
org.apache.hadoop.io.SequenceFile$Sorter$SortPass.run(SequenceFile.java:2994)
 at org.apache.hadoop.io.SequenceFile$Sorter.sortPass(SequenceFile.java:2932)
 at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2880)
 at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2921)
 at org.apache.hadoop.tools.util.DistCpUtils.sortListing(DistCpUtils.java:474)
 at org.apache.hadoop.tools.util.DistCpUtils.sortListing(DistCpUtils.java:448)
 at 
org.apache.hadoop.tools.CopyListing.validateFinalListing(CopyListing.java:155)
 at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:93)
 at 
org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:89)
 at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)
 at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:368)
 at org.apache.hadoop.tools.DistCp.prepareFileListing(DistCp.java:96)
 at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:205)
 at org.apache.hadoop.tools.DistCp.execute(DistCp.java:182)
 at org.apache.hadoop.tools.DistCp.run(DistCp.java:153)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
 at org.apache.hadoop.tools.DistCp.main(DistCp.java:432)
2019-03-22 15:47:21 DEBUG Client:97 - stopping client from cache: 
org.apache.hadoop.ipc.Client@4944252c
2019-03-22 15:47:21 DEBUG OzoneClient:55 - Call: public abstract void 
org.apache.hadoop.ozone.client.protocol.ClientProtocol.close() throws 
java.io.IOException took 25 ms
2019-03-22 15:47:21 DEBUG ShutdownHookManager:97 - Completed shutdown in 0.026 
seconds; Timeouts: 0

{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1318) Fix MalformedTracerStateStringException on DN logs

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1318:


 Summary: Fix MalformedTracerStateStringException on DN logs
 Key: HDDS-1318
 URL: https://issues.apache.org/jira/browse/HDDS-1318
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Reporter: Xiaoyu Yao


Have seen many warnings on DN logs. This ticket is opened to track the 
investigation and fix for this.

{code}

2019-03-20 19:01:33 WARN 
PropagationRegistry$ExceptionCatchingExtractorDecorator:60 - Error when 
extracting SpanContext from carrier. Handling gracefully.
io.jaegertracing.internal.exceptions.MalformedTracerStateStringException: 
String does not match tracer state format: 2c919331-9a51-4bc4-acee-df57a8dcecf0
 at org.apache.hadoop.hdds.tracing.StringCodec.extract(StringCodec.java:42)
 at org.apache.hadoop.hdds.tracing.StringCodec.extract(StringCodec.java:32)
 at 
io.jaegertracing.internal.PropagationRegistry$ExceptionCatchingExtractorDecorator.extract(PropagationRegistry.java:57)
 at io.jaegertracing.internal.JaegerTracer.extract(JaegerTracer.java:208)
 at io.jaegertracing.internal.JaegerTracer.extract(JaegerTracer.java:61)
 at io.opentracing.util.GlobalTracer.extract(GlobalTracer.java:143)
 at 
org.apache.hadoop.hdds.tracing.TracingUtil.importAndCreateScope(TracingUtil.java:96)
 at 
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:148)
 at 
org.apache.hadoop.ozone.container.common.transport.server.GrpcXceiverService$1.onNext(GrpcXceiverService.java:73)
 at 
org.apache.hadoop.ozone.container.common.transport.server.GrpcXceiverService$1.onNext(GrpcXceiverService.java:61)
 at 
org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:248)
 at 
org.apache.ratis.thirdparty.io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33)
 at 
org.apache.ratis.thirdparty.io.grpc.Contexts$ContextualizedServerCallListener.onMessage(Contexts.java:76)
 at 
org.apache.ratis.thirdparty.io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33)
 at 
org.apache.hadoop.hdds.tracing.GrpcServerInterceptor$1.onMessage(GrpcServerInterceptor.java:46)
 at 
org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:263)
 at 
org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:686)
 at 
org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
 at 
org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
 at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)

{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1317) KeyOutputStream#write throws ArrayIndexOutOfBoundsException when running RandomWrite MR examples

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1317:


 Summary: KeyOutputStream#write throws 
ArrayIndexOutOfBoundsException when running RandomWrite MR examples
 Key: HDDS-1317
 URL: https://issues.apache.org/jira/browse/HDDS-1317
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Affects Versions: 0.4.0
Reporter: Xiaoyu Yao


Repro steps:

{code} 

jar $HADOOP_MAPRED_HOME/hadoop-mapreduce-examples-*.jar randomwriter 
-Dtest.randomwrite.total_bytes=1000  o3fs://bucket1.vol1/randomwrite.out

{code}

 

Error Stack:

{code}

2019-03-20 19:02:37 INFO Job:1686 - Task Id : 
attempt_1553108378906_0002_m_00_0, Status : FAILED
Error: java.lang.ArrayIndexOutOfBoundsException: -5
 at java.util.ArrayList.elementData(ArrayList.java:422)
 at java.util.ArrayList.get(ArrayList.java:435)
 at org.apache.hadoop.hdds.scm.storage.BufferPool.getBuffer(BufferPool.java:45)
 at 
org.apache.hadoop.hdds.scm.storage.BufferPool.allocateBufferIfNeeded(BufferPool.java:59)
 at 
org.apache.hadoop.hdds.scm.storage.BlockOutputStream.write(BlockOutputStream.java:215)
 at 
org.apache.hadoop.ozone.client.io.BlockOutputStreamEntry.write(BlockOutputStreamEntry.java:130)
 at 
org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:311)
 at 
org.apache.hadoop.ozone.client.io.KeyOutputStream.write(KeyOutputStream.java:273)
 at 
org.apache.hadoop.fs.ozone.OzoneFSOutputStream.write(OzoneFSOutputStream.java:46)
 at 
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:57)
 at java.io.DataOutputStream.write(DataOutputStream.java:107)
 at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1444)
 at 
org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat$1.write(SequenceFileOutputFormat.java:83)
 at 
org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:670)
 at 
org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
 at 
org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
 at 
org.apache.hadoop.examples.RandomWriter$RandomMapper.map(RandomWriter.java:199)
 at 
org.apache.hadoop.examples.RandomWriter$RandomMapper.map(RandomWriter.java:165)
 at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
 at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
 at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:422)
 at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
 at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)

{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-747) Update MiniOzoneCluster to work with security protocol from SCM

[Xiaoyu Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoyu Yao resolved HDDS-747.
-
Resolution: Invalid

This won't work with different components require separate Kerberos login of 
different principles in the same JVM. We will look into 
[https://www.testcontainers.org/] to test secure docker compose in the next 
release. 

> Update MiniOzoneCluster to work with security protocol from SCM
> ---
>
> Key: HDDS-747
> URL: https://issues.apache.org/jira/browse/HDDS-747
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Ajay Kumar
>Priority: Major
>  Labels: ozone-security
>
> [HDDS-103] introduces a new security protocol in SCM. MiniOzoneCluster should 
> be updated to utilize it once its implementation is completed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1299) Support TokenIssuer interface to run MR/Spark with OzoneFileSystem in secure mode

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1299:


 Summary: Support TokenIssuer interface to run MR/Spark with 
OzoneFileSystem in secure mode
 Key: HDDS-1299
 URL: https://issues.apache.org/jira/browse/HDDS-1299
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


This ticket is opened to add TokenIssuer interface support to OzoneFileSystem 
so that MR and Spark jobs can run with OzoneFileSystem in secure mode. 

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1296) Fix checkstyle issue from Nightly run

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1296:


 Summary: Fix checkstyle issue from Nightly run
 Key: HDDS-1296
 URL: https://issues.apache.org/jira/browse/HDDS-1296
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Affects Versions: 0.4.0
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


https://ci.anzix.net/job/ozone-nightly/28/checkstyle/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1292) Fix nightly run findbugs and checkstyle issues

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1292:


 Summary: Fix nightly run findbugs and checkstyle issues
 Key: HDDS-1292
 URL: https://issues.apache.org/jira/browse/HDDS-1292
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Xiaoyu Yao


https://ci.anzix.net/job/ozone/3775/findbugs/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-1138) OzoneManager should return the pipeline info of the allocated block along with block info

[Xiaoyu Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoyu Yao resolved HDDS-1138.
--
   Resolution: Fixed
 Assignee: Xiaoyu Yao  (was: Mukul Kumar Singh)
Fix Version/s: 0.4.0

> OzoneManager should return the pipeline info of the allocated block along 
> with block info
> -
>
> Key: HDDS-1138
> URL: https://issues.apache.org/jira/browse/HDDS-1138
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: Ozone Client, Ozone Manager
>Affects Versions: 0.4.0
>Reporter: Mukul Kumar Singh
>Assignee: Xiaoyu Yao
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.4.0
>
> Attachments: HDDS-1138.001.patch
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Currently, while a block is allocated from OM, the request is forwarded to 
> SCM. However, even though the pipeline information is present with the OM for 
> block allocation, this information is passed through to the client.
> This optimization will help in reducing the number of hops for the client by 
> reducing 1 RPC round trip for each block allocated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1291) Set OmKeyArgs#refreshPipeline flag properly when client reads a stale pipeline

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1291:


 Summary: Set OmKeyArgs#refreshPipeline flag properly when client 
reads a stale pipeline
 Key: HDDS-1291
 URL: https://issues.apache.org/jira/browse/HDDS-1291
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


After HDDS-1138, the OM client will not talk to SCM directly to fetch the 
pipeline info. Instead the pipeline info is returned as part of the keyLocation 
cached by OM. 

 

In case SCM pipeline is changed such as closed, the client may get invalid 
pipeline exception. In this case, the client need to getKeyLocation with 
OmKeyArgs#refreshPipeline = true to force OM update its pipeline cache for this 
key. 

 

An optimization could be queue a background task to update all the keyLocations 
that is affected when OM does a refreshPipeline. (This part can be done in 0.5)
{code:java}
oldpipeline->newpipeline{code}
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1287) Change some GrpcClientProtocolClient log from INFO to DEBUG

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1287:


 Summary: Change some GrpcClientProtocolClient log from INFO to 
DEBUG
 Key: HDDS-1287
 URL: https://issues.apache.org/jira/browse/HDDS-1287
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


It is too much for every ozone sh key put.

{code}

bash-4.2$ ozone sh key put /vol1/bucket1/key1 LICENSE.txt
2019-03-14 16:35:03 INFO GrpcClientProtocolClient:302 - schedule 3000ms timeout 
check for 
RaftClientRequest:client-D48F06A03AF5->45e6a76d-7c48-4e3d-b18e-fee47055793b@group-D2B35873B94E,
 cid=0, seq=0 RW, 
org.apache.hadoop.hdds.scm.XceiverClientRatis$$Lambda$64/878991463@3e6fd0b9
2019-03-14 16:35:04 INFO GrpcClientProtocolClient:256 - 
client-D48F06A03AF5->45e6a76d-7c48-4e3d-b18e-fee47055793b: receive 
RaftClientReply:client-D48F06A03AF5->45e6a76d-7c48-4e3d-b18e-fee47055793b@group-D2B35873B94E,
 cid=0, SUCCESS, logIndex=1, commits[45e6a76d-7c48-4e3d-b18e-fee47055793b:c2]
2019-03-14 16:35:04 INFO GrpcClientProtocolClient:302 - schedule 3000ms timeout 
check for 
RaftClientRequest:client-D48F06A03AF5->45e6a76d-7c48-4e3d-b18e-fee47055793b@group-D2B35873B94E,
 cid=1, seq=1 RW, 
org.apache.hadoop.hdds.scm.XceiverClientRatis$$Lambda$64/878991463@25211d10
2019-03-14 16:35:04 INFO GrpcClientProtocolClient:256 - 
client-D48F06A03AF5->45e6a76d-7c48-4e3d-b18e-fee47055793b: receive 
RaftClientReply:client-D48F06A03AF5->45e6a76d-7c48-4e3d-b18e-fee47055793b@group-D2B35873B94E,
 cid=1, SUCCESS, logIndex=3, commits[45e6a76d-7c48-4e3d-b18e-fee47055793b:c4]

{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1253) Fix checkstyle issue from Nightly run

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1253:


 Summary: Fix checkstyle issue from Nightly run
 Key: HDDS-1253
 URL: https://issues.apache.org/jira/browse/HDDS-1253
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Affects Versions: 0.4.0
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


https://ci.anzix.net/job/ozone-nightly/28/checkstyle/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1246) Support ozone delegation token utility subcmd for Ozone CLI

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1246:


 Summary: Support ozone delegation token utility subcmd for Ozone 
CLI
 Key: HDDS-1246
 URL: https://issues.apache.org/jira/browse/HDDS-1246
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


This allow running dtutil with integration test and dev test for demo of Ozone 
security.

 

 

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1245) OM delegation expiration time should use Time.now instead of Time.monotonicNow

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1245:


 Summary: OM delegation expiration time should use Time.now instead 
of Time.monotonicNow
 Key: HDDS-1245
 URL: https://issues.apache.org/jira/browse/HDDS-1245
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


Otherwise, we will set incorrect the exp date of OM delegation like below: 

{code}
ozone dtutil print /tmp/om.dt
 
File: /tmp/om.dt
Token kind   Service  Renewer Exp date URL 
enc token

OzoneToken   om:9862  yarn*1/8/70 12:03 PM*
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1236) Fix incorrect Ozone ClientProtocol KerberosInfo annotation

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1236:


 Summary: Fix incorrect Ozone ClientProtocol KerberosInfo annotation
 Key: HDDS-1236
 URL: https://issues.apache.org/jira/browse/HDDS-1236
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


The serverPrincipal should be OMConfigKeys.OZONE_OM_KERBEROS_PRINCIPAL_KEY 
instead of ScmConfigKeys.HDDS_SCM_KERBEROS_PRINCIPAL_KEY



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1235) BaseHttpServer NPE is HTTP policy is HTTPS_ONLY

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1235:


 Summary: BaseHttpServer NPE is HTTP policy is HTTPS_ONLY
 Key: HDDS-1235
 URL: https://issues.apache.org/jira/browse/HDDS-1235
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


This needs to be fixed when Ozone is running inside DN as plugin and DN is 
running using non-privilege HTTPS port. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1215) Ozonesecure docker compose KMS instance errors due to Java11

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1215:


 Summary: Ozonesecure docker compose KMS instance errors due to 
Java11
 Key: HDDS-1215
 URL: https://issues.apache.org/jira/browse/HDDS-1215
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Reporter: Xiaoyu Yao


{code}

kms_1           | Exception in thread "main" java.lang.NoClassDefFoundError: 
javax/activation/DataSource

kms_1           | at 
com.sun.xml.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl.(RuntimeBuiltinLeafInfoImpl.java:457)

kms_1           | at 
com.sun.xml.bind.v2.model.impl.RuntimeTypeInfoSetImpl.(RuntimeTypeInfoSetImpl.java:65)

kms_1           | at 
com.sun.xml.bind.v2.model.impl.RuntimeModelBuilder.createTypeInfoSet(RuntimeModelBuilder.java:133)

kms_1           | at 
com.sun.xml.bind.v2.model.impl.RuntimeModelBuilder.createTypeInfoSet(RuntimeModelBuilder.java:85)

kms_1           | at 
com.sun.xml.bind.v2.model.impl.ModelBuilder.(ModelBuilder.java:156)

kms_1           | at 
com.sun.xml.bind.v2.model.impl.RuntimeModelBuilder.(RuntimeModelBuilder.java:93)

kms_1           | at 
com.sun.xml.bind.v2.runtime.JAXBContextImpl.getTypeInfoSet(JAXBContextImpl.java:473)

kms_1           | at 
com.sun.xml.bind.v2.runtime.JAXBContextImpl.(JAXBContextImpl.java:319)

kms_1           | at 
com.sun.xml.bind.v2.runtime.JAXBContextImpl$JAXBContextBuilder.build(JAXBContextImpl.java:1170)

kms_1           | at 
com.sun.xml.bind.v2.ContextFactory.createContext(ContextFactory.java:145)

kms_1           | at 
com.sun.xml.bind.v2.ContextFactory.createContext(ContextFactory.java:236)

kms_1           | at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

kms_1           | at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

kms_1           | at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

kms_1           | at java.base/java.lang.reflect.Method.invoke(Method.java:566)

kms_1           | at 
javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:186)

kms_1           | at 
javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:146)

kms_1           | at javax.xml.bind.ContextFinder.find(ContextFinder.java:350)

kms_1           | at 
javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:446)

kms_1           | at 
javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:409)

kms_1           | at 
com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.(WadlApplicationContextImpl.java:103)

kms_1           | at 
com.sun.jersey.server.impl.wadl.WadlFactory.init(WadlFactory.java:100)

kms_1           | at 
com.sun.jersey.server.impl.application.RootResourceUriRules.initWadl(RootResourceUriRules.java:169)

kms_1           | at 
com.sun.jersey.server.impl.application.RootResourceUriRules.(RootResourceUriRules.java:106)

kms_1           | at 
com.sun.jersey.server.impl.application.WebApplicationImpl._initiate(WebApplicationImpl.java:1359)

kms_1           | at 
com.sun.jersey.server.impl.application.WebApplicationImpl.access$700(WebApplicationImpl.java:180)

kms_1           | at 
com.sun.jersey.server.impl.application.WebApplicationImpl$13.f(WebApplicationImpl.java:799)

kms_1           | at 
com.sun.jersey.server.impl.application.WebApplicationImpl$13.f(WebApplicationImpl.java:795)

{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1204) Fix misc issue to make ozonesecure docker-compose work on Java 11

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1204:


 Summary: Fix misc issue to make ozonesecure docker-compose work on 
Java 11
 Key: HDDS-1204
 URL: https://issues.apache.org/jira/browse/HDDS-1204
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


The ozonesecure docker-compose has been changed to use hadoop-runner image 
based on Java 11. Several class has been removed from Java 8 such as 

javax.xml.bind.DatatypeConverter.parseHexBinary

 

This ticket is opened to fix issues running ozonesecure docker-compose on java 
11.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1190) Fix jdk 11 issue for ozonesecure base image and docker-compose

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1190:


 Summary: Fix jdk 11 issue for ozonesecure base image and 
docker-compose 
 Key: HDDS-1190
 URL: https://issues.apache.org/jira/browse/HDDS-1190
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


HDDS-1019 changes to use hadoop-runner as base image for ozonesecure 
docker-compose. There are a few issues that need to fixed.

 

1.The hadoop-runner uses jdk11 but the ozonesecure/docker-config assume 
openjdk8 for JAVA_HOME. 

 

2. The KEYTAB_DIR needs to be quoted with '.

 

3. keytab based login failed with Message stream modified (41), [~elek] 
mentioned in HDDS-1019 that we need to add max_renewable_life to 
"docker-image/docker-krb5/krb5.conf" like follows.
[realms]
 EXAMPLE.COM = \{
  kdc = localhost
  admin_server = localhost
  max_renewable_life = 7d
 }
Failures:

{code}

 org.apache.hadoop.security.KerberosAuthException: failure to login: for 
principal: scm/s...@example.com from keytab /etc/security/keytabs/scm.keytab 
javax.security.auth.login.LoginException: Message stream modified (41)

scm_1           | at 
org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1847)

scm_1           |

{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1183) OzoneFileSystem needs to override delegation token APIs

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1183:


 Summary: OzoneFileSystem needs to override delegation token APIs
 Key: HDDS-1183
 URL: https://issues.apache.org/jira/browse/HDDS-1183
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


This includes addDelegationToken/renewDelegationToken/cancelDelegationToken so 
that MR jobs can collect tokens correctly upon job submission time. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1179) Ozone dist build failed on Jenkins

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1179:


 Summary: Ozone dist build failed on Jenkins
 Key: HDDS-1179
 URL: https://issues.apache.org/jira/browse/HDDS-1179
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


This is part of the Jenkins execution and was reported in several latest HDDS 
Jenkins run.

I spend some today and found a simplified repro steps:

{code}

cd hadoop-ozone/dist

mvn -Phdds -DskipTests -fae clean install -DskipTests=true 
-Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Dfindbugs.skip=true 

{code}

 

The root cause is that the 
hadoop-ozone/dist/dev-support/bin/dist-layout-stitching need objectstore-sevice 
jar being build earlier but he dependency was not explicitly declared in pom. I 
will attach a fix shortly. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1119) DN get the certificate from SCM CA for token validation

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1119:


 Summary: DN get the certificate from SCM CA for token validation
 Key: HDDS-1119
 URL: https://issues.apache.org/jira/browse/HDDS-1119
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


This is needed when the OM received delegation token signed by other OM 
instances and it does not have the certificate for foreign OM.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1118) OM get the certificate from SCM CA for token validation

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-1118:


 Summary: OM get the certificate from SCM CA for token validation
 Key: HDDS-1118
 URL: https://issues.apache.org/jira/browse/HDDS-1118
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


This is needed when the OM received delegation token signed by other OM 
instances and it does not have the certificate for foreign OM.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org