[jira] [Created] (HDFS-15724) Hive should use DFSClient#isHDFSEncryptionEnabled
Xiaoyu Yao created HDFS-15724: - Summary: Hive should use DFSClient#isHDFSEncryptionEnabled Key: HDFS-15724 URL: https://issues.apache.org/jira/browse/HDFS-15724 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.3.0 Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao HIVE-23212 and HIVE-23261 adding check for hdfs encryption to determine if copy is needed. However, the API used DFSUtilClient.isHDFSEncryptionEnabled is solely based on local hadoop.security.key.provider.path configuration. This has been evolved into a HDFS internal configuration that may not exist in client side. The right API to be called here should be DFSClient.isHDFSEncryptionEnabled which will get the keyprovider info from NN directly after HDFS-11687 and HDFS-12396. This ticket is opened to fix it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15344) DataNode#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442
[ https://issues.apache.org/jira/browse/HDFS-15344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao resolved HDFS-15344. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed Thanks all for the reviews. The PR has been merged to trunk. > DataNode#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442 > > > Key: HDFS-15344 > URL: https://issues.apache.org/jira/browse/HDFS-15344 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.7.5 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Fix For: 3.4.0 > > > HADOOP-13442 added UGI#getGroups to avoid list->array->list conversions. This > ticket is opened to change DataNode#checkSuperuserPrivilege to use > UGI#getGroups. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-15345) RouterPermissionChecker#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442
Xiaoyu Yao created HDFS-15345: - Summary: RouterPermissionChecker#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442 Key: HDFS-15345 URL: https://issues.apache.org/jira/browse/HDFS-15345 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 2.7.5 Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao HADOOP-13442 added UGI#getGroups to avoid list->array->list conversions. This ticket is opened to change DataNode#checkSuperuserPrivilege to use UGI#getGroups. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-15344) DataNode#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442
Xiaoyu Yao created HDFS-15344: - Summary: DataNode#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442 Key: HDFS-15344 URL: https://issues.apache.org/jira/browse/HDFS-15344 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 2.7.5 Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao HADOOP-13442 added UGI#getGroups to avoid list->array->list conversions. This ticket is opened to change DataNode#checkSuperuserPrivilege to use UGI#getGroups. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-15192) Leaking stream when access encrypted files hit exception via webhdfs
Xiaoyu Yao created HDFS-15192: - Summary: Leaking stream when access encrypted files hit exception via webhdfs Key: HDFS-15192 URL: https://issues.apache.org/jira/browse/HDFS-15192 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.3.0 Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao This was found during fixing for HADOOP-16885. Given the code is in HDFS, the ticket is opened to address the similar issue, i.e., inner stream is not closed when wrapper stream create failed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-15061) KMSClientProvider does not work with client using ticket logged in externally
Xiaoyu Yao created HDFS-15061: - Summary: KMSClientProvider does not work with client using ticket logged in externally Key: HDFS-15061 URL: https://issues.apache.org/jira/browse/HDFS-15061 Project: Hadoop HDFS Issue Type: Bug Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao This is a regression from HDFS-13682 that checks not only the kerberos credential but also enforce the login is non-external. This breaks client applications that need to access HDFS encrypted file using kerberos ticket that logged in external in ticket cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2375) Refactor BlockOutputStream to allow flexible buffering
[ https://issues.apache.org/jira/browse/HDDS-2375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao resolved HDDS-2375. -- Resolution: Fixed Thanks [~szetszwo] for the contribution and all for the reviews. I've merged the PR to master. > Refactor BlockOutputStream to allow flexible buffering > -- > > Key: HDDS-2375 > URL: https://issues.apache.org/jira/browse/HDDS-2375 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Reporter: Tsz-wo Sze >Assignee: Tsz-wo Sze >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > In HDDS-2331, we found that Ozone client allocates a ByteBuffer with chunk > size (e.g. 16MB ) to store data, unregarded the actual data size. The > ByteBuffer will create a byte[] with chunk size. When the ByteBuffer is > wrapped to a ByteString the byte[] remains in the ByteString. > As a result, when the actual data size is small (e.g. 1MB), a lot of memory > spaces (15MB) are wasted. > In this JIRA, we refactor BlockOutputStream so that the buffering becomes > more flexible. In a later JIRA (HDDS-2386), we implement chunk buffer using > a list of smaller buffers which are allocated only if needed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2518) Ensure RATIS leader info is properly updated with pipeline report.
Xiaoyu Yao created HDDS-2518: Summary: Ensure RATIS leader info is properly updated with pipeline report. Key: HDDS-2518 URL: https://issues.apache.org/jira/browse/HDDS-2518 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao HDDS-2034 added async pipeline creation and report handling to SCM. The leader information is not properly populated as manifested in the test failures from TestSCMPipelineManager#testPipelineReport. This ticket is opened to fix it. cc: [~sammichen] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2497) SafeMode check should allow key creation on single node pipeline when replication factor is 1
Xiaoyu Yao created HDDS-2497:
Summary: SafeMode check should allow key creation on single node
pipeline when replication factor is 1
Key: HDDS-2497
URL: https://issues.apache.org/jira/browse/HDDS-2497
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Xiaoyu Yao
Start a single datanode ozone docker-compose with replication factor of 1.
{code:java}
OZONE-SITE.XML_ozone.replication=1{code}
The key creation failed with Safemode exception below.
{code:java}
>$ docker-compose exec om bash
bash-4.2$ ozone sh vol create /vol1
bash-4.2$ ozone sh bucket create /vol1/bucket1
ozone sh kbash-4.2$ ozone sh key put /vol1/bucket1/key1 README.md
SCM_IN_SAFE_MODE SafeModePrecheck failed for allocateBlock{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2496) Delegate Ozone volume create/list ACL check to authorizer plugin
Xiaoyu Yao created HDDS-2496: Summary: Delegate Ozone volume create/list ACL check to authorizer plugin Key: HDDS-2496 URL: https://issues.apache.org/jira/browse/HDDS-2496 Project: Hadoop Distributed Data Store Issue Type: Improvement Affects Versions: 0.4.1 Reporter: Vivek Ratnavel Subramanian Assignee: Vivek Ratnavel Subramanian Today Ozone volume create/list ACL check are not sent to authorization plugins. This cause problem when authorization plugin is enabled. Admin still need to modify ozone-site.xml to change ozone.administrators to configure admin to create volume This ticket is opened to have a consistent ACL check for all Ozone resources requests including admin request like volume create. This way, the admin defined by the authorization plugin can be honored during volume provision without restart ozone services. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2492) Fix test clean up issue in TestSCMPipelineManager
Xiaoyu Yao created HDDS-2492:
Summary: Fix test clean up issue in TestSCMPipelineManager
Key: HDDS-2492
URL: https://issues.apache.org/jira/browse/HDDS-2492
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Sammi Chen
Assignee: Li Cheng
This was opened based on [~sammichen]'s investigation on HDDS-2034.
{quote}Failure is caused by newly introduced function
TestSCMPipelineManager#testPipelineOpenOnlyWhenLeaderReported which doesn't
close pipelineManager at the end. It's better to fix it in a new JIRA.
{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2491) Fix TestSCMSafeModeWithPipelineRules
Xiaoyu Yao created HDDS-2491:
Summary: Fix TestSCMSafeModeWithPipelineRules
Key: HDDS-2491
URL: https://issues.apache.org/jira/browse/HDDS-2491
Project: Hadoop Distributed Data Store
Issue Type: Bug
Affects Versions: 0.4.1
Reporter: Xiaoyu Yao
This was based on [~sammichen]'s investigation on HDDS-2034.
{quote}The root cause is failing to exit the safemode. Current pipeline open
condition(HDDS-1868) is got 3 datanode reports and one datanode marked itself
as leader. In this failure case, the leader election succeeds while
XceiverServerRatis#handleLeaderChangedNotification is not called in the next 3
minutes. So cluster.waitForClusterToBeReady() timeout.
The question is is this Leader change notification reliable? What's the
expected latency between leader election succeed and notification send?
{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2463) Remove unnecessary getServiceInfo calls
Xiaoyu Yao created HDDS-2463:
Summary: Remove unnecessary getServiceInfo calls
Key: HDDS-2463
URL: https://issues.apache.org/jira/browse/HDDS-2463
Project: Hadoop Distributed Data Store
Issue Type: Bug
Affects Versions: 0.4.1
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
OzoneManagerProtocolClientSideTranslatorPB.java Line 766-772 has multiple
impl.getServiceInfo() which can be reduced by adding a local variable.
{code:java}
resp.addAllServiceInfo(impl.getServiceInfo().getServiceInfoList().stream()
.map(ServiceInfo::getProtobuf)
.collect(Collectors.toList()));
if (impl.getServiceInfo().getCaCertificate() != null) {
resp.setCaCertificate(impl.getServiceInfo().getCaCertificate()); {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2321) Ozone Block Token verify should not apply to all datanode cmd
Xiaoyu Yao created HDDS-2321: Summary: Ozone Block Token verify should not apply to all datanode cmd Key: HDDS-2321 URL: https://issues.apache.org/jira/browse/HDDS-2321 Project: Hadoop Distributed Data Store Issue Type: Bug Affects Versions: 0.4.1 Reporter: Nilotpal Nandi Assignee: Xiaoyu Yao DN container protocol has cmd send from SCM or other DN, which do not bear OM block token like OM client. We should restrict the OM Block token check only for those issued from OM client. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2295) Display log of freon on the standard output
[ https://issues.apache.org/jira/browse/HDDS-2295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao resolved HDDS-2295. -- Fix Version/s: 0.5.0 Resolution: Fixed Thanks [~elek] for the contribution and all for the reviews. I've merged the changes. > Display log of freon on the standard output > --- > > Key: HDDS-2295 > URL: https://issues.apache.org/jira/browse/HDDS-2295 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Marton Elek >Assignee: Marton Elek >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > HDDS-2042 disabled the console logging for all of the ozone command line > tools including freon. > But freon is different, it has a different error handling model. For freon we > need all the log on the console. > 1. To follow all the different errors > 2. To get information about the used (random) prefix which can be reused > during the validation phase. > > I propose to restore the original behavior for Ozone. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2298) Fix maven warning about duplicated metrics-core jar
[
https://issues.apache.org/jira/browse/HDDS-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao resolved HDDS-2298.
--
Fix Version/s: 0.5.0
Resolution: Fixed
Thanks [~elek] for the contribution and all for the reviews. The change has
been merged.
> Fix maven warning about duplicated metrics-core jar
> ---
>
> Key: HDDS-2298
> URL: https://issues.apache.org/jira/browse/HDDS-2298
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: build
>Reporter: Marton Elek
>Assignee: Marton Elek
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Maven build of Ozone is starting with a warning:
> {code:java}
> [WARNING]
> [WARNING] Some problems were encountered while building the effective model
> for org.apache.hadoop:hadoop-ozone-tools:jar:0.5.0-SNAPSHOT
> [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must
> be unique: io.dropwizard.metrics:metrics-core:jar -> version 3.2.4 vs (?) @
> line 94, column 17
> [WARNING]
> [WARNING] It is highly recommended to fix these problems because they
> threaten the stability of your build.
> [WARNING]
> [WARNING] For this reason, future Maven versions might no longer support
> building such malformed projects.
> [WARNING]
> {code}
> It's better to avoid it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2304) ozone token cli output can be improved.
Xiaoyu Yao created HDDS-2304: Summary: ozone token cli output can be improved. Key: HDDS-2304 URL: https://issues.apache.org/jira/browse/HDDS-2304 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Some output does not start a new line at the end. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2282) scmcli pipeline list command throws NullPointerException
Xiaoyu Yao created HDDS-2282:
Summary: scmcli pipeline list command throws NullPointerException
Key: HDDS-2282
URL: https://issues.apache.org/jira/browse/HDDS-2282
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Nilotpal Nandi
Assignee: Xiaoyu Yao
ozone scmcli pipeline list
{noformat}
java.lang.NullPointerException
at
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:187)
at
org.apache.hadoop.hdds.scm.XceiverClientManager.(XceiverClientManager.java:98)
at
org.apache.hadoop.hdds.scm.XceiverClientManager.(XceiverClientManager.java:83)
at
org.apache.hadoop.hdds.scm.cli.SCMCLI.createScmClient(SCMCLI.java:139)
at
org.apache.hadoop.hdds.scm.cli.pipeline.ListPipelinesSubcommand.call(ListPipelinesSubcommand.java:55)
at
org.apache.hadoop.hdds.scm.cli.pipeline.ListPipelinesSubcommand.call(ListPipelinesSubcommand.java:30)
at picocli.CommandLine.execute(CommandLine.java:1173)
at picocli.CommandLine.access$800(CommandLine.java:141)
at picocli.CommandLine$RunLast.handle(CommandLine.java:1367)
at picocli.CommandLine$RunLast.handle(CommandLine.java:1335)
at
picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1243)
at picocli.CommandLine.parseWithHandlers(CommandLine.java:1526)
at picocli.CommandLine.parseWithHandler(CommandLine.java:1465)
at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:65)
at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:56)
at org.apache.hadoop.hdds.scm.cli.SCMCLI.main(SCMCLI.java:101){noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2228) Fix NPE in OzoneDelegationTokenManager#addPersistedDelegationToken
Xiaoyu Yao created HDDS-2228: Summary: Fix NPE in OzoneDelegationTokenManager#addPersistedDelegationToken Key: HDDS-2228 URL: https://issues.apache.org/jira/browse/HDDS-2228 Project: Hadoop Distributed Data Store Issue Type: Improvement Affects Versions: 0.4.0 Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao The certClient was not initialized in proper order as a result, when OM restart with delegation token issued, the ozone delegation token secret manager NPE. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2213) Reduce key provider loading log level in OzoneFileSystem#getAdditionalTokenIssuers
Xiaoyu Yao created HDDS-2213:
Summary: Reduce key provider loading log level in
OzoneFileSystem#getAdditionalTokenIssuers
Key: HDDS-2213
URL: https://issues.apache.org/jira/browse/HDDS-2213
Project: Hadoop Distributed Data Store
Issue Type: Improvement
Reporter: Vivek Ratnavel Subramanian
OzoneFileSystem#getAdditionalTokenIssuers log an error when secure client tries
to collect ozone delegation token to run MR/Spark jobs but ozone file system
does not have a kms provider configured. In this case, we simply return null
provider here in the code below. This is a benign error and we should reduce
the log level to debug level.
\{code}
KeyProvider keyProvider;
try {
keyProvider = getKeyProvider();
} catch (IOException ioe) {
LOG.error("Error retrieving KeyProvider.", ioe);
return null;
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2172) Ozone shell should remove description about REST protocol support
Xiaoyu Yao created HDDS-2172:
Summary: Ozone shell should remove description about REST protocol
support
Key: HDDS-2172
URL: https://issues.apache.org/jira/browse/HDDS-2172
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Xiaoyu Yao
{code}
Usage: ozone sh key put [-hV] [-r=]
creates or overwrites an existing key
URI of the volume/bucket/key.
Ozone URI could start with o3:// or http(s):// or without prefix.
REST protocol will be used for http(s), RPC otherwise. URI may
contain the host and port of the OM server. Both are optional.
If they are not specified it will be identified from the config
files.
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2160) Add acceptance test for ozonesecure-mr compose
Xiaoyu Yao created HDDS-2160: Summary: Add acceptance test for ozonesecure-mr compose Key: HDDS-2160 URL: https://issues.apache.org/jira/browse/HDDS-2160 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao This will give us coverage of running basic MR jobs on security enabled OZONE cluster against YARN. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1553) Add metrics in rack aware container placement policy
[ https://issues.apache.org/jira/browse/HDDS-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao resolved HDDS-1553. -- Fix Version/s: 0.5.0 Resolution: Fixed Thanks [~Sammi] for the contribution. I merged the change to trunk. > Add metrics in rack aware container placement policy > > > Key: HDDS-1553 > URL: https://issues.apache.org/jira/browse/HDDS-1553 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Sammi Chen >Assignee: Sammi Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 3h 20m > Remaining Estimate: 0h > > To collect following statistics, > 1. total requested datanode count (A) > 2. success allocated datanode count without constrain compromise (B) > 3. success allocated datanode count with some comstrain compromise (C) > B includes C, failed allocation = (A - B) -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2099) Refactor to create pipeline via DN heartbeat response
Xiaoyu Yao created HDDS-2099: Summary: Refactor to create pipeline via DN heartbeat response Key: HDDS-2099 URL: https://issues.apache.org/jira/browse/HDDS-2099 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Xiaoyu Yao Currently, SCM directly talk to DN GRPC server to create pipeline in a background thread. We should avoid direct communication from SCM to DN for better scalability of ozone. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14828) Add TeraSort to acceptance test
Xiaoyu Yao created HDFS-14828: - Summary: Add TeraSort to acceptance test Key: HDFS-14828 URL: https://issues.apache.org/jira/browse/HDFS-14828 Project: Hadoop HDFS Issue Type: Test Reporter: Xiaoyu Yao We may begin with 1GB teragen/terasort/teravalidate. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2096) Ozone ACL document missing AddAcl API
Xiaoyu Yao created HDDS-2096: Summary: Ozone ACL document missing AddAcl API Key: HDDS-2096 URL: https://issues.apache.org/jira/browse/HDDS-2096 Project: Hadoop Distributed Data Store Issue Type: Test Reporter: Xiaoyu Yao Current Ozone Native ACL APIs document looks like below, the AddAcl is missing. h3. Ozone Native ACL APIs The ACLs can be manipulated by a set of APIs supported by Ozone. The APIs supported are: # *SetAcl* – This API will take user principal, the name, type of the ozone object and a list of ACLs. # *GetAcl* – This API will take the name and type of the ozone object and will return a list of ACLs. # *RemoveAcl* - This API will take the name, type of the ozone object and the ACL that has to be removed. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2047) Datanodes fail to come up after 10 retries in a secure environment
[
https://issues.apache.org/jira/browse/HDDS-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao resolved HDDS-2047.
--
Fix Version/s: 0.4.1
Resolution: Fixed
Thanks all for the reviews and discussions. I've merged the PR to trunk and
ozone-0.4.1.
> Datanodes fail to come up after 10 retries in a secure environment
> --
>
> Key: HDDS-2047
> URL: https://issues.apache.org/jira/browse/HDDS-2047
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode, Security
>Affects Versions: 0.4.1
>Reporter: Vivek Ratnavel Subramanian
>Assignee: Xiaoyu Yao
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.1
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> {code:java}
> 10:06:36.585 PMERRORHddsDatanodeService
> Error while storing SCM signed certificate.
> java.net.ConnectException: Call From
> jmccarthy-ozone-secure-2.vpc.cloudera.com/10.65.50.127 to
> jmccarthy-ozone-secure-1.vpc.cloudera.com:9961 failed on connection
> exception: java.net.ConnectException: Connection refused; For more details
> see: http://wiki.apache.org/hadoop/ConnectionRefused
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:755)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1515)
> at org.apache.hadoop.ipc.Client.call(Client.java:1457)
> at org.apache.hadoop.ipc.Client.call(Client.java:1367)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy15.getDataNodeCertificate(Unknown Source)
> at
> org.apache.hadoop.hdds.protocolPB.SCMSecurityProtocolClientSideTranslatorPB.getDataNodeCertificateChain(SCMSecurityProtocolClientSideTranslatorPB.java:156)
> at
> org.apache.hadoop.ozone.HddsDatanodeService.getSCMSignedCert(HddsDatanodeService.java:278)
> at
> org.apache.hadoop.ozone.HddsDatanodeService.initializeCertificateClient(HddsDatanodeService.java:248)
> at
> org.apache.hadoop.ozone.HddsDatanodeService.start(HddsDatanodeService.java:211)
> at
> org.apache.hadoop.ozone.HddsDatanodeService.start(HddsDatanodeService.java:168)
> at
> org.apache.hadoop.ozone.HddsDatanodeService.call(HddsDatanodeService.java:143)
> at
> org.apache.hadoop.ozone.HddsDatanodeService.call(HddsDatanodeService.java:70)
> at picocli.CommandLine.execute(CommandLine.java:1173)
> at picocli.CommandLine.access$800(CommandLine.java:141)
> at picocli.CommandLine$RunLast.handle(CommandLine.java:1367)
> at picocli.CommandLine$RunLast.handle(CommandLine.java:1335)
> at
> picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1243)
> at picocli.CommandLine.parseWithHandlers(CommandLine.java:1526)
> at picocli.CommandLine.parseWithHandler(CommandLine.java:1465)
> at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:65)
> at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:56)
> at
> org.apache.hadoop.ozone.HddsDatanodeService.main(HddsDatanodeService.java:126)
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
> at
> org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:690)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:794)
> at org.apache.hadoop.ipc.Client$Connection.access$3700(Client.java:411)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1572)
> at org.apache.hadoop.ipc.Client.call(Client.java:1403)
> ... 21 more
> {code}
> Datanodes try to get SCM signed certificate for just 10 times with interval
> of 1 sec. When SCM takes a little longer to come up, datanodes throw an
> exception and fail.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For
[jira] [Created] (HDFS-14791) Fix TestOzoneManagerRatisServer failure
Xiaoyu Yao created HDFS-14791: - Summary: Fix TestOzoneManagerRatisServer failure Key: HDFS-14791 URL: https://issues.apache.org/jira/browse/HDFS-14791 Project: Hadoop HDFS Issue Type: Test Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2039) Some ozone unit test takes too long to finish.
Xiaoyu Yao created HDDS-2039:
Summary: Some ozone unit test takes too long to finish.
Key: HDDS-2039
URL: https://issues.apache.org/jira/browse/HDDS-2039
Project: Hadoop Distributed Data Store
Issue Type: Test
Reporter: Xiaoyu Yao
Here are a few \{code}
[INFO] Running org.apache.hadoop.ozone.om.TestOzoneManagerHA
[INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 436.08
s - in org.apache.hadoop.ozone.om.TestOzoneManagerHA
[INFO] Running org.apache.hadoop.ozone.om.TestOzoneManager
[INFO] Tests run: 26, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 259.566
s - in org.apache.hadoop.ozone.om.TestOzoneManager
[INFO] Running org.apache.hadoop.ozone.om.TestScmSafeMode
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 129.653
s - in org.apache.hadoop.ozone.om.TestScmSafeMode
[INFO] Running org.apache.hadoop.ozone.om.TestOzoneManagerRestart
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 843.129
s - in org.apache.hadoop.ozone.om.TestOzoneManagerRestart
{code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2020) Remove mTLS from Ozone GRPC
Xiaoyu Yao created HDDS-2020: Summary: Remove mTLS from Ozone GRPC Key: HDDS-2020 URL: https://issues.apache.org/jira/browse/HDDS-2020 Project: Hadoop Distributed Data Store Issue Type: Improvement Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Generic GRPC support mTLS for mutual authentication. However, Ozone has built in block token mechanism for server to authenticate the client. We only need TLS for client to authenticate the server and wire encryption. Remove the mTLS support also simplify the GRPC server/client configuration. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1946) CertificateClient should not persist keys/certs to ozone.metadata.dir
Xiaoyu Yao created HDDS-1946: Summary: CertificateClient should not persist keys/certs to ozone.metadata.dir Key: HDDS-1946 URL: https://issues.apache.org/jira/browse/HDDS-1946 Project: Hadoop Distributed Data Store Issue Type: Improvement Reporter: Xiaoyu Yao Assignee: Vivek Ratnavel Subramanian For example, when OM and SCM are deployed on the same host with ozone.metadata.dir defined. SCM can start successfully but OM can not because the key/cert from OM will collide with SCM. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1944) Update document for Ozone HTTP SPNEGO authentication
Xiaoyu Yao created HDDS-1944: Summary: Update document for Ozone HTTP SPNEGO authentication Key: HDDS-1944 URL: https://issues.apache.org/jira/browse/HDDS-1944 Project: Hadoop Distributed Data Store Issue Type: Improvement Affects Versions: 0.4.0 Reporter: Xiaoyu Yao -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1943) TestKeyManagerImpl.testLookupKeyWithLocation is failing
Xiaoyu Yao created HDDS-1943:
Summary: TestKeyManagerImpl.testLookupKeyWithLocation is failing
Key: HDDS-1943
URL: https://issues.apache.org/jira/browse/HDDS-1943
Project: Hadoop Distributed Data Store
Issue Type: Sub-task
Reporter: Xiaoyu Yao
{code}
[ERROR] TestKeyManagerImpl.testLookupKeyWithLocation:757
expected:<102ad7e3-4226-4966-af79-2b12a56f83cb{ip: 32.53.16.224, host:
localhost-32.53.16.224, networkLocation: /default-rack, certSerialId: null}>
but was:
{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1930) Test Topology Aware Job scheduling with Ozone Topology
Xiaoyu Yao created HDDS-1930: Summary: Test Topology Aware Job scheduling with Ozone Topology Key: HDDS-1930 URL: https://issues.apache.org/jira/browse/HDDS-1930 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Xiaoyu Yao My initial results with Terasort does not seem to report the counter properly. Most of the requests are handled by rack locl but no node local. This ticket is opened to add more system testing to validate the feature. Total Allocated Containers: 3778 Each table cell represents the number of NodeLocal/RackLocal/OffSwitch containers satisfied by NodeLocal/RackLocal/OffSwitch resource requests. Node Local Request Rack Local Request Off Switch Request Num Node Local Containers (satisfied by)0 Num Rack Local Containers (satisfied by)0 3648 Num Off Switch Containers (satisfied by)0 96 34 -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1865) Use "ozone.network.topology.aware.read" to control both RPC client and server side logic
[ https://issues.apache.org/jira/browse/HDDS-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao resolved HDDS-1865. -- Resolution: Fixed Fix Version/s: 0.5.0 Thanks [~Sammi] for the contribution. I've merged the patch to trunk. > Use "ozone.network.topology.aware.read" to control both RPC client and server > side logic > - > > Key: HDDS-1865 > URL: https://issues.apache.org/jira/browse/HDDS-1865 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Sammi Chen >Assignee: Sammi Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1901) Fix Ozone HTTP WebConsole Authentication
Xiaoyu Yao created HDDS-1901:
Summary: Fix Ozone HTTP WebConsole Authentication
Key: HDDS-1901
URL: https://issues.apache.org/jira/browse/HDDS-1901
Project: Hadoop Distributed Data Store
Issue Type: Bug
Affects Versions: 0.4.0
Reporter: Vivek Ratnavel Subramanian
Assignee: Xiaoyu Yao
This was found during integration testing where the http authentication is
enabled but anonymous can still access the ozone http web console like scm:9876
or om:9874. This can be reproed with the following configurations added to the
ozonesecure docker-compose.
{code}
CORE-SITE.XML_hadoop.http.authentication.simple.anonymous.allowed=false
CORE-SITE.XML_hadoop.http.authentication.signature.secret.file=/etc/security/http_secret
CORE-SITE.XML_hadoop.http.authentication.type=kerberos
CORE-SITE.XML_hadoop.http.authentication.kerberos.principal=HTTP/_h...@example.com
CORE-SITE.XML_hadoop.http.authentication.kerberos.keytab=/etc/security/keytabs/HTTP.keytab
CORE-SITE.XML_hadoop.http.filter.initializers=org.apache.hadoop.security.AuthenticationFilterInitializer
{code}
After debugging into the KerberosAuthenticationFilter, the root cause is the
name of the keytab does not follow the AuthenticationFilter tradition. The fix
is to change
hdds.scm.http.kerberos.keytab.file to hdds.scm.http.kerberos.keytab and
hdds.om.http.kerberos.keytab.file to hdds.om.http.kerberos.keytab
I will also add an integration test for this under ozonesecure docker-compose.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1896) Suppress WARN log from NetworkTopology#getDistanceCost
Xiaoyu Yao created HDDS-1896:
Summary: Suppress WARN log from NetworkTopology#getDistanceCost
Key: HDDS-1896
URL: https://issues.apache.org/jira/browse/HDDS-1896
Project: Hadoop Distributed Data Store
Issue Type: Sub-task
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
When RackAwareness is enabled and client from outside, the distance calculation
flood SCM log with the following messages. This ticket is opened to suppress
the WARN log.
{code}
2019-08-01 23:08:05,011 WARN org.apache.hadoop.hdds.scm.net.NetworkTopology:
One of the nodes is outside of network topology
2019-08-01 23:08:05,011 WARN org.apache.hadoop.hdds.scm.net.NetworkTopology:
One of the nodes is outside of network topology
2019-08-01 23:08:05,011 WARN org.apache.hadoop.hdds.scm.net.NetworkTopology:
One of the nodes is outside of network topology
{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1894) Support listPipelines by filters in scmcli
Xiaoyu Yao created HDDS-1894:
Summary: Support listPipelines by filters in scmcli
Key: HDDS-1894
URL: https://issues.apache.org/jira/browse/HDDS-1894
Project: Hadoop Distributed Data Store
Issue Type: Sub-task
Reporter: Xiaoyu Yao
Assignee: Junjie Chen
Today scmcli has a subcmd that allow list all pipelines. This ticket is opened
to filter the results by switches, e.g., filter by Factor: THREE and State:
OPEN. This will be useful for trouble shooting in large cluster.
{code}
bin/ozone scmcli listPipelines
Pipeline[ Id: a8d1b0c9-e1d4-49ea-8746-3f61dfb5ee3f, Nodes:
cce44fde-bc8d-4063-97b3-6f557af756e1\{ip: 10.17.112.65, host:
ia0230.halxg.cloudera.com, networkLocation: /default-rack, certSerialId: null},
Type:RATIS, Factor:ONE, State:OPEN]
Pipeline[ Id: c9c453d1-d74c-4414-b87f-1d3585d78a7c, Nodes:
0b7b0b93-8323-4b82-8cc0-a9a5c10ab827\{ip: 10.17.112.29, host:
ia0138.halxg.cloudera.com, networkLocation: /default-rack, certSerialId:
null}c756a0e0-5a1b-4d03-ba5b-cafbcabac877\{ip: 10.17.112.27, host:
ia0134.halxg.cloudera.com, networkLocation: /default-rack, certSerialId:
null}bee45bd7-1ee6-4726-b3d1-81476dc1eb49\{ip: 10.17.112.28, host:
ia0136.halxg.cloudera.com, networkLocation: /default-rack, certSerialId: null},
Type:RATIS, Factor:THREE, State:OPEN]
{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1857) YARN fails on mapreduce in Kerberos enabled cluster
[
https://issues.apache.org/jira/browse/HDDS-1857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao resolved HDDS-1857.
--
Resolution: Not A Problem
> YARN fails on mapreduce in Kerberos enabled cluster
> ---
>
> Key: HDDS-1857
> URL: https://issues.apache.org/jira/browse/HDDS-1857
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Eric Yang
>Assignee: Xiaoyu Yao
>Priority: Blocker
>
> When configured Ozone as secure cluster, running mapreduce job on secure YARN
> produces this error message:
> {code}
> 2019-07-23 19:33:12,168 INFO retry.RetryInvocationHandler:
> com.google.protobuf.ServiceException: java.io.IOException: DestHost:destPort
> eyang-1.openstacklocal:9862 , LocalHost:localPort
> eyang-1.openstacklocal/172.26.111.17:0. Failed on local exception:
> java.io.IOException: Couldn't set up IO streams:
> java.util.ServiceConfigurationError: org.apache.hadoop.security.SecurityInfo:
> Provider org.apache.hadoop.yarn.server.RMNMSecurityInfoClass not a subtype,
> while invoking $Proxy13.submitRequest over
> nodeId=null,nodeAddress=eyang-1.openstacklocal:9862 after 9 failover
> attempts. Trying to failover immediately.
> 2019-07-23 19:33:12,174 ERROR ha.OMFailoverProxyProvider: Failed to connect
> to OM. Attempted 10 retries and 10 failovers
> 2019-07-23 19:33:12,176 ERROR client.OzoneClientFactory: Couldn't create
> protocol class org.apache.hadoop.ozone.client.rpc.RpcClient exception:
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:291)
> at
> org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:169)
> at
> org.apache.hadoop.fs.ozone.BasicOzoneClientAdapterImpl.(BasicOzoneClientAdapterImpl.java:137)
> at
> org.apache.hadoop.fs.ozone.BasicOzoneClientAdapterImpl.(BasicOzoneClientAdapterImpl.java:101)
> at
> org.apache.hadoop.fs.ozone.BasicOzoneClientAdapterImpl.(BasicOzoneClientAdapterImpl.java:86)
> at
> org.apache.hadoop.fs.ozone.OzoneClientAdapterImpl.(OzoneClientAdapterImpl.java:34)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.fs.ozone.OzoneClientAdapterFactory.lambda$createAdapter$1(OzoneClientAdapterFactory.java:66)
> at
> org.apache.hadoop.fs.ozone.OzoneClientAdapterFactory.createAdapter(OzoneClientAdapterFactory.java:116)
> at
> org.apache.hadoop.fs.ozone.OzoneClientAdapterFactory.createAdapter(OzoneClientAdapterFactory.java:62)
> at
> org.apache.hadoop.fs.ozone.OzoneFileSystem.createAdapter(OzoneFileSystem.java:98)
> at
> org.apache.hadoop.fs.ozone.BasicOzoneFileSystem.initialize(BasicOzoneFileSystem.java:144)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3338)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:136)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3387)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3355)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:497)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:245)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:481)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
> at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:352)
> at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:250)
> at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:233)
> at
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:104)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:177)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:327)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:390)
> Caused by: java.io.IOException: DestHost:destPort eyang-1.openstacklocal:9862
> , LocalHost:localPort eyang-1.openstacklocal/172.26.111.17:0. Failed on local
> exception:
[jira] [Created] (HDDS-1845) OMVolumeSetQuotaRequest#validateAndUpdateCache should not return OMVolumeCreateResponse
Xiaoyu Yao created HDDS-1845:
Summary: OMVolumeSetQuotaRequest#validateAndUpdateCache should not
return OMVolumeCreateResponse
Key: HDDS-1845
URL: https://issues.apache.org/jira/browse/HDDS-1845
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Xiaoyu Yao
OMVolumeSetQuotaRequest#validateAndUpdateCache Line 115, we should return
OMVolumeSetQuotaResponse in the failure case.
{code}
return new OMVolumeCreateResponse(null, null,
createErrorOMResponse(omResponse, ex));
{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1839) Change topology sorting related logs in Pipeline from INFO to DEBUG
Xiaoyu Yao created HDDS-1839:
Summary: Change topology sorting related logs in Pipeline from
INFO to DEBUG
Key: HDDS-1839
URL: https://issues.apache.org/jira/browse/HDDS-1839
Project: Hadoop Distributed Data Store
Issue Type: Sub-task
Affects Versions: 0.4.1
Reporter: Xiaoyu Yao
Assignee: Junjie Chen
This will avoid output like
{code}
2019-07-19 22:36:40 INFO Pipeline:342 - Serialize nodesInOrder
[610d4084-7cce-4691-b43a-f9dd5cdb8809\{ip: 192.168.144.3, host:
ozonesecure-mr_datanode_1.ozonesecure-mr_default, networkLocation:
/default-rack, certSerialId: null}] in pipeline
PipelineID=f9ba269c-aba9-4a42-946c-4048d02cb7d1
2019-07-19 22:36:40 INFO Pipeline:342 - Deserialize nodesInOrder
[610d4084-7cce-4691-b43a-f9dd5cdb8809\{ip: 192.168.144.3, host:
ozonesecure-mr_datanode_1.ozonesecure-mr_default, networkLocation:
/default-rack, certSerialId: null}] in pipeline
PipelineID=f9ba269c-aba9-4a42-946c-4048d02cb7d1
{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1824) IllegalArgumentException in NetworkTopologyImpl causes SCM to shutdown
[
https://issues.apache.org/jira/browse/HDDS-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao resolved HDDS-1824.
--
Resolution: Duplicate
> IllegalArgumentException in NetworkTopologyImpl causes SCM to shutdown
> --
>
> Key: HDDS-1824
> URL: https://issues.apache.org/jira/browse/HDDS-1824
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Reporter: Lokesh Jain
>Priority: Blocker
>
>
>
> {code:java}
> 2019-07-18 02:22:18,005 ERROR
> org.apache.hadoop.hdds.scm.container.ReplicationManager: Exception in
> Replication Monitor Thread.
> java.lang.IllegalArgumentException: Affinity node /default-rack/10.17.213.25
> is not a member of topology
> at
> org.apache.hadoop.hdds.scm.net.NetworkTopologyImpl.checkAffinityNode(NetworkTopologyImpl.java:780)
> at
> org.apache.hadoop.hdds.scm.net.NetworkTopologyImpl.chooseRandom(NetworkTopologyImpl.java:408)
> at
> org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRackAware.chooseNode(SCMContainerPlacementRackAware.java:242)
> at
> org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRackAware.chooseDatanodes(SCMContainerPlacementRackAware.java:168)
> at
> org.apache.hadoop.hdds.scm.container.ReplicationManager.handleUnderReplicatedContainer(ReplicationManager.java:487)
> at
> org.apache.hadoop.hdds.scm.container.ReplicationManager.processContainer(ReplicationManager.java:293)
> at
> java.util.concurrent.ConcurrentHashMap$KeySetView.forEach(ConcurrentHashMap.java:4649)
> at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)
> at
> org.apache.hadoop.hdds.scm.container.ReplicationManager.run(ReplicationManager.java:205)
> at java.lang.Thread.run(Thread.java:745)
> 2019-07-18 02:22:18,008 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1: java.lang.IllegalArgumentException: Affinity node
> /default-rack/10.17.213.25 is not a member of topology
> 2019-07-18 02:22:18,010 INFO
> org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter:
> SHUTDOWN_MSG:
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1770) SCM crashes when ReplicationManager is trying to re-replicate under replicated containers
[
https://issues.apache.org/jira/browse/HDDS-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao resolved HDDS-1770.
--
Resolution: Duplicate
> SCM crashes when ReplicationManager is trying to re-replicate under
> replicated containers
> -
>
> Key: HDDS-1770
> URL: https://issues.apache.org/jira/browse/HDDS-1770
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Reporter: Nanda kumar
>Priority: Blocker
>
> SCM crashes with the following exception when ReplicationManager is trying to
> re-replicate under replicated containers
> {noformat}
> 2019-07-08 12:46:36 ERROR ReplicationManager:215 - Exception in Replication
> Monitor Thread.
> java.lang.IllegalArgumentException: Affinity node /default-rack/aab15e2d07cc
> is not a member of topology
> at
> org.apache.hadoop.hdds.scm.net.NetworkTopologyImpl.checkAffinityNode(NetworkTopologyImpl.java:767)
> at
> org.apache.hadoop.hdds.scm.net.NetworkTopologyImpl.chooseRandom(NetworkTopologyImpl.java:407)
> at
> org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRackAware.chooseNode(SCMContainerPlacementRackAware.java:242)
> at
> org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRackAware.chooseDatanodes(SCMContainerPlacementRackAware.java:168)
> at
> org.apache.hadoop.hdds.scm.container.ReplicationManager.handleUnderReplicatedContainer(ReplicationManager.java:487)
> at
> org.apache.hadoop.hdds.scm.container.ReplicationManager.processContainer(ReplicationManager.java:293)
> at
> java.base/java.util.concurrent.ConcurrentHashMap$KeySetView.forEach(ConcurrentHashMap.java:4698)
> at
> java.base/java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1083)
> at
> org.apache.hadoop.hdds.scm.container.ReplicationManager.run(ReplicationManager.java:205)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2019-07-08 12:46:36 INFO ExitUtil:210 - Exiting with status 1:
> java.lang.IllegalArgumentException: Affinity node /default-rack/aab15e2d07cc
> is not a member of topology
> 2019-07-08 12:46:36 INFO StorageContainerManagerStarter:51 - SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down StorageContainerManager at
> 8c763563f672/192.168.112.2
> /
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1813) Fix false warning from ozones3 acceptance test
[
https://issues.apache.org/jira/browse/HDDS-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao resolved HDDS-1813.
--
Resolution: Fixed
> Fix false warning from ozones3 acceptance test
> --
>
> Key: HDDS-1813
> URL: https://issues.apache.org/jira/browse/HDDS-1813
> Project: Hadoop Distributed Data Store
> Issue Type: Test
>Affects Versions: 0.4.1
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
>Priority: Minor
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> All acceptance passed but the results are marked failed due to the following
> warnings.
> [https://ci.anzix.net/job/ozone/17381/RobotTests/log.html]
> {code}
> [ WARN ] Collapsing consecutive whitespace during parsing is deprecated. Fix
> ' # Bucket already is created in Test Setup.' in file
> '/opt/hadoop/smoketest/s3/bucketcreate.robot' on line 31.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1813) Fix false warning from ozones3 acceptance test
Xiaoyu Yao created HDDS-1813:
Summary: Fix false warning from ozones3 acceptance test
Key: HDDS-1813
URL: https://issues.apache.org/jira/browse/HDDS-1813
Project: Hadoop Distributed Data Store
Issue Type: Test
Affects Versions: 0.4.1
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
All acceptance passed but the results are marked failed due to the following
warnings.
[https://ci.anzix.net/job/ozone/17381/RobotTests/log.html]
{code}
[ WARN ] Collapsing consecutive whitespace during parsing is deprecated. Fix '
# Bucket already is created in Test Setup.' in file
'/opt/hadoop/smoketest/s3/bucketcreate.robot' on line 31.
{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1801) Make Topology Aware Replication/Read non-default for ozone 0.4.1
Xiaoyu Yao created HDDS-1801: Summary: Make Topology Aware Replication/Read non-default for ozone 0.4.1 Key: HDDS-1801 URL: https://issues.apache.org/jira/browse/HDDS-1801 Project: Hadoop Distributed Data Store Issue Type: Sub-task Affects Versions: 0.4.1 Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao This helps stablize the ozone-0.4.1 release and fix HDDS-1705, HDDS-1751, HDDS-1713 and HDDS-1770 for 0.5. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1586) Allow Ozone RPC client to read with topology awareness
[ https://issues.apache.org/jira/browse/HDDS-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao resolved HDDS-1586. -- Resolution: Fixed Fix Version/s: 0.5.0 Thanks [~Sammi] for the contribution. I've committed the patch to trunk. > Allow Ozone RPC client to read with topology awareness > -- > > Key: HDDS-1586 > URL: https://issues.apache.org/jira/browse/HDDS-1586 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Sammi Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 4h 10m > Remaining Estimate: 0h > > The idea is to leverage the node location from the block locations and perfer > read from closer block replicas. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1776) Fix typos in ozone docker-compose files
Xiaoyu Yao created HDDS-1776:
Summary: Fix typos in ozone docker-compose files
Key: HDDS-1776
URL: https://issues.apache.org/jira/browse/HDDS-1776
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
The docker compose file has invalid reference to scm images, which fails the
docker-compose up with errors like below. This ticket is opened to fix them.
{code:java}
ERROR: no such image: apache/ozone-runner::20190617-2: invalid reference format}
or
ERROR: no such image: apache/ozone-runner:latest:20190617-2: invalid reference
format{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1713) ReplicationManager fail to find proper node topology based on Datanode details from heartbeat
Xiaoyu Yao created HDDS-1713: Summary: ReplicationManager fail to find proper node topology based on Datanode details from heartbeat Key: HDDS-1713 URL: https://issues.apache.org/jira/browse/HDDS-1713 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao DN does not have the topology info included in its heartbeat message for container report/pipeline report. SCM is where the topology information is available. During the processing of heartbeat, we should not rely on the datanodedetails from report to choose datanodes for close container. Otherwise, all the datanode locations of existing container replicas will fallback to /default-rack. The fix is to retrieve the corresponding datanode locations from scm nodemanager, which has authoritative network topology information. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1695) XceiverClientGrpc#ManagedChannelReference shutdown issue when running TeraGen
Xiaoyu Yao created HDDS-1695:
Summary: XceiverClientGrpc#ManagedChannelReference shutdown issue
when running TeraGen
Key: HDDS-1695
URL: https://issues.apache.org/jira/browse/HDDS-1695
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Xiaoyu Yao
{code}
Jun 17, 2019 8:45:22 AM
org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference
cleanQueue
SEVERE: *~*~*~ Channel ManagedChannelImpl\{logId=1, target=10.17.207.52:9859}
was not shutdown properly!!! ~*~*~*
Make sure to call shutdown()/shutdownNow() and wait until awaitTermination()
returns true.
java.lang.RuntimeException: ManagedChannel allocation site
at
org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.(ManagedChannelOrphanWrapper.java:103)
at
org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:53)
at
org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:44)
at
org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:411)
at
org.apache.hadoop.hdds.scm.XceiverClientGrpc.connectToDatanode(XceiverClientGrpc.java:172)
at
org.apache.hadoop.hdds.scm.XceiverClientGrpc.reconnect(XceiverClientGrpc.java:417)
at
org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandAsync(XceiverClientGrpc.java:366)
at
org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:279)
at
org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:248)
at
org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:231)
at
org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:118)
at
org.apache.hadoop.hdds.scm.storage.BlockInputStream.getChunkInfos(BlockInputStream.java:167)
at
org.apache.hadoop.hdds.scm.storage.BlockInputStream.initialize(BlockInputStream.java:118)
at
org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:222)
at
org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:171)
at
org.apache.hadoop.fs.ozone.OzoneFSInputStream.read(OzoneFSInputStream.java:52)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.examples.terasort.TeraInputFormat$TeraRecordReader.nextKeyValue(TeraInputFormat.java:257)
at
org.apache.hadoop.examples.terasort.TeraInputFormat$1.run(TeraInputFormat.java:154)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1671) Multiple unit test fails because of assertion while validating Acls
[ https://issues.apache.org/jira/browse/HDDS-1671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao resolved HDDS-1671. -- Resolution: Fixed This has been fixed in HDDS-1545. > Multiple unit test fails because of assertion while validating Acls > --- > > Key: HDDS-1671 > URL: https://issues.apache.org/jira/browse/HDDS-1671 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Security >Reporter: Mukul Kumar Singh >Priority: Blocker > > There are multiple unit test failures because of assertion in validateAcls > https://builds.apache.org/job/hadoop-multibranch/job/PR-846/7/testReport/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1587) Support dynamically adding delegated class to filteredclass loader
[ https://issues.apache.org/jira/browse/HDDS-1587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao resolved HDDS-1587. -- Resolution: Fixed Fix Version/s: 0.4.1 Thanks [~junjie] for the contribution. I've merged the change to trunk. > Support dynamically adding delegated class to filteredclass loader > -- > > Key: HDDS-1587 > URL: https://issues.apache.org/jira/browse/HDDS-1587 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Affects Versions: 0.4.0 >Reporter: Xiaoyu Yao >Assignee: Junjie Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.4.1 > > Time Spent: 3h 10m > Remaining Estimate: 0h > > HDDS-922 added a filtered class loader with a list of delegated classes that > will be loaded with the app launcher's classloader. With security enabled on > ozone-0.4, there are some incompatible changes from Hadoop-common and > hadoop-auth module from Hadoop-2.x to Hadoop-3.x. Some examples can be seen > HDDS-1080, where the fix has to be made along with a rebuild/release. > > This ticket is opened to allow dynamically adding delegated classes or class > prefix via environment variable. This way, we can easily adjust the setting > in different deployment without rebuild/release. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1663) Add datanode to network topology cluster during node register
[ https://issues.apache.org/jira/browse/HDDS-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao resolved HDDS-1663. -- Resolution: Fixed Fix Version/s: 0.4.1 Thanks [~Sammi] for the contribution. I've commit the patch to trunk. > Add datanode to network topology cluster during node register > - > > Key: HDDS-1663 > URL: https://issues.apache.org/jira/browse/HDDS-1663 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Sammi Chen >Assignee: Sammi Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.4.1 > > Time Spent: 7h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1612) Add "scmcli printTopology" shell command to print datanode topology
[
https://issues.apache.org/jira/browse/HDDS-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao resolved HDDS-1612.
--
Resolution: Fixed
Fix Version/s: 0.4.1
Thanks [~Sammi] for the contribution. I've committed the patch to trunk.
> Add "scmcli printTopology" shell command to print datanode topology
>
>
> Key: HDDS-1612
> URL: https://issues.apache.org/jira/browse/HDDS-1612
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Sammi Chen
>Assignee: Sammi Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.1
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Command "ozone scmcli printTopology". It will show nodes of all kinds of
> state(HEALTHY,STALE,DEAD,DECOMMISSIONING,DECOMMISSIONED).
> Here is an example,
> {noformat}
> State = HEALTHY
> ozone_datanode_1.ozone_default/172.18.0.3/default-rack
> ozone_datanode_2.ozone_default/172.18.0.2/default-rack
> ozone_datanode_3.ozone_default/172.18.0.4/default-rack
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1652) HddsDispatcher should not shutdown volumeSet
Xiaoyu Yao created HDDS-1652: Summary: HddsDispatcher should not shutdown volumeSet Key: HDDS-1652 URL: https://issues.apache.org/jira/browse/HDDS-1652 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Currently both OzoneContainer#stop() and HddsDispatcher#stop() both invoke volumeSet.shutdown() explicitly. In addition, OzoneContainer#stop() will invoke HddsDispatcher#stop(). Since the volume set object is created by OzoneContainer object, it should be the responsibility of OzoneContainer to shutdown. This ticket is opened to remove the volumeSet.shutdown() from HddsDispatcher#stop(). There are benchmark tools relies on HddsDispatcher#stop() to shutdown volumeSet object, that we could fix with explict volumeSet#shutdown call. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1650) Fix Ozone tests leaking volume checker thread
Xiaoyu Yao created HDDS-1650: Summary: Fix Ozone tests leaking volume checker thread Key: HDDS-1650 URL: https://issues.apache.org/jira/browse/HDDS-1650 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao There are a few test leaking hdds volume checker thread. This ticket is opened to fix them. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1637) Fix random test failure TestSCMContainerPlacementRackAware
[ https://issues.apache.org/jira/browse/HDDS-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao resolved HDDS-1637. -- Resolution: Fixed Fix Version/s: 0.4.1 Thanks [~Sammi] for fixing this. PR has been merged to trunk. > Fix random test failure TestSCMContainerPlacementRackAware > -- > > Key: HDDS-1637 > URL: https://issues.apache.org/jira/browse/HDDS-1637 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Sammi Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.4.1 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > This has been seen randomly in latest trunk CI, e.g., > [https://ci.anzix.net/job/ozone/16980/testReport/org.apache.hadoop.hdds.scm.container.placement.algorithms/TestSCMContainerPlacementRackAware/testFallback/] > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1637) Fix random test failure TestSCMContainerPlacementRackAware
Xiaoyu Yao created HDDS-1637: Summary: Fix random test failure TestSCMContainerPlacementRackAware Key: HDDS-1637 URL: https://issues.apache.org/jira/browse/HDDS-1637 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao This has been seen randomly in latest trunk CI, e.g., [https://ci.anzix.net/job/ozone/16980/testReport/org.apache.hadoop.hdds.scm.container.placement.algorithms/TestSCMContainerPlacementRackAware/testFallback/] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1611) Evaluate ACL and DACL on volume bucket key and perfix to authorize access
Xiaoyu Yao created HDDS-1611: Summary: Evaluate ACL and DACL on volume bucket key and perfix to authorize access Key: HDDS-1611 URL: https://issues.apache.org/jira/browse/HDDS-1611 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Xiaoyu Yao -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1608) Persist Ozone Prefix ACLs in OM metadata table
Xiaoyu Yao created HDDS-1608: Summary: Persist Ozone Prefix ACLs in OM metadata table Key: HDDS-1608 URL: https://issues.apache.org/jira/browse/HDDS-1608 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1604) ContainerReader#initializeUsedBytes leaks DB reference
[ https://issues.apache.org/jira/browse/HDDS-1604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao resolved HDDS-1604. -- Resolution: Fixed Fix Version/s: 0.4.1 Thanks all for the reviews. I've committed the patch to trunk. > ContainerReader#initializeUsedBytes leaks DB reference > -- > > Key: HDDS-1604 > URL: https://issues.apache.org/jira/browse/HDDS-1604 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Labels: pull-request-available > Fix For: 0.4.1 > > Time Spent: 40m > Remaining Estimate: 0h > > This was caught by the New ContainerCache with reference counting from > HDDS-1449. The root cause is an unclosed KeyValueBlockIterator from > ContainerReader#initializeUsedBytes. > I will post a patch shortly, which will fix some UT failures exposed by > -HDDS-1449,- such as TestBCSID#testBCSID, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1536) testSCMSafeModeRestrictedOp is failing consistently
[
https://issues.apache.org/jira/browse/HDDS-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao resolved HDDS-1536.
--
Resolution: Fixed
Thanks [~msingh] for reporting the issue and [~bharatviswa] for the review.
I've committed the patch to trunk.
> testSCMSafeModeRestrictedOp is failing consistently
> ---
>
> Key: HDDS-1536
> URL: https://issues.apache.org/jira/browse/HDDS-1536
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: SCM
>Reporter: Mukul Kumar Singh
>Assignee: Xiaoyu Yao
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.1
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The test is failing with the following stack trace.
> {code}
> [ERROR]
> testSCMSafeModeRestrictedOp(org.apache.hadoop.ozone.om.TestScmSafeMode) Time
> elapsed: 9.79 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.hadoop.ozone.om.TestScmSafeMode.testSCMSafeModeRestrictedOp(TestScmSafeMode.java:304)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1604) ContainerReader#initializeUsedBytes leaks DB reference
Xiaoyu Yao created HDDS-1604: Summary: ContainerReader#initializeUsedBytes leaks DB reference Key: HDDS-1604 URL: https://issues.apache.org/jira/browse/HDDS-1604 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao This was caught by the New ContainerCache with reference counting from HDDS-1449. The root cause is an unclosed KeyValueBlockIterator from ContainerReader#initializeUsedBytes. I will post a patch shortly, which will fix some UT failures exposed by HDDS-1449. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1587) Support dynamically adding delegated class to filteredclass loader
Xiaoyu Yao created HDDS-1587: Summary: Support dynamically adding delegated class to filteredclass loader Key: HDDS-1587 URL: https://issues.apache.org/jira/browse/HDDS-1587 Project: Hadoop Distributed Data Store Issue Type: Bug Affects Versions: 0.4.0 Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao HDDS-922 added a filtered class loader with a list of delegated classes that will be loaded with the app launcher's classloader. With security enabled on ozone-0.4, there are some incompatible changes from Hadoop-common and hadoop-auth module from Hadoop-2.x to Hadoop-3.x. Some examples can be seen HDDS-1080, where the fix has to be made along with a rebuild/release. This ticket is opened to allow dynamically adding delegated classes or class prefix via environment variable. This way, we can easily adjust the setting in different deployment without rebuild/release. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1586) Allow Ozone RPC client to read with topology awareness
Xiaoyu Yao created HDDS-1586: Summary: Allow Ozone RPC client to read with topology awareness Key: HDDS-1586 URL: https://issues.apache.org/jira/browse/HDDS-1586 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Xiaoyu Yao The idea is to leverage the node location from the block locations and perfer read from closer block replicas. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-4) Implement security for Hadoop Distributed Storage Layer
[ https://issues.apache.org/jira/browse/HDDS-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao resolved HDDS-4. --- Resolution: Fixed Resolved as all subtasks are completed and merged. > Implement security for Hadoop Distributed Storage Layer > > > Key: HDDS-4 > URL: https://issues.apache.org/jira/browse/HDDS-4 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: Security >Reporter: Anu Engineer >Assignee: Xiaoyu Yao >Priority: Major > Attachments: HadoopStorageLayerSecurity.pdf > > > In HDFS-7240, we have created a scalable block layer that facilitates > separation of namespace and block layer. Hadoop Distributed Storage Layer > (HDSL) allows us to scale HDFS(HDFS-10419) and as well as create ozone > (HDFS-13074). > This JIRA is an umbrella JIRA that tracks the security-related work items for > Hadoop Distributed Storage Layer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-976) Support YAML format network topology cluster definition
[ https://issues.apache.org/jira/browse/HDDS-976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao resolved HDDS-976. - Resolution: Fixed Fix Version/s: 0.5.0 Thanks [~junjie] for the contribution. I've committed/merged the patch to trunk. > Support YAML format network topology cluster definition > --- > > Key: HDDS-976 > URL: https://issues.apache.org/jira/browse/HDDS-976 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Sammi Chen >Assignee: Junjie Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Attachments: NetworkTopologyDefault.yaml > > Time Spent: 3h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1377) OM failed to start with incorrect hostname set in CSR
Xiaoyu Yao created HDDS-1377:
Summary: OM failed to start with incorrect hostname set in CSR
Key: HDDS-1377
URL: https://issues.apache.org/jira/browse/HDDS-1377
Project: Hadoop Distributed Data Store
Issue Type: Bug
Affects Versions: 0.4.0
Reporter: Xiaoyu Yao
Assignee: Ajay Kumar
OM failed to start after HDDS-1355.
{code}
om_1 | 2019-04-03 16:23:50 ERROR OzoneManager:865 - Failed to start the
OzoneManager.
om_1 | java.lang.IllegalArgumentException: IP Address is invalid
om_1 | at org.bouncycastle.asn1.x509.GeneralName.(Unknown Source)
om_1 | at
org.apache.hadoop.hdds.security.x509.certificates.utils.CertificateSignRequest$Builder.addAltName(CertificateSignRequest.java:205)
om_1 | at
org.apache.hadoop.hdds.security.x509.certificates.utils.CertificateSignRequest$Builder.addIpAddress(CertificateSignRequest.java:197)
om_1 | at
org.apache.hadoop.ozone.om.OzoneManager.getSCMSignedCert(OzoneManager.java:1387)
om_1 | at
org.apache.hadoop.ozone.om.OzoneManager.initializeSecurity(OzoneManager.java:1018)
om_1 | at org.apache.hadoop.ozone.om.OzoneManager.omInit(OzoneManager.java:971)
om_1 | at
org.apache.hadoop.ozone.om.OzoneManager.createOm(OzoneManager.java:928)
om_1 | at org.apache.hadoop.ozone.om.OzoneManager.main(OzoneManager.java:859)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1350) Fix checkstyle issue in TestDatanodeStateMachine
Xiaoyu Yao created HDDS-1350: Summary: Fix checkstyle issue in TestDatanodeStateMachine Key: HDDS-1350 URL: https://issues.apache.org/jira/browse/HDDS-1350 Project: Hadoop Distributed Data Store Issue Type: Improvement Reporter: Xiaoyu Yao The following tests are FAILED: [checkstyle]: checkstyle check is failed ([https://ci.anzix.net/job/ozone-nightly/44/checkstyle/]) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1346) Remove hard-coded version ozone-0.5 from ReadMe of ozonesecure-mr docker-compose
Xiaoyu Yao created HDDS-1346:
Summary: Remove hard-coded version ozone-0.5 from ReadMe of
ozonesecure-mr docker-compose
Key: HDDS-1346
URL: https://issues.apache.org/jira/browse/HDDS-1346
Project: Hadoop Distributed Data Store
Issue Type: Improvement
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
As we are releasing ozone-0.4, we should not have hard-coded ozone-0.5 for
trunk.
The proposal is to use the following to replace it
{{cd}} {{$(git rev-parse
--show-toplevel)}}{{/hadoop-ozone/dist/target/ozone-}}{{*-SNAPSHOT}}{{/compose/ozone}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1326) putkey operation failed with java.lang.ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HDDS-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao resolved HDDS-1326.
--
Resolution: Duplicate
> putkey operation failed with java.lang.ArrayIndexOutOfBoundsException
> -
>
> Key: HDDS-1326
> URL: https://issues.apache.org/jira/browse/HDDS-1326
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Nilotpal Nandi
>Assignee: Shashikant Banerjee
>Priority: Blocker
>
> steps taken :
> ---
> # trying to write key in 40 node cluster.
> # write failed.
> client output
> ---
>
> {noformat}
> e530-491c-ab03-3b1c34d1a751:c80390,
> 974a806d-bf7d-4f1b-adb4-d51d802d368a:c80390,
> 469bd8c4-5da2-43bb-bc4b-7edd884931e5:c80390]
> 2019-03-22 10:56:19,592 [main] WARN - Encountered exception {}
> java.io.IOException: Unexpected Storage Container Exception:
> java.util.concurrent.ExecutionException:
> java.util.concurrent.CompletionException:
> org.apache.ratis.protocol.StateMachineException:
> org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException
> from Server 5d3eb91f-e530-491c-ab03-3b1c34d1a751: Container 1269 in CLOSED
> state
> at
> org.apache.hadoop.hdds.scm.storage.BlockOutputStream.close(BlockOutputStream.java:511)
> at
> org.apache.hadoop.ozone.client.io.BlockOutputStreamEntry.close(BlockOutputStreamEntry.java:144)
> at
> org.apache.hadoop.ozone.client.io.KeyOutputStream.handleFlushOrClose(KeyOutputStream.java:565)
> at
> org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:329)
> at
> org.apache.hadoop.ozone.client.io.KeyOutputStream.write(KeyOutputStream.java:273)
> at
> org.apache.hadoop.ozone.client.io.OzoneOutputStream.write(OzoneOutputStream.java:49)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:96)
> at
> org.apache.hadoop.ozone.web.ozShell.keys.PutKeyHandler.call(PutKeyHandler.java:111)
> at
> org.apache.hadoop.ozone.web.ozShell.keys.PutKeyHandler.call(PutKeyHandler.java:53)
> at picocli.CommandLine.execute(CommandLine.java:919)
> at picocli.CommandLine.access$700(CommandLine.java:104)
> at picocli.CommandLine$RunLast.handle(CommandLine.java:1083)
> at picocli.CommandLine$RunLast.handle(CommandLine.java:1051)
> at
> picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:959)
> at picocli.CommandLine.parseWithHandlers(CommandLine.java:1242)
> at picocli.CommandLine.parseWithHandler(CommandLine.java:1181)
> at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:61)
> at org.apache.hadoop.ozone.web.ozShell.Shell.execute(Shell.java:82)
> at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:52)
> at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:93)
> Caused by: java.util.concurrent.ExecutionException:
> java.util.concurrent.CompletionException:
> org.apache.ratis.protocol.StateMachineException:
> org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException
> from Server 5d3eb91f-e530-491c-ab03-3b1c34d1a751: Container 1269 in CLOSED
> state
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at
> org.apache.hadoop.hdds.scm.storage.BlockOutputStream.waitOnFlushFutures(BlockOutputStream.java:529)
> at
> org.apache.hadoop.hdds.scm.storage.BlockOutputStream.handleFlush(BlockOutputStream.java:481)
> at
> org.apache.hadoop.hdds.scm.storage.BlockOutputStream.close(BlockOutputStream.java:496)
> ... 19 more
> Caused by: java.util.concurrent.CompletionException:
> org.apache.ratis.protocol.StateMachineException:
> org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException
> from Server 5d3eb91f-e530-491c-ab03-3b1c34d1a751: Container 1269 in CLOSED
> state
> at
> org.apache.ratis.client.impl.RaftClientImpl.handleStateMachineException(RaftClientImpl.java:402)
> at
> org.apache.ratis.client.impl.RaftClientImpl.lambda$sendAsync$3(RaftClientImpl.java:198)
> at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at
> org.apache.ratis.client.impl.RaftClientImpl$PendingAsyncRequest.setReply(RaftClientImpl.java:95)
> at
> org.apache.ratis.client.impl.RaftClientImpl$PendingAsyncRequest.setReply(RaftClientImpl.java:75)
> at
> org.apache.ratis.util.SlidingWindow$RequestMap.setReply(SlidingWindow.java:127)
> at
>
[jira] [Resolved] (HDDS-1327) ClientProtocol#close failed with EOFException running distcp from o3fs to o3fs
[
https://issues.apache.org/jira/browse/HDDS-1327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao resolved HDDS-1327.
--
Resolution: Duplicate
This is a dup of HDDS-1317
> ClientProtocol#close failed with EOFException running distcp from o3fs to o3fs
> --
>
> Key: HDDS-1327
> URL: https://issues.apache.org/jira/browse/HDDS-1327
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Xiaoyu Yao
>Priority: Blocker
>
> {code}
> 2019-03-22 15:47:21 DEBUG OzoneFileSystem:843 - Found file (with /) for path
> o3fs://bucket1.vol1/user/hadoop/.staging: fake directory
> 2019-03-22 15:47:21 ERROR DistCp:167 - Exception encountered
> java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:197)
> at
> org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:70)
> at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:120)
> at org.apache.hadoop.io.SequenceFile$Reader.nextRaw(SequenceFile.java:2522)
> at
> org.apache.hadoop.io.SequenceFile$Sorter$SortPass.run(SequenceFile.java:2994)
> at org.apache.hadoop.io.SequenceFile$Sorter.sortPass(SequenceFile.java:2932)
> at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2880)
> at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2921)
> at org.apache.hadoop.tools.util.DistCpUtils.sortListing(DistCpUtils.java:474)
> at org.apache.hadoop.tools.util.DistCpUtils.sortListing(DistCpUtils.java:448)
> at
> org.apache.hadoop.tools.CopyListing.validateFinalListing(CopyListing.java:155)
> at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:93)
> at
> org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:89)
> at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)
> at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:368)
> at org.apache.hadoop.tools.DistCp.prepareFileListing(DistCp.java:96)
> at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:205)
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:182)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:153)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:432)
> 2019-03-22 15:47:21 DEBUG Client:97 - stopping client from cache:
> org.apache.hadoop.ipc.Client@4944252c
> 2019-03-22 15:47:21 DEBUG OzoneClient:55 - Call: public abstract void
> org.apache.hadoop.ozone.client.protocol.ClientProtocol.close() throws
> java.io.IOException took 25 ms
> 2019-03-22 15:47:21 DEBUG ShutdownHookManager:97 - Completed shutdown in
> 0.026 seconds; Timeouts: 0
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1327) ClientProtocol#close failed with EOFException running distcp from o3fs to o3fs
Xiaoyu Yao created HDDS-1327:
Summary: ClientProtocol#close failed with EOFException running
distcp from o3fs to o3fs
Key: HDDS-1327
URL: https://issues.apache.org/jira/browse/HDDS-1327
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Xiaoyu Yao
{code}
2019-03-22 15:47:21 DEBUG OzoneFileSystem:843 - Found file (with /) for path
o3fs://bucket1.vol1/user/hadoop/.staging: fake directory
2019-03-22 15:47:21 ERROR DistCp:167 - Exception encountered
java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:197)
at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:70)
at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:120)
at org.apache.hadoop.io.SequenceFile$Reader.nextRaw(SequenceFile.java:2522)
at
org.apache.hadoop.io.SequenceFile$Sorter$SortPass.run(SequenceFile.java:2994)
at org.apache.hadoop.io.SequenceFile$Sorter.sortPass(SequenceFile.java:2932)
at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2880)
at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2921)
at org.apache.hadoop.tools.util.DistCpUtils.sortListing(DistCpUtils.java:474)
at org.apache.hadoop.tools.util.DistCpUtils.sortListing(DistCpUtils.java:448)
at
org.apache.hadoop.tools.CopyListing.validateFinalListing(CopyListing.java:155)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:93)
at
org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:89)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)
at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:368)
at org.apache.hadoop.tools.DistCp.prepareFileListing(DistCp.java:96)
at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:205)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:182)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:153)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:432)
2019-03-22 15:47:21 DEBUG Client:97 - stopping client from cache:
org.apache.hadoop.ipc.Client@4944252c
2019-03-22 15:47:21 DEBUG OzoneClient:55 - Call: public abstract void
org.apache.hadoop.ozone.client.protocol.ClientProtocol.close() throws
java.io.IOException took 25 ms
2019-03-22 15:47:21 DEBUG ShutdownHookManager:97 - Completed shutdown in 0.026
seconds; Timeouts: 0
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1318) Fix MalformedTracerStateStringException on DN logs
Xiaoyu Yao created HDDS-1318:
Summary: Fix MalformedTracerStateStringException on DN logs
Key: HDDS-1318
URL: https://issues.apache.org/jira/browse/HDDS-1318
Project: Hadoop Distributed Data Store
Issue Type: Sub-task
Reporter: Xiaoyu Yao
Have seen many warnings on DN logs. This ticket is opened to track the
investigation and fix for this.
{code}
2019-03-20 19:01:33 WARN
PropagationRegistry$ExceptionCatchingExtractorDecorator:60 - Error when
extracting SpanContext from carrier. Handling gracefully.
io.jaegertracing.internal.exceptions.MalformedTracerStateStringException:
String does not match tracer state format: 2c919331-9a51-4bc4-acee-df57a8dcecf0
at org.apache.hadoop.hdds.tracing.StringCodec.extract(StringCodec.java:42)
at org.apache.hadoop.hdds.tracing.StringCodec.extract(StringCodec.java:32)
at
io.jaegertracing.internal.PropagationRegistry$ExceptionCatchingExtractorDecorator.extract(PropagationRegistry.java:57)
at io.jaegertracing.internal.JaegerTracer.extract(JaegerTracer.java:208)
at io.jaegertracing.internal.JaegerTracer.extract(JaegerTracer.java:61)
at io.opentracing.util.GlobalTracer.extract(GlobalTracer.java:143)
at
org.apache.hadoop.hdds.tracing.TracingUtil.importAndCreateScope(TracingUtil.java:96)
at
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:148)
at
org.apache.hadoop.ozone.container.common.transport.server.GrpcXceiverService$1.onNext(GrpcXceiverService.java:73)
at
org.apache.hadoop.ozone.container.common.transport.server.GrpcXceiverService$1.onNext(GrpcXceiverService.java:61)
at
org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:248)
at
org.apache.ratis.thirdparty.io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33)
at
org.apache.ratis.thirdparty.io.grpc.Contexts$ContextualizedServerCallListener.onMessage(Contexts.java:76)
at
org.apache.ratis.thirdparty.io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33)
at
org.apache.hadoop.hdds.tracing.GrpcServerInterceptor$1.onMessage(GrpcServerInterceptor.java:46)
at
org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:263)
at
org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:686)
at
org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1317) KeyOutputStream#write throws ArrayIndexOutOfBoundsException when running RandomWrite MR examples
Xiaoyu Yao created HDDS-1317:
Summary: KeyOutputStream#write throws
ArrayIndexOutOfBoundsException when running RandomWrite MR examples
Key: HDDS-1317
URL: https://issues.apache.org/jira/browse/HDDS-1317
Project: Hadoop Distributed Data Store
Issue Type: Sub-task
Affects Versions: 0.4.0
Reporter: Xiaoyu Yao
Repro steps:
{code}
jar $HADOOP_MAPRED_HOME/hadoop-mapreduce-examples-*.jar randomwriter
-Dtest.randomwrite.total_bytes=1000 o3fs://bucket1.vol1/randomwrite.out
{code}
Error Stack:
{code}
2019-03-20 19:02:37 INFO Job:1686 - Task Id :
attempt_1553108378906_0002_m_00_0, Status : FAILED
Error: java.lang.ArrayIndexOutOfBoundsException: -5
at java.util.ArrayList.elementData(ArrayList.java:422)
at java.util.ArrayList.get(ArrayList.java:435)
at org.apache.hadoop.hdds.scm.storage.BufferPool.getBuffer(BufferPool.java:45)
at
org.apache.hadoop.hdds.scm.storage.BufferPool.allocateBufferIfNeeded(BufferPool.java:59)
at
org.apache.hadoop.hdds.scm.storage.BlockOutputStream.write(BlockOutputStream.java:215)
at
org.apache.hadoop.ozone.client.io.BlockOutputStreamEntry.write(BlockOutputStreamEntry.java:130)
at
org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:311)
at
org.apache.hadoop.ozone.client.io.KeyOutputStream.write(KeyOutputStream.java:273)
at
org.apache.hadoop.fs.ozone.OzoneFSOutputStream.write(OzoneFSOutputStream.java:46)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:57)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1444)
at
org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat$1.write(SequenceFileOutputFormat.java:83)
at
org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:670)
at
org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at
org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
at
org.apache.hadoop.examples.RandomWriter$RandomMapper.map(RandomWriter.java:199)
at
org.apache.hadoop.examples.RandomWriter$RandomMapper.map(RandomWriter.java:165)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-747) Update MiniOzoneCluster to work with security protocol from SCM
[ https://issues.apache.org/jira/browse/HDDS-747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao resolved HDDS-747. - Resolution: Invalid This won't work with different components require separate Kerberos login of different principles in the same JVM. We will look into [https://www.testcontainers.org/] to test secure docker compose in the next release. > Update MiniOzoneCluster to work with security protocol from SCM > --- > > Key: HDDS-747 > URL: https://issues.apache.org/jira/browse/HDDS-747 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Priority: Major > Labels: ozone-security > > [HDDS-103] introduces a new security protocol in SCM. MiniOzoneCluster should > be updated to utilize it once its implementation is completed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1299) Support TokenIssuer interface to run MR/Spark with OzoneFileSystem in secure mode
Xiaoyu Yao created HDDS-1299: Summary: Support TokenIssuer interface to run MR/Spark with OzoneFileSystem in secure mode Key: HDDS-1299 URL: https://issues.apache.org/jira/browse/HDDS-1299 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao This ticket is opened to add TokenIssuer interface support to OzoneFileSystem so that MR and Spark jobs can run with OzoneFileSystem in secure mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1296) Fix checkstyle issue from Nightly run
Xiaoyu Yao created HDDS-1296: Summary: Fix checkstyle issue from Nightly run Key: HDDS-1296 URL: https://issues.apache.org/jira/browse/HDDS-1296 Project: Hadoop Distributed Data Store Issue Type: Sub-task Affects Versions: 0.4.0 Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao https://ci.anzix.net/job/ozone-nightly/28/checkstyle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1292) Fix nightly run findbugs and checkstyle issues
Xiaoyu Yao created HDDS-1292: Summary: Fix nightly run findbugs and checkstyle issues Key: HDDS-1292 URL: https://issues.apache.org/jira/browse/HDDS-1292 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Xiaoyu Yao https://ci.anzix.net/job/ozone/3775/findbugs/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1138) OzoneManager should return the pipeline info of the allocated block along with block info
[ https://issues.apache.org/jira/browse/HDDS-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao resolved HDDS-1138. -- Resolution: Fixed Assignee: Xiaoyu Yao (was: Mukul Kumar Singh) Fix Version/s: 0.4.0 > OzoneManager should return the pipeline info of the allocated block along > with block info > - > > Key: HDDS-1138 > URL: https://issues.apache.org/jira/browse/HDDS-1138 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client, Ozone Manager >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Xiaoyu Yao >Priority: Major > Labels: pull-request-available > Fix For: 0.4.0 > > Attachments: HDDS-1138.001.patch > > Time Spent: 1h > Remaining Estimate: 0h > > Currently, while a block is allocated from OM, the request is forwarded to > SCM. However, even though the pipeline information is present with the OM for > block allocation, this information is passed through to the client. > This optimization will help in reducing the number of hops for the client by > reducing 1 RPC round trip for each block allocated. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1291) Set OmKeyArgs#refreshPipeline flag properly when client reads a stale pipeline
Xiaoyu Yao created HDDS-1291:
Summary: Set OmKeyArgs#refreshPipeline flag properly when client
reads a stale pipeline
Key: HDDS-1291
URL: https://issues.apache.org/jira/browse/HDDS-1291
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
After HDDS-1138, the OM client will not talk to SCM directly to fetch the
pipeline info. Instead the pipeline info is returned as part of the keyLocation
cached by OM.
In case SCM pipeline is changed such as closed, the client may get invalid
pipeline exception. In this case, the client need to getKeyLocation with
OmKeyArgs#refreshPipeline = true to force OM update its pipeline cache for this
key.
An optimization could be queue a background task to update all the keyLocations
that is affected when OM does a refreshPipeline. (This part can be done in 0.5)
{code:java}
oldpipeline->newpipeline{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1287) Change some GrpcClientProtocolClient log from INFO to DEBUG
Xiaoyu Yao created HDDS-1287:
Summary: Change some GrpcClientProtocolClient log from INFO to
DEBUG
Key: HDDS-1287
URL: https://issues.apache.org/jira/browse/HDDS-1287
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
It is too much for every ozone sh key put.
{code}
bash-4.2$ ozone sh key put /vol1/bucket1/key1 LICENSE.txt
2019-03-14 16:35:03 INFO GrpcClientProtocolClient:302 - schedule 3000ms timeout
check for
RaftClientRequest:client-D48F06A03AF5->45e6a76d-7c48-4e3d-b18e-fee47055793b@group-D2B35873B94E,
cid=0, seq=0 RW,
org.apache.hadoop.hdds.scm.XceiverClientRatis$$Lambda$64/878991463@3e6fd0b9
2019-03-14 16:35:04 INFO GrpcClientProtocolClient:256 -
client-D48F06A03AF5->45e6a76d-7c48-4e3d-b18e-fee47055793b: receive
RaftClientReply:client-D48F06A03AF5->45e6a76d-7c48-4e3d-b18e-fee47055793b@group-D2B35873B94E,
cid=0, SUCCESS, logIndex=1, commits[45e6a76d-7c48-4e3d-b18e-fee47055793b:c2]
2019-03-14 16:35:04 INFO GrpcClientProtocolClient:302 - schedule 3000ms timeout
check for
RaftClientRequest:client-D48F06A03AF5->45e6a76d-7c48-4e3d-b18e-fee47055793b@group-D2B35873B94E,
cid=1, seq=1 RW,
org.apache.hadoop.hdds.scm.XceiverClientRatis$$Lambda$64/878991463@25211d10
2019-03-14 16:35:04 INFO GrpcClientProtocolClient:256 -
client-D48F06A03AF5->45e6a76d-7c48-4e3d-b18e-fee47055793b: receive
RaftClientReply:client-D48F06A03AF5->45e6a76d-7c48-4e3d-b18e-fee47055793b@group-D2B35873B94E,
cid=1, SUCCESS, logIndex=3, commits[45e6a76d-7c48-4e3d-b18e-fee47055793b:c4]
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1253) Fix checkstyle issue from Nightly run
Xiaoyu Yao created HDDS-1253: Summary: Fix checkstyle issue from Nightly run Key: HDDS-1253 URL: https://issues.apache.org/jira/browse/HDDS-1253 Project: Hadoop Distributed Data Store Issue Type: Sub-task Affects Versions: 0.4.0 Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao https://ci.anzix.net/job/ozone-nightly/28/checkstyle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1246) Support ozone delegation token utility subcmd for Ozone CLI
Xiaoyu Yao created HDDS-1246: Summary: Support ozone delegation token utility subcmd for Ozone CLI Key: HDDS-1246 URL: https://issues.apache.org/jira/browse/HDDS-1246 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao This allow running dtutil with integration test and dev test for demo of Ozone security. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1245) OM delegation expiration time should use Time.now instead of Time.monotonicNow
Xiaoyu Yao created HDDS-1245:
Summary: OM delegation expiration time should use Time.now instead
of Time.monotonicNow
Key: HDDS-1245
URL: https://issues.apache.org/jira/browse/HDDS-1245
Project: Hadoop Distributed Data Store
Issue Type: Sub-task
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
Otherwise, we will set incorrect the exp date of OM delegation like below:
{code}
ozone dtutil print /tmp/om.dt
File: /tmp/om.dt
Token kind Service Renewer Exp date URL
enc token
OzoneToken om:9862 yarn*1/8/70 12:03 PM*
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1236) Fix incorrect Ozone ClientProtocol KerberosInfo annotation
Xiaoyu Yao created HDDS-1236: Summary: Fix incorrect Ozone ClientProtocol KerberosInfo annotation Key: HDDS-1236 URL: https://issues.apache.org/jira/browse/HDDS-1236 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao The serverPrincipal should be OMConfigKeys.OZONE_OM_KERBEROS_PRINCIPAL_KEY instead of ScmConfigKeys.HDDS_SCM_KERBEROS_PRINCIPAL_KEY -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1235) BaseHttpServer NPE is HTTP policy is HTTPS_ONLY
Xiaoyu Yao created HDDS-1235: Summary: BaseHttpServer NPE is HTTP policy is HTTPS_ONLY Key: HDDS-1235 URL: https://issues.apache.org/jira/browse/HDDS-1235 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao This needs to be fixed when Ozone is running inside DN as plugin and DN is running using non-privilege HTTPS port. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1215) Ozonesecure docker compose KMS instance errors due to Java11
Xiaoyu Yao created HDDS-1215:
Summary: Ozonesecure docker compose KMS instance errors due to
Java11
Key: HDDS-1215
URL: https://issues.apache.org/jira/browse/HDDS-1215
Project: Hadoop Distributed Data Store
Issue Type: Sub-task
Reporter: Xiaoyu Yao
{code}
kms_1 | Exception in thread "main" java.lang.NoClassDefFoundError:
javax/activation/DataSource
kms_1 | at
com.sun.xml.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl.(RuntimeBuiltinLeafInfoImpl.java:457)
kms_1 | at
com.sun.xml.bind.v2.model.impl.RuntimeTypeInfoSetImpl.(RuntimeTypeInfoSetImpl.java:65)
kms_1 | at
com.sun.xml.bind.v2.model.impl.RuntimeModelBuilder.createTypeInfoSet(RuntimeModelBuilder.java:133)
kms_1 | at
com.sun.xml.bind.v2.model.impl.RuntimeModelBuilder.createTypeInfoSet(RuntimeModelBuilder.java:85)
kms_1 | at
com.sun.xml.bind.v2.model.impl.ModelBuilder.(ModelBuilder.java:156)
kms_1 | at
com.sun.xml.bind.v2.model.impl.RuntimeModelBuilder.(RuntimeModelBuilder.java:93)
kms_1 | at
com.sun.xml.bind.v2.runtime.JAXBContextImpl.getTypeInfoSet(JAXBContextImpl.java:473)
kms_1 | at
com.sun.xml.bind.v2.runtime.JAXBContextImpl.(JAXBContextImpl.java:319)
kms_1 | at
com.sun.xml.bind.v2.runtime.JAXBContextImpl$JAXBContextBuilder.build(JAXBContextImpl.java:1170)
kms_1 | at
com.sun.xml.bind.v2.ContextFactory.createContext(ContextFactory.java:145)
kms_1 | at
com.sun.xml.bind.v2.ContextFactory.createContext(ContextFactory.java:236)
kms_1 | at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
kms_1 | at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
kms_1 | at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
kms_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566)
kms_1 | at
javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:186)
kms_1 | at
javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:146)
kms_1 | at javax.xml.bind.ContextFinder.find(ContextFinder.java:350)
kms_1 | at
javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:446)
kms_1 | at
javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:409)
kms_1 | at
com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.(WadlApplicationContextImpl.java:103)
kms_1 | at
com.sun.jersey.server.impl.wadl.WadlFactory.init(WadlFactory.java:100)
kms_1 | at
com.sun.jersey.server.impl.application.RootResourceUriRules.initWadl(RootResourceUriRules.java:169)
kms_1 | at
com.sun.jersey.server.impl.application.RootResourceUriRules.(RootResourceUriRules.java:106)
kms_1 | at
com.sun.jersey.server.impl.application.WebApplicationImpl._initiate(WebApplicationImpl.java:1359)
kms_1 | at
com.sun.jersey.server.impl.application.WebApplicationImpl.access$700(WebApplicationImpl.java:180)
kms_1 | at
com.sun.jersey.server.impl.application.WebApplicationImpl$13.f(WebApplicationImpl.java:799)
kms_1 | at
com.sun.jersey.server.impl.application.WebApplicationImpl$13.f(WebApplicationImpl.java:795)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1204) Fix misc issue to make ozonesecure docker-compose work on Java 11
Xiaoyu Yao created HDDS-1204: Summary: Fix misc issue to make ozonesecure docker-compose work on Java 11 Key: HDDS-1204 URL: https://issues.apache.org/jira/browse/HDDS-1204 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao The ozonesecure docker-compose has been changed to use hadoop-runner image based on Java 11. Several class has been removed from Java 8 such as javax.xml.bind.DatatypeConverter.parseHexBinary This ticket is opened to fix issues running ozonesecure docker-compose on java 11. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1190) Fix jdk 11 issue for ozonesecure base image and docker-compose
Xiaoyu Yao created HDDS-1190:
Summary: Fix jdk 11 issue for ozonesecure base image and
docker-compose
Key: HDDS-1190
URL: https://issues.apache.org/jira/browse/HDDS-1190
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
HDDS-1019 changes to use hadoop-runner as base image for ozonesecure
docker-compose. There are a few issues that need to fixed.
1.The hadoop-runner uses jdk11 but the ozonesecure/docker-config assume
openjdk8 for JAVA_HOME.
2. The KEYTAB_DIR needs to be quoted with '.
3. keytab based login failed with Message stream modified (41), [~elek]
mentioned in HDDS-1019 that we need to add max_renewable_life to
"docker-image/docker-krb5/krb5.conf" like follows.
[realms]
EXAMPLE.COM = \{
kdc = localhost
admin_server = localhost
max_renewable_life = 7d
}
Failures:
{code}
org.apache.hadoop.security.KerberosAuthException: failure to login: for
principal: scm/s...@example.com from keytab /etc/security/keytabs/scm.keytab
javax.security.auth.login.LoginException: Message stream modified (41)
scm_1 | at
org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1847)
scm_1 |
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1183) OzoneFileSystem needs to override delegation token APIs
Xiaoyu Yao created HDDS-1183: Summary: OzoneFileSystem needs to override delegation token APIs Key: HDDS-1183 URL: https://issues.apache.org/jira/browse/HDDS-1183 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao This includes addDelegationToken/renewDelegationToken/cancelDelegationToken so that MR jobs can collect tokens correctly upon job submission time. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1179) Ozone dist build failed on Jenkins
Xiaoyu Yao created HDDS-1179:
Summary: Ozone dist build failed on Jenkins
Key: HDDS-1179
URL: https://issues.apache.org/jira/browse/HDDS-1179
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
This is part of the Jenkins execution and was reported in several latest HDDS
Jenkins run.
I spend some today and found a simplified repro steps:
{code}
cd hadoop-ozone/dist
mvn -Phdds -DskipTests -fae clean install -DskipTests=true
-Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Dfindbugs.skip=true
{code}
The root cause is that the
hadoop-ozone/dist/dev-support/bin/dist-layout-stitching need objectstore-sevice
jar being build earlier but he dependency was not explicitly declared in pom. I
will attach a fix shortly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1119) DN get the certificate from SCM CA for token validation
Xiaoyu Yao created HDDS-1119: Summary: DN get the certificate from SCM CA for token validation Key: HDDS-1119 URL: https://issues.apache.org/jira/browse/HDDS-1119 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao This is needed when the OM received delegation token signed by other OM instances and it does not have the certificate for foreign OM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1118) OM get the certificate from SCM CA for token validation
Xiaoyu Yao created HDDS-1118: Summary: OM get the certificate from SCM CA for token validation Key: HDDS-1118 URL: https://issues.apache.org/jira/browse/HDDS-1118 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao This is needed when the OM received delegation token signed by other OM instances and it does not have the certificate for foreign OM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1111) OzoneManager NPE reading private key file.
Xiaoyu Yao created HDDS-:
Summary: OzoneManager NPE reading private key file.
Key: HDDS-
URL: https://issues.apache.org/jira/browse/HDDS-
Project: Hadoop Distributed Data Store
Issue Type: Sub-task
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
{code}
ozoneManager_1 | 2019-02-14 23:21:51 ERROR OzoneManager:596 - Unable to read
key pair for OM.
ozoneManager_1 | org.apache.hadoop.ozone.security.OzoneSecurityException:
Error reading private file for OzoneManager
ozoneManager_1 | at
org.apache.hadoop.ozone.om.OzoneManager.readKeyPair(OzoneManager.java:638)
ozoneManager_1 | at
org.apache.hadoop.ozone.om.OzoneManager.startSecretManager(OzoneManager.java:594)
ozoneManager_1 | at
org.apache.hadoop.ozone.om.OzoneManager.startSecretManagerIfNecessary(OzoneManager.java:1216)
ozoneManager_1 | at
org.apache.hadoop.ozone.om.OzoneManager.start(OzoneManager.java:1007)
ozoneManager_1 | at
org.apache.hadoop.ozone.om.OzoneManager.main(OzoneManager.java:768)
ozoneManager_1 | Caused by: java.lang.NullPointerException
ozoneManager_1 | at
org.apache.hadoop.ozone.om.OzoneManager.readKeyPair(OzoneManager.java:635)
ozoneManager_1 | ... 4 more
ozoneManager_1 | 2019-02-14 23:21:51 ERROR OzoneManager:772 - Failed to start
the OzoneManager.
ozoneManager_1 | java.lang.RuntimeException:
org.apache.hadoop.ozone.security.OzoneSecurityException: Error reading private
file for OzoneManager
ozoneManager_1 | at
org.apache.hadoop.ozone.om.OzoneManager.startSecretManager(OzoneManager.java:597)
ozoneManager_1 | at
org.apache.hadoop.ozone.om.OzoneManager.startSecretManagerIfNecessary(OzoneManager.java:1216)
ozoneManager_1 | at
org.apache.hadoop.ozone.om.OzoneManager.start(OzoneManager.java:1007)
ozoneManager_1 | at
org.apache.hadoop.ozone.om.OzoneManager.main(OzoneManager.java:768)
ozoneManager_1 | Caused by:
org.apache.hadoop.ozone.security.OzoneSecurityException: Error reading private
file for OzoneManager
ozoneManager_1 | at
org.apache.hadoop.ozone.om.OzoneManager.readKeyPair(OzoneManager.java:638)
ozoneManager_1 | at
org.apache.hadoop.ozone.om.OzoneManager.startSecretManager(OzoneManager.java:594)
ozoneManager_1 | ... 3 more
ozoneManager_1 | Caused by: java.lang.NullPointerException
ozoneManager_1 | at
org.apache.hadoop.ozone.om.OzoneManager.readKeyPair(OzoneManager.java:635)
ozoneManager_1 | ... 4 more
ozoneManager_1 | 2019-02-14 23:21:51 INFO ExitUtil:210 - Exiting with status
1: java.lang.RuntimeException:
org.apache.hadoop.ozone.security.OzoneSecurityException: Error reading private
file for OzoneManager
ozoneManager_1 | 2019-02-14 23:21:51 INFO OzoneManager:51 - SHUTDOWN_MSG:
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1110) OzoneManager need to login during init when security is enabled.
Xiaoyu Yao created HDDS-1110:
Summary: OzoneManager need to login during init when security is
enabled.
Key: HDDS-1110
URL: https://issues.apache.org/jira/browse/HDDS-1110
Project: Hadoop Distributed Data Store
Issue Type: Sub-task
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
HDDS-776/HDDS-972 changed when the om login code.
Now the OM#init() may invoke SCM#getScmInfo() without a login, which failed
with the following. This ticket is opened to fix it.
{code}
ozoneManager_1 | java.io.IOException: DestHost:destPort scm:9863 ,
LocalHost:localPort om/172.19.0.4:0. Failed on local exception:
java.io.IOException: org.apache.hadoop.security.AccessControlException: Client
cannot authenticate via:[KERBEROS]
ozoneManager_1 | at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
ozoneManager_1 | at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
ozoneManager_1 | at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
ozoneManager_1 | at
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
ozoneManager_1 | at
org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
ozoneManager_1 | at
org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:806)
ozoneManager_1 | at
org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1515)
ozoneManager_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1457)
ozoneManager_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1367)
ozoneManager_1 | at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
ozoneManager_1 | at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
ozoneManager_1 | at com.sun.proxy.$Proxy28.getScmInfo(Unknown Source)
ozoneManager_1 | at
org.apache.hadoop.hdds.scm.protocolPB.ScmBlockLocationProtocolClientSideTranslatorPB.getScmInfo(ScmBlockLocationProtocolClientSideTranslatorPB.java:154)
ozoneManager_1 | at
org.apache.hadoop.ozone.om.OzoneManager.lambda$getScmInfo$1(OzoneManager.java:910)
ozoneManager_1 | at
org.apache.hadoop.utils.RetriableTask.call(RetriableTask.java:56)
ozoneManager_1 | at
org.apache.hadoop.ozone.om.OzoneManager.getScmInfo(OzoneManager.java:911)
ozoneManager_1 | at
org.apache.hadoop.ozone.om.OzoneManager.omInit(OzoneManager.java:873)
ozoneManager_1 | at
org.apache.hadoop.ozone.om.OzoneManager.createOm(OzoneManager.java:842)
ozoneManager_1 | at
org.apache.hadoop.ozone.om.OzoneManager.main(OzoneManager.java:771)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1107) Fix findbugs issues in DefaultCertificateClient#handleCase
[
https://issues.apache.org/jira/browse/HDDS-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao resolved HDDS-1107.
--
Resolution: Fixed
> Fix findbugs issues in DefaultCertificateClient#handleCase
> --
>
> Key: HDDS-1107
> URL: https://issues.apache.org/jira/browse/HDDS-1107
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Xiaoyu Yao
>Assignee: Ajay Kumar
>Priority: Major
>
> {code}
> FindBugs :
>
> module:hadoop-hdds/common
> Incompatible bit masks in (e & 0x2 == 0x1) yields a constant result in
> org.apache.hadoop.hdds.security.x509.certificate.client.DefaultCertificateClient.handleCase(DefaultCertificateClient$InitCase)
> At DefaultCertificateClient.java:& 0x2 == 0x1) yields a constant result in
> org.apache.hadoop.hdds.security.x509.certificate.client.DefaultCertificateClient.handleCase(DefaultCertificateClient$InitCase)
> At DefaultCertificateClient.java:[line 529]
> Incompatible bit masks in (e & 0x4 == 0x1) yields a constant result in
> org.apache.hadoop.hdds.security.x509.certificate.client.DefaultCertificateClient.handleCase(DefaultCertificateClient$InitCase)
> At DefaultCertificateClient.java:& 0x4 == 0x1) yields a constant result in
> org.apache.hadoop.hdds.security.x509.certificate.client.DefaultCertificateClient.handleCase(DefaultCertificateClient$InitCase)
> At DefaultCertificateClient.java:[line 529]
> Found reliance on default encoding in
> org.apache.hadoop.hdds.security.x509.certificate.client.DefaultCertificateClient.validateKeyPair(PublicKey):in
>
> org.apache.hadoop.hdds.security.x509.certificate.client.DefaultCertificateClient.validateKeyPair(PublicKey):
> String.getBytes() At DefaultCertificateClient.java:[line 587]
> Incompatible bit masks in (e & 0x2 == 0x1) yields a constant result in
> org.apache.hadoop.hdds.security.x509.certificate.client.OMCertificateClient.handleCase(DefaultCertificateClient$InitCase)
> At OMCertificateClient.java:& 0x2 == 0x1) yields a constant result in
> org.apache.hadoop.hdds.security.x509.certificate.client.OMCertificateClient.handleCase(DefaultCertificateClient$InitCase)
> At OMCertificateClient.java:[line 95]
> Incompatible bit masks in (e & 0x4 == 0x1) yields a constant result in
> org.apache.hadoop.hdds.security.x509.certificate.client.OMCertificateClient.handleCase(DefaultCertificateClient$InitCase)
> At OMCertificateClient.java:& 0x4 == 0x1) yields a constant result in
> org.apache.hadoop.hdds.security.x509.certificate.client.OMCertificateClient.handleCase(DefaultCertificateClient$InitCase)
> At OMCertificateClient.java:[line 95]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1107) Fix findbugs issues in DefaultCertificateClient#handleCase
Xiaoyu Yao created HDDS-1107:
Summary: Fix findbugs issues in DefaultCertificateClient#handleCase
Key: HDDS-1107
URL: https://issues.apache.org/jira/browse/HDDS-1107
Project: Hadoop Distributed Data Store
Issue Type: Sub-task
Reporter: Xiaoyu Yao
Assignee: Ajay Kumar
{code}
FindBugs :
module:hadoop-hdds/common
Incompatible bit masks in (e & 0x2 == 0x1) yields a constant result in
org.apache.hadoop.hdds.security.x509.certificate.client.DefaultCertificateClient.handleCase(DefaultCertificateClient$InitCase)
At DefaultCertificateClient.java:& 0x2 == 0x1) yields a constant result in
org.apache.hadoop.hdds.security.x509.certificate.client.DefaultCertificateClient.handleCase(DefaultCertificateClient$InitCase)
At DefaultCertificateClient.java:[line 529]
Incompatible bit masks in (e & 0x4 == 0x1) yields a constant result in
org.apache.hadoop.hdds.security.x509.certificate.client.DefaultCertificateClient.handleCase(DefaultCertificateClient$InitCase)
At DefaultCertificateClient.java:& 0x4 == 0x1) yields a constant result in
org.apache.hadoop.hdds.security.x509.certificate.client.DefaultCertificateClient.handleCase(DefaultCertificateClient$InitCase)
At DefaultCertificateClient.java:[line 529]
Found reliance on default encoding in
org.apache.hadoop.hdds.security.x509.certificate.client.DefaultCertificateClient.validateKeyPair(PublicKey):in
org.apache.hadoop.hdds.security.x509.certificate.client.DefaultCertificateClient.validateKeyPair(PublicKey):
String.getBytes() At DefaultCertificateClient.java:[line 587]
Incompatible bit masks in (e & 0x2 == 0x1) yields a constant result in
org.apache.hadoop.hdds.security.x509.certificate.client.OMCertificateClient.handleCase(DefaultCertificateClient$InitCase)
At OMCertificateClient.java:& 0x2 == 0x1) yields a constant result in
org.apache.hadoop.hdds.security.x509.certificate.client.OMCertificateClient.handleCase(DefaultCertificateClient$InitCase)
At OMCertificateClient.java:[line 95]
Incompatible bit masks in (e & 0x4 == 0x1) yields a constant result in
org.apache.hadoop.hdds.security.x509.certificate.client.OMCertificateClient.handleCase(DefaultCertificateClient$InitCase)
At OMCertificateClient.java:& 0x4 == 0x1) yields a constant result in
org.apache.hadoop.hdds.security.x509.certificate.client.OMCertificateClient.handleCase(DefaultCertificateClient$InitCase)
At OMCertificateClient.java:[line 95]
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1063) Implement OM init in secure cluster
[ https://issues.apache.org/jira/browse/HDDS-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao resolved HDDS-1063. -- Resolution: Duplicate > Implement OM init in secure cluster > --- > > Key: HDDS-1063 > URL: https://issues.apache.org/jira/browse/HDDS-1063 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > > Implement OM init in secure cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-581) Bootstrap DN with private/public key pair
[ https://issues.apache.org/jira/browse/HDDS-581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao resolved HDDS-581. - Resolution: Duplicate > Bootstrap DN with private/public key pair > - > > Key: HDDS-581 > URL: https://issues.apache.org/jira/browse/HDDS-581 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Ajay Kumar >Priority: Major > Attachments: HDDS-581-HDDS-4.00.patch > > > This will create public/private key pair for HDDS datanode if there isn't one > available during secure dn startup. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
