[GitHub] [hadoop-ozone] cxorm commented on pull request #1296: HDDS-4053. Volume space: add quotaUsageInBytes and update it when write and delete key.
cxorm commented on pull request #1296: URL: https://github.com/apache/hadoop-ozone/pull/1296#issuecomment-687539338 Hi @captainzmc , Could you be so kind to rebase this PR ? I would take a look on this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] smengcl edited a comment on pull request #1363: HDDS-3805. [OFS] Remove usage of OzoneClientAdapter interface
smengcl edited a comment on pull request #1363:
URL: https://github.com/apache/hadoop-ozone/pull/1363#issuecomment-687489498
> Thanks for the separation @smengcl, I think it's easier to discuss.
>
> I am not sure it's the same patch what I already commented or not. I had a
comment in the previous PR, where the discussion is stopped:
>
> > > I'm in favor of A. I'll attempt to remove the usage of
OzoneClientAdapter in OFS altogether then.
>
> > I am fine with that approach but let me add some comments to the latest
patch.
>
> > The naming of BasicRootedOzoneFileSystem and
BasicRootedOzoneFileSystemImpl is misleading. Usually the Impl postfix is used
when the class implemented a well known interface. There is no such interface
here. (It's more like the delegation design pattern not an implementation)
>
> > As a test: Can you please explain what are the differences between the
two classes and the responsibilities?
>
> > If not, we don't need two classes. Just remove the Impl and remove the
dedicated methods and directly call the proxy from the original methods of
BasicRootedOzoneFileSystem.
>
> > Wouldn't it be more simple?
>
> This patch seems to use `BasicRootedOzoneClientAdapterImpl` which is not
an `Impl` (not an interface). Do you need a client adapter here (in the old
code we need an interface and an implementation for classpath separation but
here this separation is removed.)? If yes, do you need `Impl` in the name (it
doesn't implement anything)?
I'm renaming `BasicRootedOzoneClientAdapterImpl` to
`BasicRootedOzoneFileSystemHelper`. This way the name should make more sense?
But one uncertainly is that there is one class extending
`BasicRootedOzoneClientAdapterImpl`:
```java
public class RootedOzoneClientAdapterImpl
extends BasicRootedOzoneClientAdapterImpl {
```
The class is dealing with `OzoneFSStorageStatistics`, but not used anywhere.
The o3fs counterpart is `OzoneClientAdapterImpl`, which is used in
`OzoneFileSystem`.
I'm not sure of the implication of the renaming.
What do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] smengcl edited a comment on pull request #1363: HDDS-3805. [OFS] Remove usage of OzoneClientAdapter interface
smengcl edited a comment on pull request #1363:
URL: https://github.com/apache/hadoop-ozone/pull/1363#issuecomment-687489498
> Thanks for the separation @smengcl, I think it's easier to discuss.
>
> I am not sure it's the same patch what I already commented or not. I had a
comment in the previous PR, where the discussion is stopped:
>
> > > I'm in favor of A. I'll attempt to remove the usage of
OzoneClientAdapter in OFS altogether then.
>
> > I am fine with that approach but let me add some comments to the latest
patch.
>
> > The naming of BasicRootedOzoneFileSystem and
BasicRootedOzoneFileSystemImpl is misleading. Usually the Impl postfix is used
when the class implemented a well known interface. There is no such interface
here. (It's more like the delegation design pattern not an implementation)
>
> > As a test: Can you please explain what are the differences between the
two classes and the responsibilities?
>
> > If not, we don't need two classes. Just remove the Impl and remove the
dedicated methods and directly call the proxy from the original methods of
BasicRootedOzoneFileSystem.
>
> > Wouldn't it be more simple?
>
> This patch seems to use `BasicRootedOzoneClientAdapterImpl` which is not
an `Impl` (not an interface). Do you need a client adapter here (in the old
code we need an interface and an implementation for classpath separation but
here this separation is removed.)? If yes, do you need `Impl` in the name (it
doesn't implement anything)?
I'm thinking of renaming `BasicRootedOzoneClientAdapterImpl` to
`BasicRootedOzoneFileSystemHelper`. This way the class name should make more
sense?
But one uncertainly is that there is one class extending
`BasicRootedOzoneClientAdapterImpl`:
```java
public class RootedOzoneClientAdapterImpl
extends BasicRootedOzoneClientAdapterImpl {
```
The class is dealing with `OzoneFSStorageStatistics`, but not used anywhere.
The o3fs counterpart is `OzoneClientAdapterImpl`, which is used in
`OzoneFileSystem`.
I'm not sure of the implication of the renaming. What do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] smengcl edited a comment on pull request #1363: HDDS-3805. [OFS] Remove usage of OzoneClientAdapter interface
smengcl edited a comment on pull request #1363:
URL: https://github.com/apache/hadoop-ozone/pull/1363#issuecomment-687489498
> Thanks for the separation @smengcl, I think it's easier to discuss.
>
> I am not sure it's the same patch what I already commented or not. I had a
comment in the previous PR, where the discussion is stopped:
>
> > > I'm in favor of A. I'll attempt to remove the usage of
OzoneClientAdapter in OFS altogether then.
>
> > I am fine with that approach but let me add some comments to the latest
patch.
>
> > The naming of BasicRootedOzoneFileSystem and
BasicRootedOzoneFileSystemImpl is misleading. Usually the Impl postfix is used
when the class implemented a well known interface. There is no such interface
here. (It's more like the delegation design pattern not an implementation)
>
> > As a test: Can you please explain what are the differences between the
two classes and the responsibilities?
>
> > If not, we don't need two classes. Just remove the Impl and remove the
dedicated methods and directly call the proxy from the original methods of
BasicRootedOzoneFileSystem.
>
> > Wouldn't it be more simple?
>
> This patch seems to use `BasicRootedOzoneClientAdapterImpl` which is not
an `Impl` (not an interface). Do you need a client adapter here (in the old
code we need an interface and an implementation for classpath separation but
here this separation is removed.)? If yes, do you need `Impl` in the name (it
doesn't implement anything)?
I'm renaming `BasicRootedOzoneClientAdapterImpl` to
`BasicRootedOzoneFileSystemHelper`. This way the name should make more sense?
But one uncertainly is that there is one class extending
`BasicRootedOzoneClientAdapterImpl`:
```java
public class RootedOzoneClientAdapterImpl
extends BasicRootedOzoneClientAdapterImpl {
```
The class is dealing with `OzoneFSStorageStatistics`, but not used anywhere.
The o3fs counterpart is `OzoneClientAdapterImpl`, which is used in
`OzoneFileSystem`.
I am not sure how
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] smengcl commented on pull request #1363: HDDS-3805. [OFS] Remove usage of OzoneClientAdapter interface
smengcl commented on pull request #1363: URL: https://github.com/apache/hadoop-ozone/pull/1363#issuecomment-687489498 > Thanks for the separation @smengcl, I think it's easier to discuss. > > I am not sure it's the same patch what I already commented or not. I had a comment in the previous PR, where the discussion is stopped: > > > > I'm in favor of A. I'll attempt to remove the usage of OzoneClientAdapter in OFS altogether then. > > > I am fine with that approach but let me add some comments to the latest patch. > > > The naming of BasicRootedOzoneFileSystem and BasicRootedOzoneFileSystemImpl is misleading. Usually the Impl postfix is used when the class implemented a well known interface. There is no such interface here. (It's more like the delegation design pattern not an implementation) > > > As a test: Can you please explain what are the differences between the two classes and the responsibilities? > > > If not, we don't need two classes. Just remove the Impl and remove the dedicated methods and directly call the proxy from the original methods of BasicRootedOzoneFileSystem. > > > Wouldn't it be more simple? > > This patch seems to use `BasicRootedOzoneClientAdapterImpl` which is not an `Impl` (not an interface). Do you need a client adapter here (in the old code we need an interface and an implementation for classpath separation but here this separation is removed.)? If yes, do you need `Impl` in the name (it doesn't implement anything)? I'm renaming `BasicRootedOzoneClientAdapterImpl` to `BasicRootedOzoneFileSystemHelper`. This way the name should make more sense? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on pull request #1399: HDDS-3684. Add tests for replication annotation
amaliujia commented on pull request #1399: URL: https://github.com/apache/hadoop-ozone/pull/1399#issuecomment-687463641 R: @timmylicheng This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] fapifta opened a new pull request #1400: HDDS-4141. Implement Finalize command in Ozone Manager client.
fapifta opened a new pull request #1400: URL: https://github.com/apache/hadoop-ozone/pull/1400 ## What changes were proposed in this pull request? As part of the upgrade development, this addition is aiming to create the client side implementation of the finalizeUpgrade initiation and monitoring parts. In order to test it the PR has some classes and some modifications that are related to the server side development, but are added here, to be able to test out the command implemented in FinalizeUpgradeSubCommand class. The PR also contains the proto message additions for the two newly implemented requests, finalieUpgrade and finalizeUpgreadeProgress request and response pair, and a status message sent between the client and server with these requests. The idea is the following: only one client can initialize and monitor the finalization, so that we have a client ID with the requests, but in case of a client side failure the command allows a takeover from the original initiator to finish the monitoring. The finalization is a write request, that potentially affects all table caches in OM, so CleanupTableInfo annotation was extended with a cleanupAll option. And the tests were added to accommodate that. OmMetaDataManager was extended with an ALL_TABLES psf field, to have the full table list at the same place. I was exploring the idea of using OmDbDefinition#getColumnFamilies here, but in order to use that we need to change the annotation in all the response classes to use DBColumnFamilyDefinition instead of String in cleanupTables field, as in the annotation it is impossible to point to a definition's getTableName method, and refactoring that should be done in a separate task, it affects too many files and code paths to include it here. Also during toying with the idea it turned out that OmDbDefinition#getColumnFamilies returns S3_TABLE that is not there anymore in OM as I found, so probably we should remove it. The server side handling is purely just toying with some ideas, and in a state where I got to a few problems/questions. At that point we have decided that the minimum viable solution is something like this: - finalizeUpgrade will post one request to Ratis, and all OMs will start finalization and finish finalization as part of that request. - monitoring will be supplied status from the request handling in the Leader and will emit finalization done when OM reached majority processing the finalization request. - finalization itself will ensure to run the finalization of all the new LayoutFeature-s that are enabled and applied, and will update the MLV in all 3 OMs A proposal that describes problems stemming from this approach and with a few possibilities to make this more robust will be added in a follow up JIRA later on as we finished the minimalistic first version, as ideas explore messaging between OMs outside Ratis, posting finalize requests for LayoutFeatures in a lazy manner one by one to ratis, just to mention a few from a couple possible approaches, and this for sure requires a lot of time and exploration and a consensus on how deep we want to go down this rabbit hole. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-4141 ## How was this patch tested? In ha and non-ha docker environments possible scenarios to run the command was tested by hand. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4141) Implement Finalize command in Ozone Manager client.
[ https://issues.apache.org/jira/browse/HDDS-4141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-4141: - Labels: pull-request-available (was: ) > Implement Finalize command in Ozone Manager client. > --- > > Key: HDDS-4141 > URL: https://issues.apache.org/jira/browse/HDDS-4141 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: István Fajth >Priority: Major > Labels: pull-request-available > Fix For: 1.1.0 > > > * On the client side, add a new command to finalize OM through CLI. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] vivekratnavel commented on pull request #1390: HDDS-4196. Add an endpoint in Recon to query Prometheus
vivekratnavel commented on pull request #1390: URL: https://github.com/apache/hadoop-ozone/pull/1390#issuecomment-687397374 @avijayanhwx Thanks for the review! Please take another look. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] vivekratnavel commented on a change in pull request #1390: HDDS-4196. Add an endpoint in Recon to query Prometheus
vivekratnavel commented on a change in pull request #1390:
URL: https://github.com/apache/hadoop-ozone/pull/1390#discussion_r483849269
##
File path:
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/MetricsProxyEndpoint.java
##
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api;
+
+import org.apache.hadoop.ozone.recon.MetricsServiceProviderFactory;
+import org.apache.hadoop.ozone.recon.spi.MetricsServiceProvider;
+import org.apache.hadoop.ozone.recon.spi.impl.PrometheusServiceProviderImpl;
+
+import javax.inject.Inject;
+import javax.servlet.http.HttpServletResponse;
+import javax.ws.rs.DefaultValue;
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.Produces;
+import javax.ws.rs.core.Context;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import javax.ws.rs.core.UriInfo;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.HttpURLConnection;
+import java.nio.ByteBuffer;
+import java.nio.channels.Channels;
+import java.nio.channels.ReadableByteChannel;
+import java.nio.channels.WritableByteChannel;
+
+import static
org.apache.hadoop.ozone.recon.ReconConstants.PROMETHEUS_INSTANT_QUERY_API;
+
+/**
+ * Endpoint to fetch metrics data from Prometheus HTTP endpoint.
+ */
+@Path("/metrics")
+public class MetricsProxyEndpoint {
+
+ private MetricsServiceProvider metricsServiceProvider;
+
+ @Inject
+ public MetricsProxyEndpoint(
+ MetricsServiceProviderFactory metricsServiceProviderFactory) {
+this.metricsServiceProvider =
+metricsServiceProviderFactory.getMetricsServiceProvider();
+ }
+
+ /**
+ * Return a response from the configured metrics endpoint.
+ */
+ @GET
+ @Produces(MediaType.APPLICATION_JSON)
+ @Path("/{api}")
+ public void getMetricsResponse(
+ @DefaultValue (PROMETHEUS_INSTANT_QUERY_API) @PathParam("api") String
api,
+ @Context UriInfo uriInfo,
+ @Context HttpServletResponse httpServletResponse
+ ) throws Exception {
+if (metricsServiceProvider != null) {
+ HttpURLConnection connection = metricsServiceProvider.getMetricsResponse(
+ api, uriInfo.getRequestUri().getQuery());
+ InputStream inputStream;
+ if (Response.Status.fromStatusCode(connection.getResponseCode())
+ .getFamily() == Response.Status.Family.SUCCESSFUL) {
+inputStream = connection.getInputStream();
+ } else {
+// Throw a bad gateway error if HttpResponseCode is not 2xx
+httpServletResponse.setStatus(HttpServletResponse.SC_BAD_GATEWAY);
+inputStream = connection.getErrorStream();
+ }
+ try (
+ OutputStream outputStream =
+ httpServletResponse.getOutputStream();
+ ReadableByteChannel inputChannel =
+ Channels.newChannel(inputStream);
+ WritableByteChannel outputChannel =
+ Channels.newChannel(outputStream)
+) {
+final ByteBuffer buffer = ByteBuffer.allocateDirect(16 * 1024);
+
+while(inputChannel.read(buffer) != -1) {
+ buffer.flip();
+ outputChannel.write(buffer);
+ buffer.compact();
+}
+
+buffer.flip();
+
+while(buffer.hasRemaining()) {
+ outputChannel.write(buffer);
+}
+ } finally {
+inputStream.close();
+ }
+} else {
+ // Throw a Bad Gateway Error
+ httpServletResponse.sendError(HttpServletResponse.SC_BAD_GATEWAY,
+ "Metrics endpoint is not configured. Configure " +
+ PrometheusServiceProviderImpl.getEndpointConfigKey() + " and " +
Review comment:
@avijayanhwx I thought about keeping this method as part of the
interface. But, then to construct this error message, we will have to
instantiate `PrometheusServiceProviderImpl` to get endpoint config key. This is
because, `metricsServiceProviderFactory.getMetricsServiceProvider()` will
return null when no metrics service providers are configured.
This is an automated message from the Apache Git Service.
To respond to the
[jira] [Updated] (HDDS-3684) Add tests for replication annotation
[ https://issues.apache.org/jira/browse/HDDS-3684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-3684: - Labels: pull-request-available (was: ) > Add tests for replication annotation > > > Key: HDDS-3684 > URL: https://issues.apache.org/jira/browse/HDDS-3684 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Li Cheng >Assignee: Rui Wang >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] vivekratnavel commented on a change in pull request #1390: HDDS-4196. Add an endpoint in Recon to query Prometheus
vivekratnavel commented on a change in pull request #1390:
URL: https://github.com/apache/hadoop-ozone/pull/1390#discussion_r483849269
##
File path:
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/MetricsProxyEndpoint.java
##
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api;
+
+import org.apache.hadoop.ozone.recon.MetricsServiceProviderFactory;
+import org.apache.hadoop.ozone.recon.spi.MetricsServiceProvider;
+import org.apache.hadoop.ozone.recon.spi.impl.PrometheusServiceProviderImpl;
+
+import javax.inject.Inject;
+import javax.servlet.http.HttpServletResponse;
+import javax.ws.rs.DefaultValue;
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.Produces;
+import javax.ws.rs.core.Context;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import javax.ws.rs.core.UriInfo;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.HttpURLConnection;
+import java.nio.ByteBuffer;
+import java.nio.channels.Channels;
+import java.nio.channels.ReadableByteChannel;
+import java.nio.channels.WritableByteChannel;
+
+import static
org.apache.hadoop.ozone.recon.ReconConstants.PROMETHEUS_INSTANT_QUERY_API;
+
+/**
+ * Endpoint to fetch metrics data from Prometheus HTTP endpoint.
+ */
+@Path("/metrics")
+public class MetricsProxyEndpoint {
+
+ private MetricsServiceProvider metricsServiceProvider;
+
+ @Inject
+ public MetricsProxyEndpoint(
+ MetricsServiceProviderFactory metricsServiceProviderFactory) {
+this.metricsServiceProvider =
+metricsServiceProviderFactory.getMetricsServiceProvider();
+ }
+
+ /**
+ * Return a response from the configured metrics endpoint.
+ */
+ @GET
+ @Produces(MediaType.APPLICATION_JSON)
+ @Path("/{api}")
+ public void getMetricsResponse(
+ @DefaultValue (PROMETHEUS_INSTANT_QUERY_API) @PathParam("api") String
api,
+ @Context UriInfo uriInfo,
+ @Context HttpServletResponse httpServletResponse
+ ) throws Exception {
+if (metricsServiceProvider != null) {
+ HttpURLConnection connection = metricsServiceProvider.getMetricsResponse(
+ api, uriInfo.getRequestUri().getQuery());
+ InputStream inputStream;
+ if (Response.Status.fromStatusCode(connection.getResponseCode())
+ .getFamily() == Response.Status.Family.SUCCESSFUL) {
+inputStream = connection.getInputStream();
+ } else {
+// Throw a bad gateway error if HttpResponseCode is not 2xx
+httpServletResponse.setStatus(HttpServletResponse.SC_BAD_GATEWAY);
+inputStream = connection.getErrorStream();
+ }
+ try (
+ OutputStream outputStream =
+ httpServletResponse.getOutputStream();
+ ReadableByteChannel inputChannel =
+ Channels.newChannel(inputStream);
+ WritableByteChannel outputChannel =
+ Channels.newChannel(outputStream)
+) {
+final ByteBuffer buffer = ByteBuffer.allocateDirect(16 * 1024);
+
+while(inputChannel.read(buffer) != -1) {
+ buffer.flip();
+ outputChannel.write(buffer);
+ buffer.compact();
+}
+
+buffer.flip();
+
+while(buffer.hasRemaining()) {
+ outputChannel.write(buffer);
+}
+ } finally {
+inputStream.close();
+ }
+} else {
+ // Throw a Bad Gateway Error
+ httpServletResponse.sendError(HttpServletResponse.SC_BAD_GATEWAY,
+ "Metrics endpoint is not configured. Configure " +
+ PrometheusServiceProviderImpl.getEndpointConfigKey() + " and " +
Review comment:
@avijayanhwx I thought about keeping this method as part of the
interface. But, then to construct this error message, we will have to
instantiate `PrometheusServiceProviderImpl` to get endpoint config key. This is
because, `metricsServiceProviderFactory.getMetricsServiceProvider()` will
return null when no metrics service service providers are configured.
This is an automated message from the Apache Git Service.
To
[GitHub] [hadoop-ozone] amaliujia opened a new pull request #1399: HDDS-3684. Add tests for replication annotation
amaliujia opened a new pull request #1399: URL: https://github.com/apache/hadoop-ozone/pull/1399 ## What changes were proposed in this pull request? Add tests for replication annotation ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-3684 ## How was this patch tested? Unit Test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4210) ResolveBucket during checkAcls fails
[
https://issues.apache.org/jira/browse/HDDS-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-4210:
-
Target Version/s: 1.1.0
> ResolveBucket during checkAcls fails
>
>
> Key: HDDS-4210
> URL: https://issues.apache.org/jira/browse/HDDS-4210
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Affects Versions: 1.0.0
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Blocker
> Labels: pull-request-available
>
> In HA, in validateAndUpdateCache when resolveBucket, it checks the permission
> using checkAcls. But it will have not any RpcContext and it will fail with
> NPE in checkAcls when getting hostName.
> For this same reason, we added the required information to check ACLs into
> OMRequest.
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.ozone.om.OzoneManager.checkAcls(OzoneManager.java:1604)
> at
> org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3497)
> at
> org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3465)
> at
> org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3452)
> at
> org.apache.hadoop.ozone.om.request.key.OMKeyRequest.resolveBucketLink(OMKeyRequest.java:96)
> at
> org.apache.hadoop.ozone.om.request.key.OMKeyCreateRequest.validateAndUpdateCache(OMKeyCreateRequest.java:215)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:227)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:428)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$applyTransaction$1(OzoneManagerStateMachine.java:246)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4210) ResolveBucket during checkAcls fails
[
https://issues.apache.org/jira/browse/HDDS-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-4210:
-
Labels: pull-request-available (was: )
> ResolveBucket during checkAcls fails
>
>
> Key: HDDS-4210
> URL: https://issues.apache.org/jira/browse/HDDS-4210
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Affects Versions: 1.0.0
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Blocker
> Labels: pull-request-available
>
> In HA, in validateAndUpdateCache when resolveBucket, it checks the permission
> using checkAcls. But it will have not any RpcContext and it will fail with
> NPE in checkAcls when getting hostName.
> For this same reason, we added the required information to check ACLs into
> OMRequest.
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.ozone.om.OzoneManager.checkAcls(OzoneManager.java:1604)
> at
> org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3497)
> at
> org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3465)
> at
> org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3452)
> at
> org.apache.hadoop.ozone.om.request.key.OMKeyRequest.resolveBucketLink(OMKeyRequest.java:96)
> at
> org.apache.hadoop.ozone.om.request.key.OMKeyCreateRequest.validateAndUpdateCache(OMKeyCreateRequest.java:215)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:227)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:428)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$applyTransaction$1(OzoneManagerStateMachine.java:246)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 opened a new pull request #1398: HDDS-4210. ResolveBucket during checkAcls fails.
bharatviswa504 opened a new pull request #1398: URL: https://github.com/apache/hadoop-ozone/pull/1398 ## What changes were proposed in this pull request? When bucket links are levels, When ACLS are enabled, during `checkAcls` It will fail with NPE. Finally in HA, this causes OM to be terminated. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-4210 ## How was this patch tested? Added UT. The same test will fail without the fix. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4155) Directory and filename can end up with same name in a path
[
https://issues.apache.org/jira/browse/HDDS-4155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190913#comment-17190913
]
Arpit Agarwal commented on HDDS-4155:
-
{quote}It seems to be very strange for me as this is something which should be
avoid with posix fs, but if hadoop can support it...
{quote}
Marton I am really surprised that you are asking this as a serious question.
S3A does not _"support it"._ It is not a cool exciting feature in S3A. It is an
unfixable bug because S3A cannot control what goes on at the backend which is
owned by Amazon and they are forced to display both, if they exist. We need not
emulate S3A bugs.
> Directory and filename can end up with same name in a path
> --
>
> Key: HDDS-4155
> URL: https://issues.apache.org/jira/browse/HDDS-4155
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
>
> Scenario:
> Create Key via S3, and Create Directory through Fs.
> # open key -> /a/b/c
> # CreateDirectory -> /a/b/c
> # CommitKey -> /a/b/c
> So, now in Ozone we will have directory and file with name "c"
> When created through Fs interface.
> # create file -> /a/b/c
> # CreateDirectory -> /a/b/c
> # CommitKey -> /a/b/c
> So, now in Ozone we will have directory and file with name "c"
>
> # InitiateMPU /a/b/c
> # Create Part1 /a/b/c
> # Commit Part1 /a/b/c
> # Create Directory /a/b/c
> # Complete MPU /a/b/c
> So, now in Ozone, we will have directory and file with name "c". In MPU this
> is one example scenario.
>
> Few proposals/ideas to solve this:
> # Check during commit whether a directory already exists with same name. But
> disadvantage is after user uploads the entire data during last stage we fail.
> (File system with create in progress acts similarly. Scenario: 1. vi t1 2.
> mkdir t1 3. Save t1: (Fail:"t1" is a directory)
> # During create directory check are there any open key creation with same
> name and fail.
>
> Any of the above approaches are not final, this Jira is opened to discuss
> this issue and come up with solution.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4211) [OFS] Better owner and group display for listing Ozone volumes and buckets
[ https://issues.apache.org/jira/browse/HDDS-4211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDDS-4211: - Target Version/s: 1.1.0 > [OFS] Better owner and group display for listing Ozone volumes and buckets > -- > > Key: HDDS-4211 > URL: https://issues.apache.org/jira/browse/HDDS-4211 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available > > Improve volumes' and buckets' owner and group display when listing in OFS. > 1. Display short name instead of full Kerberos principal. > 2. For volumes, get actual group of the owner (currently it is the volume > admin name which is incorrect) > 3. For buckets, display the owner and group of its parent volume. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4211) [OFS] Better owner and group display for listing Ozone volumes and buckets
[ https://issues.apache.org/jira/browse/HDDS-4211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDDS-4211: - Status: Patch Available (was: Open) > [OFS] Better owner and group display for listing Ozone volumes and buckets > -- > > Key: HDDS-4211 > URL: https://issues.apache.org/jira/browse/HDDS-4211 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available > > Improve volumes' and buckets' owner and group display when listing in OFS. > 1. Display short name instead of full Kerberos principal. > 2. For volumes, get actual group of the owner (currently it is the volume > admin name which is incorrect) > 3. For buckets, display the owner and group of its parent volume. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] smengcl opened a new pull request #1397: HDDS-4211. [OFS] Better owner and group display for listing Ozone volumes and buckets
smengcl opened a new pull request #1397: URL: https://github.com/apache/hadoop-ozone/pull/1397 ## What changes were proposed in this pull request? Improve volumes' and buckets' owner and group display when listing root and in volumes in OFS. 1. Display short name instead of full Kerberos principal. 2. For volumes, get actual group of the owner (currently it is the volume admin name which is incorrect) 3. For buckets, display the owner and group of its parent volume. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-4211 ## How was this patch tested? Will add new a test case. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4211) [OFS] Better owner and group display for listing Ozone volumes and buckets
[ https://issues.apache.org/jira/browse/HDDS-4211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-4211: - Labels: pull-request-available (was: ) > [OFS] Better owner and group display for listing Ozone volumes and buckets > -- > > Key: HDDS-4211 > URL: https://issues.apache.org/jira/browse/HDDS-4211 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available > > Improve volumes' and buckets' owner and group display when listing in OFS. > 1. Display short name instead of full Kerberos principal. > 2. For volumes, get actual group of the owner (currently it is the volume > admin name which is incorrect) > 3. For buckets, display the owner and group of its parent volume. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4211) [OFS] Better owner and group display for listing Ozone volumes and buckets
[ https://issues.apache.org/jira/browse/HDDS-4211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDDS-4211: - Summary: [OFS] Better owner and group display for listing Ozone volumes and buckets (was: [OFS] Better owner and group name display for Ozone volumes and buckets) > [OFS] Better owner and group display for listing Ozone volumes and buckets > -- > > Key: HDDS-4211 > URL: https://issues.apache.org/jira/browse/HDDS-4211 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > > Improve volumes' and buckets' owner and group display when listing in OFS. > 1. Display short name instead of full Kerberos principal. > 2. For volumes, get actual group of the owner (currently it is the volume > admin name which is incorrect) > 3. For buckets, display the owner and group of its parent volume. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4211) [OFS] Better owner and group name display for Ozone volumes and buckets
[ https://issues.apache.org/jira/browse/HDDS-4211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDDS-4211: - Description: Improve volumes' and buckets' owner and group display when listing in OFS. 1. Display short name instead of full Kerberos principal. 2. For volumes, get actual group of the owner (currently it is the volume admin name which is incorrect) 3. For buckets, display the owner and group of its parent volume. > [OFS] Better owner and group name display for Ozone volumes and buckets > --- > > Key: HDDS-4211 > URL: https://issues.apache.org/jira/browse/HDDS-4211 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > > Improve volumes' and buckets' owner and group display when listing in OFS. > 1. Display short name instead of full Kerberos principal. > 2. For volumes, get actual group of the owner (currently it is the volume > admin name which is incorrect) > 3. For buckets, display the owner and group of its parent volume. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4211) [OFS] Better owner and group name display for Ozone volumes and buckets
Siyao Meng created HDDS-4211: Summary: [OFS] Better owner and group name display for Ozone volumes and buckets Key: HDDS-4211 URL: https://issues.apache.org/jira/browse/HDDS-4211 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Siyao Meng Assignee: Siyao Meng -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] vivekratnavel opened a new pull request #1396: HDDS-4150. recon.api.TestEndpoints test is flaky
vivekratnavel opened a new pull request #1396: URL: https://github.com/apache/hadoop-ozone/pull/1396 ## What changes were proposed in this pull request? - Fix flakiness in the test ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-4150 ## How was this patch tested? Tested by running the test several times locally. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4150) recon.api.TestEndpoints is flaky
[ https://issues.apache.org/jira/browse/HDDS-4150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-4150: - Labels: pull-request-available (was: ) > recon.api.TestEndpoints is flaky > > > Key: HDDS-4150 > URL: https://issues.apache.org/jira/browse/HDDS-4150 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Marton Elek >Assignee: Vivek Ratnavel Subramanian >Priority: Blocker > Labels: pull-request-available > > Failed on the PR: > https://github.com/apache/hadoop-ozone/pull/1349 > And on the master: > https://github.com/elek/ozone-build-results/blob/master/2020/08/25/2533/unit/hadoop-ozone/recon/org.apache.hadoop.ozone.recon.api.TestEndpoints.txt > and here: > https://github.com/elek/ozone-build-results/blob/master/2020/08/22/2499/unit/hadoop-ozone/recon/org.apache.hadoop.ozone.recon.api.TestEndpoints.txt -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4088) Adding Owner info for Authorizer plugin to honor owner access rights
[ https://issues.apache.org/jira/browse/HDDS-4088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-4088: - Labels: pull-request-available (was: ) > Adding Owner info for Authorizer plugin to honor owner access rights > > > Key: HDDS-4088 > URL: https://issues.apache.org/jira/browse/HDDS-4088 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Labels: pull-request-available > > Currently external authorizer does not have the owner info of the volume > bucket keys when authorizing requests. Explicit rules/policies must be set > before volume/bucket/key creation is allowed even for owner themselves. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao opened a new pull request #1395: HDDS-4088. Adding Owner info for Authorizer plugin to honor owner acc…
xiaoyuyao opened a new pull request #1395: URL: https://github.com/apache/hadoop-ozone/pull/1395 …ess rights. ## What changes were proposed in this pull request? adding isowner flag and volume owner info to acl check request context so that Ranger and 3rd party plugin can allow volume owner has full control over the volume ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-4088 ## How was this patch tested? UT added. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4210) ResolveBucket during checkAcls fails
[
https://issues.apache.org/jira/browse/HDDS-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-4210:
-
Description:
In HA, in validateAndUpdateCache when resolveBucket, it checks the permission
using checkAcls. But it will have not any RpcContext and it will fail with NPE
in checkAcls when getting hostName.
For this same reason, we added the required information to check ACLs into
OMRequest.
{code:java}
java.lang.NullPointerException
at
org.apache.hadoop.ozone.om.OzoneManager.checkAcls(OzoneManager.java:1604)
at
org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3497)
at
org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3465)
at
org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3452)
at
org.apache.hadoop.ozone.om.request.key.OMKeyRequest.resolveBucketLink(OMKeyRequest.java:96)
at
org.apache.hadoop.ozone.om.request.key.OMKeyCreateRequest.validateAndUpdateCache(OMKeyCreateRequest.java:215)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:227)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:428)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$applyTransaction$1(OzoneManagerStateMachine.java:246)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
was:
In HA, in validateAndUpdateCache when resolveBucket, it checks the permission
using checkAcls. But it will have not any RpcContext and it will fail with NPE
in checkAcls when getting hostName.
{code:java}
java.lang.NullPointerException
at
org.apache.hadoop.ozone.om.OzoneManager.checkAcls(OzoneManager.java:1604)
at
org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3497)
at
org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3465)
at
org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3452)
at
org.apache.hadoop.ozone.om.request.key.OMKeyRequest.resolveBucketLink(OMKeyRequest.java:96)
at
org.apache.hadoop.ozone.om.request.key.OMKeyCreateRequest.validateAndUpdateCache(OMKeyCreateRequest.java:215)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:227)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:428)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$applyTransaction$1(OzoneManagerStateMachine.java:246)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
> ResolveBucket during checkAcls fails
>
>
> Key: HDDS-4210
> URL: https://issues.apache.org/jira/browse/HDDS-4210
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Affects Versions: 1.0.0
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Blocker
>
> In HA, in validateAndUpdateCache when resolveBucket, it checks the permission
> using checkAcls. But it will have not any RpcContext and it will fail with
> NPE in checkAcls when getting hostName.
> For this same reason, we added the required information to check ACLs into
> OMRequest.
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.ozone.om.OzoneManager.checkAcls(OzoneManager.java:1604)
> at
> org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3497)
> at
> org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3465)
> at
> org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3452)
> at
> org.apache.hadoop.ozone.om.request.key.OMKeyRequest.resolveBucketLink(OMKeyRequest.java:96)
> at
> org.apache.hadoop.ozone.om.request.key.OMKeyCreateRequest.validateAndUpdateCache(OMKeyCreateRequest.java:215)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:227)
> at
>
[jira] [Updated] (HDDS-4210) ResolveBucket during checkAcls fails
[
https://issues.apache.org/jira/browse/HDDS-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-4210:
-
Labels: (was: OzoneManager)
> ResolveBucket during checkAcls fails
>
>
> Key: HDDS-4210
> URL: https://issues.apache.org/jira/browse/HDDS-4210
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Blocker
>
> In HA, in validateAndUpdateCache when resolveBucket, it checks the permission
> using checkAcls. But it will have not any RpcContext and it will fail with
> NPE in checkAcls when getting hostName.
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.ozone.om.OzoneManager.checkAcls(OzoneManager.java:1604)
> at
> org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3497)
> at
> org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3465)
> at
> org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3452)
> at
> org.apache.hadoop.ozone.om.request.key.OMKeyRequest.resolveBucketLink(OMKeyRequest.java:96)
> at
> org.apache.hadoop.ozone.om.request.key.OMKeyCreateRequest.validateAndUpdateCache(OMKeyCreateRequest.java:215)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:227)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:428)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$applyTransaction$1(OzoneManagerStateMachine.java:246)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4210) ResolveBucket during checkAcls fails
[
https://issues.apache.org/jira/browse/HDDS-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-4210:
-
Labels: OzoneManager (was: )
> ResolveBucket during checkAcls fails
>
>
> Key: HDDS-4210
> URL: https://issues.apache.org/jira/browse/HDDS-4210
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Blocker
> Labels: OzoneManager
>
> In HA, in validateAndUpdateCache when resolveBucket, it checks the permission
> using checkAcls. But it will have not any RpcContext and it will fail with
> NPE in checkAcls when getting hostName.
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.ozone.om.OzoneManager.checkAcls(OzoneManager.java:1604)
> at
> org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3497)
> at
> org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3465)
> at
> org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3452)
> at
> org.apache.hadoop.ozone.om.request.key.OMKeyRequest.resolveBucketLink(OMKeyRequest.java:96)
> at
> org.apache.hadoop.ozone.om.request.key.OMKeyCreateRequest.validateAndUpdateCache(OMKeyCreateRequest.java:215)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:227)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:428)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$applyTransaction$1(OzoneManagerStateMachine.java:246)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4210) ResolveBucket during checkAcls fails
[
https://issues.apache.org/jira/browse/HDDS-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-4210:
-
Component/s: Ozone Manager
> ResolveBucket during checkAcls fails
>
>
> Key: HDDS-4210
> URL: https://issues.apache.org/jira/browse/HDDS-4210
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Affects Versions: 1.0.0
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Blocker
>
> In HA, in validateAndUpdateCache when resolveBucket, it checks the permission
> using checkAcls. But it will have not any RpcContext and it will fail with
> NPE in checkAcls when getting hostName.
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.ozone.om.OzoneManager.checkAcls(OzoneManager.java:1604)
> at
> org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3497)
> at
> org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3465)
> at
> org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3452)
> at
> org.apache.hadoop.ozone.om.request.key.OMKeyRequest.resolveBucketLink(OMKeyRequest.java:96)
> at
> org.apache.hadoop.ozone.om.request.key.OMKeyCreateRequest.validateAndUpdateCache(OMKeyCreateRequest.java:215)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:227)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:428)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$applyTransaction$1(OzoneManagerStateMachine.java:246)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4210) ResolveBucket during checkAcls fails
[
https://issues.apache.org/jira/browse/HDDS-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-4210:
-
Priority: Blocker (was: Major)
> ResolveBucket during checkAcls fails
>
>
> Key: HDDS-4210
> URL: https://issues.apache.org/jira/browse/HDDS-4210
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Blocker
>
> In HA, in validateAndUpdateCache when resolveBucket, it checks the permission
> using checkAcls. But it will have not any RpcContext and it will fail with
> NPE in checkAcls when getting hostName.
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.ozone.om.OzoneManager.checkAcls(OzoneManager.java:1604)
> at
> org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3497)
> at
> org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3465)
> at
> org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3452)
> at
> org.apache.hadoop.ozone.om.request.key.OMKeyRequest.resolveBucketLink(OMKeyRequest.java:96)
> at
> org.apache.hadoop.ozone.om.request.key.OMKeyCreateRequest.validateAndUpdateCache(OMKeyCreateRequest.java:215)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:227)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:428)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$applyTransaction$1(OzoneManagerStateMachine.java:246)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4210) ResolveBucket during checkAcls fails
Bharat Viswanadham created HDDS-4210:
Summary: ResolveBucket during checkAcls fails
Key: HDDS-4210
URL: https://issues.apache.org/jira/browse/HDDS-4210
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Bharat Viswanadham
Assignee: Bharat Viswanadham
In HA, in validateAndUpdateCache when resolveBucket, it checks the permission
using checkAcls. But it will have not any RpcContext and it will fail with NPE
in checkAcls when getting hostName.
{code:java}
java.lang.NullPointerException
at
org.apache.hadoop.ozone.om.OzoneManager.checkAcls(OzoneManager.java:1604)
at
org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3497)
at
org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3465)
at
org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3452)
at
org.apache.hadoop.ozone.om.request.key.OMKeyRequest.resolveBucketLink(OMKeyRequest.java:96)
at
org.apache.hadoop.ozone.om.request.key.OMKeyCreateRequest.validateAndUpdateCache(OMKeyCreateRequest.java:215)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:227)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:428)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$applyTransaction$1(OzoneManagerStateMachine.java:246)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4210) ResolveBucket during checkAcls fails
[
https://issues.apache.org/jira/browse/HDDS-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-4210:
-
Affects Version/s: 1.0.0
> ResolveBucket during checkAcls fails
>
>
> Key: HDDS-4210
> URL: https://issues.apache.org/jira/browse/HDDS-4210
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Blocker
>
> In HA, in validateAndUpdateCache when resolveBucket, it checks the permission
> using checkAcls. But it will have not any RpcContext and it will fail with
> NPE in checkAcls when getting hostName.
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.ozone.om.OzoneManager.checkAcls(OzoneManager.java:1604)
> at
> org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3497)
> at
> org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3465)
> at
> org.apache.hadoop.ozone.om.OzoneManager.resolveBucketLink(OzoneManager.java:3452)
> at
> org.apache.hadoop.ozone.om.request.key.OMKeyRequest.resolveBucketLink(OMKeyRequest.java:96)
> at
> org.apache.hadoop.ozone.om.request.key.OMKeyCreateRequest.validateAndUpdateCache(OMKeyCreateRequest.java:215)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:227)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:428)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$applyTransaction$1(OzoneManagerStateMachine.java:246)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request #1390: HDDS-4196. Add an endpoint in Recon to query Prometheus
avijayanhwx commented on a change in pull request #1390:
URL: https://github.com/apache/hadoop-ozone/pull/1390#discussion_r483749537
##
File path:
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/MetricsServiceProviderFactory.java
##
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.recon.ReconConfigKeys;
+import org.apache.hadoop.ozone.recon.spi.MetricsServiceProvider;
+import org.apache.hadoop.ozone.recon.spi.impl.PrometheusServiceProviderImpl;
+
+import javax.inject.Inject;
+import javax.inject.Singleton;
+
+/**
+ * Factory class that is used to get the instance of configured Metrics Service
+ * Provider.
+ */
+@Singleton
+public class MetricsServiceProviderFactory {
+
+ private OzoneConfiguration configuration;
+ private ReconUtils reconUtils;
+
+ @Inject
+ public MetricsServiceProviderFactory(OzoneConfiguration configuration,
+ ReconUtils reconUtils) {
+this.configuration = configuration;
+this.reconUtils = reconUtils;
+ }
+
+ /**
+ * Returns the configured MetricsServiceProvider implementation (defaults
+ * to prometheus).
+ * If no metrics service providers are configured, returns null.
+ *
+ * @return MetricsServiceProvider instance that is configured.
+ */
+ public MetricsServiceProvider getMetricsServiceProvider() {
+String prometheusEndpoint = getPrometheusEndpoint();
+if (prometheusEndpoint != null && !prometheusEndpoint.isEmpty()) {
Review comment:
Nit. can be StringUtils.isEmpty(prometheusEndpoint)
Nit. Let's add a log line saying we are choosing "Prometheus" as the
configured Metrics provider here or inside the constructor of
PrometheusServiceProviderImpl.
##
File path:
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/metrics/Metric.java
##
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.metrics;
+
+import org.apache.hadoop.hdds.annotation.InterfaceAudience;
+
+import java.util.Map;
+import java.util.SortedMap;
+import java.util.TreeMap;
+
+/**
+ * Class for wrapping a metric response from
+ * {@link org.apache.hadoop.ozone.recon.spi.MetricsServiceProvider}.
+ */
+@InterfaceAudience.Private
+public final class Metric {
+
+ private Map metadata;
+ private TreeMap values;
+
+ public Metric(Map metadata,
+SortedMap values) {
+this.metadata = metadata;
+this.values = (TreeMap) values;
Review comment:
We should do values.addAll() or Collections. unmodifiableMap() here,
since the caller may mutate the passed in reference.
##
File path:
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/metrics/Metric.java
##
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in
[jira] [Resolved] (HDDS-4208) Fix table rendering and logo display in docs
[ https://issues.apache.org/jira/browse/HDDS-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vivek Ratnavel Subramanian resolved HDDS-4208. -- Fix Version/s: 1.0.1 Resolution: Fixed > Fix table rendering and logo display in docs > > > Key: HDDS-4208 > URL: https://issues.apache.org/jira/browse/HDDS-4208 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: documentation >Affects Versions: 1.0.0 >Reporter: Vivek Ratnavel Subramanian >Assignee: Vivek Ratnavel Subramanian >Priority: Major > Labels: pull-request-available > Fix For: 1.0.1 > > Attachments: Screen Shot 2020-09-03 at 2.17.50 PM.png > > > The Docs page does not render the table correctly in OM and SCM architecture > pages. > Also, the ozone logo in the header is not visible. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] vivekratnavel merged pull request #1391: HDDS-4208. Fix table rendering and logo display in docs
vivekratnavel merged pull request #1391: URL: https://github.com/apache/hadoop-ozone/pull/1391 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] vivekratnavel commented on pull request #1391: HDDS-4208. Fix table rendering and logo display in docs
vivekratnavel commented on pull request #1391: URL: https://github.com/apache/hadoop-ozone/pull/1391#issuecomment-687286582 @adoroszlai Thanks for the review! You are right! 1.0.0 doc is rendering the tables correctly. But, I see that no tables are rendered in edge version of the docs - https://ci-hadoop.apache.org/view/Hadoop%20Ozone/job/ozone-doc-master/lastSuccessfulBuild/artifact/hadoop-hdds/docs/public/concept/ozonemanager.html Even though, I can see them rendered for other pages locally. I will keep an eye out on the edge version after this patch gets committed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request #1392: HDDS-4173. Implement HDDS Version management using the LayoutVersion…
avijayanhwx commented on a change in pull request #1392:
URL: https://github.com/apache/hadoop-ozone/pull/1392#discussion_r483365229
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/upgrade/SCMLayoutFeatureAPI.java
##
@@ -0,0 +1,35 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.upgrade;
+
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+import
org.apache.hadoop.hdds.scm.upgrade.SCMLayoutFeatureCatalog.SCMLayoutFeature;
+
+/**
+ * Annotation to specify if an API is backed up by a Layout Feature.
+ */
+@Target(ElementType.METHOD)
+@Retention(RetentionPolicy.RUNTIME)
+public @interface SCMLayoutFeatureAPI {
Review comment:
Can be removed.
##
File path: hadoop-hdds/server-scm/pom.xml
##
@@ -29,6 +29,19 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd;>
jar
+
+
Review comment:
Let's hold off on introducing Aspectj in HDDS layer, since we have not
fully understood the impact of bringing it in. Moreover, it suits a '**client
facing**' component like OzoneManager better since the point cut that we have
is used for disallowing '**pre-finalize state unsupported client APIs**'. Once
there are concrete API use cases for SCM, we can bring it here by moving the OM
Aspect to a common maven module.
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/upgrade/SCMLayoutFeatureCatalog.java
##
@@ -0,0 +1,93 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.upgrade;
+
+import java.util.Optional;
+
+import org.apache.hadoop.ozone.upgrade.LayoutFeature;
+
+/**
+ * Catalog of SCM features.
+ */
+public class SCMLayoutFeatureCatalog {
+
+ /**
+ * List of SCM Features.
+ */
+ public enum SCMLayoutFeature implements LayoutFeature {
Review comment:
Should be "HDDSLayoutFeature" since it is common across SCM and DN.
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/upgrade/SCMLayoutFeatureCatalog.java
##
@@ -0,0 +1,93 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.upgrade;
+
+import java.util.Optional;
+
+import org.apache.hadoop.ozone.upgrade.LayoutFeature;
+
+/**
+ * Catalog of SCM features.
+ */
+public class SCMLayoutFeatureCatalog {
+
+ /**
+ * List of SCM Features.
+ */
+ public enum SCMLayoutFeature implements LayoutFeature {
+INITIAL_VERSION(0, "Initial Layout Version"),
+CREATE_EC(1, ""),
+
[GitHub] [hadoop-ozone] rakeshadr commented on a change in pull request #1385: HDDS-3947: Sort DNs for client when the key is a file for #getFileStatus #listStatus APIs
rakeshadr commented on a change in pull request #1385: URL: https://github.com/apache/hadoop-ozone/pull/1385#discussion_r483727870 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/fs/OzoneManagerFS.java ## @@ -49,6 +69,21 @@ OpenKeySession createFile(OmKeyArgs args, boolean isOverWrite, */ OmKeyInfo lookupFile(OmKeyArgs args, String clientAddress) throws IOException; + /** + * List the status for a file or a directory and its contents. + * + * @param keyArgs the args of the key provided by client. + * @param recursive For a directory if true all the descendants of a + * particular directory are listed + * @param startKey Key from which listing needs to start. If startKey + * exists its status is included in the final list. + * @param numEntriesNumber of entries to list from the start key + * @param clientAddress a hint to key manager, order the datanode in returned + * pipeline by distance between client and datanode. + * @return list of file status + * @throws IOException if file or bucket or volume does not exist + */ List listStatus(OmKeyArgs keyArgs, boolean recursive, - String startKey, long numEntries) throws IOException; + String startKey, long numEntries, String clientAddress) Review comment: @adoroszlai , Sorry for re-opening the discussion on this comment. On a second thought `listStatus` is only used in [one place in the source code](https://github.com/apache/hadoop-ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java#L2910) and there won't be any other caller for `listStatus() without ClientAddress`. So, how about deleting the original method and can add it later if there is any need ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] rakeshadr commented on a change in pull request #1385: HDDS-3947: Sort DNs for client when the key is a file for #getFileStatus #listStatus APIs
rakeshadr commented on a change in pull request #1385: URL: https://github.com/apache/hadoop-ozone/pull/1385#discussion_r483727870 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/fs/OzoneManagerFS.java ## @@ -49,6 +69,21 @@ OpenKeySession createFile(OmKeyArgs args, boolean isOverWrite, */ OmKeyInfo lookupFile(OmKeyArgs args, String clientAddress) throws IOException; + /** + * List the status for a file or a directory and its contents. + * + * @param keyArgs the args of the key provided by client. + * @param recursive For a directory if true all the descendants of a + * particular directory are listed + * @param startKey Key from which listing needs to start. If startKey + * exists its status is included in the final list. + * @param numEntriesNumber of entries to list from the start key + * @param clientAddress a hint to key manager, order the datanode in returned + * pipeline by distance between client and datanode. + * @return list of file status + * @throws IOException if file or bucket or volume does not exist + */ List listStatus(OmKeyArgs keyArgs, boolean recursive, - String startKey, long numEntries) throws IOException; + String startKey, long numEntries, String clientAddress) Review comment: Sorry for re-opening the discussion on this comment. On a second thought `listStatus` is only used in [one place in the source code](https://github.com/apache/hadoop-ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java#L2910) and there won't be any other caller for `listStatus() without ClientAddress`. So, how about deleting the original method and can add it later if there is any need ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-4155) Directory and filename can end up with same name in a path
[
https://issues.apache.org/jira/browse/HDDS-4155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190803#comment-17190803
]
Bharat Viswanadham edited comment on HDDS-4155 at 9/4/20, 4:05 PM:
---
{quote}I am wondering what is the problem with it. It seems that Hadoop
Compatible File System can handle this and display both.
{quote}
Not tested my self, I think here to say S3AFileSystem supports it not every
Hadoop Compatible FileSystem supports it(Example below). My guess is as S3A
does not have control over AWS S3 implementation, might be they cannot avoid
this. But in the Ozone case, during KeyCommit we can catch this, and block it.
So, the question is do we want to support what kind of semantics in Ozone?
{quote}It seems to be very strange for me as this is something which should be
avoid with posix fs, but if hadoop can support it...
{quote}
An Example, HDFS disallows this, it does not allow file and a directory with
the same name in a path.
was (Author: bharatviswa):
{quote}I am wondering what is the problem with it. It seems that Hadoop
Compatible File System can handle this and display both.
{quote}
Not tested my self, I think here to say S3AFileSystem supports it not every
Hadoop Compatible FileSystem supports it(Example below). My guess is as S3A
does not have control over AWS S3 implementation, might be they cannot avoid
this. But in the Ozone case, during KeyCommit we can catch this, and block it.
So, the question is do we want to support such semantics in Ozone?
{quote}It seems to be very strange for me as this is something which should be
avoid with posix fs, but if hadoop can support it...
{quote}
An Example, HDFS disallows this, it does not allow file and a directory with
the same name in a path.
> Directory and filename can end up with same name in a path
> --
>
> Key: HDDS-4155
> URL: https://issues.apache.org/jira/browse/HDDS-4155
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
>
> Scenario:
> Create Key via S3, and Create Directory through Fs.
> # open key -> /a/b/c
> # CreateDirectory -> /a/b/c
> # CommitKey -> /a/b/c
> So, now in Ozone we will have directory and file with name "c"
> When created through Fs interface.
> # create file -> /a/b/c
> # CreateDirectory -> /a/b/c
> # CommitKey -> /a/b/c
> So, now in Ozone we will have directory and file with name "c"
>
> # InitiateMPU /a/b/c
> # Create Part1 /a/b/c
> # Commit Part1 /a/b/c
> # Create Directory /a/b/c
> # Complete MPU /a/b/c
> So, now in Ozone, we will have directory and file with name "c". In MPU this
> is one example scenario.
>
> Few proposals/ideas to solve this:
> # Check during commit whether a directory already exists with same name. But
> disadvantage is after user uploads the entire data during last stage we fail.
> (File system with create in progress acts similarly. Scenario: 1. vi t1 2.
> mkdir t1 3. Save t1: (Fail:"t1" is a directory)
> # During create directory check are there any open key creation with same
> name and fail.
>
> Any of the above approaches are not final, this Jira is opened to discuss
> this issue and come up with solution.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-4155) Directory and filename can end up with same name in a path
[
https://issues.apache.org/jira/browse/HDDS-4155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190803#comment-17190803
]
Bharat Viswanadham edited comment on HDDS-4155 at 9/4/20, 4:04 PM:
---
{quote}I am wondering what is the problem with it. It seems that Hadoop
Compatible File System can handle this and display both.
{quote}
Not tested my self, I think here to say S3AFileSystem supports it not every
Hadoop Compatible FileSystem supports it(Example below). My guess is as S3A
does not have control over AWS S3 implementation, might be they cannot avoid
this. But in the Ozone case, during KeyCommit we can catch this, and block it.
So, the question is do we want to support such semantics in Ozone?
{quote}It seems to be very strange for me as this is something which should be
avoid with posix fs, but if hadoop can support it...
{quote}
An Example, HDFS disallows this, it does not allow file and a directory with
the same name in a path.
was (Author: bharatviswa):
{quote}I am wondering what is the problem with it. It seems that Hadoop
Compatible File System can handle this and display both.
{quote}
Not tested my self, I think here to say S3AFileSystem supports it not every
Hadoop Compatible FileSystem supports it(Example below). My guess is as S3A
does not have control over AWS S3 implementation, they cannot avoid this. But
in the Ozone case, during KeyCommit we can catch this, and block it. So, the
question is do we want to support such semantics in Ozone?
{quote}It seems to be very strange for me as this is something which should be
avoid with posix fs, but if hadoop can support it...
{quote}
An Example, HDFS disallows this, it does not allow file and a directory with
the same name in a path.
> Directory and filename can end up with same name in a path
> --
>
> Key: HDDS-4155
> URL: https://issues.apache.org/jira/browse/HDDS-4155
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
>
> Scenario:
> Create Key via S3, and Create Directory through Fs.
> # open key -> /a/b/c
> # CreateDirectory -> /a/b/c
> # CommitKey -> /a/b/c
> So, now in Ozone we will have directory and file with name "c"
> When created through Fs interface.
> # create file -> /a/b/c
> # CreateDirectory -> /a/b/c
> # CommitKey -> /a/b/c
> So, now in Ozone we will have directory and file with name "c"
>
> # InitiateMPU /a/b/c
> # Create Part1 /a/b/c
> # Commit Part1 /a/b/c
> # Create Directory /a/b/c
> # Complete MPU /a/b/c
> So, now in Ozone, we will have directory and file with name "c". In MPU this
> is one example scenario.
>
> Few proposals/ideas to solve this:
> # Check during commit whether a directory already exists with same name. But
> disadvantage is after user uploads the entire data during last stage we fail.
> (File system with create in progress acts similarly. Scenario: 1. vi t1 2.
> mkdir t1 3. Save t1: (Fail:"t1" is a directory)
> # During create directory check are there any open key creation with same
> name and fail.
>
> Any of the above approaches are not final, this Jira is opened to discuss
> this issue and come up with solution.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4155) Directory and filename can end up with same name in a path
[
https://issues.apache.org/jira/browse/HDDS-4155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190803#comment-17190803
]
Bharat Viswanadham commented on HDDS-4155:
--
{quote}I am wondering what is the problem with it. It seems that Hadoop
Compatible File System can handle this and display both.
{quote}
Not tested my self, I think here to say S3AFileSystem supports it not every
Hadoop Compatible FileSystem supports it(Example below). My guess is as S3A
does not have control over AWS S3 implementation, they cannot avoid this. But
in the Ozone case, during KeyCommit we can catch this, and block it. So, the
question is do we want to support such semantics in Ozone?
{quote}It seems to be very strange for me as this is something which should be
avoid with posix fs, but if hadoop can support it...
{quote}
An Example, HDFS disallows this, it does not allow file and a directory with
the same name in a path.
> Directory and filename can end up with same name in a path
> --
>
> Key: HDDS-4155
> URL: https://issues.apache.org/jira/browse/HDDS-4155
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
>
> Scenario:
> Create Key via S3, and Create Directory through Fs.
> # open key -> /a/b/c
> # CreateDirectory -> /a/b/c
> # CommitKey -> /a/b/c
> So, now in Ozone we will have directory and file with name "c"
> When created through Fs interface.
> # create file -> /a/b/c
> # CreateDirectory -> /a/b/c
> # CommitKey -> /a/b/c
> So, now in Ozone we will have directory and file with name "c"
>
> # InitiateMPU /a/b/c
> # Create Part1 /a/b/c
> # Commit Part1 /a/b/c
> # Create Directory /a/b/c
> # Complete MPU /a/b/c
> So, now in Ozone, we will have directory and file with name "c". In MPU this
> is one example scenario.
>
> Few proposals/ideas to solve this:
> # Check during commit whether a directory already exists with same name. But
> disadvantage is after user uploads the entire data during last stage we fail.
> (File system with create in progress acts similarly. Scenario: 1. vi t1 2.
> mkdir t1 3. Save t1: (Fail:"t1" is a directory)
> # During create directory check are there any open key creation with same
> name and fail.
>
> Any of the above approaches are not final, this Jira is opened to discuss
> this issue and come up with solution.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #1385: HDDS-3947: Sort DNs for client when the key is a file for #getFileStatus #listStatus APIs
adoroszlai commented on a change in pull request #1385:
URL: https://github.com/apache/hadoop-ozone/pull/1385#discussion_r483649487
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
##
@@ -1783,7 +1802,9 @@ private OzoneFileStatus getOzoneFileStatus(String
volumeName,
// if the key is a file then do refresh pipeline info in OM by asking SCM
if (fileKeyInfo != null) {
-refreshPipeline(fileKeyInfo);
+if (refreshPipeline) {
Review comment:
Yes, both points make sense. Thanks for noticing the duplicate call.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] rakeshadr commented on a change in pull request #1385: HDDS-3947: Sort DNs for client when the key is a file for #getFileStatus #listStatus APIs
rakeshadr commented on a change in pull request #1385:
URL: https://github.com/apache/hadoop-ozone/pull/1385#discussion_r483645101
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
##
@@ -1783,7 +1802,9 @@ private OzoneFileStatus getOzoneFileStatus(String
volumeName,
// if the key is a file then do refresh pipeline info in OM by asking SCM
if (fileKeyInfo != null) {
-refreshPipeline(fileKeyInfo);
+if (refreshPipeline) {
Review comment:
Thanks @adoroszlai again for the review help.
Yes, you are correct. Test passed locally without the `if (refreshPipeline)`
condition.
Point-1) That means, the caller should call`refreshPipeline(OmKeyInfo
value)` method without the `refreshPipeline` condition.
Point-2) Also, I'm planning to move the
[refreshPipeline](https://github.com/apache/hadoop-ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java#L2131)
call outside BUCKET_LOCK and will be moving to
[here](https://github.com/apache/hadoop-ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java#L2147).
Now, its a duplicate call added as part of HDDS-3658.
Does these two changes make sense to you?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] rakeshadr commented on a change in pull request #1385: HDDS-3947: Sort DNs for client when the key is a file for #getFileStatus #listStatus APIs
rakeshadr commented on a change in pull request #1385: URL: https://github.com/apache/hadoop-ozone/pull/1385#discussion_r483645522 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/fs/OzoneManagerFS.java ## @@ -49,6 +69,21 @@ OpenKeySession createFile(OmKeyArgs args, boolean isOverWrite, */ OmKeyInfo lookupFile(OmKeyArgs args, String clientAddress) throws IOException; + /** + * List the status for a file or a directory and its contents. + * + * @param keyArgs the args of the key provided by client. + * @param recursive For a directory if true all the descendants of a + * particular directory are listed + * @param startKey Key from which listing needs to start. If startKey + * exists its status is included in the final list. + * @param numEntriesNumber of entries to list from the start key + * @param clientAddress a hint to key manager, order the datanode in returned + * pipeline by distance between client and datanode. + * @return list of file status + * @throws IOException if file or bucket or volume does not exist + */ List listStatus(OmKeyArgs keyArgs, boolean recursive, - String startKey, long numEntries) throws IOException; + String startKey, long numEntries, String clientAddress) Review comment: Sure, I will make the changes in my next commit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on pull request #1327: HDDS-4098. Improve om admin getserviceroles error message
adoroszlai commented on pull request #1327: URL: https://github.com/apache/hadoop-ozone/pull/1327#issuecomment-687171426 Thanks @dshavkanihort for working on this. The checkstyle issue @dineshchitlangia mentioned is that line 83 exceeds maximum allowed length of 80 characters: https://github.com/apache/hadoop-ozone/blob/773af08368d339a75703caef42f74d9ab8cf6bb2/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/om/OMAdmin.java#L83 You can find the problem in the _Summary of failures_ section after opening _Details_ of the failed _checkstyle_ check. Here is the direct link: https://github.com/apache/hadoop-ozone/pull/1327/checks?check_run_id=981251536#step:4:5 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] flirmnave commented on a change in pull request #1368: HDDS-4156. add hierarchical layout to Chinese doc
flirmnave commented on a change in pull request #1368: URL: https://github.com/apache/hadoop-ozone/pull/1368#discussion_r483637138 ## File path: hadoop-hdds/docs/content/concept/_index.zh.md ## @@ -2,7 +2,7 @@ title: 概念 date: "2017-10-10" menu: main -weight: 6 +weight: 3 Review comment: It’s ok. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4206) Attempt pipeline creation more frequently in acceptance tests
[ https://issues.apache.org/jira/browse/HDDS-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Doroszlai updated HDDS-4206: --- Status: Patch Available (was: In Progress) > Attempt pipeline creation more frequently in acceptance tests > - > > Key: HDDS-4206 > URL: https://issues.apache.org/jira/browse/HDDS-4206 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: test >Reporter: Attila Doroszlai >Assignee: Attila Doroszlai >Priority: Major > Labels: pull-request-available > > Acceptance tests are frequently failing recently due to SCM not being able to > exit safe mode for 3 minutes. Background pipeline creator attempts pipeline > creation every 2 minutes. It is safe to decrease this period in tests for > faster startup and less idle wait time. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4204) upgrade docker environment does not work with KEEP_RUNNING=true
[
https://issues.apache.org/jira/browse/HDDS-4204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai updated HDDS-4204:
---
Status: Patch Available (was: In Progress)
> upgrade docker environment does not work with KEEP_RUNNING=true
> ---
>
> Key: HDDS-4204
> URL: https://issues.apache.org/jira/browse/HDDS-4204
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: docker, test
>Affects Versions: 1.0.0
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Minor
> Labels: pull-request-available
>
> Ozone {{upgrade}} Docker Compose environment fails if run with
> {{KEEP_RUNNING=true}}. The variable is applied to both runs (pre- and
> post-upgrade), but pre-upgrade containers should be stopped anyway, since
> they will be replaced by the new ones.
> {code}
> $ cd hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT/compose/upgrade
> $ KEEP_RUNNING=true ./test.sh
> ...
> Failed: IO error: While lock file: scm.db/LOCK: Resource temporarily
> unavailable
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4198) Compile Ozone with multiple Java versions
[ https://issues.apache.org/jira/browse/HDDS-4198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Doroszlai updated HDDS-4198: --- Status: Patch Available (was: In Progress) > Compile Ozone with multiple Java versions > - > > Key: HDDS-4198 > URL: https://issues.apache.org/jira/browse/HDDS-4198 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: build >Reporter: Attila Doroszlai >Assignee: Attila Doroszlai >Priority: Major > Labels: pull-request-available > > Add matrix build in GitHub Actions for compiling Ozone with both Java 8 and > 11. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #1298: HDDS-3869. Use different column families for datanode block and metadata
elek commented on pull request #1298: URL: https://github.com/apache/hadoop-ozone/pull/1298#issuecomment-687089838 Thanks @errose28 the patch. I didn't check it fully, but I tried it with creating a new [half-baked patch](https://github.com/apache/hadoop-ozone/commit/80c8ecf05b310cc05aa148a1075aea8dbd763124) to try if we can support multiple RocksDB scheme (shared / per-container). I really liked the approach: Using wrapper `Table<>` implementation to add prefixes (like in `SchemaOneDeletedBlocksTable`) Also, `DatanodeStore` interface provides a good interface which can be re-used in the future (I removed `start/stop` from the interface, but in this patch it's fine). Earlier I realized that the interfaces of the containers (like `ContainerData`) is not very straightforward. I think we can use one generic `ContainerData` implementation long-term (instead of casting...), but -- again -- this is not for this patch. But I would consider moving `schemaVersion` to `ContainerData` as it seems to be a generic (and important) field (not blocking, either). +1 The high level class hierarchy, concept looks good to me (but I am not very good to find low level problems, others are more attentive than me ;-) ) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on pull request #1338: HDDS-4023. Delete closed container after all blocks have been deleted.
ChenSammi commented on pull request #1338: URL: https://github.com/apache/hadoop-ozone/pull/1338#issuecomment-687089550 Hi, @sodonnel do you have time for review? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on a change in pull request #1298: HDDS-3869. Use different column families for datanode block and metadata
elek commented on a change in pull request #1298:
URL: https://github.com/apache/hadoop-ozone/pull/1298#discussion_r483555964
##
File path:
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueBlockIterator.java
##
@@ -88,196 +86,314 @@ public void setUp() throws Exception {
conf = new OzoneConfiguration();
conf.set(HDDS_DATANODE_DIR_KEY, testRoot.getAbsolutePath());
volumeSet = new MutableVolumeSet(UUID.randomUUID().toString(), conf);
+
+containerData = new KeyValueContainerData(105L,
+layout,
+(long) StorageUnit.GB.toBytes(1), UUID.randomUUID().toString(),
+UUID.randomUUID().toString());
+// Init the container.
+container = new KeyValueContainer(containerData, conf);
+container.create(volumeSet, new RoundRobinVolumeChoosingPolicy(), UUID
+.randomUUID().toString());
+db = BlockUtils.getDB(containerData, conf);
}
@After
- public void tearDown() {
+ public void tearDown() throws Exception {
+db.close();
+db.cleanup();
volumeSet.shutdown();
FileUtil.fullyDelete(testRoot);
}
@Test
public void testKeyValueBlockIteratorWithMixedBlocks() throws Exception {
-
-long containerID = 100L;
-int deletedBlocks = 5;
+int deletingBlocks = 5;
int normalBlocks = 5;
-createContainerWithBlocks(containerID, normalBlocks, deletedBlocks);
-String containerPath = new File(containerData.getMetadataPath())
-.getParent();
-try(KeyValueBlockIterator keyValueBlockIterator = new
KeyValueBlockIterator(
-containerID, new File(containerPath))) {
+Map> blockIDs = createContainerWithBlocks(CONTAINER_ID,
+normalBlocks,
+deletingBlocks);
Review comment:
> @elek, any idea if continuation indent should be 4 or does it not
matter? Checkstyle CI seems to not care about it.
I don't think there is any rule. It can be reasonable to introduce new one
(to reduce noise during merges) but in that case we should provide good
auto-formatting rules to avoid any overhead.
(Similar to the import format, we can improve checkstyle, but we also need
autoformatter rules).
I think @maobaolong had a few ideas, but the effort is blocked by #921
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-4168) Remove reference to Skaffold in the README in dist/
[ https://issues.apache.org/jira/browse/HDDS-4168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Elek resolved HDDS-4168. --- Fix Version/s: 1.1.0 Resolution: Fixed > Remove reference to Skaffold in the README in dist/ > --- > > Key: HDDS-4168 > URL: https://issues.apache.org/jira/browse/HDDS-4168 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: documentation >Reporter: Alex Scammon >Assignee: Alex Scammon >Priority: Trivial > Labels: pull-request-available > Fix For: 1.1.0 > > > I got sidetracked when I ran into the README in the dist folder because it > referenced Skaffold which hasn't been used in a while. > So that others don't get confused as I did, I created a PR to fix the > wayward reference: > * [https://github.com/apache/hadoop-ozone/pull/1360] > This issue is merely to track the PR. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek merged pull request #1360: HDDS-4168. Remove archaic reference to Skaffold
elek merged pull request #1360: URL: https://github.com/apache/hadoop-ozone/pull/1360 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on pull request #1386: HDDS-4197. Failed to load existing service definition files: ...SubcommandWithParent
adoroszlai commented on pull request #1386: URL: https://github.com/apache/hadoop-ozone/pull/1386#issuecomment-687084333 Thanks @elek for reviewing and committing it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #1360: HDDS-4168. Remove archaic reference to Skaffold
elek commented on pull request #1360: URL: https://github.com/apache/hadoop-ozone/pull/1360#issuecomment-687084227 > I'm not sure what your stance is on force-pushing in this repo Earlier I used force-pushing, too. I think it's easier to develop locally with rebase. But @adoroszlai convinced me after a while to use merge. Sometimes it's more noisy, but in general it's easier to follow (and we can see the results of the earlier builds). (And I started to use helper scripts to support it from local...) And in the current case, It doesn't matter, I am grateful for the help anyway ;-) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-4197) Failed to load existing service definition files: ...SubcommandWithParent
[
https://issues.apache.org/jira/browse/HDDS-4197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Elek resolved HDDS-4197.
---
Fix Version/s: 1.1.0
Resolution: Fixed
> Failed to load existing service definition files: ...SubcommandWithParent
> -
>
> Key: HDDS-4197
> URL: https://issues.apache.org/jira/browse/HDDS-4197
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: build
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Major
> Labels: jdk11, pull-request-available
> Fix For: 1.1.0
>
>
> {code}
> [INFO] Apache Hadoop HDDS Tools ... FAILURE
> ...
> [ERROR] Failed to load existing service definition files:
> java.nio.file.NoSuchFileException:
> hadoop-hdds/tools/target/classes/META-INF/services/org.apache.hadoop.hdds.cli.SubcommandWithParent
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek merged pull request #1386: HDDS-4197. Failed to load existing service definition files: ...SubcommandWithParent
elek merged pull request #1386: URL: https://github.com/apache/hadoop-ozone/pull/1386 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4155) Directory and filename can end up with same name in a path
[ https://issues.apache.org/jira/browse/HDDS-4155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190662#comment-17190662 ] Marton Elek commented on HDDS-4155: --- Yesterday I learned that s3a supports this: it can display file name and directory with the same name. I am wondering what is the problem with it. It seems that Hadoop Compatible File System can handle this and display both. It seems to be very strange for me as this is something which should be avoid with posix fs, but if hadoop can support it... > Directory and filename can end up with same name in a path > -- > > Key: HDDS-4155 > URL: https://issues.apache.org/jira/browse/HDDS-4155 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > > Scenario: > Create Key via S3, and Create Directory through Fs. > # open key -> /a/b/c > # CreateDirectory -> /a/b/c > # CommitKey -> /a/b/c > So, now in Ozone we will have directory and file with name "c" > When created through Fs interface. > # create file -> /a/b/c > # CreateDirectory -> /a/b/c > # CommitKey -> /a/b/c > So, now in Ozone we will have directory and file with name "c" > > # InitiateMPU /a/b/c > # Create Part1 /a/b/c > # Commit Part1 /a/b/c > # Create Directory /a/b/c > # Complete MPU /a/b/c > So, now in Ozone, we will have directory and file with name "c". In MPU this > is one example scenario. > > Few proposals/ideas to solve this: > # Check during commit whether a directory already exists with same name. But > disadvantage is after user uploads the entire data during last stage we fail. > (File system with create in progress acts similarly. Scenario: 1. vi t1 2. > mkdir t1 3. Save t1: (Fail:"t1" is a directory) > # During create directory check are there any open key creation with same > name and fail. > > Any of the above approaches are not final, this Jira is opened to discuss > this issue and come up with solution. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc commented on a change in pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
captainzmc commented on a change in pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337#discussion_r483510372
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##
@@ -3540,6 +3546,43 @@ public boolean getEnableFileSystemPaths() {
OZONE_OM_ENABLE_FILESYSTEM_PATHS_DEFAULT);
}
+ /**
+ * Initialize volume's quota, which resolves the problem that the old quota
+ * initial value was not set correctly. This method will be removed in a
+ * later version.
+ * @throws IOException
+ */
+ private void initializeOldVolumeQuota() throws IOException {
+
+TableIterator> iterator = metadataManager.getVolumeTable().iterator();
+while(iterator.hasNext()) {
+ OmVolumeArgs omVolumeArgs = iterator.next().getValue();
+ String omVolumeKey = metadataManager.getVolumeKey(
+ omVolumeArgs.getVolume());
+ // Previously, the volume quota was created to default to 1EB. To change
+ // to -1, default does not enable quota.
+ if (omVolumeArgs.getQuotaInBytes() == 1024 * 1024 * TB ||
Review comment:
Currently, the way to determine the old quota is by seeing whether his
quota is 1 EB. So 1EB is still a special value. At present, OZone's cluster
size is not large, far from the upper limit of 1EB.
So can we ignore this case? In addition, I adjust the LOG level from INFO to
Warn (I can change it to ERROR if needed) to alert the user that the
corresponding quota has been changed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc commented on a change in pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
captainzmc commented on a change in pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337#discussion_r483510372
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##
@@ -3540,6 +3546,43 @@ public boolean getEnableFileSystemPaths() {
OZONE_OM_ENABLE_FILESYSTEM_PATHS_DEFAULT);
}
+ /**
+ * Initialize volume's quota, which resolves the problem that the old quota
+ * initial value was not set correctly. This method will be removed in a
+ * later version.
+ * @throws IOException
+ */
+ private void initializeOldVolumeQuota() throws IOException {
+
+TableIterator> iterator = metadataManager.getVolumeTable().iterator();
+while(iterator.hasNext()) {
+ OmVolumeArgs omVolumeArgs = iterator.next().getValue();
+ String omVolumeKey = metadataManager.getVolumeKey(
+ omVolumeArgs.getVolume());
+ // Previously, the volume quota was created to default to 1EB. To change
+ // to -1, default does not enable quota.
+ if (omVolumeArgs.getQuotaInBytes() == 1024 * 1024 * TB ||
Review comment:
Currently, the way to determine the old quota is by seeing whether his
quota is 1 EB. So 1EB is still a special value. At present, OZone's cluster
size is not large, far from the upper limit of 1EB.
So can we ignore this case? In addition, I adjust the LOG level from INFO to
Warn (I can change it to error if needed) to alert the user that the
corresponding quota has been changed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] GlenGeng edited a comment on pull request #1319: HDDS-4107: replace scmID with clusterID for container and volume at Datanode side
GlenGeng edited a comment on pull request #1319: URL: https://github.com/apache/hadoop-ozone/pull/1319#issuecomment-687041994 Hey @elek , I found that scmId not only exists in layout of volume dir, but also in container meta data. You may check the analysis from [here](https://issues.apache.org/jira/browse/HDDS-4107?focusedCommentId=17190638=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17190638) I consider It will make the upgrade infeasible for a huge cluster consisting of millions of containers. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] GlenGeng commented on pull request #1319: HDDS-4107: replace scmID with clusterID for container and volume at Datanode side
GlenGeng commented on pull request #1319: URL: https://github.com/apache/hadoop-ozone/pull/1319#issuecomment-687041994 Hey @elek , I found that scmId not only exists in layout of volume dir, but also in container meta data. I consider It will make the upgrade infeasible for a huge cluster consisting of millions of containers. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc commented on a change in pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
captainzmc commented on a change in pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337#discussion_r483501073
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##
@@ -3540,6 +3546,43 @@ public boolean getEnableFileSystemPaths() {
OZONE_OM_ENABLE_FILESYSTEM_PATHS_DEFAULT);
}
+ /**
+ * Initialize volume's quota, which resolves the problem that the old quota
+ * initial value was not set correctly. This method will be removed in a
+ * later version.
+ * @throws IOException
+ */
+ private void initializeOldVolumeQuota() throws IOException {
+
+TableIterator> iterator = metadataManager.getVolumeTable().iterator();
+while(iterator.hasNext()) {
+ OmVolumeArgs omVolumeArgs = iterator.next().getValue();
+ String omVolumeKey = metadataManager.getVolumeKey(
+ omVolumeArgs.getVolume());
+ // Previously, the volume quota was created to default to 1EB. To change
+ // to -1, default does not enable quota.
+ if (omVolumeArgs.getQuotaInBytes() == 1024 * 1024 * TB ||
+ omVolumeArgs.getQuotaInBytes() == OzoneConsts.MAX_QUOTA_IN_BYTES) {
Review comment:
This judgment is redundant, will delete it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc edited a comment on pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
captainzmc edited a comment on pull request #1337: URL: https://github.com/apache/hadoop-ozone/pull/1337#issuecomment-686945316 Hi @ChenSammi , I had added the initialization and change the old volume quota default value to -1 when OM starts. Could you help review this PR again? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #1385: HDDS-3947: Sort DNs for client when the key is a file for #getFileStatus #listStatus APIs
adoroszlai commented on a change in pull request #1385:
URL: https://github.com/apache/hadoop-ozone/pull/1385#discussion_r483453091
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
##
@@ -1783,7 +1802,9 @@ private OzoneFileStatus getOzoneFileStatus(String
volumeName,
// if the key is a file then do refresh pipeline info in OM by asking SCM
if (fileKeyInfo != null) {
-refreshPipeline(fileKeyInfo);
+if (refreshPipeline) {
Review comment:
I think `refreshPipeline` condition was removed intentionally in
HDDS-3658 and should not be restored. (This is also causing failure of
`TestKeyManagerUnit.testLookupFileWithDnFailure` unit test.)
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/fs/OzoneManagerFS.java
##
@@ -49,6 +69,21 @@ OpenKeySession createFile(OmKeyArgs args, boolean
isOverWrite,
*/
OmKeyInfo lookupFile(OmKeyArgs args, String clientAddress) throws
IOException;
+ /**
+ * List the status for a file or a directory and its contents.
+ *
+ * @param keyArgs the args of the key provided by client.
+ * @param recursive For a directory if true all the descendants of a
+ * particular directory are listed
+ * @param startKey Key from which listing needs to start. If startKey
+ * exists its status is included in the final list.
+ * @param numEntriesNumber of entries to list from the start key
+ * @param clientAddress a hint to key manager, order the datanode in returned
+ * pipeline by distance between client and datanode.
+ * @return list of file status
+ * @throws IOException if file or bucket or volume does not exist
+ */
List listStatus(OmKeyArgs keyArgs, boolean recursive,
- String startKey, long numEntries) throws IOException;
+ String startKey, long numEntries, String clientAddress)
Review comment:
In my
[comment](https://github.com/apache/hadoop-ozone/pull/1192#discussion_r481025587)
on the previous PR:
> Should we also keep the two original methods?
I meant both `getFileStatus` and `listStatus`. Sorry if it was unclear.
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
##
@@ -2144,10 +2166,14 @@ private void listStatusFindKeyInTableCache(
metadataManager.getLock().releaseReadLock(BUCKET_LOCK, volumeName,
bucketName);
}
-if (args.getRefreshPipeline()) {
- for(OzoneFileStatus fileStatus : fileStatusList){
+
+for (OzoneFileStatus fileStatus : fileStatusList) {
+ if (args.getRefreshPipeline()) {
Review comment:
@ChenSammi Was this instance of `getRefreshPipeline` condition retained
in HDDS-3658 intentionally?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc commented on a change in pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
captainzmc commented on a change in pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337#discussion_r483493420
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##
@@ -3540,6 +3546,43 @@ public boolean getEnableFileSystemPaths() {
OZONE_OM_ENABLE_FILESYSTEM_PATHS_DEFAULT);
}
+ /**
+ * Initialize volume's quota, which resolves the problem that the old quota
+ * initial value was not set correctly. This method will be removed in a
+ * later version.
+ * @throws IOException
+ */
+ private void initializeOldVolumeQuota() throws IOException {
Review comment:
Yes, the corresponding [UT has been
added](https://github.com/apache/hadoop-ozone/pull/1337/files#diff-d914f1d4b911db08ba389a30c322a338R308).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-4107) replace scmID with clusterID for container and volume at Datanode side
[
https://issues.apache.org/jira/browse/HDDS-4107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190638#comment-17190638
]
Glen Geng edited comment on HDDS-4107 at 9/4/20, 9:06 AM:
--
I am working on the upgrade issues for this PR, found that renaming the volume
dir is not enough.
The scmId is not only in the volume dir, but in meta data of container as well,
which will make the upgrade procedure for a huge cluster to be impossible.
Here is an example from the misc/upgrade case of acceptance.
*Version file as follows:*
{code:java}
[hadoop@9
~/glengeng/hadoop-ozone/hadoop-ozone/dist/target/ozone-0.6.0-SNAPSHOT/compose/upgrade/data]$
cat scm/metadata/scm/current/VERSION
#Fri Sep 04 07:08:47 UTC 2020
cTime=1599203327270
clusterID=CID-a1d16168-8a93-4ab2-b276-a68e0bf4dbb5
nodeType=SCM
scmUuid=9176f875-c8f2-4dcd-8d1d-5b988ad25914
{code}
*the layout after rename the volume layout from scmId to clusterId*
{code:java}
[hadoop@9
~/glengeng/hadoop-ozone/hadoop-ozone/dist/target/ozone-0.6.0-SNAPSHOT/compose/upgrade/data/dn2/hdds]$
tree .
.
|-- hdds
| |-- CID-a1d16168-8a93-4ab2-b276-a68e0bf4dbb5
| | `-- current
| | `-- containerDir0
| | `-- 1
| | |-- chunks
| | | `-- 104805390775943168_chunk_1
| | `-- metadata
| | |-- 1-dn-container.db
| | | |-- 06.log
| | | |-- CURRENT
| | | |-- IDENTITY
| | | |-- LOCK
| | | |-- LOG
| | | |-- LOG.old.1599203351819700
| | | |-- MANIFEST-05
| | | |-- OPTIONS-05
| | | `-- OPTIONS-08
| | `-- 1.container
| `-- VERSION
`-- scmUsed8 directories, 13 files
{code}
*grep scmId, and found them from meta data of container.*
{code:java}
[hadoop@9
~/glengeng/hadoop-ozone/hadoop-ozone/dist/target/ozone-0.6.0-SNAPSHOT/compose/upgrade/data/dn2]$
find . -type f | xargs grep 9176f875
./hdds/hdds/CID-a1d16168-8a93-4ab2-b276-a68e0bf4dbb5/current/containerDir0/1/metadata/1-dn-container.db/OPTIONS-08:
wal_dir=/data/hdds/hdds/9176f875-c8f2-4dcd-8d1d-5b988ad25914/current/containerDir0/1/metadata/1-dn-container.db
./hdds/hdds/CID-a1d16168-8a93-4ab2-b276-a68e0bf4dbb5/current/containerDir0/1/metadata/1-dn-container.db/LOG:2020/09/04-07:09:11.819945
7fe6f187b700 SST files in
/data/hdds/hdds/9176f875-c8f2-4dcd-8d1d-5b988ad25914/current/containerDir0/1/metadata/1-dn-container.db
dir, Total Num: 0, files:
./hdds/hdds/CID-a1d16168-8a93-4ab2-b276-a68e0bf4dbb5/current/containerDir0/1/metadata/1-dn-container.db/LOG:2020/09/04-07:09:11.819947
7fe6f187b700 Write Ahead Log file in
/data/hdds/hdds/9176f875-c8f2-4dcd-8d1d-5b988ad25914/current/containerDir0/1/metadata/1-dn-container.db:
03.log size: 0 ;
./hdds/hdds/CID-a1d16168-8a93-4ab2-b276-a68e0bf4dbb5/current/containerDir0/1/metadata/1-dn-container.db/LOG:2020/09/04-07:09:11.819972
7fe6f187b700 Options.wal_dir:
/data/hdds/hdds/9176f875-c8f2-4dcd-8d1d-5b988ad25914/current/containerDir0/1/metadata/1-dn-container.db
./hdds/hdds/CID-a1d16168-8a93-4ab2-b276-a68e0bf4dbb5/current/containerDir0/1/metadata/1-dn-container.db/LOG:2020/09/04-07:09:11.820954
7fe6f187b700 [/version_set.cc:3731] Recovered from manifest
file:/data/hdds/hdds/9176f875-c8f2-4dcd-8d1d-5b988ad25914/current/containerDir0/1/metadata/1-dn-container.db/MANIFEST-01
succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is
0, log_number is 0,prev_log_number is 0,max_column_family is
0,min_log_number_to_keep is 0
./hdds/hdds/CID-a1d16168-8a93-4ab2-b276-a68e0bf4dbb5/current/containerDir0/1/metadata/1-dn-container.db/OPTIONS-05:
wal_dir=/data/hdds/hdds/9176f875-c8f2-4dcd-8d1d-5b988ad25914/current/containerDir0/1/metadata/1-dn-container.db
./hdds/hdds/CID-a1d16168-8a93-4ab2-b276-a68e0bf4dbb5/current/containerDir0/1/metadata/1-dn-container.db/LOG.old.1599203351819700:2020/09/04-07:09:11.716714
7fe702591700 SST files in
/data/hdds/hdds/9176f875-c8f2-4dcd-8d1d-5b988ad25914/current/containerDir0/1/metadata/1-dn-container.db
dir, Total Num: 0, files:
./hdds/hdds/CID-a1d16168-8a93-4ab2-b276-a68e0bf4dbb5/current/containerDir0/1/metadata/1-dn-container.db/LOG.old.1599203351819700:2020/09/04-07:09:11.716716
7fe702591700 Write Ahead Log file in
/data/hdds/hdds/9176f875-c8f2-4dcd-8d1d-5b988ad25914/current/containerDir0/1/metadata/1-dn-container.db:
./hdds/hdds/CID-a1d16168-8a93-4ab2-b276-a68e0bf4dbb5/current/containerDir0/1/metadata/1-dn-container.db/LOG.old.1599203351819700:2020/09/04-07:09:11.716737
7fe702591700 Options.wal_dir:
/data/hdds/hdds/9176f875-c8f2-4dcd-8d1d-5b988ad25914/current/containerDir0/1/metadata/1-dn-container.db
[jira] [Commented] (HDDS-4107) replace scmID with clusterID for container and volume at Datanode side
[
https://issues.apache.org/jira/browse/HDDS-4107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190638#comment-17190638
]
Glen Geng commented on HDDS-4107:
-
I am working on the upgrade issues for this PR, found that renaming the volume
dir is not enough.
The scmId is not only in the volume dir, but in meta data of container as well,
which will make the upgrade procedure for a huge cluster to be impossible.
Here is an example from the misc/upgrade case of acceptance.
Version file as follows:
{code:java}
[hadoop@9
~/glengeng/hadoop-ozone/hadoop-ozone/dist/target/ozone-0.6.0-SNAPSHOT/compose/upgrade/data]$
cat scm/metadata/scm/current/VERSION
#Fri Sep 04 07:08:47 UTC 2020
cTime=1599203327270
clusterID=CID-a1d16168-8a93-4ab2-b276-a68e0bf4dbb5
nodeType=SCM
scmUuid=9176f875-c8f2-4dcd-8d1d-5b988ad25914
{code}
the layout after rename the volume layout from scmId to clusterId
{code:java}
[hadoop@9
~/glengeng/hadoop-ozone/hadoop-ozone/dist/target/ozone-0.6.0-SNAPSHOT/compose/upgrade/data/dn2/hdds]$
tree .
.
|-- hdds
| |-- CID-a1d16168-8a93-4ab2-b276-a68e0bf4dbb5
| | `-- current
| | `-- containerDir0
| | `-- 1
| | |-- chunks
| | | `-- 104805390775943168_chunk_1
| | `-- metadata
| | |-- 1-dn-container.db
| | | |-- 06.log
| | | |-- CURRENT
| | | |-- IDENTITY
| | | |-- LOCK
| | | |-- LOG
| | | |-- LOG.old.1599203351819700
| | | |-- MANIFEST-05
| | | |-- OPTIONS-05
| | | `-- OPTIONS-08
| | `-- 1.container
| `-- VERSION
`-- scmUsed8 directories, 13 files
{code}
grep scmId, and found them from meta data of container.
{code:java}
[hadoop@9
~/glengeng/hadoop-ozone/hadoop-ozone/dist/target/ozone-0.6.0-SNAPSHOT/compose/upgrade/data/dn2]$
find . -type f | xargs grep 9176f875
./hdds/hdds/CID-a1d16168-8a93-4ab2-b276-a68e0bf4dbb5/current/containerDir0/1/metadata/1-dn-container.db/OPTIONS-08:
wal_dir=/data/hdds/hdds/9176f875-c8f2-4dcd-8d1d-5b988ad25914/current/containerDir0/1/metadata/1-dn-container.db
./hdds/hdds/CID-a1d16168-8a93-4ab2-b276-a68e0bf4dbb5/current/containerDir0/1/metadata/1-dn-container.db/LOG:2020/09/04-07:09:11.819945
7fe6f187b700 SST files in
/data/hdds/hdds/9176f875-c8f2-4dcd-8d1d-5b988ad25914/current/containerDir0/1/metadata/1-dn-container.db
dir, Total Num: 0, files:
./hdds/hdds/CID-a1d16168-8a93-4ab2-b276-a68e0bf4dbb5/current/containerDir0/1/metadata/1-dn-container.db/LOG:2020/09/04-07:09:11.819947
7fe6f187b700 Write Ahead Log file in
/data/hdds/hdds/9176f875-c8f2-4dcd-8d1d-5b988ad25914/current/containerDir0/1/metadata/1-dn-container.db:
03.log size: 0 ;
./hdds/hdds/CID-a1d16168-8a93-4ab2-b276-a68e0bf4dbb5/current/containerDir0/1/metadata/1-dn-container.db/LOG:2020/09/04-07:09:11.819972
7fe6f187b700 Options.wal_dir:
/data/hdds/hdds/9176f875-c8f2-4dcd-8d1d-5b988ad25914/current/containerDir0/1/metadata/1-dn-container.db
./hdds/hdds/CID-a1d16168-8a93-4ab2-b276-a68e0bf4dbb5/current/containerDir0/1/metadata/1-dn-container.db/LOG:2020/09/04-07:09:11.820954
7fe6f187b700 [/version_set.cc:3731] Recovered from manifest
file:/data/hdds/hdds/9176f875-c8f2-4dcd-8d1d-5b988ad25914/current/containerDir0/1/metadata/1-dn-container.db/MANIFEST-01
succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is
0, log_number is 0,prev_log_number is 0,max_column_family is
0,min_log_number_to_keep is 0
./hdds/hdds/CID-a1d16168-8a93-4ab2-b276-a68e0bf4dbb5/current/containerDir0/1/metadata/1-dn-container.db/OPTIONS-05:
wal_dir=/data/hdds/hdds/9176f875-c8f2-4dcd-8d1d-5b988ad25914/current/containerDir0/1/metadata/1-dn-container.db
./hdds/hdds/CID-a1d16168-8a93-4ab2-b276-a68e0bf4dbb5/current/containerDir0/1/metadata/1-dn-container.db/LOG.old.1599203351819700:2020/09/04-07:09:11.716714
7fe702591700 SST files in
/data/hdds/hdds/9176f875-c8f2-4dcd-8d1d-5b988ad25914/current/containerDir0/1/metadata/1-dn-container.db
dir, Total Num: 0, files:
./hdds/hdds/CID-a1d16168-8a93-4ab2-b276-a68e0bf4dbb5/current/containerDir0/1/metadata/1-dn-container.db/LOG.old.1599203351819700:2020/09/04-07:09:11.716716
7fe702591700 Write Ahead Log file in
/data/hdds/hdds/9176f875-c8f2-4dcd-8d1d-5b988ad25914/current/containerDir0/1/metadata/1-dn-container.db:
./hdds/hdds/CID-a1d16168-8a93-4ab2-b276-a68e0bf4dbb5/current/containerDir0/1/metadata/1-dn-container.db/LOG.old.1599203351819700:2020/09/04-07:09:11.716737
7fe702591700 Options.wal_dir:
/data/hdds/hdds/9176f875-c8f2-4dcd-8d1d-5b988ad25914/current/containerDir0/1/metadata/1-dn-container.db
[GitHub] [hadoop-ozone] adoroszlai merged pull request #1394: HDDS-4205. Make coverage upload to codecov optional
adoroszlai merged pull request #1394: URL: https://github.com/apache/hadoop-ozone/pull/1394 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4205) Fix or disable coverage upload to codecov (pull requests)
[
https://issues.apache.org/jira/browse/HDDS-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai updated HDDS-4205:
---
Labels: (was: pull-request-available)
> Fix or disable coverage upload to codecov (pull requests)
> -
>
> Key: HDDS-4205
> URL: https://issues.apache.org/jira/browse/HDDS-4205
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Marton Elek
>Assignee: Marton Elek
>Priority: Major
> Fix For: 1.1.0
>
>
> I would like to start a conversation about disabling the coverage data
> uploading to codecov
> 1. It seems to be unreliable, HTTP 400 is a very common answer. I checked
> and the @v1 is the latest version from the action
> {code}
> -> Pinging Codecov
> https://codecov.io/upload/v4?package=bash-20200825-997b141=secret=HDDS-4193=361ddccd0666a8bd24240cda2652a903d4b69013=237476928_url=http%3A%2F%2Fgithub.com%2Fapache%2Fhadoop-ozone%2Factions%2Fruns%2F237476928=codecov-umbrella==apache%2Fhadoop-ozone=github-actions==1384=_args=f,n,F,Z
> HTTP 400
> {code}
> 2. The reported coverage data is meaningless as there is no good mechanism
> to use the latest stable build as baseline.
> I think we should disable codecov (sonar is still available) until we will
> have a solution for these issue.
> [~adoroszlai], [~vivekratnavel]: What do you think?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-4205) Fix or disable coverage upload to codecov (pull requests)
[
https://issues.apache.org/jira/browse/HDDS-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai resolved HDDS-4205.
Fix Version/s: 1.1.0
Resolution: Done
> Fix or disable coverage upload to codecov (pull requests)
> -
>
> Key: HDDS-4205
> URL: https://issues.apache.org/jira/browse/HDDS-4205
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Marton Elek
>Assignee: Marton Elek
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.1.0
>
>
> I would like to start a conversation about disabling the coverage data
> uploading to codecov
> 1. It seems to be unreliable, HTTP 400 is a very common answer. I checked
> and the @v1 is the latest version from the action
> {code}
> -> Pinging Codecov
> https://codecov.io/upload/v4?package=bash-20200825-997b141=secret=HDDS-4193=361ddccd0666a8bd24240cda2652a903d4b69013=237476928_url=http%3A%2F%2Fgithub.com%2Fapache%2Fhadoop-ozone%2Factions%2Fruns%2F237476928=codecov-umbrella==apache%2Fhadoop-ozone=github-actions==1384=_args=f,n,F,Z
> HTTP 400
> {code}
> 2. The reported coverage data is meaningless as there is no good mechanism
> to use the latest stable build as baseline.
> I think we should disable codecov (sonar is still available) until we will
> have a solution for these issue.
> [~adoroszlai], [~vivekratnavel]: What do you think?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
ChenSammi commented on a change in pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337#discussion_r483480047
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##
@@ -3540,6 +3546,43 @@ public boolean getEnableFileSystemPaths() {
OZONE_OM_ENABLE_FILESYSTEM_PATHS_DEFAULT);
}
+ /**
+ * Initialize volume's quota, which resolves the problem that the old quota
+ * initial value was not set correctly. This method will be removed in a
+ * later version.
+ * @throws IOException
+ */
+ private void initializeOldVolumeQuota() throws IOException {
Review comment:
Can we add a UT for this function?
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##
@@ -3540,6 +3546,43 @@ public boolean getEnableFileSystemPaths() {
OZONE_OM_ENABLE_FILESYSTEM_PATHS_DEFAULT);
}
+ /**
+ * Initialize volume's quota, which resolves the problem that the old quota
+ * initial value was not set correctly. This method will be removed in a
+ * later version.
+ * @throws IOException
+ */
+ private void initializeOldVolumeQuota() throws IOException {
+
+TableIterator> iterator = metadataManager.getVolumeTable().iterator();
+while(iterator.hasNext()) {
+ OmVolumeArgs omVolumeArgs = iterator.next().getValue();
+ String omVolumeKey = metadataManager.getVolumeKey(
+ omVolumeArgs.getVolume());
+ // Previously, the volume quota was created to default to 1EB. To change
+ // to -1, default does not enable quota.
+ if (omVolumeArgs.getQuotaInBytes() == 1024 * 1024 * TB ||
+ omVolumeArgs.getQuotaInBytes() == OzoneConsts.MAX_QUOTA_IN_BYTES) {
+omVolumeArgs.setQuotaInBytes(-1);
+// Commit to DB.
+BatchOperation batchOperation =
+metadataManager.getStore().initBatchOperation();
+metadataManager.getVolumeTable().putWithBatch(batchOperation,
+omVolumeKey, omVolumeArgs);
+metadataManager.getStore().commitBatchOperation(batchOperation);
+
+// Add to cache.
+long transactionID = (Long.MAX_VALUE - 1) >> 8;
+metadataManager.getVolumeTable().addCacheEntry(
+new CacheKey<>(omVolumeKey),
+new CacheValue<>(Optional.of(omVolumeArgs), transactionID));
+LOG.info("Update Volume {}'s quota to -1, , default does not enable" +
Review comment:
duplicate ","
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##
@@ -3540,6 +3546,43 @@ public boolean getEnableFileSystemPaths() {
OZONE_OM_ENABLE_FILESYSTEM_PATHS_DEFAULT);
}
+ /**
+ * Initialize volume's quota, which resolves the problem that the old quota
+ * initial value was not set correctly. This method will be removed in a
+ * later version.
+ * @throws IOException
+ */
+ private void initializeOldVolumeQuota() throws IOException {
+
+TableIterator> iterator = metadataManager.getVolumeTable().iterator();
+while(iterator.hasNext()) {
+ OmVolumeArgs omVolumeArgs = iterator.next().getValue();
+ String omVolumeKey = metadataManager.getVolumeKey(
+ omVolumeArgs.getVolume());
+ // Previously, the volume quota was created to default to 1EB. To change
+ // to -1, default does not enable quota.
+ if (omVolumeArgs.getQuotaInBytes() == 1024 * 1024 * TB ||
Review comment:
There is one risk here, if user set the quota to 1EB specifically, we
cann't reset it to -1 on next OM restart.
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##
@@ -3540,6 +3546,43 @@ public boolean getEnableFileSystemPaths() {
OZONE_OM_ENABLE_FILESYSTEM_PATHS_DEFAULT);
}
+ /**
+ * Initialize volume's quota, which resolves the problem that the old quota
+ * initial value was not set correctly. This method will be removed in a
+ * later version.
+ * @throws IOException
+ */
+ private void initializeOldVolumeQuota() throws IOException {
+
+TableIterator> iterator = metadataManager.getVolumeTable().iterator();
+while(iterator.hasNext()) {
+ OmVolumeArgs omVolumeArgs = iterator.next().getValue();
+ String omVolumeKey = metadataManager.getVolumeKey(
+ omVolumeArgs.getVolume());
+ // Previously, the volume quota was created to default to 1EB. To change
+ // to -1, default does not enable quota.
+ if (omVolumeArgs.getQuotaInBytes() == 1024 * 1024 * TB ||
+ omVolumeArgs.getQuotaInBytes() == OzoneConsts.MAX_QUOTA_IN_BYTES) {
Review comment:
Why omVolumeArgs.getQuotaInBytes() == OzoneConsts.MAX_QUOTA_IN_BYTES
check is required here?
This is an automated message from the Apache Git Service.
To respond to the message,
[GitHub] [hadoop-ozone] ChenSammi merged pull request #1233: HDDS-3725. Ozone sh volume client support quota option.
ChenSammi merged pull request #1233: URL: https://github.com/apache/hadoop-ozone/pull/1233 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on pull request #1233: HDDS-3725. Ozone sh volume client support quota option.
ChenSammi commented on pull request #1233: URL: https://github.com/apache/hadoop-ozone/pull/1233#issuecomment-687007534 LGTM +1. Thanks @captainzmc for the contribution and @cxorm for review the patch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-4186) Adjust RetryPolicy of SCMConnectionManager for SCM/Recon
[
https://issues.apache.org/jira/browse/HDDS-4186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glen Geng resolved HDDS-4186.
-
Resolution: Fixed
> Adjust RetryPolicy of SCMConnectionManager for SCM/Recon
>
>
> Key: HDDS-4186
> URL: https://issues.apache.org/jira/browse/HDDS-4186
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Reporter: Glen Geng
>Assignee: Glen Geng
>Priority: Critical
> Labels: pull-request-available
>
> *The problem is:*
> If setup one Recon and one SCM, then shutdown the Recon server, all Datanodes
> will be stale/dead very soon at SCM side.
>
> *The root cause is:*
> Current RetryPolicy of Datanode for SCM is retryForeverWithFixedSleep:
> {code:java}
> RetryPolicy retryPolicy =
> RetryPolicies.retryForeverWithFixedSleep(
> 1000, TimeUnit.MILLISECONDS);
> StorageContainerDatanodeProtocolPB rpcProxy = RPC.getProtocolProxy(
> StorageContainerDatanodeProtocolPB.class, version,
> address, UserGroupInformation.getCurrentUser(), hadoopConfig,
> NetUtils.getDefaultSocketFactory(hadoopConfig), getRpcTimeout(),
> retryPolicy).getProxy();{code}
> that for Recon is retryUpToMaximumCountWithFixedSleep:
> {code:java}
> RetryPolicy retryPolicy =
> RetryPolicies.retryUpToMaximumCountWithFixedSleep(10,
> 6, TimeUnit.MILLISECONDS);
> ReconDatanodeProtocolPB rpcProxy = RPC.getProtocolProxy(
> ReconDatanodeProtocolPB.class, version,
> address, UserGroupInformation.getCurrentUser(), hadoopConfig,
> NetUtils.getDefaultSocketFactory(hadoopConfig), getRpcTimeout(),
> retryPolicy).getProxy();
> {code}
>
> The executorService in DatanodeStateMachine is
> Executors.newFixedThreadPool(...), whose default pool size is 2, one for
> Recon, another for SCM.
>
> When encounter rpc failure, call() of RegisterEndpointTask,
> VersionEndpointTask, HeartbeatEndpointTask will retry while holding the
> rpcEndpoint.lock(). For example:
> {code:java}
> public EndpointStateMachine.EndPointStates call() throws Exception {
> rpcEndpoint.lock();
> try {
>
> SCMHeartbeatResponseProto reponse = rpcEndpoint.getEndPoint()
> .sendHeartbeat(request);
>
> } finally {
> rpcEndpoint.unlock();
> }
> return rpcEndpoint.getState();
> }
> {code}
>
> If Recon is down, the thread running Recon task will retry due to rpc
> failure, meanwhile holds the lock of EndpointStateMachine for Recon. When
> DatanodeStateMachine schedule the next round of SCM/Recon task, the only left
> thread will be assigned to run Recon task, and blocked at waiting for the
> lock of EndpointStateMachine for Recon.
> {code:java}
> public EndpointStateMachine.EndPointStates call() throws Exception {
> rpcEndpoint.lock();
> ...{code}
>
> *The solution is:*
> Since DatanodeStateMachine will periodically schedule SCM/Recon tasks, we may
> adjust RetryPolicy so that won't retry for longer that 1min.
>
> *The change has no side effect:*
> 1) VersionEndpointTask.call() is fine
> 2) RegisterEndpointTask.call() will query containerReport, nodeReport,
> pipelineReports from OzoneContainer, which is fine.
> 3) HeartbeatEndpointTask.call() will putBackReports(), which is fine.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4186) Adjust RetryPolicy of SCMConnectionManager for SCM/Recon
[
https://issues.apache.org/jira/browse/HDDS-4186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sammi Chen updated HDDS-4186:
-
Priority: Critical (was: Blocker)
> Adjust RetryPolicy of SCMConnectionManager for SCM/Recon
>
>
> Key: HDDS-4186
> URL: https://issues.apache.org/jira/browse/HDDS-4186
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Reporter: Glen Geng
>Assignee: Glen Geng
>Priority: Critical
> Labels: pull-request-available
>
> *The problem is:*
> If setup one Recon and one SCM, then shutdown the Recon server, all Datanodes
> will be stale/dead very soon at SCM side.
>
> *The root cause is:*
> Current RetryPolicy of Datanode for SCM is retryForeverWithFixedSleep:
> {code:java}
> RetryPolicy retryPolicy =
> RetryPolicies.retryForeverWithFixedSleep(
> 1000, TimeUnit.MILLISECONDS);
> StorageContainerDatanodeProtocolPB rpcProxy = RPC.getProtocolProxy(
> StorageContainerDatanodeProtocolPB.class, version,
> address, UserGroupInformation.getCurrentUser(), hadoopConfig,
> NetUtils.getDefaultSocketFactory(hadoopConfig), getRpcTimeout(),
> retryPolicy).getProxy();{code}
> that for Recon is retryUpToMaximumCountWithFixedSleep:
> {code:java}
> RetryPolicy retryPolicy =
> RetryPolicies.retryUpToMaximumCountWithFixedSleep(10,
> 6, TimeUnit.MILLISECONDS);
> ReconDatanodeProtocolPB rpcProxy = RPC.getProtocolProxy(
> ReconDatanodeProtocolPB.class, version,
> address, UserGroupInformation.getCurrentUser(), hadoopConfig,
> NetUtils.getDefaultSocketFactory(hadoopConfig), getRpcTimeout(),
> retryPolicy).getProxy();
> {code}
>
> The executorService in DatanodeStateMachine is
> Executors.newFixedThreadPool(...), whose default pool size is 2, one for
> Recon, another for SCM.
>
> When encounter rpc failure, call() of RegisterEndpointTask,
> VersionEndpointTask, HeartbeatEndpointTask will retry while holding the
> rpcEndpoint.lock(). For example:
> {code:java}
> public EndpointStateMachine.EndPointStates call() throws Exception {
> rpcEndpoint.lock();
> try {
>
> SCMHeartbeatResponseProto reponse = rpcEndpoint.getEndPoint()
> .sendHeartbeat(request);
>
> } finally {
> rpcEndpoint.unlock();
> }
> return rpcEndpoint.getState();
> }
> {code}
>
> If Recon is down, the thread running Recon task will retry due to rpc
> failure, meanwhile holds the lock of EndpointStateMachine for Recon. When
> DatanodeStateMachine schedule the next round of SCM/Recon task, the only left
> thread will be assigned to run Recon task, and blocked at waiting for the
> lock of EndpointStateMachine for Recon.
> {code:java}
> public EndpointStateMachine.EndPointStates call() throws Exception {
> rpcEndpoint.lock();
> ...{code}
>
> *The solution is:*
> Since DatanodeStateMachine will periodically schedule SCM/Recon tasks, we may
> adjust RetryPolicy so that won't retry for longer that 1min.
>
> *The change has no side effect:*
> 1) VersionEndpointTask.call() is fine
> 2) RegisterEndpointTask.call() will query containerReport, nodeReport,
> pipelineReports from OzoneContainer, which is fine.
> 3) HeartbeatEndpointTask.call() will putBackReports(), which is fine.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi merged pull request #1373: HDDS-4186: Adjust RetryPolicy of SCMConnectionManager for SCM/Recon
ChenSammi merged pull request #1373: URL: https://github.com/apache/hadoop-ozone/pull/1373 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4186) Adjust RetryPolicy of SCMConnectionManager for SCM/Recon
[
https://issues.apache.org/jira/browse/HDDS-4186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sammi Chen updated HDDS-4186:
-
Issue Type: Bug (was: Improvement)
> Adjust RetryPolicy of SCMConnectionManager for SCM/Recon
>
>
> Key: HDDS-4186
> URL: https://issues.apache.org/jira/browse/HDDS-4186
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Reporter: Glen Geng
>Assignee: Glen Geng
>Priority: Blocker
> Labels: pull-request-available
>
> *The problem is:*
> If setup one Recon and one SCM, then shutdown the Recon server, all Datanodes
> will be stale/dead very soon at SCM side.
>
> *The root cause is:*
> Current RetryPolicy of Datanode for SCM is retryForeverWithFixedSleep:
> {code:java}
> RetryPolicy retryPolicy =
> RetryPolicies.retryForeverWithFixedSleep(
> 1000, TimeUnit.MILLISECONDS);
> StorageContainerDatanodeProtocolPB rpcProxy = RPC.getProtocolProxy(
> StorageContainerDatanodeProtocolPB.class, version,
> address, UserGroupInformation.getCurrentUser(), hadoopConfig,
> NetUtils.getDefaultSocketFactory(hadoopConfig), getRpcTimeout(),
> retryPolicy).getProxy();{code}
> that for Recon is retryUpToMaximumCountWithFixedSleep:
> {code:java}
> RetryPolicy retryPolicy =
> RetryPolicies.retryUpToMaximumCountWithFixedSleep(10,
> 6, TimeUnit.MILLISECONDS);
> ReconDatanodeProtocolPB rpcProxy = RPC.getProtocolProxy(
> ReconDatanodeProtocolPB.class, version,
> address, UserGroupInformation.getCurrentUser(), hadoopConfig,
> NetUtils.getDefaultSocketFactory(hadoopConfig), getRpcTimeout(),
> retryPolicy).getProxy();
> {code}
>
> The executorService in DatanodeStateMachine is
> Executors.newFixedThreadPool(...), whose default pool size is 2, one for
> Recon, another for SCM.
>
> When encounter rpc failure, call() of RegisterEndpointTask,
> VersionEndpointTask, HeartbeatEndpointTask will retry while holding the
> rpcEndpoint.lock(). For example:
> {code:java}
> public EndpointStateMachine.EndPointStates call() throws Exception {
> rpcEndpoint.lock();
> try {
>
> SCMHeartbeatResponseProto reponse = rpcEndpoint.getEndPoint()
> .sendHeartbeat(request);
>
> } finally {
> rpcEndpoint.unlock();
> }
> return rpcEndpoint.getState();
> }
> {code}
>
> If Recon is down, the thread running Recon task will retry due to rpc
> failure, meanwhile holds the lock of EndpointStateMachine for Recon. When
> DatanodeStateMachine schedule the next round of SCM/Recon task, the only left
> thread will be assigned to run Recon task, and blocked at waiting for the
> lock of EndpointStateMachine for Recon.
> {code:java}
> public EndpointStateMachine.EndPointStates call() throws Exception {
> rpcEndpoint.lock();
> ...{code}
>
> *The solution is:*
> Since DatanodeStateMachine will periodically schedule SCM/Recon tasks, we may
> adjust RetryPolicy so that won't retry for longer that 1min.
>
> *The change has no side effect:*
> 1) VersionEndpointTask.call() is fine
> 2) RegisterEndpointTask.call() will query containerReport, nodeReport,
> pipelineReports from OzoneContainer, which is fine.
> 3) HeartbeatEndpointTask.call() will putBackReports(), which is fine.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on pull request #1373: HDDS-4186: Adjust RetryPolicy of SCMConnectionManager for SCM/Recon
ChenSammi commented on pull request #1373: URL: https://github.com/apache/hadoop-ozone/pull/1373#issuecomment-686961310 The patch LGTM. +1. Thanks @GlenGeng to fix it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4205) Fix or disable coverage upload to codecov (pull requests)
[
https://issues.apache.org/jira/browse/HDDS-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-4205:
-
Labels: pull-request-available (was: )
> Fix or disable coverage upload to codecov (pull requests)
> -
>
> Key: HDDS-4205
> URL: https://issues.apache.org/jira/browse/HDDS-4205
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Marton Elek
>Assignee: Marton Elek
>Priority: Major
> Labels: pull-request-available
>
> I would like to start a conversation about disabling the coverage data
> uploading to codecov
> 1. It seems to be unreliable, HTTP 400 is a very common answer. I checked
> and the @v1 is the latest version from the action
> {code}
> -> Pinging Codecov
> https://codecov.io/upload/v4?package=bash-20200825-997b141=secret=HDDS-4193=361ddccd0666a8bd24240cda2652a903d4b69013=237476928_url=http%3A%2F%2Fgithub.com%2Fapache%2Fhadoop-ozone%2Factions%2Fruns%2F237476928=codecov-umbrella==apache%2Fhadoop-ozone=github-actions==1384=_args=f,n,F,Z
> HTTP 400
> {code}
> 2. The reported coverage data is meaningless as there is no good mechanism
> to use the latest stable build as baseline.
> I think we should disable codecov (sonar is still available) until we will
> have a solution for these issue.
> [~adoroszlai], [~vivekratnavel]: What do you think?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek opened a new pull request #1394: HDDS-4205. Make coverage upload to codecov optional
elek opened a new pull request #1394: URL: https://github.com/apache/hadoop-ozone/pull/1394 ## What changes were proposed in this pull request? Motivation is discussed on the Apache jira: 1. We make codecov upload optional (accept failures) 2. And execute it only for the master branch ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-4205 ## How was this patch tested? As modification explicit says that it will be executed on non-apache repositories: it's not easy to test. But it's a small change and copied the same condition from the Sonar upload. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-4201) Improve the performance of OmKeyLocationInfoGroup
[ https://issues.apache.org/jira/browse/HDDS-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Doroszlai resolved HDDS-4201. Fix Version/s: 1.1.0 Resolution: Implemented > Improve the performance of OmKeyLocationInfoGroup > - > > Key: HDDS-4201 > URL: https://issues.apache.org/jira/browse/HDDS-4201 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: runzhiwang >Assignee: runzhiwang >Priority: Major > Labels: pull-request-available > Fix For: 1.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai merged pull request #1381: HDDS-4201. Improve the performance of OmKeyLocationInfoGroup
adoroszlai merged pull request #1381: URL: https://github.com/apache/hadoop-ozone/pull/1381 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4201) Improve the performance of OmKeyLocationInfoGroup
[ https://issues.apache.org/jira/browse/HDDS-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Doroszlai updated HDDS-4201: --- Labels: (was: pull-request-available) > Improve the performance of OmKeyLocationInfoGroup > - > > Key: HDDS-4201 > URL: https://issues.apache.org/jira/browse/HDDS-4201 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: runzhiwang >Assignee: runzhiwang >Priority: Major > Fix For: 1.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc commented on pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
captainzmc commented on pull request #1337: URL: https://github.com/apache/hadoop-ozone/pull/1337#issuecomment-686945316 Hi @ChenSammi @adoroszlai , I had added the initialization and change the old volume quota default value to -1 when OM starts. Could you help review this PR again? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc commented on pull request #1233: HDDS-3725. Ozone sh volume client support quota option.
captainzmc commented on pull request #1233: URL: https://github.com/apache/hadoop-ozone/pull/1233#issuecomment-686931995 Thanks for @ChenSammi 's review, the issues has been fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc commented on a change in pull request #1233: HDDS-3725. Ozone sh volume client support quota option.
captainzmc commented on a change in pull request #1233:
URL: https://github.com/apache/hadoop-ozone/pull/1233#discussion_r482918376
##
File path:
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/volume/CreateVolumeHandler.java
##
@@ -40,10 +41,13 @@
description = "Owner of the volume")
private String ownerName;
+ @Option(names = {"--spaceQuota", "-s"},
+ description = "Quota in bytes of the newly created volume (eg. 1GB)")
+ private String quotaInBytes;
+
@Option(names = {"--quota", "-q"},
Review comment:
This command follows the shell of volume (ozone sh volume
update/create), so use bucketQuota feels unclear, may be volume's quota of
bucket number, or misinterpreted as bucket's quota. So, Here, the "quota" is
named similar to HDFS, and more detailed description added.
More detailed naming rules can be found in the [design
documentation](https://issues.apache.org/jira/secure/attachment/13010452/Ozone%20Quota%20Design.pdf).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
