[jira] [Created] (HDFS-15897) SCM HA should be disabled in secure cluster
Bharat Viswanadham created HDFS-15897: - Summary: SCM HA should be disabled in secure cluster Key: HDFS-15897 URL: https://issues.apache.org/jira/browse/HDFS-15897 Project: Hadoop HDFS Issue Type: Task Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham SCM HA security work is still in progress. [~elek] Brought up the point that until before merge of SCM HA branch we should add safeguard check to fail bringing up the cluster -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-15165) In Du missed calling getAttributesProvider
Bharat Viswanadham created HDFS-15165: - Summary: In Du missed calling getAttributesProvider Key: HDFS-15165 URL: https://issues.apache.org/jira/browse/HDFS-15165 Project: Hadoop HDFS Issue Type: Bug Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham HDFS-12130 has changed the behavior of DU. During that change to getInodeAttributes, it missed calling getAttributesProvider().getAttributes when it is configured. Because of this, when sentry is configured for hdfs path, and attributeProvider class is set. We missed calling this, and AclFeature from Sentry was missing. Because of this when DU command is run on a sentry managed hdfs path, we are seeing AccessControlException. This Jira is to fix this issue. {code:java} dfs.namenode.inode.attributes.provider.class org.apache.sentry.hdfs.SentryINodeAttributesProvider {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2536) Add ozone.om.internal.service.id to OM HA configuration
[ https://issues.apache.org/jira/browse/HDDS-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2536. -- Fix Version/s: 0.5.0 Resolution: Fixed > Add ozone.om.internal.service.id to OM HA configuration > --- > > Key: HDDS-2536 > URL: https://issues.apache.org/jira/browse/HDDS-2536 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > This Jira is to add ozone.om.internal.serviceid to let OM knows it belong to > a particular service. > > As now we have ozone.om.service.ids -≥ where we can define all service id's > in a cluster.(This can happen if the same config is shared across the cluster) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2594) S3 RangeReads failing with NumberFormatException
Bharat Viswanadham created HDDS-2594: Summary: S3 RangeReads failing with NumberFormatException Key: HDDS-2594 URL: https://issues.apache.org/jira/browse/HDDS-2594 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham {code:java} 2019-11-20 15:32:04,684 WARN org.eclipse.jetty.servlet.ServletHandler: javax.servlet.ServletException: java.lang.NumberFormatException: For input string: "3977248768" at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:432) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:539) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2241) Optimize the refresh pipeline logic used by KeyManagerImpl to obtain the pipelines for a key
[ https://issues.apache.org/jira/browse/HDDS-2241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2241. -- Fix Version/s: 0.5.0 Resolution: Fixed > Optimize the refresh pipeline logic used by KeyManagerImpl to obtain the > pipelines for a key > > > Key: HDDS-2241 > URL: https://issues.apache.org/jira/browse/HDDS-2241 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Currently, while looking up a key, the Ozone Manager gets the pipeline > information from SCM through an RPC for every block in the key. For large > files > 1GB, we may end up making a lot of RPC calls for this. This can be > optimized in a couple of ways > * We can implement a batch getContainerWithPipeline API in SCM using which we > can get the pipeline info locations for all the blocks for a file. To keep > the number of containers passed in to SCM in a single call, we can have a > fixed container batch size on the OM side. _Here, Number of calls = 1 (or k > depending on batch size)_ > * Instead, a simpler change would be to have a map (method local) of > ContainerID -> Pipeline that we get from SCM so that we don't need to make > repeated calls to SCM for the same containerID for a key. _Here, Number of > calls = Number of unique containerIDs_ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2247) Delete FileEncryptionInfo from KeyInfo when a Key is deleted
[ https://issues.apache.org/jira/browse/HDDS-2247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2247. -- Fix Version/s: 0.5.0 Resolution: Fixed > Delete FileEncryptionInfo from KeyInfo when a Key is deleted > > > Key: HDDS-2247 > URL: https://issues.apache.org/jira/browse/HDDS-2247 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Dinesh Chitlangia >Assignee: Dinesh Chitlangia >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > As part of HDDS-2174 we are deleting GDPR Encryption Key on delete file > operation. > However, if KMS is enabled, we are skipping GDPR Encryption Key approach when > writing file in a GDPR enforced Bucket. > {code:java} > final FileEncryptionInfo feInfo = keyOutputStream.getFileEncryptionInfo(); > if (feInfo != null) { > KeyProvider.KeyVersion decrypted = getDEK(feInfo); > final CryptoOutputStream cryptoOut = > new CryptoOutputStream(keyOutputStream, > OzoneKMSUtil.getCryptoCodec(conf, feInfo), > decrypted.getMaterial(), feInfo.getIV()); > return new OzoneOutputStream(cryptoOut); > } else { > try{ > GDPRSymmetricKey gk; > Map openKeyMetadata = > openKey.getKeyInfo().getMetadata(); > if(Boolean.valueOf(openKeyMetadata.get(OzoneConsts.GDPR_FLAG))){ > gk = new GDPRSymmetricKey( > openKeyMetadata.get(OzoneConsts.GDPR_SECRET), > openKeyMetadata.get(OzoneConsts.GDPR_ALGORITHM) > ); > gk.getCipher().init(Cipher.ENCRYPT_MODE, gk.getSecretKey()); > return new OzoneOutputStream( > new CipherOutputStream(keyOutputStream, gk.getCipher())); > } > }catch (Exception ex){ > throw new IOException(ex); > } > {code} > In such scenario, when KMS is enabled & GDPR enforced on a bucket, if user > deletes a file, we should delete the {{FileEncryptionInfo}} from KeyInfo, > before moving it to deletedTable, else we cannot guarantee Right to Erasure. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2581) Make OM Ha config to use Java Configs
Bharat Viswanadham created HDDS-2581: Summary: Make OM Ha config to use Java Configs Key: HDDS-2581 URL: https://issues.apache.org/jira/browse/HDDS-2581 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham This Jira is created based on the comments from [~aengineer] during HDDS-2536 review. Can we please use the Java Configs instead of this old-style config to add a config? This Jira it to make all HA OM config to the new style (Java config based approach) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2536) Add ozone.om.internal.service.id to OM HA configuration
Bharat Viswanadham created HDDS-2536: Summary: Add ozone.om.internal.service.id to OM HA configuration Key: HDDS-2536 URL: https://issues.apache.org/jira/browse/HDDS-2536 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham This Jira is to add ozone.om.internal.serviceid to let OM knows it belong to a particular service. As now we have ozone.om.service.ids -≥ where we can define all service id's in a cluster.(This can happen if the same config is shared across the cluster) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2461) Logging by ChunkUtils is misleading
[ https://issues.apache.org/jira/browse/HDDS-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2461. -- Fix Version/s: 0.5.0 Resolution: Fixed > Logging by ChunkUtils is misleading > --- > > Key: HDDS-2461 > URL: https://issues.apache.org/jira/browse/HDDS-2461 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Marton Elek >Assignee: Marton Elek >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > During a k8s based test I found a lot of log message like: > {code:java} > 2019-11-12 14:27:13 WARN ChunkManagerImpl:209 - Duplicate write chunk > request. Chunk overwrite without explicit request. > ChunkInfo{chunkName='A9UrLxiEUN_testdata_chunk_4465025, offset=0, len=1024} > {code} > I was very surprised as at ChunkManagerImpl:209 there was no similar lines. > It turned out that it's logged by ChunkUtils but it's used the logger of > ChunkManagerImpl. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2513) Remove this unused "COMPONENT" private field.
[ https://issues.apache.org/jira/browse/HDDS-2513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2513. -- Fix Version/s: 0.5.0 Resolution: Fixed > Remove this unused "COMPONENT" private field. > - > > Key: HDDS-2513 > URL: https://issues.apache.org/jira/browse/HDDS-2513 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Abhishek Purohit >Assignee: Abhishek Purohit >Priority: Minor > Labels: pull-request-available, sonar > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Remove this unused "COMPONENT" private field in class > XceiverClientGrpc > [https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md_AGKcVY8lQ4ZsWG=false] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2502) Close ScmClient in RatisInsight
[ https://issues.apache.org/jira/browse/HDDS-2502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2502. -- Fix Version/s: 0.5.0 Resolution: Fixed > Close ScmClient in RatisInsight > --- > > Key: HDDS-2502 > URL: https://issues.apache.org/jira/browse/HDDS-2502 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Attila Doroszlai >Assignee: Siddharth Wagle >Priority: Major > Labels: pull-request-available, sonar > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > {{ScmClient}} in {{RatisInsight}} should be closed after use. > https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-mYKcVY8lQ4Zr_s=AW5md-mYKcVY8lQ4Zr_s > Also two other minor issues reported in the same file: > https://sonarcloud.io/project/issues?fileUuids=AW5md-HeKcVY8lQ4ZrXL=hadoop-ozone=false -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2507) Remove the hard-coded exclusion of TestMiniChaosOzoneCluster
[ https://issues.apache.org/jira/browse/HDDS-2507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2507. -- Fix Version/s: 0.5.0 Resolution: Fixed > Remove the hard-coded exclusion of TestMiniChaosOzoneCluster > > > Key: HDDS-2507 > URL: https://issues.apache.org/jira/browse/HDDS-2507 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Marton Elek >Assignee: Marton Elek >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > We excluded the execution of TestMiniChaosOzoneCluster from the > hadoop-ozone/dev-support/checks/integration.sh because it was not stable > enough. > Unfortunately this exclusion makes it impossible to use custom exclusion > lists (-Dsurefire.excludesFile=) as excludesFile can't be used if > -Dtest=!... is already used. > I propose to remove this exclusion to make it possible to use different > exclusion for different runs (pr check, daily, etc.) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2511) Fix Sonar issues in OzoneManagerServiceProviderImpl
[ https://issues.apache.org/jira/browse/HDDS-2511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2511. -- Resolution: Fixed > Fix Sonar issues in OzoneManagerServiceProviderImpl > --- > > Key: HDDS-2511 > URL: https://issues.apache.org/jira/browse/HDDS-2511 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Recon >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available, sonar > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Link to the list of issues : > https://sonarcloud.io/project/issues?fileUuids=AW5md-HdKcVY8lQ4ZrUn=hadoop-ozone=false -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2515) No need to call "toString()" method as formatting and string conversion is done by the Formatter
[ https://issues.apache.org/jira/browse/HDDS-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2515. -- Fix Version/s: 0.5.0 Resolution: Fixed > No need to call "toString()" method as formatting and string conversion is > done by the Formatter > > > Key: HDDS-2515 > URL: https://issues.apache.org/jira/browse/HDDS-2515 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Abhishek Purohit >Assignee: Abhishek Purohit >Priority: Major > Labels: pull-request-available, sonar > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > [https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md_AGKcVY8lQ4ZsV4=false] > Class: XceiverClientGrpc > {code:java} > if (LOG.isDebugEnabled()) { LOG.debug("Nodes in pipeline : {}", > pipeline.getNodes().toString()); > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2472) Use try-with-resources while creating FlushOptions in RDBStore.
[ https://issues.apache.org/jira/browse/HDDS-2472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2472. -- Resolution: Fixed > Use try-with-resources while creating FlushOptions in RDBStore. > --- > > Key: HDDS-2472 > URL: https://issues.apache.org/jira/browse/HDDS-2472 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Affects Versions: 0.5.0 >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available, sonar > Fix For: 0.5.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Link to the sonar issue flag - > https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-zwKcVY8lQ4ZsJ4=AW5md-zwKcVY8lQ4ZsJ4. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2494) Sonar - BigDecimal(double) should not be used
[ https://issues.apache.org/jira/browse/HDDS-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2494. -- Fix Version/s: 0.5.0 Resolution: Fixed > Sonar - BigDecimal(double) should not be used > - > > Key: HDDS-2494 > URL: https://issues.apache.org/jira/browse/HDDS-2494 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Matthew Sharp >Assignee: Matthew Sharp >Priority: Minor > Labels: pull-request-available, sonar > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Sonar Issue: > [https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-0AKcVY8lQ4ZsKR=AW5md-0AKcVY8lQ4ZsKR] > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2477) TableCache cleanup issue for OM non-HA
Bharat Viswanadham created HDDS-2477: Summary: TableCache cleanup issue for OM non-HA Key: HDDS-2477 URL: https://issues.apache.org/jira/browse/HDDS-2477 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham In OM in non-HA case, the ratisTransactionLogIndex is generated by OmProtocolServersideTranslatorPB.java. And in OM non-HA validateAndUpdateCache is called from multipleHandler threads. So think of a case where one thread which has an index - 10 has added to doubleBuffer. (0-9 still have not added). DoubleBuffer flush thread flushes and call cleanup. (So, now cleanup will go and cleanup all cache entries with less than 10 epoch) This should not have cleanup those which might have put in to cache later and which are in process of flush to DB. This will cause inconsitency for few OM requests. Example: 4 threads Committing 4 parts. 1st thread - part 1 - ratis Index - 3 2nd thread - part 2 - ratis index - 2 3rd thread - part3 - ratis index - 1 First thread got lock, and put in to doubleBuffer and cache with OmMultipartInfo (with part1). And cleanup is called to cleanup all entries in cache with less than 3. In the mean time 2nd thread and 1st thread put 2,3 parts in to OmMultipartInfo in to Cache and doubleBuffer. But first thread might cleanup those entries, as it is called with index 3 for cleanup. Now when the 4th part upload came -> when it is commit Multipart upload when it gets multipartinfo it get Only part1 in OmMultipartInfo, as the OmMultipartInfo (with 1,2,3 is still in process of committing to DB). So now after 4th part upload is complete in DB and Cache we will have 1,4 parts only. We will miss part2,3 information. So for non-HA case cleanup will be called with list of epochs that need to be cleanedup. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2471) Improve exception message for CompleteMultipartUpload
Bharat Viswanadham created HDDS-2471: Summary: Improve exception message for CompleteMultipartUpload Key: HDDS-2471 URL: https://issues.apache.org/jira/browse/HDDS-2471 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham When InvalidPart error occurs, the exception message does not have any information about partName and partNumber, it will be good to have this information. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2470) Add partName, partNumber for CommitMultipartUpload
Bharat Viswanadham created HDDS-2470: Summary: Add partName, partNumber for CommitMultipartUpload Key: HDDS-2470 URL: https://issues.apache.org/jira/browse/HDDS-2470 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham Right now when complete Multipart Upload is not printing partName and partNumber into the audit log. 2019-11-13 15:14:10,191 | INFO | OMAudit | user=root | ip=9.134.50.210 | op=COMMIT_MULTIPART_UPLOAD_PARTKEY {volume=s325d55ad283aa400af464c76d713c07ad, bucket=ozone-test, key=plc_1570850798896_2991, dataSize=5242880, replicationType=RATIS, replicationFactor=ONE, keyLocationInfo=[blockID { containerBlockID { containerID: 2 localID: 103129366531867089 } blockCommitSequenceId: 4978 } offset: 0 length: 5242880 createVersion: 0 pipeline { leaderID: "" members { uuid: "5d03aed5-cfb3-4689-b168-0c9a94316551" ipAddress: "9.134.51.232" hostName: "9.134.51.232" ports { name: "RATIS" value: 9858 } ports { name: "STANDALONE" value: 9859 } networkName: "5d03aed5-cfb3-4689-b168-0c9a94316551" networkLocation: "/default-rack" } members { uuid: "a71462ae-7865-4ed5-b84e-60616df60a0d" ipAddress: "9.134.51.25" hostName: "9.134.51.25" ports { name: "RATIS" value: 9858 } ports { name: "STANDALONE" value: 9859 } networkName: "a71462ae-7865-4ed5-b84e-60616df60a0d" networkLocation: "/default-rack" } members { uuid: "79bf7bdf-ed29-49d4-bf7c-e88fdbd2ce03" ipAddress: "9.134.51.215" hostName: "9.134.51.215" ports { name: "RATIS" value: 9858 } ports { name: "STANDALONE" value: 9859 } networkName: "79bf7bdf-ed29-49d4-bf7c-e88fdbd2ce03" networkLocation: "/default-rack" } state: PIPELINE_OPEN type: RATIS factor: THREE id { id: "ec6b06c5-193f-4c30-879b-5a12284dc4f8" } } ]} | ret=SUCCESS | -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2465) S3 Multipart upload failing
Bharat Viswanadham created HDDS-2465: Summary: S3 Multipart upload failing Key: HDDS-2465 URL: https://issues.apache.org/jira/browse/HDDS-2465 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham When I run attached java program, facing below error, during completeMultipartUpload. {code:java} ERROR StatusLogger No Log4j 2 configuration file found. Using default configuration (logging only errors to the console), or user programmatically provided configurations. Set system property 'log4j2.debug' to show Log4j 2 internal initialization logging. See https://logging.apache.org/log4j/2.x/manual/configuration.html for instructions on how to configure Log4j 2ERROR StatusLogger No Log4j 2 configuration file found. Using default configuration (logging only errors to the console), or user programmatically provided configurations. Set system property 'log4j2.debug' to show Log4j 2 internal initialization logging. See https://logging.apache.org/log4j/2.x/manual/configuration.html for instructions on how to configure Log4j 2Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: c7b87393-955b-4c93-85f6-b02945e293ca; S3 Extended Request ID: 7tnVbqgc4bgb), S3 Extended Request ID: 7tnVbqgc4bgb at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4921) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4867) at com.amazonaws.services.s3.AmazonS3Client.completeMultipartUpload(AmazonS3Client.java:3464) at org.apache.hadoop.ozone.freon.MPU.main(MPU.java:96){code} When I debug it is not the request is not been received by S3Gateway, and I don't see any trace of this in audit log. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2453) Add Freon tests for S3Bucket/MPU Keys
Bharat Viswanadham created HDDS-2453: Summary: Add Freon tests for S3Bucket/MPU Keys Key: HDDS-2453 URL: https://issues.apache.org/jira/browse/HDDS-2453 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham This Jira is to create freon tests for # S3Bucket creation. # S3 MPU Key uploads. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2410) Ozoneperf docker cluster should use privileged containers
[ https://issues.apache.org/jira/browse/HDDS-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2410. -- Fix Version/s: 0.5.0 Resolution: Fixed > Ozoneperf docker cluster should use privileged containers > - > > Key: HDDS-2410 > URL: https://issues.apache.org/jira/browse/HDDS-2410 > Project: Hadoop Distributed Data Store > Issue Type: Task >Reporter: Marton Elek >Assignee: Marton Elek >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > The profiler > [servlet|https://github.com/elek/hadoop-ozone/blob/master/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/ProfileServlet.java] > (which helps to run java profiler in the background and publishes the result > on the web interface) requires privileged docker containers. > > This flag is missing from the ozoneperf docker-compose cluster (which is > designed to run performance tests). > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2399) Update mailing list information in CONTRIBUTION and README files
[ https://issues.apache.org/jira/browse/HDDS-2399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2399. -- Fix Version/s: 0.5.0 Resolution: Fixed > Update mailing list information in CONTRIBUTION and README files > > > Key: HDDS-2399 > URL: https://issues.apache.org/jira/browse/HDDS-2399 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Marton Elek >Priority: Major > Labels: newbie, pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > We have new mailing lists: > [ozone-...@hadoop.apache.org|mailto:ozone-...@hadoop.apache.org] > [ozone-iss...@hadoop.apache.org|mailto:ozone-iss...@hadoop.apache.org] > [ozone-comm...@hadoop.apache.org|mailto:ozone-comm...@hadoop.apache.org] > > We need to update CONTRIBUTION.md and README.md to use ozone-dev instead of > hdfs-dev (optionally we can mention the issues/commits lists, but only in > CONTRIBUTION.md) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2395) Handle Ozone S3 completeMPU to match with aws s3 behavior.
[ https://issues.apache.org/jira/browse/HDDS-2395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2395. -- Fix Version/s: 0.5.0 Resolution: Fixed > Handle Ozone S3 completeMPU to match with aws s3 behavior. > -- > > Key: HDDS-2395 > URL: https://issues.apache.org/jira/browse/HDDS-2395 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > # When uploaded 2 parts, and when complete upload 1 part no error > # During complete multipart upload name/part number not matching with > uploaded part and part number then InvalidPart error > # When parts are not specified in sorted order InvalidPartOrder > # During complete multipart upload when no uploaded parts, and we specify > some parts then also InvalidPart > # Uploaded parts 1,2,3 and during complete we can do upload 1,3 (No error) > # When part 3 uploaded, complete with part 3 can be done -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2427) Exclude webapps from hadoop-ozone-filesystem-lib-current uber jar
Bharat Viswanadham created HDDS-2427: Summary: Exclude webapps from hadoop-ozone-filesystem-lib-current uber jar Key: HDDS-2427 URL: https://issues.apache.org/jira/browse/HDDS-2427 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham This has caused issue for DN UI loading. hadoop-ozone-filesystem-lib-current-xx.jar is in the classpath which accidentally loaded Ozone datanode web application instead of Hadoop datanode application. This leads to the reported error. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2377) Speed up TestOzoneManagerHA#testOMRetryProxy and #testTwoOMNodesDown
[ https://issues.apache.org/jira/browse/HDDS-2377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2377. -- Fix Version/s: 0.5.0 Resolution: Fixed > Speed up TestOzoneManagerHA#testOMRetryProxy and #testTwoOMNodesDown > > > Key: HDDS-2377 > URL: https://issues.apache.org/jira/browse/HDDS-2377 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Marton's comment: > https://github.com/apache/hadoop-ozone/pull/30#pullrequestreview-302465440 > Out of curiosity, I ran entire TestOzoneManagerHA locally. The entire test > class finished in 10m 30s. I discovered {{testOMRetryProxy}} and > {{testTwoOMNodesDown}} are taking the most time (2m and 2m 30s respectively) > to finish. Most time are wasted on retry and wait. We could reasonably reduce > the amount of time on the wait. > As I tested, with the patch, {{testOMRetryProxy}} and {{testTwoOMNodesDown}} > finish in 20 sec each, saving almost 4 min runtime on those two tests alone. > The whole TestOzoneManagerHA test finishes in 5m 44s with the patch. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1643) Send hostName also part of OMRequest
[ https://issues.apache.org/jira/browse/HDDS-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-1643. -- Fix Version/s: 0.5.0 Resolution: Fixed > Send hostName also part of OMRequest > > > Key: HDDS-1643 > URL: https://issues.apache.org/jira/browse/HDDS-1643 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: YiSheng Lien >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > This Jira is created based on the comment from [~eyang] on HDDS-1600 jira. > [~bharatviswa] can hostname be used as part of OM request? For running in > docker container, virtual private network address may not be routable or > exposed to outside world. Using IP to identify the source client location may > not be enough. It would be nice to have ability support hostname based > request too. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2064) Add tests for incorrect OM HA config when node ID or RPC address is not configured
[ https://issues.apache.org/jira/browse/HDDS-2064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2064. -- Fix Version/s: 0.5.0 Resolution: Fixed > Add tests for incorrect OM HA config when node ID or RPC address is not > configured > -- > > Key: HDDS-2064 > URL: https://issues.apache.org/jira/browse/HDDS-2064 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > -OM will NPE and crash when `ozone.om.service.ids=id1,id2` is configured but > `ozone.om.nodes.id1` doesn't exist; or `ozone.om.address.id1.omX` doesn't > exist.- > -Root cause:- > -`OzoneManager#loadOMHAConfigs()` didn't check the case where `found == 0`. > This happens when local OM doesn't match any `ozone.om.address.idX.omX` in > the config.- > Due to the refactoring done in HDDS-2162. This fix has been included in that > commit. I will repurpose the jira to add some tests for the HA config. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2359) Seeking randomly in a key with more than 2 blocks of data leads to inconsistent reads
[ https://issues.apache.org/jira/browse/HDDS-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2359. -- Fix Version/s: 0.5.0 Resolution: Fixed > Seeking randomly in a key with more than 2 blocks of data leads to > inconsistent reads > - > > Key: HDDS-2359 > URL: https://issues.apache.org/jira/browse/HDDS-2359 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Istvan Fajth >Assignee: Shashikant Banerjee >Priority: Critical > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > During Hive testing we found the following exception: > {code} > TaskAttempt 3 failed, info=[Error: Error while running task ( failure ) : > attempt_1569246922012_0214_1_03_00_3:java.lang.RuntimeException: > org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: > java.io.IOException: error iterating > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at > com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108) > at > com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41) > at > com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.io.IOException: java.io.IOException: error iterating > at > org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:80) > at > org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:426) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267) > ... 16 more > Caused by: java.io.IOException: java.io.IOException: error iterating > at > org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121) > at > org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77) > at > org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:366) > at > org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79) > at > org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33) > at > org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116) > at > org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151) > at > org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116) > at > org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68) > ... 18 more > Caused by: java.io.IOException: error iterating > at > org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.next(VectorizedOrcAcidRowBatchReader.java:835) > at > org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.next(VectorizedOrcAcidRowBatchReader.java:74) > at > org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:361) > ... 24 more > Caused by: java.io.IOException: Error reading file: > o3fs://hive.warehouse.vc0136.halxg.cloudera.com:9862/data/inventory/delta_001_001_/bucket_0 > at >
[jira] [Resolved] (HDDS-2255) Improve Acl Handler Messages
[ https://issues.apache.org/jira/browse/HDDS-2255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2255. -- Fix Version/s: 0.5.0 Resolution: Fixed > Improve Acl Handler Messages > > > Key: HDDS-2255 > URL: https://issues.apache.org/jira/browse/HDDS-2255 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: om >Reporter: Hanisha Koneru >Assignee: YiSheng Lien >Priority: Minor > Labels: newbie, pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > In Add/Remove/Set Acl Key/Bucket/Volume Handlers, we print a message about > whether the operation was successful or not. If we are trying to add an ACL > which is already existing, we convey the message that the operation failed. > It would be better if the message conveyed more clearly why the operation > failed i.e. the ACL already exists. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2398) Remove usage of LogUtils class from ratis-common
[ https://issues.apache.org/jira/browse/HDDS-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2398. -- Fix Version/s: 0.5.0 Resolution: Fixed > Remove usage of LogUtils class from ratis-common > > > Key: HDDS-2398 > URL: https://issues.apache.org/jira/browse/HDDS-2398 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > MiniOzoneChaoasCluster.java for setting log level it uses LogUtils from > ratis-common. But this is removed from LogUtils as part of Ratis-508. > We can avoid depending on ratis for this, and use GenericTestUtils from > hadoop-common test. > LogUtils.setLogLevel(GrpcClientProtocolClient.LOG, Level.WARN); -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2398) Remove usage of LogUtils class from ratis-common
Bharat Viswanadham created HDDS-2398: Summary: Remove usage of LogUtils class from ratis-common Key: HDDS-2398 URL: https://issues.apache.org/jira/browse/HDDS-2398 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham MiniOzoneChaoasCluster.java for setting log level it uses LogUtils from ratis-common. But this is removed from LogUtils as part of Ratis-508. We can avoid depending on ratis for this, and use GenericTestUtils from hadoop-common test. LogUtils.setLogLevel(GrpcClientProtocolClient.LOG, Level.WARN); -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2397) Fix calling cleanup for few missing tables in OM
Bharat Viswanadham created HDDS-2397: Summary: Fix calling cleanup for few missing tables in OM Key: HDDS-2397 URL: https://issues.apache.org/jira/browse/HDDS-2397 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham After DoubleBuffer flushes, we call cleanup cache to cleanup tables cache. For few tables cleanup of cache is missed: # PrefixTable # S3SecretTable # DelegationTable -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2395) Handle completeMPU scenarios to match with aws s3 behavior.
Bharat Viswanadham created HDDS-2395: Summary: Handle completeMPU scenarios to match with aws s3 behavior. Key: HDDS-2395 URL: https://issues.apache.org/jira/browse/HDDS-2395 Project: Hadoop Distributed Data Store Issue Type: Task Reporter: Bharat Viswanadham # When uploaded 2 parts, and when complete upload 1 part no error # During complete multipart upload name/part number not matching with uploaded part and part number then InvalidPart error # When parts are not specified in sorted order InvalidPartOrder # During complete multipart upload when no uploaded parts, and we specify some parts then also InvalidPart # Uploaded parts 1,2,3 and during complete we can do upload 1,3 (No error) # When part 3 uploaded, complete with part 3 can be done -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2355) Om double buffer flush termination with rocksdb error
[ https://issues.apache.org/jira/browse/HDDS-2355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2355. -- Resolution: Fixed > Om double buffer flush termination with rocksdb error > - > > Key: HDDS-2355 > URL: https://issues.apache.org/jira/browse/HDDS-2355 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Aravindan Vijayan >Priority: Blocker > Fix For: 0.5.0 > > > om_1 |java.io.IOException: Unable to write the batch. > om_1 | at > [org.apache.hadoop.hdds.utils.db.RDBBatchOperation.commit(RDBBatchOperation.java:48|http://org.apache.hadoop.hdds.utils.db.rdbbatchoperation.commit%28rdbbatchoperation.java:48/]) > om_1 | at > [org.apache.hadoop.hdds.utils.db.RDBStore.commitBatchOperation(RDBStore.java:240|http://org.apache.hadoop.hdds.utils.db.rdbstore.commitbatchoperation%28rdbstore.java:240/]) > om_1 |at > org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:146) > om_1 |at java.base/java.lang.Thread.run(Thread.java:834) > om_1 |Caused by: org.rocksdb.RocksDBException: > WritePrepared/WriteUnprepared txn tag when write_after_commit_ is enabled (in > default WriteCommitted mode). If it is not due to corruption, the WAL must be > emptied before changing the WritePolicy. > om_1 |at org.rocksdb.RocksDB.write0(Native Method) > om_1 |at org.rocksdb.RocksDB.write(RocksDB.java:1421) > om_1 | at > [org.apache.hadoop.hdds.utils.db.RDBBatchOperation.commit(RDBBatchOperation.java:46|http://org.apache.hadoop.hdds.utils.db.rdbbatchoperation.commit%28rdbbatchoperation.java:46/]) > > In few of my test run's i see this error and OM is terminated. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2379) OM terminates with RocksDB error while continuously writing keys.
[ https://issues.apache.org/jira/browse/HDDS-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2379. -- Resolution: Fixed > OM terminates with RocksDB error while continuously writing keys. > - > > Key: HDDS-2379 > URL: https://issues.apache.org/jira/browse/HDDS-2379 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Bharat Viswanadham >Priority: Critical > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Exception trace after writing around 800,000 keys. > {code} > 2019-10-29 11:15:15,131 ERROR > org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer: Terminating with > exit status 1: During flush to DB encountered err > or in OMDoubleBuffer flush thread OMDoubleBufferFlushThread > java.io.IOException: Unable to write the batch. > at > org.apache.hadoop.hdds.utils.db.RDBBatchOperation.commit(RDBBatchOperation.java:48) > at > org.apache.hadoop.hdds.utils.db.RDBStore.commitBatchOperation(RDBStore.java:240) > at > org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:146) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.rocksdb.RocksDBException: unknown WriteBatch tag > at org.rocksdb.RocksDB.write0(Native Method) > at org.rocksdb.RocksDB.write(RocksDB.java:1421) > at > org.apache.hadoop.hdds.utils.db.RDBBatchOperation.commit(RDBBatchOperation.java:46) > ... 3 more > {code} > Assigning to [~bharat] since he has already started work on this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2381) In ExcludeList, add if not exist only
Bharat Viswanadham created HDDS-2381: Summary: In ExcludeList, add if not exist only Key: HDDS-2381 URL: https://issues.apache.org/jira/browse/HDDS-2381 Project: Hadoop Distributed Data Store Issue Type: Task Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham Created based on comment from [~chinseone] in HDDS-2356 https://issues.apache.org/jira/browse/HDDS-2356?focusedCommentId=16960796=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16960796 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2345) Add a UT for newly added clone() in OmBucketInfo
[ https://issues.apache.org/jira/browse/HDDS-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2345. -- Fix Version/s: 0.5.0 Resolution: Fixed > Add a UT for newly added clone() in OmBucketInfo > > > Key: HDDS-2345 > URL: https://issues.apache.org/jira/browse/HDDS-2345 > Project: Hadoop Distributed Data Store > Issue Type: Task >Reporter: Bharat Viswanadham >Assignee: YiSheng Lien >Priority: Major > Labels: newbie, pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Add a UT for newly added clone() method in OMBucketInfo as part of HDDS-2333. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2361) Ozone Manager init & start command prints out unnecessary line in the beginning.
[ https://issues.apache.org/jira/browse/HDDS-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2361. -- Fix Version/s: 0.5.0 Resolution: Fixed > Ozone Manager init & start command prints out unnecessary line in the > beginning. > > > Key: HDDS-2361 > URL: https://issues.apache.org/jira/browse/HDDS-2361 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Aravindan Vijayan >Assignee: YiSheng Lien >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > {code} > [root@avijayan-om-1 ozone-0.5.0-SNAPSHOT]# bin/ozone --daemon start om > Ozone Manager classpath extended by > {code} > We could probably print this line only when extra elements are added to OM > classpath or skip printing this line altogether. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2366) Remove ozone.enabled flag
Bharat Viswanadham created HDDS-2366: Summary: Remove ozone.enabled flag Key: HDDS-2366 URL: https://issues.apache.org/jira/browse/HDDS-2366 Project: Hadoop Distributed Data Store Issue Type: Task Reporter: Bharat Viswanadham Now when ozone is started the start-ozone.sh/stop-ozone.sh script check whether this property is enabled or not to start ozone services. Now, this property and this check can be removed. This was needed when ozone is part of Hadoop, and we don't want to start ozone services by default. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2296) ozoneperf compose cluster shouln't start freon by default
[ https://issues.apache.org/jira/browse/HDDS-2296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2296. -- Fix Version/s: 0.5.0 Resolution: Fixed > ozoneperf compose cluster shouln't start freon by default > - > > Key: HDDS-2296 > URL: https://issues.apache.org/jira/browse/HDDS-2296 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: docker >Reporter: Marton Elek >Assignee: Marton Elek >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > During the original creation of the compose/ozoneperf we added an example > freon execution to make it clean how the data can be generated. This freon > process starts all the time when ozoneperf cluster is started (usually I > notice it when my CPU starts to use 100% of the available resources). > Since the creation of this cluster definition we implemented multiple type of > freon tests and it's hard predict which tests should be executed. I propose > to remove the default execution of the random key generation but keep the > opportunity to run any of the tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1015) Cleanup snapshot repository settings
[ https://issues.apache.org/jira/browse/HDDS-1015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-1015. -- Resolution: Won't Fix > Cleanup snapshot repository settings > > > Key: HDDS-1015 > URL: https://issues.apache.org/jira/browse/HDDS-1015 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDDS-1015.00.patch > > > Now we can clean up snapshot repository settings from hadoop-hdds/pom.xml and > hadoop-ozone/pom.xml > As now we have moved our dependencies from Hadoop 3.2.1-SNAPSHOT to 3.2.0 as > part of HDDS-993, we don't require them. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2360) Update Ratis snapshot to d6d58d0
[ https://issues.apache.org/jira/browse/HDDS-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2360. -- Fix Version/s: 0.5.0 Resolution: Fixed > Update Ratis snapshot to d6d58d0 > > > Key: HDDS-2360 > URL: https://issues.apache.org/jira/browse/HDDS-2360 > Project: Hadoop Distributed Data Store > Issue Type: Task > Components: Ozone Client, Ozone Datanode >Reporter: Attila Doroszlai >Assignee: Attila Doroszlai >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Update Ratis dependency version to snapshot > [d6d58d0|https://github.com/apache/incubator-ratis/commit/d6d58d0], to fix > memory issues (RATIS-726, RATIS-728). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2356. -- Resolution: Fixed This will be fixed as part of HDDS-2322. > Multipart upload report errors while writing to ozone Ratis pipeline > > > Key: HDDS-2356 > URL: https://issues.apache.org/jira/browse/HDDS-2356 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Affects Versions: 0.4.1 > Environment: Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM > on a separate VM >Reporter: Li Cheng >Assignee: Bharat Viswanadham >Priority: Blocker > > Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM on a separate VM, say > it's VM0. > I use goofys as a fuse and enable ozone S3 gateway to mount ozone to a path > on VM0, while reading data from VM0 local disk and write to mount path. The > dataset has various sizes of files from 0 byte to GB-level and it has a > number of ~50,000 files. > The writing is slow (1GB for ~10 mins) and it stops after around 4GB. As I > look at hadoop-root-om-VM_50_210_centos.out log, I see OM throwing errors > related with Multipart upload. This error eventually causes the writing to > terminate and OM to be closed. > > 2019-10-24 16:01:59,527 [OMDoubleBufferFlushThread] ERROR - Terminating with > exit status 2: OMDoubleBuffer flush > threadOMDoubleBufferFlushThreadencountered Throwable error > java.util.ConcurrentModificationException > at java.util.TreeMap.forEach(TreeMap.java:1004) > at > org.apache.hadoop.ozone.om.helpers.OmMultipartKeyInfo.getProto(OmMultipartKeyInfo.java:111) > at > org.apache.hadoop.ozone.om.codec.OmMultipartKeyInfoCodec.toPersistedFormat(OmMultipartKeyInfoCodec.java:38) > at > org.apache.hadoop.ozone.om.codec.OmMultipartKeyInfoCodec.toPersistedFormat(OmMultipartKeyInfoCodec.java:31) > at > org.apache.hadoop.hdds.utils.db.CodecRegistry.asRawData(CodecRegistry.java:68) > at > org.apache.hadoop.hdds.utils.db.TypedTable.putWithBatch(TypedTable.java:125) > at > org.apache.hadoop.ozone.om.response.s3.multipart.S3MultipartUploadCommitPartResponse.addToDBBatch(S3MultipartUploadCommitPartResponse.java:112) > at > org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.lambda$flushTransactions$0(OzoneManagerDoubleBuffer.java:137) > at java.util.Iterator.forEachRemaining(Iterator.java:116) > at > org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:135) > at java.lang.Thread.run(Thread.java:745) > 2019-10-24 16:01:59,629 [shutdown-hook-0] INFO - SHUTDOWN_MSG: -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2297) Enable Opentracing for new Freon tests
[ https://issues.apache.org/jira/browse/HDDS-2297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2297. -- Fix Version/s: 0.5.0 Resolution: Fixed > Enable Opentracing for new Freon tests > -- > > Key: HDDS-2297 > URL: https://issues.apache.org/jira/browse/HDDS-2297 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: freon >Reporter: Marton Elek >Assignee: Marton Elek >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > HDDS-2022 introduced new freon tests, but the initial root span of > opentracing is not created before the test execution. We need to enable > opentracing to get better view about the executions of the new freon test. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2355) Om double buffer flush termination with rocksdb error
Bharat Viswanadham created HDDS-2355: Summary: Om double buffer flush termination with rocksdb error Key: HDDS-2355 URL: https://issues.apache.org/jira/browse/HDDS-2355 Project: Hadoop Distributed Data Store Issue Type: Task Reporter: Bharat Viswanadham om_1 |java.io.IOException: Unable to write the batch. om_1 | at [org.apache.hadoop.hdds.utils.db.RDBBatchOperation.commit(RDBBatchOperation.java:48|http://org.apache.hadoop.hdds.utils.db.rdbbatchoperation.commit%28rdbbatchoperation.java:48/]) om_1 | at [org.apache.hadoop.hdds.utils.db.RDBStore.commitBatchOperation(RDBStore.java:240|http://org.apache.hadoop.hdds.utils.db.rdbstore.commitbatchoperation%28rdbstore.java:240/]) om_1 |at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:146) om_1 |at java.base/java.lang.Thread.run(Thread.java:834) om_1 |Caused by: org.rocksdb.RocksDBException: WritePrepared/WriteUnprepared txn tag when write_after_commit_ is enabled (in default WriteCommitted mode). If it is not due to corruption, the WAL must be emptied before changing the WritePolicy. om_1 |at org.rocksdb.RocksDB.write0(Native Method) om_1 |at org.rocksdb.RocksDB.write(RocksDB.java:1421) om_1 | at [org.apache.hadoop.hdds.utils.db.RDBBatchOperation.commit(RDBBatchOperation.java:46|http://org.apache.hadoop.hdds.utils.db.rdbbatchoperation.commit%28rdbbatchoperation.java:46/]) In few of my test run's i see this error and OM is terminated. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2354) SCM log is full of AllocateBlock logs
Bharat Viswanadham created HDDS-2354: Summary: SCM log is full of AllocateBlock logs Key: HDDS-2354 URL: https://issues.apache.org/jira/browse/HDDS-2354 Project: Hadoop Distributed Data Store Issue Type: Task Reporter: Bharat Viswanadham 2019-10-24 03:17:43,087 INFO server.SCMBlockProtocolServer: Allocating 1 blocks of size 268435456, with ExcludeList \{datanodes = [], containerIds = [], pipelineIds = []} scm_1 | 2019-10-24 03:17:43,088 INFO server.SCMBlockProtocolServer: Allocating 1 blocks of size 268435456, with ExcludeList \{datanodes = [], containerIds = [], pipelineIds = []} scm_1 | 2019-10-24 03:17:43,089 INFO server.SCMBlockProtocolServer: Allocating 1 blocks of size 268435456, with ExcludeList \{datanodes = [], containerIds = [], pipelineIds = []} scm_1 | 2019-10-24 03:17:43,093 INFO server.SCMBlockProtocolServer: Allocating 1 blocks of size 268435456, with ExcludeList \{datanodes = [], containerIds = [], pipelineIds = []} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2353) Cleanup old write-path code in OM
Bharat Viswanadham created HDDS-2353: Summary: Cleanup old write-path code in OM Key: HDDS-2353 URL: https://issues.apache.org/jira/browse/HDDS-2353 Project: Hadoop Distributed Data Store Issue Type: Task Reporter: Bharat Viswanadham This Jira is to cleanup old write path code in OM. As the newly added request/response code is also used for non-HA, we can cleanup the old code. And also this integrated code is tested for few days. So, this will be good time to cleanup old code. Cleaning up old code is also causing trouble for some patches fixing write path now they need to update in 2 places.(Because if we change a constructor, then it will require change in 2 places) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2131) Optimize replication type and creation time calculation in S3 MPU list call
[ https://issues.apache.org/jira/browse/HDDS-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2131. -- Fix Version/s: 0.5.0 Resolution: Fixed > Optimize replication type and creation time calculation in S3 MPU list call > --- > > Key: HDDS-2131 > URL: https://issues.apache.org/jira/browse/HDDS-2131 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Marton Elek >Assignee: Siddharth Wagle >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Based on the review from [~bharatviswa]: > {code} > > hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java > metadataManager.getOpenKeyTable(); > OmKeyInfo omKeyInfo = > openKeyTable.get(upload.getDbKey()); > {code} > {quote}Here we are reading openKeyTable only for getting creation time. If we > can have this information in omMultipartKeyInfo, we could avoid DB calls for > openKeyTable. > To do this, We can set creationTime in OmMultipartKeyInfo during > initiateMultipartUpload . In this way, we can get all the required > information from the MultipartKeyInfo table. > And also StorageClass is missing from the returned OmMultipartUpload, as > listMultipartUploads shows StorageClass information. For this, if we can > return replicationType and depending on this value, we can set StorageClass > in the listMultipartUploads Response. > {quote} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2345) Add a UT for newly added clone() in OmBucketInfo
Bharat Viswanadham created HDDS-2345: Summary: Add a UT for newly added clone() in OmBucketInfo Key: HDDS-2345 URL: https://issues.apache.org/jira/browse/HDDS-2345 Project: Hadoop Distributed Data Store Issue Type: Task Reporter: Bharat Viswanadham Add a UT for newly added clone() method in OMBucketInfo as part of HDDS-2333. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2344) CLONE - Add immutable entries in to the DoubleBuffer for Volume requests.
Bharat Viswanadham created HDDS-2344: Summary: CLONE - Add immutable entries in to the DoubleBuffer for Volume requests. Key: HDDS-2344 URL: https://issues.apache.org/jira/browse/HDDS-2344 Project: Hadoop Distributed Data Store Issue Type: Task Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham Fix For: 0.5.0 OMBucketCreateRequest.java L181: omClientResponse = new OMBucketCreateResponse(omBucketInfo, omResponse.build()); We add this to double-buffer, and double-buffer flushThread which is running in the background when picks up, converts to protoBuf and to ByteArray and write to rocksDB tables. So, during this conversion(This conversion will be done without any lock acquire), if any other request changes internal structure(like acls list) of OMBucketInfo we might get ConcurrentModificationException. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2310) Add support to add ozone ranger plugin to Ozone Manager classpath
[ https://issues.apache.org/jira/browse/HDDS-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2310. -- Fix Version/s: 0.5.0 Resolution: Fixed > Add support to add ozone ranger plugin to Ozone Manager classpath > - > > Key: HDDS-2310 > URL: https://issues.apache.org/jira/browse/HDDS-2310 > Project: Hadoop Distributed Data Store > Issue Type: Task > Components: Ozone Manager >Affects Versions: 0.5.0 >Reporter: Vivek Ratnavel Subramanian >Assignee: Vivek Ratnavel Subramanian >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Currently, there is no way to add Ozone Ranger plugin to Ozone Manager > classpath. > We should be able to set an environment variable that will be respected by > ozone and added to Ozone Manager classpath. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2320) Negative value seen for OM NumKeys Metric in JMX.
[ https://issues.apache.org/jira/browse/HDDS-2320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2320. -- Fix Version/s: 0.5.0 Resolution: Fixed > Negative value seen for OM NumKeys Metric in JMX. > - > > Key: HDDS-2320 > URL: https://issues.apache.org/jira/browse/HDDS-2320 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Attachments: Screen Shot 2019-10-17 at 11.31.08 AM.png > > Time Spent: 20m > Remaining Estimate: 0h > > While running teragen/terasort on a cluster and verifying number of keys > created on Ozone Manager, I noticed that the value of NumKeys counter metric > to be a negative value !Screen Shot 2019-10-17 at 11.31.08 AM.png! . -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2340) Update RATIS snapshot version
[ https://issues.apache.org/jira/browse/HDDS-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2340. -- Resolution: Fixed > Update RATIS snapshot version > - > > Key: HDDS-2340 > URL: https://issues.apache.org/jira/browse/HDDS-2340 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: build >Affects Versions: 0.5.0 >Reporter: Siddharth Wagle >Assignee: Siddharth Wagle >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Update RATIS version to incorporate fix that went into RATIS-707 among others. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2343) Add immutable entries in to the DoubleBuffer.
Bharat Viswanadham created HDDS-2343: Summary: Add immutable entries in to the DoubleBuffer. Key: HDDS-2343 URL: https://issues.apache.org/jira/browse/HDDS-2343 Project: Hadoop Distributed Data Store Issue Type: Task Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham OMBucketCreateRequest.java L181: omClientResponse = new OMBucketCreateResponse(omBucketInfo, omResponse.build()); We add this to double-buffer, and double-buffer flushThread which is running in the background when picks up, converts to protoBuf and to ByteArray and write to rocksDB tables. So, during this conversion, if any other request changes internal structure(like acls list) of OMBucketInfo we might get ConcurrentModificationException. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2326) Http server of Freon is not started for new Freon tests
[ https://issues.apache.org/jira/browse/HDDS-2326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2326. -- Fix Version/s: 0.5.0 Resolution: Fixed > Http server of Freon is not started for new Freon tests > --- > > Key: HDDS-2326 > URL: https://issues.apache.org/jira/browse/HDDS-2326 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: freon >Reporter: Marton Elek >Assignee: Marton Elek >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > HDDS-2022 introduced new Freon tests but the Freon http server is not started > for the new tests. > Freon includes a http server which can be turned on with the '–server' flag. > It helps to monitor and profile the freon as the http server contains by > default the prometheus and profiler servlets. > The server should be started if's requested. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2333) Enable sync option for OM non-HA
Bharat Viswanadham created HDDS-2333: Summary: Enable sync option for OM non-HA Key: HDDS-2333 URL: https://issues.apache.org/jira/browse/HDDS-2333 Project: Hadoop Distributed Data Store Issue Type: Task Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham In OM non-HA when double buffer flushes, it should commit with sync turned on. As in non-HA when power failure/system crashes, the operations which are acknowledged by OM might be lost. (As in rocks DB with Sync false, the flush is asynchronous and it will not persist to storage system) In non-HA, this is not a problem because the guarantee is provided by ratis and ratis logs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2318) Avoid proto::tostring in preconditions to save CPU cycles
[ https://issues.apache.org/jira/browse/HDDS-2318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2318. -- Fix Version/s: 0.5.0 Resolution: Fixed > Avoid proto::tostring in preconditions to save CPU cycles > - > > Key: HDDS-2318 > URL: https://issues.apache.org/jira/browse/HDDS-2318 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Reporter: Rajesh Balamohan >Assignee: Bharat Viswanadham >Priority: Major > Labels: performance, pull-request-available > Fix For: 0.5.0 > > Attachments: Screenshot 2019-10-17 at 6.10.22 PM.png > > Time Spent: 20m > Remaining Estimate: 0h > > [https://github.com/apache/hadoop-ozone/blob/61f4aa30f502b34fd778d9b37b1168721abafb2f/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerProtocolServerSideTranslatorPB.java#L117] > > This ends up converting proto toString in precondition checks and burns CPU > cycles. {{request.toString()}} can be added in debug log on need basis. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2322) DoubleBuffer flush termination and OM is shutdown
Bharat Viswanadham created HDDS-2322: Summary: DoubleBuffer flush termination and OM is shutdown Key: HDDS-2322 URL: https://issues.apache.org/jira/browse/HDDS-2322 Project: Hadoop Distributed Data Store Issue Type: Task Reporter: Bharat Viswanadham om1_1 | 2019-10-18 00:34:45,317 [OMDoubleBufferFlushThread] ERROR - Terminating with exit status 2: OMDoubleBuffer flush threadOMDoubleBufferFlushThreadencountered Throwable error om1_1 | java.util.ConcurrentModificationException om1_1 | at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1660) om1_1 | at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) om1_1 | at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) om1_1 | at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) om1_1 | at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) om1_1 | at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) om1_1 | at org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup.getProtobuf(OmKeyLocationInfoGroup.java:65) om1_1 | at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) om1_1 | at java.base/java.util.Collections$2.tryAdvance(Collections.java:4745) om1_1 | at java.base/java.util.Collections$2.forEachRemaining(Collections.java:4753) om1_1 | at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) om1_1 | at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) om1_1 | at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) om1_1 | at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) om1_1 | at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) om1_1 | at org.apache.hadoop.ozone.om.helpers.OmKeyInfo.getProtobuf(OmKeyInfo.java:362) om1_1 | at org.apache.hadoop.ozone.om.codec.OmKeyInfoCodec.toPersistedFormat(OmKeyInfoCodec.java:37) om1_1 | at org.apache.hadoop.ozone.om.codec.OmKeyInfoCodec.toPersistedFormat(OmKeyInfoCodec.java:31) om1_1 | at org.apache.hadoop.hdds.utils.db.CodecRegistry.asRawData(CodecRegistry.java:68) om1_1 | at org.apache.hadoop.hdds.utils.db.TypedTable.putWithBatch(TypedTable.java:125) om1_1 | at org.apache.hadoop.ozone.om.response.key.OMKeyCreateResponse.addToDBBatch(OMKeyCreateResponse.java:58) om1_1 | at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.lambda$flushTransactions$0(OzoneManagerDoubleBuffer.java:139) om1_1 | at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) om1_1 | at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:137) om1_1 | at java.base/java.lang.Thread.run(Thread.java:834) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2311) Fix logic in RetryPolicy in OzoneClientSideTranslatorPB
Bharat Viswanadham created HDDS-2311: Summary: Fix logic in RetryPolicy in OzoneClientSideTranslatorPB Key: HDDS-2311 URL: https://issues.apache.org/jira/browse/HDDS-2311 Project: Hadoop Distributed Data Store Issue Type: Task Reporter: Bharat Viswanadham OzoneManagerProtocolClientSideTranslatorPB.java L251: if (cause instanceof NotLeaderException) { NotLeaderException notLeaderException = (NotLeaderException) cause; omFailoverProxyProvider.performFailoverIfRequired( notLeaderException.getSuggestedLeaderNodeId()); return getRetryAction(RetryAction.RETRY, retries, failovers); } The suggested leader returned from Server is not used during failOver, as the cause is a type of RemoteException. So with current code, it does not use suggested leader for failOver at all and by default with each OM, it tries max retries. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1230) Update OzoneServiceProvider in s3 gateway to handle OM ha
[ https://issues.apache.org/jira/browse/HDDS-1230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-1230. -- Fix Version/s: 0.5.0 Resolution: Fixed Fixed as part of HDDS-2019. > Update OzoneServiceProvider in s3 gateway to handle OM ha > - > > Key: HDDS-1230 > URL: https://issues.apache.org/jira/browse/HDDS-1230 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Priority: Major > Fix For: 0.5.0 > > > Update OzoneServiceProvider in s3 gateway to handle OM ha -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1986) Fix listkeys API
[ https://issues.apache.org/jira/browse/HDDS-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-1986. -- Fix Version/s: 0.5.0 Resolution: Fixed > Fix listkeys API > > > Key: HDDS-1986 > URL: https://issues.apache.org/jira/browse/HDDS-1986 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 7h 10m > Remaining Estimate: 0h > > This Jira is to fix listKeys API in HA code path. > In HA, we have an in-memory cache, where we put the result to in-memory cache > and return the response, later it will be picked by double buffer thread and > it will flush to disk. So, now when do listkeys, it should use both in-memory > cache and rocksdb key table to list keys in a bucket. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2279) S3 commands not working on HA cluster
Bharat Viswanadham created HDDS-2279: Summary: S3 commands not working on HA cluster Key: HDDS-2279 URL: https://issues.apache.org/jira/browse/HDDS-2279 Project: Hadoop Distributed Data Store Issue Type: Task Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham ozone s3 getSecret ozone s3 path are not working on OM HA cluster Because these commands do not take URI as a parameter. And for shell in HA, passing URI is mandatory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2278) Run S3 test suite on OM HA cluster
Bharat Viswanadham created HDDS-2278: Summary: Run S3 test suite on OM HA cluster Key: HDDS-2278 URL: https://issues.apache.org/jira/browse/HDDS-2278 Project: Hadoop Distributed Data Store Issue Type: Task Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham This will add a new compose setup with 3 OM's and start SCM, S3G, Datanode. Run the existing test suite against this new docker-compose cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2191) Handle bucket create request in OzoneNativeAuthorizer
[ https://issues.apache.org/jira/browse/HDDS-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2191. -- Resolution: Fixed > Handle bucket create request in OzoneNativeAuthorizer > - > > Key: HDDS-2191 > URL: https://issues.apache.org/jira/browse/HDDS-2191 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Security >Affects Versions: 0.5.0 >Reporter: Vivek Ratnavel Subramanian >Assignee: Vivek Ratnavel Subramanian >Priority: Major > > OzoneNativeAuthorizer should handle bucket create request when the bucket > object is not yet created. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2269) Provide config for fair/non-fair for OM RW Lock
Bharat Viswanadham created HDDS-2269: Summary: Provide config for fair/non-fair for OM RW Lock Key: HDDS-2269 URL: https://issues.apache.org/jira/browse/HDDS-2269 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham Provide config in OzoneManager Lock for fair/non-fair for OM RW Lock. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2244) Use new ReadWrite lock in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2244. -- Resolution: Fixed > Use new ReadWrite lock in OzoneManager > -- > > Key: HDDS-2244 > URL: https://issues.apache.org/jira/browse/HDDS-2244 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 3h 20m > Remaining Estimate: 0h > > Use new ReadWriteLock added in HDDS-2223. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2260) Avoid evaluation of LOG.trace and LOG.debug statement in the read/write path (HDDS)
[ https://issues.apache.org/jira/browse/HDDS-2260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2260. -- Fix Version/s: 0.5.0 Resolution: Fixed > Avoid evaluation of LOG.trace and LOG.debug statement in the read/write path > (HDDS) > --- > > Key: HDDS-2260 > URL: https://issues.apache.org/jira/browse/HDDS-2260 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client, Ozone Datanode >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Siddharth Wagle >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 4h 40m > Remaining Estimate: 0h > > LOG.trace and LOG.debug with logging information will be evaluated even when > debug/trace logging is disabled. This jira proposes to wrap all the > trace/debug logging with > LOG.isDebugEnabled and LOG.isTraceEnabled to prevent the logging. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2258) Fix checkstyle issues introduced by HDDS-2222
[ https://issues.apache.org/jira/browse/HDDS-2258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2258. -- Resolution: Duplicate > Fix checkstyle issues introduced by HDDS- > - > > Key: HDDS-2258 > URL: https://issues.apache.org/jira/browse/HDDS-2258 > Project: Hadoop Distributed Data Store > Issue Type: Task >Reporter: Vivek Ratnavel Subramanian >Assignee: Vivek Ratnavel Subramanian >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2244) Use new ReadWrite lock in OzoneManager
Bharat Viswanadham created HDDS-2244: Summary: Use new ReadWrite lock in OzoneManager Key: HDDS-2244 URL: https://issues.apache.org/jira/browse/HDDS-2244 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham Assignee: Nanda kumar Fix For: 0.5.0 Currently {{LockManager}} is using exclusive lock, instead we should support {{ReadWrite}} lock. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2239) Fix TestOzoneFsHAUrls
Bharat Viswanadham created HDDS-2239: Summary: Fix TestOzoneFsHAUrls Key: HDDS-2239 URL: https://issues.apache.org/jira/browse/HDDS-2239 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham [https://github.com/elek/ozone-ci-q4/blob/master/pr/pr-hdds-2162-pj84x/integration/hadoop-ozone/ozonefs/org.apache.hadoop.fs.ozone.TestOzoneFsHAURLs.txt] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2236) Remove default http-bind-host from ozone-default.xml
[ https://issues.apache.org/jira/browse/HDDS-2236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2236. -- Resolution: Not A Problem Discussed offline with [~arp], this was an intentional choice to use http.bind.host to 0.0.0.0 so that on multihomed environments/normal clusters it binds to all interfaces. > Remove default http-bind-host from ozone-default.xml > > > Key: HDDS-2236 > URL: https://issues.apache.org/jira/browse/HDDS-2236 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Priority: Major > > Right now, in the code to get HttpBindAddress > > final Optional bindHost = > getHostNameFromConfigKeys(conf, bindHostKey); > final Optional addressPort = > getPortNumberFromConfigKeys(conf, addressKey); > final Optional addressHost = > getHostNameFromConfigKeys(conf, addressKey); > String hostName = bindHost.orElse(addressHost.orElse(bindHostDefault)); > return NetUtils.createSocketAddr( > hostName + ":" + addressPort.orElse(bindPortdefault)); > > So, if http-address is mentioned in the config with some hostname, still > bind-host (0.0.0.0) will be used, as ozone-default.xml has value for > http-bind-host with 0.0.0.0. > > Like this, we need to delete the default 0.0.0.0 for recon,freon,datanode. > > ozone.om.http-bind-host > 0.0.0.0 > OM, MANAGEMENT > > The actual address the OM web server will bind to. If this optional > the address is set, it overrides only the hostname portion of > ozone.om.http-address. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2236) Remove default http-bind-host from ozone-default.xml
Bharat Viswanadham created HDDS-2236: Summary: Remove default http-bind-host from ozone-default.xml Key: HDDS-2236 URL: https://issues.apache.org/jira/browse/HDDS-2236 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham Right now, in the code to get HttpBindAddress final Optional bindHost = getHostNameFromConfigKeys(conf, bindHostKey); final Optional addressPort = getPortNumberFromConfigKeys(conf, addressKey); final Optional addressHost = getHostNameFromConfigKeys(conf, addressKey); String hostName = bindHost.orElse(addressHost.orElse(bindHostDefault)); return NetUtils.createSocketAddr( hostName + ":" + addressPort.orElse(bindPortdefault)); So, if http-address is mentioned with some hostname, still bind-host (0.0.0.0) will be used. Like this, we need to delete the default 0.0.0.0 for recon,freon,datanode. ozone.om.http-bind-host 0.0.0.0 OM, MANAGEMENT The actual address the OM web server will bind to. If this optional the address is set, it overrides only the hostname portion of ozone.om.http-address. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2224) Fix loadup cache for cache cleanup policy NEVER
Bharat Viswanadham created HDDS-2224: Summary: Fix loadup cache for cache cleanup policy NEVER Key: HDDS-2224 URL: https://issues.apache.org/jira/browse/HDDS-2224 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham During initial startup/restart of OM, if table has cache cleanup policy set to NEVER, we fill the table cache and also epochEntries. We do not need to add entries to epochEntries, as the epochEntries is used for eviction from the cache, once double buffer flushes to disk. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2193) Adding container related metrics in SCM
Bharat Viswanadham created HDDS-2193: Summary: Adding container related metrics in SCM Key: HDDS-2193 URL: https://issues.apache.org/jira/browse/HDDS-2193 Project: Hadoop Distributed Data Store Issue Type: Improvement Components: SCM Reporter: Bharat Viswanadham Assignee: Supratim Deka This jira aims to add more container related metrics to SCM. Following metrics will be added as part of this jira: * Number of containers * Number of open containers * Number of closed containers * Number of quasi closed containers * Number of closing containers * Number of successful create container calls * Number of failed create container calls * Number of successful delete container calls * Number of failed delete container calls * Number of successful container report processing * Number of failed container report processing * Number of successful incremental container report processing * Number of failed incremental container report processing -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2168) TestOzoneManagerDoubleBufferWithOMResponse sometimes fails with out of memory error
[ https://issues.apache.org/jira/browse/HDDS-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2168. -- Fix Version/s: 0.5.0 Resolution: Fixed > TestOzoneManagerDoubleBufferWithOMResponse sometimes fails with out of memory > error > --- > > Key: HDDS-2168 > URL: https://issues.apache.org/jira/browse/HDDS-2168 > Project: Hadoop Distributed Data Store > Issue Type: Task > Components: Ozone Manager >Affects Versions: 0.4.1 >Reporter: Vivek Ratnavel Subramanian >Assignee: Vivek Ratnavel Subramanian >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > testDoubleBuffer() in TestOzoneManagerDoubleBufferWithOMResponse fails with > outofmemory exceptions at times in dev machines. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2163) Add "Replication factor" to the output of list keys
[ https://issues.apache.org/jira/browse/HDDS-2163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2163. -- Fix Version/s: 0.5.0 Resolution: Fixed > Add "Replication factor" to the output of list keys > > > Key: HDDS-2163 > URL: https://issues.apache.org/jira/browse/HDDS-2163 > Project: Hadoop Distributed Data Store > Issue Type: Task > Components: Ozone CLI >Affects Versions: 0.4.1 >Reporter: Vivek Ratnavel Subramanian >Assignee: Vivek Ratnavel Subramanian >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 50m > Remaining Estimate: 0h > > The output of "ozone sh key list /vol1/bucket1" does not include replication > factor and it will be good to have it in the output. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2162) Make KeyTab configuration support HA config
Bharat Viswanadham created HDDS-2162: Summary: Make KeyTab configuration support HA config Key: HDDS-2162 URL: https://issues.apache.org/jira/browse/HDDS-2162 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham To have a single configuration to use across OM cluster, few of the configs like OZONE_OM_KERBEROS_KEYTAB_FILE_KEY, OZONE_OM_KERBEROS_PRINCIPAL_KEY, OZONE_OM_HTTP_KERBEROS_KEYTAB_FILE, OZONE_OM_HTTP_KERBEROS_PRINCIPAL_KEY need to support configs which append with service id and node id. This Jira is to fix the above configs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2121) Create a shaded ozone filesystem (client) jar
[ https://issues.apache.org/jira/browse/HDDS-2121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2121. -- Fix Version/s: 0.5.0 Resolution: Fixed > Create a shaded ozone filesystem (client) jar > - > > Key: HDDS-2121 > URL: https://issues.apache.org/jira/browse/HDDS-2121 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: build >Reporter: Arpit Agarwal >Assignee: Elek, Marton >Priority: Blocker > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > We need a shaded Ozonefs jar that does not include Hadoop ecosystem > components (Hadoop, HDFS, Ratis, Zookeeper). > A common expected use case for Ozone is Hadoop clients (3.2.0 and later) > wanting to access Ozone via the Ozone Filesystem interface. For these > clients, we want to add Ozone file system jar to the classpath, however we > want to use Hadoop ecosystem dependencies that are `provided` and already > expected to be in the client classpath. > Note that this is different from the legacy jar which bundles a shaded Hadoop > 3.2.0. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2139) Update BeanUtils and Jackson Databind dependency versions
[ https://issues.apache.org/jira/browse/HDDS-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2139. -- Fix Version/s: 0.5.0 Resolution: Fixed Thank You [~hanishakoneru] for the contribution. I have committed this to the trunk. > Update BeanUtils and Jackson Databind dependency versions > - > > Key: HDDS-2139 > URL: https://issues.apache.org/jira/browse/HDDS-2139 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 40m > Remaining Estimate: 0h > > The following Ozone dependencies have known security vulnerabilities. We > should update them to newer/ latest versions. > * Apache Common BeanUtils version 1.9.3 > * Fasterxml Jackson version 2.9.5 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2144) MR job failing on secure Ozone cluster
Bharat Viswanadham created HDDS-2144: Summary: MR job failing on secure Ozone cluster Key: HDDS-2144 URL: https://issues.apache.org/jira/browse/HDDS-2144 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham Failing with below error: Caused by: Client cannot authenticate via:[TOKEN, KERBEROS] org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:173) at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390) at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:617) at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:411) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:804) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:800) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:800) at org.apache.hadoop.ipc.Client$Connection.access$3700(Client.java:411) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1572) at org.apache.hadoop.ipc.Client.call(Client.java:1403) at org.apache.hadoop.ipc.Client.call(Client.java:1367) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) at com.sun.proxy.$Proxy79.submitRequest(Unknown Source) at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy79.submitRequest(Unknown Source) at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:332) at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceList(OzoneManagerProtocolClientSideTranslatorPB.java:1163) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hdds.tracing.TraceAllMethod.invoke(TraceAllMethod.java:66) at com.sun.proxy.$Proxy80.getServiceList(Unknown Source) at org.apache.hadoop.ozone.client.rpc.RpcClient.getScmAddressForClient(RpcClient.java:248) at org.apache.hadoop.ozone.client.rpc.RpcClient.(RpcClient.java:167) at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:256) at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:239) at org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:203) at org.apache.hadoop.fs.ozone.BasicOzoneClientAdapterImpl.(BasicOzoneClientAdapterImpl.java:161) at org.apache.hadoop.fs.ozone.OzoneClientAdapterImpl.(OzoneClientAdapterImpl.java:50) at org.apache.hadoop.fs.ozone.OzoneFileSystem.createAdapter(OzoneFileSystem.java:102) at org.apache.hadoop.fs.ozone.BasicOzoneFileSystem.initialize(BasicOzoneFileSystem.java:155) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3303) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361) at org.apache.hadoop.yarn.util.FSDownload.verifyAndCopy(FSDownload.java:268) at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:67) at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:414) at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:411) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at
[jira] [Created] (HDDS-2143) Rename classes under package org.apache.hadoop.utils
Bharat Viswanadham created HDDS-2143: Summary: Rename classes under package org.apache.hadoop.utils Key: HDDS-2143 URL: https://issues.apache.org/jira/browse/HDDS-2143 Project: Hadoop Distributed Data Store Issue Type: Bug Environment: Rename classes under package org.apache.hadoop.utils -> org.apache.hadoop.hdds.utils in hadoop-hdds-common Now, with current way, we might collide with hadoop classes. Reporter: Bharat Viswanadham -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2098) Ozone shell command prints out ERROR when the log4j file is not present.
[ https://issues.apache.org/jira/browse/HDDS-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2098. -- Resolution: Fixed > Ozone shell command prints out ERROR when the log4j file is not present. > > > Key: HDDS-2098 > URL: https://issues.apache.org/jira/browse/HDDS-2098 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone CLI >Affects Versions: 0.5.0 >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > *Exception Trace* > {code} > log4j:ERROR Could not read configuration file from URL > [file:/etc/ozone/conf/ozone-shell-log4j.properties]. > java.io.FileNotFoundException: /etc/ozone/conf/ozone-shell-log4j.properties > (No such file or directory) > at java.io.FileInputStream.open0(Native Method) > at java.io.FileInputStream.open(FileInputStream.java:195) > at java.io.FileInputStream.(FileInputStream.java:138) > at java.io.FileInputStream.(FileInputStream.java:93) > at > sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90) > at > sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188) > at > org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:557) > at > org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526) > at org.apache.log4j.LogManager.(LogManager.java:127) > at org.slf4j.impl.Log4jLoggerFactory.(Log4jLoggerFactory.java:66) > at org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:72) > at > org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:45) > at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150) > at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124) > at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412) > at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357) > at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383) > at org.apache.hadoop.ozone.web.ozShell.Shell.(Shell.java:35) > log4j:ERROR Ignoring configuration file > [file:/etc/ozone/conf/ozone-shell-log4j.properties]. > log4j:WARN No appenders could be found for logger > (io.jaegertracing.thrift.internal.senders.ThriftSenderFactory). > log4j:WARN Please initialize the log4j system properly. > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more > info. > { > "metadata" : { }, > "name" : "vol-test-putfile-1567740142", > "admin" : "root", > "owner" : "root", > "creationTime" : 1567740146501, > "acls" : [ { > "type" : "USER", > "name" : "root", > "aclScope" : "ACCESS", > "aclList" : [ "ALL" ] > }, { > "type" : "GROUP", > "name" : "root", > "aclScope" : "ACCESS", > "aclList" : [ "ALL" ] > } ], > "quota" : 1152921504606846976 > } > {code} > *Fix* > When a log4j file is not present, the default should be console. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2007) Make ozone fs shell command work with OM HA service ids
[ https://issues.apache.org/jira/browse/HDDS-2007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2007. -- Fix Version/s: 0.5.0 Resolution: Fixed > Make ozone fs shell command work with OM HA service ids > --- > > Key: HDDS-2007 > URL: https://issues.apache.org/jira/browse/HDDS-2007 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Client >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 6h 50m > Remaining Estimate: 0h > > Build an HDFS HA-like nameservice for OM HA so that Ozone client can access > Ozone HA cluster with ease. > The majority of the work is already done in HDDS-972. But the problem is that > the client would crash if there are more than one service ids > (ozone.om.service.ids) configured in ozone-site.xml. This need to be address > on client side. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2087) Remove the hard coded config key in ChunkManager
[ https://issues.apache.org/jira/browse/HDDS-2087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2087. -- Fix Version/s: 0.5.0 Resolution: Fixed > Remove the hard coded config key in ChunkManager > > > Key: HDDS-2087 > URL: https://issues.apache.org/jira/browse/HDDS-2087 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Anu Engineer >Assignee: Vivek Ratnavel Subramanian >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > We have a hard-coded config key in the {{ChunkManagerFactory.java.}} > > {code} > boolean scrubber = config.getBoolean( > "hdds.containerscrub.enabled", > false); > {code} -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2080) Document details regarding how to implement write request in OzoneManager
Bharat Viswanadham created HDDS-2080: Summary: Document details regarding how to implement write request in OzoneManager Key: HDDS-2080 URL: https://issues.apache.org/jira/browse/HDDS-2080 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2079) Fix TestSecureOzoneManager
Bharat Viswanadham created HDDS-2079: Summary: Fix TestSecureOzoneManager Key: HDDS-2079 URL: https://issues.apache.org/jira/browse/HDDS-2079 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham [https://github.com/elek/ozone-ci/blob/master/pr/pr-hdds-1909-plfbr/integration/hadoop-ozone/integration-test/org.apache.hadoop.ozone.TestSecureOzoneCluster.txt] -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2078) Fix TestSecureOzoneCluster
Bharat Viswanadham created HDDS-2078: Summary: Fix TestSecureOzoneCluster Key: HDDS-2078 URL: https://issues.apache.org/jira/browse/HDDS-2078 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1941) Unused executor in SimpleContainerDownloader
[ https://issues.apache.org/jira/browse/HDDS-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-1941. -- Fix Version/s: 0.5.0 Resolution: Fixed > Unused executor in SimpleContainerDownloader > > > Key: HDDS-1941 > URL: https://issues.apache.org/jira/browse/HDDS-1941 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Doroszlai, Attila >Assignee: Doroszlai, Attila >Priority: Minor > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 50m > Remaining Estimate: 0h > > {{SimpleContainerDownloader}} has an {{executor}} that's created and shut > down, but never used. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2038) Add Auditlog for ACL operations
Bharat Viswanadham created HDDS-2038: Summary: Add Auditlog for ACL operations Key: HDDS-2038 URL: https://issues.apache.org/jira/browse/HDDS-2038 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham This is to add audit log support for Acl operations in HA code path. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Reopened] (HDDS-1975) Implement default acls for bucket/volume/key for OM HA code
[ https://issues.apache.org/jira/browse/HDDS-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham reopened HDDS-1975: -- > Implement default acls for bucket/volume/key for OM HA code > --- > > Key: HDDS-1975 > URL: https://issues.apache.org/jira/browse/HDDS-1975 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 2h > Remaining Estimate: 0h > > This Jira is to implement default ACLs for Ozone volume/bucket/key. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2019) Handle Set DtService of token in S3Gateway for OM HA
Bharat Viswanadham created HDDS-2019: Summary: Handle Set DtService of token in S3Gateway for OM HA Key: HDDS-2019 URL: https://issues.apache.org/jira/browse/HDDS-2019 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham When OM HA is enabled, when tokens are generated, the service name should be set with address of all OM's. Current without HA, it is set with Om RpcAddress string. This Jira is to handle: # Set dtService with all OM address. # Update token selector to return tokens if there is a match with Service. Because SaslRpcClient calls token selector with server address. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2018) Handle Set DtService of token for OM HA
Bharat Viswanadham created HDDS-2018: Summary: Handle Set DtService of token for OM HA Key: HDDS-2018 URL: https://issues.apache.org/jira/browse/HDDS-2018 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham When OM HA is enabled, when tokens are generated, the service name should be set with address of all OM's. Current without HA, it is set with Om RpcAddress string. This Jira is to handle: # Set dtService with all OM address. # Update token selector to return tokens if there is a match with Service. Because SaslRpcClient calls token selector with server address. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2008) Wrong package for RatisHelper class in hadoop-hdds/common module.
[ https://issues.apache.org/jira/browse/HDDS-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2008. -- Fix Version/s: 0.5.0 Resolution: Fixed > Wrong package for RatisHelper class in hadoop-hdds/common module. > - > > Key: HDDS-2008 > URL: https://issues.apache.org/jira/browse/HDDS-2008 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Minor > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 50m > Remaining Estimate: 0h > > It is currently org.apache.ratis.RatisHelper. > It should be org.apache.hadoop.hdds.ratis.RatisHelper. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1995) Generate renewTime on OMLeader for GetDelegationToken
Bharat Viswanadham created HDDS-1995: Summary: Generate renewTime on OMLeader for GetDelegationToken Key: HDDS-1995 URL: https://issues.apache.org/jira/browse/HDDS-1995 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham Use renewTime generated by OM leader, across quorum of OM's. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1993) Merge OzoneManagerRequestHandler and OzoneManagerHARequestHandlerImpl
Bharat Viswanadham created HDDS-1993: Summary: Merge OzoneManagerRequestHandler and OzoneManagerHARequestHandlerImpl Key: HDDS-1993 URL: https://issues.apache.org/jira/browse/HDDS-1993 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham Once HA and Non-HA code are merged to use newly OM HA code. We can merge these classes, and remove the unused code. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1992) Have a separate CacheTable class for cache based tables.
Bharat Viswanadham created HDDS-1992: Summary: Have a separate CacheTable class for cache based tables. Key: HDDS-1992 URL: https://issues.apache.org/jira/browse/HDDS-1992 Project: Hadoop Distributed Data Store Issue Type: Bug Components: om Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham Fix For: 0.4.1 HDDS-1499 introduced a new caching layer together with a double-buffer based db writer to support OM HA. TLDR: I think the caching layer is not updated for new volume creation. And (slightly related to this problem) I suggest to separated the TypedTable and the caching layer. ## How to reproduce the problem? 1. Start a docker compose cluster 2. Create one volume (let's say `/vol1`) 3. Restart the om (!) 4. Try to create an _other_ volume twice! ``` bash-4.2$ ozone sh volume create /vol2 2019-08-07 12:29:47 INFO RpcClient:288 - Creating Volume: vol2, with hadoop as owner. bash-4.2$ ozone sh volume create /vol2 2019-08-07 12:29:50 INFO RpcClient:288 - Creating Volume: vol2, with hadoop as owner. ``` Expected behavior is an error: {code} bash-4.2$ ozone sh volume create /vol1 2019-08-07 09:48:39 INFO RpcClient:288 - Creating Volume: vol1, with hadoop as owner. bash-4.2$ ozone sh volume create /vol1 2019-08-07 09:48:42 INFO RpcClient:288 - Creating Volume: vol1, with hadoop as owner. VOLUME_ALREADY_EXISTS {code} The problem is that the new cache is used even for the old code path (TypedTable): {code} @Override public VALUE get(KEY key) throws IOException { // Here the metadata lock will guarantee that cache is not updated for same // key during get key. CacheResult> cacheResult = cache.lookup(new CacheKey<>(key)); if (cacheResult.getCacheStatus() == EXISTS) { return cacheResult.getValue().getCacheValue(); } else if (cacheResult.getCacheStatus() == NOT_EXIST) { return null; } else { return getFromTable(key); } } {code} For volume table after the FIRST start it always returns with `getFromTable(key)` due to the condition in the `TableCacheImpl.lookup`: {code} public CacheResult lookup(CACHEKEY cachekey) { if (cache.size() == 0) { return new CacheResult<>(CacheResult.CacheStatus.MAY_EXIST, null); } {code} But after a restart the cache is pre-loaded by the TypedTable.constructor. After the restart, the real caching logic will be used (as cache.size()>0), which cause a problem as the cache is NOT updated from the old code path. An additional problem is that the cache is turned on for all the metadata table even if the cache is not required... ## Proposed solution As I commented at HDDS-1499 this caching layer is not a "traditional cache". It's not updated during the typedTable.put() call but updated by a separated component during double-buffer flash. I would suggest to remove the cache related methods from TypedTable (move to a separated implementation). I think this kind of caching can be independent from the TypedTable implementation. We can continue to use the simple TypedTable everywhere where we don't need to use any kind of caching. For caching we can use a separated object. It would make it more visible that the cache should always be updated manually all the time. This separated caching utility may include a reference to the original TypedTable/Table. With this approach we can separate the different responsibilities but provide the same functionality. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Reopened] (HDDS-1123) Add timeout when submitting request to Raft server on leader OM
[ https://issues.apache.org/jira/browse/HDDS-1123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham reopened HDDS-1123: -- Assignee: (was: Bharat Viswanadham) > Add timeout when submitting request to Raft server on leader OM > --- > > Key: HDDS-1123 > URL: https://issues.apache.org/jira/browse/HDDS-1123 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Priority: Major > > Add timeout when submitting request to Raft server on leader OM. > As raft server keeps on trying, even if the followers are down, so we need to > add some time out logic when submitting request in OM, to send back response > to client -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1991) Remove RatisClient in OM HA
Bharat Viswanadham created HDDS-1991: Summary: Remove RatisClient in OM HA Key: HDDS-1991 URL: https://issues.apache.org/jira/browse/HDDS-1991 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham In OM, we use ratis server api's to submit request. We can remove the RatisClient code from OM, which is no more used in submitting requests to ratis. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1990) Make expiry of Delegation tokens to OM HA model.
Bharat Viswanadham created HDDS-1990: Summary: Make expiry of Delegation tokens to OM HA model. Key: HDDS-1990 URL: https://issues.apache.org/jira/browse/HDDS-1990 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham Currently, when tokens expire, there is background thread which runs and check if the token is expired, delete from in-memory map and DB. In HA this deletion should go through ratis, to keep all OM's in sync. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org