[jira] [Assigned] (HDDS-4391) UnixPath.toUri() is expensive
[
https://issues.apache.org/jira/browse/HDDS-4391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal reassigned HDDS-4391:
---
Assignee: Bharat Viswanadham
> UnixPath.toUri() is expensive
> -
>
> Key: HDDS-4391
> URL: https://issues.apache.org/jira/browse/HDDS-4391
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Manager
>Affects Versions: 1.0.0
>Reporter: Wei-Chiu Chuang
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: Screen Shot 2020-10-20 at 12.34.52 PM.png
>
>
> OM makes call this API to look up a key.
> This call accounts for 20% (give or take) of OM request handler overhead.
> Would be great if we can get rid of this call.
> !Screen Shot 2020-10-20 at 12.34.52 PM.png!
> OMClientRequest.java
> {code:java}
> @SuppressFBWarnings("DMI_HARDCODED_ABSOLUTE_FILENAME")
> public static String validateAndNormalizeKey(String keyName)
> throws OMException {
> String normalizedKeyName;
> if (keyName.startsWith(OM_KEY_PREFIX)) {
> normalizedKeyName = Paths.get(keyName).toUri().normalize().getPath();
> } else {
> normalizedKeyName = Paths.get(OM_KEY_PREFIX, keyName).toUri()
> .normalize().getPath();
> } {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4333) Ozone supports append operation
[ https://issues.apache.org/jira/browse/HDDS-4333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17221752#comment-17221752 ] Arpit Agarwal commented on HDDS-4333: - I don't think we should implement the append/truncate features right now. I think we should focus on stability and performance of the write pipeline and recent features like multi-RAFT. > Ozone supports append operation > --- > > Key: HDDS-4333 > URL: https://issues.apache.org/jira/browse/HDDS-4333 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: mingchao zhao >Priority: Major > > Currently HDDS does not support modifying append operations on data. We had > this need in production, so we needed to make HDDS support this feature -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-4351) DN crash while RatisApplyTransactionExecutor tries to putBlock to rocksDB
[
https://issues.apache.org/jira/browse/HDDS-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal resolved HDDS-4351.
-
Fix Version/s: 1.1.0
Resolution: Done
> DN crash while RatisApplyTransactionExecutor tries to putBlock to rocksDB
> -
>
> Key: HDDS-4351
> URL: https://issues.apache.org/jira/browse/HDDS-4351
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Affects Versions: 1.1.0
>Reporter: Glen Geng
>Assignee: Ethan Rose
>Priority: Major
> Fix For: 1.1.0
>
>
> In Tencent, we monthly pick up the latest mater and deploy them to our
> production environment.
> This time, we tested c956ce6 (HDDS-4262 [. Use ClientID and CallID from Rpc
> Client to detect retry
> re…|https://github.com/apache/hadoop-ozone/commit/c956ce6b7537a0286c01b15d496a7ffeba90]
> ), encountered frequently crash in datanode while putBlock.
>
> *The setup* is 3 DN, each engage in 8 pipelines. 1 OM 1 SCM and 1 Gateway.
> *The repo procedure* is simple. Continually writing 10GB size files to s3g
> from python (the aws lib boto3), after write tens of files, DN might crash
> while applying putBlock operations.
> After running the test on the version that revert HDDS-3869 for 10 hours,,
> no DN crash occurred.
> Will schedule a long run test on the latest master with HDDS-4327, to check
> if adding try-with-resource to BatchOperation could fix the crash issue.
>
> *Example1: segment fault while putBlock.*
> {code:java}
> Current thread (0x7eff34524000): JavaThread
> "RatisApplyTransactionExecutor 9" daemon [_thread_in_native, id=20401,
> stack(0x7efef4a14000,0x7efef4b15000)]siginfo: si_signo: 11 (SIGSEGV),
> si_code: 2 (SEGV_ACCERR), si_addr: 0x7eff37eb9000Registers:
> RAX=0x7efe8bbfb024, RBX=0x, RCX=0x,
> RDX=0x007688e4
> RSP=0x7efef4b11e38, RBP=0x7efef4b11f60, RSI=0x7eff37eb8feb,
> RDI=0x7efe8f892640
> R8 =0x7efe8bbfb024, R9 =0x0080, R10=0x0022,
> R11=0x1000
> R12=0x7efef4b12100, R13=0x7eff340badc0, R14=0x7eff340bb7b0,
> R15=0x0440
> RIP=0x7eff4fa04bae, EFLAGS=0x00010206, CSGSFS=0x0033,
> ERR=0x0004
> TRAPNO=0x000eStack: [0x7efef4a14000,0x7efef4b15000],
> sp=0x7efef4b11e38, free space=1015k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> C [libc.so.6+0x151bae] __memmove_ssse3_back+0x192e
> C [librocksdbjni3701435679326554484.so+0x3b2263]
> rocksdb::MemTableInserter::DeleteCF(unsigned int, rocksdb::Slice const&)+0x253
> C [librocksdbjni3701435679326554484.so+0x3a889f]
> rocksdb::WriteBatchInternal::Iterate(rocksdb::WriteBatch const*,
> rocksdb::WriteBatch::Handler*, unsigned long, unsigned long)+0x75f
> C [librocksdbjni3701435679326554484.so+0x3a8d44]
> rocksdb::WriteBatch::Iterate(rocksdb::WriteBatch::Handler*) const+0x24
> C [librocksdbjni3701435679326554484.so+0x3ac3f9]
> rocksdb::WriteBatchInternal::InsertInto(rocksdb::WriteThread::WriteGroup&,
> unsigned long, rocksdb::ColumnFamilyMemTables*, rocksdb::FlushScheduler*,
> rocksdb::TrimHistoryScheduler*, bool, unsigned long, rocksdb::DB*, bool,
> bool, bool)+0x249
> C [librocksdbjni3701435679326554484.so+0x2f6308]
> rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&,
> rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long,
> bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*)+0x1e98
> C [librocksdbjni3701435679326554484.so+0x2f70c1]
> rocksdb::DBImpl::Write(rocksdb::WriteOptions const&,
> rocksdb::WriteBatch*)+0x21
> C [librocksdbjni3701435679326554484.so+0x1dd0cc]
> Java_org_rocksdb_RocksDB_write0+0xcc
> j org.rocksdb.RocksDB.write0(JJJ)V+0
> J 8597 C1
> org.apache.hadoop.ozone.container.keyvalue.impl.BlockManagerImpl.putBlock(Lorg/apache/hadoop/ozone/container/common/interfaces/Container;Lorg/apache/hadoop/ozone/container/common/helpers/BlockData;Z)J
> (487 bytes) @ 0x7eff3a8dd84c [0x7eff3a8db8e0+0x1f6c]
> J 8700 C1
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handlePutBlock(Lorg/apache/hadoop/hdds/protocol/datanode/proto/ContainerProtos$ContainerCommandRequestProto;Lorg/apache/hadoop/ozone/container/keyvalue/KeyValueContainer;Lorg/apache/hadoop/ozone/container/common/transport/server/ratis/DispatcherContext;)Lorg/apache/hadoop/hdds/protocol/datanode/proto/ContainerProtos$ContainerCommandResponseProto;
> (211 bytes) @ 0x7eff3a927ebc [0x7eff3a926220+0x1c9c]
> J 6685 C1
>
[jira] [Commented] (HDDS-4351) DN crash while RatisApplyTransactionExecutor tries to putBlock to rocksDB
[
https://issues.apache.org/jira/browse/HDDS-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17215477#comment-17215477
]
Arpit Agarwal commented on HDDS-4351:
-
Thank you very much for reporting this [~glengeng] and for verifying the fix. I
will resolve this.
> DN crash while RatisApplyTransactionExecutor tries to putBlock to rocksDB
> -
>
> Key: HDDS-4351
> URL: https://issues.apache.org/jira/browse/HDDS-4351
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Affects Versions: 1.1.0
>Reporter: Glen Geng
>Assignee: Ethan Rose
>Priority: Major
>
> In Tencent, we monthly pick up the latest mater and deploy them to our
> production environment.
> This time, we tested c956ce6 (HDDS-4262 [. Use ClientID and CallID from Rpc
> Client to detect retry
> re…|https://github.com/apache/hadoop-ozone/commit/c956ce6b7537a0286c01b15d496a7ffeba90]
> ), encountered frequently crash in datanode while putBlock.
>
> *The setup* is 3 DN, each engage in 8 pipelines. 1 OM 1 SCM and 1 Gateway.
> *The repo procedure* is simple. Continually writing 10GB size files to s3g
> from python (the aws lib boto3), after write tens of files, DN might crash
> while applying putBlock operations.
> After running the test on the version that revert HDDS-3869 for 10 hours,,
> no DN crash occurred.
> Will schedule a long run test on the latest master with HDDS-4327, to check
> if adding try-with-resource to BatchOperation could fix the crash issue.
>
> *Example1: segment fault while putBlock.*
> {code:java}
> Current thread (0x7eff34524000): JavaThread
> "RatisApplyTransactionExecutor 9" daemon [_thread_in_native, id=20401,
> stack(0x7efef4a14000,0x7efef4b15000)]siginfo: si_signo: 11 (SIGSEGV),
> si_code: 2 (SEGV_ACCERR), si_addr: 0x7eff37eb9000Registers:
> RAX=0x7efe8bbfb024, RBX=0x, RCX=0x,
> RDX=0x007688e4
> RSP=0x7efef4b11e38, RBP=0x7efef4b11f60, RSI=0x7eff37eb8feb,
> RDI=0x7efe8f892640
> R8 =0x7efe8bbfb024, R9 =0x0080, R10=0x0022,
> R11=0x1000
> R12=0x7efef4b12100, R13=0x7eff340badc0, R14=0x7eff340bb7b0,
> R15=0x0440
> RIP=0x7eff4fa04bae, EFLAGS=0x00010206, CSGSFS=0x0033,
> ERR=0x0004
> TRAPNO=0x000eStack: [0x7efef4a14000,0x7efef4b15000],
> sp=0x7efef4b11e38, free space=1015k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> C [libc.so.6+0x151bae] __memmove_ssse3_back+0x192e
> C [librocksdbjni3701435679326554484.so+0x3b2263]
> rocksdb::MemTableInserter::DeleteCF(unsigned int, rocksdb::Slice const&)+0x253
> C [librocksdbjni3701435679326554484.so+0x3a889f]
> rocksdb::WriteBatchInternal::Iterate(rocksdb::WriteBatch const*,
> rocksdb::WriteBatch::Handler*, unsigned long, unsigned long)+0x75f
> C [librocksdbjni3701435679326554484.so+0x3a8d44]
> rocksdb::WriteBatch::Iterate(rocksdb::WriteBatch::Handler*) const+0x24
> C [librocksdbjni3701435679326554484.so+0x3ac3f9]
> rocksdb::WriteBatchInternal::InsertInto(rocksdb::WriteThread::WriteGroup&,
> unsigned long, rocksdb::ColumnFamilyMemTables*, rocksdb::FlushScheduler*,
> rocksdb::TrimHistoryScheduler*, bool, unsigned long, rocksdb::DB*, bool,
> bool, bool)+0x249
> C [librocksdbjni3701435679326554484.so+0x2f6308]
> rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&,
> rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long,
> bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*)+0x1e98
> C [librocksdbjni3701435679326554484.so+0x2f70c1]
> rocksdb::DBImpl::Write(rocksdb::WriteOptions const&,
> rocksdb::WriteBatch*)+0x21
> C [librocksdbjni3701435679326554484.so+0x1dd0cc]
> Java_org_rocksdb_RocksDB_write0+0xcc
> j org.rocksdb.RocksDB.write0(JJJ)V+0
> J 8597 C1
> org.apache.hadoop.ozone.container.keyvalue.impl.BlockManagerImpl.putBlock(Lorg/apache/hadoop/ozone/container/common/interfaces/Container;Lorg/apache/hadoop/ozone/container/common/helpers/BlockData;Z)J
> (487 bytes) @ 0x7eff3a8dd84c [0x7eff3a8db8e0+0x1f6c]
> J 8700 C1
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handlePutBlock(Lorg/apache/hadoop/hdds/protocol/datanode/proto/ContainerProtos$ContainerCommandRequestProto;Lorg/apache/hadoop/ozone/container/keyvalue/KeyValueContainer;Lorg/apache/hadoop/ozone/container/common/transport/server/ratis/DispatcherContext;)Lorg/apache/hadoop/hdds/protocol/datanode/proto/ContainerProtos$ContainerCommandResponseProto;
> (211 bytes) @ 0x7eff3a927ebc [0x7eff3a926220+0x1c9c]
> J 6685 C1
>
[jira] [Updated] (HDDS-4327) Potential resource leakage using BatchOperation
[ https://issues.apache.org/jira/browse/HDDS-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-4327: Target Version/s: 1.1.0 > Potential resource leakage using BatchOperation > --- > > Key: HDDS-4327 > URL: https://issues.apache.org/jira/browse/HDDS-4327 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Wei-Chiu Chuang >Assignee: Bharat Viswanadham >Priority: Blocker > > there are a number of places in the code where BatchOperation is used but not > closed. As a best practice, better to close them explicitly. > I have a stress test code that uses BatchOperation to insert into OM rocksdb. > Without closing BatchOperation explicitly, the process crashes after just a > few minutes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-4327) Potential resource leakage using BatchOperation
[ https://issues.apache.org/jira/browse/HDDS-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal reassigned HDDS-4327: --- Assignee: Bharat Viswanadham > Potential resource leakage using BatchOperation > --- > > Key: HDDS-4327 > URL: https://issues.apache.org/jira/browse/HDDS-4327 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Wei-Chiu Chuang >Assignee: Bharat Viswanadham >Priority: Major > > there are a number of places in the code where BatchOperation is used but not > closed. As a best practice, better to close them explicitly. > I have a stress test code that uses BatchOperation to insert into OM rocksdb. > Without closing BatchOperation explicitly, the process crashes after just a > few minutes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4327) Potential resource leakage using BatchOperation
[ https://issues.apache.org/jira/browse/HDDS-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-4327: Priority: Blocker (was: Major) > Potential resource leakage using BatchOperation > --- > > Key: HDDS-4327 > URL: https://issues.apache.org/jira/browse/HDDS-4327 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Wei-Chiu Chuang >Assignee: Bharat Viswanadham >Priority: Blocker > > there are a number of places in the code where BatchOperation is used but not > closed. As a best practice, better to close them explicitly. > I have a stress test code that uses BatchOperation to insert into OM rocksdb. > Without closing BatchOperation explicitly, the process crashes after just a > few minutes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4209) S3A Filesystem does not work with Ozone S3 in file system compat mode
[
https://issues.apache.org/jira/browse/HDDS-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-4209:
Summary: S3A Filesystem does not work with Ozone S3 in file system compat
mode (was: S3A Filesystem does not work with Ozone S3)
> S3A Filesystem does not work with Ozone S3 in file system compat mode
> -
>
> Key: HDDS-4209
> URL: https://issues.apache.org/jira/browse/HDDS-4209
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Blocker
> Labels: OzoneS3, S3A, pull-request-available
>
> When *ozone.om.enable.filesystem.paths* is enabled
>
> hdfs dfs -mkdir -p s3a://b12345/d11/d12 -> Success
> hdfs dfs -put /tmp/file1 s3a://b12345/d11/d12/file1 -> fails with below error
>
> {code:java}
> 2020-09-04 03:53:51,377 ERROR
> org.apache.hadoop.ozone.om.request.key.OMKeyCreateRequest: Key creation
> failed. Volume:s3v, Bucket:b1234, Keyd11/d12/file1._COPYING_. Exception:{}
> NOT_A_FILE org.apache.hadoop.ozone.om.exceptions.OMException: Can not create
> file: cp/k1._COPYING_ as there is already file in the given path
> at
> org.apache.hadoop.ozone.om.request.key.OMKeyCreateRequest.validateAndUpdateCache(OMKeyCreateRequest.java:256)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:227)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:428)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$applyTransaction$1(OzoneManagerStateMachine.java:246)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748){code}
> *Reason for this*
> S3A filesystem when create directory creates an empty file
> *Now entries in Ozone KeyTable after create directory*
> d11/
> d11/d12
> Because of this in OMFileRequest.VerifyInFilesPath fails with
> FILE_EXISTS_IN_GIVEN_PATH because d11/d12 is considered as file not a
> directory. (As in ozone currently, directories end with trailing "/")
> So, when d11/d12/file is created, we check parent exists, now d11/d12 is
> considered as file and fails with NOT_A_FILE
> When disabled it works fine, as when disabled during key create we do not
> check any filesystem semantics and also does not create intermediate
> directories.
> {code:java}
> [root@bvoz-1 ~]# hdfs dfs -mkdir -p s3a://b12345/d11/d12
> [root@bvoz-1 ~]# hdfs dfs -put /etc/hadoop/conf/ozone-site.xml
> s3a://b12345/d11/d12/k1
> [root@bvoz-1 ~]# hdfs dfs -ls s3a://b12345/d11/d12
> Found 1 items
> -rw-rw-rw- 1 systest systest 2373 2020-09-04 04:45
> s3a://b12345/d11/d12/k1
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-4124) Verify parent directories and file system semantics during initiate MPU
[ https://issues.apache.org/jira/browse/HDDS-4124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal resolved HDDS-4124. - Resolution: Won't Fix > Verify parent directories and file system semantics during initiate MPU > --- > > Key: HDDS-4124 > URL: https://issues.apache.org/jira/browse/HDDS-4124 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > > Now for each and every part upload, we check parent directories exist and > create missing directories if not exist. This is an unnecessary check during > each and every part upload. This can be moved to initiate Multipart upload > which is done only one time. > With this we have we don't need any checks during each and every part upload > operation, which also improve speed of MPU when > ozone.om.enable.filesystem.paths is enabled. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-4209) S3A Filesystem does not work with Ozone S3
[
https://issues.apache.org/jira/browse/HDDS-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal resolved HDDS-4209.
-
Resolution: Won't Fix
S3A is not supported in file system compatibility mode.
> S3A Filesystem does not work with Ozone S3
> --
>
> Key: HDDS-4209
> URL: https://issues.apache.org/jira/browse/HDDS-4209
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Blocker
> Labels: OzoneS3, S3A, pull-request-available
>
> When *ozone.om.enable.filesystem.paths* is enabled
>
> hdfs dfs -mkdir -p s3a://b12345/d11/d12 -> Success
> hdfs dfs -put /tmp/file1 s3a://b12345/d11/d12/file1 -> fails with below error
>
> {code:java}
> 2020-09-04 03:53:51,377 ERROR
> org.apache.hadoop.ozone.om.request.key.OMKeyCreateRequest: Key creation
> failed. Volume:s3v, Bucket:b1234, Keyd11/d12/file1._COPYING_. Exception:{}
> NOT_A_FILE org.apache.hadoop.ozone.om.exceptions.OMException: Can not create
> file: cp/k1._COPYING_ as there is already file in the given path
> at
> org.apache.hadoop.ozone.om.request.key.OMKeyCreateRequest.validateAndUpdateCache(OMKeyCreateRequest.java:256)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:227)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:428)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$applyTransaction$1(OzoneManagerStateMachine.java:246)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748){code}
> *Reason for this*
> S3A filesystem when create directory creates an empty file
> *Now entries in Ozone KeyTable after create directory*
> d11/
> d11/d12
> Because of this in OMFileRequest.VerifyInFilesPath fails with
> FILE_EXISTS_IN_GIVEN_PATH because d11/d12 is considered as file not a
> directory. (As in ozone currently, directories end with trailing "/")
> So, when d11/d12/file is created, we check parent exists, now d11/d12 is
> considered as file and fails with NOT_A_FILE
> When disabled it works fine, as when disabled during key create we do not
> check any filesystem semantics and also does not create intermediate
> directories.
> {code:java}
> [root@bvoz-1 ~]# hdfs dfs -mkdir -p s3a://b12345/d11/d12
> [root@bvoz-1 ~]# hdfs dfs -put /etc/hadoop/conf/ozone-site.xml
> s3a://b12345/d11/d12/k1
> [root@bvoz-1 ~]# hdfs dfs -ls s3a://b12345/d11/d12
> Found 1 items
> -rw-rw-rw- 1 systest systest 2373 2020-09-04 04:45
> s3a://b12345/d11/d12/k1
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3102) ozone getconf command should use the GenericCli parent class
[ https://issues.apache.org/jira/browse/HDDS-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17208835#comment-17208835 ] Arpit Agarwal commented on HDDS-3102: - This was an incompatible change [~elek]. > ozone getconf command should use the GenericCli parent class > > > Key: HDDS-3102 > URL: https://issues.apache.org/jira/browse/HDDS-3102 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone CLI >Reporter: Marton Elek >Assignee: Rui Wang >Priority: Major > Labels: incompatible, newbie, pull-request-available > Fix For: 1.1.0 > > > org.apache.hadoop.ozone.freon.OzoneGetCOnf implements a tool to print out > current configuration values > With all the other CLI tools we already started to use picocli and the > GenericCli parent class. > To provide better user experience we should migrate the tool to use > GenericCli (+move it to the tools project + remove freon from the package > name) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3102) ozone getconf command should use the GenericCli parent class
[ https://issues.apache.org/jira/browse/HDDS-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3102: Labels: incompatible newbie pull-request-available (was: newbie pull-request-available) > ozone getconf command should use the GenericCli parent class > > > Key: HDDS-3102 > URL: https://issues.apache.org/jira/browse/HDDS-3102 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone CLI >Reporter: Marton Elek >Assignee: Rui Wang >Priority: Major > Labels: incompatible, newbie, pull-request-available > Fix For: 1.1.0 > > > org.apache.hadoop.ozone.freon.OzoneGetCOnf implements a tool to print out > current configuration values > With all the other CLI tools we already started to use picocli and the > GenericCli parent class. > To provide better user experience we should migrate the tool to use > GenericCli (+move it to the tools project + remove freon from the package > name) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4310) Ozone getconf broke the compatibility
[
https://issues.apache.org/jira/browse/HDDS-4310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-4310:
Description:
Currently ozone getconf '-confKey' does not work as 'HDDS-3102' removed the
need of prepending - with options.
{code:java}
RUNNING: ozone getconf -confKey ozone.om.service.ids 2020-10-05
19:10:09,110|INFO|MainThread|machine.py:180 -
run()||GUID=8644ce5b-cfe9-4e6b-9b3f-55c29c950489|Unknown options: '-confKey',
'ozone.om.service.ids' 2020-10-05 19:10:09,111|INFO|MainThread|machine.py:180 -
run()||GUID=8644ce5b-cfe9-4e6b-9b3f-55c29c950489|Possible solutions: -conf
{code}
There are some users which did the automation with the commands and this change
broke them.
was:
Currently ozone getconf '-confKey' does not work as 'HDDS-3102' removed the
need of prepending - with options.
{code:java}
RUNNING: /opt/cloudera/parcels/CDH/bin/ozone getconf -confKey
ozone.om.service.ids 2020-10-05 19:10:09,110|INFO|MainThread|machine.py:180 -
run()||GUID=8644ce5b-cfe9-4e6b-9b3f-55c29c950489|Unknown options: '-confKey',
'ozone.om.service.ids' 2020-10-05 19:10:09,111|INFO|MainThread|machine.py:180 -
run()||GUID=8644ce5b-cfe9-4e6b-9b3f-55c29c950489|Possible solutions: -conf
{code}
There are some users which did the automation with the commands and this change
broke them.
> Ozone getconf broke the compatibility
> -
>
> Key: HDDS-4310
> URL: https://issues.apache.org/jira/browse/HDDS-4310
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone CLI
>Affects Versions: 1.0.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Fix For: 1.1.0
>
>
> Currently ozone getconf '-confKey' does not work as 'HDDS-3102' removed the
> need of prepending - with options.
> {code:java}
> RUNNING: ozone getconf -confKey ozone.om.service.ids 2020-10-05
> 19:10:09,110|INFO|MainThread|machine.py:180 -
> run()||GUID=8644ce5b-cfe9-4e6b-9b3f-55c29c950489|Unknown options: '-confKey',
> 'ozone.om.service.ids' 2020-10-05 19:10:09,111|INFO|MainThread|machine.py:180
> - run()||GUID=8644ce5b-cfe9-4e6b-9b3f-55c29c950489|Possible solutions: -conf
> {code}
> There are some users which did the automation with the commands and this
> change broke them.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4155) Directory and filename can end up with same name in a path
[
https://issues.apache.org/jira/browse/HDDS-4155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192408#comment-17192408
]
Arpit Agarwal commented on HDDS-4155:
-
{quote}Sorry, I am not sure what does this answer mean exactly. Can you please
define what ofs/o3fs behaviour is suggested by you?
{quote}
I am proposing don't guarantee any ofs/o3fs behavior when the filesystem
compatibility is off. i.e. state that it is undefined.
> Directory and filename can end up with same name in a path
> --
>
> Key: HDDS-4155
> URL: https://issues.apache.org/jira/browse/HDDS-4155
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
>
> Scenario:
> Create Key via S3, and Create Directory through Fs.
> # open key -> /a/b/c
> # CreateDirectory -> /a/b/c
> # CommitKey -> /a/b/c
> So, now in Ozone we will have directory and file with name "c"
> When created through Fs interface.
> # create file -> /a/b/c
> # CreateDirectory -> /a/b/c
> # CommitKey -> /a/b/c
> So, now in Ozone we will have directory and file with name "c"
>
> # InitiateMPU /a/b/c
> # Create Part1 /a/b/c
> # Commit Part1 /a/b/c
> # Create Directory /a/b/c
> # Complete MPU /a/b/c
> So, now in Ozone, we will have directory and file with name "c". In MPU this
> is one example scenario.
>
> Few proposals/ideas to solve this:
> # Check during commit whether a directory already exists with same name. But
> disadvantage is after user uploads the entire data during last stage we fail.
> (File system with create in progress acts similarly. Scenario: 1. vi t1 2.
> mkdir t1 3. Save t1: (Fail:"t1" is a directory)
> # During create directory check are there any open key creation with same
> name and fail.
>
> Any of the above approaches are not final, this Jira is opened to discuss
> this issue and come up with solution.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4155) Directory and filename can end up with same name in a path
[ https://issues.apache.org/jira/browse/HDDS-4155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192265#comment-17192265 ] Arpit Agarwal commented on HDDS-4155: - bq. s3a can handle both dir/file with the same name s3a does not handle it. It just displays both if they exist. > Directory and filename can end up with same name in a path > -- > > Key: HDDS-4155 > URL: https://issues.apache.org/jira/browse/HDDS-4155 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > > Scenario: > Create Key via S3, and Create Directory through Fs. > # open key -> /a/b/c > # CreateDirectory -> /a/b/c > # CommitKey -> /a/b/c > So, now in Ozone we will have directory and file with name "c" > When created through Fs interface. > # create file -> /a/b/c > # CreateDirectory -> /a/b/c > # CommitKey -> /a/b/c > So, now in Ozone we will have directory and file with name "c" > > # InitiateMPU /a/b/c > # Create Part1 /a/b/c > # Commit Part1 /a/b/c > # Create Directory /a/b/c > # Complete MPU /a/b/c > So, now in Ozone, we will have directory and file with name "c". In MPU this > is one example scenario. > > Few proposals/ideas to solve this: > # Check during commit whether a directory already exists with same name. But > disadvantage is after user uploads the entire data during last stage we fail. > (File system with create in progress acts similarly. Scenario: 1. vi t1 2. > mkdir t1 3. Save t1: (Fail:"t1" is a directory) > # During create directory check are there any open key creation with same > name and fail. > > Any of the above approaches are not final, this Jira is opened to discuss > this issue and come up with solution. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-4164) OM client request fails with "failed to commit as key is not found in OpenKey table"
[
https://issues.apache.org/jira/browse/HDDS-4164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal reassigned HDDS-4164:
---
Assignee: Bharat Viswanadham
> OM client request fails with "failed to commit as key is not found in OpenKey
> table"
>
>
> Key: HDDS-4164
> URL: https://issues.apache.org/jira/browse/HDDS-4164
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: OM HA
>Reporter: Lokesh Jain
>Assignee: Bharat Viswanadham
>Priority: Blocker
>
> OM client request fails with "failed to commit as key is not found in OpenKey
> table"
> {code:java}
> 20/08/28 03:21:53 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28868 $Proxy17.submitRequest over
> nodeId=om3,nodeAddress=vc1330.halxg.cloudera.com:9862
> 20/08/28 03:21:53 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28870 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:53 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28869 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:54 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28871 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:54 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28872 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:54 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28866 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:54 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28867 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:54 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28874 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:54 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28875 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:54 ERROR freon.BaseFreonGenerator: Error on executing task
> 14424
> KEY_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Failed to
> commit key, as /vol1/bucket1/akjkdz4hoj/14424/104766512182520809entry is not
> found in the OpenKey table
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:593)
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.commitKey(OzoneManagerProtocolClientSideTranslatorPB.java:650)
> at
> org.apache.hadoop.ozone.client.io.BlockOutputStreamEntryPool.commitKey(BlockOutputStreamEntryPool.java:306)
> at
> org.apache.hadoop.ozone.client.io.KeyOutputStream.close(KeyOutputStream.java:514)
> at
> org.apache.hadoop.ozone.client.io.OzoneOutputStream.close(OzoneOutputStream.java:60)
> at
> org.apache.hadoop.ozone.freon.OzoneClientKeyGenerator.lambda$createKey$0(OzoneClientKeyGenerator.java:118)
> at com.codahale.metrics.Timer.time(Timer.java:101)
> at
> org.apache.hadoop.ozone.freon.OzoneClientKeyGenerator.createKey(OzoneClientKeyGenerator.java:113)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.tryNextTask(BaseFreonGenerator.java:178)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.taskLoop(BaseFreonGenerator.java:167)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.lambda$startTaskRunners$0(BaseFreonGenerator.java:150)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4155) Directory and filename can end up with same name in a path
[ https://issues.apache.org/jira/browse/HDDS-4155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191746#comment-17191746 ] Arpit Agarwal commented on HDDS-4155: - We will not "support" them, they will just exist. With strict S3 compatibility there is no guarantee that ofs/o3fs will work correctly. > Directory and filename can end up with same name in a path > -- > > Key: HDDS-4155 > URL: https://issues.apache.org/jira/browse/HDDS-4155 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > > Scenario: > Create Key via S3, and Create Directory through Fs. > # open key -> /a/b/c > # CreateDirectory -> /a/b/c > # CommitKey -> /a/b/c > So, now in Ozone we will have directory and file with name "c" > When created through Fs interface. > # create file -> /a/b/c > # CreateDirectory -> /a/b/c > # CommitKey -> /a/b/c > So, now in Ozone we will have directory and file with name "c" > > # InitiateMPU /a/b/c > # Create Part1 /a/b/c > # Commit Part1 /a/b/c > # Create Directory /a/b/c > # Complete MPU /a/b/c > So, now in Ozone, we will have directory and file with name "c". In MPU this > is one example scenario. > > Few proposals/ideas to solve this: > # Check during commit whether a directory already exists with same name. But > disadvantage is after user uploads the entire data during last stage we fail. > (File system with create in progress acts similarly. Scenario: 1. vi t1 2. > mkdir t1 3. Save t1: (Fail:"t1" is a directory) > # During create directory check are there any open key creation with same > name and fail. > > Any of the above approaches are not final, this Jira is opened to discuss > this issue and come up with solution. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4155) Directory and filename can end up with same name in a path
[ https://issues.apache.org/jira/browse/HDDS-4155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191059#comment-17191059 ] Arpit Agarwal commented on HDDS-4155: - Which is why we want to support two modes, pure S3 compatibility or HCFS compatibility. If the filesystem compatibility flag is off the user is welcome to create both entries. > Directory and filename can end up with same name in a path > -- > > Key: HDDS-4155 > URL: https://issues.apache.org/jira/browse/HDDS-4155 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > > Scenario: > Create Key via S3, and Create Directory through Fs. > # open key -> /a/b/c > # CreateDirectory -> /a/b/c > # CommitKey -> /a/b/c > So, now in Ozone we will have directory and file with name "c" > When created through Fs interface. > # create file -> /a/b/c > # CreateDirectory -> /a/b/c > # CommitKey -> /a/b/c > So, now in Ozone we will have directory and file with name "c" > > # InitiateMPU /a/b/c > # Create Part1 /a/b/c > # Commit Part1 /a/b/c > # Create Directory /a/b/c > # Complete MPU /a/b/c > So, now in Ozone, we will have directory and file with name "c". In MPU this > is one example scenario. > > Few proposals/ideas to solve this: > # Check during commit whether a directory already exists with same name. But > disadvantage is after user uploads the entire data during last stage we fail. > (File system with create in progress acts similarly. Scenario: 1. vi t1 2. > mkdir t1 3. Save t1: (Fail:"t1" is a directory) > # During create directory check are there any open key creation with same > name and fail. > > Any of the above approaches are not final, this Jira is opened to discuss > this issue and come up with solution. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4155) Directory and filename can end up with same name in a path
[
https://issues.apache.org/jira/browse/HDDS-4155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190913#comment-17190913
]
Arpit Agarwal commented on HDDS-4155:
-
{quote}It seems to be very strange for me as this is something which should be
avoid with posix fs, but if hadoop can support it...
{quote}
Marton I am really surprised that you are asking this as a serious question.
S3A does not _"support it"._ It is not a cool exciting feature in S3A. It is an
unfixable bug because S3A cannot control what goes on at the backend which is
owned by Amazon and they are forced to display both, if they exist. We need not
emulate S3A bugs.
> Directory and filename can end up with same name in a path
> --
>
> Key: HDDS-4155
> URL: https://issues.apache.org/jira/browse/HDDS-4155
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
>
> Scenario:
> Create Key via S3, and Create Directory through Fs.
> # open key -> /a/b/c
> # CreateDirectory -> /a/b/c
> # CommitKey -> /a/b/c
> So, now in Ozone we will have directory and file with name "c"
> When created through Fs interface.
> # create file -> /a/b/c
> # CreateDirectory -> /a/b/c
> # CommitKey -> /a/b/c
> So, now in Ozone we will have directory and file with name "c"
>
> # InitiateMPU /a/b/c
> # Create Part1 /a/b/c
> # Commit Part1 /a/b/c
> # Create Directory /a/b/c
> # Complete MPU /a/b/c
> So, now in Ozone, we will have directory and file with name "c". In MPU this
> is one example scenario.
>
> Few proposals/ideas to solve this:
> # Check during commit whether a directory already exists with same name. But
> disadvantage is after user uploads the entire data during last stage we fail.
> (File system with create in progress acts similarly. Scenario: 1. vi t1 2.
> mkdir t1 3. Save t1: (Fail:"t1" is a directory)
> # During create directory check are there any open key creation with same
> name and fail.
>
> Any of the above approaches are not final, this Jira is opened to discuss
> this issue and come up with solution.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4147) Add OFS to FileSystem META-INF
[
https://issues.apache.org/jira/browse/HDDS-4147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-4147:
Fix Version/s: 1.1.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Add OFS to FileSystem META-INF
> --
>
> Key: HDDS-4147
> URL: https://issues.apache.org/jira/browse/HDDS-4147
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.1.0
>
>
> So that {{fs.ofs.impl}} won't have to be explicitly set in core-site.xml.
> Derived from HDDS-3805.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-4097) S3/Ozone Filesystem inter-op
[
https://issues.apache.org/jira/browse/HDDS-4097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17189624#comment-17189624
]
Arpit Agarwal edited comment on HDDS-4097 at 9/2/20, 6:30 PM:
--
Thanks for the doc, I will take a look.
However this is only partially related to the original proposal from Bharat.
The proposal here states:
# There is a set of use-cases where files are ingested via S3 and accessed via
HCFS. E.g. someone ingesting logs into Hive external tables via fluentd/S3. For
those use cases, the key names must look like valid paths and we need to ensure
they are checked and normalized appropriately and directory prefixes created.
# There is another set of pure object store use cases where the paths are
random strings and may have arbitrary characters including {{/}} or other
characters which are not valid in an FS path. Ingestion of such keys should be
successful and must not fail.
These are mutually exclusive use cases. In case #2, you cannot ingest invalid
paths while maintaining filesystem integrity. The point of providing a config
key or switch is to allow the user to choose, with the current default being
#2. This umbrella task is proposing to improve the first use case. What you are
proposing is for the second use case and it is a completely separate
discussion. So what's the overlap?
was (Author: arpitagarwal):
Thanks for the doc, I will take a look.
However this is only partially related to the original proposal from Bharat.
The proposal here states:
# There is a set of use-cases where files are ingested via S3 and accessed via
HCFS. E.g. someone ingesting logs into Hive external tables via fluentd/S3. For
those use cases, the key names must look like valid paths and we need to ensure
they are checked and normalized appropriately and directory prefixes created.
# There is another set of pure object store use cases where the paths are
random strings and may have arbitrary characters including {{/}} or other
characters which are not valid in an FS path. Ingestion of such keys should be
successful and must not fail.
These are mutually exclusive use cases. In case #2, you cannot ingest invalid
paths while maintaining filesystem integrity. The point of providing a config
key or switch is to allow the user to choose, with the current default being
#2. This umbrella task is proposing to fix the first use case. What you are
proposing is for the second use case. So what's the overlap?
> S3/Ozone Filesystem inter-op
>
>
> Key: HDDS-4097
> URL: https://issues.apache.org/jira/browse/HDDS-4097
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: Ozone FileSystem Paths Enabled.docx, Ozone filesystem
> path enabled.xlsx
>
>
> This Jira is to implement changes required to use Ozone buckets when data is
> ingested via S3 and use the bucket/volume via OzoneFileSystem. Initial
> implementation for this is done as part of HDDS-3955. There are few API's
> which have missed the changes during the implementation of HDDS-3955.
> Attached design document which discusses each API, and what changes are
> required.
> Excel sheet has information about each API, from what all interfaces the OM
> API is used, and what changes are required for the API to support
> inter-operability.
> Note: The proposal for delete/rename is still under discussion, not yet
> finalized.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4097) S3/Ozone Filesystem inter-op
[
https://issues.apache.org/jira/browse/HDDS-4097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17189624#comment-17189624
]
Arpit Agarwal commented on HDDS-4097:
-
Thanks for the doc, I will take a look.
However this is only partially related to the original proposal from Bharat.
The proposal here states:
# There is a set of use-cases where files are ingested via S3 and accessed via
HCFS. E.g. someone ingesting logs into Hive external tables via fluentd/S3. For
those use cases, the key names must look like valid paths and we need to ensure
they are checked and normalized appropriately and directory prefixes created.
# There is another set of pure object store use cases where the paths are
random strings and may have arbitrary characters including {{/}} or other
characters which are not valid in an FS path. Ingestion of such keys should be
successful and must not fail.
These are mutually exclusive use cases. In case #2, you cannot ingest invalid
paths while maintaining filesystem integrity. The point of providing a config
key or switch is to allow the user to choose, with the current default being
#2. This umbrella task is proposing to fix the first use case. What you are
proposing is for the second use case. So what's the overlap?
> S3/Ozone Filesystem inter-op
>
>
> Key: HDDS-4097
> URL: https://issues.apache.org/jira/browse/HDDS-4097
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: Ozone FileSystem Paths Enabled.docx, Ozone filesystem
> path enabled.xlsx
>
>
> This Jira is to implement changes required to use Ozone buckets when data is
> ingested via S3 and use the bucket/volume via OzoneFileSystem. Initial
> implementation for this is done as part of HDDS-3955. There are few API's
> which have missed the changes during the implementation of HDDS-3955.
> Attached design document which discusses each API, and what changes are
> required.
> Excel sheet has information about each API, from what all interfaces the OM
> API is used, and what changes are required for the API to support
> inter-operability.
> Note: The proposal for delete/rename is still under discussion, not yet
> finalized.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4097) S3/Ozone Filesystem inter-op
[ https://issues.apache.org/jira/browse/HDDS-4097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188586#comment-17188586 ] Arpit Agarwal commented on HDDS-4097: - If you always create them, then you are basically interpreting key names as filesystem paths, so then they have to normalized and interpreted as paths. There is no middle ground. > S3/Ozone Filesystem inter-op > > > Key: HDDS-4097 > URL: https://issues.apache.org/jira/browse/HDDS-4097 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: Ozone FileSystem Paths Enabled.docx, Ozone filesystem > path enabled.xlsx > > > This Jira is to implement changes required to use Ozone buckets when data is > ingested via S3 and use the bucket/volume via OzoneFileSystem. Initial > implementation for this is done as part of HDDS-3955. There are few API's > which have missed the changes during the implementation of HDDS-3955. > Attached design document which discusses each API, and what changes are > required. > Excel sheet has information about each API, from what all interfaces the OM > API is used, and what changes are required for the API to support > inter-operability. > Note: The proposal for delete/rename is still under discussion, not yet > finalized. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-4097) S3/Ozone Filesystem inter-op
[ https://issues.apache.org/jira/browse/HDDS-4097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188538#comment-17188538 ] Arpit Agarwal edited comment on HDDS-4097 at 9/1/20, 3:12 PM: -- bq. Using simple, acceptable key names (/a/b/c, /a/b/c/d) both s3 and HCFS should work out-of-the box, without any additional settings. Unfortunately there is no way you can guarantee that. A filesystem client will need all the intermediate directories to exist for navigating the tree. was (Author: arpitagarwal): bq. Using simple, acceptable key names (/a/b/c, /a/b/c/d) both s3 and HCFS should work out-of-the box, without any additional settings. Unfortunately there is no way you cannot guarantee that. A filesystem client will need all the intermediate directories to exist for navigating the tree. > S3/Ozone Filesystem inter-op > > > Key: HDDS-4097 > URL: https://issues.apache.org/jira/browse/HDDS-4097 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: Ozone FileSystem Paths Enabled.docx, Ozone filesystem > path enabled.xlsx > > > This Jira is to implement changes required to use Ozone buckets when data is > ingested via S3 and use the bucket/volume via OzoneFileSystem. Initial > implementation for this is done as part of HDDS-3955. There are few API's > which have missed the changes during the implementation of HDDS-3955. > Attached design document which discusses each API, and what changes are > required. > Excel sheet has information about each API, from what all interfaces the OM > API is used, and what changes are required for the API to support > inter-operability. > Note: The proposal for delete/rename is still under discussion, not yet > finalized. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4097) S3/Ozone Filesystem inter-op
[ https://issues.apache.org/jira/browse/HDDS-4097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188538#comment-17188538 ] Arpit Agarwal commented on HDDS-4097: - bq. Using simple, acceptable key names (/a/b/c, /a/b/c/d) both s3 and HCFS should work out-of-the box, without any additional settings. Unfortunately there is no way you cannot guarantee that. A filesystem client will need all the intermediate directories to exist for navigating the tree. > S3/Ozone Filesystem inter-op > > > Key: HDDS-4097 > URL: https://issues.apache.org/jira/browse/HDDS-4097 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: Ozone FileSystem Paths Enabled.docx, Ozone filesystem > path enabled.xlsx > > > This Jira is to implement changes required to use Ozone buckets when data is > ingested via S3 and use the bucket/volume via OzoneFileSystem. Initial > implementation for this is done as part of HDDS-3955. There are few API's > which have missed the changes during the implementation of HDDS-3955. > Attached design document which discusses each API, and what changes are > required. > Excel sheet has information about each API, from what all interfaces the OM > API is used, and what changes are required for the API to support > inter-operability. > Note: The proposal for delete/rename is still under discussion, not yet > finalized. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-4097) S3/Ozone Filesystem inter-op
[ https://issues.apache.org/jira/browse/HDDS-4097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17187892#comment-17187892 ] Arpit Agarwal edited comment on HDDS-4097 at 8/31/20, 5:27 PM: --- bq. Even when the flag is false, intermediate directory creation should be supported, without any check for file system semantics. No, FS interaction with data ingested via S3 should simply not be supported when the flag is off. Let's not try to support some halfway half-baked option, it will be hard to get right. To clarify a bit more, how do you message half support to a potential user of Ozone? The FS side expects a hierarchical directory tree and FS operations to work like HDFS. The S3 side expects a flat key value namespace. It is one or the other, there is no in-between option. was (Author: arpitagarwal): bq. Even when the flag is false, intermediate directory creation should be supported, without any check for file system semantics. No, FS interaction with data ingested via S3 should simply not be supported when the flag is off. Let's not try to support some halfway half-baked option, it will be hard to get right. > S3/Ozone Filesystem inter-op > > > Key: HDDS-4097 > URL: https://issues.apache.org/jira/browse/HDDS-4097 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: Ozone FileSystem Paths Enabled.docx, Ozone filesystem > path enabled.xlsx > > > This Jira is to implement changes required to use Ozone buckets when data is > ingested via S3 and use the bucket/volume via OzoneFileSystem. Initial > implementation for this is done as part of HDDS-3955. There are few API's > which have missed the changes during the implementation of HDDS-3955. > Attached design document which discusses each API, and what changes are > required. > Excel sheet has information about each API, from what all interfaces the OM > API is used, and what changes are required for the API to support > inter-operability. > Note: The proposal for delete/rename is still under discussion, not yet > finalized. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4097) S3/Ozone Filesystem inter-op
[ https://issues.apache.org/jira/browse/HDDS-4097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17187892#comment-17187892 ] Arpit Agarwal commented on HDDS-4097: - bq. Even when the flag is false, intermediate directory creation should be supported, without any check for file system semantics. No, FS interaction with data ingested via S3 should simply not be supported when the flag is off. Let's not try to support some halfway half-baked option, it will be hard to get right. > S3/Ozone Filesystem inter-op > > > Key: HDDS-4097 > URL: https://issues.apache.org/jira/browse/HDDS-4097 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: Ozone FileSystem Paths Enabled.docx, Ozone filesystem > path enabled.xlsx > > > This Jira is to implement changes required to use Ozone buckets when data is > ingested via S3 and use the bucket/volume via OzoneFileSystem. Initial > implementation for this is done as part of HDDS-3955. There are few API's > which have missed the changes during the implementation of HDDS-3955. > Attached design document which discusses each API, and what changes are > required. > Excel sheet has information about each API, from what all interfaces the OM > API is used, and what changes are required for the API to support > inter-operability. > Note: The proposal for delete/rename is still under discussion, not yet > finalized. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4097) S3/Ozone Filesystem inter-op
[ https://issues.apache.org/jira/browse/HDDS-4097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185953#comment-17185953 ] Arpit Agarwal commented on HDDS-4097: - Yes that will be the case if filesystem compatibility flag is not enabled. If someone enables the flag then they are choosing to forego strict AWS compatibility and interpret key names as file paths. [~bharat] can you confirm? > S3/Ozone Filesystem inter-op > > > Key: HDDS-4097 > URL: https://issues.apache.org/jira/browse/HDDS-4097 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: Ozone FileSystem Paths Enabled.docx, Ozone filesystem > path enabled.xlsx > > > This Jira is to implement changes required to use Ozone buckets when data is > ingested via S3 and use the bucket/volume via OzoneFileSystem. Initial > implementation for this is done as part of HDDS-3955. There are few API's > which have missed the changes during the implementation of HDDS-3955. > Attached design document which discusses each API, and what changes are > required. > Excel sheet has information about each API, from what all interfaces the OM > API is used, and what changes are required for the API to support > inter-operability. > Note: The proposal for delete/rename is still under discussion, not yet > finalized. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4097) S3/Ozone Filesystem inter-op
[ https://issues.apache.org/jira/browse/HDDS-4097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185262#comment-17185262 ] Arpit Agarwal commented on HDDS-4097: - bq. I would prefer to keep 100% AWS compatibility even with the new approach, unless we have very strong arguments why is it impossible [~elek] this comment is unclear. What do you mean by keep 100% AWS compatibility? bq. Can you please add more information about this?. In current Ozone we guarantee 100% AWS compatibility and HCFS compatiblity at the same time Not true, HCFS compatibility is incompletely simulated on top of the object store. > S3/Ozone Filesystem inter-op > > > Key: HDDS-4097 > URL: https://issues.apache.org/jira/browse/HDDS-4097 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: Ozone FileSystem Paths Enabled.docx, Ozone filesystem > path enabled.xlsx > > > This Jira is to implement changes required to use Ozone buckets when data is > ingested via S3 and use the bucket/volume via OzoneFileSystem. Initial > implementation for this is done as part of HDDS-3955. There are few API's > which have missed the changes during the implementation of HDDS-3955. > Attached design document which discusses each API, and what changes are > required. > Excel sheet has information about each API, from what all interfaces the OM > API is used, and what changes are required for the API to support > inter-operability. > Note: The proposal for delete/rename is still under discussion, not yet > finalized. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4097) S3/Ozone Filesystem inter-op
[ https://issues.apache.org/jira/browse/HDDS-4097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-4097: Target Version/s: 0.7.0 > S3/Ozone Filesystem inter-op > > > Key: HDDS-4097 > URL: https://issues.apache.org/jira/browse/HDDS-4097 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: Ozone FileSystem Paths Enabled.docx, Ozone filesystem > path enabled.xlsx > > > This Jira is to implement changes required to use Ozone buckets when data is > ingested via S3. Initial implementation for this is done as part of > HDDS-3955. There are few API's which have missed the changes during the > implementation of HDDS-3955. > Attached design document which discusses about each API, and what changes are > required. > Excel sheet has information about each API. > Note: The proposal for delete/rename is still under discussion, not yet > finalized. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4097) S3/Ozone Filesystem inter-op
[ https://issues.apache.org/jira/browse/HDDS-4097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-4097: Issue Type: New Feature (was: Bug) > S3/Ozone Filesystem inter-op > > > Key: HDDS-4097 > URL: https://issues.apache.org/jira/browse/HDDS-4097 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: Ozone FileSystem Paths Enabled.docx, Ozone filesystem > path enabled.xlsx > > > This Jira is to implement changes required to use Ozone buckets when data is > ingested via S3. Initial implementation for this is done as part of > HDDS-3955. There are few API's which have missed the changes during the > implementation of HDDS-3955. > Attached design document which discusses about each API, and what changes are > required. > Excel sheet has information about each API. > Note: The proposal for delete/rename is still under discussion, not yet > finalized. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4058) Wrong use of AtomicBoolean in HddsDatanodeService
[
https://issues.apache.org/jira/browse/HDDS-4058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-4058:
Target Version/s: 0.6.0 (was: 0.7.0)
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Wrong use of AtomicBoolean in HddsDatanodeService
> -
>
> Key: HDDS-4058
> URL: https://issues.apache.org/jira/browse/HDDS-4058
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Minor
> Labels: pull-request-available
>
> * {{AtomicBoolean isStopped}} should be {{final}}, not {{volatile}}, since
> the reference is not being changed
> * {{stop()}} should use atomic {{getAndSet()}} instead of {{get()}} followed
> by {{set()}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2949) mkdir : store directory entries in a separate table
[ https://issues.apache.org/jira/browse/HDDS-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-2949: Status: Patch Available (was: Open) > mkdir : store directory entries in a separate table > --- > > Key: HDDS-2949 > URL: https://issues.apache.org/jira/browse/HDDS-2949 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Supratim Deka >Assignee: Rakesh Radhakrishnan >Priority: Major > Labels: backward-incompatible, pull-request-available > > As of HDDS-2940, all the directories from the path prefix get created as > entries in the key table. as per the namespace proposal attached to > HDDS-2939, directory entries need to be stored in a separate "directory" > table. Files will continue to be stored in the key table, which can be > thought of as the "file" table. > The advantage of a separate directory table is to make directory lookup more > efficient - the entire table would fit into memory for a typical file based > dataset. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3976) KeyValueBlockIterator#nextBlock skips valid blocks
[ https://issues.apache.org/jira/browse/HDDS-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3976: Status: Patch Available (was: Open) > KeyValueBlockIterator#nextBlock skips valid blocks > -- > > Key: HDDS-3976 > URL: https://issues.apache.org/jira/browse/HDDS-3976 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Ethan Rose >Assignee: Ethan Rose >Priority: Major > Labels: pull-request-available > > HDDS-3854 fixed a bug in KeyValueBlockIterator#hasNext, but introduced > another one in KeyValueBlockIterator#nextBlock, which depends on the behavior > of that method. When the first key encountered does not pass the filter, the > internal nextBlock field is never intialized. Then a call to nextBlock() > results in call to hasNext() which returns true, which recursively calls > nextBlock(), again calling hasNext(), etc until the end of the set is reached > and an exception is thrown. This skips all valid keys that may occur past the > first invalid key. > Additionally, the current implementation of KeyValueBlockIterator#seekLast > depends on the internal RocksDB iterators seekLast() method, which will skip > to the last key in the DB regardless of whether it matches the filter or not. > This could be different from last key according to the filter. > This bug was identified while working on HDDS-3869, which adds a strong > typing layer before objects are serialized into RocksDB for datanode. Due to > RocksDB internals, this changes the database layout so that all prefixed keys > are returned at the beginning of the key set, instead of in the end. Since > the original layout returned all prefixed keys at the end of the key set, > this bug was not evident in any of the original unit tests, since the > behavior described above could not occur. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-4026) Dir rename failed when sets 'ozone.om.enable.filesystem.paths' to true
[
https://issues.apache.org/jira/browse/HDDS-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal resolved HDDS-4026.
-
Fix Version/s: 0.6.0
Target Version/s: (was: 0.6.0)
Resolution: Fixed
> Dir rename failed when sets 'ozone.om.enable.filesystem.paths' to true
> --
>
> Key: HDDS-4026
> URL: https://issues.apache.org/jira/browse/HDDS-4026
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Rakesh Radhakrishnan
>Assignee: Bharat Viswanadham
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 0.6.0
>
>
> Sets ozone.om.enable.filesystem.paths=true, then starts the Ozone cluster.
> {code:java}
> [root~]$ ozone fs -mkdir o3fs://bucket2.vol2.ozone1/subdir2
> [root~]$ ozone fs -mv o3fs://bucket2.vol2.ozone1/subdir2
> o3fs://bucket2.vol2.ozone1/subdir2-renamedmv: Key not found
> /vol2/bucket2/subdir2
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-4026) Dir rename failed when sets 'ozone.om.enable.filesystem.paths' to true
[
https://issues.apache.org/jira/browse/HDDS-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal reassigned HDDS-4026:
---
Assignee: Bharat Viswanadham
> Dir rename failed when sets 'ozone.om.enable.filesystem.paths' to true
> --
>
> Key: HDDS-4026
> URL: https://issues.apache.org/jira/browse/HDDS-4026
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Rakesh Radhakrishnan
>Assignee: Bharat Viswanadham
>Priority: Blocker
>
> Sets ozone.om.enable.filesystem.paths=true, then starts the Ozone cluster.
> {code:java}
> [root~]$ ozone fs -mkdir o3fs://bucket2.vol2.ozone1/subdir2
> [root~]$ ozone fs -mv o3fs://bucket2.vol2.ozone1/subdir2
> o3fs://bucket2.vol2.ozone1/subdir2-renamedmv: Key not found
> /vol2/bucket2/subdir2
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3999) OM NPE and shutdown while S3MultipartUploadCommitPartResponse#checkAndUpdateDB
[ https://issues.apache.org/jira/browse/HDDS-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal resolved HDDS-3999. - Fix Version/s: 0.6.0 Target Version/s: (was: 0.6.0) Resolution: Fixed > OM NPE and shutdown while S3MultipartUploadCommitPartResponse#checkAndUpdateDB > -- > > Key: HDDS-3999 > URL: https://issues.apache.org/jira/browse/HDDS-3999 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Affects Versions: 0.6.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Blocker > Labels: pull-request-available > Fix For: 0.6.0 > > Attachments: OM-NPE-Full.log, screenshot-1.png, test-repro.patch > > > The related bad code. > !screenshot-1.png! > The following is the part log of OM. If you want to see the full log, you can > download the attachment. > 2020-07-20 16:28:56,395 ERROR ratis.OzoneManagerDoubleBuffer > (ExitUtils.java:terminate(133)) - Terminating with exit status 2: > OMDoubleBuffer flush threadOMDoubleBufferFlushThreadencountered Throwable > error > java.lang.NullPointerException > at > org.apache.hadoop.ozone.om.response.s3.multipart.S3MultipartUploadCommitPartResponse.checkAndUpdateDB(S3MultipartUploadCommitPartResponse.java:99) > at > org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.lambda$null$0(OzoneManagerDoubleBuffer.java:256) > at > org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatchWithTrace(OzoneManagerDoubleBuffer.java:201) > at > org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.lambda$flushTransactions$1(OzoneManagerDoubleBuffer.java:254) > at java.util.Iterator.forEachRemaining(Iterator.java:116) > at > org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:250) > at java.lang.Thread.run(Thread.java:748) > 2020-07-20 16:28:56,426 INFO om.OzoneManagerStarter > (StringUtils.java:lambda$startupShutdownMessage$0(124)) - SHUTDOWN_MSG: > / > SHUTDOWN_MSG: Shutting down OzoneManager at BAOLOONGMAO-MB0/10.78.32.49 > / -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4026) Dir rename failed when sets 'ozone.om.enable.filesystem.paths' to true
[
https://issues.apache.org/jira/browse/HDDS-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-4026:
Target Version/s: 0.6.0
Priority: Blocker (was: Major)
> Dir rename failed when sets 'ozone.om.enable.filesystem.paths' to true
> --
>
> Key: HDDS-4026
> URL: https://issues.apache.org/jira/browse/HDDS-4026
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Rakesh Radhakrishnan
>Priority: Blocker
>
> Sets ozone.om.enable.filesystem.paths=true, then starts the Ozone cluster.
> {code:java}
> [root~]$ ozone fs -mkdir o3fs://bucket2.vol2.ozone1/subdir2
> [root~]$ ozone fs -mv o3fs://bucket2.vol2.ozone1/subdir2
> o3fs://bucket2.vol2.ozone1/subdir2-renamedmv: Key not found
> /vol2/bucket2/subdir2
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-4006) Disallow MPU on encrypted buckets.
[ https://issues.apache.org/jira/browse/HDDS-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal resolved HDDS-4006. - Fix Version/s: 0.6.0 Resolution: Fixed > Disallow MPU on encrypted buckets. > -- > > Key: HDDS-4006 > URL: https://issues.apache.org/jira/browse/HDDS-4006 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Blocker > Labels: pull-request-available > Fix For: 0.6.0 > > > With HDDS-3612 buckets created via ozone are also accessible via S3. > This has caused a problem when the bucket is encrypted, the keys are not > encrypted on disk. > *2 Issues:* > 1. On OM, for each part a new encryption info is generated. During complete > Multipart upload, the encryption info is not stored in KeyInfo. > 2. On the client, for part upload, the encryption info is silently ignored. > If we don't throw an error, on an encrypted bucket, key data is not encrypted > on disks. > For 0.6.0 release, we can mark this as not supported, and this will be fixed > in next release by HDDS-4005 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4008) Recon should fallback to ozone.om.service.ids when the internal service id is not defined.
[ https://issues.apache.org/jira/browse/HDDS-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-4008: Target Version/s: 0.6.0 Priority: Blocker (was: Critical) > Recon should fallback to ozone.om.service.ids when the internal service id is > not defined. > -- > > Key: HDDS-4008 > URL: https://issues.apache.org/jira/browse/HDDS-4008 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Recon >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Blocker > Labels: pull-request-available > > Recon connects to OM via RPC using the "ozone.om.internal.service.id" to get > updates. If the above config is not defined, but the ozone.om.service.ids is > defined, Recon should use the latter as a fallback. Currently, a single Recon > instance supports only 1 OM HA cluster at a time. Hence, if multiple ids are > defined, Recon will pick the first. > Thanks to [~vivekratnavel] for reporting the issue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4008) Recon should fallback to ozone.om.service.ids when the internal service id is not defined.
[ https://issues.apache.org/jira/browse/HDDS-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163067#comment-17163067 ] Arpit Agarwal commented on HDDS-4008: - bq. Hence, if multiple ids are defined, Recon will pick the first. If there are multiple service IDs and internal service ID is not defined, then Recon startup should fail. > Recon should fallback to ozone.om.service.ids when the internal service id is > not defined. > -- > > Key: HDDS-4008 > URL: https://issues.apache.org/jira/browse/HDDS-4008 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Recon >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Critical > > Recon connects to OM via RPC using the "ozone.om.internal.service.id" to get > updates. If the above config is not defined, but the ozone.om.service.ids is > defined, Recon should use the latter as a fallback. Currently, a single Recon > instance supports only 1 OM HA cluster at a time. Hence, if multiple ids are > defined, Recon will pick the first. > Thanks to [~vivekratnavel] for reporting the issue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3969) Validate KeyNames created in FileSystem requests.
[ https://issues.apache.org/jira/browse/HDDS-3969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal resolved HDDS-3969. - Fix Version/s: 0.6.0 Resolution: Fixed > Validate KeyNames created in FileSystem requests. > - > > Key: HDDS-3969 > URL: https://issues.apache.org/jira/browse/HDDS-3969 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.6.0 > > > This jira is to validate KeyNames which are created with OzoneFileSystem. > Similar to how hdfs handles using DFSUtil. isValidName() -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3993) Create volume required for S3G during OM startup
[ https://issues.apache.org/jira/browse/HDDS-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal resolved HDDS-3993. - Fix Version/s: 0.6.0 Resolution: Fixed > Create volume required for S3G during OM startup > > > Key: HDDS-3993 > URL: https://issues.apache.org/jira/browse/HDDS-3993 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.6.0 > > > Create volume required for S3G operations during OM startup -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3982) Disable moveToTrash in o3fs and ofs temporarily
[
https://issues.apache.org/jira/browse/HDDS-3982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3982:
Fix Version/s: (was: 0.7.0)
0.6.0
> Disable moveToTrash in o3fs and ofs temporarily
> ---
>
> Key: HDDS-3982
> URL: https://issues.apache.org/jira/browse/HDDS-3982
> Project: Hadoop Distributed Data Store
> Issue Type: Task
> Components: Ozone Client
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.6.0
>
>
> A proper server-side trash cleanup solution (HDDS-3915) might not land any
> time soon.
> This jira aims to completely disable "move to trash" when a client is
> deleting files and {{fs.trash.interval > 0}} by intercepting the deprecated
> {{fs.rename(src, dst, options)}} call used by
> {{TrashPolicyDefault#moveToTrash}}.
> This should be reverted when trash cleanup is implemented.
> CC [~arp] and [~msingh]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3987) Encrypted bucket creation failed with INVALID_REQUEST Encryption cannot be set for bucket links
[
https://issues.apache.org/jira/browse/HDDS-3987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3987:
Fix Version/s: 0.6.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
+1. Merged via Gerrit. Thanks [~nilotpalnandi] and [~adoroszlai].
> Encrypted bucket creation failed with INVALID_REQUEST Encryption cannot be
> set for bucket links
> ---
>
> Key: HDDS-3987
> URL: https://issues.apache.org/jira/browse/HDDS-3987
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Affects Versions: 0.7.0
>Reporter: Nilotpal Nandi
>Assignee: Attila Doroszlai
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 0.6.0
>
>
> Bucket creation with encrypted key fails.
> Steps:
> # Created encryption key
> # Created volume
> # Tried to create bucket with encryption key
> Result:
> {code}
> INVALID_REQUEST Encryption cannot be set for bucket links
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3955) Unable to list intermediate paths on keys created using S3G.
[ https://issues.apache.org/jira/browse/HDDS-3955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3955: Resolution: Fixed Status: Resolved (was: Patch Available) +1 merged via Gerrit. Thanks for the contribution [~bharat]. > Unable to list intermediate paths on keys created using S3G. > > > Key: HDDS-3955 > URL: https://issues.apache.org/jira/browse/HDDS-3955 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Bharat Viswanadham >Priority: Blocker > Labels: pull-request-available > > Keys created via the S3 Gateway currently use the createKey OM API to create > the ozone key. Hence, when using a hdfs client to list intermediate > directories in the key, OM returns key not found error. This was encountered > while using fluentd to write Hive logs to Ozone via the s3 gateway. > cc [~bharat] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3955) Unable to list intermediate paths on keys created using S3G.
[ https://issues.apache.org/jira/browse/HDDS-3955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3955: Status: Patch Available (was: Open) > Unable to list intermediate paths on keys created using S3G. > > > Key: HDDS-3955 > URL: https://issues.apache.org/jira/browse/HDDS-3955 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Bharat Viswanadham >Priority: Blocker > Labels: pull-request-available > > Keys created via the S3 Gateway currently use the createKey OM API to create > the ozone key. Hence, when using a hdfs client to list intermediate > directories in the key, OM returns key not found error. This was encountered > while using fluentd to write Hive logs to Ozone via the s3 gateway. > cc [~bharat] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3354) OM HA replay optimization
[ https://issues.apache.org/jira/browse/HDDS-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3354: Resolution: Done Status: Resolved (was: Patch Available) Resolving as all the sub-tasks are complete. > OM HA replay optimization > - > > Key: HDDS-3354 > URL: https://issues.apache.org/jira/browse/HDDS-3354 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: OM HA, Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: Triaged > Attachments: OM HA Replay.pdf, Screen Shot 2020-05-20 at 1.28.48 > PM.png > > > This Jira is to improve the OM HA replay scenario. > Attached the design document which discusses about the proposal and issue in > detail. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3747) Remove the redundancy if condition code in ListPipelinesSubcommand
[ https://issues.apache.org/jira/browse/HDDS-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17154712#comment-17154712 ] Arpit Agarwal commented on HDDS-3747: - Done. > Remove the redundancy if condition code in ListPipelinesSubcommand > -- > > Key: HDDS-3747 > URL: https://issues.apache.org/jira/browse/HDDS-3747 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone CLI >Reporter: maobaolong >Assignee: HemanthBoyina >Priority: Minor > Fix For: 0.7.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-3747) Remove the redundancy if condition code in ListPipelinesSubcommand
[ https://issues.apache.org/jira/browse/HDDS-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal reassigned HDDS-3747: --- Assignee: HemanthBoyina > Remove the redundancy if condition code in ListPipelinesSubcommand > -- > > Key: HDDS-3747 > URL: https://issues.apache.org/jira/browse/HDDS-3747 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone CLI >Reporter: maobaolong >Assignee: HemanthBoyina >Priority: Minor > Fix For: 0.7.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3936) OM client failover ignores suggested leader info
[
https://issues.apache.org/jira/browse/HDDS-3936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17153817#comment-17153817
]
Arpit Agarwal commented on HDDS-3936:
-
bq. One idea is to send along hostname also
In multi-homed setups, hostnames can be different between servers and clients.
> OM client failover ignores suggested leader info
>
>
> Key: HDDS-3936
> URL: https://issues.apache.org/jira/browse/HDDS-3936
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: OM HA
>Affects Versions: 0.6.0
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Major
>
> If OM client hits follower OM, failover is performed sequentially, ignoring
> suggested leader info:
> {code}
> 2020-07-08 17:20:05,249 [main] DEBUG Hadoop3OmTransport:140 - RetryProxy:
> OM:om1 is not the leader. Suggested leader is OM:om3.
> 2020-07-08 17:20:05,277 [main] DEBUG Hadoop3OmTransport:140 - RetryProxy:
> OM:om2 is not the leader. Suggested leader is OM:om3.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3936) OM client failover ignores suggested leader info
[
https://issues.apache.org/jira/browse/HDDS-3936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17153790#comment-17153790
]
Arpit Agarwal commented on HDDS-3936:
-
[~adoroszlai] this is tricky to get right. The node ID returned by the server
may not match the node ID known by the clients.
> OM client failover ignores suggested leader info
>
>
> Key: HDDS-3936
> URL: https://issues.apache.org/jira/browse/HDDS-3936
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: OM HA
>Affects Versions: 0.6.0
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Major
>
> If OM client hits follower OM, failover is performed sequentially, ignoring
> suggested leader info:
> {code}
> 2020-07-08 17:20:05,249 [main] DEBUG Hadoop3OmTransport:140 - RetryProxy:
> OM:om1 is not the leader. Suggested leader is OM:om3.
> 2020-07-08 17:20:05,277 [main] DEBUG Hadoop3OmTransport:140 - RetryProxy:
> OM:om2 is not the leader. Suggested leader is OM:om3.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3935) Change latest snapshot log to debug
[
https://issues.apache.org/jira/browse/HDDS-3935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3935:
Priority: Blocker (was: Major)
> Change latest snapshot log to debug
> ---
>
> Key: HDDS-3935
> URL: https://issues.apache.org/jira/browse/HDDS-3935
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: OM HA
>Affects Versions: 0.6.0
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Blocker
>
> In OM HA, followers log latest snapshot information twice per second:
> {code}
> om1_1 | 2020-07-08 15:46:47,097 [grpc-default-executor-3] INFO
> ratis.OzoneManagerStateMachine: Latest Snapshot Info 0#-1
> om2_1 | 2020-07-08 15:46:47,097 [grpc-default-executor-0] INFO
> ratis.OzoneManagerStateMachine: Latest Snapshot Info 0#-1
> om1_1 | 2020-07-08 15:46:47,604 [grpc-default-executor-3] INFO
> ratis.OzoneManagerStateMachine: Latest Snapshot Info 0#-1
> om2_1 | 2020-07-08 15:46:47,604 [grpc-default-executor-0] INFO
> ratis.OzoneManagerStateMachine: Latest Snapshot Info 0#-1
> om1_1 | 2020-07-08 15:46:48,110 [grpc-default-executor-3] INFO
> ratis.OzoneManagerStateMachine: Latest Snapshot Info 0#-1
> om2_1 | 2020-07-08 15:46:48,110 [grpc-default-executor-0] INFO
> ratis.OzoneManagerStateMachine: Latest Snapshot Info 0#-1
> {code}
> I think this should be debug-level message.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3214) Unhealthy datanodes repeatedly participate in pipeline creation
[ https://issues.apache.org/jira/browse/HDDS-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148135#comment-17148135 ] Arpit Agarwal commented on HDDS-3214: - Moved to 0.7.0. > Unhealthy datanodes repeatedly participate in pipeline creation > --- > > Key: HDDS-3214 > URL: https://issues.apache.org/jira/browse/HDDS-3214 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Reporter: Nilotpal Nandi >Assignee: Prashant Pogde >Priority: Blocker > Labels: TriagePending, fault_injection > > steps taken : > 1) Mounted noise injection FUSE on all datanodes > 2) Selected 1 datanode from each open pipeline (factor=3) > 3) Injected WRITE FAILURE noise with error code - ENOENT on > "hdds.datanode.dir" path of list of datanodes selected in step 2) > 4) start PUT key operation of size 32 MB. > > Observation : > > # Commit failed, pipelines were moved to exclusion list. > # Client retries , new pipeline is created with same set of datanodes. > Container creation fails as WRITE FAILURE injection present. > # Pipeline is closed and the process is repeated for > "ozone.client.max.retries" retries. > Everytime, same set of datanodes are selected for pipeline creation which > include 1 unhealthy datanode. > Expectation - pipeline should have been created by selecting 3 healthy > datanodes available. > > cc - [~ljain] > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3214) Unhealthy datanodes repeatedly participate in pipeline creation
[ https://issues.apache.org/jira/browse/HDDS-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3214: Target Version/s: 0.7.0 (was: 0.6.0) > Unhealthy datanodes repeatedly participate in pipeline creation > --- > > Key: HDDS-3214 > URL: https://issues.apache.org/jira/browse/HDDS-3214 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Reporter: Nilotpal Nandi >Assignee: Prashant Pogde >Priority: Blocker > Labels: TriagePending, fault_injection > > steps taken : > 1) Mounted noise injection FUSE on all datanodes > 2) Selected 1 datanode from each open pipeline (factor=3) > 3) Injected WRITE FAILURE noise with error code - ENOENT on > "hdds.datanode.dir" path of list of datanodes selected in step 2) > 4) start PUT key operation of size 32 MB. > > Observation : > > # Commit failed, pipelines were moved to exclusion list. > # Client retries , new pipeline is created with same set of datanodes. > Container creation fails as WRITE FAILURE injection present. > # Pipeline is closed and the process is repeated for > "ozone.client.max.retries" retries. > Everytime, same set of datanodes are selected for pipeline creation which > include 1 unhealthy datanode. > Expectation - pipeline should have been created by selecting 3 healthy > datanodes available. > > cc - [~ljain] > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3741) Reload old OM state if Install Snapshot from Leader fails
[ https://issues.apache.org/jira/browse/HDDS-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3741: Component/s: OM HA > Reload old OM state if Install Snapshot from Leader fails > - > > Key: HDDS-3741 > URL: https://issues.apache.org/jira/browse/HDDS-3741 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: OM HA >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Critical > Labels: pull-request-available > > Follower OM issues a pause on its services before installing new checkpoint > from Leader OM (Install Snapshot). If this installation fails for some > reason, the OM stays in paused state. It should be unpaused and the old state > should be reloaded. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3612) Allow mounting bucket under other volume
[ https://issues.apache.org/jira/browse/HDDS-3612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3612: Target Version/s: 0.7.0 (was: 0.6.0) > Allow mounting bucket under other volume > > > Key: HDDS-3612 > URL: https://issues.apache.org/jira/browse/HDDS-3612 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: Ozone Manager >Reporter: Attila Doroszlai >Assignee: Attila Doroszlai >Priority: Critical > Labels: Triaged, pull-request-available > > Step 2 from S3 [volume mapping design > doc|https://github.com/apache/hadoop-ozone/blob/master/hadoop-hdds/docs/content/design/ozone-volume-management.md#solving-the-mapping-problem-2-4-from-the-problem-listing]: > Implement a bind mount mechanic which makes it possible to mount any > volume/buckets to the specific "s3" volume. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3612) Allow mounting bucket under other volume
[ https://issues.apache.org/jira/browse/HDDS-3612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3612: Target Version/s: 0.6.0 (was: 0.7.0) > Allow mounting bucket under other volume > > > Key: HDDS-3612 > URL: https://issues.apache.org/jira/browse/HDDS-3612 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: Ozone Manager >Reporter: Attila Doroszlai >Assignee: Attila Doroszlai >Priority: Critical > Labels: Triaged, pull-request-available > > Step 2 from S3 [volume mapping design > doc|https://github.com/apache/hadoop-ozone/blob/master/hadoop-hdds/docs/content/design/ozone-volume-management.md#solving-the-mapping-problem-2-4-from-the-problem-listing]: > Implement a bind mount mechanic which makes it possible to mount any > volume/buckets to the specific "s3" volume. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3402) Use proper acls for sub directories created during CreateDirectory operation
[ https://issues.apache.org/jira/browse/HDDS-3402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3402: Target Version/s: 0.7.0 (was: 0.6.0) > Use proper acls for sub directories created during CreateDirectory operation > > > Key: HDDS-3402 > URL: https://issues.apache.org/jira/browse/HDDS-3402 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Rakesh Radhakrishnan >Priority: Blocker > Labels: TriagePending > > Use proper ACLS for subdirectories created during create directory operation. > All subdirectories/missing directories should inherit the ACLS from the > bucket if ancestors are not present in key table. If present should inherit > the ACLS from its ancestor. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3612) Allow mounting bucket under other volume
[ https://issues.apache.org/jira/browse/HDDS-3612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3612: Priority: Blocker (was: Critical) > Allow mounting bucket under other volume > > > Key: HDDS-3612 > URL: https://issues.apache.org/jira/browse/HDDS-3612 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: Ozone Manager >Reporter: Attila Doroszlai >Assignee: Attila Doroszlai >Priority: Blocker > Labels: Triaged, pull-request-available > > Step 2 from S3 [volume mapping design > doc|https://github.com/apache/hadoop-ozone/blob/master/hadoop-hdds/docs/content/design/ozone-volume-management.md#solving-the-mapping-problem-2-4-from-the-problem-listing]: > Implement a bind mount mechanic which makes it possible to mount any > volume/buckets to the specific "s3" volume. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3685) Remove replay logic from actual request logic
[ https://issues.apache.org/jira/browse/HDDS-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3685: Priority: Blocker (was: Critical) > Remove replay logic from actual request logic > - > > Key: HDDS-3685 > URL: https://issues.apache.org/jira/browse/HDDS-3685 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: OM HA >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Blocker > Labels: pull-request-available > > HDDS-3476 used the transaction info persisted in OM DB during double buffer > flush when OM is restarted. This transaction info log index and the term are > used as a snapshot index. So, we can remove the replay logic from actual > request logic. (As now we shall never have the transaction which is applied > to OM DB will never be again replayed to DB) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3599) [OFS] Add contract test for HA
[ https://issues.apache.org/jira/browse/HDDS-3599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3599: Target Version/s: 0.7.0 (was: 0.6.0) > [OFS] Add contract test for HA > -- > > Key: HDDS-3599 > URL: https://issues.apache.org/jira/browse/HDDS-3599 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: test >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Blocker > Labels: Triaged, pull-request-available > > Add contract tests for HA as well. > Since adding HA contract tests will be another ~10 new classes. [~xyao] and I > decided to put HA OFS contract tests in another jira. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3869) Use different column families for datanode block and metadata
[ https://issues.apache.org/jira/browse/HDDS-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146583#comment-17146583 ] Arpit Agarwal commented on HDDS-3869: - I am removing the backward incompatible label since we can mitigate this via auto-migration. > Use different column families for datanode block and metadata > - > > Key: HDDS-3869 > URL: https://issues.apache.org/jira/browse/HDDS-3869 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Ethan Rose >Assignee: Ethan Rose >Priority: Critical > Labels: upgrade > > Currently datanodes place all of their data under the default column family > in RocksDB. This differs from OM and SCM which organize their data into > different column families based on its type. By having the datanode place its > metadata in one column family, and block data in another, it will be easier > to view for debugging and organizational purposes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3869) Use different column families for datanode block and metadata
[ https://issues.apache.org/jira/browse/HDDS-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3869: Labels: upgrade (was: backward-incompatible upgrade) > Use different column families for datanode block and metadata > - > > Key: HDDS-3869 > URL: https://issues.apache.org/jira/browse/HDDS-3869 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Ethan Rose >Assignee: Ethan Rose >Priority: Critical > Labels: upgrade > > Currently datanodes place all of their data under the default column family > in RocksDB. This differs from OM and SCM which organize their data into > different column families based on its type. By having the datanode place its > metadata in one column family, and block data in another, it will be easier > to view for debugging and organizational purposes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3290) Remove deprecated RandomKeyGenerator
[ https://issues.apache.org/jira/browse/HDDS-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3290: Labels: TriagePending (was: TriagePending pull-request-available) > Remove deprecated RandomKeyGenerator > > > Key: HDDS-3290 > URL: https://issues.apache.org/jira/browse/HDDS-3290 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Marton Elek >Assignee: Marton Elek >Priority: Major > Labels: TriagePending > Time Spent: 20m > Remaining Estimate: 0h > > Our first Freon test (RandomKeyGenerator) is depracated as we have all the > functionalities with a simplified architecture (BaseFreonGenerator). We can > remove it (especially as it's flaky...) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2268) Incorrect container checksum upon downgrade
[
https://issues.apache.org/jira/browse/HDDS-2268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-2268:
Labels: upgrade (was: Triaged upgrade)
> Incorrect container checksum upon downgrade
> ---
>
> Key: HDDS-2268
> URL: https://issues.apache.org/jira/browse/HDDS-2268
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Reporter: Attila Doroszlai
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: upgrade
>
> Container file checksum is calculated based on all YAML fields in a given
> Ozone version. If the same container file is used in older Ozone, which has
> fewer fields, the expected checksum will be different.
> Example: origin pipeline ID and origin node ID were added for HDDS-837 in
> Ozone 0.4.0. Starting Ozone 0.3.0 with the same data results in checksum
> error.
> {noformat}
> datanode_1 | ... ERROR ContainerReader:166 - Failed to parse ContainerFile
> for ContainerID: 1
> datanode_1 |
> org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
> Container checksum error for ContainerID: 1.
> datanode_1 | Stored Checksum:
> 7a6ec508d6e3796c5fe5fd52574b3d3437b0a0eaa4e053f7a96a5e39f4abb374
> datanode_1 | Expected Checksum:
> fee023a02d3ced2f7b0b42c116cce5f03da6b57b29965ca878dc46d1213230b6
> datanode_1 | at
> org.apache.hadoop.ozone.container.common.helpers.ContainerUtils.verifyChecksum(ContainerUtils.java:259)
> datanode_1 | at
> org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerUtil.parseKVContainerData(KeyValueContainerUtil.java:165)
> datanode_1 | at
> org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.verifyContainerData(ContainerReader.java:180)
> datanode_1 | at
> org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.verifyContainerFile(ContainerReader.java:164)
> datanode_1 | at
> org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.readVolume(ContainerReader.java:142)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2268) Incorrect container checksum upon downgrade
[
https://issues.apache.org/jira/browse/HDDS-2268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-2268:
Target Version/s: 0.7.0 (was: 0.6.0)
> Incorrect container checksum upon downgrade
> ---
>
> Key: HDDS-2268
> URL: https://issues.apache.org/jira/browse/HDDS-2268
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Reporter: Attila Doroszlai
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: upgrade
>
> Container file checksum is calculated based on all YAML fields in a given
> Ozone version. If the same container file is used in older Ozone, which has
> fewer fields, the expected checksum will be different.
> Example: origin pipeline ID and origin node ID were added for HDDS-837 in
> Ozone 0.4.0. Starting Ozone 0.3.0 with the same data results in checksum
> error.
> {noformat}
> datanode_1 | ... ERROR ContainerReader:166 - Failed to parse ContainerFile
> for ContainerID: 1
> datanode_1 |
> org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
> Container checksum error for ContainerID: 1.
> datanode_1 | Stored Checksum:
> 7a6ec508d6e3796c5fe5fd52574b3d3437b0a0eaa4e053f7a96a5e39f4abb374
> datanode_1 | Expected Checksum:
> fee023a02d3ced2f7b0b42c116cce5f03da6b57b29965ca878dc46d1213230b6
> datanode_1 | at
> org.apache.hadoop.ozone.container.common.helpers.ContainerUtils.verifyChecksum(ContainerUtils.java:259)
> datanode_1 | at
> org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerUtil.parseKVContainerData(KeyValueContainerUtil.java:165)
> datanode_1 | at
> org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.verifyContainerData(ContainerReader.java:180)
> datanode_1 | at
> org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.verifyContainerFile(ContainerReader.java:164)
> datanode_1 | at
> org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.readVolume(ContainerReader.java:142)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2268) Incorrect container checksum upon downgrade
[
https://issues.apache.org/jira/browse/HDDS-2268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17145054#comment-17145054
]
Arpit Agarwal commented on HDDS-2268:
-
Yeah this can probably be moved out to 0.7.0.
> Incorrect container checksum upon downgrade
> ---
>
> Key: HDDS-2268
> URL: https://issues.apache.org/jira/browse/HDDS-2268
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Reporter: Attila Doroszlai
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: Triaged, upgrade
>
> Container file checksum is calculated based on all YAML fields in a given
> Ozone version. If the same container file is used in older Ozone, which has
> fewer fields, the expected checksum will be different.
> Example: origin pipeline ID and origin node ID were added for HDDS-837 in
> Ozone 0.4.0. Starting Ozone 0.3.0 with the same data results in checksum
> error.
> {noformat}
> datanode_1 | ... ERROR ContainerReader:166 - Failed to parse ContainerFile
> for ContainerID: 1
> datanode_1 |
> org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
> Container checksum error for ContainerID: 1.
> datanode_1 | Stored Checksum:
> 7a6ec508d6e3796c5fe5fd52574b3d3437b0a0eaa4e053f7a96a5e39f4abb374
> datanode_1 | Expected Checksum:
> fee023a02d3ced2f7b0b42c116cce5f03da6b57b29965ca878dc46d1213230b6
> datanode_1 | at
> org.apache.hadoop.ozone.container.common.helpers.ContainerUtils.verifyChecksum(ContainerUtils.java:259)
> datanode_1 | at
> org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerUtil.parseKVContainerData(KeyValueContainerUtil.java:165)
> datanode_1 | at
> org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.verifyContainerData(ContainerReader.java:180)
> datanode_1 | at
> org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.verifyContainerFile(ContainerReader.java:164)
> datanode_1 | at
> org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.readVolume(ContainerReader.java:142)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3367) Trash Support for Ozone FileSystem
[ https://issues.apache.org/jira/browse/HDDS-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3367: Priority: Blocker (was: Minor) > Trash Support for Ozone FileSystem > -- > > Key: HDDS-3367 > URL: https://issues.apache.org/jira/browse/HDDS-3367 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: Ozone Filesystem >Affects Versions: 0.6.0 >Reporter: Sadanand Shenoy >Assignee: Sadanand Shenoy >Priority: Blocker > Labels: TriagePending > Attachments: OzoneFsTrash.patch > > > The aim of this jira to add the trash feature capability in ozone that will > help prevent accidental deletion of files and directories. When a file gets > deleted in Ozone, the file is not immediately expelled from Ozone but will be > moved to trash. Similarly as in HDFS, to immediately delete a file, a > skipTrash option will be provided. The jira also aims to integrate the Ozone > FS Trash to the Trash API developed as a part of HDDS-2416. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3367) Trash Support for Ozone FileSystem
[ https://issues.apache.org/jira/browse/HDDS-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3367: Target Version/s: 0.6.0 (was: 0.7.0) > Trash Support for Ozone FileSystem > -- > > Key: HDDS-3367 > URL: https://issues.apache.org/jira/browse/HDDS-3367 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: Ozone Filesystem >Affects Versions: 0.6.0 >Reporter: Sadanand Shenoy >Assignee: Sadanand Shenoy >Priority: Minor > Labels: TriagePending > Attachments: OzoneFsTrash.patch > > > The aim of this jira to add the trash feature capability in ozone that will > help prevent accidental deletion of files and directories. When a file gets > deleted in Ozone, the file is not immediately expelled from Ozone but will be > moved to trash. Similarly as in HDFS, to immediately delete a file, a > skipTrash option will be provided. The jira also aims to integrate the Ozone > FS Trash to the Trash API developed as a part of HDDS-2416. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3367) Trash Support for Ozone FileSystem
[ https://issues.apache.org/jira/browse/HDDS-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17144350#comment-17144350 ] Arpit Agarwal commented on HDDS-3367: - Does this fix include trash cleanup support on the OM? We'll need it for GA. > Trash Support for Ozone FileSystem > -- > > Key: HDDS-3367 > URL: https://issues.apache.org/jira/browse/HDDS-3367 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: Ozone Filesystem >Affects Versions: 0.6.0 >Reporter: Sadanand Shenoy >Assignee: Sadanand Shenoy >Priority: Blocker > Labels: TriagePending > Attachments: OzoneFsTrash.patch > > > The aim of this jira to add the trash feature capability in ozone that will > help prevent accidental deletion of files and directories. When a file gets > deleted in Ozone, the file is not immediately expelled from Ozone but will be > moved to trash. Similarly as in HDFS, to immediately delete a file, a > skipTrash option will be provided. The jira also aims to integrate the Ozone > FS Trash to the Trash API developed as a part of HDDS-2416. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3705) [OFS] Implement getTrashRoots for trash cleanup
[
https://issues.apache.org/jira/browse/HDDS-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3705:
Target Version/s: 0.6.0
> [OFS] Implement getTrashRoots for trash cleanup
> ---
>
> Key: HDDS-3705
> URL: https://issues.apache.org/jira/browse/HDDS-3705
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Blocker
> Labels: pull-request-available
>
> We need to override {{getTrashRoots()}} as well in order to allow for easier
> future OM trash cleanup impl.
> This jira doesn't directly implement the trash cleanup feature itself, but a
> prerequisite for this feature.
> This is a follow-up jira to HDDS-3574:
> https://github.com/apache/hadoop-ozone/pull/941#discussion_r428212741
> May raise PR after vote for OFS merge to master
> Dev commit:
> https://github.com/smengcl/hadoop-ozone/commit/c563260011aa00a36568c57cda7254e82c15c59c
> (Crude code finished. Untested. Pending unit test.)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3705) [OFS] Implement getTrashRoots for trash cleanup
[
https://issues.apache.org/jira/browse/HDDS-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3705:
Priority: Blocker (was: Major)
> [OFS] Implement getTrashRoots for trash cleanup
> ---
>
> Key: HDDS-3705
> URL: https://issues.apache.org/jira/browse/HDDS-3705
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Blocker
> Labels: pull-request-available
>
> We need to override {{getTrashRoots()}} as well in order to allow for easier
> future OM trash cleanup impl.
> This jira doesn't directly implement the trash cleanup feature itself, but a
> prerequisite for this feature.
> This is a follow-up jira to HDDS-3574:
> https://github.com/apache/hadoop-ozone/pull/941#discussion_r428212741
> May raise PR after vote for OFS merge to master
> Dev commit:
> https://github.com/smengcl/hadoop-ozone/commit/c563260011aa00a36568c57cda7254e82c15c59c
> (Crude code finished. Untested. Pending unit test.)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3545) MR Jobhistory cannot work well with o3fs hadoop compatible file system
[ https://issues.apache.org/jira/browse/HDDS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3545: Labels: (was: TriagePending) > MR Jobhistory cannot work well with o3fs hadoop compatible file system > -- > > Key: HDDS-3545 > URL: https://issues.apache.org/jira/browse/HDDS-3545 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Filesystem >Reporter: maobaolong >Priority: Major > > After take a look at the code of JobHistory, i see jobhistory use > `fs.getModificationTime()` to get the directory modification time, and use it > as a condition of starting scan the job directories. > But, for ozone, wile insert a child to a `directory`, the modification time > of the `directory` don't update now. > So we should update the modification time of `directory`, otherwise, MR > Jobhistory and some other component which use the modification time of > `directory` cannot work as expected. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3545) MR Jobhistory cannot work well with o3fs hadoop compatible file system
[ https://issues.apache.org/jira/browse/HDDS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143164#comment-17143164 ] Arpit Agarwal commented on HDDS-3545: - I see thanks for the link [~maobaolong] and for reporting this. Yes this should get addressed by HDDS-2939 when we have a better notion of directories or prefixes. Certainly something for us to keep in mind as we work on HDDS-2939. > MR Jobhistory cannot work well with o3fs hadoop compatible file system > -- > > Key: HDDS-3545 > URL: https://issues.apache.org/jira/browse/HDDS-3545 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Filesystem >Reporter: maobaolong >Priority: Major > > After take a look at the code of JobHistory, i see jobhistory use > `fs.getModificationTime()` to get the directory modification time, and use it > as a condition of starting scan the job directories. > But, for ozone, wile insert a child to a `directory`, the modification time > of the `directory` don't update now. > So we should update the modification time of `directory`, otherwise, MR > Jobhistory and some other component which use the modification time of > `directory` cannot work as expected. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-3545) MR Jobhistory cannot work well with o3fs hadoop compatible file system
[ https://issues.apache.org/jira/browse/HDDS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142212#comment-17142212 ] Arpit Agarwal edited comment on HDDS-3545 at 6/22/20, 3:54 PM: --- HDDS-2939 should be done for 0.7.0 release. How does Jobhistory handle this when working over S3A? cc [~msingh] [~rakeshr]. was (Author: arpitagarwal): HDDS-2939 should be done for 0.7.0 release. How does Jobhistory handle this when working over S3A? > MR Jobhistory cannot work well with o3fs hadoop compatible file system > -- > > Key: HDDS-3545 > URL: https://issues.apache.org/jira/browse/HDDS-3545 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Filesystem >Reporter: maobaolong >Priority: Major > Labels: TriagePending > > After take a look at the code of JobHistory, i see jobhistory use > `fs.getModificationTime()` to get the directory modification time, and use it > as a condition of starting scan the job directories. > But, for ozone, wile insert a child to a `directory`, the modification time > of the `directory` don't update now. > So we should update the modification time of `directory`, otherwise, MR > Jobhistory and some other component which use the modification time of > `directory` cannot work as expected. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3545) MR Jobhistory cannot work well with o3fs hadoop compatible file system
[ https://issues.apache.org/jira/browse/HDDS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142212#comment-17142212 ] Arpit Agarwal commented on HDDS-3545: - HDDS-2939 should be done for 0.7.0 release. How does Jobhistory handle this when working over S3A? > MR Jobhistory cannot work well with o3fs hadoop compatible file system > -- > > Key: HDDS-3545 > URL: https://issues.apache.org/jira/browse/HDDS-3545 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Filesystem >Reporter: maobaolong >Priority: Major > Labels: TriagePending > > After take a look at the code of JobHistory, i see jobhistory use > `fs.getModificationTime()` to get the directory modification time, and use it > as a condition of starting scan the job directories. > But, for ozone, wile insert a child to a `directory`, the modification time > of the `directory` don't update now. > So we should update the modification time of `directory`, otherwise, MR > Jobhistory and some other component which use the modification time of > `directory` cannot work as expected. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2939) Ozone FS namespace
[ https://issues.apache.org/jira/browse/HDDS-2939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142209#comment-17142209 ] Arpit Agarwal commented on HDDS-2939: - [~Sammi] let's move it to 0.7.0. Thanks. > Ozone FS namespace > -- > > Key: HDDS-2939 > URL: https://issues.apache.org/jira/browse/HDDS-2939 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: Ozone Manager >Reporter: Supratim Deka >Assignee: Rakesh Radhakrishnan >Priority: Major > Labels: Triaged > Attachments: Ozone FS Namespace Proposal v1.0.docx > > > Create the structures and metadata layout required to support efficient FS > namespace operations in Ozone - operations involving folders/directories > required to support the Hadoop compatible Filesystem interface. > The details are described in the attached document. The work is divided up > into sub-tasks as per the task list in the document. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3721) Implement getContentSummary to provide replicated size properly to dfs -du command
[
https://issues.apache.org/jira/browse/HDDS-3721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17140694#comment-17140694
]
Arpit Agarwal commented on HDDS-3721:
-
bq. Arpit Agarwal, can you give some insight why you would like to avoid
implementing this on OM side, perhaps why at the end was it implemented on the
server side for HDFS?
IMO getContentSummary on NameNode is the single biggest mistake in HDFS. :) In
general we should have no recursive calls on the server which can run for a
long time while holding locks and tying up RPC handler threads.
getContentSummary is the largest contributor to HDFS performance issues among
our customers since they naively assume it is a simple RPC call and issue it
very frequently. I would like to avoid giving Ozone users such a powerful tool
to shoot themselves with.
bq. The problem of approaching this from the client side, is that it leads to 4
calls per subdirectory (14k calls in this case)... 1 READ_BUCKET, then 1
GET_FILE_STATUS (to see if this is a file or a dir), then if it is a directory
1 READ_BUCKET again, and finally a LIST_STATUS, which then can not be
controlled or throttled by the server side much as these are coming from the
client side and from possibly multiple clients at some times.
Can we fix the client to avoid this multiplication of calls?
Also I have a more high level question - why do we want to optimize
getContentSummary? I want to get people to move away from using this call on
arbitrary directories. Perhaps we can provide an efficient implementation at
the bucket level by tracking the usage per bucket and updating it on key
commit/key delete.
> Implement getContentSummary to provide replicated size properly to dfs -du
> command
> --
>
> Key: HDDS-3721
> URL: https://issues.apache.org/jira/browse/HDDS-3721
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Major
> Labels: Triaged
>
> Currently when you run hdfs dfs -du command against a path on Ozone, it uses
> the default implementation from FileSystem class in the Hadoop project, and
> that does not care to calculate with replication factor by default. In
> DistributedFileSystem and in a couple of FileSystem implementation there is
> an override to calculate the full replicated size properly.
> Currently the output is something like this for a folder which has file with
> replication factor of 3:
> {code}
> hdfs dfs -du -s -h o3fs://perfbucket.volume.ozone1/terasort/datagen
> 931.3 G 931.3 G o3fs://perfbucket.volume.ozone1/terasort/datagen
> {code}
> The command in Ozone's case as well should report the replicated size az the
> second number so something around 2.7TB in this case.
> In order to do so, we should implement getContentSummary and calculate the
> replicated size in the response properly in order to get there.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-3793) Use Hadoop 2.7.2 for ozone-mr/hadoop27 acceptance tests
[ https://issues.apache.org/jira/browse/HDDS-3793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal reassigned HDDS-3793: --- Assignee: Attila Doroszlai (was: Marton Elek) > Use Hadoop 2.7.2 for ozone-mr/hadoop27 acceptance tests > --- > > Key: HDDS-3793 > URL: https://issues.apache.org/jira/browse/HDDS-3793 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Marton Elek >Assignee: Attila Doroszlai >Priority: Major > Labels: pull-request-available > > Today 2.7.7 is used but to be compatible with older Hadoop (including 2.7.4 > which is included in spark), it's easier to test with an older version -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3755) Storage-class support for Ozone
[ https://issues.apache.org/jira/browse/HDDS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17135959#comment-17135959 ] Arpit Agarwal commented on HDDS-3755: - Folks, I cannot support this idea of replication factor TWO on the write path. The Ratis based write pipeline is designed to work with consensus and handle single node failure on the write path. With factor of TWO the nature of the write pipeline changes completely. > Storage-class support for Ozone > --- > > Key: HDDS-3755 > URL: https://issues.apache.org/jira/browse/HDDS-3755 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Marton Elek >Assignee: Marton Elek >Priority: Major > > Use a storage-class as an abstraction which combines replication > configuration, container states and transitions. > See this thread for the detailed design doc: > > [https://lists.apache.org/thread.html/r1e2a5d5581abe9dd09834305ca65a6807f37bd229a07b8b31bda32ad%40%3Cozone-dev.hadoop.apache.org%3E] > which is also uploaded to here: > https://hackmd.io/4kxufJBOQNaKn7PKFK_6OQ?edit -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-741) all containers are in 'CLOSING' state after service restart
[
https://issues.apache.org/jira/browse/HDDS-741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-741:
---
Priority: Critical (was: Blocker)
> all containers are in 'CLOSING' state after service restart
> ---
>
> Key: HDDS-741
> URL: https://issues.apache.org/jira/browse/HDDS-741
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 0.3.0
>Reporter: Nilotpal Nandi
>Assignee: Nanda kumar
>Priority: Critical
> Labels: TriagePending
> Attachments: all-node-ozone-logs-1540556458.tar.gz
>
>
> all containers are in closing state after service restart. None of the writes
> are working after restart.
> The cluster contains 11 live datanodes.
> **
> {noformat}
> [
> {
> "nodeType": "OM",
> "hostname": "ctr-e138-1518143905142-53-01-08.hwx.site",
> "ports": {
> "RPC": 9889,
> "HTTP": 9874
> }
> },
> {
> "nodeType": "SCM",
> "hostname": "ctr-e138-1518143905142-53-01-03.hwx.site",
> "ports": {
> "RPC": 9860
> }
> },
> {
> "nodeType": "DATANODE",
> "hostname": "ctr-e138-1518143905142-541661-01-03.hwx.site",
> "ports": {
> "HTTP": 9880
> }
> },
> {
> "nodeType": "DATANODE",
> "hostname": "ctr-e138-1518143905142-541661-01-07.hwx.site",
> "ports": {
> "HTTP": 9880
> }
> },
> {
> "nodeType": "DATANODE",
> "hostname": "ctr-e138-1518143905142-53-01-03.hwx.site",
> "ports": {
> "HTTP": 9880
> }
> },
> {
> "nodeType": "DATANODE",
> "hostname": "ctr-e138-1518143905142-541661-01-04.hwx.site",
> "ports": {
> "HTTP": 9880
> }
> },
> {
> "nodeType": "DATANODE",
> "hostname": "ctr-e138-1518143905142-53-01-04.hwx.site",
> "ports": {
> "HTTP": 9880
> }
> },
> {
> "nodeType": "DATANODE",
> "hostname": "ctr-e138-1518143905142-53-01-08.hwx.site",
> "ports": {
> "HTTP": 9880
> }
> },
> {
> "nodeType": "DATANODE",
> "hostname": "ctr-e138-1518143905142-541661-01-02.hwx.site",
> "ports": {
> "HTTP": 9880
> }
> },
> {
> "nodeType": "DATANODE",
> "hostname": "ctr-e138-1518143905142-53-01-05.hwx.site",
> "ports": {
> "HTTP": 9880
> }
> },
> {
> "nodeType": "DATANODE",
> "hostname": "ctr-e138-1518143905142-541661-01-06.hwx.site",
> "ports": {
> "HTTP": 9880
> }
> },
> {
> "nodeType": "DATANODE",
> "hostname": "ctr-e138-1518143905142-53-01-07.hwx.site",
> "ports": {
> "HTTP": 9880
> }
> },
> {
> "nodeType": "DATANODE",
> "hostname": "ctr-e138-1518143905142-53-01-06.hwx.site",
> "ports": {
> "HTTP": 9880
> }
> }
> ]{noformat}
> error thrown while write :
>
> {noformat}
> [root@ctr-e138-1518143905142-541661-01-07 test_files]# ozone fs -put
> /etc/passwd /testdir5/
> 2018-10-26 12:09:43,822 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 2018-10-26 12:09:47,882 ERROR io.ChunkGroupOutputStream: Try to allocate more
> blocks for write failed, already allocated 0 blocks for this write.
> put: Allocate block failed, error:INTERNAL_ERROR{noformat}
>
>
> pipelines in the cluster :
>
> {noformat}
> [root@ctr-e138-1518143905142-541661-01-07 test_files]# ozone scmcli
> listPipelines
> Pipeline[ Id: 29b68cc2-2d18-4db0-a11a-587ae4abc715, Nodes:
> e3d89961-fe38-4ed0-8a32-cd1849c58e0c{ip: 172.27.20.96, host:
> ctr-e138-1518143905142-53-01-08.hwx.site}b33a30d9-f1e2-448e-aabb-61a970445cea{ip:
> 172.27.85.64, host: ctr-e138-1518143905142-541661-01-07.hwx.site},
> Type:RATIS, Factor:THREE, State:CLOSING]
> Pipeline[ Id: 05061f87-4c68-443b-ae27-984da2d0a2cd, Nodes:
> dc002a73-fc63-4e76-be3e-3c6d16ede5f6{ip: 172.27.38.9, host:
> ctr-e138-1518143905142-53-01-04.hwx.site}4e6bd2a2-6802-4e67-9710-612a2cdb9dc1{ip:
> 172.27.24.90, host:
> ctr-e138-1518143905142-53-01-05.hwx.site}be3f0db4-3a19-44a5-bd6e-0da47d2ed92e{ip:
> 172.27.20.91, host: ctr-e138-1518143905142-53-01-03.hwx.site},
> Type:RATIS, Factor:THREE, State:CLOSING]
> Pipeline[ Id: 80893f87-5e73-49a2-8f38-2adb2b13140a, Nodes:
> 63833540-bf93-410c-b081-243a56f93c88{ip: 172.27.10.199, host:
> ctr-e138-1518143905142-53-01-07.hwx.site}6e8b7129-8615-45fe-81e0-848a2e0ba520{ip:
> 172.27.15.139, host:
> ctr-e138-1518143905142-53-01-06.hwx.site}aab1f2e5-1cf0-430d-b1bf-04be8630a8ee{ip:
> 172.27.57.0, host: ctr-e138-1518143905142-541661-01-03.hwx.site},
> Type:RATIS, Factor:THREE, State:CLOSING]
> Pipeline[ Id: f0a14cb9-d37a-4c7c-b3e6-b7e3830dfd5f, Nodes:
> 61e271bf-68ad-435e-8a6e-582be90ebb6f{ip: 172.27.19.74, host:
> ctr-e138-1518143905142-541661-01-06.hwx.site}3622352c-b136-4c74-b952-34e938cbda94{ip:
> 172.27.15.131, host:
>
[jira] [Commented] (HDDS-3039) SCM sometimes cannot exit safe mode
[
https://issues.apache.org/jira/browse/HDDS-3039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17133530#comment-17133530
]
Arpit Agarwal commented on HDDS-3039:
-
Let's examine logs for a few recent runs to figure out why the SCM is not
exiting safe mode. This seems like a must fix for the next release.
> SCM sometimes cannot exit safe mode
> ---
>
> Key: HDDS-3039
> URL: https://issues.apache.org/jira/browse/HDDS-3039
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Reporter: Attila Doroszlai
>Priority: Critical
> Labels: Triaged
>
> Sometimes SCM cannot exit safe mode:
> {code:title=https://github.com/apache/hadoop-ozone/pull/563/checks?check_run_id=453543576}
> 2020-02-18T19:12:28.1108180Z [ERROR] Tests run: 1, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 139.821 s <<< FAILURE! - in
> org.apache.hadoop.ozone.fsck.TestContainerMapper
> 2020-02-18T19:12:28.1169327Z [ERROR]
> org.apache.hadoop.ozone.fsck.TestContainerMapper Time elapsed: 139.813 s
> <<< ERROR!
> 2020-02-18T19:12:28.1202534Z java.util.concurrent.TimeoutException:
> ...
> at
> org.apache.hadoop.ozone.MiniOzoneClusterImpl.waitForClusterToBeReady(MiniOzoneClusterImpl.java:164)
> at
> org.apache.hadoop.ozone.fsck.TestContainerMapper.init(TestContainerMapper.java:71)
> {code}
> despite nodes and pipeline being ready:
> {code}
> 2020-02-18 19:10:18,045 [main] INFO ozone.MiniOzoneClusterImpl
> (MiniOzoneClusterImpl.java:lambda$waitForClusterToBeReady$0(169)) - Nodes are
> ready. Got 3 of 3 DN Heartbeats.
> ...
> 2020-02-18 19:10:18,847 [RatisPipelineUtilsThread] INFO
> pipeline.PipelineStateManager (PipelineStateManager.java:addPipeline(54)) -
> Created pipeline Pipeline[ Id: b56478a3-8816-459e-a007-db5ee4a5572e, Nodes:
> 86e97873-2dbd-4f1b-b418-cf9fba405476{ip: 172.17.0.2, host: bedb6e0ff851,
> networkLocation: /default-rack, certSerialId:
> null}0fb407c1-4cda-4b3e-8e64-20c845872684{ip: 172.17.0.2, host: bedb6e0ff851,
> networkLocation: /default-rack, certSerialId:
> null}31baa82d-441c-41be-94c9-8dd7468b728e{ip: 172.17.0.2, host: bedb6e0ff851,
> networkLocation: /default-rack, certSerialId: null}, Type:RATIS,
> Factor:THREE, State:ALLOCATED, leaderId:null ]
> ...
> 2020-02-18 19:12:17,108 [main] INFO ozone.MiniOzoneClusterImpl
> (MiniOzoneClusterImpl.java:lambda$waitForClusterToBeReady$0(169)) - Nodes are
> ready. Got 3 of 3 DN Heartbeats.
> 2020-02-18 19:12:17,108 [main] INFO ozone.MiniOzoneClusterImpl
> (MiniOzoneClusterImpl.java:lambda$waitForClusterToBeReady$0(172)) - Waiting
> for cluster to exit safe mode
> 2020-02-18 19:12:17,151 [main] INFO ozone.MiniOzoneClusterImpl
> (MiniOzoneClusterImpl.java:shutdown(370)) - Shutting down the Mini Ozone
> Cluster
> {code}
> [~shashikant] also noticed this in other integration tests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3039) SCM sometimes cannot exit safe mode
[
https://issues.apache.org/jira/browse/HDDS-3039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-3039:
Labels: Triaged (was: TriagePending)
> SCM sometimes cannot exit safe mode
> ---
>
> Key: HDDS-3039
> URL: https://issues.apache.org/jira/browse/HDDS-3039
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Reporter: Attila Doroszlai
>Priority: Critical
> Labels: Triaged
>
> Sometimes SCM cannot exit safe mode:
> {code:title=https://github.com/apache/hadoop-ozone/pull/563/checks?check_run_id=453543576}
> 2020-02-18T19:12:28.1108180Z [ERROR] Tests run: 1, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 139.821 s <<< FAILURE! - in
> org.apache.hadoop.ozone.fsck.TestContainerMapper
> 2020-02-18T19:12:28.1169327Z [ERROR]
> org.apache.hadoop.ozone.fsck.TestContainerMapper Time elapsed: 139.813 s
> <<< ERROR!
> 2020-02-18T19:12:28.1202534Z java.util.concurrent.TimeoutException:
> ...
> at
> org.apache.hadoop.ozone.MiniOzoneClusterImpl.waitForClusterToBeReady(MiniOzoneClusterImpl.java:164)
> at
> org.apache.hadoop.ozone.fsck.TestContainerMapper.init(TestContainerMapper.java:71)
> {code}
> despite nodes and pipeline being ready:
> {code}
> 2020-02-18 19:10:18,045 [main] INFO ozone.MiniOzoneClusterImpl
> (MiniOzoneClusterImpl.java:lambda$waitForClusterToBeReady$0(169)) - Nodes are
> ready. Got 3 of 3 DN Heartbeats.
> ...
> 2020-02-18 19:10:18,847 [RatisPipelineUtilsThread] INFO
> pipeline.PipelineStateManager (PipelineStateManager.java:addPipeline(54)) -
> Created pipeline Pipeline[ Id: b56478a3-8816-459e-a007-db5ee4a5572e, Nodes:
> 86e97873-2dbd-4f1b-b418-cf9fba405476{ip: 172.17.0.2, host: bedb6e0ff851,
> networkLocation: /default-rack, certSerialId:
> null}0fb407c1-4cda-4b3e-8e64-20c845872684{ip: 172.17.0.2, host: bedb6e0ff851,
> networkLocation: /default-rack, certSerialId:
> null}31baa82d-441c-41be-94c9-8dd7468b728e{ip: 172.17.0.2, host: bedb6e0ff851,
> networkLocation: /default-rack, certSerialId: null}, Type:RATIS,
> Factor:THREE, State:ALLOCATED, leaderId:null ]
> ...
> 2020-02-18 19:12:17,108 [main] INFO ozone.MiniOzoneClusterImpl
> (MiniOzoneClusterImpl.java:lambda$waitForClusterToBeReady$0(169)) - Nodes are
> ready. Got 3 of 3 DN Heartbeats.
> 2020-02-18 19:12:17,108 [main] INFO ozone.MiniOzoneClusterImpl
> (MiniOzoneClusterImpl.java:lambda$waitForClusterToBeReady$0(172)) - Waiting
> for cluster to exit safe mode
> 2020-02-18 19:12:17,151 [main] INFO ozone.MiniOzoneClusterImpl
> (MiniOzoneClusterImpl.java:shutdown(370)) - Shutting down the Mini Ozone
> Cluster
> {code}
> [~shashikant] also noticed this in other integration tests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1953) Remove pipeline persistent in SCM
[ https://issues.apache.org/jira/browse/HDDS-1953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-1953: Target Version/s: 0.7.0 (was: 0.6.0) > Remove pipeline persistent in SCM > - > > Key: HDDS-1953 > URL: https://issues.apache.org/jira/browse/HDDS-1953 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Sammi Chen >Assignee: Sammi Chen >Priority: Major > Labels: TriagePending > > Currently, SCM will persistent pipeline in metastore with datanode > information locally. After SCM restart, it will reload all the pipelines from > the metastore. If there is any datanode information change during the whole > SCMc lifecycle, the persisted pipeline is not updated. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1336) Better block allocation policy in SCM
[ https://issues.apache.org/jira/browse/HDDS-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-1336: Labels: Triaged (was: TriagePending) > Better block allocation policy in SCM > - > > Key: HDDS-1336 > URL: https://issues.apache.org/jira/browse/HDDS-1336 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Labels: Triaged > > In case, a client hits close container exception and asks SCM to allocate > block for the next write, SCM might allocate blocks from containers which are > almost full and can get closed while the client write is still on. Ozone > client by default tries 5 times on hitting an exception to write Data to a > different block in case it hits CloseContainerException 5 times, it will give > up. > The issue was found while testing with HDDS-1295. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2534) scmcli container delete not working
[
https://issues.apache.org/jira/browse/HDDS-2534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-2534:
Target Version/s: 0.7.0 (was: 0.6.0)
> scmcli container delete not working
> ---
>
> Key: HDDS-2534
> URL: https://issues.apache.org/jira/browse/HDDS-2534
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Reporter: luhuachao
>Assignee: luhuachao
>Priority: Major
> Labels: Triaged, pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {code:java}
> java.lang.IllegalArgumentException: Unknown command type: DeleteContainer
> at
> org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.processRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:219)
> at
> org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:72)
> at
> org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.submitRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:112)
> at
> org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos$StorageContainerLocationProtocolService$2.callBlockingMethod(StorageContainerLocationProtocolProtos.java:30454)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1933) Datanode should use hostname in place of ip addresses to allow DN's to work when ipaddress change
[ https://issues.apache.org/jira/browse/HDDS-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-1933: Priority: Major (was: Blocker) > Datanode should use hostname in place of ip addresses to allow DN's to work > when ipaddress change > - > > Key: HDDS-1933 > URL: https://issues.apache.org/jira/browse/HDDS-1933 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode, SCM >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: runzhiwang >Priority: Major > Labels: TriagePending > > This was noticed by [~elek] while deploying Ozone on Kubernetes based > environment. > When the datanode ip address change on restart, the Datanode details cease to > be correct for the datanode. and this prevents the cluster from functioning > after a restart. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-698) Support Topology Awareness for Ozone
[ https://issues.apache.org/jira/browse/HDDS-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-698: --- Target Version/s: (was: 0.6.0) > Support Topology Awareness for Ozone > > > Key: HDDS-698 > URL: https://issues.apache.org/jira/browse/HDDS-698 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: SCM >Reporter: Xiaoyu Yao >Assignee: Sammi Chen >Priority: Blocker > Fix For: 0.6.0 > > Attachments: HDDS-698.000.patch, network-topology-default.xml, > network-topology-nodegroup.xml > > > This is an umbrella JIRA to add topology aware support for Ozone Pipelines, > Containers and Blocks. Long time since HDFS is created, we provide > rack/nodegroup awareness for reliability and high performance for data > access. Ozone need a similar mechanism and this can be more flexible for > cloud scenarios. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-3707) UUID can be non unique for a huge samples
[ https://issues.apache.org/jira/browse/HDDS-3707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17132675#comment-17132675 ] Arpit Agarwal edited comment on HDDS-3707 at 6/10/20, 7:52 PM: --- Hi [~maobaolong] the probability is so infinitesimal I don't think it is worth trying to change it. :) was (Author: arpitagarwal): Hi [~maobaolong] the probability is so infinitesimal I don't think it is worth trying to fix it. :) > UUID can be non unique for a huge samples > - > > Key: HDDS-3707 > URL: https://issues.apache.org/jira/browse/HDDS-3707 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode, Ozone Manager, SCM >Affects Versions: 0.7.0 >Reporter: maobaolong >Priority: Minor > Labels: Triaged > > Now, we have used UUID as id for many places, for example, DataNodeId, > pipelineId. I believe that it should be pretty less chance to met collision, > but, if met the collision, we are in trouble. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3707) UUID can be non unique for a huge samples
[ https://issues.apache.org/jira/browse/HDDS-3707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17132675#comment-17132675 ] Arpit Agarwal commented on HDDS-3707: - Hi [~maobaolong] the probability is so infinitesimal I don't think it is worth trying to fix it. :) > UUID can be non unique for a huge samples > - > > Key: HDDS-3707 > URL: https://issues.apache.org/jira/browse/HDDS-3707 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode, Ozone Manager, SCM >Affects Versions: 0.7.0 >Reporter: maobaolong >Priority: Minor > Labels: Triaged > > Now, we have used UUID as id for many places, for example, DataNodeId, > pipelineId. I believe that it should be pretty less chance to met collision, > but, if met the collision, we are in trouble. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3755) Storage-class support for Ozone
[ https://issues.apache.org/jira/browse/HDDS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17129832#comment-17129832 ] Arpit Agarwal commented on HDDS-3755: - Thanks for kicking this off [~elek]. This is a pretty detailed writeup. I have couple of very high-level thoughts (also commented on the doc): # Would like to see better defined use-cases, and some discussion on the use-cases before we get into the design. # It may make sense to decouple EC from this feature. I think they are big enough to stand independently, and EC can be plugged into the storage-class framework later. Still going through the doc in detail. > Storage-class support for Ozone > --- > > Key: HDDS-3755 > URL: https://issues.apache.org/jira/browse/HDDS-3755 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Marton Elek >Assignee: Marton Elek >Priority: Major > > Use a storage-class as an abstraction which combines replication > configuration, container states and transitions. > See this thread for the detailed design doc: > > [https://lists.apache.org/thread.html/r1e2a5d5581abe9dd09834305ca65a6807f37bd229a07b8b31bda32ad%40%3Cozone-dev.hadoop.apache.org%3E] > which is also uploaded to here: > https://hackmd.io/4kxufJBOQNaKn7PKFK_6OQ?edit -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3090) Fix logging in OMFileRequest and OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-3090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3090: Labels: Triaged (was: Triaged pull-request-available) > Fix logging in OMFileRequest and OzoneManager > - > > Key: HDDS-3090 > URL: https://issues.apache.org/jira/browse/HDDS-3090 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Supratim Deka >Assignee: Supratim Deka >Priority: Trivial > Labels: Triaged > Time Spent: 10m > Remaining Estimate: 0h > > HDDS-2940 introduced a INFO level log in > hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileCreateRequest.java > This needs to be a TRACE because it occurs in the regular file create path. > Also, trace logs introduced in OzoneManager and OMFileRequest.java need to be > parameterized. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3691) Update gRPC to 1.29.0
[ https://issues.apache.org/jira/browse/HDDS-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3691: Component/s: (was: Ratis) build > Update gRPC to 1.29.0 > - > > Key: HDDS-3691 > URL: https://issues.apache.org/jira/browse/HDDS-3691 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: build >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Labels: pull-request-available > Fix For: 0.6.0 > > > This is to update the grpc compile version to 1.29.0 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3659) put a new file to exist key with different factor or type don't update the omkeyinfo
[ https://issues.apache.org/jira/browse/HDDS-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3659: Component/s: (was: om) > put a new file to exist key with different factor or type don't update the > omkeyinfo > > > Key: HDDS-3659 > URL: https://issues.apache.org/jira/browse/HDDS-3659 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Affects Versions: 0.6.0 >Reporter: maobaolong >Priority: Critical > Labels: Triaged > > bin/ozone sh key put -r THREE /myvol/mybucket/NOTICE.txt NOTICE.txt > bin/ozone sh key put -r ONE /myvol/mybucket/NOTICE.txt NOTICE.txt > bin/ozone sh key info /myvol/mybucket/NOTICE.txt NOTICE.txt > it should be ONE -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3575) Implement trash clean up on OM
[ https://issues.apache.org/jira/browse/HDDS-3575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-3575: Component/s: (was: om) > Implement trash clean up on OM > -- > > Key: HDDS-3575 > URL: https://issues.apache.org/jira/browse/HDDS-3575 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: Ozone Manager >Reporter: Siyao Meng >Priority: Critical > Labels: Triaged > > We need to implement trash clean up on Ozone Manager. Otherwise files deleted > with Hadoop shell will not be cleaned up. > Inspired from HDDS-3574. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-371) Add RetriableException class in Ozone
[ https://issues.apache.org/jira/browse/HDDS-371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-371: --- Component/s: Ozone Client > Add RetriableException class in Ozone > - > > Key: HDDS-371 > URL: https://issues.apache.org/jira/browse/HDDS-371 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Labels: TriagePending > > Certain Exception thrown by a server can be because server is in a state > where request cannot be processed temporarily. > Ozone Client may retry the request. If the service is up, the server may be > able to > process a retried request. This Jira aims to introduce notion of > RetriableException in Ozone. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-537) ozone sh vol complains for valid length even when the length is valid for 3 characters
[
https://issues.apache.org/jira/browse/HDDS-537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-537:
---
Component/s: Ozone CLI
> ozone sh vol complains for valid length even when the length is valid for 3
> characters
> --
>
> Key: HDDS-537
> URL: https://issues.apache.org/jira/browse/HDDS-537
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone CLI
>Reporter: Namit Maheshwari
>Assignee: Namit Maheshwari
>Priority: Major
> Labels: Triaged
>
> {code:java}
> [root@ctr-e138-1518143905142-481027-01-02 bin]# ./ozone sh vol info abc
> Bucket or Volume length is illegal, valid length is 3-63 characters
> {code}
> Here, the length is already 3 characters still it throws errors, that valid
> length is 3-63 characters
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-698) Support Topology Awareness for Ozone
[ https://issues.apache.org/jira/browse/HDDS-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-698: --- Component/s: SCM > Support Topology Awareness for Ozone > > > Key: HDDS-698 > URL: https://issues.apache.org/jira/browse/HDDS-698 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: SCM >Reporter: Xiaoyu Yao >Assignee: Sammi Chen >Priority: Blocker > Fix For: 0.6.0 > > Attachments: HDDS-698.000.patch, network-topology-default.xml, > network-topology-nodegroup.xml > > > This is an umbrella JIRA to add topology aware support for Ozone Pipelines, > Containers and Blocks. Long time since HDFS is created, we provide > rack/nodegroup awareness for reliability and high performance for data > access. Ozone need a similar mechanism and this can be more flexible for > cloud scenarios. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1042) Class-compatibility: Remove or shade Guava + leveldbjni in Ozonefs
[
https://issues.apache.org/jira/browse/HDDS-1042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-1042:
Component/s: build
> Class-compatibility: Remove or shade Guava + leveldbjni in Ozonefs
> ---
>
> Key: HDDS-1042
> URL: https://issues.apache.org/jira/browse/HDDS-1042
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: build
>Affects Versions: 0.4.0
>Reporter: Gopal Vijayaraghavan
>Priority: Blocker
> Labels: Triaged
>
> Ozonefs client JARs have unshaded guava which has version incompatibilities.
> https://github.com/apache/hadoop/blob/trunk/hadoop-ozone/ozonefs/pom.xml#L53
> {code}
> com.google.guava:guava
> org.fusesource.leveldbjni:leveldbjni-all
> {code}
> Not sure if the ratis-server or rocksdb are needed either.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
