[jira] [Updated] (HDFS-12337) Ozone: Concurrent RocksDB open calls fail because of "No locks available"

2017-08-23 Thread Anu Engineer (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anu Engineer updated HDFS-12337:

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

[~msingh] Thank you for taking care of this issue. I have committed this to the 
feature branch.

> Ozone: Concurrent RocksDB open calls fail because of "No locks available"
> -
>
> Key: HDFS-12337
> URL: https://issues.apache.org/jira/browse/HDFS-12337
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Fix For: HDFS-7240
>
> Attachments: HDFS-12337-HDFS-7240.001.patch
>
>
> HDFS-12216 fixes the issue with the static container cache and re-using the 
> same container port on datanode restart. However TestKeys still fails after 
> HDFS-12216 is fixed.
> The test is now failing because Concurrent RocksDB open calls fail. In the 
> current code BlockDeleting service and Dispatcher tries to open the db 
> concurrently.
> This jira will also fix the keepPort property for Ratis Container port and 
> set the reuse address property for XceiverServerRatis correctly.
> {code}
> 2017-08-22 16:51:34,453 [BlockDeletingService#1] INFO  utils.RocksDBStore 
> (RocksDBStore.java:(64)) - opening db file 
> /Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.db
> 2017-08-22 16:51:34,460 [nioEventLoopGroup-9-1] INFO  logging.LoggingHandler 
> (Slf4JLogger.java:info(101)) - [id: 0x8822cd3d, /0.0.0.0:57044] RECEIVED: 
> [id: 0x61367e6f, /127.0.0.1:57173 => /127.0.0
> .1:57044]
> 2017-08-22 16:51:34,461 [nioEventLoopGroup-10-1] INFO  utils.RocksDBStore 
> (RocksDBStore.java:(64)) - opening db file 
> /Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.db
> 2017-08-22 16:51:34,465 [nioEventLoopGroup-10-1] INFO  utils.RocksDBStore 
> (RocksDBStore.java:(67)) - Failed init RocksDB, db path : 
> /Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.dbexception
>  org.rocksdb.RocksDBException: lock 
> /Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.db/LOCK:
>  No locks available
> 2017-08-22 16:51:34,465 [BlockDeletingService#1] INFO  
> background.BlockDeletingService (BlockDeletingService.java:call(203))  - 
> The elapsed time of task@70a576ee for deleting blocks: 12ms.
> 2017-08-22 16:51:34,474 [nioEventLoopGroup-10-1] INFO  impl.Dispatcher 
> (ContainerUtils.java:logAndReturnError(129))  - Operation: GetKey : Trace 
> ID: 73f19131-f63b-459a-8f09-9a3db893a296 : Message: 
> 621a3b15-b9fc-4d49-a6e6-29d4c40cc91f : Result: UNABLE_TO_READ_METADATA_DB
> 2017-08-22 16:51:34,475 [Thread-382] INFO  exceptions.OzoneExceptionMapper 
> (OzoneExceptionMapper.java:toResponse(39)) ozone  
> c2a23759-c76f-49ea-b574-f0802a4e5b75/c0df3a48-f75b-4b5e-b1bd-c189ce698056/13b3d486-3d7a-49e4-bc9d-1ef63e674548
>  hdfs 73f19131-f63b-459a-8f09-9a3db893a296 - Returning exception. ex: 
> {"httpCode":500,"shortMessage":"internalServerError","resource":"hdfs","message":"621a3b15-b9fc-4d49-a6e6-29d4c40cc91f","requestID":"73f19131-f63b-459a-8f09-9a3db893a296","hostName":"hw13605.local"}
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12337) Ozone: Concurrent RocksDB open calls fail because of "No locks available"

2017-08-22 Thread Mukul Kumar Singh (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mukul Kumar Singh updated HDFS-12337:
-
Attachment: HDFS-12337-HDFS-7240.001.patch

> Ozone: Concurrent RocksDB open calls fail because of "No locks available"
> -
>
> Key: HDFS-12337
> URL: https://issues.apache.org/jira/browse/HDFS-12337
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Fix For: HDFS-7240
>
> Attachments: HDFS-12337-HDFS-7240.001.patch
>
>
> HDFS-12216 fixes the issue with the static container cache and re-using the 
> same container port on datanode restart. However TestKeys still fails after 
> HDFS-12216 is fixed.
> The test is now failing because Concurrent RocksDB open calls fail. In the 
> current code BlockDeleting service and Dispatcher tries to open the db 
> concurrently.
> This jira will also fix the keepPort property for Ratis Container port and 
> set the reuse address property for XceiverServerRatis correctly.
> {code}
> 2017-08-22 16:51:34,453 [BlockDeletingService#1] INFO  utils.RocksDBStore 
> (RocksDBStore.java:(64)) - opening db file 
> /Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.db
> 2017-08-22 16:51:34,460 [nioEventLoopGroup-9-1] INFO  logging.LoggingHandler 
> (Slf4JLogger.java:info(101)) - [id: 0x8822cd3d, /0.0.0.0:57044] RECEIVED: 
> [id: 0x61367e6f, /127.0.0.1:57173 => /127.0.0
> .1:57044]
> 2017-08-22 16:51:34,461 [nioEventLoopGroup-10-1] INFO  utils.RocksDBStore 
> (RocksDBStore.java:(64)) - opening db file 
> /Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.db
> 2017-08-22 16:51:34,465 [nioEventLoopGroup-10-1] INFO  utils.RocksDBStore 
> (RocksDBStore.java:(67)) - Failed init RocksDB, db path : 
> /Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.dbexception
>  org.rocksdb.RocksDBException: lock 
> /Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.db/LOCK:
>  No locks available
> 2017-08-22 16:51:34,465 [BlockDeletingService#1] INFO  
> background.BlockDeletingService (BlockDeletingService.java:call(203))  - 
> The elapsed time of task@70a576ee for deleting blocks: 12ms.
> 2017-08-22 16:51:34,474 [nioEventLoopGroup-10-1] INFO  impl.Dispatcher 
> (ContainerUtils.java:logAndReturnError(129))  - Operation: GetKey : Trace 
> ID: 73f19131-f63b-459a-8f09-9a3db893a296 : Message: 
> 621a3b15-b9fc-4d49-a6e6-29d4c40cc91f : Result: UNABLE_TO_READ_METADATA_DB
> 2017-08-22 16:51:34,475 [Thread-382] INFO  exceptions.OzoneExceptionMapper 
> (OzoneExceptionMapper.java:toResponse(39)) ozone  
> c2a23759-c76f-49ea-b574-f0802a4e5b75/c0df3a48-f75b-4b5e-b1bd-c189ce698056/13b3d486-3d7a-49e4-bc9d-1ef63e674548
>  hdfs 73f19131-f63b-459a-8f09-9a3db893a296 - Returning exception. ex: 
> {"httpCode":500,"shortMessage":"internalServerError","resource":"hdfs","message":"621a3b15-b9fc-4d49-a6e6-29d4c40cc91f","requestID":"73f19131-f63b-459a-8f09-9a3db893a296","hostName":"hw13605.local"}
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12337) Ozone: Concurrent RocksDB open calls fail because of "No locks available"

2017-08-22 Thread Mukul Kumar Singh (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mukul Kumar Singh updated HDFS-12337:
-
Status: Patch Available  (was: Open)

> Ozone: Concurrent RocksDB open calls fail because of "No locks available"
> -
>
> Key: HDFS-12337
> URL: https://issues.apache.org/jira/browse/HDFS-12337
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Fix For: HDFS-7240
>
> Attachments: HDFS-12337-HDFS-7240.001.patch
>
>
> HDFS-12216 fixes the issue with the static container cache and re-using the 
> same container port on datanode restart. However TestKeys still fails after 
> HDFS-12216 is fixed.
> The test is now failing because Concurrent RocksDB open calls fail. In the 
> current code BlockDeleting service and Dispatcher tries to open the db 
> concurrently.
> This jira will also fix the keepPort property for Ratis Container port and 
> set the reuse address property for XceiverServerRatis correctly.
> {code}
> 2017-08-22 16:51:34,453 [BlockDeletingService#1] INFO  utils.RocksDBStore 
> (RocksDBStore.java:(64)) - opening db file 
> /Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.db
> 2017-08-22 16:51:34,460 [nioEventLoopGroup-9-1] INFO  logging.LoggingHandler 
> (Slf4JLogger.java:info(101)) - [id: 0x8822cd3d, /0.0.0.0:57044] RECEIVED: 
> [id: 0x61367e6f, /127.0.0.1:57173 => /127.0.0
> .1:57044]
> 2017-08-22 16:51:34,461 [nioEventLoopGroup-10-1] INFO  utils.RocksDBStore 
> (RocksDBStore.java:(64)) - opening db file 
> /Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.db
> 2017-08-22 16:51:34,465 [nioEventLoopGroup-10-1] INFO  utils.RocksDBStore 
> (RocksDBStore.java:(67)) - Failed init RocksDB, db path : 
> /Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.dbexception
>  org.rocksdb.RocksDBException: lock 
> /Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.db/LOCK:
>  No locks available
> 2017-08-22 16:51:34,465 [BlockDeletingService#1] INFO  
> background.BlockDeletingService (BlockDeletingService.java:call(203))  - 
> The elapsed time of task@70a576ee for deleting blocks: 12ms.
> 2017-08-22 16:51:34,474 [nioEventLoopGroup-10-1] INFO  impl.Dispatcher 
> (ContainerUtils.java:logAndReturnError(129))  - Operation: GetKey : Trace 
> ID: 73f19131-f63b-459a-8f09-9a3db893a296 : Message: 
> 621a3b15-b9fc-4d49-a6e6-29d4c40cc91f : Result: UNABLE_TO_READ_METADATA_DB
> 2017-08-22 16:51:34,475 [Thread-382] INFO  exceptions.OzoneExceptionMapper 
> (OzoneExceptionMapper.java:toResponse(39)) ozone  
> c2a23759-c76f-49ea-b574-f0802a4e5b75/c0df3a48-f75b-4b5e-b1bd-c189ce698056/13b3d486-3d7a-49e4-bc9d-1ef63e674548
>  hdfs 73f19131-f63b-459a-8f09-9a3db893a296 - Returning exception. ex: 
> {"httpCode":500,"shortMessage":"internalServerError","resource":"hdfs","message":"621a3b15-b9fc-4d49-a6e6-29d4c40cc91f","requestID":"73f19131-f63b-459a-8f09-9a3db893a296","hostName":"hw13605.local"}
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org