[jira] [Work logged] (HDFS-15714) HDFS Provided Storage Read/Write Mount Support On-the-fly
[ https://issues.apache.org/jira/browse/HDFS-15714?focusedWorklogId=692198=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-692198 ] ASF GitHub Bot logged work on HDFS-15714: - Author: ASF GitHub Bot Created on: 08/Dec/21 02:51 Start Date: 08/Dec/21 02:51 Worklog Time Spent: 10m Work Description: Zhangshunyu removed a comment on pull request #2655: URL: https://github.com/apache/hadoop/pull/2655#issuecomment-815380601 it seems the 'cacheUsed' in SimpleReadCacheManager will be 0 when namenode restart, this value in stored only in memory. So the 'evictBlocks' would not get correct used value to evict the blocks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 692198) Time Spent: 2h 50m (was: 2h 40m) > HDFS Provided Storage Read/Write Mount Support On-the-fly > - > > Key: HDFS-15714 > URL: https://issues.apache.org/jira/browse/HDFS-15714 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 3.4.0 >Reporter: Feilong He >Assignee: Feilong He >Priority: Major > Labels: pull-request-available > Attachments: HDFS-15714-01.patch, HDFS-15714-02.patch, > HDFS_Provided_Storage_Design-V1.pdf, HDFS_Provided_Storage_Performance-V1.pdf > > Time Spent: 2h 50m > Remaining Estimate: 0h > > HDFS Provided Storage (PS) is a feature to tier HDFS over other file systems. > In HDFS-9806, PROVIDED storage type was introduced to HDFS. Through > configuring external storage with PROVIDED tag for DataNode, user can enable > application to access data stored externally from HDFS side. However, there > are two issues need to be addressed. Firstly, mounting external storage > on-the-fly, namely dynamic mount, is lacking. It is necessary to get it > supported to flexibly combine HDFS with an external storage at runtime. > Secondly, PS write is not supported by current HDFS. But in real > applications, it is common to transfer data bi-directionally for read/write > between HDFS and external storage. > Through this JIRA, we are presenting our work for PS write support and > dynamic mount support for both read & write. Please note in the community > several JIRAs have been filed for these topics. Our work is based on these > previous community work, with new design & implementation to support called > writeBack mount and enable admin to add any mount on-the-fly. We appreciate > those folks in the community for their great contribution! See their pending > JIRAs: HDFS-14805 & HDFS-12090. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15714) HDFS Provided Storage Read/Write Mount Support On-the-fly
[ https://issues.apache.org/jira/browse/HDFS-15714?focusedWorklogId=692197=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-692197 ] ASF GitHub Bot logged work on HDFS-15714: - Author: ASF GitHub Bot Created on: 08/Dec/21 02:50 Start Date: 08/Dec/21 02:50 Worklog Time Spent: 10m Work Description: Zhangshunyu removed a comment on pull request #2655: URL: https://github.com/apache/hadoop/pull/2655#issuecomment-818374335 Currentlly, alias map is based on LevelDB, and it does not support namenode HA, right? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 692197) Time Spent: 2h 40m (was: 2.5h) > HDFS Provided Storage Read/Write Mount Support On-the-fly > - > > Key: HDFS-15714 > URL: https://issues.apache.org/jira/browse/HDFS-15714 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 3.4.0 >Reporter: Feilong He >Assignee: Feilong He >Priority: Major > Labels: pull-request-available > Attachments: HDFS-15714-01.patch, HDFS-15714-02.patch, > HDFS_Provided_Storage_Design-V1.pdf, HDFS_Provided_Storage_Performance-V1.pdf > > Time Spent: 2h 40m > Remaining Estimate: 0h > > HDFS Provided Storage (PS) is a feature to tier HDFS over other file systems. > In HDFS-9806, PROVIDED storage type was introduced to HDFS. Through > configuring external storage with PROVIDED tag for DataNode, user can enable > application to access data stored externally from HDFS side. However, there > are two issues need to be addressed. Firstly, mounting external storage > on-the-fly, namely dynamic mount, is lacking. It is necessary to get it > supported to flexibly combine HDFS with an external storage at runtime. > Secondly, PS write is not supported by current HDFS. But in real > applications, it is common to transfer data bi-directionally for read/write > between HDFS and external storage. > Through this JIRA, we are presenting our work for PS write support and > dynamic mount support for both read & write. Please note in the community > several JIRAs have been filed for these topics. Our work is based on these > previous community work, with new design & implementation to support called > writeBack mount and enable admin to add any mount on-the-fly. We appreciate > those folks in the community for their great contribution! See their pending > JIRAs: HDFS-14805 & HDFS-12090. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15714) HDFS Provided Storage Read/Write Mount Support On-the-fly
[ https://issues.apache.org/jira/browse/HDFS-15714?focusedWorklogId=581505=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-581505 ] ASF GitHub Bot logged work on HDFS-15714: - Author: ASF GitHub Bot Created on: 13/Apr/21 03:11 Start Date: 13/Apr/21 03:11 Worklog Time Spent: 10m Work Description: PHILO-HE commented on pull request #2655: URL: https://github.com/apache/hadoop/pull/2655#issuecomment-818398537 1) Yes, LevelDB based AliasMap is recommended to user. And text based AliasMap is just for the purpose of unit test. In this patch, we made few code changes for AliasMap. You may note that it was initially introduced by the community few years ago. 2) For namenode HA, we had not tested this feature on that. I think there are two main considerations. Firstly, mount operation should be recovered in new NN. Thus, the mounted remote storages are "visible" to new active NN. Since we log mount info in edit log for each mount request, this may be not a problem. Secondly, key info currently kept in memory should be available to other NNs, e.g., key tracking info in syncing data to remote storage to guarantee data consistency even though active NN is shifted. Frankly speaking, provided storage is still an experimental feature. So there may still exist a large gap for productization. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 581505) Time Spent: 2.5h (was: 2h 20m) > HDFS Provided Storage Read/Write Mount Support On-the-fly > - > > Key: HDFS-15714 > URL: https://issues.apache.org/jira/browse/HDFS-15714 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 3.4.0 >Reporter: Feilong He >Assignee: Feilong He >Priority: Major > Labels: pull-request-available > Attachments: HDFS-15714-01.patch, > HDFS_Provided_Storage_Design-V1.pdf, HDFS_Provided_Storage_Performance-V1.pdf > > Time Spent: 2.5h > Remaining Estimate: 0h > > HDFS Provided Storage (PS) is a feature to tier HDFS over other file systems. > In HDFS-9806, PROVIDED storage type was introduced to HDFS. Through > configuring external storage with PROVIDED tag for DataNode, user can enable > application to access data stored externally from HDFS side. However, there > are two issues need to be addressed. Firstly, mounting external storage > on-the-fly, namely dynamic mount, is lacking. It is necessary to get it > supported to flexibly combine HDFS with an external storage at runtime. > Secondly, PS write is not supported by current HDFS. But in real > applications, it is common to transfer data bi-directionally for read/write > between HDFS and external storage. > Through this JIRA, we are presenting our work for PS write support and > dynamic mount support for both read & write. Please note in the community > several JIRAs have been filed for these topics. Our work is based on these > previous community work, with new design & implementation to support called > writeBack mount and enable admin to add any mount on-the-fly. We appreciate > those folks in the community for their great contribution! See their pending > JIRAs: HDFS-14805 & HDFS-12090. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15714) HDFS Provided Storage Read/Write Mount Support On-the-fly
[ https://issues.apache.org/jira/browse/HDFS-15714?focusedWorklogId=581491=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-581491 ] ASF GitHub Bot logged work on HDFS-15714: - Author: ASF GitHub Bot Created on: 13/Apr/21 01:58 Start Date: 13/Apr/21 01:58 Worklog Time Spent: 10m Work Description: Zhangshunyu edited a comment on pull request #2655: URL: https://github.com/apache/hadoop/pull/2655#issuecomment-818374335 Currentlly, alias map is based on LevelDB, and it does not support namenode HA, right? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 581491) Time Spent: 2h 20m (was: 2h 10m) > HDFS Provided Storage Read/Write Mount Support On-the-fly > - > > Key: HDFS-15714 > URL: https://issues.apache.org/jira/browse/HDFS-15714 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 3.4.0 >Reporter: Feilong He >Assignee: Feilong He >Priority: Major > Labels: pull-request-available > Attachments: HDFS-15714-01.patch, > HDFS_Provided_Storage_Design-V1.pdf, HDFS_Provided_Storage_Performance-V1.pdf > > Time Spent: 2h 20m > Remaining Estimate: 0h > > HDFS Provided Storage (PS) is a feature to tier HDFS over other file systems. > In HDFS-9806, PROVIDED storage type was introduced to HDFS. Through > configuring external storage with PROVIDED tag for DataNode, user can enable > application to access data stored externally from HDFS side. However, there > are two issues need to be addressed. Firstly, mounting external storage > on-the-fly, namely dynamic mount, is lacking. It is necessary to get it > supported to flexibly combine HDFS with an external storage at runtime. > Secondly, PS write is not supported by current HDFS. But in real > applications, it is common to transfer data bi-directionally for read/write > between HDFS and external storage. > Through this JIRA, we are presenting our work for PS write support and > dynamic mount support for both read & write. Please note in the community > several JIRAs have been filed for these topics. Our work is based on these > previous community work, with new design & implementation to support called > writeBack mount and enable admin to add any mount on-the-fly. We appreciate > those folks in the community for their great contribution! See their pending > JIRAs: HDFS-14805 & HDFS-12090. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15714) HDFS Provided Storage Read/Write Mount Support On-the-fly
[ https://issues.apache.org/jira/browse/HDFS-15714?focusedWorklogId=581490=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-581490 ] ASF GitHub Bot logged work on HDFS-15714: - Author: ASF GitHub Bot Created on: 13/Apr/21 01:57 Start Date: 13/Apr/21 01:57 Worklog Time Spent: 10m Work Description: Zhangshunyu commented on pull request #2655: URL: https://github.com/apache/hadoop/pull/2655#issuecomment-818374335 Currentlly, alias map is based on LevelDB, and it is not support namenode HA, right? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 581490) Time Spent: 2h 10m (was: 2h) > HDFS Provided Storage Read/Write Mount Support On-the-fly > - > > Key: HDFS-15714 > URL: https://issues.apache.org/jira/browse/HDFS-15714 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 3.4.0 >Reporter: Feilong He >Assignee: Feilong He >Priority: Major > Labels: pull-request-available > Attachments: HDFS-15714-01.patch, > HDFS_Provided_Storage_Design-V1.pdf, HDFS_Provided_Storage_Performance-V1.pdf > > Time Spent: 2h 10m > Remaining Estimate: 0h > > HDFS Provided Storage (PS) is a feature to tier HDFS over other file systems. > In HDFS-9806, PROVIDED storage type was introduced to HDFS. Through > configuring external storage with PROVIDED tag for DataNode, user can enable > application to access data stored externally from HDFS side. However, there > are two issues need to be addressed. Firstly, mounting external storage > on-the-fly, namely dynamic mount, is lacking. It is necessary to get it > supported to flexibly combine HDFS with an external storage at runtime. > Secondly, PS write is not supported by current HDFS. But in real > applications, it is common to transfer data bi-directionally for read/write > between HDFS and external storage. > Through this JIRA, we are presenting our work for PS write support and > dynamic mount support for both read & write. Please note in the community > several JIRAs have been filed for these topics. Our work is based on these > previous community work, with new design & implementation to support called > writeBack mount and enable admin to add any mount on-the-fly. We appreciate > those folks in the community for their great contribution! See their pending > JIRAs: HDFS-14805 & HDFS-12090. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15714) HDFS Provided Storage Read/Write Mount Support On-the-fly
[ https://issues.apache.org/jira/browse/HDFS-15714?focusedWorklogId=578845=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-578845 ] ASF GitHub Bot logged work on HDFS-15714: - Author: ASF GitHub Bot Created on: 08/Apr/21 02:59 Start Date: 08/Apr/21 02:59 Worklog Time Spent: 10m Work Description: Zhangshunyu commented on pull request #2655: URL: https://github.com/apache/hadoop/pull/2655#issuecomment-815409363 @PHILO-HE Thanks for this reply. "block not found" exception is fixed, its our config issue not a problem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 578845) Time Spent: 2h (was: 1h 50m) > HDFS Provided Storage Read/Write Mount Support On-the-fly > - > > Key: HDFS-15714 > URL: https://issues.apache.org/jira/browse/HDFS-15714 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 3.4.0 >Reporter: Feilong He >Assignee: Feilong He >Priority: Major > Labels: pull-request-available > Attachments: HDFS-15714-01.patch, > HDFS_Provided_Storage_Design-V1.pdf, HDFS_Provided_Storage_Performance-V1.pdf > > Time Spent: 2h > Remaining Estimate: 0h > > HDFS Provided Storage (PS) is a feature to tier HDFS over other file systems. > In HDFS-9806, PROVIDED storage type was introduced to HDFS. Through > configuring external storage with PROVIDED tag for DataNode, user can enable > application to access data stored externally from HDFS side. However, there > are two issues need to be addressed. Firstly, mounting external storage > on-the-fly, namely dynamic mount, is lacking. It is necessary to get it > supported to flexibly combine HDFS with an external storage at runtime. > Secondly, PS write is not supported by current HDFS. But in real > applications, it is common to transfer data bi-directionally for read/write > between HDFS and external storage. > Through this JIRA, we are presenting our work for PS write support and > dynamic mount support for both read & write. Please note in the community > several JIRAs have been filed for these topics. Our work is based on these > previous community work, with new design & implementation to support called > writeBack mount and enable admin to add any mount on-the-fly. We appreciate > those folks in the community for their great contribution! See their pending > JIRAs: HDFS-14805 & HDFS-12090. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15714) HDFS Provided Storage Read/Write Mount Support On-the-fly
[ https://issues.apache.org/jira/browse/HDFS-15714?focusedWorklogId=578844=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-578844 ] ASF GitHub Bot logged work on HDFS-15714: - Author: ASF GitHub Bot Created on: 08/Apr/21 02:56 Start Date: 08/Apr/21 02:56 Worklog Time Spent: 10m Work Description: PHILO-HE commented on pull request #2655: URL: https://github.com/apache/hadoop/pull/2655#issuecomment-815408368 Thanks for your feedback! Indeed, as you mentioned, `cacheUsed` is not persisted. We will fix this issue in the future. BTW, we cannot reproduce your previously mentioned "block not found" exception. Please share your key config and deployment info. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 578844) Time Spent: 1h 50m (was: 1h 40m) > HDFS Provided Storage Read/Write Mount Support On-the-fly > - > > Key: HDFS-15714 > URL: https://issues.apache.org/jira/browse/HDFS-15714 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 3.4.0 >Reporter: Feilong He >Assignee: Feilong He >Priority: Major > Labels: pull-request-available > Attachments: HDFS-15714-01.patch, > HDFS_Provided_Storage_Design-V1.pdf, HDFS_Provided_Storage_Performance-V1.pdf > > Time Spent: 1h 50m > Remaining Estimate: 0h > > HDFS Provided Storage (PS) is a feature to tier HDFS over other file systems. > In HDFS-9806, PROVIDED storage type was introduced to HDFS. Through > configuring external storage with PROVIDED tag for DataNode, user can enable > application to access data stored externally from HDFS side. However, there > are two issues need to be addressed. Firstly, mounting external storage > on-the-fly, namely dynamic mount, is lacking. It is necessary to get it > supported to flexibly combine HDFS with an external storage at runtime. > Secondly, PS write is not supported by current HDFS. But in real > applications, it is common to transfer data bi-directionally for read/write > between HDFS and external storage. > Through this JIRA, we are presenting our work for PS write support and > dynamic mount support for both read & write. Please note in the community > several JIRAs have been filed for these topics. Our work is based on these > previous community work, with new design & implementation to support called > writeBack mount and enable admin to add any mount on-the-fly. We appreciate > those folks in the community for their great contribution! See their pending > JIRAs: HDFS-14805 & HDFS-12090. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15714) HDFS Provided Storage Read/Write Mount Support On-the-fly
[ https://issues.apache.org/jira/browse/HDFS-15714?focusedWorklogId=578821=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-578821 ] ASF GitHub Bot logged work on HDFS-15714: - Author: ASF GitHub Bot Created on: 08/Apr/21 01:29 Start Date: 08/Apr/21 01:29 Worklog Time Spent: 10m Work Description: Zhangshunyu commented on pull request #2655: URL: https://github.com/apache/hadoop/pull/2655#issuecomment-815380601 it seems the 'cacheUsed' in SimpleReadCacheManager will be 0 when namenode restart, this value in stored only in memory. So the 'evictBlocks' would not get correct used value to evict the blocks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 578821) Time Spent: 1h 40m (was: 1.5h) > HDFS Provided Storage Read/Write Mount Support On-the-fly > - > > Key: HDFS-15714 > URL: https://issues.apache.org/jira/browse/HDFS-15714 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 3.4.0 >Reporter: Feilong He >Assignee: Feilong He >Priority: Major > Labels: pull-request-available > Attachments: HDFS-15714-01.patch, > HDFS_Provided_Storage_Design-V1.pdf, HDFS_Provided_Storage_Performance-V1.pdf > > Time Spent: 1h 40m > Remaining Estimate: 0h > > HDFS Provided Storage (PS) is a feature to tier HDFS over other file systems. > In HDFS-9806, PROVIDED storage type was introduced to HDFS. Through > configuring external storage with PROVIDED tag for DataNode, user can enable > application to access data stored externally from HDFS side. However, there > are two issues need to be addressed. Firstly, mounting external storage > on-the-fly, namely dynamic mount, is lacking. It is necessary to get it > supported to flexibly combine HDFS with an external storage at runtime. > Secondly, PS write is not supported by current HDFS. But in real > applications, it is common to transfer data bi-directionally for read/write > between HDFS and external storage. > Through this JIRA, we are presenting our work for PS write support and > dynamic mount support for both read & write. Please note in the community > several JIRAs have been filed for these topics. Our work is based on these > previous community work, with new design & implementation to support called > writeBack mount and enable admin to add any mount on-the-fly. We appreciate > those folks in the community for their great contribution! See their pending > JIRAs: HDFS-14805 & HDFS-12090. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15714) HDFS Provided Storage Read/Write Mount Support On-the-fly
[
https://issues.apache.org/jira/browse/HDFS-15714?focusedWorklogId=578818=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-578818
]
ASF GitHub Bot logged work on HDFS-15714:
-
Author: ASF GitHub Bot
Created on: 08/Apr/21 01:22
Start Date: 08/Apr/21 01:22
Worklog Time Spent: 10m
Work Description: Zhangshunyu removed a comment on pull request #2655:
URL: https://github.com/apache/hadoop/pull/2655#issuecomment-787926710
@PHILO-HE by the way, the read write lock in
org/apache/hadoop/hdfs/server/datanode/BlockSender.java:267
`try(AutoCloseableLock lock = datanode.data.acquireDatasetReadLock()) {`
org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java:155
`try (AutoCloseableLock l = writeLock.acquire()) {`
will dead lock
pls check the code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 578818)
Time Spent: 1.5h (was: 1h 20m)
> HDFS Provided Storage Read/Write Mount Support On-the-fly
> -
>
> Key: HDFS-15714
> URL: https://issues.apache.org/jira/browse/HDFS-15714
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, namenode
>Affects Versions: 3.4.0
>Reporter: Feilong He
>Assignee: Feilong He
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-15714-01.patch,
> HDFS_Provided_Storage_Design-V1.pdf, HDFS_Provided_Storage_Performance-V1.pdf
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> HDFS Provided Storage (PS) is a feature to tier HDFS over other file systems.
> In HDFS-9806, PROVIDED storage type was introduced to HDFS. Through
> configuring external storage with PROVIDED tag for DataNode, user can enable
> application to access data stored externally from HDFS side. However, there
> are two issues need to be addressed. Firstly, mounting external storage
> on-the-fly, namely dynamic mount, is lacking. It is necessary to get it
> supported to flexibly combine HDFS with an external storage at runtime.
> Secondly, PS write is not supported by current HDFS. But in real
> applications, it is common to transfer data bi-directionally for read/write
> between HDFS and external storage.
> Through this JIRA, we are presenting our work for PS write support and
> dynamic mount support for both read & write. Please note in the community
> several JIRAs have been filed for these topics. Our work is based on these
> previous community work, with new design & implementation to support called
> writeBack mount and enable admin to add any mount on-the-fly. We appreciate
> those folks in the community for their great contribution! See their pending
> JIRAs: HDFS-14805 & HDFS-12090.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15714) HDFS Provided Storage Read/Write Mount Support On-the-fly
[
https://issues.apache.org/jira/browse/HDFS-15714?focusedWorklogId=559352=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559352
]
ASF GitHub Bot logged work on HDFS-15714:
-
Author: ASF GitHub Bot
Created on: 01/Mar/21 12:53
Start Date: 01/Mar/21 12:53
Worklog Time Spent: 10m
Work Description: Zhangshunyu commented on pull request #2655:
URL: https://github.com/apache/hadoop/pull/2655#issuecomment-787926710
@PHILO-HE by the way, the read write lock in
org/apache/hadoop/hdfs/server/datanode/BlockSender.java:267
`try(AutoCloseableLock lock = datanode.data.acquireDatasetReadLock()) {`
org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java:155
`try (AutoCloseableLock l = writeLock.acquire()) {`
will dead lock
pls check the code
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 559352)
Time Spent: 1h 20m (was: 1h 10m)
> HDFS Provided Storage Read/Write Mount Support On-the-fly
> -
>
> Key: HDFS-15714
> URL: https://issues.apache.org/jira/browse/HDFS-15714
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, namenode
>Affects Versions: 3.4.0
>Reporter: Feilong He
>Assignee: Feilong He
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-15714-01.patch,
> HDFS_Provided_Storage_Design-V1.pdf, HDFS_Provided_Storage_Performance-V1.pdf
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> HDFS Provided Storage (PS) is a feature to tier HDFS over other file systems.
> In HDFS-9806, PROVIDED storage type was introduced to HDFS. Through
> configuring external storage with PROVIDED tag for DataNode, user can enable
> application to access data stored externally from HDFS side. However, there
> are two issues need to be addressed. Firstly, mounting external storage
> on-the-fly, namely dynamic mount, is lacking. It is necessary to get it
> supported to flexibly combine HDFS with an external storage at runtime.
> Secondly, PS write is not supported by current HDFS. But in real
> applications, it is common to transfer data bi-directionally for read/write
> between HDFS and external storage.
> Through this JIRA, we are presenting our work for PS write support and
> dynamic mount support for both read & write. Please note in the community
> several JIRAs have been filed for these topics. Our work is based on these
> previous community work, with new design & implementation to support called
> writeBack mount and enable admin to add any mount on-the-fly. We appreciate
> those folks in the community for their great contribution! See their pending
> JIRAs: HDFS-14805 & HDFS-12090.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15714) HDFS Provided Storage Read/Write Mount Support On-the-fly
[ https://issues.apache.org/jira/browse/HDFS-15714?focusedWorklogId=559209=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559209 ] ASF GitHub Bot logged work on HDFS-15714: - Author: ASF GitHub Bot Created on: 01/Mar/21 06:21 Start Date: 01/Mar/21 06:21 Worklog Time Spent: 10m Work Description: Zhangshunyu commented on pull request #2655: URL: https://github.com/apache/hadoop/pull/2655#issuecomment-787680182 @PHILO-HE I checked the config value and it is 1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 559209) Time Spent: 1h 10m (was: 1h) > HDFS Provided Storage Read/Write Mount Support On-the-fly > - > > Key: HDFS-15714 > URL: https://issues.apache.org/jira/browse/HDFS-15714 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 3.4.0 >Reporter: Feilong He >Assignee: Feilong He >Priority: Major > Labels: pull-request-available > Attachments: HDFS-15714-01.patch, > HDFS_Provided_Storage_Design-V1.pdf, HDFS_Provided_Storage_Performance-V1.pdf > > Time Spent: 1h 10m > Remaining Estimate: 0h > > HDFS Provided Storage (PS) is a feature to tier HDFS over other file systems. > In HDFS-9806, PROVIDED storage type was introduced to HDFS. Through > configuring external storage with PROVIDED tag for DataNode, user can enable > application to access data stored externally from HDFS side. However, there > are two issues need to be addressed. Firstly, mounting external storage > on-the-fly, namely dynamic mount, is lacking. It is necessary to get it > supported to flexibly combine HDFS with an external storage at runtime. > Secondly, PS write is not supported by current HDFS. But in real > applications, it is common to transfer data bi-directionally for read/write > between HDFS and external storage. > Through this JIRA, we are presenting our work for PS write support and > dynamic mount support for both read & write. Please note in the community > several JIRAs have been filed for these topics. Our work is based on these > previous community work, with new design & implementation to support called > writeBack mount and enable admin to add any mount on-the-fly. We appreciate > those folks in the community for their great contribution! See their pending > JIRAs: HDFS-14805 & HDFS-12090. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15714) HDFS Provided Storage Read/Write Mount Support On-the-fly
[ https://issues.apache.org/jira/browse/HDFS-15714?focusedWorklogId=559193=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559193 ] ASF GitHub Bot logged work on HDFS-15714: - Author: ASF GitHub Bot Created on: 01/Mar/21 05:25 Start Date: 01/Mar/21 05:25 Worklog Time Spent: 10m Work Description: PHILO-HE commented on pull request #2655: URL: https://github.com/apache/hadoop/pull/2655#issuecomment-787648343 Hi, @Zhangshunyu, in your HDFS config, did you set `dfs.provided.overreplication.factor` with a number larger than 0? This property is used to control the number of cached block (cached to HDFS from remote storage). If it is set to 0 (default setting), the redundant cached block will be deleted by HDFS, which can trigger replica not found exception. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 559193) Time Spent: 1h (was: 50m) > HDFS Provided Storage Read/Write Mount Support On-the-fly > - > > Key: HDFS-15714 > URL: https://issues.apache.org/jira/browse/HDFS-15714 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 3.4.0 >Reporter: Feilong He >Assignee: Feilong He >Priority: Major > Labels: pull-request-available > Attachments: HDFS-15714-01.patch, > HDFS_Provided_Storage_Design-V1.pdf, HDFS_Provided_Storage_Performance-V1.pdf > > Time Spent: 1h > Remaining Estimate: 0h > > HDFS Provided Storage (PS) is a feature to tier HDFS over other file systems. > In HDFS-9806, PROVIDED storage type was introduced to HDFS. Through > configuring external storage with PROVIDED tag for DataNode, user can enable > application to access data stored externally from HDFS side. However, there > are two issues need to be addressed. Firstly, mounting external storage > on-the-fly, namely dynamic mount, is lacking. It is necessary to get it > supported to flexibly combine HDFS with an external storage at runtime. > Secondly, PS write is not supported by current HDFS. But in real > applications, it is common to transfer data bi-directionally for read/write > between HDFS and external storage. > Through this JIRA, we are presenting our work for PS write support and > dynamic mount support for both read & write. Please note in the community > several JIRAs have been filed for these topics. Our work is based on these > previous community work, with new design & implementation to support called > writeBack mount and enable admin to add any mount on-the-fly. We appreciate > those folks in the community for their great contribution! See their pending > JIRAs: HDFS-14805 & HDFS-12090. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15714) HDFS Provided Storage Read/Write Mount Support On-the-fly
[
https://issues.apache.org/jira/browse/HDFS-15714?focusedWorklogId=559160=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559160
]
ASF GitHub Bot logged work on HDFS-15714:
-
Author: ASF GitHub Bot
Created on: 01/Mar/21 02:54
Start Date: 01/Mar/21 02:54
Worklog Time Spent: 10m
Work Description: Zhangshunyu commented on pull request #2655:
URL: https://github.com/apache/hadoop/pull/2655#issuecomment-787600101
@PHILO-HE Thanks for the reply. Please find the error msg:
```
ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in
creating BlockSender: {}
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not
found for BP-1079229821-192.168.0.156-1614366591560:blk_1073791203_50379. The
block may have been removed recently by the balancer or by intentionally
reducing the replication factor. This condition is usually harmless. To be
certain, please check the preceding datanode log messages for signs of a more
serious issue.
```
we use readOnly mount mode, the data is stored on s3 store, after mount we
read the data use readthrough mode, we find that some times the datanode will
throw this exception, it seems the read op request send to the wrong datanode?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 559160)
Time Spent: 50m (was: 40m)
> HDFS Provided Storage Read/Write Mount Support On-the-fly
> -
>
> Key: HDFS-15714
> URL: https://issues.apache.org/jira/browse/HDFS-15714
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, namenode
>Affects Versions: 3.4.0
>Reporter: Feilong He
>Assignee: Feilong He
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-15714-01.patch,
> HDFS_Provided_Storage_Design-V1.pdf, HDFS_Provided_Storage_Performance-V1.pdf
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> HDFS Provided Storage (PS) is a feature to tier HDFS over other file systems.
> In HDFS-9806, PROVIDED storage type was introduced to HDFS. Through
> configuring external storage with PROVIDED tag for DataNode, user can enable
> application to access data stored externally from HDFS side. However, there
> are two issues need to be addressed. Firstly, mounting external storage
> on-the-fly, namely dynamic mount, is lacking. It is necessary to get it
> supported to flexibly combine HDFS with an external storage at runtime.
> Secondly, PS write is not supported by current HDFS. But in real
> applications, it is common to transfer data bi-directionally for read/write
> between HDFS and external storage.
> Through this JIRA, we are presenting our work for PS write support and
> dynamic mount support for both read & write. Please note in the community
> several JIRAs have been filed for these topics. Our work is based on these
> previous community work, with new design & implementation to support called
> writeBack mount and enable admin to add any mount on-the-fly. We appreciate
> those folks in the community for their great contribution! See their pending
> JIRAs: HDFS-14805 & HDFS-12090.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15714) HDFS Provided Storage Read/Write Mount Support On-the-fly
[ https://issues.apache.org/jira/browse/HDFS-15714?focusedWorklogId=559159=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559159 ] ASF GitHub Bot logged work on HDFS-15714: - Author: ASF GitHub Bot Created on: 01/Mar/21 02:50 Start Date: 01/Mar/21 02:50 Worklog Time Spent: 10m Work Description: PHILO-HE commented on pull request #2655: URL: https://github.com/apache/hadoop/pull/2655#issuecomment-787599087 Hi, @Zhangshunyu, thanks for your feedback. Could you please give more details? Did you mount an external storage in readOnly mode? Or writeBack mount? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 559159) Time Spent: 40m (was: 0.5h) > HDFS Provided Storage Read/Write Mount Support On-the-fly > - > > Key: HDFS-15714 > URL: https://issues.apache.org/jira/browse/HDFS-15714 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 3.4.0 >Reporter: Feilong He >Assignee: Feilong He >Priority: Major > Labels: pull-request-available > Attachments: HDFS-15714-01.patch, > HDFS_Provided_Storage_Design-V1.pdf, HDFS_Provided_Storage_Performance-V1.pdf > > Time Spent: 40m > Remaining Estimate: 0h > > HDFS Provided Storage (PS) is a feature to tier HDFS over other file systems. > In HDFS-9806, PROVIDED storage type was introduced to HDFS. Through > configuring external storage with PROVIDED tag for DataNode, user can enable > application to access data stored externally from HDFS side. However, there > are two issues need to be addressed. Firstly, mounting external storage > on-the-fly, namely dynamic mount, is lacking. It is necessary to get it > supported to flexibly combine HDFS with an external storage at runtime. > Secondly, PS write is not supported by current HDFS. But in real > applications, it is common to transfer data bi-directionally for read/write > between HDFS and external storage. > Through this JIRA, we are presenting our work for PS write support and > dynamic mount support for both read & write. Please note in the community > several JIRAs have been filed for these topics. Our work is based on these > previous community work, with new design & implementation to support called > writeBack mount and enable admin to add any mount on-the-fly. We appreciate > those folks in the community for their great contribution! See their pending > JIRAs: HDFS-14805 & HDFS-12090. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15714) HDFS Provided Storage Read/Write Mount Support On-the-fly
[ https://issues.apache.org/jira/browse/HDFS-15714?focusedWorklogId=558844=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-558844 ] ASF GitHub Bot logged work on HDFS-15714: - Author: ASF GitHub Bot Created on: 27/Feb/21 02:13 Start Date: 27/Feb/21 02:13 Worklog Time Spent: 10m Work Description: Zhangshunyu commented on pull request #2655: URL: https://github.com/apache/hadoop/pull/2655#issuecomment-786986457 Some bugs exist on cluster that while reading data, datanode will throw block not found exception but after some retries the block can be read. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 558844) Time Spent: 0.5h (was: 20m) > HDFS Provided Storage Read/Write Mount Support On-the-fly > - > > Key: HDFS-15714 > URL: https://issues.apache.org/jira/browse/HDFS-15714 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 3.4.0 >Reporter: Feilong He >Assignee: Feilong He >Priority: Major > Labels: pull-request-available > Attachments: HDFS-15714-01.patch, > HDFS_Provided_Storage_Design-V1.pdf, HDFS_Provided_Storage_Performance-V1.pdf > > Time Spent: 0.5h > Remaining Estimate: 0h > > HDFS Provided Storage (PS) is a feature to tier HDFS over other file systems. > In HDFS-9806, PROVIDED storage type was introduced to HDFS. Through > configuring external storage with PROVIDED tag for DataNode, user can enable > application to access data stored externally from HDFS side. However, there > are two issues need to be addressed. Firstly, mounting external storage > on-the-fly, namely dynamic mount, is lacking. It is necessary to get it > supported to flexibly combine HDFS with an external storage at runtime. > Secondly, PS write is not supported by current HDFS. But in real > applications, it is common to transfer data bi-directionally for read/write > between HDFS and external storage. > Through this JIRA, we are presenting our work for PS write support and > dynamic mount support for both read & write. Please note in the community > several JIRAs have been filed for these topics. Our work is based on these > previous community work, with new design & implementation to support called > writeBack mount and enable admin to add any mount on-the-fly. We appreciate > those folks in the community for their great contribution! See their pending > JIRAs: HDFS-14805 & HDFS-12090. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15714) HDFS Provided Storage Read/Write Mount Support On-the-fly
[
https://issues.apache.org/jira/browse/HDFS-15714?focusedWorklogId=542754=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-542754
]
ASF GitHub Bot logged work on HDFS-15714:
-
Author: ASF GitHub Bot
Created on: 27/Jan/21 10:32
Start Date: 27/Jan/21 10:32
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2655:
URL: https://github.com/apache/hadoop/pull/2655#issuecomment-768192307
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 27s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files
found. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 79 new or modified test files. |
_ HDFS-15714 Compile Tests _ |
| +0 :ok: | mvndep | 13m 53s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 50s | | HDFS-15714 passed |
| +1 :green_heart: | compile | 21m 54s | | HDFS-15714 passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 18m 22s | | HDFS-15714 passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 4m 9s | | HDFS-15714 passed |
| +1 :green_heart: | mvnsite | 6m 3s | | HDFS-15714 passed |
| +1 :green_heart: | shadedclient | 27m 51s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 4m 30s | | HDFS-15714 passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 5m 53s | | HDFS-15714 passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 46s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 11m 26s | | HDFS-15714 passed |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 31s | | the patch passed |
| +1 :green_heart: | compile | 21m 7s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| -1 :x: | cc | 21m 7s |
[/diff-compile-cc-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2655/1/artifact/out/diff-compile-cc-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 30 new + 142 unchanged - 30
fixed = 172 total (was 172) |
| -1 :x: | javac | 21m 7s |
[/diff-compile-javac-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2655/1/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt)
| root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 67 new + 2006 unchanged - 27
fixed = 2073 total (was 2033) |
| +1 :green_heart: | compile | 22m 21s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| -1 :x: | cc | 22m 21s |
[/diff-compile-cc-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2655/1/artifact/out/diff-compile-cc-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
| root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 generated 33 new + 139
unchanged - 33 fixed = 172 total (was 172) |
| -1 :x: | javac | 22m 21s |
[/diff-compile-javac-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2655/1/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt)
| root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 generated 67 new + 1901
unchanged - 27 fixed = 1968 total (was 1928) |
| -0 :warning: | checkstyle | 4m 43s |
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2655/1/artifact/out/diff-checkstyle-root.txt)
| root: The patch generated 144 new + 4280 unchanged - 35 fixed = 4424 total
(was 4315) |
| +1 :green_heart: | mvnsite | 9m 18s | | the patch passed |
| -1 :x: | whitespace | 0m
[jira] [Work logged] (HDFS-15714) HDFS Provided Storage Read/Write Mount Support On-the-fly
[ https://issues.apache.org/jira/browse/HDFS-15714?focusedWorklogId=542609=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-542609 ] ASF GitHub Bot logged work on HDFS-15714: - Author: ASF GitHub Bot Created on: 27/Jan/21 03:42 Start Date: 27/Jan/21 03:42 Worklog Time Spent: 10m Work Description: PHILO-HE opened a new pull request #2655: URL: https://github.com/apache/hadoop/pull/2655 We have uploaded design doc & performance results to [Jira HDFS-15714](https://issues.apache.org/jira/browse/HDFS-15714). Please have a review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 542609) Remaining Estimate: 0h Time Spent: 10m > HDFS Provided Storage Read/Write Mount Support On-the-fly > - > > Key: HDFS-15714 > URL: https://issues.apache.org/jira/browse/HDFS-15714 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 3.4.0 >Reporter: Feilong He >Assignee: Feilong He >Priority: Major > Attachments: HDFS-15714-01.patch, > HDFS_Provided_Storage_Design-V1.pdf, HDFS_Provided_Storage_Performance-V1.pdf > > Time Spent: 10m > Remaining Estimate: 0h > > HDFS Provided Storage (PS) is a feature to tier HDFS over other file systems. > In HDFS-9806, PROVIDED storage type was introduced to HDFS. Through > configuring external storage with PROVIDED tag for DataNode, user can enable > application to access data stored externally from HDFS side. However, there > are two issues need to be addressed. Firstly, mounting external storage > on-the-fly, namely dynamic mount, is lacked. It is necessary to get it > supported to flexibly combine HDFS with an external storage at runtime. > Secondly, PS write is not supported by current HDFS. But in real > applications, it is common to transfer data bi-directionally for read/write > between HDFS and external storage. > Through this JIRA, we are presenting our work for PS write support and > dynamic mount support for both read Please note in the community > several JIRAs have been filed for these topics. Our work is based on these > previous community work, with new design & implementation to support called > writeBack mount and enable admin to add any mount on-the-fly. We appreciate > those folks in the community for their great contribution! See their pending > JIRAs: HDFS-14805 & HDFS-12090. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
