[jira] [Comment Edited] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[ https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939232#comment-16939232 ] Sammi Chen edited comment on HADOOP-15616 at 9/27/19 9:07 AM: -- Last patch LGTM, +1. Will commit soon. Thanks [~yuyang733] for the contribution. Thanks [~djp] [~ste...@apache.org] [~xyao] [~wujinhu] for review the code. Tencent COS is a very popular cloud object storage in China Market. It's good to have it in Hadoop. was (Author: sammi): Last patch LGTM, +1. Will commit soon. Thanks [~yuyang733] for the contribution. Thanks [~djp] [~ste...@apache.org] [~wujinhu] for review the code. > Incorporate Tencent Cloud COS File System Implementation > > > Key: HADOOP-15616 > URL: https://issues.apache.org/jira/browse/HADOOP-15616 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/cos >Reporter: Junping Du >Assignee: YangY >Priority: Major > Attachments: HADOOP-15616.001.patch, HADOOP-15616.002.patch, > HADOOP-15616.003.patch, HADOOP-15616.004.patch, HADOOP-15616.005.patch, > HADOOP-15616.006.patch, HADOOP-15616.007.patch, HADOOP-15616.008.patch, > HADOOP-15616.009.patch, HADOOP-15616.010.patch, HADOOP-15616.011.patch, > Tencent-COS-Integrated-v2.pdf, Tencent-COS-Integrated.pdf > > > Tencent cloud is top 2 cloud vendors in China market and the object store COS > （[https://intl.cloud.tencent.com/product/cos]） is widely used among China’s > cloud users but now it is hard for hadoop user to access data laid on COS > storage as no native support for COS in Hadoop. > This work aims to integrate Tencent cloud COS with Hadoop/Spark/Hive, just > like what we do before for S3, ADL, OSS, etc. With simple configuration, > Hadoop applications can read/write data from COS without any code change. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[ https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868835#comment-16868835 ] YangY edited comment on HADOOP-15616 at 6/20/19 6:37 PM: - [~wujinhu] Thank you for your reply. I have fixed this issue. Could you continue to help me review the other parts of this code? If any other issues found, I hope you can continue to give me a feedback. Thanks for your attention to this work again. was (Author: yuyang733): [~wujinhu] Thank you for your. I have fixed this issue. Could you continue to help me review the other parts of this code? If any other issues found, I hope you can continue to give me a feedback. Thanks for your attention to this work again. > Incorporate Tencent Cloud COS File System Implementation > > > Key: HADOOP-15616 > URL: https://issues.apache.org/jira/browse/HADOOP-15616 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/cos >Reporter: Junping Du >Assignee: YangY >Priority: Major > Attachments: HADOOP-15616.001.patch, HADOOP-15616.002.patch, > HADOOP-15616.003.patch, HADOOP-15616.004.patch, HADOOP-15616.005.patch, > HADOOP-15616.006.patch, HADOOP-15616.007.patch, HADOOP-15616.008.patch, > HADOOP-15616.009.patch, HADOOP-15616.010.patch, > Tencent-COS-Integrated-v2.pdf, Tencent-COS-Integrated.pdf > > > Tencent cloud is top 2 cloud vendors in China market and the object store COS > （[https://intl.cloud.tencent.com/product/cos]） is widely used among China’s > cloud users but now it is hard for hadoop user to access data laid on COS > storage as no native support for COS in Hadoop. > This work aims to integrate Tencent cloud COS with Hadoop/Spark/Hive, just > like what we do before for S3, ADL, OSS, etc. With simple configuration, > Hadoop applications can read/write data from COS without any code change. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[ https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16843378#comment-16843378 ] YangY edited comment on HADOOP-15616 at 5/19/19 10:09 AM: -- [~ste...@apache.org] hi, Steve, do you have time to review the comments submitted last week? or, have you encountered any problems that require me to help? I am looking forward to your reply. Thanks. was (Author: yuyang733): [~ste...@apache.org] hi，steve，do you have time to review the comments submitted last week? or, have you encountered any problems that require me to help? I am looking forward to your reply. Thanks. > Incorporate Tencent Cloud COS File System Implementation > > > Key: HADOOP-15616 > URL: https://issues.apache.org/jira/browse/HADOOP-15616 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/cos >Reporter: Junping Du >Assignee: YangY >Priority: Major > Attachments: HADOOP-15616.001.patch, HADOOP-15616.002.patch, > HADOOP-15616.003.patch, HADOOP-15616.004.patch, HADOOP-15616.005.patch, > HADOOP-15616.006.patch, HADOOP-15616.007.patch, HADOOP-15616.008.patch, > HADOOP-15616.009.patch, HADOOP-15616.010.patch, > Tencent-COS-Integrated-v2.pdf, Tencent-COS-Integrated.pdf > > > Tencent cloud is top 2 cloud vendors in China market and the object store COS > （[https://intl.cloud.tencent.com/product/cos]） is widely used among China’s > cloud users but now it is hard for hadoop user to access data laid on COS > storage as no native support for COS in Hadoop. > This work aims to integrate Tencent cloud COS with Hadoop/Spark/Hive, just > like what we do before for S3, ADL, OSS, etc. With simple configuration, > Hadoop applications can read/write data from COS without any code change. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[
https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837911#comment-16837911
]
YangY edited comment on HADOOP-15616 at 5/11/19 7:15 PM:
-
[~ste...@apache.org] Thanks for your reply. I forked the trunk branch of the
Hadoop repository and committed all contents of this patch, whose URL is
[[yuyang733/hadoop|https://github.com/yuyang733/hadoop]|[https://github.com/yuyang733/hadoop]].
You can clone it to your local path.
In order to run all unit tests, the 'auth-keys.xml' and
'contract-test-options.xml' need to be created under the directory:
'${HADOOP_HOME}/hadoop-cloud-storage-project/hadoop-cos/src/test/resource/'.
Both the template for the above two files and the test account have been sent
to you by email, and please check it out.
Alternatively, you can create it by yourself by referring to 'index.md'.
was (Author: yuyang733):
[~ste...@apache.org] Thanks for your reply. I forked the trunk branch of the
Hadoop repository and committed all contents of this patch, whose URL is
[[https://github.com/yuyang733/hadoop]|[https://github.com/yuyang733/hadoop]].
You can clone it to your local path.
In order to run all unit tests, the 'auth-keys.xml' and
'contract-test-options.xml' need to be created under the directory:
'${HADOOP_HOME}/hadoop-cloud-storage-project/hadoop-cos/src/test/resource/'.
Both the template for the above two files and the test account have been sent
to you by email, and please check it out.
Alternatively, you can create it by yourself by referring to 'index.md'.
> Incorporate Tencent Cloud COS File System Implementation
>
>
> Key: HADOOP-15616
> URL: https://issues.apache.org/jira/browse/HADOOP-15616
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/cos
>Reporter: Junping Du
>Assignee: YangY
>Priority: Major
> Attachments: HADOOP-15616.001.patch, HADOOP-15616.002.patch,
> HADOOP-15616.003.patch, HADOOP-15616.004.patch, HADOOP-15616.005.patch,
> HADOOP-15616.006.patch, HADOOP-15616.007.patch, HADOOP-15616.008.patch,
> HADOOP-15616.009.patch, HADOOP-15616.010.patch,
> Tencent-COS-Integrated-v2.pdf, Tencent-COS-Integrated.pdf
>
>
> Tencent cloud is top 2 cloud vendors in China market and the object store COS
> （[https://intl.cloud.tencent.com/product/cos]） is widely used among China’s
> cloud users but now it is hard for hadoop user to access data laid on COS
> storage as no native support for COS in Hadoop.
> This work aims to integrate Tencent cloud COS with Hadoop/Spark/Hive, just
> like what we do before for S3, ADL, OSS, etc. With simple configuration,
> Hadoop applications can read/write data from COS without any code change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[ https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836932#comment-16836932 ] YangY edited comment on HADOOP-15616 at 5/10/19 6:29 AM: - Thank [~Sammi], [~ste...@apache.org] and [~DanielZhou] for your comments. I have carefully revised the patch according to your suggestions, and uploaded a new patch named *HADOOP-15616.010.patch*. Here are the main fixed points for this patch: 1. Fixed the code style and a small number of Chinese comment issue according to the [~Sammi]'s comments; 2. Limited the use of thread pools and completed the file system contract tests according to the [~ste...@apache.org]'s comments; 3. Adjusted the order of java package import according to the [~DanielZhou]'s suggestions. 4. This revision also formats the log output. All revisions passed the unit tests. Can you help me to review it again. At the same time, I can provide a test account to verify all necessary tests. Thanks. was (Author: yuyang733): Thank [~Sammi], [~ste...@apache.org] and [~DanielZhou] for your comments. I have carefully revised the patch according to your suggestions, and uploaded a new patch named *HADOOP-15616.010.patch*. Here are the main fix points for this patch: 1. Fixed the code style and a small number of Chinese comment issue according to the [~Sammi]'s comments; 2. Limited the use of thread pools and completed the file system contract tests according to the [~ste...@apache.org]'s comments; 3. Adjusted the order of java package import according to the [~DanielZhou]'s suggestions. 4. This revision also formats the log output. All revisions passed the unit tests. Can you help me to review it again. At the same time, I can provide a test account to verify all necessary tests. Thanks. > Incorporate Tencent Cloud COS File System Implementation > > > Key: HADOOP-15616 > URL: https://issues.apache.org/jira/browse/HADOOP-15616 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/cos >Reporter: Junping Du >Assignee: YangY >Priority: Major > Attachments: HADOOP-15616.001.patch, HADOOP-15616.002.patch, > HADOOP-15616.003.patch, HADOOP-15616.004.patch, HADOOP-15616.005.patch, > HADOOP-15616.006.patch, HADOOP-15616.007.patch, HADOOP-15616.008.patch, > HADOOP-15616.009.patch, HADOOP-15616.010.patch, Junping Du.url, > Tencent-COS-Integrated-v2.pdf, Tencent-COS-Integrated.pdf > > > Tencent cloud is top 2 cloud vendors in China market and the object store COS > （[https://intl.cloud.tencent.com/product/cos]） is widely used among China’s > cloud users but now it is hard for hadoop user to access data laid on COS > storage as no native support for COS in Hadoop. > This work aims to integrate Tencent cloud COS with Hadoop/Spark/Hive, just > like what we do before for S3, ADL, OSS, etc. With simple configuration, > Hadoop applications can read/write data from COS without any code change. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[
https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829186#comment-16829186
]
Sammi Chen edited comment on HADOOP-15616 at 4/30/19 6:25 AM:
--
Comments for 009.patch,
1. CosNConfigKeys, default pool size is too big. CosN will share CPU resource
with other modules in Hadoop and with other services on in the big data system.
Be Conservative when allocating resources. Suggest 4 threads for upload pool
and 8 threads for copy pool.
{code:java}
public static final int DEFAULT_UPLOAD_THREAD_POOL_SIZE =
Runtime.getRuntime().availableProcessors() * 5;
public static final int DEFAULT_COPY_THREAD_POOL_SIZE =
Runtime.getRuntime().availableProcessors() * 3;
{code}
add comments for the unit of
{code:java}
public static final long DEFAULT_THREAD_KEEP_ALIVE_TIME = 60L;
{code}
2. function format, prefer a compact format, keep as many as parameters in a
line. There are many such places in the patch to be fixed, not just the
following one.
{code:java}
public CosNFileReadTask(
Configuration conf,
String key, NativeFileSystemStore store,
CosNInputStream.ReadBuffer readBuffer) {
{code}
preferred
{code:java}
public CosNFileReadTask(Configuration conf, String key,
NativeFileSystemStore store, CosNInputStream.ReadBuffer readBuffer) {
{code}
3. CosNFileSystem#initialize, suggest use two variable to control pool size and
waiting queue size, pool size should be relative smaller number, while waiting
queue can be bigger number, such as 256, etc.
{code:java}
this.boundedIOThreadPool = BlockingThreadPoolExecutorService.newInstance(
ioThreadPoolSize / 2, ioThreadPoolSize,
threadKeepAlive, TimeUnit.SECONDS,
"cos-transfer-thread-pool");
this.boundedCopyThreadPool =
BlockingThreadPoolExecutorService.newInstance(
copyThreadPoolSize / 2, copyThreadPoolSize, threadKeepAliveTime,
TimeUnit.SECONDS, "cos-copy-thread-pool");
{code}
4. CosNFileSystem, use PATH_DELIMITER to replace all "/", following is not
necessary, can be removed to keep code concise.
{code:java}
if (LOG.isDebugEnabled())
{code}
5. ByteBufferWrapper#close，use nested try/catch to guarantee every resource get
the chance to be released
{code:java}
void close() throws IOException {
if (null != this.byteBuffer) {
this.byteBuffer.clear();
}
if (null != randomAccessFile) {
this.randomAccessFile.close();
}
if (this.byteBuffer instanceof MappedByteBuffer) {
munmap((MappedByteBuffer) this.byteBuffer);
}
if (null != this.file && this.file.exists()) {
this.file.delete();
}
}
{code}
6. CosNOutputStream#uploadPart#, byte buffer is not released if
(store).uploadPart throw any exception
7. Enhanced the unit tests. You can leverage the code coverage report function
in intellij.
was (Author: sammi):
Comments for 009.patch,
1. CosNConfigKeys, default pool size is too big. CosN will CPU resource with
other modules in Hadoop and with other services on in the big data system. Be
Conservative when allocating resources. Suggest 4 threads for upload pool and
8 threads for copy pool.
{code:java}
public static final int DEFAULT_UPLOAD_THREAD_POOL_SIZE =
Runtime.getRuntime().availableProcessors() * 5;
public static final int DEFAULT_COPY_THREAD_POOL_SIZE =
Runtime.getRuntime().availableProcessors() * 3;
{code}
add comments for the unit of
{code:java}
public static final long DEFAULT_THREAD_KEEP_ALIVE_TIME = 60L;
{code}
2. function format, prefer a compact format, keep as many as parameters in a
line. There are many such places in the patch to be fixed, not just the
following one.
{code:java}
public CosNFileReadTask(
Configuration conf,
String key, NativeFileSystemStore store,
CosNInputStream.ReadBuffer readBuffer) {
{code}
preferred
{code:java}
public CosNFileReadTask(Configuration conf, String key,
NativeFileSystemStore store, CosNInputStream.ReadBuffer readBuffer) {
{code}
3. CosNFileSystem#initialize, suggest use two variable to control pool size and
waiting queue size, pool size should be relative smaller number, while waiting
queue can be bigger number, such as 256, etc.
{code:java}
this.boundedIOThreadPool = BlockingThreadPoolExecutorService.newInstance(
ioThreadPoolSize / 2, ioThreadPoolSize,
threadKeepAlive, TimeUnit.SECONDS,
"cos-transfer-thread-pool");
this.boundedCopyThreadPool =
BlockingThreadPoolExecutorService.newInstance(
copyThreadPoolSize / 2, copyThreadPoolSize, threadKeepAliveTime,
TimeUnit.SECONDS, "cos-copy-thread-pool");
{code}
4. CosNFileSystem, use PATH_DELIMITER to replace all "/", following is not
necessary, can be removed to keep code concise.
{code:java}
if (LOG.isDebugEnabled())
{code}
5. ByteBufferWrapper#close，use nested try/catch to guarantee
[jira] [Comment Edited] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[
https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829186#comment-16829186
]
Sammi Chen edited comment on HADOOP-15616 at 4/29/19 12:11 PM:
---
Comments for 009.patch,
1. CosNConfigKeys, default pool size is too big. CosN will CPU resource with
other modules in Hadoop and with other services on in the big data system. Be
Conservative when allocating resources. Suggest 4 threads for upload pool and
8 threads for copy pool.
{code:java}
public static final int DEFAULT_UPLOAD_THREAD_POOL_SIZE =
Runtime.getRuntime().availableProcessors() * 5;
public static final int DEFAULT_COPY_THREAD_POOL_SIZE =
Runtime.getRuntime().availableProcessors() * 3;
{code}
add comments for the unit of
{code:java}
public static final long DEFAULT_THREAD_KEEP_ALIVE_TIME = 60L;
{code}
2. function format, prefer a compact format, keep as many as parameters in a
line. There are many such places in the patch to be fixed, not just the
following one.
{code:java}
public CosNFileReadTask(
Configuration conf,
String key, NativeFileSystemStore store,
CosNInputStream.ReadBuffer readBuffer) {
{code}
preferred
{code:java}
public CosNFileReadTask(Configuration conf, String key,
NativeFileSystemStore store, CosNInputStream.ReadBuffer readBuffer) {
{code}
3. CosNFileSystem#initialize, suggest use two variable to control pool size and
waiting queue size, pool size should be relative smaller number, while waiting
queue can be bigger number, such as 256, etc.
{code:java}
this.boundedIOThreadPool = BlockingThreadPoolExecutorService.newInstance(
ioThreadPoolSize / 2, ioThreadPoolSize,
threadKeepAlive, TimeUnit.SECONDS,
"cos-transfer-thread-pool");
this.boundedCopyThreadPool =
BlockingThreadPoolExecutorService.newInstance(
copyThreadPoolSize / 2, copyThreadPoolSize, threadKeepAliveTime,
TimeUnit.SECONDS, "cos-copy-thread-pool");
{code}
4. CosNFileSystem, use PATH_DELIMITER to replace all "/", following is not
necessary, can be removed to keep code concise.
{code:java}
if (LOG.isDebugEnabled())
{code}
5. ByteBufferWrapper#close，use nested try/catch to guarantee every resource get
the chance to be released
{code:java}
void close() throws IOException {
if (null != this.byteBuffer) {
this.byteBuffer.clear();
}
if (null != randomAccessFile) {
this.randomAccessFile.close();
}
if (this.byteBuffer instanceof MappedByteBuffer) {
munmap((MappedByteBuffer) this.byteBuffer);
}
if (null != this.file && this.file.exists()) {
this.file.delete();
}
}
{code}
6. CosNOutputStream#uploadPart#, byte buffer is not released if
(store).uploadPart throw any exception
7. Enhanced the unit tests. You can leverage the code coverage report function
in intellij.
was (Author: sammi):
Comments for 009.patch,
1. CosNConfigKeys, default pool size is too big. CosN will CPU resource with
other modules in Hadoop and with other services on in the big data system. Be
Conservative when allocating resources. Suggest 4 threads for upload pool and
8 threads for copy pool.
{code:java}
public static final int DEFAULT_UPLOAD_THREAD_POOL_SIZE =
Runtime.getRuntime().availableProcessors() * 5;
public static final int DEFAULT_COPY_THREAD_POOL_SIZE =
Runtime.getRuntime().availableProcessors() * 3;
{code}
add comments for the unit of
{code:java}
public static final long DEFAULT_THREAD_KEEP_ALIVE_TIME = 60L;
{code}
2. function format, prefer a compact format, keep as many as parameters in a
line. There are many such places in the patch to be fixed, not just the
following one.
{code:java}
public CosNFileReadTask(
Configuration conf,
String key, NativeFileSystemStore store,
CosNInputStream.ReadBuffer readBuffer) {
{code}
preferred
{code:java}
public CosNFileReadTask(Configuration conf, String key,
NativeFileSystemStore store, CosNInputStream.ReadBuffer readBuffer) {
{code}
3. CosNFileSystem#initialize, suggest use two variable to control pool size and
waiting queue size, pool size should be relative smaller number, while waiting
queue can be bigger number, such as 256, etc.
{code:java}
this.boundedIOThreadPool = BlockingThreadPoolExecutorService.newInstance(
ioThreadPoolSize / 2, ioThreadPoolSize,
threadKeepAlive, TimeUnit.SECONDS,
"cos-transfer-thread-pool");
this.boundedCopyThreadPool =
BlockingThreadPoolExecutorService.newInstance(
copyThreadPoolSize / 2, copyThreadPoolSize, threadKeepAliveTime,
TimeUnit.SECONDS, "cos-copy-thread-pool");
{code}
4. CosNFileSystem, use PATH_DELIMITER to replace all "/", following is not
necessary, can be removed to keep code concise.
{code:java}
if (LOG.isDebugEnabled())
{code}
5. ByteBufferWrapper#close，use
[jira] [Comment Edited] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[ https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824396#comment-16824396 ] YangY edited comment on HADOOP-15616 at 4/23/19 6:13 PM: - [~ste...@apache.org] Thank you very much for your reply. I do understand the effort to support and maintain an object store in the Apache Hadoop Community, and will try my best to achieve it. I will improve the test code as soon as possible based on your comments. Thanks again for your continued attention to this work. was (Author: yuyang733): [~ste...@apache.org] Thank you very much for your reply. I do understand the effort to support and maintain an object store in the Apache Hadoop Community, and will try my best to achieve it. I will improve the test code as soon as possible based on your comments. Thanks again for your continued attention to this patch. > Incorporate Tencent Cloud COS File System Implementation > > > Key: HADOOP-15616 > URL: https://issues.apache.org/jira/browse/HADOOP-15616 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/cos >Reporter: Junping Du >Assignee: YangY >Priority: Major > Attachments: HADOOP-15616.001.patch, HADOOP-15616.002.patch, > HADOOP-15616.003.patch, HADOOP-15616.004.patch, HADOOP-15616.005.patch, > HADOOP-15616.006.patch, HADOOP-15616.007.patch, HADOOP-15616.008.patch, > HADOOP-15616.009.patch, Tencent-COS-Integrated-v2.pdf, > Tencent-COS-Integrated.pdf > > > Tencent cloud is top 2 cloud vendors in China market and the object store COS > （[https://intl.cloud.tencent.com/product/cos]） is widely used among China’s > cloud users but now it is hard for hadoop user to access data laid on COS > storage as no native support for COS in Hadoop. > This work aims to integrate Tencent cloud COS with Hadoop/Spark/Hive, just > like what we do before for S3, ADL, OSS, etc. With simple configuration, > Hadoop applications can read/write data from COS without any code change. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[ https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824396#comment-16824396 ] YangY edited comment on HADOOP-15616 at 4/23/19 6:13 PM: - [~ste...@apache.org] Thank you very much for your reply. I do understand the effort to support and maintain an object store in the Apache Hadoop Community, and will try my best to achieve it. I will improve the test code as soon as possible based on your comments. Thanks again for your continued attention to this patch. was (Author: yuyang733): [~ste...@apache.org] Thank you very much for your reply. I do understand the effort to support and maintain an object store in the Apache Hadoop Community, and will try my best to achieve it. I will improve the test code as soon as possible based on your comments. > Incorporate Tencent Cloud COS File System Implementation > > > Key: HADOOP-15616 > URL: https://issues.apache.org/jira/browse/HADOOP-15616 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/cos >Reporter: Junping Du >Assignee: YangY >Priority: Major > Attachments: HADOOP-15616.001.patch, HADOOP-15616.002.patch, > HADOOP-15616.003.patch, HADOOP-15616.004.patch, HADOOP-15616.005.patch, > HADOOP-15616.006.patch, HADOOP-15616.007.patch, HADOOP-15616.008.patch, > HADOOP-15616.009.patch, Tencent-COS-Integrated-v2.pdf, > Tencent-COS-Integrated.pdf > > > Tencent cloud is top 2 cloud vendors in China market and the object store COS > （[https://intl.cloud.tencent.com/product/cos]） is widely used among China’s > cloud users but now it is hard for hadoop user to access data laid on COS > storage as no native support for COS in Hadoop. > This work aims to integrate Tencent cloud COS with Hadoop/Spark/Hive, just > like what we do before for S3, ADL, OSS, etc. With simple configuration, > Hadoop applications can read/write data from COS without any code change. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[ https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16823857#comment-16823857 ] YangY edited comment on HADOOP-15616 at 4/23/19 8:59 AM: - Considering that more and more Hadoop Users choose COS as an underlying storage system, we do hope to merge it into the official community of Apache Hadoop as soon as possible. Thanks everyone for taking the time to pay attention to this patch again. was (Author: yuyang733): Considering that more and more Hadoop Users choose COS as an underlying storage system. So, we do hope to merge it into the official community of Apache Hadoop as soon as possible. Thanks everyone for taking the time to pay attention to this patch again. > Incorporate Tencent Cloud COS File System Implementation > > > Key: HADOOP-15616 > URL: https://issues.apache.org/jira/browse/HADOOP-15616 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/cos >Reporter: Junping Du >Assignee: YangY >Priority: Major > Attachments: HADOOP-15616.001.patch, HADOOP-15616.002.patch, > HADOOP-15616.003.patch, HADOOP-15616.004.patch, HADOOP-15616.005.patch, > HADOOP-15616.006.patch, HADOOP-15616.007.patch, HADOOP-15616.008.patch, > HADOOP-15616.009.patch, Tencent-COS-Integrated-v2.pdf, > Tencent-COS-Integrated.pdf > > > Tencent cloud is top 2 cloud vendors in China market and the object store COS > （[https://intl.cloud.tencent.com/product/cos]） is widely used among China’s > cloud users but now it is hard for hadoop user to access data laid on COS > storage as no native support for COS in Hadoop. > This work aims to integrate Tencent cloud COS with Hadoop/Spark/Hive, just > like what we do before for S3, ADL, OSS, etc. With simple configuration, > Hadoop applications can read/write data from COS without any code change. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[ https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16823103#comment-16823103 ] YangY edited comment on HADOOP-15616 at 4/22/19 1:30 PM: - [~Sammi] Thanks for your patch review. According to your suggestions, I have replaced all keyword `cos` to `cosn`, and fixed the related description in our document. A new patch and a new document have been uploaded. Please take a look at it. Thanks again for your time. was (Author: yuyang733): [~Sammi] Thanks for your patch review. According to your suggestions, I have replaced all keyword `cos` to `cosn`, and fixed the related description in our document. A new patch and a new document has been uploaded. Please take a look at it. Thanks again for your time. > Incorporate Tencent Cloud COS File System Implementation > > > Key: HADOOP-15616 > URL: https://issues.apache.org/jira/browse/HADOOP-15616 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/cos >Reporter: Junping Du >Assignee: YangY >Priority: Major > Attachments: HADOOP-15616.001.patch, HADOOP-15616.002.patch, > HADOOP-15616.003.patch, HADOOP-15616.004.patch, HADOOP-15616.005.patch, > HADOOP-15616.006.patch, HADOOP-15616.007.patch, HADOOP-15616.008.patch, > HADOOP-15616.009.patch, Tencent-COS-Integrated-v2.pdf, > Tencent-COS-Integrated.pdf > > > Tencent cloud is top 2 cloud vendors in China market and the object store COS > （[https://intl.cloud.tencent.com/product/cos]） is widely used among China’s > cloud users but now it is hard for hadoop user to access data laid on COS > storage as no native support for COS in Hadoop. > This work aims to integrate Tencent cloud COS with Hadoop/Spark/Hive, just > like what we do before for S3, ADL, OSS, etc. With simple configuration, > Hadoop applications can read/write data from COS without any code change. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[ https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16823103#comment-16823103 ] YangY edited comment on HADOOP-15616 at 4/22/19 1:29 PM: - [~Sammi] Thanks for your patch review. According to your suggestions, I have replaced all keyword `cos` to `cosn`, and fixed the related description in our document. A new patch and a new document has been uploaded. Please take a look at it. Thanks again for your time. was (Author: yuyang733): [~Sammi] Thank you for your patch review. According to your suggestions, I have replaced all keyword `cos` to `cosn`, and fixed the related description in our document. A new patch and a new document has been uploaded. Please take a look at it. Thanks again for your time. > Incorporate Tencent Cloud COS File System Implementation > > > Key: HADOOP-15616 > URL: https://issues.apache.org/jira/browse/HADOOP-15616 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/cos >Reporter: Junping Du >Assignee: YangY >Priority: Major > Attachments: HADOOP-15616.001.patch, HADOOP-15616.002.patch, > HADOOP-15616.003.patch, HADOOP-15616.004.patch, HADOOP-15616.005.patch, > HADOOP-15616.006.patch, HADOOP-15616.007.patch, HADOOP-15616.008.patch, > HADOOP-15616.009.patch, Tencent-COS-Integrated-v2.pdf, > Tencent-COS-Integrated.pdf > > > Tencent cloud is top 2 cloud vendors in China market and the object store COS > （[https://intl.cloud.tencent.com/product/cos]） is widely used among China’s > cloud users but now it is hard for hadoop user to access data laid on COS > storage as no native support for COS in Hadoop. > This work aims to integrate Tencent cloud COS with Hadoop/Spark/Hive, just > like what we do before for S3, ADL, OSS, etc. With simple configuration, > Hadoop applications can read/write data from COS without any code change. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[ https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718572#comment-16718572 ] YangY edited comment on HADOOP-15616 at 12/12/18 8:01 AM: -- Thanks [~xyao] for comment on this code. Here are the answers to your comments: 1. Changes under hadoop-tools/hadoop-aliyun unrelated to this patch. This may be a misoperation when formatting my code, and the error has been corrected in the new patch. 2. Should we put hadoop-cos under hadoop-tools project like s3a, adsl, etc. instead of hadoop-cloud-storage-project? At first, I also thought I should put it under the hadoop-tools project. However, as steve's comment above, using "hadoop-cloud-storage-project" seems more appropriate，isn't it? 3. More description to keys. Thank you for your reminder, I will add some detailed descriptions in our document. 4. BufferPool.java: since it sets the disk buffer file delete on exist, does it support recovery if client restart? BufferPool is a shared buffer pool. It initially provides two buffer types: Memory and Disk. The latter uses the memory file mapping to construct a byte buffer that can be used by other classes uniformly. Therefore, it can not support recovery if client restart. After all, the disk buffer is mapped a temporal file, and it will be cleaned up automatically when the Java Virtual Machine exists. In the latest patch, I further optimize it by combining two buffer types and gain two improvements: memory usage and buffer performance. For this reason, the type of buffers here will not be visible to the user. Finally, I look forward to your more comments. was (Author: yuyang733): Thanks [~xyao] for comment on this code. Here are the answers to your comments: 1. Changes under hadoop-tools/hadoop-aliyun unrelated to this patch. This may be a misoperation when formatting my code, and the error has been corrected in the new patch. 2. Should we put hadoop-cos under hadoop-tools project like s3a, adsl, etc. instead of hadoop-cloud-storage-project? At first, I also thought I should put it under the hadoop-tools project. However, as steve's comment above, using "hadoop-cloud-storage-project" seems more appropriate，isn't it? 3. More description to keys. Thank you for your reminder, I will add some detailed descriptions in our document. 4. BufferPool.java: since it sets the disk buffer file delete on exist, does it support recovery if client restart? BufferPool is a shared buffer pool. It initially provides two buffer types: Memory and Disk. The latter uses the memory file mapping to construct a byte buffer that can be used by other classes uniformly. Therefore, it can not support recovery if client restart. After all, the disk buffer is mapped a temporal file, and it will be cleaned up automatically when the Java Virtual Machine exists. In the latest patch, I further optimize it by combining two buffer types: memory usage and buffer performance. For this reason, the type of buffers here will not be visible to the user. Finally, I look forward to your more comments. > Incorporate Tencent Cloud COS File System Implementation > > > Key: HADOOP-15616 > URL: https://issues.apache.org/jira/browse/HADOOP-15616 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/cos >Reporter: Junping Du >Assignee: YangY >Priority: Major > Attachments: HADOOP-15616.001.patch, HADOOP-15616.002.patch, > HADOOP-15616.003.patch, HADOOP-15616.004.patch, HADOOP-15616.005.patch, > Tencent-COS-Integrated.pdf > > > Tencent cloud is top 2 cloud vendors in China market and the object store COS > （[https://intl.cloud.tencent.com/product/cos]） is widely used among China’s > cloud users but now it is hard for hadoop user to access data laid on COS > storage as no native support for COS in Hadoop. > This work aims to integrate Tencent cloud COS with Hadoop/Spark/Hive, just > like what we do before for S3, ADL, OSS, etc. With simple configuration, > Hadoop applications can read/write data from COS without any code change. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[ https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718572#comment-16718572 ] YangY edited comment on HADOOP-15616 at 12/12/18 7:58 AM: -- Thanks [~xyao] for comment on this code. Here are the answers to your comments: 1. Changes under hadoop-tools/hadoop-aliyun unrelated to this patch. This may be a misoperation when formatting my code, and the error has been corrected in the new patch. 2. Should we put hadoop-cos under hadoop-tools project like s3a, adsl, etc. instead of hadoop-cloud-storage-project? At first, I also thought I should put it under the hadoop-tools project. However, as steve's comment above, using "hadoop-cloud-storage-project" seems more appropriate，isn't it? 3. More description to keys. Thank you for your reminder, I will add some detailed descriptions in our document. 4. BufferPool.java: since it sets the disk buffer file delete on exist, does it support recovery if client restart? BufferPool is a shared buffer pool. It initially provides two buffer types: Memory and Disk. The latter uses the memory file mapping to construct a byte buffer that can be used by other classes uniformly. Therefore, it can not support recovery if client restart. After all, the disk buffer is mapped a temporal file, and it will be cleaned up automatically when the Java Virtual Machine exists. In the latest patch, I further optimize it by combining two buffer types: memory usage and buffer performance. For this reason, the type of buffers here will not be visible to the user. Finally, I look forward to your more comments. was (Author: yuyang733): Thanks [~xyao] for comment on this code. Here are the answers to your comments: 1. Changes under hadoop-tools/hadoop-aliyun unrelated to this patch. This may be a misoperation when formatting my code, and the error has been corrected in the new patch. 2. Should we put hadoop-cos under hadoop-tools project like s3a, adsl, etc. instead of hadoop-cloud-storage-project? At first, I also thought I should put it under the hadoop-tools project. However, as steve's comment above, using "hadoop-cloud-storage-project" seems more appropriate，isn't it? 3. More description to SecretKey -> SecretID. Thank you for your reminder, I will add a detailed description for getting SecretKey and SecretID in our document. 4. BufferPool.java: since it sets the disk buffer file delete on exist, does it support recovery if client restart? BufferPool is a shared buffer pool. It initially provides two buffer types: Memory and Disk. The latter uses the memory file mapping to construct a byte buffer that can be used by other classes uniformly. Therefore, it can not support recovery if client restart. After all, the disk buffer is mapped a temporal file, and it will be cleaned up automatically when the Java Virtual Machine exists. In the latest patch, I further optimize it by combining two buffer types: memory usage and buffer performance. For this reason, the type of buffers here will not be visible to the user. Finally, I look forward to your more comments. > Incorporate Tencent Cloud COS File System Implementation > > > Key: HADOOP-15616 > URL: https://issues.apache.org/jira/browse/HADOOP-15616 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/cos >Reporter: Junping Du >Assignee: YangY >Priority: Major > Attachments: HADOOP-15616.001.patch, HADOOP-15616.002.patch, > HADOOP-15616.003.patch, HADOOP-15616.004.patch, HADOOP-15616.005.patch, > Tencent-COS-Integrated.pdf > > > Tencent cloud is top 2 cloud vendors in China market and the object store COS > （[https://intl.cloud.tencent.com/product/cos]） is widely used among China’s > cloud users but now it is hard for hadoop user to access data laid on COS > storage as no native support for COS in Hadoop. > This work aims to integrate Tencent cloud COS with Hadoop/Spark/Hive, just > like what we do before for S3, ADL, OSS, etc. With simple configuration, > Hadoop applications can read/write data from COS without any code change. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[ https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718572#comment-16718572 ] YangY edited comment on HADOOP-15616 at 12/12/18 7:57 AM: -- Thanks [~xyao] for comment on this code. Here are the answers to your comments: 1. Changes under hadoop-tools/hadoop-aliyun unrelated to this patch. This may be a misoperation when formatting my code, and the error has been corrected in the new patch. 2. Should we put hadoop-cos under hadoop-tools project like s3a, adsl, etc. instead of hadoop-cloud-storage-project? At first, I also thought I should put it under the hadoop-tools project. However, as steve's comment above, using "hadoop-cloud-storage-project" seems more appropriate，isn't it? 3. More description to SecretKey -> SecretID. Thank you for your reminder, I will add a detailed description for getting SecretKey and SecretID in our document. 4. BufferPool.java: since it sets the disk buffer file delete on exist, does it support recovery if client restart? BufferPool is a shared buffer pool. It initially provides two buffer types: Memory and Disk. The latter uses the memory file mapping to construct a byte buffer that can be used by other classes uniformly. Therefore, it can not support recovery if client restart. After all, the disk buffer is mapped a temporal file, and it will be cleaned up automatically when the Java Virtual Machine exists. In the latest patch, I further optimize it by combining two buffer types: memory usage and buffer performance. For this reason, the type of buffers here will not be visible to the user. Finally, I look forward to your more comments. was (Author: yuyang733): Thanks [~xyao] for comment on this code. Here are the answers to your comments: 1. Changes under hadoop-tools/hadoop-aliyun unrelated to this patch. This may be a misoperation when formatting my code, and the error has been corrected in the new path. 2. Should we put hadoop-cos under hadoop-tools project like s3a, adsl, etc. instead of hadoop-cloud-storage-project? At first, I also thought I should put it under the hadoop-tools project. However, as steve's comment above, using "hadoop-cloud-storage-project" seems more appropriate，isn't it? 3. More description to SecretKey -> SecretID. Thank you for your reminder, I will add a detailed description for getting SecretKey and SecretID in our document. 4. BufferPool.java: since it sets the disk buffer file delete on exist, does it support recovery if client restart? BufferPool is a shared buffer pool. It initially provides two buffer types: Memory and Disk. The latter uses the memory file mapping to construct a byte buffer that can be used by other classes uniformly. Therefore, it can not support recovery if client restart. After all, the disk buffer is mapped a temporal file, and it will be cleaned up automatically when the Java Virtual Machine exists. In the latest patch, I further optimize it by combining two buffer types: memory usage and buffer performance. For this reason, the type of buffers here will not be visible to the user. Finally, I look forward to your more comments. > Incorporate Tencent Cloud COS File System Implementation > > > Key: HADOOP-15616 > URL: https://issues.apache.org/jira/browse/HADOOP-15616 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/cos >Reporter: Junping Du >Assignee: YangY >Priority: Major > Attachments: HADOOP-15616.001.patch, HADOOP-15616.002.patch, > HADOOP-15616.003.patch, HADOOP-15616.004.patch, HADOOP-15616.005.patch, > Tencent-COS-Integrated.pdf > > > Tencent cloud is top 2 cloud vendors in China market and the object store COS > （[https://intl.cloud.tencent.com/product/cos]） is widely used among China’s > cloud users but now it is hard for hadoop user to access data laid on COS > storage as no native support for COS in Hadoop. > This work aims to integrate Tencent cloud COS with Hadoop/Spark/Hive, just > like what we do before for S3, ADL, OSS, etc. With simple configuration, > Hadoop applications can read/write data from COS without any code change. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[ https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718572#comment-16718572 ] YangY edited comment on HADOOP-15616 at 12/12/18 7:57 AM: -- Thanks [~xyao] for comment on this code. Here are the answers to your comments: 1. Changes under hadoop-tools/hadoop-aliyun unrelated to this patch. This may be a misoperation when formatting my code, and the error has been corrected in the new path. 2. Should we put hadoop-cos under hadoop-tools project like s3a, adsl, etc. instead of hadoop-cloud-storage-project? At first, I also thought I should put it under the hadoop-tools project. However, as steve's comment above, using "hadoop-cloud-storage-project" seems more appropriate，isn't it? 3. More description to SecretKey -> SecretID. Thank you for your reminder, I will add a detailed description for getting SecretKey and SecretID in our document. 4. BufferPool.java: since it sets the disk buffer file delete on exist, does it support recovery if client restart? BufferPool is a shared buffer pool. It initially provides two buffer types: Memory and Disk. The latter uses the memory file mapping to construct a byte buffer that can be used by other classes uniformly. Therefore, it can not support recovery if client restart. After all, the disk buffer is mapped a temporal file, and it will be cleaned up automatically when the Java Virtual Machine exists. In the latest patch, I further optimize it by combining two buffer types: memory usage and buffer performance. For this reason, the type of buffers here will not be visible to the user. Finally, I look forward to your more comments. was (Author: yuyang733): Thanks [~xyao] for comment on this code. Here are the answers to your comments: 1. Changes under hadoop-tools/hadoop-aliyun unrelated to this patch. This may be a misoperation when formatting my code, and the error has been corrected in the new path. 2. Should we put hadoop-cos under hadoop-tools project like s3a, adsl, etc. instead of hadoop-cloud-storage-project? At first, I also thought I should put it under the hadoop-tools project. However, as steve's comment above, using "hadoop-cloud-storage-project" seems more appropriate，isn't it? 3. More description to SecretKey -> SecretID. Thank you for your reminder, I will add a detailed description for getting SecretKey and SecretID in our document. 4. BufferPool.java: since it sets the disk buffer file delete on exist, does it support recovery if client restart? BufferPool is a shared buffer pool. It initially provides two buffer types: Memory and Disk. The latter uses the memory file mapping to construct a byte buffer that can be used by other classes uniformly. Therefore, it can not support recovery if client restart. After all, the disk buffer is mapped a temporal file, and it will be cleaned up automatically when the Java Virtual Machine exists. Finally, I look forward to your more comments. > Incorporate Tencent Cloud COS File System Implementation > > > Key: HADOOP-15616 > URL: https://issues.apache.org/jira/browse/HADOOP-15616 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/cos >Reporter: Junping Du >Assignee: YangY >Priority: Major > Attachments: HADOOP-15616.001.patch, HADOOP-15616.002.patch, > HADOOP-15616.003.patch, HADOOP-15616.004.patch, HADOOP-15616.005.patch, > Tencent-COS-Integrated.pdf > > > Tencent cloud is top 2 cloud vendors in China market and the object store COS > （[https://intl.cloud.tencent.com/product/cos]） is widely used among China’s > cloud users but now it is hard for hadoop user to access data laid on COS > storage as no native support for COS in Hadoop. > This work aims to integrate Tencent cloud COS with Hadoop/Spark/Hive, just > like what we do before for S3, ADL, OSS, etc. With simple configuration, > Hadoop applications can read/write data from COS without any code change. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[ https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548780#comment-16548780 ] Junping Du edited comment on HADOOP-15616 at 7/19/18 4:14 AM: -- Thank you, Steve! Yes. My colleague and myself will help to maintain this. The initial plan is we will share design and code in next few days for community review. The code has been running in production environment for a while but we are very open for comments and review from community. For test part, if necessary, we can setup some test account for community reviewer. :) bq. as its a new module, can you put it under hadoop-cloud-storage-project I noticed most previous work like s3, adl, oss, etc. are all put under hadoop-tools. Do we have plan to move them all to under hadoop-cloud-storage-project? bq. Anything you can point us to about the store in terms of: consistency, rename performance, atomicity, etc, as in "how close does this come to a filesystem" That's very key to hadoop behavior over a cloud storage. We will discuss this in our design doc and we can start discussion from that. bq. not got any time to review this, going to have to find others to do anything...i'll try to keep an eye on the process though. Understand. Thanks for heads up for our effort here. was (Author: djp): Thank you, Steve! Yes. My colleague and myself will help to maintain this. The initial plan is we will share design and code in next few days for community review. The code has been running in production environment for a while but we are very open for comments and review from community. For test part, if necessary, we can setup some test account for community reviewer. :) bq. as its a new module, can you put it under hadoop-cloud-storage-project I noticed most previous work like s3, adl, oss, etc. are all put under hadoop-tools. Do we have plan to move them all to under hadoop-cloud-storage-project? bq. Anything you can point us to about the store in terms of: consistency, rename performance, atomicity, etc, as in "how close does this come to a filesystem" That's very key to hadoop behavior over a cloud storage. We will discuss this in our design doc and we can start discussion from that. bq. not got any time to review this, going to have to find others to do anything...i'll try to keep an eye on the process though. Understand. Thanks for heads up for our effort here. > Incorporate Tencent Cloud COS File System Implementation > > > Key: HADOOP-15616 > URL: https://issues.apache.org/jira/browse/HADOOP-15616 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/cos >Reporter: Junping Du >Priority: Major > > Tencent cloud is top 2 cloud vendors in China market and the object store COS > is widely used among China’s cloud users but now it is hard for hadoop user > to access data laid on COS storage as no native support for COS in Hadoop. > This work aims to integrate Tencent cloud COS with Hadoop, just like what we > do before for S3, ADL, OSS, etc. With simple configuration, Hadoop > applications can read/write data from COS without any code change. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
