[jira] [Updated] (HADOOP-18851) Performance improvement for DelegationTokenSecretManager
[
https://issues.apache.org/jira/browse/HADOOP-18851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sammi Chen updated HADOOP-18851:
Fix Version/s: 3.5.0
> Performance improvement for DelegationTokenSecretManager
>
>
> Key: HADOOP-18851
> URL: https://issues.apache.org/jira/browse/HADOOP-18851
> Project: Hadoop Common
> Issue Type: Task
> Components: common
>Affects Versions: 3.4.0
>Reporter: Vikas Kumar
>Assignee: Vikas Kumar
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.5.0
>
> Attachments:
> 0001-HADOOP-18851-Perfm-improvement-for-ZKDT-management.patch, Screenshot
> 2023-08-16 at 5.36.57 PM.png
>
>
> *Context:*
> KMS depends on hadoop-common for DT management. Recently we were analysing
> one performance issue and following is out findings:
> # Around 96% (196 out of 200) KMS container threads were in BLOCKED state at
> following:
> ## *AbstractDelegationTokenSecretManager.verifyToken()*
> ## *AbstractDelegationTokenSecretManager.createPassword()*
> # And then process crashed.
>
> {code:java}
> http-nio-9292-exec-200PRIORITY : 5THREAD ID : 0X7F075C157800NATIVE ID :
> 0X2C87FNATIVE ID (DECIMAL) : 182399STATE : BLOCKED
> stackTrace:
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.verifyToken(AbstractDelegationTokenSecretManager.java:474)
> - waiting to lock <0x0005f2f545e8> (a
> org.apache.hadoop.security.token.delegation.web.DelegationTokenManager$ZKSecretManager)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenManager.verifyToken(DelegationTokenManager.java:213)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:396)
> at {code}
> All the 199 out of 200 were blocked at above point.
> And the lock they are waiting for is acquired by a thread that was trying to
> createPassword and publishing the same on ZK.
>
> {code:java}
> stackTrace:
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1598)
> - locked <0x000749263ec0> (a org.apache.zookeeper.ClientCnxn$Packet)
> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1570)
> at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:2235)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl$7.call(SetDataBuilderImpl.java:398)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl$7.call(SetDataBuilderImpl.java:385)
> at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:93)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl.pathInForeground(SetDataBuilderImpl.java:382)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:358)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:36)
> at
> org.apache.curator.framework.recipes.shared.SharedValue.trySetValue(SharedValue.java:201)
> at
> org.apache.curator.framework.recipes.shared.SharedCount.trySetCount(SharedCount.java:116)
> at
> org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.incrSharedCount(ZKDelegationTokenSecretManager.java:586)
> at
> org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.incrementDelegationTokenSeqNum(ZKDelegationTokenSecretManager.java:601)
> at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.createPassword(AbstractDelegationTokenSecretManager.java:402)
> - locked <0x0005f2f545e8> (a
> org.apache.hadoop.security.token.delegation.web.DelegationTokenManager$ZKSecretManager)
> at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.createPassword(AbstractDelegationTokenSecretManager.java:48)
> at org.apache.hadoop.security.token.Token.(Token.java:67)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenManager.createToken(DelegationTokenManager.java:183)
> {code}
> We can say that this thread is slow and has blocked remaining all. But
> following is my observation:
>
> # verifyToken() and createPaswword() has been synchronized because one is
> reading the tokenMap and another is updating the map. If it's only to protect
> the map, then can't we simply use ConcurrentHashMap and remove the
> "synchronized" keyword. Because due to this, all reader threads ( to
> verifyToken()) are also blocking each other.
> # IN HA env, It is recommended to use ZK to store DTs. We know that
> CuratorFramework is thread safe.
>
[jira] [Updated] (HADOOP-18851) Performance improvement for DelegationTokenSecretManager
[
https://issues.apache.org/jira/browse/HADOOP-18851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sammi Chen updated HADOOP-18851:
Target Version/s: 3.4.0, 3.5.0 (was: 3.4.0)
> Performance improvement for DelegationTokenSecretManager
>
>
> Key: HADOOP-18851
> URL: https://issues.apache.org/jira/browse/HADOOP-18851
> Project: Hadoop Common
> Issue Type: Task
> Components: common
>Affects Versions: 3.4.0
>Reporter: Vikas Kumar
>Assignee: Vikas Kumar
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments:
> 0001-HADOOP-18851-Perfm-improvement-for-ZKDT-management.patch, Screenshot
> 2023-08-16 at 5.36.57 PM.png
>
>
> *Context:*
> KMS depends on hadoop-common for DT management. Recently we were analysing
> one performance issue and following is out findings:
> # Around 96% (196 out of 200) KMS container threads were in BLOCKED state at
> following:
> ## *AbstractDelegationTokenSecretManager.verifyToken()*
> ## *AbstractDelegationTokenSecretManager.createPassword()*
> # And then process crashed.
>
> {code:java}
> http-nio-9292-exec-200PRIORITY : 5THREAD ID : 0X7F075C157800NATIVE ID :
> 0X2C87FNATIVE ID (DECIMAL) : 182399STATE : BLOCKED
> stackTrace:
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.verifyToken(AbstractDelegationTokenSecretManager.java:474)
> - waiting to lock <0x0005f2f545e8> (a
> org.apache.hadoop.security.token.delegation.web.DelegationTokenManager$ZKSecretManager)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenManager.verifyToken(DelegationTokenManager.java:213)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:396)
> at {code}
> All the 199 out of 200 were blocked at above point.
> And the lock they are waiting for is acquired by a thread that was trying to
> createPassword and publishing the same on ZK.
>
> {code:java}
> stackTrace:
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1598)
> - locked <0x000749263ec0> (a org.apache.zookeeper.ClientCnxn$Packet)
> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1570)
> at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:2235)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl$7.call(SetDataBuilderImpl.java:398)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl$7.call(SetDataBuilderImpl.java:385)
> at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:93)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl.pathInForeground(SetDataBuilderImpl.java:382)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:358)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:36)
> at
> org.apache.curator.framework.recipes.shared.SharedValue.trySetValue(SharedValue.java:201)
> at
> org.apache.curator.framework.recipes.shared.SharedCount.trySetCount(SharedCount.java:116)
> at
> org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.incrSharedCount(ZKDelegationTokenSecretManager.java:586)
> at
> org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.incrementDelegationTokenSeqNum(ZKDelegationTokenSecretManager.java:601)
> at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.createPassword(AbstractDelegationTokenSecretManager.java:402)
> - locked <0x0005f2f545e8> (a
> org.apache.hadoop.security.token.delegation.web.DelegationTokenManager$ZKSecretManager)
> at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.createPassword(AbstractDelegationTokenSecretManager.java:48)
> at org.apache.hadoop.security.token.Token.(Token.java:67)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenManager.createToken(DelegationTokenManager.java:183)
> {code}
> We can say that this thread is slow and has blocked remaining all. But
> following is my observation:
>
> # verifyToken() and createPaswword() has been synchronized because one is
> reading the tokenMap and another is updating the map. If it's only to protect
> the map, then can't we simply use ConcurrentHashMap and remove the
> "synchronized" keyword. Because due to this, all reader threads ( to
> verifyToken()) are also blocking each other.
> # IN HA env, It is recommended to use ZK to store DTs. We know that
> CuratorFramework is thread safe.
>
[jira] [Resolved] (HADOOP-18851) Performance improvement for DelegationTokenSecretManager
[
https://issues.apache.org/jira/browse/HADOOP-18851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sammi Chen resolved HADOOP-18851.
-
Resolution: Fixed
> Performance improvement for DelegationTokenSecretManager
>
>
> Key: HADOOP-18851
> URL: https://issues.apache.org/jira/browse/HADOOP-18851
> Project: Hadoop Common
> Issue Type: Task
> Components: common
>Affects Versions: 3.4.0
>Reporter: Vikas Kumar
>Assignee: Vikas Kumar
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0, 3.4.0
>
> Attachments:
> 0001-HADOOP-18851-Perfm-improvement-for-ZKDT-management.patch, Screenshot
> 2023-08-16 at 5.36.57 PM.png
>
>
> *Context:*
> KMS depends on hadoop-common for DT management. Recently we were analysing
> one performance issue and following is out findings:
> # Around 96% (196 out of 200) KMS container threads were in BLOCKED state at
> following:
> ## *AbstractDelegationTokenSecretManager.verifyToken()*
> ## *AbstractDelegationTokenSecretManager.createPassword()*
> # And then process crashed.
>
> {code:java}
> http-nio-9292-exec-200PRIORITY : 5THREAD ID : 0X7F075C157800NATIVE ID :
> 0X2C87FNATIVE ID (DECIMAL) : 182399STATE : BLOCKED
> stackTrace:
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.verifyToken(AbstractDelegationTokenSecretManager.java:474)
> - waiting to lock <0x0005f2f545e8> (a
> org.apache.hadoop.security.token.delegation.web.DelegationTokenManager$ZKSecretManager)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenManager.verifyToken(DelegationTokenManager.java:213)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:396)
> at {code}
> All the 199 out of 200 were blocked at above point.
> And the lock they are waiting for is acquired by a thread that was trying to
> createPassword and publishing the same on ZK.
>
> {code:java}
> stackTrace:
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1598)
> - locked <0x000749263ec0> (a org.apache.zookeeper.ClientCnxn$Packet)
> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1570)
> at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:2235)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl$7.call(SetDataBuilderImpl.java:398)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl$7.call(SetDataBuilderImpl.java:385)
> at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:93)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl.pathInForeground(SetDataBuilderImpl.java:382)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:358)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:36)
> at
> org.apache.curator.framework.recipes.shared.SharedValue.trySetValue(SharedValue.java:201)
> at
> org.apache.curator.framework.recipes.shared.SharedCount.trySetCount(SharedCount.java:116)
> at
> org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.incrSharedCount(ZKDelegationTokenSecretManager.java:586)
> at
> org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.incrementDelegationTokenSeqNum(ZKDelegationTokenSecretManager.java:601)
> at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.createPassword(AbstractDelegationTokenSecretManager.java:402)
> - locked <0x0005f2f545e8> (a
> org.apache.hadoop.security.token.delegation.web.DelegationTokenManager$ZKSecretManager)
> at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.createPassword(AbstractDelegationTokenSecretManager.java:48)
> at org.apache.hadoop.security.token.Token.(Token.java:67)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenManager.createToken(DelegationTokenManager.java:183)
> {code}
> We can say that this thread is slow and has blocked remaining all. But
> following is my observation:
>
> # verifyToken() and createPaswword() has been synchronized because one is
> reading the tokenMap and another is updating the map. If it's only to protect
> the map, then can't we simply use ConcurrentHashMap and remove the
> "synchronized" keyword. Because due to this, all reader threads ( to
> verifyToken()) are also blocking each other.
> # IN HA env, It is recommended to use ZK to store DTs. We know that
> CuratorFramework is thread safe.
>
[jira] [Updated] (HADOOP-18851) Performance improvement for DelegationTokenSecretManager
[
https://issues.apache.org/jira/browse/HADOOP-18851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sammi Chen updated HADOOP-18851:
Summary: Performance improvement for DelegationTokenSecretManager (was:
Performance improvement for DelegationTokenSecretManager.)
> Performance improvement for DelegationTokenSecretManager
>
>
> Key: HADOOP-18851
> URL: https://issues.apache.org/jira/browse/HADOOP-18851
> Project: Hadoop Common
> Issue Type: Task
> Components: common
>Affects Versions: 3.4.0
>Reporter: Vikas Kumar
>Assignee: Vikas Kumar
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments:
> 0001-HADOOP-18851-Perfm-improvement-for-ZKDT-management.patch, Screenshot
> 2023-08-16 at 5.36.57 PM.png
>
>
> *Context:*
> KMS depends on hadoop-common for DT management. Recently we were analysing
> one performance issue and following is out findings:
> # Around 96% (196 out of 200) KMS container threads were in BLOCKED state at
> following:
> ## *AbstractDelegationTokenSecretManager.verifyToken()*
> ## *AbstractDelegationTokenSecretManager.createPassword()*
> # And then process crashed.
>
> {code:java}
> http-nio-9292-exec-200PRIORITY : 5THREAD ID : 0X7F075C157800NATIVE ID :
> 0X2C87FNATIVE ID (DECIMAL) : 182399STATE : BLOCKED
> stackTrace:
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.verifyToken(AbstractDelegationTokenSecretManager.java:474)
> - waiting to lock <0x0005f2f545e8> (a
> org.apache.hadoop.security.token.delegation.web.DelegationTokenManager$ZKSecretManager)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenManager.verifyToken(DelegationTokenManager.java:213)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:396)
> at {code}
> All the 199 out of 200 were blocked at above point.
> And the lock they are waiting for is acquired by a thread that was trying to
> createPassword and publishing the same on ZK.
>
> {code:java}
> stackTrace:
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1598)
> - locked <0x000749263ec0> (a org.apache.zookeeper.ClientCnxn$Packet)
> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1570)
> at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:2235)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl$7.call(SetDataBuilderImpl.java:398)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl$7.call(SetDataBuilderImpl.java:385)
> at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:93)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl.pathInForeground(SetDataBuilderImpl.java:382)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:358)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:36)
> at
> org.apache.curator.framework.recipes.shared.SharedValue.trySetValue(SharedValue.java:201)
> at
> org.apache.curator.framework.recipes.shared.SharedCount.trySetCount(SharedCount.java:116)
> at
> org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.incrSharedCount(ZKDelegationTokenSecretManager.java:586)
> at
> org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.incrementDelegationTokenSeqNum(ZKDelegationTokenSecretManager.java:601)
> at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.createPassword(AbstractDelegationTokenSecretManager.java:402)
> - locked <0x0005f2f545e8> (a
> org.apache.hadoop.security.token.delegation.web.DelegationTokenManager$ZKSecretManager)
> at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.createPassword(AbstractDelegationTokenSecretManager.java:48)
> at org.apache.hadoop.security.token.Token.(Token.java:67)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenManager.createToken(DelegationTokenManager.java:183)
> {code}
> We can say that this thread is slow and has blocked remaining all. But
> following is my observation:
>
> # verifyToken() and createPaswword() has been synchronized because one is
> reading the tokenMap and another is updating the map. If it's only to protect
> the map, then can't we simply use ConcurrentHashMap and remove the
> "synchronized" keyword. Because due to this, all reader threads ( to
> verifyToken()) are also blocking each other.
> # IN HA env, It is recommended to use
[jira] [Reopened] (HADOOP-18851) Performance improvement for DelegationTokenSecretManager.
[
https://issues.apache.org/jira/browse/HADOOP-18851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sammi Chen reopened HADOOP-18851:
-
Revert the previous commit which removes the synchronized keywords. Will have a
new implementation using ReentrantReadWriteLock.
> Performance improvement for DelegationTokenSecretManager.
> -
>
> Key: HADOOP-18851
> URL: https://issues.apache.org/jira/browse/HADOOP-18851
> Project: Hadoop Common
> Issue Type: Task
> Components: common
>Affects Versions: 3.4.0
>Reporter: Vikas Kumar
>Assignee: Vikas Kumar
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments:
> 0001-HADOOP-18851-Perfm-improvement-for-ZKDT-management.patch, Screenshot
> 2023-08-16 at 5.36.57 PM.png
>
>
> *Context:*
> KMS depends on hadoop-common for DT management. Recently we were analysing
> one performance issue and following is out findings:
> # Around 96% (196 out of 200) KMS container threads were in BLOCKED state at
> following:
> ## *AbstractDelegationTokenSecretManager.verifyToken()*
> ## *AbstractDelegationTokenSecretManager.createPassword()*
> # And then process crashed.
>
> {code:java}
> http-nio-9292-exec-200PRIORITY : 5THREAD ID : 0X7F075C157800NATIVE ID :
> 0X2C87FNATIVE ID (DECIMAL) : 182399STATE : BLOCKED
> stackTrace:
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.verifyToken(AbstractDelegationTokenSecretManager.java:474)
> - waiting to lock <0x0005f2f545e8> (a
> org.apache.hadoop.security.token.delegation.web.DelegationTokenManager$ZKSecretManager)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenManager.verifyToken(DelegationTokenManager.java:213)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:396)
> at {code}
> All the 199 out of 200 were blocked at above point.
> And the lock they are waiting for is acquired by a thread that was trying to
> createPassword and publishing the same on ZK.
>
> {code:java}
> stackTrace:
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1598)
> - locked <0x000749263ec0> (a org.apache.zookeeper.ClientCnxn$Packet)
> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1570)
> at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:2235)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl$7.call(SetDataBuilderImpl.java:398)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl$7.call(SetDataBuilderImpl.java:385)
> at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:93)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl.pathInForeground(SetDataBuilderImpl.java:382)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:358)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:36)
> at
> org.apache.curator.framework.recipes.shared.SharedValue.trySetValue(SharedValue.java:201)
> at
> org.apache.curator.framework.recipes.shared.SharedCount.trySetCount(SharedCount.java:116)
> at
> org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.incrSharedCount(ZKDelegationTokenSecretManager.java:586)
> at
> org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.incrementDelegationTokenSeqNum(ZKDelegationTokenSecretManager.java:601)
> at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.createPassword(AbstractDelegationTokenSecretManager.java:402)
> - locked <0x0005f2f545e8> (a
> org.apache.hadoop.security.token.delegation.web.DelegationTokenManager$ZKSecretManager)
> at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.createPassword(AbstractDelegationTokenSecretManager.java:48)
> at org.apache.hadoop.security.token.Token.(Token.java:67)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenManager.createToken(DelegationTokenManager.java:183)
> {code}
> We can say that this thread is slow and has blocked remaining all. But
> following is my observation:
>
> # verifyToken() and createPaswword() has been synchronized because one is
> reading the tokenMap and another is updating the map. If it's only to protect
> the map, then can't we simply use ConcurrentHashMap and remove the
> "synchronized" keyword. Because due to this, all reader threads ( to
> verifyToken()) are also blocking each other.
> # IN HA env, It is recommended to use
[jira] [Comment Edited] (HADOOP-18851) Performance improvement for DelegationTokenSecretManager.
[
https://issues.apache.org/jira/browse/HADOOP-18851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844149#comment-17844149
]
Sammi Chen edited comment on HADOOP-18851 at 5/7/24 6:12 AM:
-
Reverted the previous commit which removes the synchronized keywords. Will have
a new implementation using ReentrantReadWriteLock.
was (Author: sammi):
Revert the previous commit which removes the synchronized keywords. Will have a
new implementation using ReentrantReadWriteLock.
> Performance improvement for DelegationTokenSecretManager.
> -
>
> Key: HADOOP-18851
> URL: https://issues.apache.org/jira/browse/HADOOP-18851
> Project: Hadoop Common
> Issue Type: Task
> Components: common
>Affects Versions: 3.4.0
>Reporter: Vikas Kumar
>Assignee: Vikas Kumar
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments:
> 0001-HADOOP-18851-Perfm-improvement-for-ZKDT-management.patch, Screenshot
> 2023-08-16 at 5.36.57 PM.png
>
>
> *Context:*
> KMS depends on hadoop-common for DT management. Recently we were analysing
> one performance issue and following is out findings:
> # Around 96% (196 out of 200) KMS container threads were in BLOCKED state at
> following:
> ## *AbstractDelegationTokenSecretManager.verifyToken()*
> ## *AbstractDelegationTokenSecretManager.createPassword()*
> # And then process crashed.
>
> {code:java}
> http-nio-9292-exec-200PRIORITY : 5THREAD ID : 0X7F075C157800NATIVE ID :
> 0X2C87FNATIVE ID (DECIMAL) : 182399STATE : BLOCKED
> stackTrace:
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.verifyToken(AbstractDelegationTokenSecretManager.java:474)
> - waiting to lock <0x0005f2f545e8> (a
> org.apache.hadoop.security.token.delegation.web.DelegationTokenManager$ZKSecretManager)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenManager.verifyToken(DelegationTokenManager.java:213)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:396)
> at {code}
> All the 199 out of 200 were blocked at above point.
> And the lock they are waiting for is acquired by a thread that was trying to
> createPassword and publishing the same on ZK.
>
> {code:java}
> stackTrace:
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1598)
> - locked <0x000749263ec0> (a org.apache.zookeeper.ClientCnxn$Packet)
> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1570)
> at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:2235)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl$7.call(SetDataBuilderImpl.java:398)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl$7.call(SetDataBuilderImpl.java:385)
> at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:93)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl.pathInForeground(SetDataBuilderImpl.java:382)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:358)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:36)
> at
> org.apache.curator.framework.recipes.shared.SharedValue.trySetValue(SharedValue.java:201)
> at
> org.apache.curator.framework.recipes.shared.SharedCount.trySetCount(SharedCount.java:116)
> at
> org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.incrSharedCount(ZKDelegationTokenSecretManager.java:586)
> at
> org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.incrementDelegationTokenSeqNum(ZKDelegationTokenSecretManager.java:601)
> at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.createPassword(AbstractDelegationTokenSecretManager.java:402)
> - locked <0x0005f2f545e8> (a
> org.apache.hadoop.security.token.delegation.web.DelegationTokenManager$ZKSecretManager)
> at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.createPassword(AbstractDelegationTokenSecretManager.java:48)
> at org.apache.hadoop.security.token.Token.(Token.java:67)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenManager.createToken(DelegationTokenManager.java:183)
> {code}
> We can say that this thread is slow and has blocked remaining all. But
> following is my observation:
>
> # verifyToken() and createPaswword() has been synchronized because one is
> reading the tokenMap and another is updating the map. If it's only to
[jira] [Resolved] (HADOOP-18458) AliyunOSS: AliyunOSSBlockOutputStream to support heap/off-heap buffer before uploading data to OSS
[ https://issues.apache.org/jira/browse/HADOOP-18458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen resolved HADOOP-18458. - Resolution: Fixed > AliyunOSS: AliyunOSSBlockOutputStream to support heap/off-heap buffer before > uploading data to OSS > -- > > Key: HADOOP-18458 > URL: https://issues.apache.org/jira/browse/HADOOP-18458 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Affects Versions: 3.0.3, 3.1.4, 2.10.2, 3.2.4, 3.3.4 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Recently, our customers raise a requirement: AliyunOSSBlockOutputStream > should support heap/off-heap buffer before uploading data to OSS. > Currently, AliyunOSSBlockOutputStream buffers data in local directory before > uploading to OSS, it is not efficient compared to memory. > Changes: > # Adds heap/off-heap buffers > # Adds limitation of memory used, and fallback to disk -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18458) AliyunOSS: AliyunOSSBlockOutputStream to support heap/off-heap buffer before uploading data to OSS
[ https://issues.apache.org/jira/browse/HADOOP-18458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HADOOP-18458: Fix Version/s: 3.4.0 > AliyunOSS: AliyunOSSBlockOutputStream to support heap/off-heap buffer before > uploading data to OSS > -- > > Key: HADOOP-18458 > URL: https://issues.apache.org/jira/browse/HADOOP-18458 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Affects Versions: 3.0.3, 3.1.4, 2.10.2, 3.2.4, 3.3.4 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Recently, our customers raise a requirement: AliyunOSSBlockOutputStream > should support heap/off-heap buffer before uploading data to OSS. > Currently, AliyunOSSBlockOutputStream buffers data in local directory before > uploading to OSS, it is not efficient compared to memory. > Changes: > # Adds heap/off-heap buffers > # Adds limitation of memory used, and fallback to disk -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16959) Resolve hadoop-cos dependency conflict
[ https://issues.apache.org/jira/browse/HADOOP-16959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HADOOP-16959: Fix Version/s: 3.3.1 > Resolve hadoop-cos dependency conflict > -- > > Key: HADOOP-16959 > URL: https://issues.apache.org/jira/browse/HADOOP-16959 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/cos >Reporter: YangY >Assignee: YangY >Priority: Major > Fix For: 3.3.1, 3.4.0 > > Attachments: HADOOP-16959-branch-3.3.001.patch, > HADOOP-16959-branch-3.3.002.patch, HADOOP-16959-branch-3.3.003.patch, > HADOOP-16959-branch-3.3.004.patch, HADOOP-16959-branch-3.3.005.patch > > > There are some dependency conflicts between the Hadoop-common and Hadoop-cos. > For example, joda time lib, HTTP client lib and etc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16959) Resolve hadoop-cos dependency conflict
[ https://issues.apache.org/jira/browse/HADOOP-16959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HADOOP-16959: Resolution: Fixed Status: Resolved (was: Patch Available) > Resolve hadoop-cos dependency conflict > -- > > Key: HADOOP-16959 > URL: https://issues.apache.org/jira/browse/HADOOP-16959 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/cos >Reporter: YangY >Assignee: YangY >Priority: Major > Fix For: 3.3.1, 3.4.0 > > Attachments: HADOOP-16959-branch-3.3.001.patch, > HADOOP-16959-branch-3.3.002.patch, HADOOP-16959-branch-3.3.003.patch, > HADOOP-16959-branch-3.3.004.patch, HADOOP-16959-branch-3.3.005.patch > > > There are some dependency conflicts between the Hadoop-common and Hadoop-cos. > For example, joda time lib, HTTP client lib and etc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16959) Resolve hadoop-cos dependency conflict
[ https://issues.apache.org/jira/browse/HADOOP-16959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HADOOP-16959: Fix Version/s: 3.4.0 > Resolve hadoop-cos dependency conflict > -- > > Key: HADOOP-16959 > URL: https://issues.apache.org/jira/browse/HADOOP-16959 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/cos >Reporter: YangY >Assignee: YangY >Priority: Major > Fix For: 3.4.0 > > Attachments: HADOOP-16959-branch-3.3.001.patch, > HADOOP-16959-branch-3.3.002.patch, HADOOP-16959-branch-3.3.003.patch, > HADOOP-16959-branch-3.3.004.patch, HADOOP-16959-branch-3.3.005.patch > > > There are some dependency conflicts between the Hadoop-common and Hadoop-cos. > For example, joda time lib, HTTP client lib and etc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16959) Resolve hadoop-cos dependency conflict
[ https://issues.apache.org/jira/browse/HADOOP-16959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17087571#comment-17087571 ] Sammi Chen commented on HADOOP-16959: - Thanks [~yuyang733] for continuously improving the patch based on our offline discussion. The last patch LGTM. +1. Will commit to trunk soon. > Resolve hadoop-cos dependency conflict > -- > > Key: HADOOP-16959 > URL: https://issues.apache.org/jira/browse/HADOOP-16959 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/cos >Reporter: YangY >Assignee: YangY >Priority: Major > Attachments: HADOOP-16959-branch-3.3.001.patch, > HADOOP-16959-branch-3.3.002.patch, HADOOP-16959-branch-3.3.003.patch, > HADOOP-16959-branch-3.3.004.patch, HADOOP-16959-branch-3.3.005.patch > > > There are some dependency conflicts between the Hadoop-common and Hadoop-cos. > For example, joda time lib, HTTP client lib and etc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16955) Umbrella Jira for improving the Hadoop-cos support in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17085734#comment-17085734 ] Sammi Chen commented on HADOOP-16955: - [~weichiu], sure, will try best to work with [~yuyang733] to make sure the feature going mature. > Umbrella Jira for improving the Hadoop-cos support in Hadoop > > > Key: HADOOP-16955 > URL: https://issues.apache.org/jira/browse/HADOOP-16955 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/cos >Reporter: YangY >Assignee: YangY >Priority: Major > Attachments: HADOOP-16955-branch-3.3.001.patch > > Original Estimate: 48h > Time Spent: 4h > Remaining Estimate: 44h > > This Umbrella Jira focus on fixing some known bugs and adding some important > features. > > bugfix: > # resolve the dependency conflict; > # fix the upload buffer returning failed when some exceptions occur; > # fix the issue that the single file upload can not be retried; > # fix the bug of checking if a file exists through listing the file > frequently. > features: > # support SessionCredentialsProvider and InstanceCredentialsProvider, which > allows users to specify the credentials in URI or get it from the CVM > (Tencent Cloud Virtual Machine) bound to the CAM role that can access the COS > bucket; > # support the server encryption based on SSE-COS and SSE-C; > # support the HTTP proxy settings; > # support the storage class settings; > # support the CRC64 checksum. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[ https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16947327#comment-16947327 ] Sammi Chen commented on HADOOP-15616: - It's on trunk. Fix Version updated. [~weichiu], thanks for the reminder. > Incorporate Tencent Cloud COS File System Implementation > > > Key: HADOOP-15616 > URL: https://issues.apache.org/jira/browse/HADOOP-15616 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/cos >Reporter: Junping Du >Assignee: YangY >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15616.001.patch, HADOOP-15616.002.patch, > HADOOP-15616.003.patch, HADOOP-15616.004.patch, HADOOP-15616.005.patch, > HADOOP-15616.006.patch, HADOOP-15616.007.patch, HADOOP-15616.008.patch, > HADOOP-15616.009.patch, HADOOP-15616.010.patch, HADOOP-15616.011.patch, > Tencent-COS-Integrated-v2.pdf, Tencent-COS-Integrated.pdf > > > Tencent cloud is top 2 cloud vendors in China market and the object store COS > （[https://intl.cloud.tencent.com/product/cos]） is widely used among China’s > cloud users but now it is hard for hadoop user to access data laid on COS > storage as no native support for COS in Hadoop. > This work aims to integrate Tencent cloud COS with Hadoop/Spark/Hive, just > like what we do before for S3, ADL, OSS, etc. With simple configuration, > Hadoop applications can read/write data from COS without any code change. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[ https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HADOOP-15616: Fix Version/s: 3.3.0 > Incorporate Tencent Cloud COS File System Implementation > > > Key: HADOOP-15616 > URL: https://issues.apache.org/jira/browse/HADOOP-15616 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/cos >Reporter: Junping Du >Assignee: YangY >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15616.001.patch, HADOOP-15616.002.patch, > HADOOP-15616.003.patch, HADOOP-15616.004.patch, HADOOP-15616.005.patch, > HADOOP-15616.006.patch, HADOOP-15616.007.patch, HADOOP-15616.008.patch, > HADOOP-15616.009.patch, HADOOP-15616.010.patch, HADOOP-15616.011.patch, > Tencent-COS-Integrated-v2.pdf, Tencent-COS-Integrated.pdf > > > Tencent cloud is top 2 cloud vendors in China market and the object store COS > （[https://intl.cloud.tencent.com/product/cos]） is widely used among China’s > cloud users but now it is hard for hadoop user to access data laid on COS > storage as no native support for COS in Hadoop. > This work aims to integrate Tencent cloud COS with Hadoop/Spark/Hive, just > like what we do before for S3, ADL, OSS, etc. With simple configuration, > Hadoop applications can read/write data from COS without any code change. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[ https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HADOOP-15616: Release Note: Tencent cloud is top 2 cloud vendors in China market and the object store COS is widely used among China’s cloud users. This task implements a COSN filesytem to support Tencent cloud COS natively in Hadoop. With simple configuration, Hadoop applications, like Spark and Hive, can read/write data from COS without any code change. Resolution: Fixed Status: Resolved (was: Patch Available) > Incorporate Tencent Cloud COS File System Implementation > > > Key: HADOOP-15616 > URL: https://issues.apache.org/jira/browse/HADOOP-15616 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/cos >Reporter: Junping Du >Assignee: YangY >Priority: Major > Attachments: HADOOP-15616.001.patch, HADOOP-15616.002.patch, > HADOOP-15616.003.patch, HADOOP-15616.004.patch, HADOOP-15616.005.patch, > HADOOP-15616.006.patch, HADOOP-15616.007.patch, HADOOP-15616.008.patch, > HADOOP-15616.009.patch, HADOOP-15616.010.patch, HADOOP-15616.011.patch, > Tencent-COS-Integrated-v2.pdf, Tencent-COS-Integrated.pdf > > > Tencent cloud is top 2 cloud vendors in China market and the object store COS > （[https://intl.cloud.tencent.com/product/cos]） is widely used among China’s > cloud users but now it is hard for hadoop user to access data laid on COS > storage as no native support for COS in Hadoop. > This work aims to integrate Tencent cloud COS with Hadoop/Spark/Hive, just > like what we do before for S3, ADL, OSS, etc. With simple configuration, > Hadoop applications can read/write data from COS without any code change. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[ https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939232#comment-16939232 ] Sammi Chen edited comment on HADOOP-15616 at 9/27/19 9:07 AM: -- Last patch LGTM, +1. Will commit soon. Thanks [~yuyang733] for the contribution. Thanks [~djp] [~ste...@apache.org] [~xyao] [~wujinhu] for review the code. Tencent COS is a very popular cloud object storage in China Market. It's good to have it in Hadoop. was (Author: sammi): Last patch LGTM, +1. Will commit soon. Thanks [~yuyang733] for the contribution. Thanks [~djp] [~ste...@apache.org] [~wujinhu] for review the code. > Incorporate Tencent Cloud COS File System Implementation > > > Key: HADOOP-15616 > URL: https://issues.apache.org/jira/browse/HADOOP-15616 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/cos >Reporter: Junping Du >Assignee: YangY >Priority: Major > Attachments: HADOOP-15616.001.patch, HADOOP-15616.002.patch, > HADOOP-15616.003.patch, HADOOP-15616.004.patch, HADOOP-15616.005.patch, > HADOOP-15616.006.patch, HADOOP-15616.007.patch, HADOOP-15616.008.patch, > HADOOP-15616.009.patch, HADOOP-15616.010.patch, HADOOP-15616.011.patch, > Tencent-COS-Integrated-v2.pdf, Tencent-COS-Integrated.pdf > > > Tencent cloud is top 2 cloud vendors in China market and the object store COS > （[https://intl.cloud.tencent.com/product/cos]） is widely used among China’s > cloud users but now it is hard for hadoop user to access data laid on COS > storage as no native support for COS in Hadoop. > This work aims to integrate Tencent cloud COS with Hadoop/Spark/Hive, just > like what we do before for S3, ADL, OSS, etc. With simple configuration, > Hadoop applications can read/write data from COS without any code change. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[ https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939232#comment-16939232 ] Sammi Chen commented on HADOOP-15616: - Last patch LGTM, +1. Will commit soon. Thanks [~yuyang733] for the contribution. Thanks [~djp] [~ste...@apache.org] [~wujinhu] for review the code. > Incorporate Tencent Cloud COS File System Implementation > > > Key: HADOOP-15616 > URL: https://issues.apache.org/jira/browse/HADOOP-15616 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/cos >Reporter: Junping Du >Assignee: YangY >Priority: Major > Attachments: HADOOP-15616.001.patch, HADOOP-15616.002.patch, > HADOOP-15616.003.patch, HADOOP-15616.004.patch, HADOOP-15616.005.patch, > HADOOP-15616.006.patch, HADOOP-15616.007.patch, HADOOP-15616.008.patch, > HADOOP-15616.009.patch, HADOOP-15616.010.patch, HADOOP-15616.011.patch, > Tencent-COS-Integrated-v2.pdf, Tencent-COS-Integrated.pdf > > > Tencent cloud is top 2 cloud vendors in China market and the object store COS > （[https://intl.cloud.tencent.com/product/cos]） is widely used among China’s > cloud users but now it is hard for hadoop user to access data laid on COS > storage as no native support for COS in Hadoop. > This work aims to integrate Tencent cloud COS with Hadoop/Spark/Hive, just > like what we do before for S3, ADL, OSS, etc. With simple configuration, > Hadoop applications can read/write data from COS without any code change. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[ https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846689#comment-16846689 ] Sammi Chen commented on HADOOP-15616: - [~ste...@apache.org] Hi Steve, do you happen to got the chance to try the latest patch? Let me or [~yuyang733] know if you got any issue. > Incorporate Tencent Cloud COS File System Implementation > > > Key: HADOOP-15616 > URL: https://issues.apache.org/jira/browse/HADOOP-15616 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/cos >Reporter: Junping Du >Assignee: YangY >Priority: Major > Attachments: HADOOP-15616.001.patch, HADOOP-15616.002.patch, > HADOOP-15616.003.patch, HADOOP-15616.004.patch, HADOOP-15616.005.patch, > HADOOP-15616.006.patch, HADOOP-15616.007.patch, HADOOP-15616.008.patch, > HADOOP-15616.009.patch, HADOOP-15616.010.patch, > Tencent-COS-Integrated-v2.pdf, Tencent-COS-Integrated.pdf > > > Tencent cloud is top 2 cloud vendors in China market and the object store COS > （[https://intl.cloud.tencent.com/product/cos]） is widely used among China’s > cloud users but now it is hard for hadoop user to access data laid on COS > storage as no native support for COS in Hadoop. > This work aims to integrate Tencent cloud COS with Hadoop/Spark/Hive, just > like what we do before for S3, ADL, OSS, etc. With simple configuration, > Hadoop applications can read/write data from COS without any code change. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[ https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HADOOP-15616: Attachment: (was: Junping Du.url) > Incorporate Tencent Cloud COS File System Implementation > > > Key: HADOOP-15616 > URL: https://issues.apache.org/jira/browse/HADOOP-15616 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/cos >Reporter: Junping Du >Assignee: YangY >Priority: Major > Attachments: HADOOP-15616.001.patch, HADOOP-15616.002.patch, > HADOOP-15616.003.patch, HADOOP-15616.004.patch, HADOOP-15616.005.patch, > HADOOP-15616.006.patch, HADOOP-15616.007.patch, HADOOP-15616.008.patch, > HADOOP-15616.009.patch, HADOOP-15616.010.patch, > Tencent-COS-Integrated-v2.pdf, Tencent-COS-Integrated.pdf > > > Tencent cloud is top 2 cloud vendors in China market and the object store COS > （[https://intl.cloud.tencent.com/product/cos]） is widely used among China’s > cloud users but now it is hard for hadoop user to access data laid on COS > storage as no native support for COS in Hadoop. > This work aims to integrate Tencent cloud COS with Hadoop/Spark/Hive, just > like what we do before for S3, ADL, OSS, etc. With simple configuration, > Hadoop applications can read/write data from COS without any code change. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[
https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829186#comment-16829186
]
Sammi Chen edited comment on HADOOP-15616 at 4/30/19 6:25 AM:
--
Comments for 009.patch,
1. CosNConfigKeys, default pool size is too big. CosN will share CPU resource
with other modules in Hadoop and with other services on in the big data system.
Be Conservative when allocating resources. Suggest 4 threads for upload pool
and 8 threads for copy pool.
{code:java}
public static final int DEFAULT_UPLOAD_THREAD_POOL_SIZE =
Runtime.getRuntime().availableProcessors() * 5;
public static final int DEFAULT_COPY_THREAD_POOL_SIZE =
Runtime.getRuntime().availableProcessors() * 3;
{code}
add comments for the unit of
{code:java}
public static final long DEFAULT_THREAD_KEEP_ALIVE_TIME = 60L;
{code}
2. function format, prefer a compact format, keep as many as parameters in a
line. There are many such places in the patch to be fixed, not just the
following one.
{code:java}
public CosNFileReadTask(
Configuration conf,
String key, NativeFileSystemStore store,
CosNInputStream.ReadBuffer readBuffer) {
{code}
preferred
{code:java}
public CosNFileReadTask(Configuration conf, String key,
NativeFileSystemStore store, CosNInputStream.ReadBuffer readBuffer) {
{code}
3. CosNFileSystem#initialize, suggest use two variable to control pool size and
waiting queue size, pool size should be relative smaller number, while waiting
queue can be bigger number, such as 256, etc.
{code:java}
this.boundedIOThreadPool = BlockingThreadPoolExecutorService.newInstance(
ioThreadPoolSize / 2, ioThreadPoolSize,
threadKeepAlive, TimeUnit.SECONDS,
"cos-transfer-thread-pool");
this.boundedCopyThreadPool =
BlockingThreadPoolExecutorService.newInstance(
copyThreadPoolSize / 2, copyThreadPoolSize, threadKeepAliveTime,
TimeUnit.SECONDS, "cos-copy-thread-pool");
{code}
4. CosNFileSystem, use PATH_DELIMITER to replace all "/", following is not
necessary, can be removed to keep code concise.
{code:java}
if (LOG.isDebugEnabled())
{code}
5. ByteBufferWrapper#close，use nested try/catch to guarantee every resource get
the chance to be released
{code:java}
void close() throws IOException {
if (null != this.byteBuffer) {
this.byteBuffer.clear();
}
if (null != randomAccessFile) {
this.randomAccessFile.close();
}
if (this.byteBuffer instanceof MappedByteBuffer) {
munmap((MappedByteBuffer) this.byteBuffer);
}
if (null != this.file && this.file.exists()) {
this.file.delete();
}
}
{code}
6. CosNOutputStream#uploadPart#, byte buffer is not released if
(store).uploadPart throw any exception
7. Enhanced the unit tests. You can leverage the code coverage report function
in intellij.
was (Author: sammi):
Comments for 009.patch,
1. CosNConfigKeys, default pool size is too big. CosN will CPU resource with
other modules in Hadoop and with other services on in the big data system. Be
Conservative when allocating resources. Suggest 4 threads for upload pool and
8 threads for copy pool.
{code:java}
public static final int DEFAULT_UPLOAD_THREAD_POOL_SIZE =
Runtime.getRuntime().availableProcessors() * 5;
public static final int DEFAULT_COPY_THREAD_POOL_SIZE =
Runtime.getRuntime().availableProcessors() * 3;
{code}
add comments for the unit of
{code:java}
public static final long DEFAULT_THREAD_KEEP_ALIVE_TIME = 60L;
{code}
2. function format, prefer a compact format, keep as many as parameters in a
line. There are many such places in the patch to be fixed, not just the
following one.
{code:java}
public CosNFileReadTask(
Configuration conf,
String key, NativeFileSystemStore store,
CosNInputStream.ReadBuffer readBuffer) {
{code}
preferred
{code:java}
public CosNFileReadTask(Configuration conf, String key,
NativeFileSystemStore store, CosNInputStream.ReadBuffer readBuffer) {
{code}
3. CosNFileSystem#initialize, suggest use two variable to control pool size and
waiting queue size, pool size should be relative smaller number, while waiting
queue can be bigger number, such as 256, etc.
{code:java}
this.boundedIOThreadPool = BlockingThreadPoolExecutorService.newInstance(
ioThreadPoolSize / 2, ioThreadPoolSize,
threadKeepAlive, TimeUnit.SECONDS,
"cos-transfer-thread-pool");
this.boundedCopyThreadPool =
BlockingThreadPoolExecutorService.newInstance(
copyThreadPoolSize / 2, copyThreadPoolSize, threadKeepAliveTime,
TimeUnit.SECONDS, "cos-copy-thread-pool");
{code}
4. CosNFileSystem, use PATH_DELIMITER to replace all "/", following is not
necessary, can be removed to keep code concise.
{code:java}
if (LOG.isDebugEnabled())
{code}
5. ByteBufferWrapper#close，use nested try/catch to guarantee
[jira] [Commented] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[ https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829202#comment-16829202 ] Sammi Chen commented on HADOOP-15616: - Hi [~ste...@apache.org]，thanks for the comments. I know you are fully occupied and don't have much time for this. But you are the expert of object storage support in Hadoop, I would like to ask your favor for having an eye on this JIRA and give some high level comments or suggestions if possible. [~djp] will also have an eye on this patch too and I will cover the detail patch review part. > Incorporate Tencent Cloud COS File System Implementation > > > Key: HADOOP-15616 > URL: https://issues.apache.org/jira/browse/HADOOP-15616 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/cos >Reporter: Junping Du >Assignee: YangY >Priority: Major > Attachments: HADOOP-15616.001.patch, HADOOP-15616.002.patch, > HADOOP-15616.003.patch, HADOOP-15616.004.patch, HADOOP-15616.005.patch, > HADOOP-15616.006.patch, HADOOP-15616.007.patch, HADOOP-15616.008.patch, > HADOOP-15616.009.patch, Junping Du.url, Tencent-COS-Integrated-v2.pdf, > Tencent-COS-Integrated.pdf > > > Tencent cloud is top 2 cloud vendors in China market and the object store COS > （[https://intl.cloud.tencent.com/product/cos]） is widely used among China’s > cloud users but now it is hard for hadoop user to access data laid on COS > storage as no native support for COS in Hadoop. > This work aims to integrate Tencent cloud COS with Hadoop/Spark/Hive, just > like what we do before for S3, ADL, OSS, etc. With simple configuration, > Hadoop applications can read/write data from COS without any code change. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[ https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HADOOP-15616: Attachment: Junping Du.url > Incorporate Tencent Cloud COS File System Implementation > > > Key: HADOOP-15616 > URL: https://issues.apache.org/jira/browse/HADOOP-15616 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/cos >Reporter: Junping Du >Assignee: YangY >Priority: Major > Attachments: HADOOP-15616.001.patch, HADOOP-15616.002.patch, > HADOOP-15616.003.patch, HADOOP-15616.004.patch, HADOOP-15616.005.patch, > HADOOP-15616.006.patch, HADOOP-15616.007.patch, HADOOP-15616.008.patch, > HADOOP-15616.009.patch, Junping Du.url, Tencent-COS-Integrated-v2.pdf, > Tencent-COS-Integrated.pdf > > > Tencent cloud is top 2 cloud vendors in China market and the object store COS > （[https://intl.cloud.tencent.com/product/cos]） is widely used among China’s > cloud users but now it is hard for hadoop user to access data laid on COS > storage as no native support for COS in Hadoop. > This work aims to integrate Tencent cloud COS with Hadoop/Spark/Hive, just > like what we do before for S3, ADL, OSS, etc. With simple configuration, > Hadoop applications can read/write data from COS without any code change. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[
https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829186#comment-16829186
]
Sammi Chen edited comment on HADOOP-15616 at 4/29/19 12:11 PM:
---
Comments for 009.patch,
1. CosNConfigKeys, default pool size is too big. CosN will CPU resource with
other modules in Hadoop and with other services on in the big data system. Be
Conservative when allocating resources. Suggest 4 threads for upload pool and
8 threads for copy pool.
{code:java}
public static final int DEFAULT_UPLOAD_THREAD_POOL_SIZE =
Runtime.getRuntime().availableProcessors() * 5;
public static final int DEFAULT_COPY_THREAD_POOL_SIZE =
Runtime.getRuntime().availableProcessors() * 3;
{code}
add comments for the unit of
{code:java}
public static final long DEFAULT_THREAD_KEEP_ALIVE_TIME = 60L;
{code}
2. function format, prefer a compact format, keep as many as parameters in a
line. There are many such places in the patch to be fixed, not just the
following one.
{code:java}
public CosNFileReadTask(
Configuration conf,
String key, NativeFileSystemStore store,
CosNInputStream.ReadBuffer readBuffer) {
{code}
preferred
{code:java}
public CosNFileReadTask(Configuration conf, String key,
NativeFileSystemStore store, CosNInputStream.ReadBuffer readBuffer) {
{code}
3. CosNFileSystem#initialize, suggest use two variable to control pool size and
waiting queue size, pool size should be relative smaller number, while waiting
queue can be bigger number, such as 256, etc.
{code:java}
this.boundedIOThreadPool = BlockingThreadPoolExecutorService.newInstance(
ioThreadPoolSize / 2, ioThreadPoolSize,
threadKeepAlive, TimeUnit.SECONDS,
"cos-transfer-thread-pool");
this.boundedCopyThreadPool =
BlockingThreadPoolExecutorService.newInstance(
copyThreadPoolSize / 2, copyThreadPoolSize, threadKeepAliveTime,
TimeUnit.SECONDS, "cos-copy-thread-pool");
{code}
4. CosNFileSystem, use PATH_DELIMITER to replace all "/", following is not
necessary, can be removed to keep code concise.
{code:java}
if (LOG.isDebugEnabled())
{code}
5. ByteBufferWrapper#close，use nested try/catch to guarantee every resource get
the chance to be released
{code:java}
void close() throws IOException {
if (null != this.byteBuffer) {
this.byteBuffer.clear();
}
if (null != randomAccessFile) {
this.randomAccessFile.close();
}
if (this.byteBuffer instanceof MappedByteBuffer) {
munmap((MappedByteBuffer) this.byteBuffer);
}
if (null != this.file && this.file.exists()) {
this.file.delete();
}
}
{code}
6. CosNOutputStream#uploadPart#, byte buffer is not released if
(store).uploadPart throw any exception
7. Enhanced the unit tests. You can leverage the code coverage report function
in intellij.
was (Author: sammi):
Comments for 009.patch,
1. CosNConfigKeys, default pool size is too big. CosN will CPU resource with
other modules in Hadoop and with other services on in the big data system. Be
Conservative when allocating resources. Suggest 4 threads for upload pool and
8 threads for copy pool.
{code:java}
public static final int DEFAULT_UPLOAD_THREAD_POOL_SIZE =
Runtime.getRuntime().availableProcessors() * 5;
public static final int DEFAULT_COPY_THREAD_POOL_SIZE =
Runtime.getRuntime().availableProcessors() * 3;
{code}
add comments for the unit of
{code:java}
public static final long DEFAULT_THREAD_KEEP_ALIVE_TIME = 60L;
{code}
2. function format, prefer a compact format, keep as many as parameters in a
line. There are many such places in the patch to be fixed, not just the
following one.
{code:java}
public CosNFileReadTask(
Configuration conf,
String key, NativeFileSystemStore store,
CosNInputStream.ReadBuffer readBuffer) {
{code}
preferred
{code:java}
public CosNFileReadTask(Configuration conf, String key,
NativeFileSystemStore store, CosNInputStream.ReadBuffer readBuffer) {
{code}
3. CosNFileSystem#initialize, suggest use two variable to control pool size and
waiting queue size, pool size should be relative smaller number, while waiting
queue can be bigger number, such as 256, etc.
{code:java}
this.boundedIOThreadPool = BlockingThreadPoolExecutorService.newInstance(
ioThreadPoolSize / 2, ioThreadPoolSize,
threadKeepAlive, TimeUnit.SECONDS,
"cos-transfer-thread-pool");
this.boundedCopyThreadPool =
BlockingThreadPoolExecutorService.newInstance(
copyThreadPoolSize / 2, copyThreadPoolSize, threadKeepAliveTime,
TimeUnit.SECONDS, "cos-copy-thread-pool");
{code}
4. CosNFileSystem, use PATH_DELIMITER to replace all "/", following is not
necessary, can be removed to keep code concise.
{code:java}
if (LOG.isDebugEnabled())
{code}
5. ByteBufferWrapper#close，use
[jira] [Commented] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[
https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829186#comment-16829186
]
Sammi Chen commented on HADOOP-15616:
-
Comments for 009.patch,
1. CosNConfigKeys, default pool size is too big. CosN will CPU resource with
other modules in Hadoop and with other services on in the big data system. Be
Conservative when allocating resources. Suggest 4 threads for upload pool and
8 threads for copy pool.
{code:java}
public static final int DEFAULT_UPLOAD_THREAD_POOL_SIZE =
Runtime.getRuntime().availableProcessors() * 5;
public static final int DEFAULT_COPY_THREAD_POOL_SIZE =
Runtime.getRuntime().availableProcessors() * 3;
{code}
add comments for the unit of
{code:java}
public static final long DEFAULT_THREAD_KEEP_ALIVE_TIME = 60L;
{code}
2. function format, prefer a compact format, keep as many as parameters in a
line. There are many such places in the patch to be fixed, not just the
following one.
{code:java}
public CosNFileReadTask(
Configuration conf,
String key, NativeFileSystemStore store,
CosNInputStream.ReadBuffer readBuffer) {
{code}
preferred
{code:java}
public CosNFileReadTask(Configuration conf, String key,
NativeFileSystemStore store, CosNInputStream.ReadBuffer readBuffer) {
{code}
3. CosNFileSystem#initialize, suggest use two variable to control pool size and
waiting queue size, pool size should be relative smaller number, while waiting
queue can be bigger number, such as 256, etc.
{code:java}
this.boundedIOThreadPool = BlockingThreadPoolExecutorService.newInstance(
ioThreadPoolSize / 2, ioThreadPoolSize,
threadKeepAlive, TimeUnit.SECONDS,
"cos-transfer-thread-pool");
this.boundedCopyThreadPool =
BlockingThreadPoolExecutorService.newInstance(
copyThreadPoolSize / 2, copyThreadPoolSize, threadKeepAliveTime,
TimeUnit.SECONDS, "cos-copy-thread-pool");
{code}
4. CosNFileSystem, use PATH_DELIMITER to replace all "/", following is not
necessary, can be removed to keep code concise.
{code:java}
if (LOG.isDebugEnabled())
{code}
5. ByteBufferWrapper#close，use nested try/catch to guarantee every resource get
the chance to be released
{code:java}
void close() throws IOException {
if (null != this.byteBuffer) {
this.byteBuffer.clear();
}
if (null != randomAccessFile) {
this.randomAccessFile.close();
}
if (this.byteBuffer instanceof MappedByteBuffer) {
munmap((MappedByteBuffer) this.byteBuffer);
}
if (null != this.file && this.file.exists()) {
this.file.delete();
}
}
{code}
6. CosNOutputStream#uploadPart#, byte buffer is not released if
(store).uploadPart throw any exception
> Incorporate Tencent Cloud COS File System Implementation
>
>
> Key: HADOOP-15616
> URL: https://issues.apache.org/jira/browse/HADOOP-15616
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/cos
>Reporter: Junping Du
>Assignee: YangY
>Priority: Major
> Attachments: HADOOP-15616.001.patch, HADOOP-15616.002.patch,
> HADOOP-15616.003.patch, HADOOP-15616.004.patch, HADOOP-15616.005.patch,
> HADOOP-15616.006.patch, HADOOP-15616.007.patch, HADOOP-15616.008.patch,
> HADOOP-15616.009.patch, Tencent-COS-Integrated-v2.pdf,
> Tencent-COS-Integrated.pdf
>
>
> Tencent cloud is top 2 cloud vendors in China market and the object store COS
> （[https://intl.cloud.tencent.com/product/cos]） is widely used among China’s
> cloud users but now it is hard for hadoop user to access data laid on COS
> storage as no native support for COS in Hadoop.
> This work aims to integrate Tencent cloud COS with Hadoop/Spark/Hive, just
> like what we do before for S3, ADL, OSS, etc. With simple configuration,
> Hadoop applications can read/write data from COS without any code change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[ https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821966#comment-16821966 ] Sammi Chen commented on HADOOP-15616: - [~yuyang733], there are still many places where "cos" are used, which should be replaced by "cosn", including the package name, would you please upload a new patch to unify them first? Also comments in Chinese is not preferred because many people don't know Chinese. > Incorporate Tencent Cloud COS File System Implementation > > > Key: HADOOP-15616 > URL: https://issues.apache.org/jira/browse/HADOOP-15616 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/cos >Reporter: Junping Du >Assignee: YangY >Priority: Major > Attachments: HADOOP-15616.001.patch, HADOOP-15616.002.patch, > HADOOP-15616.003.patch, HADOOP-15616.004.patch, HADOOP-15616.005.patch, > HADOOP-15616.006.patch, HADOOP-15616.007.patch, HADOOP-15616.008.patch, > Tencent-COS-Integrated.pdf > > > Tencent cloud is top 2 cloud vendors in China market and the object store COS > （[https://intl.cloud.tencent.com/product/cos]） is widely used among China’s > cloud users but now it is hard for hadoop user to access data laid on COS > storage as no native support for COS in Hadoop. > This work aims to integrate Tencent cloud COS with Hadoop/Spark/Hive, just > like what we do before for S3, ADL, OSS, etc. With simple configuration, > Hadoop applications can read/write data from COS without any code change. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[ https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744103#comment-16744103 ] Sammi Chen commented on HADOOP-15616: - [~yuyang733], thanks for working on this. I have gone through the 007 patch briefly. Here are some findings. # Is file/directory last access time tracked by cos? There are applicaitons which depend on these information # "cosn" is used as the schma of Tencent Cos. Any specific reason to add "n" to the schema? There are both "cos" and "cosn" used in config keys, a unified one is expected. # Site content is not provided. Site content is used to educate user what this feature is and how to use it. The site will looks like this https://hadoop.apache.org/docs/r3.1.1/hadoop-azure-datalake/index.html # suggest putting all final static constant values into one configuration file instead of spreading over several files. for example, move final static fields like "SCHEME", "COS_MAX_LISTING_LENGTH" to Constants.java # "cos-hadoop-plugin-v5.3" is v5.3 the cos_api API version. Is it able to get the version dynamically? # The size of default thread pool of upload, download and copy action seems too large, use a reasonable value. Cannot occupy all resources by one module. # The thread pool of upload, download and copy are shared between all files of one cos filesystem instance, one big file may starve other files, suggest using SemaphoredDelegatingExecutor to fairly share resources between files. It's just an improvement suggestion. We can do it as a follow-on. # Is it too long to wait Long.MAX_VALUE milliseconds for the close of the thread pool? this.boundedCopyThreadPool.awaitTermination(Long.MAX_VALUE, TimeUnit.MILLISECONDS); # import * is not a recommended coding style in Hadoop > Incorporate Tencent Cloud COS File System Implementation > > > Key: HADOOP-15616 > URL: https://issues.apache.org/jira/browse/HADOOP-15616 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/cos >Reporter: Junping Du >Assignee: YangY >Priority: Major > Attachments: HADOOP-15616.001.patch, HADOOP-15616.002.patch, > HADOOP-15616.003.patch, HADOOP-15616.004.patch, HADOOP-15616.005.patch, > HADOOP-15616.006.patch, HADOOP-15616.007.patch, Tencent-COS-Integrated.pdf > > > Tencent cloud is top 2 cloud vendors in China market and the object store COS > （[https://intl.cloud.tencent.com/product/cos]） is widely used among China’s > cloud users but now it is hard for hadoop user to access data laid on COS > storage as no native support for COS in Hadoop. > This work aims to integrate Tencent cloud COS with Hadoop/Spark/Hive, just > like what we do before for S3, ADL, OSS, etc. With simple configuration, > Hadoop applications can read/write data from COS without any code change. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15917) AliyunOSS: fix incorrect ReadOps and WriteOps in statistics
[ https://issues.apache.org/jira/browse/HADOOP-15917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HADOOP-15917: Affects Version/s: (was: 2.10.0) > AliyunOSS: fix incorrect ReadOps and WriteOps in statistics > --- > > Key: HADOOP-15917 > URL: https://issues.apache.org/jira/browse/HADOOP-15917 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 2.9.1, 3.2.0, 3.1.1, 3.0.3 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Fix For: 2.10.0, 3.0.4, 3.3.0, 3.1.2, 3.2.1, 2.9.3 > > Attachments: HADOOP-15917.001.patch, HADOOP-15917.002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15917) AliyunOSS: fix incorrect ReadOps and WriteOps in statistics
[ https://issues.apache.org/jira/browse/HADOOP-15917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HADOOP-15917: Fix Version/s: 2.9.3 3.2.1 3.1.2 3.3.0 3.0.4 2.10.0 > AliyunOSS: fix incorrect ReadOps and WriteOps in statistics > --- > > Key: HADOOP-15917 > URL: https://issues.apache.org/jira/browse/HADOOP-15917 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 2.10.0, 2.9.1, 3.2.0, 3.1.1, 3.0.3 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Fix For: 2.10.0, 3.0.4, 3.3.0, 3.1.2, 3.2.1, 2.9.3 > > Attachments: HADOOP-15917.001.patch, HADOOP-15917.002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15917) AliyunOSS: fix incorrect ReadOps and WriteOps in statistics
[
https://issues.apache.org/jira/browse/HADOOP-15917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684911#comment-16684911
]
Sammi Chen commented on HADOOP-15917:
-
[~wujinhu], some comments,
1. TestAliyunOSSBlockOutputStream. java.
would you add some comments about why read ops is "7" and write ops is "3"
here?
assertEquals({color:#ff}7{color}, statistics.getReadOps());
assertEquals({color:#ff}3{color}, statistics.getWriteOps());
2. The statistics counts the successful read and write operations. Do we care
about the failure operations counts?
3. FSDataOutputStream is created with a null statistics
> AliyunOSS: fix incorrect ReadOps and WriteOps in statistics
> ---
>
> Key: HADOOP-15917
> URL: https://issues.apache.org/jira/browse/HADOOP-15917
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/oss
>Affects Versions: 2.10.0, 2.9.1, 3.2.0, 3.1.1, 3.0.3
>Reporter: wujinhu
>Assignee: wujinhu
>Priority: Major
> Attachments: HADOOP-15917.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15917) AliyunOSS: fix incorrect ReadOps and WriteOps in statistics
[ https://issues.apache.org/jira/browse/HADOOP-15917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen reassigned HADOOP-15917: --- Assignee: Sammi Chen (was: wujinhu) > AliyunOSS: fix incorrect ReadOps and WriteOps in statistics > --- > > Key: HADOOP-15917 > URL: https://issues.apache.org/jira/browse/HADOOP-15917 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 2.10.0, 2.9.1, 3.2.0, 3.1.1, 3.0.3 >Reporter: wujinhu >Assignee: Sammi Chen >Priority: Major > Attachments: HADOOP-15917.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15917) AliyunOSS: fix incorrect ReadOps and WriteOps in statistics
[ https://issues.apache.org/jira/browse/HADOOP-15917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen reassigned HADOOP-15917: --- Assignee: wujinhu (was: Sammi Chen) > AliyunOSS: fix incorrect ReadOps and WriteOps in statistics > --- > > Key: HADOOP-15917 > URL: https://issues.apache.org/jira/browse/HADOOP-15917 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 2.10.0, 2.9.1, 3.2.0, 3.1.1, 3.0.3 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Attachments: HADOOP-15917.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15917) AliyunOSS: fix incorrect ReadOps and WriteOps in statistics
[ https://issues.apache.org/jira/browse/HADOOP-15917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen reassigned HADOOP-15917: --- Assignee: wujinhu (was: Sammi Chen) > AliyunOSS: fix incorrect ReadOps and WriteOps in statistics > --- > > Key: HADOOP-15917 > URL: https://issues.apache.org/jira/browse/HADOOP-15917 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 2.10.0, 2.9.1, 3.2.0, 3.1.1, 3.0.3 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Attachments: HADOOP-15917.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15917) AliyunOSS: fix incorrect ReadOps and WriteOps in statistics
[ https://issues.apache.org/jira/browse/HADOOP-15917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen reassigned HADOOP-15917: --- Assignee: Sammi Chen (was: wujinhu) > AliyunOSS: fix incorrect ReadOps and WriteOps in statistics > --- > > Key: HADOOP-15917 > URL: https://issues.apache.org/jira/browse/HADOOP-15917 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 2.10.0, 2.9.1, 3.2.0, 3.1.1, 3.0.3 >Reporter: wujinhu >Assignee: Sammi Chen >Priority: Major > Attachments: HADOOP-15917.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15868) AliyunOSS: update document for properties of multiple part download, multiple part upload and directory copy
[ https://issues.apache.org/jira/browse/HADOOP-15868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HADOOP-15868: Resolution: Fixed Status: Resolved (was: Patch Available) > AliyunOSS: update document for properties of multiple part download, multiple > part upload and directory copy > > > Key: HADOOP-15868 > URL: https://issues.apache.org/jira/browse/HADOOP-15868 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Affects Versions: 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.1.2 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Fix For: 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.3.0, 3.1.2 > > Attachments: HADOOP-15868.001.patch, HADOOP-15868.002.patch > > > Update document for properties of multiple part download, multiple part > upload and directory copy. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15868) AliyunOSS: update document for properties of multiple part download, multiple part upload and directory copy
[ https://issues.apache.org/jira/browse/HADOOP-15868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HADOOP-15868: Fix Version/s: 3.1.2 3.3.0 3.0.4 2.9.2 3.2.0 2.10.0 > AliyunOSS: update document for properties of multiple part download, multiple > part upload and directory copy > > > Key: HADOOP-15868 > URL: https://issues.apache.org/jira/browse/HADOOP-15868 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Affects Versions: 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.1.2 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Fix For: 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.3.0, 3.1.2 > > Attachments: HADOOP-15868.001.patch, HADOOP-15868.002.patch > > > Update document for properties of multiple part download, multiple part > upload and directory copy. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15868) AliyunOSS: update document for properties of multiple part download, multiple part upload and directory copy
[ https://issues.apache.org/jira/browse/HADOOP-15868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664801#comment-16664801 ] Sammi Chen commented on HADOOP-15868: - + 1. Committed to trunk, branch-3.0, branch-3.1, branch-3.2, branch-2, branch-2.9. > AliyunOSS: update document for properties of multiple part download, multiple > part upload and directory copy > > > Key: HADOOP-15868 > URL: https://issues.apache.org/jira/browse/HADOOP-15868 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Affects Versions: 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.1.2 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Attachments: HADOOP-15868.001.patch, HADOOP-15868.002.patch > > > Update document for properties of multiple part download, multiple part > upload and directory copy. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15868) AliyunOSS: update document for properties of multiple part download, multiple part upload and directory copy
[ https://issues.apache.org/jira/browse/HADOOP-15868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660424#comment-16660424 ] Sammi Chen commented on HADOOP-15868: - Hi [~wujinhu], the original JIRA title and description are too vague. Please try to be as specific as possible when creating new JIRAs. It will help other people to easily understand the Jira. I refined them a little bit. You can change as you want. Regarding the patch, there are some grammar corrections, use "when reading a file" instead of "when reads a file". "when uploads a file" is the same issue. > AliyunOSS: update document for properties of multiple part download, multiple > part upload and directory copy > > > Key: HADOOP-15868 > URL: https://issues.apache.org/jira/browse/HADOOP-15868 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Affects Versions: 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.1.2 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Attachments: HADOOP-15868.001.patch > > > Update document for properties of multiple part download, multiple part > upload and directory copy. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15868) AliyunOSS: update document for properties of multiple part download, multiple part upload and directory copy
[ https://issues.apache.org/jira/browse/HADOOP-15868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HADOOP-15868: Description: h1. Update document for properties of multiple part download, multiple part upload and directory copy. (was: Adds some configurations' documentation.) > AliyunOSS: update document for properties of multiple part download, multiple > part upload and directory copy > > > Key: HADOOP-15868 > URL: https://issues.apache.org/jira/browse/HADOOP-15868 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Affects Versions: 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.1.2 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Attachments: HADOOP-15868.001.patch > > > h1. Update document for properties of multiple part download, multiple part > upload and directory copy. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15868) AliyunOSS: update document for properties of multiple part download, multiple part upload and directory copy
[ https://issues.apache.org/jira/browse/HADOOP-15868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HADOOP-15868: Description: Update document for properties of multiple part download, multiple part upload and directory copy. (was: h1. Update document for properties of multiple part download, multiple part upload and directory copy.) > AliyunOSS: update document for properties of multiple part download, multiple > part upload and directory copy > > > Key: HADOOP-15868 > URL: https://issues.apache.org/jira/browse/HADOOP-15868 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Affects Versions: 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.1.2 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Attachments: HADOOP-15868.001.patch > > > Update document for properties of multiple part download, multiple part > upload and directory copy. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15868) AliyunOSS: update document for properties of multiple part download, multiple part upload and directory copy
[ https://issues.apache.org/jira/browse/HADOOP-15868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HADOOP-15868: Summary: AliyunOSS: update document for properties of multiple part download, multiple part upload and directory copy (was: AliyunOSS: update document for properties of multiple download and upload) > AliyunOSS: update document for properties of multiple part download, multiple > part upload and directory copy > > > Key: HADOOP-15868 > URL: https://issues.apache.org/jira/browse/HADOOP-15868 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Affects Versions: 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.1.2 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Attachments: HADOOP-15868.001.patch > > > Adds some configurations' documentation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15868) AliyunOSS: update document for properties of multiple download and upload
[ https://issues.apache.org/jira/browse/HADOOP-15868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HADOOP-15868: Summary: AliyunOSS: update document for properties of multiple download and upload (was: AliyunOSS: add some configurations' documentation) > AliyunOSS: update document for properties of multiple download and upload > - > > Key: HADOOP-15868 > URL: https://issues.apache.org/jira/browse/HADOOP-15868 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Affects Versions: 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.1.2 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Attachments: HADOOP-15868.001.patch > > > Adds some configurations' documentation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15671) AliyunOSS: Support Assume Roles in AliyunOSS
[ https://issues.apache.org/jira/browse/HADOOP-15671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HADOOP-15671: Fix Version/s: 3.1.2 3.0.4 2.9.2 3.2.0 2.10.0 > AliyunOSS: Support Assume Roles in AliyunOSS > > > Key: HADOOP-15671 > URL: https://issues.apache.org/jira/browse/HADOOP-15671 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Affects Versions: 3.1.0, 2.10.0, 2.9.1, 3.2.0, 3.0.3 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Fix For: 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.1.2 > > Attachments: HADOOP-15671.001.patch, HADOOP-15671.002.patch, > HADOOP-15671.003.patch, HADOOP-15671.004.patch, HADOOP-15671.005.patch, > HADOOP-15671.006.patch > > > We will add assume role function in Aliyun OSS. > For details about assume role and sts token, click the link below: > [https://www.alibabacloud.com/help/doc-detail/31935.html?spm=a2c5t.11065259.1996646101.searchclickresult.1fad155aKOUvJZ] > > Major Changes: > # Stabilise the constructor of CredentialsProvider so that other developers > can have their own implementations. > # add assumed role functions for hadoop aliyun module -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15671) AliyunOSS: Support Assume Roles in AliyunOSS
[ https://issues.apache.org/jira/browse/HADOOP-15671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HADOOP-15671: Resolution: Fixed Status: Resolved (was: Patch Available) > AliyunOSS: Support Assume Roles in AliyunOSS > > > Key: HADOOP-15671 > URL: https://issues.apache.org/jira/browse/HADOOP-15671 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Affects Versions: 3.1.0, 2.10.0, 2.9.1, 3.2.0, 3.0.3 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Fix For: 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.1.2 > > Attachments: HADOOP-15671.001.patch, HADOOP-15671.002.patch, > HADOOP-15671.003.patch, HADOOP-15671.004.patch, HADOOP-15671.005.patch, > HADOOP-15671.006.patch > > > We will add assume role function in Aliyun OSS. > For details about assume role and sts token, click the link below: > [https://www.alibabacloud.com/help/doc-detail/31935.html?spm=a2c5t.11065259.1996646101.searchclickresult.1fad155aKOUvJZ] > > Major Changes: > # Stabilise the constructor of CredentialsProvider so that other developers > can have their own implementations. > # add assumed role functions for hadoop aliyun module -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15671) AliyunOSS: Support Assume Roles in AliyunOSS
[ https://issues.apache.org/jira/browse/HADOOP-15671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16627218#comment-16627218 ] Sammi Chen commented on HADOOP-15671: - +1. Thanks [~wujinhu] for the contribution. Committed to trunk, branch-3.1, branch-3.0, branch-2 and branch-2.9. > AliyunOSS: Support Assume Roles in AliyunOSS > > > Key: HADOOP-15671 > URL: https://issues.apache.org/jira/browse/HADOOP-15671 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Affects Versions: 3.1.0, 2.10.0, 2.9.1, 3.2.0, 3.0.3 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Attachments: HADOOP-15671.001.patch, HADOOP-15671.002.patch, > HADOOP-15671.003.patch, HADOOP-15671.004.patch, HADOOP-15671.005.patch, > HADOOP-15671.006.patch > > > We will add assume role function in Aliyun OSS. > For details about assume role and sts token, click the link below: > [https://www.alibabacloud.com/help/doc-detail/31935.html?spm=a2c5t.11065259.1996646101.searchclickresult.1fad155aKOUvJZ] > > Major Changes: > # Stabilise the constructor of CredentialsProvider so that other developers > can have their own implementations. > # add assumed role functions for hadoop aliyun module -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15671) AliyunOSS: Support Assume Roles in AliyunOSS
[
https://issues.apache.org/jira/browse/HADOOP-15671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16626710#comment-16626710
]
Sammi Chen edited comment on HADOOP-15671 at 9/25/18 3:35 AM:
--
[~wujinhu], thanks for the update. A few small issues,
# TestAliyunCredentials.java, use predefined constant field to replace string
"{color:#008000}fs.oss.credentials.provider{color}"
# TestAliyunCredentials.java, InvalidCredentialsException exception is handled
in two catch branches of validateCredential. Is it expected?
# AliyunOSSUtils.java, the following two lines can be merged to 1 line.
{code:java}
URI.class, Configuration.class){code}
{code:java}
.newInstance(uri, conf);
{code}
# Kindly verify all the UTs with assume roles on branch-2, and past the result
report here.
Also please check the issue reported by check style.
was (Author: sammi):
[~wujinhu], thanks for the update. A few small issues,
# TestAliyunCredentials.java, use predefined constant field to replace string
"{color:#008000}fs.oss.credentials.provider{color}"
# TestAliyunCredentials.java, InvalidCredentialsException exception is handled
in two catch branches of validateCredential. Is it expected?
# AliyunOSSUtils.java, the following two lines can be merged to 1 line.
{code:java}
URI.class, Configuration.class){code}
{code:java}
.newInstance(uri, conf);
{code}
# Kindly verify all the UTs with assume roles on branch-2, and past the result
report here.
> AliyunOSS: Support Assume Roles in AliyunOSS
>
>
> Key: HADOOP-15671
> URL: https://issues.apache.org/jira/browse/HADOOP-15671
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/oss
>Affects Versions: 3.1.0, 2.10.0, 2.9.1, 3.2.0, 3.0.3
>Reporter: wujinhu
>Assignee: wujinhu
>Priority: Major
> Attachments: HADOOP-15671.001.patch, HADOOP-15671.002.patch,
> HADOOP-15671.003.patch, HADOOP-15671.004.patch, HADOOP-15671.005.patch
>
>
> We will add assume role function in Aliyun OSS.
> For details about assume role and sts token, click the link below:
> [https://www.alibabacloud.com/help/doc-detail/31935.html?spm=a2c5t.11065259.1996646101.searchclickresult.1fad155aKOUvJZ]
>
> Major Changes:
> # Stabilise the constructor of CredentialsProvider so that other developers
> can have their own implementations.
> # add assumed role functions for hadoop aliyun module
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15671) AliyunOSS: Support Assume Roles in AliyunOSS
[
https://issues.apache.org/jira/browse/HADOOP-15671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16626710#comment-16626710
]
Sammi Chen commented on HADOOP-15671:
-
[~wujinhu], thanks for the update. A few small issues,
# TestAliyunCredentials.java, use predefined constant field to replace string
"{color:#008000}fs.oss.credentials.provider{color}"
# TestAliyunCredentials.java, InvalidCredentialsException exception is handled
in two catch branches of validateCredential. Is it expected?
# AliyunOSSUtils.java, the following two lines can be merged to 1 line.
{code:java}
URI.class, Configuration.class){code}
{code:java}
.newInstance(uri, conf);
{code}
# Kindly verify all the UTs with assume roles on branch-2, and past the result
report here.
> AliyunOSS: Support Assume Roles in AliyunOSS
>
>
> Key: HADOOP-15671
> URL: https://issues.apache.org/jira/browse/HADOOP-15671
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/oss
>Affects Versions: 3.1.0, 2.10.0, 2.9.1, 3.2.0, 3.0.3
>Reporter: wujinhu
>Assignee: wujinhu
>Priority: Major
> Attachments: HADOOP-15671.001.patch, HADOOP-15671.002.patch,
> HADOOP-15671.003.patch, HADOOP-15671.004.patch, HADOOP-15671.005.patch
>
>
> We will add assume role function in Aliyun OSS.
> For details about assume role and sts token, click the link below:
> [https://www.alibabacloud.com/help/doc-detail/31935.html?spm=a2c5t.11065259.1996646101.searchclickresult.1fad155aKOUvJZ]
>
> Major Changes:
> # Stabilise the constructor of CredentialsProvider so that other developers
> can have their own implementations.
> # add assumed role functions for hadoop aliyun module
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15759) AliyunOSS: update oss-sdk version to 3.0.0
[ https://issues.apache.org/jira/browse/HADOOP-15759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HADOOP-15759: Resolution: Fixed Status: Resolved (was: Patch Available) > AliyunOSS: update oss-sdk version to 3.0.0 > -- > > Key: HADOOP-15759 > URL: https://issues.apache.org/jira/browse/HADOOP-15759 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Affects Versions: 2.9.1, 3.1.1, 3.0.3 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Fix For: 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.1.2 > > Attachments: HADOOP-15759.001.patch > > > OSS Java SDK 3.0.0 adds 7 CredentialsProviders. And Assumes Role Credentials > function bases on > [STSAssumeRoleSessionCredentialsProvider|https://github.com/aliyun/aliyun-oss-java-sdk/blob/master/src/main/java/com/aliyun/oss/common/auth/STSAssumeRoleSessionCredentialsProvider.java] > that sdk offers, so we update its version. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15759) AliyunOSS: update oss-sdk version to 3.0.0
[ https://issues.apache.org/jira/browse/HADOOP-15759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16619045#comment-16619045 ] Sammi Chen commented on HADOOP-15759: - Thanks [~wujinhu] for the contribution. + 1, committed to trunk, branch-3.1, branch-3.2, branch-2 and branch-2.9. > AliyunOSS: update oss-sdk version to 3.0.0 > -- > > Key: HADOOP-15759 > URL: https://issues.apache.org/jira/browse/HADOOP-15759 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Affects Versions: 2.9.1, 3.1.1, 3.0.3 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Fix For: 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.1.2 > > Attachments: HADOOP-15759.001.patch > > > OSS Java SDK 3.0.0 adds 7 CredentialsProviders. And Assumes Role Credentials > function bases on > [STSAssumeRoleSessionCredentialsProvider|https://github.com/aliyun/aliyun-oss-java-sdk/blob/master/src/main/java/com/aliyun/oss/common/auth/STSAssumeRoleSessionCredentialsProvider.java] > that sdk offers, so we update its version. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15759) AliyunOSS: update oss-sdk version to 3.0.0
[ https://issues.apache.org/jira/browse/HADOOP-15759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HADOOP-15759: Fix Version/s: 3.1.2 3.0.4 2.9.2 3.2.0 2.10.0 > AliyunOSS: update oss-sdk version to 3.0.0 > -- > > Key: HADOOP-15759 > URL: https://issues.apache.org/jira/browse/HADOOP-15759 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Affects Versions: 2.9.1, 3.1.1, 3.0.3 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Fix For: 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.1.2 > > Attachments: HADOOP-15759.001.patch > > > OSS Java SDK 3.0.0 adds 7 CredentialsProviders. And Assumes Role Credentials > function bases on > [STSAssumeRoleSessionCredentialsProvider|https://github.com/aliyun/aliyun-oss-java-sdk/blob/master/src/main/java/com/aliyun/oss/common/auth/STSAssumeRoleSessionCredentialsProvider.java] > that sdk offers, so we update its version. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15759) AliyunOSS: update oss-sdk version to 3.0.0
[ https://issues.apache.org/jira/browse/HADOOP-15759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HADOOP-15759: Affects Version/s: (was: 3.2.0) (was: 2.8.4) (was: 2.10.0) > AliyunOSS: update oss-sdk version to 3.0.0 > -- > > Key: HADOOP-15759 > URL: https://issues.apache.org/jira/browse/HADOOP-15759 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Affects Versions: 2.9.1, 3.1.1, 3.0.3 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Fix For: 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.1.2 > > Attachments: HADOOP-15759.001.patch > > > OSS Java SDK 3.0.0 adds 7 CredentialsProviders. And Assumes Role Credentials > function bases on > [STSAssumeRoleSessionCredentialsProvider|https://github.com/aliyun/aliyun-oss-java-sdk/blob/master/src/main/java/com/aliyun/oss/common/auth/STSAssumeRoleSessionCredentialsProvider.java] > that sdk offers, so we update its version. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15558) Implementation of Clay Codes plugin (Coupled Layer MSR codes)
[
https://issues.apache.org/jira/browse/HADOOP-15558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588900#comment-16588900
]
Sammi Chen edited comment on HADOOP-15558 at 8/22/18 2:11 PM:
--
Hi [~shreya2205], several comments after gone through the slides and code.
1. The encoding and decoding of Clay Codes involve PFT, PRT and RS computation.
So basically the idea is to reduce the network throughput and disk read during
data repair phase by adding additional computations. In single data node
failure case, Clay Codes can save 2/3 network bandwidth compared with RS. In
worst case, Clay Codes will behave the same as RS from network bandwidth wise.
Given that most of the failures are single node failure in the storage cluster,
cluster can benefit from Clay Codes with no doubt. I assume all the benchmark
data in slides are collected in single data node failure case. Correct me if
it's not correct.
2. On P22 of slides, it says " total encoding time remains the same while Clay
Codec has 70% higher encode computation time". Confused, could you explain it
further?
3. On P21 of slices, Fragmented Read, it says there is no impact on SSD when
sub-chunk size reaches 4KB. DO you have any data for HDD? Since the
Hadoop/HDFS, HDD is still the majority.
4. P23, what does the "Degraded I/O" scenario means in the slides?
5. From the slices, we can see to configure a Clay Codec, k, m, d and sub-trunk
size all have matter. While in the implementation, only k and m are
configurable. What about d and sub-trunk?
6. I googled a lot but found very few links about PFT and PRT matrix. Do you
have any documents for them?
7. For implementation part, is clone input blocks a must when
prepareEncodingStep? Also could you add more comments, such as whih part is
PFT computation, and PRT computation. I will go through the code again later.
Also ClayCodeUtil is better to be placed in a new file.
8. Code style. Here are a list of Hadoop code styles to follow.
* a. Import * is not recommended
* b. a line cannot exceed 80 characters
* c. tab is 4 spaces
* d. new line indent is 2 spaces
* e. cross line indent is 4 spaces
* f. remove unnecessary empty line
* g. 1 space between operator and value,
for examples,
{code:java}
if (rsRawDecoder==null) { => if (rsRawDecoder == null) {
new ErasureCoderOptions(2,2); => new ErasureCoderOptions(2, 2);
if(erasedIndexes.length==1){ =>if (erasedIndexes.length == 1) {
{code}
was (Author: sammi):
Hi [~shreya2205], several comments after gone through the slides and code.
1. The encoding and decoding of Clay Codes involve PFT, PRT and RS computation.
So basically the idea is to reduce the network throughput and disk read during
data repair phase by adding additional computations. In single data node
failure case, Clay Codes can save 2/3 network bandwidth compared with RS. In
worst case, Clay Codes will behave the same as RS from network bandwidth wise.
Given that most of the failures are single node failure in the storage cluster,
cluster can benefit from Clay Codes with no doubt. I assume all the benchmark
data in slides are collected in single data node failure case. Correct me if
it's not correct.
2. On P22 of slides, it says " total encoding time remains the same while Clay
Codec has 70% higher encode computation time". Confused, could you explain it
further?
3. On P21 of slices, Fragmented Read, it says there is no impact on SSD when
sub-chunk size reaches 4KB. DO you have any data for HDD? Since the
Hadoop/HDFS, HDD is still the majority.
4. P23, what does the "Degraded I/O" scenario means in the slides?
5. From the slices, we can see to configure a Clay Codec, k, m, d and sub-trunk
size all have matter. While in the implementation, only k and m are
configurable. What about d and sub-trunk?
6. I googled a lot but found very few links about PFT and PRT matrix. Do you
have any documents for them?
7. For implementation part, is clone input blocks a must when
prepareEncodingStep? Also could you add more comments, such as whih part is
PFT computation, and PRT computation. I will go through the code again later.
Also ClayCodeUtil is better to be placed in a new file.
8. Code style. Here are a list of Hadoop code styles to follow.
a. Import * is not recommended
b. a line cannot exceed 80 characters
c. tab is 4 spaces
d. new line indent is 2 spaces
e. cross line indent is 4 spaces
f. remove unnecessary empty line
g. 1 space between operator and value,
for examples,
if (rsRawDecoder==null) { => if (rsRawDecoder == null) {
new ErasureCoderOptions(2,2); => new ErasureCoderOptions(2,
2);
if(erasedIndexes.length==1){ =>if
[jira] [Commented] (HADOOP-15558) Implementation of Clay Codes plugin (Coupled Layer MSR codes)
[
https://issues.apache.org/jira/browse/HADOOP-15558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588900#comment-16588900
]
Sammi Chen commented on HADOOP-15558:
-
Hi [~shreya2205], several comments after gone through the slides and code.
1. The encoding and decoding of Clay Codes involve PFT, PRT and RS computation.
So basically the idea is to reduce the network throughput and disk read during
data repair phase by adding additional computations. In single data node
failure case, Clay Codes can save 2/3 network bandwidth compared with RS. In
worst case, Clay Codes will behave the same as RS from network bandwidth wise.
Given that most of the failures are single node failure in the storage cluster,
cluster can benefit from Clay Codes with no doubt. I assume all the benchmark
data in slides are collected in single data node failure case. Correct me if
it's not correct.
2. On P22 of slides, it says " total encoding time remains the same while Clay
Codec has 70% higher encode computation time". Confused, could you explain it
further?
3. On P21 of slices, Fragmented Read, it says there is no impact on SSD when
sub-chunk size reaches 4KB. DO you have any data for HDD? Since the
Hadoop/HDFS, HDD is still the majority.
4. P23, what does the "Degraded I/O" scenario means in the slides?
5. From the slices, we can see to configure a Clay Codec, k, m, d and sub-trunk
size all have matter. While in the implementation, only k and m are
configurable. What about d and sub-trunk?
6. I googled a lot but found very few links about PFT and PRT matrix. Do you
have any documents for them?
7. For implementation part, is clone input blocks a must when
prepareEncodingStep? Also could you add more comments, such as whih part is
PFT computation, and PRT computation. I will go through the code again later.
Also ClayCodeUtil is better to be placed in a new file.
8. Code style. Here are a list of Hadoop code styles to follow.
a. Import * is not recommended
b. a line cannot exceed 80 characters
c. tab is 4 spaces
d. new line indent is 2 spaces
e. cross line indent is 4 spaces
f. remove unnecessary empty line
g. 1 space between operator and value,
for examples,
if (rsRawDecoder==null) { => if (rsRawDecoder == null) {
new ErasureCoderOptions(2,2); => new ErasureCoderOptions(2,
2);
if(erasedIndexes.length==1){ =>if (erasedIndexes.length ==
1) {
> Implementation of Clay Codes plugin (Coupled Layer MSR codes)
> --
>
> Key: HADOOP-15558
> URL: https://issues.apache.org/jira/browse/HADOOP-15558
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Chaitanya Mukka
>Assignee: Chaitanya Mukka
>Priority: Major
> Attachments: ClayCodeCodecDesign-20180630.pdf,
> HADOOP-15558.001.patch, HADOOP-15558.002.patch
>
>
> [Clay Codes|https://www.usenix.org/conference/fast18/presentation/vajha] are
> new erasure codes developed as a research project at Codes and Signal Design
> Lab, IISc Bangalore. A particular Clay code, with storage overhead 1.25x, has
> been shown to reduce repair network traffic, disk read and repair times by
> factors of 2.9, 3.4 and 3 respectively compared to the RS codes with the same
> parameters.
> This Jira aims to introduce Clay Codes to HDFS-EC as one of the pluggable
> erasure codec.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15558) Implementation of Clay Codes plugin (Coupled Layer MSR codes)
[ https://issues.apache.org/jira/browse/HADOOP-15558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen reassigned HADOOP-15558: --- Assignee: Chaitanya Mukka (was: Sammi Chen) > Implementation of Clay Codes plugin (Coupled Layer MSR codes) > -- > > Key: HADOOP-15558 > URL: https://issues.apache.org/jira/browse/HADOOP-15558 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Chaitanya Mukka >Assignee: Chaitanya Mukka >Priority: Major > Attachments: ClayCodeCodecDesign-20180630.pdf, > HADOOP-15558.001.patch, HADOOP-15558.002.patch > > > [Clay Codes|https://www.usenix.org/conference/fast18/presentation/vajha] are > new erasure codes developed as a research project at Codes and Signal Design > Lab, IISc Bangalore. A particular Clay code, with storage overhead 1.25x, has > been shown to reduce repair network traffic, disk read and repair times by > factors of 2.9, 3.4 and 3 respectively compared to the RS codes with the same > parameters. > This Jira aims to introduce Clay Codes to HDFS-EC as one of the pluggable > erasure codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15558) Implementation of Clay Codes plugin (Coupled Layer MSR codes)
[ https://issues.apache.org/jira/browse/HADOOP-15558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen reassigned HADOOP-15558: --- Assignee: Sammi Chen (was: Chaitanya Mukka) > Implementation of Clay Codes plugin (Coupled Layer MSR codes) > -- > > Key: HADOOP-15558 > URL: https://issues.apache.org/jira/browse/HADOOP-15558 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Chaitanya Mukka >Assignee: Sammi Chen >Priority: Major > Attachments: ClayCodeCodecDesign-20180630.pdf, > HADOOP-15558.001.patch, HADOOP-15558.002.patch > > > [Clay Codes|https://www.usenix.org/conference/fast18/presentation/vajha] are > new erasure codes developed as a research project at Codes and Signal Design > Lab, IISc Bangalore. A particular Clay code, with storage overhead 1.25x, has > been shown to reduce repair network traffic, disk read and repair times by > factors of 2.9, 3.4 and 3 respectively compared to the RS codes with the same > parameters. > This Jira aims to introduce Clay Codes to HDFS-EC as one of the pluggable > erasure codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15671) AliyunOSS: Support Assume Roles in AliyunOSS
[
https://issues.apache.org/jira/browse/HADOOP-15671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588319#comment-16588319
]
Sammi Chen commented on HADOOP-15671:
-
Thanks [~wujinhu] for the work. Some comments,
1. What's the time unit of the default value. Would you please add a comment?
public static final long ASSUMED_ROLE_DURATION_DEFAULT = 30 * 60;
2. "will be generated" and “." period is missed at the end.
Session name for the assumed role, must be valid characters
according to Aliyun API. It is optional, will generated by
oss java sdk if it is empty
same "." missing at end of "Default is 30 minutes"
3. when "fs.oss.credentials.provider" is
"org.apache.hadoop.fs.aliyun.oss.AssumedRoleCredentialProvider", are
"fs.oss.accessKeyId" and "fs.oss.accessKeySecret" still mandatory property?
4. import * is now recommended in Hadoop code style.
import static org.apache.hadoop.fs.aliyun.oss.Constants.*;
import com.aliyun.oss.common.auth.*;
5. Invalid parameter cases are verified in TestAliyunCredentials while there is
no success cases. Would you please add more UT to verify that the new
credential provider works as expected.
6. Java doc is not updated for new getCredentialsProvider API. The " throws
IOException {" statement can be in the same line with " URI uri, Configuration
conf)"
7. In AssumedRoleCredentialProvider, when getCredentials is called,
crendentials are retrieved from stsCredentialsProvider, which makes
setCredentials useless. Is it an expected behavior? Also please try to add
"final" to fields as long as it fits.
8. I agree with Steve Loughran. Please fire another JIRA to raise the OSS JDK
version. That one can be committed first.
> AliyunOSS: Support Assume Roles in AliyunOSS
>
>
> Key: HADOOP-15671
> URL: https://issues.apache.org/jira/browse/HADOOP-15671
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/oss
>Affects Versions: 3.1.0, 2.10.0, 2.9.1, 3.2.0, 3.0.3
>Reporter: wujinhu
>Assignee: wujinhu
>Priority: Major
> Attachments: HADOOP-15671.001.patch, HADOOP-15671.002.patch
>
>
> We will add assume role function in Aliyun OSS.
> For details about assume role and sts token, click the link below:
> [https://www.alibabacloud.com/help/doc-detail/31935.html?spm=a2c5t.11065259.1996646101.searchclickresult.1fad155aKOUvJZ]
>
> Major Changes:
> # Stabilise the constructor of CredentialsProvider so that other developers
> can have their own implementations.
> # change version of oss java sdk from 2.8.3 to 3.0.0 since new version
> contains the basic function of assumed role credentials provider.
> # add assumed role functions for hadoop aliyun module
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15558) Implementation of Clay Codes plugin (Coupled Layer MSR codes)
[ https://issues.apache.org/jira/browse/HADOOP-15558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16566317#comment-16566317 ] Sammi Chen commented on HADOOP-15558: - Hi [~cmukka20] and [~shreya2205], thanks for firing this JIRA. I will take some time to go through the design document & patch, and provide feedback later. > Implementation of Clay Codes plugin (Coupled Layer MSR codes) > -- > > Key: HADOOP-15558 > URL: https://issues.apache.org/jira/browse/HADOOP-15558 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Chaitanya Mukka >Assignee: Chaitanya Mukka >Priority: Major > Attachments: ClayCodeCodecDesign-20180630.pdf, > HADOOP-15558.001.patch, HADOOP-15558.002.patch > > > [Clay Codes|https://www.usenix.org/conference/fast18/presentation/vajha] are > new erasure codes developed as a research project at Codes and Signal Design > Lab, IISc Bangalore. A particular Clay code, with storage overhead 1.25x, has > been shown to reduce repair network traffic, disk read and repair times by > factors of 2.9, 3.4 and 3 respectively compared to the RS codes with the same > parameters. > This Jira aims to introduce Clay Codes to HDFS-EC as one of the pluggable > erasure codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15607) AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
[
https://issues.apache.org/jira/browse/HADOOP-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sammi Chen updated HADOOP-15607:
Priority: Critical (was: Major)
> AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
> -
>
> Key: HADOOP-15607
> URL: https://issues.apache.org/jira/browse/HADOOP-15607
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.1.0, 2.10.0, 2.9.1, 3.2.0, 3.0.3
>Reporter: wujinhu
>Assignee: wujinhu
>Priority: Critical
> Fix For: 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.1.2
>
> Attachments: HADOOP-15607-branch-2.001.patch, HADOOP-15607.001.patch,
> HADOOP-15607.002.patch, HADOOP-15607.003.patch, HADOOP-15607.004.patch
>
>
> When I generated data with hive-tpcds tool, I got exception below:
> 2018-07-16 14:50:43,680 INFO mapreduce.Job: Task Id :
> attempt_1531723399698_0001_m_52_0, Status : FAILED
> Error: com.aliyun.oss.OSSException: The list of parts was not in ascending
> order. Parts list must specified in order by part number.
> [ErrorCode]: InvalidPartOrder
> [RequestId]: 5B4C40425FCC208D79D1EAF5
> [HostId]: 100.103.0.137
> [ResponseError]:
>
>
> InvalidPartOrder
> The list of parts was not in ascending order. Parts list must
> specified in order by part number.
> 5B4C40425FCC208D79D1EAF5
> xx.xx.xx.xx
> current PartNumber 3, you given part number 3is not in
> ascending order
>
> at
> com.aliyun.oss.common.utils.ExceptionFactory.createOSSException(ExceptionFactory.java:99)
> at
> com.aliyun.oss.internal.OSSErrorResponseHandler.handle(OSSErrorResponseHandler.java:69)
> at
> com.aliyun.oss.common.comm.ServiceClient.handleResponse(ServiceClient.java:248)
> at
> com.aliyun.oss.common.comm.ServiceClient.sendRequestImpl(ServiceClient.java:130)
> at
> com.aliyun.oss.common.comm.ServiceClient.sendRequest(ServiceClient.java:68)
> at com.aliyun.oss.internal.OSSOperation.send(OSSOperation.java:94)
> at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:149)
> at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:113)
> at
> com.aliyun.oss.internal.OSSMultipartOperation.completeMultipartUpload(OSSMultipartOperation.java:185)
> at com.aliyun.oss.OSSClient.completeMultipartUpload(OSSClient.java:790)
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.completeMultipartUpload(AliyunOSSFileSystemStore.java:643)
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSBlockOutputStream.close(AliyunOSSBlockOutputStream.java:120)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
> at
> org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106)
> at
> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.close(MultipleOutputs.java:574)
> at org.notmysock.tpcds.GenTable$DSDGen.cleanup(GenTable.java:169)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686)
>
> I reviewed code below,
> {code:java}
> blockId {code}
> has thread synchronization problem
> {code:java}
> // code placeholder
> private void uploadCurrentPart() throws IOException {
> blockFiles.add(blockFile);
> blockStream.flush();
> blockStream.close();
> if (blockId == 0) {
> uploadId = store.getUploadId(key);
> }
> ListenableFuture partETagFuture =
> executorService.submit(() -> {
> PartETag partETag = store.uploadPart(blockFile, key, uploadId,
> blockId + 1);
> return partETag;
> });
> partETagsFutures.add(partETagFuture);
> blockFile = newBlockFile();
> blockId++;
> blockStream = new BufferedOutputStream(new FileOutputStream(blockFile));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15607) AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
[
https://issues.apache.org/jira/browse/HADOOP-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sammi Chen updated HADOOP-15607:
Fix Version/s: 2.9.2
2.10.0
> AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
> -
>
> Key: HADOOP-15607
> URL: https://issues.apache.org/jira/browse/HADOOP-15607
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.1.0, 2.10.0, 2.9.1, 3.2.0, 3.0.3
>Reporter: wujinhu
>Assignee: wujinhu
>Priority: Major
> Fix For: 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.1.2
>
> Attachments: HADOOP-15607-branch-2.001.patch, HADOOP-15607.001.patch,
> HADOOP-15607.002.patch, HADOOP-15607.003.patch, HADOOP-15607.004.patch
>
>
> When I generated data with hive-tpcds tool, I got exception below:
> 2018-07-16 14:50:43,680 INFO mapreduce.Job: Task Id :
> attempt_1531723399698_0001_m_52_0, Status : FAILED
> Error: com.aliyun.oss.OSSException: The list of parts was not in ascending
> order. Parts list must specified in order by part number.
> [ErrorCode]: InvalidPartOrder
> [RequestId]: 5B4C40425FCC208D79D1EAF5
> [HostId]: 100.103.0.137
> [ResponseError]:
>
>
> InvalidPartOrder
> The list of parts was not in ascending order. Parts list must
> specified in order by part number.
> 5B4C40425FCC208D79D1EAF5
> xx.xx.xx.xx
> current PartNumber 3, you given part number 3is not in
> ascending order
>
> at
> com.aliyun.oss.common.utils.ExceptionFactory.createOSSException(ExceptionFactory.java:99)
> at
> com.aliyun.oss.internal.OSSErrorResponseHandler.handle(OSSErrorResponseHandler.java:69)
> at
> com.aliyun.oss.common.comm.ServiceClient.handleResponse(ServiceClient.java:248)
> at
> com.aliyun.oss.common.comm.ServiceClient.sendRequestImpl(ServiceClient.java:130)
> at
> com.aliyun.oss.common.comm.ServiceClient.sendRequest(ServiceClient.java:68)
> at com.aliyun.oss.internal.OSSOperation.send(OSSOperation.java:94)
> at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:149)
> at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:113)
> at
> com.aliyun.oss.internal.OSSMultipartOperation.completeMultipartUpload(OSSMultipartOperation.java:185)
> at com.aliyun.oss.OSSClient.completeMultipartUpload(OSSClient.java:790)
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.completeMultipartUpload(AliyunOSSFileSystemStore.java:643)
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSBlockOutputStream.close(AliyunOSSBlockOutputStream.java:120)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
> at
> org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106)
> at
> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.close(MultipleOutputs.java:574)
> at org.notmysock.tpcds.GenTable$DSDGen.cleanup(GenTable.java:169)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686)
>
> I reviewed code below,
> {code:java}
> blockId {code}
> has thread synchronization problem
> {code:java}
> // code placeholder
> private void uploadCurrentPart() throws IOException {
> blockFiles.add(blockFile);
> blockStream.flush();
> blockStream.close();
> if (blockId == 0) {
> uploadId = store.getUploadId(key);
> }
> ListenableFuture partETagFuture =
> executorService.submit(() -> {
> PartETag partETag = store.uploadPart(blockFile, key, uploadId,
> blockId + 1);
> return partETag;
> });
> partETagsFutures.add(partETagFuture);
> blockFile = newBlockFile();
> blockId++;
> blockStream = new BufferedOutputStream(new FileOutputStream(blockFile));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15607) AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
[
https://issues.apache.org/jira/browse/HADOOP-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564760#comment-16564760
]
Sammi Chen commented on HADOOP-15607:
-
I fired YETUS-652 to track the build issue. Will commit the patch to branch-2
and branch-2.9 soon.
> AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
> -
>
> Key: HADOOP-15607
> URL: https://issues.apache.org/jira/browse/HADOOP-15607
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.1.0, 2.10.0, 2.9.1, 3.2.0, 3.0.3
>Reporter: wujinhu
>Assignee: wujinhu
>Priority: Major
> Fix For: 3.2.0, 3.0.4, 3.1.2
>
> Attachments: HADOOP-15607-branch-2.001.patch, HADOOP-15607.001.patch,
> HADOOP-15607.002.patch, HADOOP-15607.003.patch, HADOOP-15607.004.patch
>
>
> When I generated data with hive-tpcds tool, I got exception below:
> 2018-07-16 14:50:43,680 INFO mapreduce.Job: Task Id :
> attempt_1531723399698_0001_m_52_0, Status : FAILED
> Error: com.aliyun.oss.OSSException: The list of parts was not in ascending
> order. Parts list must specified in order by part number.
> [ErrorCode]: InvalidPartOrder
> [RequestId]: 5B4C40425FCC208D79D1EAF5
> [HostId]: 100.103.0.137
> [ResponseError]:
>
>
> InvalidPartOrder
> The list of parts was not in ascending order. Parts list must
> specified in order by part number.
> 5B4C40425FCC208D79D1EAF5
> xx.xx.xx.xx
> current PartNumber 3, you given part number 3is not in
> ascending order
>
> at
> com.aliyun.oss.common.utils.ExceptionFactory.createOSSException(ExceptionFactory.java:99)
> at
> com.aliyun.oss.internal.OSSErrorResponseHandler.handle(OSSErrorResponseHandler.java:69)
> at
> com.aliyun.oss.common.comm.ServiceClient.handleResponse(ServiceClient.java:248)
> at
> com.aliyun.oss.common.comm.ServiceClient.sendRequestImpl(ServiceClient.java:130)
> at
> com.aliyun.oss.common.comm.ServiceClient.sendRequest(ServiceClient.java:68)
> at com.aliyun.oss.internal.OSSOperation.send(OSSOperation.java:94)
> at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:149)
> at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:113)
> at
> com.aliyun.oss.internal.OSSMultipartOperation.completeMultipartUpload(OSSMultipartOperation.java:185)
> at com.aliyun.oss.OSSClient.completeMultipartUpload(OSSClient.java:790)
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.completeMultipartUpload(AliyunOSSFileSystemStore.java:643)
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSBlockOutputStream.close(AliyunOSSBlockOutputStream.java:120)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
> at
> org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106)
> at
> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.close(MultipleOutputs.java:574)
> at org.notmysock.tpcds.GenTable$DSDGen.cleanup(GenTable.java:169)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686)
>
> I reviewed code below,
> {code:java}
> blockId {code}
> has thread synchronization problem
> {code:java}
> // code placeholder
> private void uploadCurrentPart() throws IOException {
> blockFiles.add(blockFile);
> blockStream.flush();
> blockStream.close();
> if (blockId == 0) {
> uploadId = store.getUploadId(key);
> }
> ListenableFuture partETagFuture =
> executorService.submit(() -> {
> PartETag partETag = store.uploadPart(blockFile, key, uploadId,
> blockId + 1);
> return partETag;
> });
> partETagsFutures.add(partETagFuture);
> blockFile = newBlockFile();
> blockId++;
> blockStream = new BufferedOutputStream(new FileOutputStream(blockFile));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
