[jira] [Updated] (HADOOP-9975) Adding relogin() method to UGI
[ https://issues.apache.org/jira/browse/HADOOP-9975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-9975: -- Attachment: HADOOP-9975.patch Attached a patch based on HADOOP-9926 and will submit when the dep is resolved. Adding relogin() method to UGI -- Key: HADOOP-9975 URL: https://issues.apache.org/jira/browse/HADOOP-9975 Project: Hadoop Common Issue Type: Improvement Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HADOOP-9975.patch In current Hadoop UGI implementation, it has API methods like reloginFromKeytab() and reloginFromTicketCache(). However, such methods are too Kerberos specific and also involves login implementation details, it would be better to add generic relogin() method regardless authentication mechanism. This is possible since relevant authentication specific parameters like principal and keytab are already passed and saved in the UGI object after initial login. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9797) Pluggable and compatible UGI change
[ https://issues.apache.org/jira/browse/HADOOP-9797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-9797: -- Attachment: HADOOP-9797-v3.patch Updated the overall patch for convenient review. Pluggable and compatible UGI change --- Key: HADOOP-9797 URL: https://issues.apache.org/jira/browse/HADOOP-9797 Project: Hadoop Common Issue Type: Sub-task Components: security Reporter: Kai Zheng Assignee: Kai Zheng Labels: Rhino Fix For: 3.0.0 Attachments: HADOOP-9797-v1.patch, HADOOP-9797-v2.patch, HADOOP-9797-v3.patch, Pluggable and Compatible UGI Change.pdf As already widely discussed current UGI related classes needs to be improved in many aspects. This is to improve and make UGI so that it can be: * Pluggable, new authentication method with its login module can be dynamically registered and plugged without having to change the UGI class; * Extensible, login modules with their options can be dynamically extended and customized so that can be reusable elsewhere, like in TokenAuth; * No Kerberos relevant, remove any Kerberos relevant functionalities out of it to make it simple and suitable for other login mechanisms; * Of appropriate abstraction and API, with improved abstraction and API it’s possible to allow authentication implementations not using JAAS modules; * Compatible, should be compatible with previous deployment and authentication methods, so the existing APIs won’t be removed and some of them are just to be deprecated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9852) UGI login user keytab and principal should not be static
[ https://issues.apache.org/jira/browse/HADOOP-9852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13776109#comment-13776109 ] Kai Zheng commented on HADOOP-9852: --- Could I have some comments on this? I understand to make the static keytab and principal not static any more the patch wraps them in the loginConf and then keeps the loginConf in the private credentials of the UGI's user object. How about keeping the loginConf in the UGI object directly as HADOOP-9797 does similarly? In HADOOP-9797, UGI keeps the HadoopLogin object, which contains the needed login configuration, and only for Kerberos keytab login configuration, it saves the keytab and principal variables. I understand HADOOP-9797 can't serve here since it's too large, though. One minor, since it adds getLoginConfiguration(), would be better to have setLoginConfiguration(), instead of changing the existing to have setLogin(LoginContext login, HadoopConfiguration loginConf). UGI login user keytab and principal should not be static Key: HADOOP-9852 URL: https://issues.apache.org/jira/browse/HADOOP-9852 Project: Hadoop Common Issue Type: Sub-task Components: security Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HADOOP-9852.patch The static keytab and principal for the login user is problematic. The login conf explicitly references these statics. As a result, loginUserFromKeytabAndReturnUGI is unnecessarily synch'ed on the class to swap out the login user's keytab and principal, login, then restore the keytab/principal. This method's synch blocks further de-synching of other methods. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8315) Support SASL-authenticated ZooKeeper in ActiveStandbyElector
[ https://issues.apache.org/jira/browse/HADOOP-8315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777230#comment-13777230 ] Kai Zheng commented on HADOOP-8315: --- Correction: the issue HDFS-5152 mentioned above was transferred to HADOOP-9938. Support SASL-authenticated ZooKeeper in ActiveStandbyElector Key: HADOOP-8315 URL: https://issues.apache.org/jira/browse/HADOOP-8315 Project: Hadoop Common Issue Type: Improvement Components: auto-failover, ha Affects Versions: Auto Failover (HDFS-3042) Reporter: Todd Lipcon Assignee: Todd Lipcon Attachments: hadoop-8315.txt, hadoop-8315_v2.txt Currently, if you try to use SASL-authenticated ZK with the ActiveStandbyElector, you run into a couple issues: 1) We hit ZOOKEEPER-1437 - we need to wait until we see SaslAuthenticated before we can make any requests 2) We currently throw a fatalError when we see the SaslAuthenticated callback on the connection watcher We need to wait for ZK-1437 upstream, and then upgrade to the fixed version for #1. For #2 we just need to add a case there and ignore it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9797) Pluggable and compatible UGI change
[ https://issues.apache.org/jira/browse/HADOOP-9797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13808656#comment-13808656 ] Kai Zheng commented on HADOOP-9797: --- Hi Larry, Thanks for taking care of this. You're right you found a coding error in getJaasLoginConfiguration(). The code was obsolete when introduced HadoopLoginFactory, and I should have removed the dead code. Thanks anyway. Sure to simply add the TokenAuthn method, we don't have to need this, as can be seen in HADOOP-9804. I'm probably running this for the long term trying various possible enhancements with incremental patches. Pluggable and compatible UGI change --- Key: HADOOP-9797 URL: https://issues.apache.org/jira/browse/HADOOP-9797 Project: Hadoop Common Issue Type: Sub-task Components: security Reporter: Kai Zheng Assignee: Kai Zheng Labels: Rhino Fix For: 3.0.0 Attachments: HADOOP-9797-v1.patch, HADOOP-9797-v2.patch, HADOOP-9797-v3.patch, Pluggable and Compatible UGI Change.pdf As already widely discussed current UGI related classes needs to be improved in many aspects. This is to improve and make UGI so that it can be: * Pluggable, new authentication method with its login module can be dynamically registered and plugged without having to change the UGI class; * Extensible, login modules with their options can be dynamically extended and customized so that can be reusable elsewhere, like in TokenAuth; * No Kerberos relevant, remove any Kerberos relevant functionalities out of it to make it simple and suitable for other login mechanisms; * Of appropriate abstraction and API, with improved abstraction and API it’s possible to allow authentication implementations not using JAAS modules; * Compatible, should be compatible with previous deployment and authentication methods, so the existing APIs won’t be removed and some of them are just to be deprecated. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-10959) A complement solution to TokenAuth based on Kerberos pre-authentication framework
[ https://issues.apache.org/jira/browse/HADOOP-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257998#comment-14257998 ] Kai Zheng commented on HADOOP-10959: Just some update. We're working on defining the token-preauth and access-token-profile drafts with MIT Kerberos team. As it's of low priority it's running very slow. Meanwhile, we have also initiated [Haox|https://github.com/drankye/haox] project, targeting a Java Kerberos implementation and based on it, we're going to prototype the Kerberos extensions in not so long future. https://github.com/drankye/haox A complement solution to TokenAuth based on Kerberos pre-authentication framework - Key: HADOOP-10959 URL: https://issues.apache.org/jira/browse/HADOOP-10959 Project: Hadoop Common Issue Type: New Feature Components: security Reporter: Kai Zheng Assignee: Kai Zheng Labels: Rhino Attachments: KerbToken-v2.pdf To implement and integrate pluggable authentication providers, enhance desirable single sign on for end users, and help enforce centralized access control on the platform, the community has widely discussed and concluded token based authentication could be the appropriate approach. TokenAuth (HADOOP-9392) was proposed and is under development to implement another Authentication Method in lieu with Simple and Kerberos. It is a big and long term effort to support TokenAuth across the entire ecosystem. We here propose a short term replacement based on Kerberos that can complement to TokenAuth. Our solution involves less codes changes with limited risk and the main development work has already been done in our POC. Users can use our solution as a short term solution to support token inside Hadoop. This effort and resultant solution will be fully described in the design document to be attached. And the brief introduction will be commented. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11535) TableMapping related tests failed due to 'correct' resolving for test hostname
[
https://issues.apache.org/jira/browse/HADOOP-11535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303163#comment-14303163
]
Kai Zheng commented on HADOOP-11535:
Hi Steve, thanks for your response.
bq. the IETF mandates that these hostnames must never resolve
Would you clarify which hostnames not to resolve ? Does a.b.c be such one ?
Which IETF spec states this ?
I'd like to investigate a bit further, because it always resolves in my company
dev environment, which fails the tests.
Thanks.
TableMapping related tests failed due to 'correct' resolving for test hostname
--
Key: HADOOP-11535
URL: https://issues.apache.org/jira/browse/HADOOP-11535
Project: Hadoop Common
Issue Type: Test
Reporter: Kai Zheng
Assignee: Kai Zheng
Priority: Minor
When mvn test in my environment, it reported the following.
{noformat}
Failed tests:
TestTableMapping.testClearingCachedMappings:144 expected:/[rack1] but
was:/[default-rack]
TestTableMapping.testTableCaching:79 expected:/[rack1] but
was:/[default-rack]
TestTableMapping.testResolve:56 expected:/[rack1] but
was:/[default-rack]
{noformat}
It's caused by the good resolving for the 'bad test' hostname 'a.b.c' as
follows.
{noformat}
[drankye@zkdesk hadoop-common-project]$ ping a.b.c
PING a.b.c (220.250.64.228) 56(84) bytes of data.
{noformat}
I understand it may happen in just my local environment, and document this
just in case others also meet this. We may use even worse hostname than
'a.b.c' to avoid such situation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11540) Raw Reed-Solomon coder using Intel ISA-L library
[ https://issues.apache.org/jira/browse/HADOOP-11540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14306516#comment-14306516 ] Kai Zheng commented on HADOOP-11540: bq.You are going to use the Intel ISA-L library but not actually implementing it. I'd clarify further that we actually implement one in pure Java for RS code scheme, which should make sense in some environment where native ones aren't available in case. Please see HADOOP-11542. Raw Reed-Solomon coder using Intel ISA-L library Key: HADOOP-11540 URL: https://issues.apache.org/jira/browse/HADOOP-11540 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HDFS-EC Reporter: Zhe Zhang Assignee: Kai Zheng This is to provide RS codec implementation using Intel ISA-L library for encoding and decoding. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11542) Raw Reed-Solomon coder in pure Java
[ https://issues.apache.org/jira/browse/HADOOP-11542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11542: --- Attachment: HADOOP-11542-v1.patch Updated the patch sync-ing with other relevant patches. More tests added and passed. Raw Reed-Solomon coder in pure Java --- Key: HADOOP-11542 URL: https://issues.apache.org/jira/browse/HADOOP-11542 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HADOOP-11542-v1.patch, HDFS-7418-v1.patch, HDFS-7418-v2.patch This will implement RS coder by porting existing codes in HDFS-RAID in the new codec and coder framework, which could be useful in case native support isn't available or convenient in some environments or platforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11542) Raw Reed-Solomon coder in pure Java
[ https://issues.apache.org/jira/browse/HADOOP-11542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11542: --- Attachment: (was: HDFS-7418-v2.patch) Raw Reed-Solomon coder in pure Java --- Key: HADOOP-11542 URL: https://issues.apache.org/jira/browse/HADOOP-11542 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HADOOP-11542-v1.patch This will implement RS coder by porting existing codes in HDFS-RAID in the new codec and coder framework, which could be useful in case native support isn't available or convenient in some environments or platforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11542) Raw Reed-Solomon coder in pure Java
[ https://issues.apache.org/jira/browse/HADOOP-11542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11542: --- Attachment: (was: HDFS-7418-v1.patch) Raw Reed-Solomon coder in pure Java --- Key: HADOOP-11542 URL: https://issues.apache.org/jira/browse/HADOOP-11542 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HADOOP-11542-v1.patch This will implement RS coder by porting existing codes in HDFS-RAID in the new codec and coder framework, which could be useful in case native support isn't available or convenient in some environments or platforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11535) TableMapping related tests failed due to 'correct' resolving for test hostname
Kai Zheng created HADOOP-11535:
--
Summary: TableMapping related tests failed due to 'correct'
resolving for test hostname
Key: HADOOP-11535
URL: https://issues.apache.org/jira/browse/HADOOP-11535
Project: Hadoop Common
Issue Type: Test
Reporter: Kai Zheng
Assignee: Kai Zheng
Priority: Minor
When mvn test in my environment, it reported the following.
{noformat}
Failed tests:
TestTableMapping.testClearingCachedMappings:144 expected:/[rack1] but
was:/[default-rack]
TestTableMapping.testTableCaching:79 expected:/[rack1] but
was:/[default-rack]
TestTableMapping.testResolve:56 expected:/[rack1] but was:/[default-rack]
{noformat}
It's caused by the good resolving for the 'bad test' hostname 'a.b.c' as
follows.
{noformat}
[drankye@zkdesk hadoop-common-project]$ ping a.b.c
PING a.b.c (220.250.64.228) 56(84) bytes of data.
{noformat}
I understand it may happen in just my local environment, and document this just
in case others also meet this. We may use even worse hostname than 'a.b.c' to
avoid such situation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11535) TableMapping related tests failed due to 'correct' resolving for test hostname
[
https://issues.apache.org/jira/browse/HADOOP-11535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308682#comment-14308682
]
Kai Zheng commented on HADOOP-11535:
Hmmm, looks like it's not the case. Actually whatever bad or invalid hostname
is used, it can resolve to the fixed IP address in my network, as follows.
{noformat}
nslookup abc.invalid.test
Server: 10.248.2.5
Address:10.248.2.5#53
Non-authoritative answer:
Name: abc.invalid.test
Address: 220.250.64.228
{noformat}
So I thought in the tests we should not use hostname, instead use IP address,
to avoid such bad situation at all. I'm not sure if it's the typical or not,
but Java as nslookup does return non-authoritative answer when resolving a
domain name. Using domain name isn't necessary for the tests.
[~ste...@apache.org], do you agree we can change the test, not using domain
name like 'a.b.c', instead using IP address like '1.2.3.4' ? Thanks.
TableMapping related tests failed due to 'correct' resolving for test hostname
--
Key: HADOOP-11535
URL: https://issues.apache.org/jira/browse/HADOOP-11535
Project: Hadoop Common
Issue Type: Test
Reporter: Kai Zheng
Assignee: Kai Zheng
Priority: Minor
When mvn test in my environment, it reported the following.
{noformat}
Failed tests:
TestTableMapping.testClearingCachedMappings:144 expected:/[rack1] but
was:/[default-rack]
TestTableMapping.testTableCaching:79 expected:/[rack1] but
was:/[default-rack]
TestTableMapping.testResolve:56 expected:/[rack1] but
was:/[default-rack]
{noformat}
It's caused by the good resolving for the 'bad test' hostname 'a.b.c' as
follows.
{noformat}
[drankye@zkdesk hadoop-common-project]$ ping a.b.c
PING a.b.c (220.250.64.228) 56(84) bytes of data.
{noformat}
I understand it may happen in just my local environment, and document this
just in case others also meet this. We may use even worse hostname than
'a.b.c' to avoid such situation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11535) TableMapping related tests failed due to 'correct' resolving for test hostname
[
https://issues.apache.org/jira/browse/HADOOP-11535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng updated HADOOP-11535:
---
Attachment: HADOOP-11535-v1.patch
Uploaded a patch, using IP directly.
TableMapping related tests failed due to 'correct' resolving for test hostname
--
Key: HADOOP-11535
URL: https://issues.apache.org/jira/browse/HADOOP-11535
Project: Hadoop Common
Issue Type: Test
Reporter: Kai Zheng
Assignee: Kai Zheng
Priority: Minor
Attachments: HADOOP-11535-v1.patch
When mvn test in my environment, it reported the following.
{noformat}
Failed tests:
TestTableMapping.testClearingCachedMappings:144 expected:/[rack1] but
was:/[default-rack]
TestTableMapping.testTableCaching:79 expected:/[rack1] but
was:/[default-rack]
TestTableMapping.testResolve:56 expected:/[rack1] but
was:/[default-rack]
{noformat}
It's caused by the good resolving for the 'bad test' hostname 'a.b.c' as
follows.
{noformat}
[drankye@zkdesk hadoop-common-project]$ ping a.b.c
PING a.b.c (220.250.64.228) 56(84) bytes of data.
{noformat}
I understand it may happen in just my local environment, and document this
just in case others also meet this. We may use even worse hostname than
'a.b.c' to avoid such situation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11535) TableMapping related tests failed due to 'correct' resolving for test hostname
[
https://issues.apache.org/jira/browse/HADOOP-11535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng updated HADOOP-11535:
---
Status: Patch Available (was: Open)
I submit the patch any way in case it's liked.
TableMapping related tests failed due to 'correct' resolving for test hostname
--
Key: HADOOP-11535
URL: https://issues.apache.org/jira/browse/HADOOP-11535
Project: Hadoop Common
Issue Type: Test
Reporter: Kai Zheng
Assignee: Kai Zheng
Priority: Minor
Attachments: HADOOP-11535-v1.patch
When mvn test in my environment, it reported the following.
{noformat}
Failed tests:
TestTableMapping.testClearingCachedMappings:144 expected:/[rack1] but
was:/[default-rack]
TestTableMapping.testTableCaching:79 expected:/[rack1] but
was:/[default-rack]
TestTableMapping.testResolve:56 expected:/[rack1] but
was:/[default-rack]
{noformat}
It's caused by the good resolving for the 'bad test' hostname 'a.b.c' as
follows.
{noformat}
[drankye@zkdesk hadoop-common-project]$ ping a.b.c
PING a.b.c (220.250.64.228) 56(84) bytes of data.
{noformat}
I understand it may happen in just my local environment, and document this
just in case others also meet this. We may use even worse hostname than
'a.b.c' to avoid such situation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11541) Raw XOR coder
[ https://issues.apache.org/jira/browse/HADOOP-11541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310635#comment-14310635 ] Kai Zheng commented on HADOOP-11541: bq.you need a +1 from someone else (not yourself). OK. It clarifies the process for me, thanks. I thought it has been relaxed somehow for the branch but not sure how I got the impression. Thanks for your +1. I'll get it in so am able to sort out other patches. Raw XOR coder - Key: HADOOP-11541 URL: https://issues.apache.org/jira/browse/HADOOP-11541 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Fix For: HDFS-EC Attachments: HADOOP-11541-v1.patch, HADOOP-11541-v2.patch This will implement XOR codes by porting the codes from HDFS-RAID. The coder in the algorithm is needed by some high level codecs like LRC. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11541) Raw XOR coder
[ https://issues.apache.org/jira/browse/HADOOP-11541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310452#comment-14310452 ] Kai Zheng commented on HADOOP-11541: I think the updated patch addressed above review comments and I give it +1. Will commit it today if no more feedback coming. Raw XOR coder - Key: HADOOP-11541 URL: https://issues.apache.org/jira/browse/HADOOP-11541 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Fix For: HDFS-EC Attachments: HADOOP-11541-v1.patch, HADOOP-11541-v2.patch This will implement XOR codes by porting the codes from HDFS-RAID. The coder in the algorithm is needed by some high level codecs like LRC. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11550) Ehance ErasureCoder API to support multiple encoding/decoding steps
[ https://issues.apache.org/jira/browse/HADOOP-11550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11550: --- Summary: Ehance ErasureCoder API to support multiple encoding/decoding steps (was: Ehance ErasureCoder API to support complex cases requiring multiple steps for encoding/decoding of a block group) Ehance ErasureCoder API to support multiple encoding/decoding steps --- Key: HADOOP-11550 URL: https://issues.apache.org/jira/browse/HADOOP-11550 Project: Hadoop Common Issue Type: Improvement Reporter: Kai Zheng Assignee: Kai Zheng In HDFS-7662, *ErasureCoder* API is defined to support encoding/decoding of a block group considering the most simple but often case that only one coding step is involved. This is to ehance the API to support complex cases requiring multiple steps for encoding/decoding of a block group. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11542) Raw Reed-Solomon coder in pure Java
[
https://issues.apache.org/jira/browse/HADOOP-11542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310473#comment-14310473
]
Kai Zheng commented on HADOOP-11542:
Hi [~zhz],
Thanks for your helpful review to move a little bit faster.
bq.I don't think we need to put Java in the class name
I want to distinguish it from native coders. We'll also have ISA one for RS
code scheme. How about {{JRSRawEncoder}} ? Note as it's in pure Java its
efficiency is very low so I thought it would be good not to have users think
it's the default and right good RS implementation.
bq.I think we should remove test usage. They'll be used all-around if native
coders are not available.
OK, agree.
Regarding {{GaloisField}} and related, it's almost the same from HDFS-RAID. I
did add some functions to support ByteBuffer. As you suggested, I will compare
and give comments for the updated parts.
Will update the patch when HADOOP-11541 is in.
Raw Reed-Solomon coder in pure Java
---
Key: HADOOP-11542
URL: https://issues.apache.org/jira/browse/HADOOP-11542
Project: Hadoop Common
Issue Type: Sub-task
Affects Versions: HDFS-EC
Reporter: Kai Zheng
Assignee: Kai Zheng
Attachments: HADOOP-11542-v1.patch, HADOOP-11542-v2.patch
This will implement RS coder by porting existing codes in HDFS-RAID in the
new codec and coder framework, which could be useful in case native support
isn't available or convenient in some environments or platforms.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11535) TableMapping related tests failed due to 'successful' resolving of invalid test hostname
[
https://issues.apache.org/jira/browse/HADOOP-11535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310482#comment-14310482
]
Kai Zheng commented on HADOOP-11535:
Thanks Steve for committing the patch.
TableMapping related tests failed due to 'successful' resolving of invalid
test hostname
Key: HADOOP-11535
URL: https://issues.apache.org/jira/browse/HADOOP-11535
Project: Hadoop Common
Issue Type: Test
Reporter: Kai Zheng
Assignee: Kai Zheng
Priority: Minor
Attachments: HADOOP-11535-v1.patch
When mvn test in my environment, it reported the following.
{noformat}
Failed tests:
TestTableMapping.testClearingCachedMappings:144 expected:/[rack1] but
was:/[default-rack]
TestTableMapping.testTableCaching:79 expected:/[rack1] but
was:/[default-rack]
TestTableMapping.testResolve:56 expected:/[rack1] but
was:/[default-rack]
{noformat}
It's caused by the good resolving for the 'bad test' hostname 'a.b.c' as
follows.
{noformat}
[drankye@zkdesk hadoop-common-project]$ ping a.b.c
PING a.b.c (220.250.64.228) 56(84) bytes of data.
{noformat}
I understand it may happen in just my local environment, and document this
just in case others also meet this. We may use even worse hostname than
'a.b.c' to avoid such situation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11541) Raw XOR coder
[ https://issues.apache.org/jira/browse/HADOOP-11541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310640#comment-14310640 ] Kai Zheng commented on HADOOP-11541: commit e031adecc5cb7414395c6708e8d6fb12dcf6f8d8 Author: Kai Zheng dran...@apache.org Date: Sun Feb 8 01:40:27 2015 +0800 HADOOP-11541. Raw XOR coder Raw XOR coder - Key: HADOOP-11541 URL: https://issues.apache.org/jira/browse/HADOOP-11541 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Fix For: HDFS-EC Attachments: HADOOP-11541-v1.patch, HADOOP-11541-v2.patch This will implement XOR codes by porting the codes from HDFS-RAID. The coder in the algorithm is needed by some high level codecs like LRC. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11541) Raw XOR coder
[ https://issues.apache.org/jira/browse/HADOOP-11541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11541: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Raw XOR coder - Key: HADOOP-11541 URL: https://issues.apache.org/jira/browse/HADOOP-11541 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Fix For: HDFS-EC Attachments: HADOOP-11541-v1.patch, HADOOP-11541-v2.patch This will implement XOR codes by porting the codes from HDFS-RAID. The coder in the algorithm is needed by some high level codecs like LRC. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11541) Raw XOR coder
[ https://issues.apache.org/jira/browse/HADOOP-11541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11541: --- Attachment: (was: HDFS-7417-v3.patch) Raw XOR coder - Key: HADOOP-11541 URL: https://issues.apache.org/jira/browse/HADOOP-11541 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Fix For: HDFS-EC This will implement XOR codes by porting the codes from HDFS-RAID. The coder in the algorithm is needed by some high level codecs like LRC. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11541) Raw XOR coder
[ https://issues.apache.org/jira/browse/HADOOP-11541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11541: --- Attachment: (was: HDFS-7417-v1.patch) Raw XOR coder - Key: HADOOP-11541 URL: https://issues.apache.org/jira/browse/HADOOP-11541 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Fix For: HDFS-EC This will implement XOR codes by porting the codes from HDFS-RAID. The coder in the algorithm is needed by some high level codecs like LRC. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11542) Raw Reed-Solomon coder in pure Java
[ https://issues.apache.org/jira/browse/HADOOP-11542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11542: --- Attachment: HADOOP-11542-v2.patch Updated the patch cleaning up unrelevant codes. Raw Reed-Solomon coder in pure Java --- Key: HADOOP-11542 URL: https://issues.apache.org/jira/browse/HADOOP-11542 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HADOOP-11542-v1.patch, HADOOP-11542-v2.patch This will implement RS coder by porting existing codes in HDFS-RAID in the new codec and coder framework, which could be useful in case native support isn't available or convenient in some environments or platforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11541) Raw XOR coder
[ https://issues.apache.org/jira/browse/HADOOP-11541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11541: --- Attachment: HADOOP-11541-v1.patch Uploaded the patch with added and passed tests. Raw XOR coder - Key: HADOOP-11541 URL: https://issues.apache.org/jira/browse/HADOOP-11541 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Fix For: HDFS-EC Attachments: HADOOP-11541-v1.patch This will implement XOR codes by porting the codes from HDFS-RAID. The coder in the algorithm is needed by some high level codecs like LRC. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11541) Raw XOR coder
[ https://issues.apache.org/jira/browse/HADOOP-11541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11541: --- Attachment: (was: HDFS-7417-v2.patch) Raw XOR coder - Key: HADOOP-11541 URL: https://issues.apache.org/jira/browse/HADOOP-11541 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Fix For: HDFS-EC This will implement XOR codes by porting the codes from HDFS-RAID. The coder in the algorithm is needed by some high level codecs like LRC. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11550) Ehance ErasureCoder API to support complex cases requiring multiple steps for encoding/decoding of a block group
Kai Zheng created HADOOP-11550: -- Summary: Ehance ErasureCoder API to support complex cases requiring multiple steps for encoding/decoding of a block group Key: HADOOP-11550 URL: https://issues.apache.org/jira/browse/HADOOP-11550 Project: Hadoop Common Issue Type: Improvement Reporter: Kai Zheng Assignee: Kai Zheng In HDFS-7662, *ErasureCoder* API is defined to support encoding/decoding of a block group considering the most simple but often case that only one coding step is involved. This is to ehance the API to support complex cases requiring multiple steps for encoding/decoding of a block group. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11541) Raw XOR coder
[
https://issues.apache.org/jira/browse/HADOOP-11541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308438#comment-14308438
]
Kai Zheng commented on HADOOP-11541:
Hi [~zhz],
Thanks for your comments.
bq.What's the downside of converting to a unified format first?
I'm thinking it's doing the real calculating and performance concerned, so I
don't have much trick here. When we have more experience for implementing the
following more raw coders, I guess we can have some idea about how to unifying
them.
bq.TestCoderBase needs better documentation
Agree, will do.
bq.What does {{getErasedIndexesForDecoding}} do? What is {{erasedDataIndexes}}
for?
To test recovering, we intentively erase some data units specified by
{{erasedDataIndexes}}. {{getErasedIndexesForDecoding}} is to adjust the indexes
to pass to raw coder to invoke the decode() function. Will have more comments
for them.
bq.{{testCoding}} can be in {{TestCoderBase}}
{{TestCoderBase}} is not only for tests of raw coders, but also will be used
for tests of block coders, which can be seen in the patch for HDFS-7662.
Therefore I don't put raw coder specific codes in the base class.
Raw XOR coder
-
Key: HADOOP-11541
URL: https://issues.apache.org/jira/browse/HADOOP-11541
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
Fix For: HDFS-EC
Attachments: HADOOP-11541-v1.patch
This will implement XOR codes by porting the codes from HDFS-RAID. The coder
in the algorithm is needed by some high level codecs like LRC.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11535) TableMapping related tests failed due to 'correct' resolving for test hostname
[
https://issues.apache.org/jira/browse/HADOOP-11535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308495#comment-14308495
]
Kai Zheng commented on HADOOP-11535:
Thanks Steve for the pointing. I checked it out and see the following domain
names are reserved.
{noformat}
.test
.example
.invalid
.localhost
example.com
example.net
example.org
{noformat}
So according to this, we should check the tests and replace the domain names
with the reserved ones, to make sure they won't resolve. Will provide a patch
accordingly.
TableMapping related tests failed due to 'correct' resolving for test hostname
--
Key: HADOOP-11535
URL: https://issues.apache.org/jira/browse/HADOOP-11535
Project: Hadoop Common
Issue Type: Test
Reporter: Kai Zheng
Assignee: Kai Zheng
Priority: Minor
When mvn test in my environment, it reported the following.
{noformat}
Failed tests:
TestTableMapping.testClearingCachedMappings:144 expected:/[rack1] but
was:/[default-rack]
TestTableMapping.testTableCaching:79 expected:/[rack1] but
was:/[default-rack]
TestTableMapping.testResolve:56 expected:/[rack1] but
was:/[default-rack]
{noformat}
It's caused by the good resolving for the 'bad test' hostname 'a.b.c' as
follows.
{noformat}
[drankye@zkdesk hadoop-common-project]$ ping a.b.c
PING a.b.c (220.250.64.228) 56(84) bytes of data.
{noformat}
I understand it may happen in just my local environment, and document this
just in case others also meet this. We may use even worse hostname than
'a.b.c' to avoid such situation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11563) Add the missed entry for CHANGES.txt
Kai Zheng created HADOOP-11563: -- Summary: Add the missed entry for CHANGES.txt Key: HADOOP-11563 URL: https://issues.apache.org/jira/browse/HADOOP-11563 Project: Hadoop Common Issue Type: Bug Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Priority: Trivial Fix For: HDFS-EC When committing HADOOP-11541, it forgot to update the hadoop-common/CHANGES-HDFS-EC-7285.txt file. This is to add the missed entry. Thanks [~hitliuyi] for pointing this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11541) Raw XOR coder
[ https://issues.apache.org/jira/browse/HADOOP-11541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311320#comment-14311320 ] Kai Zheng commented on HADOOP-11541: I provided a minor patch to save it in HADOOP-11563. [~hitliuyi] can you review it ? Thanks. Raw XOR coder - Key: HADOOP-11541 URL: https://issues.apache.org/jira/browse/HADOOP-11541 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Fix For: HDFS-EC Attachments: HADOOP-11541-v1.patch, HADOOP-11541-v2.patch This will implement XOR codes by porting the codes from HDFS-RAID. The coder in the algorithm is needed by some high level codecs like LRC. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11541) Raw XOR coder
[
https://issues.apache.org/jira/browse/HADOOP-11541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311299#comment-14311299
]
Kai Zheng commented on HADOOP-11541:
Thanks [~hitliuyi] for looking at this.
bq.You should also write the contribution name
Yes I should have followed the convention, though I did notice some exceptions.
bq.you should change the corresponding CHANGES.txt
Yes we have the {{CHANGES.txt}}, sorry I forgot updating it. Do we need to fire
a JIRA to save this ?
Raw XOR coder
-
Key: HADOOP-11541
URL: https://issues.apache.org/jira/browse/HADOOP-11541
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
Fix For: HDFS-EC
Attachments: HADOOP-11541-v1.patch, HADOOP-11541-v2.patch
This will implement XOR codes by porting the codes from HDFS-RAID. The coder
in the algorithm is needed by some high level codecs like LRC.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11563) Add the missed entry for CHANGES.txt
[ https://issues.apache.org/jira/browse/HADOOP-11563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11563: --- Attachment: HADOOP-11563-v1.patch Uploaded a patch adding the entry. Add the missed entry for CHANGES.txt Key: HADOOP-11563 URL: https://issues.apache.org/jira/browse/HADOOP-11563 Project: Hadoop Common Issue Type: Bug Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Priority: Trivial Fix For: HDFS-EC Attachments: HADOOP-11563-v1.patch When committing HADOOP-11541, it forgot to update the hadoop-common/CHANGES-HDFS-EC-7285.txt file. This is to add the missed entry. Thanks [~hitliuyi] for pointing this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11542) Raw Reed-Solomon coder in pure Java
[ https://issues.apache.org/jira/browse/HADOOP-11542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11542: --- Attachment: HADOOP-11542-v3.patch Updated the patch according to above review and discussion. Raw Reed-Solomon coder in pure Java --- Key: HADOOP-11542 URL: https://issues.apache.org/jira/browse/HADOOP-11542 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HADOOP-11542-v1.patch, HADOOP-11542-v2.patch, HADOOP-11542-v3.patch This will implement RS coder by porting existing codes in HDFS-RAID in the new codec and coder framework, which could be useful in case native support isn't available or convenient in some environments or platforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11563) Add the missed entry for CHANGES.txt
[ https://issues.apache.org/jira/browse/HADOOP-11563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11563: --- Status: Patch Available (was: Open) Add the missed entry for CHANGES.txt Key: HADOOP-11563 URL: https://issues.apache.org/jira/browse/HADOOP-11563 Project: Hadoop Common Issue Type: Bug Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Priority: Trivial Fix For: HDFS-EC Attachments: HADOOP-11563-v1.patch When committing HADOOP-11541, it forgot to update the hadoop-common/CHANGES-HDFS-EC-7285.txt file. This is to add the missed entry. Thanks [~hitliuyi] for pointing this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11563) Add the missed entry for CHANGES.txt
[ https://issues.apache.org/jira/browse/HADOOP-11563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311687#comment-14311687 ] Kai Zheng commented on HADOOP-11563: Hmm, I'm not sure. It's hard to say CHANGES.txt is code or not. If you'd like to search or just access the following URL, you can get many issues for such things. [CHANGES.txt related issues | https://issues.apache.org/jira/browse/HADOOP-3266?jql=project%20in%20%28HADOOP%2C%20HDFS%29%20AND%20text%20~%20CHANGES.txt] bq.You can change the CHANGES.txt and commit log directly through git. I thought it also works for me. I will get it done directly. Thanks. Add the missed entry for CHANGES.txt Key: HADOOP-11563 URL: https://issues.apache.org/jira/browse/HADOOP-11563 Project: Hadoop Common Issue Type: Bug Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Priority: Trivial Fix For: HDFS-EC Attachments: HADOOP-11563-v1.patch When committing HADOOP-11541, it forgot to update the hadoop-common/CHANGES-HDFS-EC-7285.txt file. This is to add the missed entry. Thanks [~hitliuyi] for pointing this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11541) Raw XOR coder
[ https://issues.apache.org/jira/browse/HADOOP-11541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311902#comment-14311902 ] Kai Zheng commented on HADOOP-11541: Yes, I've done so already that way. Thanks Uma, and Yi. Raw XOR coder - Key: HADOOP-11541 URL: https://issues.apache.org/jira/browse/HADOOP-11541 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Fix For: HDFS-EC Attachments: HADOOP-11541-v1.patch, HADOOP-11541-v2.patch This will implement XOR codes by porting the codes from HDFS-RAID. The coder in the algorithm is needed by some high level codecs like LRC. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-9477) posixGroups support for LDAP groups mapping service
[ https://issues.apache.org/jira/browse/HADOOP-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14317436#comment-14317436 ] Kai Zheng commented on HADOOP-9477: --- [~yzhangal], Glad to know you're interested in this. Yes, we can move forward. Thanks ! posixGroups support for LDAP groups mapping service --- Key: HADOOP-9477 URL: https://issues.apache.org/jira/browse/HADOOP-9477 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Kai Zheng Assignee: Kai Zheng Fix For: 2.7.0 Attachments: HADOOP-9477.003.patch, HADOOP-9477.patch, HADOOP-9477.patch Original Estimate: 168h Remaining Estimate: 168h It would be nice to support posixGroups for LdapGroupsMapping service. Below is from current description for the provider: hadoop.security.group.mapping.ldap.search.filter.group: An additional filter to use when searching for LDAP groups. This should be changed when resolving groups against a non-Active Directory installation. posixGroups are currently not a supported group class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11542) Raw Reed-Solomon coder in pure Java
[ https://issues.apache.org/jira/browse/HADOOP-11542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11542: --- Attachment: HADOOP-11542-v5.patch Updated the patch according to above discussion. Raw Reed-Solomon coder in pure Java --- Key: HADOOP-11542 URL: https://issues.apache.org/jira/browse/HADOOP-11542 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HADOOP-11542-v1.patch, HADOOP-11542-v2.patch, HADOOP-11542-v3.patch, HADOOP-11542-v4.patch, HADOOP-11542-v5.patch, HDFS-RAID-GaloisField.java This will implement RS coder by porting existing codes in HDFS-RAID in the new codec and coder framework, which could be useful in case native support isn't available or convenient in some environments or platforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-9477) posixGroups support for LDAP groups mapping service
[ https://issues.apache.org/jira/browse/HADOOP-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14317445#comment-14317445 ] Kai Zheng commented on HADOOP-9477: --- Hi [~clamb] Thanks for your update and feedback. It's great and no problem at all. My colleague [~sundapeng] worked on this, would you mind if he continues to help on this with your work ? Thanks ! posixGroups support for LDAP groups mapping service --- Key: HADOOP-9477 URL: https://issues.apache.org/jira/browse/HADOOP-9477 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Kai Zheng Assignee: Kai Zheng Fix For: 2.7.0 Attachments: HADOOP-9477.003.patch, HADOOP-9477.patch, HADOOP-9477.patch Original Estimate: 168h Remaining Estimate: 168h It would be nice to support posixGroups for LdapGroupsMapping service. Below is from current description for the provider: hadoop.security.group.mapping.ldap.search.filter.group: An additional filter to use when searching for LDAP groups. This should be changed when resolving groups against a non-Active Directory installation. posixGroups are currently not a supported group class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11542) Raw Reed-Solomon coder in pure Java
[
https://issues.apache.org/jira/browse/HADOOP-11542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14317514#comment-14317514
]
Kai Zheng commented on HADOOP-11542:
Hi [~zhz] for your good catches. I will update patch to resolve 1 2.
bq.Do we need to apply the mask before casting to byte?
I thought it applied the mask before the casting, as necessarily, and we don't
need extra masking in the statement. Please correct me if I'm not getting you.
Thanks.
bq.we can also add a direct test for the added bulk methods
For performance consideration we currently only use the bulk version. For the
single {{int}} level versions, we don't use or haven't got the chance to use
yet. Tests for them are good to have, though.
Raw Reed-Solomon coder in pure Java
---
Key: HADOOP-11542
URL: https://issues.apache.org/jira/browse/HADOOP-11542
Project: Hadoop Common
Issue Type: Sub-task
Affects Versions: HDFS-EC
Reporter: Kai Zheng
Assignee: Kai Zheng
Attachments: HADOOP-11542-v1.patch, HADOOP-11542-v2.patch,
HADOOP-11542-v3.patch, HADOOP-11542-v4.patch, HDFS-RAID-GaloisField.java
This will implement RS coder by porting existing codes in HDFS-RAID in the
new codec and coder framework, which could be useful in case native support
isn't available or convenient in some environments or platforms.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11514) Raw Erasure Coder API for concrete encoding and decoding
[ https://issues.apache.org/jira/browse/HADOOP-11514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294328#comment-14294328 ] Kai Zheng commented on HADOOP-11514: Zhe and Tsz, as I'm traveling today and not convenient to hit my dev environment, I'm not able to update the patch changing the package name. Maybe I can get it done in a follow up JIRA? Raw Erasure Coder API for concrete encoding and decoding Key: HADOOP-11514 URL: https://issues.apache.org/jira/browse/HADOOP-11514 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Fix For: HDFS-EC Attachments: HDFS-7353-v1.patch, HDFS-7353-v2.patch, HDFS-7353-v3.patch, HDFS-7353-v4.patch, HDFS-7353-v5.patch, HDFS-7353-v6.patch, HDFS-7353-v7.patch This is to abstract and define raw erasure coder API across different codes algorithms like RS, XOR and etc. Such API can be implemented by utilizing various library support, such as Intel ISA library and Jerasure library. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11514) Raw Erasure Coder API for concrete encoding and decoding
[ https://issues.apache.org/jira/browse/HADOOP-11514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14296325#comment-14296325 ] Kai Zheng commented on HADOOP-11514: Hi [~umamaheswararao], Thanks for looking at this. bq.Seems like we may need to update this interface comments here. RawErasureCoder is not really doing encoding/decoding by its interfaces declared. This sounds to some helper interface for real encode/decoder interfaces. It's a good thought. Yes itself doesn't do any real work but is just a helper, on the other hand, it also serves as a good concept for our discussion since it's not convenient to mention encoder or decoder in many general aspects. You're right I need to update the comments to make it accurate. Thanks ! Raw Erasure Coder API for concrete encoding and decoding Key: HADOOP-11514 URL: https://issues.apache.org/jira/browse/HADOOP-11514 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HDFS-7353-v1.patch, HDFS-7353-v2.patch, HDFS-7353-v3.patch, HDFS-7353-v4.patch, HDFS-7353-v5.patch, HDFS-7353-v6.patch, HDFS-7353-v7.patch This is to abstract and define raw erasure coder API across different codes algorithms like RS, XOR and etc. Such API can be implemented by utilizing various library support, such as Intel ISA library and Jerasure library. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11514) Raw Erasure Coder API for concrete encoding and decoding
[ https://issues.apache.org/jira/browse/HADOOP-11514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14296289#comment-14296289 ] Kai Zheng commented on HADOOP-11514: Hi [~zhz], bq. dataSize, paritySize, and chunkSize apply to all descendants of this interface (RawErasureCoder). Shouldn't they become member variables? Sorry for missing this. *RawErasureCoder* is an interface which has already contains the related methods. The *AbstractRawErasureCoder* class that implements the interface does has the mentioned variables. So would you clarify a bit what you meant? Thanks. Raw Erasure Coder API for concrete encoding and decoding Key: HADOOP-11514 URL: https://issues.apache.org/jira/browse/HADOOP-11514 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HDFS-7353-v1.patch, HDFS-7353-v2.patch, HDFS-7353-v3.patch, HDFS-7353-v4.patch, HDFS-7353-v5.patch, HDFS-7353-v6.patch, HDFS-7353-v7.patch This is to abstract and define raw erasure coder API across different codes algorithms like RS, XOR and etc. Such API can be implemented by utilizing various library support, such as Intel ISA library and Jerasure library. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11514) Raw Erasure Coder API for concrete encoding and decoding
[ https://issues.apache.org/jira/browse/HADOOP-11514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14296323#comment-14296323 ] Kai Zheng commented on HADOOP-11514: Hi [~anu], Thanks for you question. bq. how does it know if the ECBlock (or ECChunk) is valid ? In high level constructs like ECManager, it will check EC blocks are healthy or not (like lost at all due to DataNode failure or corrupt by block checksum detected), if some block is found in question then the erasure codec framework will play in the work and this *RawErasureCoder* will be involved. The bad block will be marked as isErasred being true and the codec/coder will know which block in a ECGroup or BlockGroup is to be recovered, and then ECChunks will be extracted for decoding. Raw Erasure Coder API for concrete encoding and decoding Key: HADOOP-11514 URL: https://issues.apache.org/jira/browse/HADOOP-11514 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HDFS-7353-v1.patch, HDFS-7353-v2.patch, HDFS-7353-v3.patch, HDFS-7353-v4.patch, HDFS-7353-v5.patch, HDFS-7353-v6.patch, HDFS-7353-v7.patch This is to abstract and define raw erasure coder API across different codes algorithms like RS, XOR and etc. Such API can be implemented by utilizing various library support, such as Intel ISA library and Jerasure library. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11514) Raw Erasure Coder API for concrete encoding and decoding
[ https://issues.apache.org/jira/browse/HADOOP-11514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14296370#comment-14296370 ] Kai Zheng commented on HADOOP-11514: Hi [~vinayrpet], Thanks for your view. bq.In fact whole ECChunk class itself is unnecessary in Common, as this is specific to 'striped' design of the erasure encoding, but current Jira targets concrete APIs only. ECChunk can be converted 'ByteBuffer' before passing to encode()/decode() of RawEncoder/RawDecoder in higher levels. As covered in HDFS-7662, the higher level construct *ErasureCoder* will mainly use ECChunk type to call the raw coder, and how to converting ECChunk to bytes or ByteBuffer is better to be located in the central place as I put in the base. That's why I thought *RawErasureCoder* better support it in the API level. We're considering to place the whole erasure codec framework in Common side, including the *ErasureCodec* API to be refined in HDFS-7699 and *ErasureCoder* API defined in HDFS-7662. I understand there will be some questions as Zhe has already raised in HDFS-7662, I guess we can discuss this there. To ease the discussion and commit of this, I would remove the ECChunk related codes from the updated patch first and then put them back in the patch for HDFS-7662. Hope this works. Raw Erasure Coder API for concrete encoding and decoding Key: HADOOP-11514 URL: https://issues.apache.org/jira/browse/HADOOP-11514 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HDFS-7353-v1.patch, HDFS-7353-v2.patch, HDFS-7353-v3.patch, HDFS-7353-v4.patch, HDFS-7353-v5.patch, HDFS-7353-v6.patch, HDFS-7353-v7.patch This is to abstract and define raw erasure coder API across different codes algorithms like RS, XOR and etc. Such API can be implemented by utilizing various library support, such as Intel ISA library and Jerasure library. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11514) Raw Erasure Coder API for concrete encoding and decoding
[ https://issues.apache.org/jira/browse/HADOOP-11514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14296380#comment-14296380 ] Kai Zheng commented on HADOOP-11514: Sorry for the point below. bq.I would remove the ECChunk related codes from the updated patch first and then put them back in the patch for HDFS-7662. Please let me take back what I said as above. As the raw coders that implements this API, like the one HDFS-7417 and HDFS-7418, all depend on this and require the ECChunk related codes, I wish they could be committed together here. Raw Erasure Coder API for concrete encoding and decoding Key: HADOOP-11514 URL: https://issues.apache.org/jira/browse/HADOOP-11514 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HDFS-7353-v1.patch, HDFS-7353-v2.patch, HDFS-7353-v3.patch, HDFS-7353-v4.patch, HDFS-7353-v5.patch, HDFS-7353-v6.patch, HDFS-7353-v7.patch This is to abstract and define raw erasure coder API across different codes algorithms like RS, XOR and etc. Such API can be implemented by utilizing various library support, such as Intel ISA library and Jerasure library. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11514) Raw Erasure Coder API for concrete encoding and decoding
[ https://issues.apache.org/jira/browse/HADOOP-11514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14296381#comment-14296381 ] Kai Zheng commented on HADOOP-11514: bq.this is specific to 'striped' design of the erasure encoding This might not be true. I see no reason it's specific to stripping instead I wish it can be used in all the EC related cases we're going to come up. In non-stripping EC or block level EC, a block can/should be divided into chunks and for each chunk to perform the encoding/decoding using the raw coder here. Raw Erasure Coder API for concrete encoding and decoding Key: HADOOP-11514 URL: https://issues.apache.org/jira/browse/HADOOP-11514 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HDFS-7353-v1.patch, HDFS-7353-v2.patch, HDFS-7353-v3.patch, HDFS-7353-v4.patch, HDFS-7353-v5.patch, HDFS-7353-v6.patch, HDFS-7353-v7.patch This is to abstract and define raw erasure coder API across different codes algorithms like RS, XOR and etc. Such API can be implemented by utilizing various library support, such as Intel ISA library and Jerasure library. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11514) Raw Erasure Coder API for concrete encoding and decoding
[
https://issues.apache.org/jira/browse/HADOOP-11514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14296522#comment-14296522
]
Kai Zheng commented on HADOOP-11514:
Hi [~vinayrpet],
Thanks a lot for your time and the summary. I have uploaded a patch with new
name resolving all the new comments hopefully as follows:
1. Rename the package name, using erasurecode;
2. Updated some comments;
3. Added comment to the codes you pasted above like you commented;
4. Added a new method in *RawErasureCoder* to allow callers to know the chunk
buffer allocation preference as below.
{code}
public boolean preferNativeBuffer()
{code}
Would you or anybody else review it one more time ? Thanks.
Raw Erasure Coder API for concrete encoding and decoding
Key: HADOOP-11514
URL: https://issues.apache.org/jira/browse/HADOOP-11514
Project: Hadoop Common
Issue Type: Sub-task
Affects Versions: HDFS-EC
Reporter: Kai Zheng
Assignee: Kai Zheng
Attachments: HADOOP-11514-v8.patch, HDFS-7353-v1.patch,
HDFS-7353-v2.patch, HDFS-7353-v3.patch, HDFS-7353-v4.patch,
HDFS-7353-v5.patch, HDFS-7353-v6.patch, HDFS-7353-v7.patch
This is to abstract and define raw erasure coder API across different codes
algorithms like RS, XOR and etc. Such API can be implemented by utilizing
various library support, such as Intel ISA library and Jerasure library.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11514) Raw Erasure Coder API for concrete encoding and decoding
[ https://issues.apache.org/jira/browse/HADOOP-11514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11514: --- Attachment: HADOOP-11514-v8.patch Raw Erasure Coder API for concrete encoding and decoding Key: HADOOP-11514 URL: https://issues.apache.org/jira/browse/HADOOP-11514 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HADOOP-11514-v8.patch, HDFS-7353-v1.patch, HDFS-7353-v2.patch, HDFS-7353-v3.patch, HDFS-7353-v4.patch, HDFS-7353-v5.patch, HDFS-7353-v6.patch, HDFS-7353-v7.patch This is to abstract and define raw erasure coder API across different codes algorithms like RS, XOR and etc. Such API can be implemented by utilizing various library support, such as Intel ISA library and Jerasure library. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11534) Minor improvements for raw erasure coders
[ https://issues.apache.org/jira/browse/HADOOP-11534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11534: --- Attachment: HADOOP-11534-v1.patch Uploaded the patch. The updated codes were passed test in other issues. Minor improvements for raw erasure coders - Key: HADOOP-11534 URL: https://issues.apache.org/jira/browse/HADOOP-11534 Project: Hadoop Common Issue Type: Improvement Reporter: Kai Zheng Assignee: Kai Zheng Priority: Minor Attachments: HADOOP-11534-v1.patch For the raw erasure coder API codes introduced by HADOOP-11514, there're some minor improvements that were noticed and can be done separately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11534) Minor improvements for raw erasure coders
[ https://issues.apache.org/jira/browse/HADOOP-11534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11534: --- Status: Patch Available (was: Open) Minor improvements for raw erasure coders - Key: HADOOP-11534 URL: https://issues.apache.org/jira/browse/HADOOP-11534 Project: Hadoop Common Issue Type: Improvement Reporter: Kai Zheng Assignee: Kai Zheng Priority: Minor Attachments: HADOOP-11534-v1.patch For the raw erasure coder API codes introduced by HADOOP-11514, there're some minor improvements that were noticed and can be done separately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11534) Minor improvements for raw erasure coders
[ https://issues.apache.org/jira/browse/HADOOP-11534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11534: --- Attachment: HADOOP-11534-v2.patch Minor improvements for raw erasure coders - Key: HADOOP-11534 URL: https://issues.apache.org/jira/browse/HADOOP-11534 Project: Hadoop Common Issue Type: Improvement Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Priority: Minor Fix For: HDFS-EC Attachments: HADOOP-11534-v1.patch, HADOOP-11534-v2.patch For the raw erasure coder API codes introduced by HADOOP-11514, there're some minor improvements that were noticed and can be done separately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11534) Minor improvements for raw erasure coders
[ https://issues.apache.org/jira/browse/HADOOP-11534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14300948#comment-14300948 ] Kai Zheng commented on HADOOP-11534: Hi [~vinayrpet], Thanks for your quick review. I updated the patch according to your idea, thanks ! Minor improvements for raw erasure coders - Key: HADOOP-11534 URL: https://issues.apache.org/jira/browse/HADOOP-11534 Project: Hadoop Common Issue Type: Improvement Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Priority: Minor Fix For: HDFS-EC Attachments: HADOOP-11534-v1.patch, HADOOP-11534-v2.patch For the raw erasure coder API codes introduced by HADOOP-11514, there're some minor improvements that were noticed and can be done separately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11534) Minor improvements for raw erasure coders
Kai Zheng created HADOOP-11534: -- Summary: Minor improvements for raw erasure coders Key: HADOOP-11534 URL: https://issues.apache.org/jira/browse/HADOOP-11534 Project: Hadoop Common Issue Type: Improvement Reporter: Kai Zheng Assignee: Kai Zheng Priority: Minor For the raw erasure coder API codes introduced by HADOOP-11514, there're some minor improvements that were noticed and can be done separately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11534) Minor improvements for raw erasure coders
[ https://issues.apache.org/jira/browse/HADOOP-11534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11534: --- Affects Version/s: HDFS-EC Minor improvements for raw erasure coders - Key: HADOOP-11534 URL: https://issues.apache.org/jira/browse/HADOOP-11534 Project: Hadoop Common Issue Type: Improvement Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Priority: Minor Fix For: HDFS-EC Attachments: HADOOP-11534-v1.patch For the raw erasure coder API codes introduced by HADOOP-11514, there're some minor improvements that were noticed and can be done separately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11534) Minor improvements for raw erasure coders
[ https://issues.apache.org/jira/browse/HADOOP-11534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11534: --- Fix Version/s: HDFS-EC Minor improvements for raw erasure coders - Key: HADOOP-11534 URL: https://issues.apache.org/jira/browse/HADOOP-11534 Project: Hadoop Common Issue Type: Improvement Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Priority: Minor Fix For: HDFS-EC Attachments: HADOOP-11534-v1.patch For the raw erasure coder API codes introduced by HADOOP-11514, there're some minor improvements that were noticed and can be done separately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11534) Minor improvements for raw erasure coders
[
https://issues.apache.org/jira/browse/HADOOP-11534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14300955#comment-14300955
]
Kai Zheng commented on HADOOP-11534:
There must be something wrong with the building. The error said as below, which
isn't the case.
{code}
cp: cannot stat ‘/home/jenkins/buildSupport/lib/*’: No such file or directory
Checking patch
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECChunk.java...
error:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECChunk.java:
No such file or directory
Checking patch
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/AbstractRawErasureCoder.java...
error:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/AbstractRawErasureCoder.java:
No such file or directory
PATCH APPLICATION FAILED
{code}
Minor improvements for raw erasure coders
-
Key: HADOOP-11534
URL: https://issues.apache.org/jira/browse/HADOOP-11534
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: HDFS-EC
Reporter: Kai Zheng
Assignee: Kai Zheng
Priority: Minor
Fix For: HDFS-EC
Attachments: HADOOP-11534-v1.patch, HADOOP-11534-v2.patch
For the raw erasure coder API codes introduced by HADOOP-11514, there're some
minor improvements that were noticed and can be done separately.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11534) Minor improvements for raw erasure coders
[ https://issues.apache.org/jira/browse/HADOOP-11534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301042#comment-14301042 ] Kai Zheng commented on HADOOP-11534: Thanks [~vinayrpet] for the commit. Minor improvements for raw erasure coders - Key: HADOOP-11534 URL: https://issues.apache.org/jira/browse/HADOOP-11534 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Priority: Minor Fix For: HDFS-EC Attachments: HADOOP-11534-v1.patch, HADOOP-11534-v2.patch For the raw erasure coder API codes introduced by HADOOP-11514, there're some minor improvements that were noticed and can be done separately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11542) Raw Reed-Solomon coder in pure Java
[ https://issues.apache.org/jira/browse/HADOOP-11542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11542: --- Status: Open (was: Patch Available) To avoid the unnecessary Hadoop QA mess, not marking it as patch available, though it is. Raw Reed-Solomon coder in pure Java --- Key: HADOOP-11542 URL: https://issues.apache.org/jira/browse/HADOOP-11542 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HADOOP-11542-v1.patch, HADOOP-11542-v2.patch, HADOOP-11542-v3.patch, HADOOP-11542-v4.patch, HDFS-RAID-GaloisField.java This will implement RS coder by porting existing codes in HDFS-RAID in the new codec and coder framework, which could be useful in case native support isn't available or convenient in some environments or platforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11542) Raw Reed-Solomon coder in pure Java
[ https://issues.apache.org/jira/browse/HADOOP-11542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11542: --- Attachment: HADOOP-11542-v4.patch Updated the patch according to above review and discussion. Raw Reed-Solomon coder in pure Java --- Key: HADOOP-11542 URL: https://issues.apache.org/jira/browse/HADOOP-11542 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HADOOP-11542-v1.patch, HADOOP-11542-v2.patch, HADOOP-11542-v3.patch, HADOOP-11542-v4.patch, HDFS-RAID-GaloisField.java This will implement RS coder by porting existing codes in HDFS-RAID in the new codec and coder framework, which could be useful in case native support isn't available or convenient in some environments or platforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11542) Raw Reed-Solomon coder in pure Java
[ https://issues.apache.org/jira/browse/HADOOP-11542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14312991#comment-14312991 ] Kai Zheng commented on HADOOP-11542: Hi [~zhz], I checked it and am sure what I updated are adding some ByteBuffer version of functions, and adjusting some formats necessarily, and more comments. I just uploaded the original version for your reference. I agree it's good to check the arithmetic logic using tests, that's why I provided quite a few. When other RS coders like ISA one are coming, I will update the tests so that the decoding of the encoded data can be done in another coder, which will make us further ensure about its correctness. Raw Reed-Solomon coder in pure Java --- Key: HADOOP-11542 URL: https://issues.apache.org/jira/browse/HADOOP-11542 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HADOOP-11542-v1.patch, HADOOP-11542-v2.patch, HADOOP-11542-v3.patch, HDFS-RAID-GaloisField.java This will implement RS coder by porting existing codes in HDFS-RAID in the new codec and coder framework, which could be useful in case native support isn't available or convenient in some environments or platforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11542) Raw Reed-Solomon coder in pure Java
[ https://issues.apache.org/jira/browse/HADOOP-11542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11542: --- Attachment: HDFS-RAID-GaloisField.java For helping review, I provided the HDFS-RAID version of GaloisField.java. I could provide a diff version instead but looks like it's quite messy. I checked it out from branch-2.0.2-alpha branch. Raw Reed-Solomon coder in pure Java --- Key: HADOOP-11542 URL: https://issues.apache.org/jira/browse/HADOOP-11542 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HADOOP-11542-v1.patch, HADOOP-11542-v2.patch, HADOOP-11542-v3.patch, HDFS-RAID-GaloisField.java This will implement RS coder by porting existing codes in HDFS-RAID in the new codec and coder framework, which could be useful in case native support isn't available or convenient in some environments or platforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11542) Raw Reed-Solomon coder in pure Java
[ https://issues.apache.org/jira/browse/HADOOP-11542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14313000#comment-14313000 ] Kai Zheng commented on HADOOP-11542: The other comments are great as well, I will update the patch. Thanks a lot ! Raw Reed-Solomon coder in pure Java --- Key: HADOOP-11542 URL: https://issues.apache.org/jira/browse/HADOOP-11542 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HADOOP-11542-v1.patch, HADOOP-11542-v2.patch, HADOOP-11542-v3.patch, HDFS-RAID-GaloisField.java This will implement RS coder by porting existing codes in HDFS-RAID in the new codec and coder framework, which could be useful in case native support isn't available or convenient in some environments or platforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11542) Raw Reed-Solomon coder in pure Java
[
https://issues.apache.org/jira/browse/HADOOP-11542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14313053#comment-14313053
]
Kai Zheng commented on HADOOP-11542:
bq.testCoding actually only tested lost data chunks; do we plan to test lost
parity chunks as well?
Yes indeed. It's planned to have follow up a JIRA to enhance the tests for
parity chunks as well. Like {{erasedDataIndexes}}, {{erasedParityIndexes}} will
be added to specify which parity units are to be erased and recovered then.
bq.In prepareInputChunksForDecoding, why are parity chunks put in front of data
chunks? Functionally it doesn't really matter, but it'd be better to have a
consistent ordering.
I thought the consistent ordering should be: parity units + data units, which
respects the implementation of {{GaloisField}}. I will make sure other coders
like {{IsaRSRawCoder}} will also follow that.
Raw Reed-Solomon coder in pure Java
---
Key: HADOOP-11542
URL: https://issues.apache.org/jira/browse/HADOOP-11542
Project: Hadoop Common
Issue Type: Sub-task
Affects Versions: HDFS-EC
Reporter: Kai Zheng
Assignee: Kai Zheng
Attachments: HADOOP-11542-v1.patch, HADOOP-11542-v2.patch,
HADOOP-11542-v3.patch, HDFS-RAID-GaloisField.java
This will implement RS coder by porting existing codes in HDFS-RAID in the
new codec and coder framework, which could be useful in case native support
isn't available or convenient in some environments or platforms.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11566) Add tests for raw erasure coders to cover erasure of parity units
Kai Zheng created HADOOP-11566: -- Summary: Add tests for raw erasure coders to cover erasure of parity units Key: HADOOP-11566 URL: https://issues.apache.org/jira/browse/HADOOP-11566 Project: Hadoop Common Issue Type: Test Reporter: Kai Zheng Assignee: Kai Zheng Discussing with [~zhz] in HADOOP-11542: it's planned to have follow up a JIRA to enhance the tests for parity chunks as well. Like erasedDataIndexes, erasedParityIndexes will be added to specify which parity units are to be erased and recovered then. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11542) Raw Reed-Solomon coder in pure Java
[ https://issues.apache.org/jira/browse/HADOOP-11542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14313072#comment-14313072 ] Kai Zheng commented on HADOOP-11542: HADOOP-11566 was opened to follow up for adding the planned tests to cover erasure of parity units. Raw Reed-Solomon coder in pure Java --- Key: HADOOP-11542 URL: https://issues.apache.org/jira/browse/HADOOP-11542 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HADOOP-11542-v1.patch, HADOOP-11542-v2.patch, HADOOP-11542-v3.patch, HDFS-RAID-GaloisField.java This will implement RS coder by porting existing codes in HDFS-RAID in the new codec and coder framework, which could be useful in case native support isn't available or convenient in some environments or platforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11588) Benchmark framework and test for erasure coders
[ https://issues.apache.org/jira/browse/HADOOP-11588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11588: --- Fix Version/s: HDFS-7285 Benchmark framework and test for erasure coders --- Key: HADOOP-11588 URL: https://issues.apache.org/jira/browse/HADOOP-11588 Project: Hadoop Common Issue Type: Sub-task Components: io Reporter: Kai Zheng Assignee: Kai Zheng Fix For: HDFS-7285 Given more than one erasure coders are implemented for a code scheme, we need benchmark and test to help evaluate which one outperforms in certain environment. This is to implement the benchmark framework. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11588) Benchmark framework and test for erasure coders
Kai Zheng created HADOOP-11588: -- Summary: Benchmark framework and test for erasure coders Key: HADOOP-11588 URL: https://issues.apache.org/jira/browse/HADOOP-11588 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Given more than one erasure coders are implemented for a code scheme, we need benchmark and test to help evaluate which one outperforms in certain environment. This is to implement the benchmark framework. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HADOOP-11542) Raw Reed-Solomon coder in pure Java
[ https://issues.apache.org/jira/browse/HADOOP-11542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng resolved HADOOP-11542. Resolution: Fixed Fix Version/s: HDFS-EC Hadoop Flags: Reviewed Raw Reed-Solomon coder in pure Java --- Key: HADOOP-11542 URL: https://issues.apache.org/jira/browse/HADOOP-11542 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Fix For: HDFS-EC Attachments: HADOOP-11542-v1.patch, HADOOP-11542-v2.patch, HADOOP-11542-v3.patch, HADOOP-11542-v4.patch, HADOOP-11542-v5.patch, HDFS-RAID-GaloisField.java This will implement RS coder by porting existing codes in HDFS-RAID in the new codec and coder framework, which could be useful in case native support isn't available or convenient in some environments or platforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11542) Raw Reed-Solomon coder in pure Java
[ https://issues.apache.org/jira/browse/HADOOP-11542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14317623#comment-14317623 ] Kai Zheng commented on HADOOP-11542: Thanks [~zhz] for going thru this. I committed it in both HDFS-EC and HDFS-7285 branches. Raw Reed-Solomon coder in pure Java --- Key: HADOOP-11542 URL: https://issues.apache.org/jira/browse/HADOOP-11542 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HADOOP-11542-v1.patch, HADOOP-11542-v2.patch, HADOOP-11542-v3.patch, HADOOP-11542-v4.patch, HADOOP-11542-v5.patch, HDFS-RAID-GaloisField.java This will implement RS coder by porting existing codes in HDFS-RAID in the new codec and coder framework, which could be useful in case native support isn't available or convenient in some environments or platforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-9477) posixGroups support for LDAP groups mapping service
[ https://issues.apache.org/jira/browse/HADOOP-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14317542#comment-14317542 ] Kai Zheng commented on HADOOP-9477: --- Hi [~dapengsun], Thanks ! Please note you can continue based on [~clamb]'s version. I thought his may be already sync-ed with latest codes though I didn't check. posixGroups support for LDAP groups mapping service --- Key: HADOOP-9477 URL: https://issues.apache.org/jira/browse/HADOOP-9477 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Kai Zheng Assignee: Dapeng Sun Fix For: 2.7.0 Attachments: HADOOP-9477.003.patch, HADOOP-9477.patch, HADOOP-9477.patch Original Estimate: 168h Remaining Estimate: 168h It would be nice to support posixGroups for LdapGroupsMapping service. Below is from current description for the provider: hadoop.security.group.mapping.ldap.search.filter.group: An additional filter to use when searching for LDAP groups. This should be changed when resolving groups against a non-Active Directory installation. posixGroups are currently not a supported group class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11717) Add Redirecting WebSSO behavior with JWT Token in Hadoop Auth
[ https://issues.apache.org/jira/browse/HADOOP-11717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14362086#comment-14362086 ] Kai Zheng commented on HADOOP-11717: Hi [~lmccay], I'm glad to see this, thanks for taking this ! As mentioned in HADOOP-10959, I had a prototype implementing a Kerberos based token (JWT token) authentication approach, covering both terminal command use case and web UI case. I attempted to break down the work but looks like it doesn't go smoothly, as you can see in HADOOP-10670 and HADOOP-10671. I built the similar web SSO flow for Hadoop web UI starting with a JWT token. So with that experience, I will look at your patch and see if anything I can help with. One thing to clarify is, in the Hadoop auth handler you enhanced, if a JWT token is there in the session after redirected back, you will validate the token in the handler itself, right ? No delegate to another service to authenticate the token, right ? If so, I'm wondering if you could leave the chance in your codes, so that other effort like HADOOP-10959 can pluggin or customize the token validation mechanism or behavior. Thanks. By the way a minor, nimbus-jose-jwt library is a good choice, as also made in Apache Kerby, where the TokenPreauth is being implemented for the Kerberos library and KDC. I thought we're much aligned in this part. Add Redirecting WebSSO behavior with JWT Token in Hadoop Auth - Key: HADOOP-11717 URL: https://issues.apache.org/jira/browse/HADOOP-11717 Project: Hadoop Common Issue Type: Improvement Components: security Reporter: Larry McCay Assignee: Larry McCay Attachments: HADOOP-11717-1.patch, HADOOP-11717-2.patch Extend AltKerberosAuthenticationHandler to provide WebSSO flow for UIs. The actual authentication is done by some external service that the handler will redirect to when there is no hadoop.auth cookie and no JWT token found in the incoming request. Using JWT provides a number of benefits: * It is not tied to any specific authentication mechanism - so buys us many SSO integrations * It is cryptographically verifiable for determining whether it can be trusted * Checking for expiration allows for a limited lifetime and window for compromised use This will introduce the use of nimbus-jose-jwt library for processing, validating and parsing JWT tokens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-10959) A Kerberos based token authentication approach
[ https://issues.apache.org/jira/browse/HADOOP-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-10959: --- Summary: A Kerberos based token authentication approach (was: A complement solution to TokenAuth based on Kerberos pre-authentication framework) A Kerberos based token authentication approach -- Key: HADOOP-10959 URL: https://issues.apache.org/jira/browse/HADOOP-10959 Project: Hadoop Common Issue Type: New Feature Components: security Reporter: Kai Zheng Assignee: Kai Zheng Labels: Rhino Attachments: KerbToken-v2.pdf To implement and integrate pluggable authentication providers, enhance desirable single sign on for end users, and help enforce centralized access control on the platform, the community has widely discussed and concluded token based authentication could be the appropriate approach. TokenAuth (HADOOP-9392) was proposed and is under development to implement another Authentication Method in lieu with Simple and Kerberos. It is a big and long term effort to support TokenAuth across the entire ecosystem. We here propose a short term replacement based on Kerberos that can complement to TokenAuth. Our solution involves less codes changes with limited risk and the main development work has already been done in our POC. Users can use our solution as a short term solution to support token inside Hadoop. This effort and resultant solution will be fully described in the design document to be attached. And the brief introduction will be commented. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10959) A Kerberos based token authentication approach
[ https://issues.apache.org/jira/browse/HADOOP-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14362074#comment-14362074 ] Kai Zheng commented on HADOOP-10959: Status update. Haox was accepted by ApacheDS and *Apache Kerby* was launched. We're working on it and implementing the tokenPreauth mechanism in Kerby first. With the major work done there, we'll be back here soon to realize the token support for Hadoop based on Kerberos by leveraging Kerby. A Kerberos based token authentication approach -- Key: HADOOP-10959 URL: https://issues.apache.org/jira/browse/HADOOP-10959 Project: Hadoop Common Issue Type: New Feature Components: security Reporter: Kai Zheng Assignee: Kai Zheng Labels: Rhino Attachments: KerbToken-v2.pdf To implement and integrate pluggable authentication providers, enhance desirable single sign on for end users, and help enforce centralized access control on the platform, the community has widely discussed and concluded token based authentication could be the appropriate approach. TokenAuth (HADOOP-9392) was proposed and is under development to implement another Authentication Method in lieu with Simple and Kerberos. It is a big and long term effort to support TokenAuth across the entire ecosystem. We here propose a short term replacement based on Kerberos that can complement to TokenAuth. Our solution involves less codes changes with limited risk and the main development work has already been done in our POC. Users can use our solution as a short term solution to support token inside Hadoop. This effort and resultant solution will be fully described in the design document to be attached. And the brief introduction will be commented. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11717) Add Redirecting WebSSO behavior with JWT Token in Hadoop Auth
[ https://issues.apache.org/jira/browse/HADOOP-11717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364393#comment-14364393 ] Kai Zheng commented on HADOOP-11717: Thanks [~lmccay] for the update. It looks nice. Just a minor reminding, wouldn't you double check the coding style and formats ? Add Redirecting WebSSO behavior with JWT Token in Hadoop Auth - Key: HADOOP-11717 URL: https://issues.apache.org/jira/browse/HADOOP-11717 Project: Hadoop Common Issue Type: Improvement Components: security Reporter: Larry McCay Assignee: Larry McCay Attachments: HADOOP-11717-1.patch, HADOOP-11717-2.patch, HADOOP-11717-3.patch, HADOOP-11717-4.patch Extend AltKerberosAuthenticationHandler to provide WebSSO flow for UIs. The actual authentication is done by some external service that the handler will redirect to when there is no hadoop.auth cookie and no JWT token found in the incoming request. Using JWT provides a number of benefits: * It is not tied to any specific authentication mechanism - so buys us many SSO integrations * It is cryptographically verifiable for determining whether it can be trusted * Checking for expiration allows for a limited lifetime and window for compromised use This will introduce the use of nimbus-jose-jwt library for processing, validating and parsing JWT tokens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11647) Reed-Solomon ErasureCoder
[
https://issues.apache.org/jira/browse/HADOOP-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366577#comment-14366577
]
Kai Zheng commented on HADOOP-11647:
Discussed more offline with Uma and will update a more time for the patch, as
{{doEncode}} wouldn't do much better than {{performDecoding}}.
Reed-Solomon ErasureCoder
-
Key: HADOOP-11647
URL: https://issues.apache.org/jira/browse/HADOOP-11647
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
Attachments: HADOOP-11647-v2.patch, HADOOP-11647-v4.patch,
HADOOP-11647-v5.patch, HDFS-7664-v1.patch
This is to implement Reed-Solomon ErasureCoder using the API defined in
HADOOP-11646. It supports to plugin via configuration for concrete
RawErasureCoder, using either JRSErasureCoder added in HDFS-7418 or
IsaRSErasureCoder added in HDFS-7338.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HADOOP-11706) Refine a little bit erasure coder API
[ https://issues.apache.org/jira/browse/HADOOP-11706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng resolved HADOOP-11706. Resolution: Fixed Hadoop Flags: Reviewed Refine a little bit erasure coder API - Key: HADOOP-11706 URL: https://issues.apache.org/jira/browse/HADOOP-11706 Project: Hadoop Common Issue Type: Improvement Reporter: Kai Zheng Assignee: Kai Zheng Priority: Minor Fix For: HDFS-7285 Attachments: HADOOP-11706-v1.patch In HADOOP-11705 it allows erasure coder class to be configurable, but not the interface, thus introduces the not elegant codes of type casting. This is minor to refine the codes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11706) Refine a little bit erasure coder API
[ https://issues.apache.org/jira/browse/HADOOP-11706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366579#comment-14366579 ] Kai Zheng commented on HADOOP-11706: Thanks Uma for the review. I've committed it in the branch. Refine a little bit erasure coder API - Key: HADOOP-11706 URL: https://issues.apache.org/jira/browse/HADOOP-11706 Project: Hadoop Common Issue Type: Improvement Reporter: Kai Zheng Assignee: Kai Zheng Priority: Minor Fix For: HDFS-7285 Attachments: HADOOP-11706-v1.patch In HADOOP-11705 it allows erasure coder class to be configurable, but not the interface, thus introduces the not elegant codes of type casting. This is minor to refine the codes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11647) Reed-Solomon ErasureCoder
[ https://issues.apache.org/jira/browse/HADOOP-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11647: --- Attachment: HADOOP-11647-v6.patch Updated the patch according to discussion with Uma. Reed-Solomon ErasureCoder - Key: HADOOP-11647 URL: https://issues.apache.org/jira/browse/HADOOP-11647 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HADOOP-11647-v2.patch, HADOOP-11647-v4.patch, HADOOP-11647-v5.patch, HADOOP-11647-v6.patch, HDFS-7664-v1.patch This is to implement Reed-Solomon ErasureCoder using the API defined in HADOOP-11646. It supports to plugin via configuration for concrete RawErasureCoder, using either JRSErasureCoder added in HDFS-7418 or IsaRSErasureCoder added in HDFS-7338. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11717) Add Redirecting WebSSO behavior with JWT Token in Hadoop Auth
[
https://issues.apache.org/jira/browse/HADOOP-11717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14362388#comment-14362388
]
Kai Zheng commented on HADOOP-11717:
I read the non-trivial patch, it's really decent and of very good quality. A
good job !
My comments are so far:
1. Why we need to add BC and nimbus library deps to hadoop-project, since
they're already in hadoop-auth project ?
2. For secure protecting JWT token, we should use SSL for the web flow. We
might need to add such security consideration texts in the new handler header
comment.
3. I'm not sure we could avoid using cookie to pass the JWT token, since it's
not a good practice. By post and putting it in the body instead ?
4. Anyway, please limit cookie just as one method to convey token, so better to
avoid cookie stuffs in the many places (variables, words in logs and etc.).
5. I guess in somewhere we need document how to configure the new
authentication handler, to feed the new properties like the login url.
6. Do we support the new mechanism for the both web UI and web hdfs ? Allow SSO
between the two ? How would you go ? In HADOOP-10671, it allows the same
configurations set for the both, thus SSO effect can be achieved.
7. Do we consider JWT token lifetime ? I thought maybe we should limit the
lifetime of the resultant authentication token (hadoop-auth) to the lifetime of
the JWT token.
8. Where {{originalUrl}} is used ? A constant for it ?
9. Can you construct {{loginURL}} only when necessary ? I thought it makes
sense.
10. I thought {{handleJWTToken}} instead of {{handleJWTCookie}}. Anyway, for it:
1) Why we have a userName parameter ? Looks like not used.
2) Would we rewrite it for better reading and extension. Suggest:
{code}
handleJWTCookie(jwtToken) {
boolean validated = validateToken(jwtToken);
...
}
validateToken(jwtToken) {
validateSignature(jwtToken);
validateAudiences(jwtToken);
validateExpiration(jwtToken);
}
{code}
Other effort like HADOOP-10959 can easily override validateToken method.
3) I thought the coding style here might be a little different from the project.
11. Only {{userName}} is used as the result of web sso, but I'm not sure that's
enough to ensure its uniqueness.
12. Ref. below, the message isn't correct. By the way, looks like we only
support PEM format.
{code}
+ if (pem.startsWith(PEM_HEADER)) {
+message = CertificateException - do not include PEM header and
footer;
+ }
{code}
Add Redirecting WebSSO behavior with JWT Token in Hadoop Auth
-
Key: HADOOP-11717
URL: https://issues.apache.org/jira/browse/HADOOP-11717
Project: Hadoop Common
Issue Type: Improvement
Components: security
Reporter: Larry McCay
Assignee: Larry McCay
Attachments: HADOOP-11717-1.patch, HADOOP-11717-2.patch
Extend AltKerberosAuthenticationHandler to provide WebSSO flow for UIs.
The actual authentication is done by some external service that the handler
will redirect to when there is no hadoop.auth cookie and no JWT token found
in the incoming request.
Using JWT provides a number of benefits:
* It is not tied to any specific authentication mechanism - so buys us many
SSO integrations
* It is cryptographically verifiable for determining whether it can be trusted
* Checking for expiration allows for a limited lifetime and window for
compromised use
This will introduce the use of nimbus-jose-jwt library for processing,
validating and parsing JWT tokens.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11706) Refine a little bit erasure coder API
[ https://issues.apache.org/jira/browse/HADOOP-11706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11706: --- Description: In HADOOP-11705 it allows erasure coder class to be configurable, but not the interface, thus introduces the not elegant codes of type casting. This is minor to refine the codes. (was: In HADOOP-11705 it allows erasure coder class to be configurable, but not the interface, thus introcues not elegant codes of type casting. This is minor to refine the codes.) Refine a little bit erasure coder API - Key: HADOOP-11706 URL: https://issues.apache.org/jira/browse/HADOOP-11706 Project: Hadoop Common Issue Type: Improvement Reporter: Kai Zheng Assignee: Kai Zheng Priority: Minor Fix For: HDFS-7285 Attachments: HADOOP-11706-v1.patch In HADOOP-11705 it allows erasure coder class to be configurable, but not the interface, thus introduces the not elegant codes of type casting. This is minor to refine the codes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11717) Add Redirecting WebSSO behavior with JWT Token in Hadoop Auth
[
https://issues.apache.org/jira/browse/HADOOP-11717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365135#comment-14365135
]
Kai Zheng commented on HADOOP-11717:
More comments to complete the review, mainly about tests.
1. The new handler looks little heavier to me. One thing we can do for now is
to have a utility class like {{JwtTokenUtil}} and remove at least
{{getPublicKey}} related logic and variables there. So some related tests like
{{testValidPEM}} will not need a handler instance. How about having
{{parsePublicKey}} (instead of {{getPublicKey}}), {{parseAudiences}}, and
{{parseJwtToken}} (even trivial, restore JWT from a string like from cookie)
for the new utility class if it sounds better to have ?
2. It would be good not to couple the new test with
{{KerberosSecurityTestcase}} since all the test cases won't relate to Kerberos
at all.
3. For all the handler really logic tests, better to move the following to
{{setup}} or {{teardown}}.
{code}
JWTRedirectAuthenticationHandler handler = new
JWTRedirectAuthenticationHandler();
...
handler.destroy();
{code}
4. All the handler logic tests are different in token preparing. It's possible
to have the following in a function where token is a parameter to avoid
repeating.
{code}
+ Properties props = getProperties();
+ handler.init(props);
+
+ SignedJWT jwt = getJWT(bob, new Date(new Date().getTime() + 5000),
+ privateKey);
+
+ Cookie cookie = new Cookie(hadoop-jwt, jwt.serialize());
+ HttpServletRequest request = Mockito.mock(HttpServletRequest.class);
+ Mockito.when(request.getCookies()).thenReturn(new Cookie[] { cookie });
+ Mockito.when(request.getRequestURL()).thenReturn(
+ new StringBuffer(SERVICE_URL));
+ HttpServletResponse response = Mockito.mock(HttpServletResponse.class);
+ Mockito.when(response.encodeRedirectURL(SERVICE_URL)).thenReturn(
+ SERVICE_URL);
+
+ AuthenticationToken token = handler.alternateAuthenticate(request,
+ response);
{code}
5. In the tests, we have repeated values like bar, bob here and there. How
about having variables for them ?
6. In the following codes, {{aud}} and {{sigInput}} aren't really used.
{code}
+ListString aud = new ArrayListString();
+aud.add(bar);
+claimsSet.setAudience(bar);
+
+JWSHeader header = new JWSHeader.Builder(JWSAlgorithm.RS256).build();
+
+SignedJWT signedJWT = new SignedJWT(header, claimsSet);
+Base64URL sigInput = Base64URL.encode(signedJWT.getSigningInput());
+JWSSigner signer = new RSASSASigner(privateKey);
+
+signedJWT.sign(signer);
+
+return signedJWT;
{code}
Add Redirecting WebSSO behavior with JWT Token in Hadoop Auth
-
Key: HADOOP-11717
URL: https://issues.apache.org/jira/browse/HADOOP-11717
Project: Hadoop Common
Issue Type: Improvement
Components: security
Reporter: Larry McCay
Assignee: Larry McCay
Attachments: HADOOP-11717-1.patch, HADOOP-11717-2.patch,
HADOOP-11717-3.patch, HADOOP-11717-4.patch, HADOOP-11717-5.patch
Extend AltKerberosAuthenticationHandler to provide WebSSO flow for UIs.
The actual authentication is done by some external service that the handler
will redirect to when there is no hadoop.auth cookie and no JWT token found
in the incoming request.
Using JWT provides a number of benefits:
* It is not tied to any specific authentication mechanism - so buys us many
SSO integrations
* It is cryptographically verifiable for determining whether it can be trusted
* Checking for expiration allows for a limited lifetime and window for
compromised use
This will introduce the use of nimbus-jose-jwt library for processing,
validating and parsing JWT tokens.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11647) Reed-Solomon ErasureCoder
[ https://issues.apache.org/jira/browse/HADOOP-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14370318#comment-14370318 ] Kai Zheng commented on HADOOP-11647: The dep of HADOOP-11707 was committed. The patch doesn't need to update and is ready for another round of review. Reed-Solomon ErasureCoder - Key: HADOOP-11647 URL: https://issues.apache.org/jira/browse/HADOOP-11647 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HADOOP-11647-v2.patch, HADOOP-11647-v4.patch, HADOOP-11647-v5.patch, HADOOP-11647-v6.patch, HDFS-7664-v1.patch This is to implement Reed-Solomon ErasureCoder using the API defined in HADOOP-11646. It supports to plugin via configuration for concrete RawErasureCoder, using either JRSErasureCoder added in HDFS-7418 or IsaRSErasureCoder added in HDFS-7338. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11707) Add factory to create raw erasure coder
[
https://issues.apache.org/jira/browse/HADOOP-11707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14370304#comment-14370304
]
Kai Zheng commented on HADOOP-11707:
Thanks Uma for the review. I've committed it in the branch.
Add factory to create raw erasure coder
---
Key: HADOOP-11707
URL: https://issues.apache.org/jira/browse/HADOOP-11707
Project: Hadoop Common
Issue Type: Sub-task
Components: io
Reporter: Kai Zheng
Assignee: Kai Zheng
Fix For: HDFS-7285
Attachments: HADOOP-11707-v1.patch
We have {{RawErasureEncoder}} and {{RawErasureDecoder}} interface separately,
which simplifies the implementation of raw coders. This would require to
configure raw encoder and decoder respectively for a {{ErasureCoder}}, which
isn't convenient. To simplify the configuration, we would have coder factory
to group encoder and decoder together so only a factory class needs to be
configured.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HADOOP-11707) Add factory to create raw erasure coder
[
https://issues.apache.org/jira/browse/HADOOP-11707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng resolved HADOOP-11707.
Resolution: Fixed
Target Version/s: HDFS-7285
Hadoop Flags: Reviewed
Add factory to create raw erasure coder
---
Key: HADOOP-11707
URL: https://issues.apache.org/jira/browse/HADOOP-11707
Project: Hadoop Common
Issue Type: Sub-task
Components: io
Reporter: Kai Zheng
Assignee: Kai Zheng
Fix For: HDFS-7285
Attachments: HADOOP-11707-v1.patch
We have {{RawErasureEncoder}} and {{RawErasureDecoder}} interface separately,
which simplifies the implementation of raw coders. This would require to
configure raw encoder and decoder respectively for a {{ErasureCoder}}, which
isn't convenient. To simplify the configuration, we would have coder factory
to group encoder and decoder together so only a factory class needs to be
configured.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11647) Reed-Solomon ErasureCoder
[ https://issues.apache.org/jira/browse/HADOOP-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14370637#comment-14370637 ] Kai Zheng commented on HADOOP-11647: Thanks Uma for the review. I've committed it in the branch. Reed-Solomon ErasureCoder - Key: HADOOP-11647 URL: https://issues.apache.org/jira/browse/HADOOP-11647 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HADOOP-11647-v2.patch, HADOOP-11647-v4.patch, HADOOP-11647-v5.patch, HADOOP-11647-v6.patch, HDFS-7664-v1.patch This is to implement Reed-Solomon ErasureCoder using the API defined in HADOOP-11646. It supports to plugin via configuration for concrete RawErasureCoder, using either JRSErasureCoder added in HDFS-7418 or IsaRSErasureCoder added in HDFS-7338. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HADOOP-11647) Reed-Solomon ErasureCoder
[ https://issues.apache.org/jira/browse/HADOOP-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng resolved HADOOP-11647. Resolution: Fixed Fix Version/s: HDFS-7285 Hadoop Flags: Reviewed Reed-Solomon ErasureCoder - Key: HADOOP-11647 URL: https://issues.apache.org/jira/browse/HADOOP-11647 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Fix For: HDFS-7285 Attachments: HADOOP-11647-v2.patch, HADOOP-11647-v4.patch, HADOOP-11647-v5.patch, HADOOP-11647-v6.patch, HDFS-7664-v1.patch This is to implement Reed-Solomon ErasureCoder using the API defined in HADOOP-11646. It supports to plugin via configuration for concrete RawErasureCoder, using either JRSErasureCoder added in HDFS-7418 or IsaRSErasureCoder added in HDFS-7338. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11646) Erasure Coder API for encoding and decoding of block group
[ https://issues.apache.org/jira/browse/HADOOP-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351143#comment-14351143 ] Kai Zheng commented on HADOOP-11646: Thanks [~vinayrpet] for the codes ! Erasure Coder API for encoding and decoding of block group -- Key: HADOOP-11646 URL: https://issues.apache.org/jira/browse/HADOOP-11646 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Fix For: HDFS-7285 Attachments: HADOOP-11646-v4.patch, HDFS-7662-v1.patch, HDFS-7662-v2.patch, HDFS-7662-v3.patch This is to define ErasureCoder API for encoding and decoding of BlockGroup. Given a BlockGroup, ErasureCoder extracts data chunks from the blocks and leverages RawErasureCoder defined in HADOOP-11514 to perform concrete encoding or decoding. Note this mainly focuses on the basic fundamental aspects, and solves encoding, data blocks recovering and etc. Regarding parity blocks recovering, as it involves multiple steps, HADOOP-11550 will handle it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11683) Need a plugin API to translate long principal names to local OS user names arbitrarily
[
https://issues.apache.org/jira/browse/HADOOP-11683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351116#comment-14351116
]
Kai Zheng commented on HADOOP-11683:
bq.we already have user-code running in the NN now
{{UserGroupsMappingProvider}} pluggable interface is a good example, which even
allows to query external LDAP server to perform user-groups mapping. We might
borrow similar idea from it for this.
To allow such an interface for the mapping would also allow to implement the
translation rules in modular approach, even not by user code.
I understand the NameNode concern, yes it's possible to involve overhead for NN
if user provided plugin performs the mapping not fast every time. To alleviate
the pain, we could consider to support cache of the mapping results in the
framework.
Need a plugin API to translate long principal names to local OS user names
arbitrarily
--
Key: HADOOP-11683
URL: https://issues.apache.org/jira/browse/HADOOP-11683
Project: Hadoop Common
Issue Type: Improvement
Components: security
Reporter: Sunny Cheung
We need a plugin API to translate long principal names (e.g.
john@example.com) to local OS user names (e.g. user123456) arbitrarily.
For some organizations the name translation is straightforward (e.g.
john@example.com to john_doe), and the hadoop.security.auth_to_local
configurable mapping is sufficient to resolve this (see HADOOP-6526).
However, in some other cases the name translation is arbitrary and cannot be
generalized by a set of translation rules easily.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11683) Need a plugin API to translate long principal names to local OS user names arbitrarily
[ https://issues.apache.org/jira/browse/HADOOP-11683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354240#comment-14354240 ] Kai Zheng commented on HADOOP-11683: bq.What do we mean by modular approach vs. user code here ? I mean by having an interface here, we can have different modular implementation classes for each mechanism, instead of mixing all of mechanisms together in one *BIG* class. bq.Perhaps the plugin could forward requests to a local daemon with cache capability I'm not sure that would eliminate the necessity of caching results in Java side, particularly considering external daemon or service might not be connectable or reliable to NameNode. Need a plugin API to translate long principal names to local OS user names arbitrarily -- Key: HADOOP-11683 URL: https://issues.apache.org/jira/browse/HADOOP-11683 Project: Hadoop Common Issue Type: Improvement Components: security Reporter: Sunny Cheung We need a plugin API to translate long principal names (e.g. john@example.com) to local OS user names (e.g. user123456) arbitrarily. For some organizations the name translation is straightforward (e.g. john@example.com to john_doe), and the hadoop.security.auth_to_local configurable mapping is sufficient to resolve this (see HADOOP-6526). However, in some other cases the name translation is arbitrary and cannot be generalized by a set of translation rules easily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10671) Unify configurations for authentication filters between web console and web hdfs
[ https://issues.apache.org/jira/browse/HADOOP-10671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354186#comment-14354186 ] Kai Zheng commented on HADOOP-10671: Hi [~wheat9], Thanks for your comment and guiding. bq.Can you please list all the configurations and then we can discuss what is the best way to move forward ? Yes I will list all the affected configuration properties for the discussion in the week. To better reflect what's actually done here, I modified the JIRA description. Actually SSO effect between web console and web hdfs is just a result of this change. Unify configurations for authentication filters between web console and web hdfs Key: HADOOP-10671 URL: https://issues.apache.org/jira/browse/HADOOP-10671 Project: Hadoop Common Issue Type: Improvement Components: security Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HADOOP-10671-v3.patch, hadoop-10671-v2.patch, hadoop-10671.patch Currently it's not able to single sign on between hadoop web console and webhdfs since they don't share common configurations as required to, such as signature secret to sign authenticaton token, and domain cookie etc. This improvement would allow sso between the two, and also simplify the configuration by removing the duplicate effort for the two parts. The sso makes sense because in current web console, it integrates webhdfs and we should avoid redundant sign on in different mechanisms. This is necessary when a certain authentication mechanism other than SPNEGO is desired across web console and webhdfs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-10671) Unify and simplify common configurations for authentication filters between web console and web hdfs
[ https://issues.apache.org/jira/browse/HADOOP-10671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-10671: --- Summary: Unify and simplify common configurations for authentication filters between web console and web hdfs (was: Unify configurations for authentication filters between web console and web hdfs) Unify and simplify common configurations for authentication filters between web console and web hdfs Key: HADOOP-10671 URL: https://issues.apache.org/jira/browse/HADOOP-10671 Project: Hadoop Common Issue Type: Improvement Components: security Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HADOOP-10671-v3.patch, hadoop-10671-v2.patch, hadoop-10671.patch Currently it's not able to single sign on between hadoop web console and webhdfs since they don't share common configurations as required to, such as signature secret to sign authenticaton token, and domain cookie etc. This improvement would allow sso between the two, and also simplify the configuration by removing the duplicate effort for the two parts. The sso makes sense because in current web console, it integrates webhdfs and we should avoid redundant sign on in different mechanisms. This is necessary when a certain authentication mechanism other than SPNEGO is desired across web console and webhdfs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-10671) Unify configurations for authentication filters between web console and web hdfs
[ https://issues.apache.org/jira/browse/HADOOP-10671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-10671: --- Summary: Unify configurations for authentication filters between web console and web hdfs (was: Single sign on between web console and webhdfs) Unify configurations for authentication filters between web console and web hdfs Key: HADOOP-10671 URL: https://issues.apache.org/jira/browse/HADOOP-10671 Project: Hadoop Common Issue Type: Improvement Components: security Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HADOOP-10671-v3.patch, hadoop-10671-v2.patch, hadoop-10671.patch Currently it's not able to single sign on between hadoop web console and webhdfs since they don't share common configurations as required to, such as signature secret to sign authenticaton token, and domain cookie etc. This improvement would allow sso between the two, and also simplify the configuration by removing the duplicate effort for the two parts. The sso makes sense because in current web console, it integrates webhdfs and we should avoid redundant sign on in different mechanisms. This is necessary when a certain authentication mechanism other than SPNEGO is desired across web console and webhdfs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11645) Erasure Codec API covering the essential aspects for an erasure code
[ https://issues.apache.org/jira/browse/HADOOP-11645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11645: --- Attachment: HADOOP-11645-v1.patch Uploaded an initial patch to illustrate the general idea. Erasure Codec API covering the essential aspects for an erasure code Key: HADOOP-11645 URL: https://issues.apache.org/jira/browse/HADOOP-11645 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HADOOP-11645-v1.patch This is to define the even higher level API *ErasureCodec* to possiblly consider all the essential aspects for an erasure code, as discussed in in HDFS-7337 in details. Generally, it will cover the necessary configurations about which *RawErasureCoder* to use for the code scheme, how to form and layout the BlockGroup, and etc. It will also discuss how an *ErasureCodec* will be used in both client and DataNode, in all the supported modes related to EC. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11705) Make erasure coder configurable
[ https://issues.apache.org/jira/browse/HADOOP-11705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358222#comment-14358222 ] Kai Zheng commented on HADOOP-11705: Committed in HDFS-7285 branch. Make erasure coder configurable --- Key: HADOOP-11705 URL: https://issues.apache.org/jira/browse/HADOOP-11705 Project: Hadoop Common Issue Type: Sub-task Components: io Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HADOOP-11705-v1.patch, HADOOP-11705-v2.patch This is to enhance erasure coder and raw coder to make them configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HADOOP-11705) Make erasure coder configurable
[ https://issues.apache.org/jira/browse/HADOOP-11705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng resolved HADOOP-11705. Resolution: Fixed Fix Version/s: HDFS-7285 Hadoop Flags: Reviewed Make erasure coder configurable --- Key: HADOOP-11705 URL: https://issues.apache.org/jira/browse/HADOOP-11705 Project: Hadoop Common Issue Type: Sub-task Components: io Reporter: Kai Zheng Assignee: Kai Zheng Fix For: HDFS-7285 Attachments: HADOOP-11705-v1.patch, HADOOP-11705-v2.patch This is to enhance erasure coder and raw coder to make them configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11647) Reed-Solomon ErasureCoder
[ https://issues.apache.org/jira/browse/HADOOP-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11647: --- Attachment: HADOOP-11647-v4.patch Updated the patch accordingly. Reed-Solomon ErasureCoder - Key: HADOOP-11647 URL: https://issues.apache.org/jira/browse/HADOOP-11647 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HADOOP-11647-v2.patch, HADOOP-11647-v4.patch, HDFS-7664-v1.patch This is to implement Reed-Solomon ErasureCoder using the API defined in HADOOP-11646. It supports to plugin via configuration for concrete RawErasureCoder, using either JRSErasureCoder added in HDFS-7418 or IsaRSErasureCoder added in HDFS-7338. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11647) Reed-Solomon ErasureCoder
[ https://issues.apache.org/jira/browse/HADOOP-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358330#comment-14358330 ] Kai Zheng commented on HADOOP-11647: In the update I left the configuration of raw coder for other issue to think about, because it involves non-trivial work that's better to be done elsewhere. Reed-Solomon ErasureCoder - Key: HADOOP-11647 URL: https://issues.apache.org/jira/browse/HADOOP-11647 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HADOOP-11647-v2.patch, HADOOP-11647-v4.patch, HDFS-7664-v1.patch This is to implement Reed-Solomon ErasureCoder using the API defined in HADOOP-11646. It supports to plugin via configuration for concrete RawErasureCoder, using either JRSErasureCoder added in HDFS-7418 or IsaRSErasureCoder added in HDFS-7338. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
