[jira] [Comment Edited] (HADOOP-14920) KMSClientProvider won't work with KMS delegation token retrieved from non-Java client.
[ https://issues.apache.org/jira/browse/HADOOP-14920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189292#comment-16189292 ] Xiaoyu Yao edited comment on HADOOP-14920 at 10/3/17 5:51 AM: -- Thanks [~xiaochen] for the pointers. I briefly looked into the discussion on HADOOP-14445. I seems that we agree the client should be able to set the service as needed based on [~daryn]'s latest summary "client gets tokens and sets service to kp uri" The issue here is orthogonal to HADOOP-14445 where we want to enable *non-java client* like curl+http to set the service for its requested delegation token. This will be needed for non-java client even with HADOOP-14445 unless we remove the service check in KMSClientProvider completely which is the solution 2 mentioned in the description. was (Author: xyao): Thanks [~xiaochen] for the pointers. I briefly looked into the discussion on HADOOP-14445. I seems that we agree the client should be able to set the service as needed based on [~daryn]'s latest summary "client gets tokens and sets service to kp uri" The issue here is orthogonal to HADOOP-14445 where we want to enable *non-java client* like curl+http to set the service for its requested delegation token. This will be needed for non-java client even with HADOOP-14445 unless we remove the service check in KMSClientProvider completely. > KMSClientProvider won't work with KMS delegation token retrieved from > non-Java client. > -- > > Key: HADOOP-14920 > URL: https://issues.apache.org/jira/browse/HADOOP-14920 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HADOOP-14920.001.patch > > > HADOOP-13381 added support to use KMS delegation token to connect to KMS > server for key operations. However, the logic to check if the UGI container > KMS delegation token assumes that the token must contain a service attribute. > Otherwise, a KMS delegation token won't be recognized. > For delegation token obtained via non-java client such curl (http), the > default DelegationTokenAuthenticationHandler only support *renewer* parameter > and assume the client itself will add the service attribute. This makes a > java client with KMSClientProvdier can't use for KMS delegation token > retrieved form non-java client because the token does not contain a service > attribute. > I did some investigation on this and found two solutions: > 1. Similar use case exists for webhdfs, and webhdfs supports it with a > ["service" > parameter|https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Get_Delegation_Token]. > We can do this similarly by allowing client to specify a service attribute in > the request URL and included in the token returned like webhdfs. Even though > this will change in DelegationTokenAuthenticationHandler and may affect many > other web component, this seems to be a clean and low risk solution because > it will be an optional parameter. Also, other components get non-java client > interop support for free if they have the similar use case. > 2. The other way to solve this is to release the token check in > KMSClientProvider to check only the token kind instead of the service. This > is an easy work around but seems less optimal to me. > cc: [~xiaochen] for additional input. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14920) KMSClientProvider won't work with KMS delegation token retrieved from non-Java client.
[ https://issues.apache.org/jira/browse/HADOOP-14920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189292#comment-16189292 ] Xiaoyu Yao commented on HADOOP-14920: - Thanks [~xiaochen] for the pointers. I briefly looked into the discussion on HADOOP-14445. I seems that we agree the client should be able to set the service as needed based on [~daryn]'s latest summary "client gets tokens and sets service to kp uri" The issue here is orthogonal to HADOOP-14445 where we want to enable *non-java client* like curl+http to set the service for its requested delegation token. This will be needed for non-java client even with HADOOP-14445 unless we remove the service check in KMSClientProvider completely. > KMSClientProvider won't work with KMS delegation token retrieved from > non-Java client. > -- > > Key: HADOOP-14920 > URL: https://issues.apache.org/jira/browse/HADOOP-14920 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HADOOP-14920.001.patch > > > HADOOP-13381 added support to use KMS delegation token to connect to KMS > server for key operations. However, the logic to check if the UGI container > KMS delegation token assumes that the token must contain a service attribute. > Otherwise, a KMS delegation token won't be recognized. > For delegation token obtained via non-java client such curl (http), the > default DelegationTokenAuthenticationHandler only support *renewer* parameter > and assume the client itself will add the service attribute. This makes a > java client with KMSClientProvdier can't use for KMS delegation token > retrieved form non-java client because the token does not contain a service > attribute. > I did some investigation on this and found two solutions: > 1. Similar use case exists for webhdfs, and webhdfs supports it with a > ["service" > parameter|https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Get_Delegation_Token]. > We can do this similarly by allowing client to specify a service attribute in > the request URL and included in the token returned like webhdfs. Even though > this will change in DelegationTokenAuthenticationHandler and may affect many > other web component, this seems to be a clean and low risk solution because > it will be an optional parameter. Also, other components get non-java client > interop support for free if they have the similar use case. > 2. The other way to solve this is to release the token check in > KMSClientProvider to check only the token kind instead of the service. This > is an easy work around but seems less optimal to me. > cc: [~xiaochen] for additional input. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14910) Upgrade netty-all jar to latest 4.0.x.Final
[ https://issues.apache.org/jira/browse/HADOOP-14910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189278#comment-16189278 ] Vinayakumar B commented on HADOOP-14910: bq. Is this change related to netty-all version upgrade? This is related to ParameterParser bugs about decoding '+' and '%' in URI. With latest jar workarounds in HDFS code was no more necessary, Decoding tests passes without those workarounds and fails with those. > Upgrade netty-all jar to latest 4.0.x.Final > --- > > Key: HADOOP-14910 > URL: https://issues.apache.org/jira/browse/HADOOP-14910 > Project: Hadoop Common > Issue Type: Bug >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Critical > Attachments: HADOOP-14910-01.patch, HADOOP-14910-02.patch > > > Upgrade netty-all jar to 4.0.37.Final version to fix latest vulnerabilities > reported. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14920) KMSClientProvider won't work with KMS delegation token retrieved from non-Java client.
[ https://issues.apache.org/jira/browse/HADOOP-14920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189243#comment-16189243 ] Xiao Chen commented on HADOOP-14920: Thanks [~xyao] for reporting this and attaching a patch. Option 1 would work and be consistent with webhdfs, but I think [~shahrs87] was working on HADOOP-14445 where the service field could be dynamically set. I'm not sure whether that would impact the case here, but good to check. Depending on how that's done, the service check may or may not be needed IIUC. Rushabh, any thoughts? > KMSClientProvider won't work with KMS delegation token retrieved from > non-Java client. > -- > > Key: HADOOP-14920 > URL: https://issues.apache.org/jira/browse/HADOOP-14920 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HADOOP-14920.001.patch > > > HADOOP-13381 added support to use KMS delegation token to connect to KMS > server for key operations. However, the logic to check if the UGI container > KMS delegation token assumes that the token must contain a service attribute. > Otherwise, a KMS delegation token won't be recognized. > For delegation token obtained via non-java client such curl (http), the > default DelegationTokenAuthenticationHandler only support *renewer* parameter > and assume the client itself will add the service attribute. This makes a > java client with KMSClientProvdier can't use for KMS delegation token > retrieved form non-java client because the token does not contain a service > attribute. > I did some investigation on this and found two solutions: > 1. Similar use case exists for webhdfs, and webhdfs supports it with a > ["service" > parameter|https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Get_Delegation_Token]. > We can do this similarly by allowing client to specify a service attribute in > the request URL and included in the token returned like webhdfs. Even though > this will change in DelegationTokenAuthenticationHandler and may affect many > other web component, this seems to be a clean and low risk solution because > it will be an optional parameter. Also, other components get non-java client > interop support for free if they have the similar use case. > 2. The other way to solve this is to release the token check in > KMSClientProvider to check only the token kind instead of the service. This > is an easy work around but seems less optimal to me. > cc: [~xiaochen] for additional input. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[
https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189239#comment-16189239
]
Hadoop QA commented on HADOOP-14600:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
53s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 34m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 37m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 44s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
49s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 28m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 28m
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 28m 17s{color}
| {color:red} root generated 1 new + 1272 unchanged - 0 fixed = 1273 total (was
1272) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 39s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 126 new + 227 unchanged - 1 fixed = 353 total (was 228) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 47s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 20s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}156m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestRaceWhenRelogin |
| | hadoop.security.TestKDiag |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HADOOP-14600 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12890083/HADOOP-14600.004.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle cc |
| uname | Linux e305e8aa1f0c 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d4d2fd1 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| javac |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13442/artifact/patchprocess/diff-compile-javac-root.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13442/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
|
[jira] [Commented] (HADOOP-14918) remove the Local Dynamo DB test option
[ https://issues.apache.org/jira/browse/HADOOP-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189188#comment-16189188 ] Aaron Fabbri commented on HADOOP-14918: --- The local dynamo lib was useful during development I think, but now I'm in favor of removing it. I typically test local (i.e. -Ddynamo *not* specified) and live dynamo. As we've seen, live dynamo behaves differently so we end up having to test it anyways. I think you still want -Ddynamo flag to choose LocalMetadataStore test case, no? Having the local MetadataStore available helps make the case for removing localdynamo, IMO. We get some coverage without requiring live dynamo at least. > remove the Local Dynamo DB test option > -- > > Key: HADOOP-14918 > URL: https://issues.apache.org/jira/browse/HADOOP-14918 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0 >Reporter: Steve Loughran > > I'm going to propose cutting out the localdynamo test option for s3guard > * the local DDB JAR is unmaintained/lags the SDK We work with...eventually > there'll be differences in API. > * as the local dynamo DB is unshaded. it complicates classpath setup for the > build. Remove it and there's no need to worry about versions of anything > other than the shaded AWS > * it complicates test runs. Now we need to test for both localdynamo *and* > real dynamo > * but we can't ignore real dynamo, because that's the one which matters > While the local option promises to reduce test costs, really, it's just > adding complexity. If you are testing with s3guard, you need to have a real > table to test against., And with the exception of those people testing s3a > against non-AWS, consistent endpoints, everyone should be testing with > S3Guard. > Straightforward to remove. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13514) Upgrade maven surefire plugin to 2.19.1
[
https://issues.apache.org/jira/browse/HADOOP-13514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189179#comment-16189179
]
Akira Ajisaka commented on HADOOP-13514:
Hi [~ste...@apache.org], would you review the 003 patch?
> Upgrade maven surefire plugin to 2.19.1
> ---
>
> Key: HADOOP-13514
> URL: https://issues.apache.org/jira/browse/HADOOP-13514
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 2.8.0
>Reporter: Ewan Higgs
>Assignee: Akira Ajisaka
> Attachments: HADOOP-13514.002.patch, HADOOP-13514.003.patch,
> HADOOP-13514-addendum.01.patch, HADOOP-13514-testing.001.patch,
> HADOOP-13514-testing.002.patch, HADOOP-13514-testing.003.patch,
> HADOOP-13514-testing.004.patch, surefire-2.19.patch
>
>
> A lot of people working on Hadoop don't want to run all the tests when they
> develop; only the bits they're working on. Surefire 2.19 introduced more
> useful test filters which let us run a subset of the tests that brings the
> build time down from 'come back tomorrow' to 'grab a coffee'.
> For instance, if I only care about the S3 adaptor, I might run:
> {code}
> mvn test -Dmaven.javadoc.skip=true -Pdist,native -Djava.awt.headless=true
> \"-Dtest=org.apache.hadoop.fs.*, org.apache.hadoop.hdfs.*,
> org.apache.hadoop.fs.s3a.*\"
> {code}
> We can work around this by specifying the surefire version on the command
> line but it would be better, imo, to just update the default surefire used.
> {code}
> mvn test -Dmaven.javadoc.skip=true -Pdist,native -Djava.awt.headless=true
> \"-Dtest=org.apache.hadoop.fs.*, org.apache.hadoop.hdfs.*,
> org.apache.hadoop.fs.s3a.*\" -Dmaven-surefire-plugin.version=2.19.1
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13514) Upgrade maven surefire plugin to 2.19.1
[
https://issues.apache.org/jira/browse/HADOOP-13514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-13514:
---
Target Version/s: 3.0.0 (was: 2.9.0, 3.0.0)
No. This issue is required for migrating from JUnit 4 to 5 (HADOOP-14775) and
2.9.0 is not in the target version of HADOOP-14775.
> Upgrade maven surefire plugin to 2.19.1
> ---
>
> Key: HADOOP-13514
> URL: https://issues.apache.org/jira/browse/HADOOP-13514
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 2.8.0
>Reporter: Ewan Higgs
>Assignee: Akira Ajisaka
> Attachments: HADOOP-13514.002.patch, HADOOP-13514.003.patch,
> HADOOP-13514-addendum.01.patch, HADOOP-13514-testing.001.patch,
> HADOOP-13514-testing.002.patch, HADOOP-13514-testing.003.patch,
> HADOOP-13514-testing.004.patch, surefire-2.19.patch
>
>
> A lot of people working on Hadoop don't want to run all the tests when they
> develop; only the bits they're working on. Surefire 2.19 introduced more
> useful test filters which let us run a subset of the tests that brings the
> build time down from 'come back tomorrow' to 'grab a coffee'.
> For instance, if I only care about the S3 adaptor, I might run:
> {code}
> mvn test -Dmaven.javadoc.skip=true -Pdist,native -Djava.awt.headless=true
> \"-Dtest=org.apache.hadoop.fs.*, org.apache.hadoop.hdfs.*,
> org.apache.hadoop.fs.s3a.*\"
> {code}
> We can work around this by specifying the surefire version on the command
> line but it would be better, imo, to just update the default surefire used.
> {code}
> mvn test -Dmaven.javadoc.skip=true -Pdist,native -Djava.awt.headless=true
> \"-Dtest=org.apache.hadoop.fs.*, org.apache.hadoop.hdfs.*,
> org.apache.hadoop.fs.s3a.*\" -Dmaven-surefire-plugin.version=2.19.1
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[
https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ping Liu updated HADOOP-14600:
--
Attachment: HADOOP-14600.004.patch
=> Fixed license issue
- added ASF license header to newly added Helper.java
=> Fixed deprecation issue
- added Suppression anotation in
TestRawLocalFileSystemContract.testPermission()
=> Fixed leftover test directory and file, which should have be cleaned up
- changed to use test base directory (getTestBaseDir()) in
TestRawLocalFileSystemContract.testPermission()
- this directory will be automatically recycled in tearDown()
Please note that there are totally three test failures as follows.
{quote}Tests in error:
TestKDiag.testKeytabAndPrincipal:162->kdiag:119 ? KerberosAuth Login failure
f...
TestKDiag.testFileOutput:186->kdiag:119 ? KerberosAuth Login failure for
user:...
TestKDiag.testLoadResource:196->kdiag:119 ? KerberosAuth Login failure for
use...
Tests run: 3927, Failures: 0, Errors: 3, Skipped: 206{quote}
The failures are all from TestKDiag which is not related to HADOOP-14600.
> LocatedFileStatus constructor forces RawLocalFS to exec a process to get the
> permissions
>
>
> Key: HADOOP-14600
> URL: https://issues.apache.org/jira/browse/HADOOP-14600
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.3
> Environment: file:// in a dir with many files
>Reporter: Steve Loughran
>Assignee: Ping Liu
> Attachments: HADOOP-14600.001.patch, HADOOP-14600.002.patch,
> HADOOP-14600.003.patch, HADOOP-14600.004.patch,
> TestRawLocalFileSystemContract.java
>
>
> Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws
> against the local FS, because {{FileStatus.getPemissions}} call forces
> {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI
> values.
> That is: for every other FS, what's a field lookup or even a no-op, on the
> local FS it's a process exec/spawn, with all the costs. This gets expensive
> if you have many files.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14920) KMSClientProvider won't work with KMS delegation token retrieved from non-Java client.
[
https://issues.apache.org/jira/browse/HADOOP-14920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189111#comment-16189111
]
Hadoop QA commented on HADOOP-14920:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
48s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 32s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
49s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 44s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 2 new + 59 unchanged - 1 fixed = 61 total (was 60) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
8m 45s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
7s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 85m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HADOOP-14920 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12890064/HADOOP-14920.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 6c8c3cbf377f 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 015abcd |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13441/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13441/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13441/console |
| Powered by | Apache
[jira] [Commented] (HADOOP-12672) RPC timeout should not override IPC ping interval
[ https://issues.apache.org/jira/browse/HADOOP-12672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189105#comment-16189105 ] Masatake Iwasaki commented on HADOOP-12672: --- [~yufeigu] I think using 0 as no timeout is natural because the rpc timeout results in java.net.Socket#setSoTimeout which interpret 0 as an infinite timeout. The many part of the code seems to use the timeout value 0 as infinite timeout. While Client#getTimeout returns -1 implying that ipc.client.ping is disabled, it is remains of the past code in which rpc timeout is tightly coupled with ipc ping. I agree to improve documentation to make it clear that the rpc timeout is disabled by default (for compatibility reason). > RPC timeout should not override IPC ping interval > - > > Key: HADOOP-12672 > URL: https://issues.apache.org/jira/browse/HADOOP-12672 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Affects Versions: 2.8.0, 2.7.3, 2.6.4 >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HADOOP-12672.001.patch, HADOOP-12672.002.patch, > HADOOP-12672.003.patch, HADOOP-12672.004.patch, HADOOP-12672.005.patch, > HADOOP-12672.006.patch > > > Currently if the value of ipc.client.rpc-timeout.ms is greater than 0, the > timeout overrides the ipc.ping.interval and client will throw exception > instead of sending ping when the interval is passed. RPC timeout should work > without effectively disabling IPC ping. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13055) Implement linkMergeSlash for ViewFileSystem
[
https://issues.apache.org/jira/browse/HADOOP-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189080#comment-16189080
]
Lei (Eddy) Xu commented on HADOOP-13055:
Thanks for updates, [~manojg]. Great work.
* Should we consider move the "fallback link" from {{INodeDir}}to
{{InodeTree}}, as conceptually it is at most 1 per tree / root. It can simplify
code a bit by removing many precondition checks. For example, it can eliminate
{{isRoot}} AND {{hasFallbackLink}} checks from INodeDir.
* Could you rephrase the doc about the concept of {{internalDir}}.
* Is {{INodeDir#resolve()}} called? Can we remove it?
* Please add comments for {{static class LinkEntry }}
* for {{INodeDir#getChildren}} you might want to return a unmodified map.
* It'd be nice to raise a user readable error if mutliple MERGE_SLASH or
SINGLE_FALLBACK were configured.
* Maybe check {{mergeSlashTarget == null}} ?
{code}
if (linkType != LinkType.MERGE_SLASH) {
..
} else {
...
mergeSlashTarget = target;
}
{code}
* Test cases are awesome !
> Implement linkMergeSlash for ViewFileSystem
> ---
>
> Key: HADOOP-13055
> URL: https://issues.apache.org/jira/browse/HADOOP-13055
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, viewfs
>Affects Versions: 2.7.5
>Reporter: Zhe Zhang
>Assignee: Manoj Govindassamy
> Attachments: HADOOP-13055.00.patch, HADOOP-13055.01.patch,
> HADOOP-13055.02.patch, HADOOP-13055.03.patch, HADOOP-13055.04.patch,
> HADOOP-13055.05.patch, HADOOP-13055.06.patch, HADOOP-13055.07.patch,
> HADOOP-13055.08.patch
>
>
> In a multi-cluster environment it is sometimes useful to operate on the root
> / slash directory of an HDFS cluster. E.g., list all top level directories.
> Quoting the comment in {{ViewFs}}:
> {code}
> * A special case of the merge mount is where mount table's root is merged
> * with the root (slash) of another file system:
> *
> * fs.viewfs.mounttable.default.linkMergeSlash=hdfs://nn99/
> *
> * In this cases the root of the mount table is merged with the root of
> *hdfs://nn99/
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14920) KMSClientProvider won't work with KMS delegation token retrieved from non-Java client.
[ https://issues.apache.org/jira/browse/HADOOP-14920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HADOOP-14920: Status: Patch Available (was: Open) > KMSClientProvider won't work with KMS delegation token retrieved from > non-Java client. > -- > > Key: HADOOP-14920 > URL: https://issues.apache.org/jira/browse/HADOOP-14920 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HADOOP-14920.001.patch > > > HADOOP-13381 added support to use KMS delegation token to connect to KMS > server for key operations. However, the logic to check if the UGI container > KMS delegation token assumes that the token must contain a service attribute. > Otherwise, a KMS delegation token won't be recognized. > For delegation token obtained via non-java client such curl (http), the > default DelegationTokenAuthenticationHandler only support *renewer* parameter > and assume the client itself will add the service attribute. This makes a > java client with KMSClientProvdier can't use for KMS delegation token > retrieved form non-java client because the token does not contain a service > attribute. > I did some investigation on this and found two solutions: > 1. Similar use case exists for webhdfs, and webhdfs supports it with a > ["service" > parameter|https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Get_Delegation_Token]. > We can do this similarly by allowing client to specify a service attribute in > the request URL and included in the token returned like webhdfs. Even though > this will change in DelegationTokenAuthenticationHandler and may affect many > other web component, this seems to be a clean and low risk solution because > it will be an optional parameter. Also, other components get non-java client > interop support for free if they have the similar use case. > 2. The other way to solve this is to release the token check in > KMSClientProvider to check only the token kind instead of the service. This > is an easy work around but seems less optimal to me. > cc: [~xiaochen] for additional input. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14920) KMSClientProvider won't work with KMS delegation token retrieved from non-Java client.
[ https://issues.apache.org/jira/browse/HADOOP-14920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HADOOP-14920: Attachment: HADOOP-14920.001.patch Attach a patch that illustrated solution 1 for discussion, will add unit test later. > KMSClientProvider won't work with KMS delegation token retrieved from > non-Java client. > -- > > Key: HADOOP-14920 > URL: https://issues.apache.org/jira/browse/HADOOP-14920 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HADOOP-14920.001.patch > > > HADOOP-13381 added support to use KMS delegation token to connect to KMS > server for key operations. However, the logic to check if the UGI container > KMS delegation token assumes that the token must contain a service attribute. > Otherwise, a KMS delegation token won't be recognized. > For delegation token obtained via non-java client such curl (http), the > default DelegationTokenAuthenticationHandler only support *renewer* parameter > and assume the client itself will add the service attribute. This makes a > java client with KMSClientProvdier can't use for KMS delegation token > retrieved form non-java client because the token does not contain a service > attribute. > I did some investigation on this and found two solutions: > 1. Similar use case exists for webhdfs, and webhdfs supports it with a > ["service" > parameter|https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Get_Delegation_Token]. > We can do this similarly by allowing client to specify a service attribute in > the request URL and included in the token returned like webhdfs. Even though > this will change in DelegationTokenAuthenticationHandler and may affect many > other web component, this seems to be a clean and low risk solution because > it will be an optional parameter. Also, other components get non-java client > interop support for free if they have the similar use case. > 2. The other way to solve this is to release the token check in > KMSClientProvider to check only the token kind instead of the service. This > is an easy work around but seems less optimal to me. > cc: [~xiaochen] for additional input. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14910) Upgrade netty-all jar to latest 4.0.x.Final
[ https://issues.apache.org/jira/browse/HADOOP-14910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188993#comment-16188993 ] Jitendra Nath Pandey commented on HADOOP-14910: --- Thanks for the patch [~vinayrpet]. bq. Basically reverted changes of HDFS-6662 and HDFS-7816 which seems to be fixed automatically in netty. Is this change related to netty-all version upgrade? > Upgrade netty-all jar to latest 4.0.x.Final > --- > > Key: HADOOP-14910 > URL: https://issues.apache.org/jira/browse/HADOOP-14910 > Project: Hadoop Common > Issue Type: Bug >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Critical > Attachments: HADOOP-14910-01.patch, HADOOP-14910-02.patch > > > Upgrade netty-all jar to 4.0.37.Final version to fix latest vulnerabilities > reported. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9122) Add powermock library for writing better unit tests
[
https://issues.apache.org/jira/browse/HADOOP-9122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated HADOOP-9122:
--
Resolution: Won't Fix
Assignee: (was: Eric Yang)
Status: Resolved (was: Patch Available)
Mock provides tool chains to run simulation for a piece of code. It helps to
prevent null pointer exception, and reduce unexpected runtime exceptions. When
a piece of code is finished with a well-defined unit test, it provides great
insights to see author’s intention and reasoning to write the code. However,
everyone looks at code from a different perspective, and it is often easier to
rewrite the code than modifying and update the tests. The short coming of
writing new code, there is always danger of losing existing purpose, workaround
buried deep in the code. On the other hand, if a test program is filling with
several pages of initialization code, and override. It is hard to get context
of the test case, and easy to lose the original meaning of the test case.
Hence, there are drawback for using mock or full integration test.
I was in favor of using Powermock in favor of giving user the ability to unit
test a class and reduce external interference initially. However, I quickly
come to realization that Hadoop usage of protocol buffer serialization
technique and java reflection serialization technique have some difference
which prevents powermock to work for certain Hadoop classes.
Hadoop unit tests are written to be bigger than one class, and frequently, a
mini-cluster is spawned to test 5-10 lines of code. Any simple API test will
trigger large portion of Hadoop code to be initialized. Hadoop code base will
require too much effort to work with Powermock. Programs outside of Hadoop can
use powermock annotation to prevent mocking Hadoop classes, such as:
@powermockignore({"javax.management_", "javax.xml.", "org.w3c.",
"org.apache.hadoop._", "com.sun.*"}) . However, working in Hadoop code base,
this technique is not practical because every class in Hadoop prefix with
org.apache.hadoop. It will be heavy upkeep to maintain the list of prefix
packages that can not work with powermock reflection.
Hence, I rest my case for re-opening this issue.
> Add powermock library for writing better unit tests
> ---
>
> Key: HADOOP-9122
> URL: https://issues.apache.org/jira/browse/HADOOP-9122
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Radim Kolar
>Priority: Critical
> Attachments: HADOOP-9122.001.patch, powermock.txt
>
>
> This library is extension of mockito and easymock libraries. It allows
> testing of private methods, private variables and other otherwise untestable
> stuff.
> http://code.google.com/p/powermock/
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14921) Conflicts when starting daemons with the same name
[
https://issues.apache.org/jira/browse/HADOOP-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188817#comment-16188817
]
Íñigo Goiri commented on HADOOP-14921:
--
Thanks for the thoughts [~aw].
At this point, I think it may make sense to rename the {{Router}} from the HDFS
side.
Before I move this question to another JIRA, do you have any suggestion for
this renaming?
> Conflicts when starting daemons with the same name
> --
>
> Key: HADOOP-14921
> URL: https://issues.apache.org/jira/browse/HADOOP-14921
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Íñigo Goiri
>
> HDFS-10467 is adding a {{Router}} while YARN already added one for YARN-2915.
> Both of them are ultimately started from {{hadoop-functions.sh}}. For the PID
> file both of them end up using {{/tmp/hadoop-hadoop-router.pid}}. I propose
> to use the command name also for the PID file.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14921) Conflicts when starting daemons with the same name
[
https://issues.apache.org/jira/browse/HADOOP-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188799#comment-16188799
]
Allen Wittenauer commented on HADOOP-14921:
---
A few thoughts:
a) In production systems, the user part of the string likely won't be the same,
much less running on the same system.
b) If people REALLY want to run the two daemons on the same node as the same
user, then HADOOP_IDENT_STRING is already available to provide a way for users
to do their own differentiation, just like they can with running e.g., two
DataNode processes on the same node.
c) Or, custom entries in hadoop-user-function to do exactly what you're
proposing.
d) Having multiple, different daemons called the same thing is awful. jps,
logs, pids, documentation ... it's going to be terrible for admins. ("No, I
meant the YARN router." "No, I meant the HFDS router.") I'd HIGHLY recommend
renaming it to something else.
> Conflicts when starting daemons with the same name
> --
>
> Key: HADOOP-14921
> URL: https://issues.apache.org/jira/browse/HADOOP-14921
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Íñigo Goiri
>
> HDFS-10467 is adding a {{Router}} while YARN already added one for YARN-2915.
> Both of them are ultimately started from {{hadoop-functions.sh}}. For the PID
> file both of them end up using {{/tmp/hadoop-hadoop-router.pid}}. I propose
> to use the command name also for the PID file.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14752) TestCopyFromLocal#testCopyFromLocalWithThreads is fleaky
[ https://issues.apache.org/jira/browse/HADOOP-14752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188781#comment-16188781 ] Hudson commented on HADOOP-14752: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13005 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13005/]) HADOOP-14752. TestCopyFromLocal#testCopyFromLocalWithThreads is fleaky. (aengineer: rev 442ea85545ce3591f45211773a10346340abaed5) * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCopyFromLocal.java > TestCopyFromLocal#testCopyFromLocalWithThreads is fleaky > > > Key: HADOOP-14752 > URL: https://issues.apache.org/jira/browse/HADOOP-14752 > Project: Hadoop Common > Issue Type: Bug > Components: test >Reporter: Andras Bokor >Assignee: Andras Bokor > Fix For: 3.1.0 > > Attachments: HADOOP-14752.01.patch > > > In the test, we find out the number of threads using a random generator. When > the random number is 0/1 we call copyFromLocal with one thread that means > executor will not be used so the number of completed tasks will not equal the > number of generated files but zero. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14752) TestCopyFromLocal#testCopyFromLocalWithThreads is fleaky
[ https://issues.apache.org/jira/browse/HADOOP-14752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HADOOP-14752: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.1.0 Target Version/s: 3.1.0 (was: 3.0.0) Status: Resolved (was: Patch Available) [~msingh] Thanks for the reviews. [~boky01] Thank you for the contribution. I have committed this patch to the trunk. > TestCopyFromLocal#testCopyFromLocalWithThreads is fleaky > > > Key: HADOOP-14752 > URL: https://issues.apache.org/jira/browse/HADOOP-14752 > Project: Hadoop Common > Issue Type: Bug > Components: test >Reporter: Andras Bokor >Assignee: Andras Bokor > Fix For: 3.1.0 > > Attachments: HADOOP-14752.01.patch > > > In the test, we find out the number of threads using a random generator. When > the random number is 0/1 we call copyFromLocal with one thread that means > executor will not be used so the number of completed tasks will not equal the > number of generated files but zero. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14752) TestCopyFromLocal#testCopyFromLocalWithThreads is fleaky
[ https://issues.apache.org/jira/browse/HADOOP-14752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188739#comment-16188739 ] Anu Engineer commented on HADOOP-14752: --- +1, I will commit this shortly. > TestCopyFromLocal#testCopyFromLocalWithThreads is fleaky > > > Key: HADOOP-14752 > URL: https://issues.apache.org/jira/browse/HADOOP-14752 > Project: Hadoop Common > Issue Type: Bug > Components: test >Reporter: Andras Bokor >Assignee: Andras Bokor > Attachments: HADOOP-14752.01.patch > > > In the test, we find out the number of threads using a random generator. When > the random number is 0/1 we call copyFromLocal with one thread that means > executor will not be used so the number of completed tasks will not equal the > number of generated files but zero. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13032) Refactor FileSystem$Statistics to use StorageStatistics
[
https://issues.apache.org/jira/browse/HADOOP-13032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188695#comment-16188695
]
Andrew Wang commented on HADOOP-13032:
--
Sure, if there's a compatible way of doing this, then no problem.
> Refactor FileSystem$Statistics to use StorageStatistics
> ---
>
> Key: HADOOP-13032
> URL: https://issues.apache.org/jira/browse/HADOOP-13032
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-13032.000.patch, HADOOP-13032.001.patch,
> HADOOP-13032.002.patch, HADOOP-13032.003.patch, HADOOP-13032.004.patch,
> HADOOP-13032.005.patch
>
>
> [HADOOP-13065] added a new interface for retrieving FS and FC Statistics.
> This jira is to track the effort of moving the {{Statistics}} class out of
> {{FileSystem}}, and make it use that new interface.
> We should keep the thread local implementation. Benefits are:
> # they could be used in both {{FileContext}} and {{FileSystem}}
> # unified stats data structure
> # shorter source code
> Please note this will be an backwards-incompatible change.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13759) Split SFTP FileSystem into its own artifact
[ https://issues.apache.org/jira/browse/HADOOP-13759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188690#comment-16188690 ] Andrew Wang commented on HADOOP-13759: -- Adding new artifacts is fine, moving code to a new artifact is not okay since downstreams will need to make pom changes to retain their dependency. Related, I remember chatting with [~busbey] about how it's totally undocumented which of our Maven artifacts are public or private. We talk about hadoop-client being public and hadoop-hdfs private, but even this is only documented in JIRA, and there's no easy reference location for downstream developers. > Split SFTP FileSystem into its own artifact > --- > > Key: HADOOP-13759 > URL: https://issues.apache.org/jira/browse/HADOOP-13759 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.3 >Reporter: Andrew Wang >Assignee: Yuanbo Liu > > As discussed on HADOOP-13696, if we split the SFTP FileSystem into its own > artifact, we can save a jsch dependency in Hadoop Common. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14921) Conflicts when starting daemons with the same name
[
https://issues.apache.org/jira/browse/HADOOP-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188625#comment-16188625
]
Íñigo Goiri commented on HADOOP-14921:
--
[~aw], you have been working on the shell script part. What are your thoughts
on this?
> Conflicts when starting daemons with the same name
> --
>
> Key: HADOOP-14921
> URL: https://issues.apache.org/jira/browse/HADOOP-14921
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Íñigo Goiri
>
> HDFS-10467 is adding a {{Router}} while YARN already added one for YARN-2915.
> Both of them are ultimately started from {{hadoop-functions.sh}}. For the PID
> file both of them end up using {{/tmp/hadoop-hadoop-router.pid}}. I propose
> to use the command name also for the PID file.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14921) Conflicts when starting daemons with the same name
[
https://issues.apache.org/jira/browse/HADOOP-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HADOOP-14921:
-
Description: HDFS-10467 is adding a {{Router}} while YARN already added one
for YARN-2915. Both of them are ultimately started from
{{hadoop-functions.sh}}. For the PID file both of them end up using
{{/tmp/hadoop-hadoop-router.pid}}. I propose to use the command name also for
the PID file.
> Conflicts when starting daemons with the same name
> --
>
> Key: HADOOP-14921
> URL: https://issues.apache.org/jira/browse/HADOOP-14921
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Íñigo Goiri
>
> HDFS-10467 is adding a {{Router}} while YARN already added one for YARN-2915.
> Both of them are ultimately started from {{hadoop-functions.sh}}. For the PID
> file both of them end up using {{/tmp/hadoop-hadoop-router.pid}}. I propose
> to use the command name also for the PID file.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14921) Conflicts when starting daemons with the same name
Íñigo Goiri created HADOOP-14921: Summary: Conflicts when starting daemons with the same name Key: HADOOP-14921 URL: https://issues.apache.org/jira/browse/HADOOP-14921 Project: Hadoop Common Issue Type: Bug Reporter: Íñigo Goiri -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14919) BZip2 drops records when reading data in splits
[
https://issues.apache.org/jira/browse/HADOOP-14919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aki Tanaka updated HADOOP-14919:
Description:
BZip2 can drop records when reading data in splits. This problem was already
discussed before in HADOOP-11445 and HADOOP-13270. But we still have a problem
in corner case, causing lost data blocks.
I attached a unit test for this issue. You can reproduce the problem if you run
the unit test.
First, this issue happens when position of newly created stream is equal to
start of split. Hadoop has some test cases for this (blockEndingInCR.txt.bz2
file for TestLineRecordReader#testBzip2SplitStartAtBlockMarker, etc). However,
the issue I am reporting does not happen when we run these tests because this
issue happens only when the start of split byte block includes both block
marker and compressed data.
BZip2 block marker - 0x314159265359
(00110001010101011001001001100101001101011001)
blockEndingInCR.txt.bz2 (Start of Split - 136504):
{code:java}
$ xxd -l 6 -g 1 -b -seek 136498
./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/target/test-classes/blockEndingInCR.txt.bz2
0021532: 00110001 0101 01011001 00100110 01010011 01011001 1AY
{code}
Test bz2 File (Start of Split - 203426)
{code:java}
$ xxd -l 7 -g 1 -b -seek 203419 25.bz2
0031a9b: 11100110 00101000 00101011 00100100 11001010 01101011 .(+$.k
0031aa1: 0010 /
{code}
Let's say a job splits this test bz2 file into two splits at the start of split
(position 203426).
The former split does not read records which start position 203426 because
BZip2 says the position of these dropped records is 203427. The latter split
does not read the records because BZip2CompressionInputStream read the block
from position 320955.
Due to this behavior, records between 203427 and 320955 are lost.
Also, if we reverted the changes in HADOOP-13270, we will not see this issue.
We will see HADOOP-13270 issue though.
was:
BZip2 can drop records when reading data in splits. This problem was already
discussed before in HADOOP-11445 and HADOOP-13270. But we still have a problem
in corner case, causing lost data blocks.
I attached a unit test for this issue. You can reproduce the problem if you run
the unit test.
First, this issue happens when position of newly created stream is equal to
start of split. Hadoop has some test cases for this (blockEndingInCR.txt.bz2
file for TestLineRecordReader#testBzip2SplitStartAtBlockMarker, etc). However,
the issue I am reporting does not happen when we run these tests because this
issue happens only when the start of split byte block includes both block
marker and compressed data.
BZip2 block marker - 0x314159265359
(00110001010101011001001001100101001101011001)
blockEndingInCR.txt.bz2 (Start of Split - 136504):
{code:java}
$ xxd -l 6 -g 1 -b -seek 136498
./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/target/test-classes/blockEndingInCR.txt.bz2
0021532: 00110001 0101 01011001 00100110 01010011 01011001 1AY
{code}
Test bz2 File (Start of Split - 203426)
{code:java}
$ xxd -l 7 -g 1 -b -seek 203419 25.bz2
0031a9b: 11100110 00101000 00101011 00100100 11001010 01101011 .(+$.k
0031aa1: 0010 /
{code}
Let's say a job splits this test bz2 file into two splits at the start of split
(position 203426).
The former split does not read records which start position 203426 because
BZip2 says the position of these dropped records is 203427. The latter split
does not read the records because BZip2CompressionInputStream read the block
from position 320955.
Due to this behavior, records between 203427 and 320955 are lost.
> BZip2 drops records when reading data in splits
> ---
>
> Key: HADOOP-14919
> URL: https://issues.apache.org/jira/browse/HADOOP-14919
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Aki Tanaka
> Attachments: 25.bz2, HADOOP-14919-test.patch
>
>
> BZip2 can drop records when reading data in splits. This problem was already
> discussed before in HADOOP-11445 and HADOOP-13270. But we still have a
> problem in corner case, causing lost data blocks.
>
> I attached a unit test for this issue. You can reproduce the problem if you
> run the unit test.
>
> First, this issue happens when position of newly created stream is equal to
> start of split. Hadoop has some test cases for this (blockEndingInCR.txt.bz2
> file for TestLineRecordReader#testBzip2SplitStartAtBlockMarker, etc).
> However, the issue I am reporting does not happen when we run these tests
> because this issue happens only when the start of split byte block includes
> both block marker and compressed data.
[jira] [Commented] (HADOOP-14915) method name is incorrect in ConfServlet
[ https://issues.apache.org/jira/browse/HADOOP-14915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188549#comment-16188549 ] Bharat Viswanadham commented on HADOOP-14915: - Thank You [~vagarychen] for committing the patch. > method name is incorrect in ConfServlet > --- > > Key: HADOOP-14915 > URL: https://issues.apache.org/jira/browse/HADOOP-14915 > Project: Hadoop Common > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Minor > Fix For: 3.0.0 > > Attachments: HADOOP-14915.00.patch > > > method name is parseAccecptHeader. > Modify it as parseAcceptHeader -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14915) method name is incorrect in ConfServlet
[ https://issues.apache.org/jira/browse/HADOOP-14915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188550#comment-16188550 ] Hudson commented on HADOOP-14915: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13002 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13002/]) HADOOP-14915. method name is incorrect in ConfServlet. Contributed by (cliang: rev 563dcdfc1de7ea9ee7ce296163cf2678dfe5349c) * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfServlet.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/ConfServlet.java > method name is incorrect in ConfServlet > --- > > Key: HADOOP-14915 > URL: https://issues.apache.org/jira/browse/HADOOP-14915 > Project: Hadoop Common > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Minor > Fix For: 3.0.0 > > Attachments: HADOOP-14915.00.patch > > > method name is parseAccecptHeader. > Modify it as parseAcceptHeader -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14915) method name is incorrect in ConfServlet
[ https://issues.apache.org/jira/browse/HADOOP-14915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HADOOP-14915: Resolution: Fixed Fix Version/s: 3.0.0 Target Version/s: 3.0.0 Status: Resolved (was: Patch Available) > method name is incorrect in ConfServlet > --- > > Key: HADOOP-14915 > URL: https://issues.apache.org/jira/browse/HADOOP-14915 > Project: Hadoop Common > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Minor > Fix For: 3.0.0 > > Attachments: HADOOP-14915.00.patch > > > method name is parseAccecptHeader. > Modify it as parseAcceptHeader -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14915) method name is incorrect in ConfServlet
[ https://issues.apache.org/jira/browse/HADOOP-14915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188520#comment-16188520 ] Chen Liang commented on HADOOP-14915: - +1 on v00 patch. I've committed to trunk, thanks [~bharatviswa] for the contribution! > method name is incorrect in ConfServlet > --- > > Key: HADOOP-14915 > URL: https://issues.apache.org/jira/browse/HADOOP-14915 > Project: Hadoop Common > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Minor > Attachments: HADOOP-14915.00.patch > > > method name is parseAccecptHeader. > Modify it as parseAcceptHeader -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14920) KMSClientProvider won't work with KMS delegation token retrieved from non-Java client.
Xiaoyu Yao created HADOOP-14920: --- Summary: KMSClientProvider won't work with KMS delegation token retrieved from non-Java client. Key: HADOOP-14920 URL: https://issues.apache.org/jira/browse/HADOOP-14920 Project: Hadoop Common Issue Type: Bug Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao HADOOP-13381 added support to use KMS delegation token to connect to KMS server for key operations. However, the logic to check if the UGI container KMS delegation token assumes that the token must contain a service attribute. Otherwise, a KMS delegation token won't be recognized. For delegation token obtained via non-java client such curl (http), the default DelegationTokenAuthenticationHandler only support *renewer* parameter and assume the client itself will add the service attribute. This makes a java client with KMSClientProvdier can't use for KMS delegation token retrieved form non-java client because the token does not contain a service attribute. I did some investigation on this and found two solutions: 1. Similar use case exists for webhdfs, and webhdfs supports it with a ["service" parameter|https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Get_Delegation_Token]. We can do this similarly by allowing client to specify a service attribute in the request URL and included in the token returned like webhdfs. Even though this will change in DelegationTokenAuthenticationHandler and may affect many other web component, this seems to be a clean and low risk solution because it will be an optional parameter. Also, other components get non-java client interop support for free if they have the similar use case. 2. The other way to solve this is to release the token check in KMSClientProvider to check only the token kind instead of the service. This is an easy work around but seems less optimal to me. cc: [~xiaochen] for additional input. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14919) BZip2 drops records when reading data in splits
[
https://issues.apache.org/jira/browse/HADOOP-14919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aki Tanaka updated HADOOP-14919:
Attachment: 25.bz2
Adding the test bz2 file (The bz2 file that the attached unit test generates)
> BZip2 drops records when reading data in splits
> ---
>
> Key: HADOOP-14919
> URL: https://issues.apache.org/jira/browse/HADOOP-14919
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Aki Tanaka
> Attachments: 25.bz2, HADOOP-14919-test.patch
>
>
> BZip2 can drop records when reading data in splits. This problem was already
> discussed before in HADOOP-11445 and HADOOP-13270. But we still have a
> problem in corner case, causing lost data blocks.
>
> I attached a unit test for this issue. You can reproduce the problem if you
> run the unit test.
>
> First, this issue happens when position of newly created stream is equal to
> start of split. Hadoop has some test cases for this (blockEndingInCR.txt.bz2
> file for TestLineRecordReader#testBzip2SplitStartAtBlockMarker, etc).
> However, the issue I am reporting does not happen when we run these tests
> because this issue happens only when the start of split byte block includes
> both block marker and compressed data.
>
> BZip2 block marker - 0x314159265359
> (00110001010101011001001001100101001101011001)
>
> blockEndingInCR.txt.bz2 (Start of Split - 136504):
> {code:java}
> $ xxd -l 6 -g 1 -b -seek 136498
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/target/test-classes/blockEndingInCR.txt.bz2
> 0021532: 00110001 0101 01011001 00100110 01010011 01011001 1AY
> {code}
>
> Test bz2 File (Start of Split - 203426)
> {code:java}
> $ xxd -l 7 -g 1 -b -seek 203419 25.bz2
> 0031a9b: 11100110 00101000 00101011 00100100 11001010 01101011 .(+$.k
> 0031aa1: 0010 /
> {code}
>
> Let's say a job splits this test bz2 file into two splits at the start of
> split (position 203426).
> The former split does not read records which start position 203426 because
> BZip2 says the position of these dropped records is 203427. The latter split
> does not read the records because BZip2CompressionInputStream read the block
> from position 320955.
> Due to this behavior, records between 203427 and 320955 are lost.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14919) BZip2 drops records when reading data in splits
[
https://issues.apache.org/jira/browse/HADOOP-14919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aki Tanaka updated HADOOP-14919:
Attachment: HADOOP-14919-test.patch
Add patch for the unit test.
> BZip2 drops records when reading data in splits
> ---
>
> Key: HADOOP-14919
> URL: https://issues.apache.org/jira/browse/HADOOP-14919
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Aki Tanaka
> Attachments: HADOOP-14919-test.patch
>
>
> BZip2 can drop records when reading data in splits. This problem was already
> discussed before in HADOOP-11445 and HADOOP-13270. But we still have a
> problem in corner case, causing lost data blocks.
>
> I attached a unit test for this issue. You can reproduce the problem if you
> run the unit test.
>
> First, this issue happens when position of newly created stream is equal to
> start of split. Hadoop has some test cases for this (blockEndingInCR.txt.bz2
> file for TestLineRecordReader#testBzip2SplitStartAtBlockMarker, etc).
> However, the issue I am reporting does not happen when we run these tests
> because this issue happens only when the start of split byte block includes
> both block marker and compressed data.
>
> BZip2 block marker - 0x314159265359
> (00110001010101011001001001100101001101011001)
>
> blockEndingInCR.txt.bz2 (Start of Split - 136504):
> {code:java}
> $ xxd -l 6 -g 1 -b -seek 136498
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/target/test-classes/blockEndingInCR.txt.bz2
> 0021532: 00110001 0101 01011001 00100110 01010011 01011001 1AY
> {code}
>
> Test bz2 File (Start of Split - 203426)
> {code:java}
> $ xxd -l 7 -g 1 -b -seek 203419 25.bz2
> 0031a9b: 11100110 00101000 00101011 00100100 11001010 01101011 .(+$.k
> 0031aa1: 0010 /
> {code}
>
> Let's say a job splits this test bz2 file into two splits at the start of
> split (position 203426).
> The former split does not read records which start position 203426 because
> BZip2 says the position of these dropped records is 203427. The latter split
> does not read the records because BZip2CompressionInputStream read the block
> from position 320955.
> Due to this behavior, records between 203427 and 320955 are lost.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14919) BZip2 drops records when reading data in splits
Aki Tanaka created HADOOP-14919:
---
Summary: BZip2 drops records when reading data in splits
Key: HADOOP-14919
URL: https://issues.apache.org/jira/browse/HADOOP-14919
Project: Hadoop Common
Issue Type: Bug
Reporter: Aki Tanaka
BZip2 can drop records when reading data in splits. This problem was already
discussed before in HADOOP-11445 and HADOOP-13270. But we still have a problem
in corner case, causing lost data blocks.
I attached a unit test for this issue. You can reproduce the problem if you run
the unit test.
First, this issue happens when position of newly created stream is equal to
start of split. Hadoop has some test cases for this (blockEndingInCR.txt.bz2
file for TestLineRecordReader#testBzip2SplitStartAtBlockMarker, etc). However,
the issue I am reporting does not happen when we run these tests because this
issue happens only when the start of split byte block includes both block
marker and compressed data.
BZip2 block marker - 0x314159265359
(00110001010101011001001001100101001101011001)
blockEndingInCR.txt.bz2 (Start of Split - 136504):
{code:java}
$ xxd -l 6 -g 1 -b -seek 136498
./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/target/test-classes/blockEndingInCR.txt.bz2
0021532: 00110001 0101 01011001 00100110 01010011 01011001 1AY
{code}
Test bz2 File (Start of Split - 203426)
{code:java}
$ xxd -l 7 -g 1 -b -seek 203419 25.bz2
0031a9b: 11100110 00101000 00101011 00100100 11001010 01101011 .(+$.k
0031aa1: 0010 /
{code}
Let's say a job splits this test bz2 file into two splits at the start of split
(position 203426).
The former split does not read records which start position 203426 because
BZip2 says the position of these dropped records is 203427. The latter split
does not read the records because BZip2CompressionInputStream read the block
from position 320955.
Due to this behavior, records between 203427 and 320955 are lost.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11715) azureFs::getFileStatus doesn't check the file system scheme and thus could throw a misleading exception.
[
https://issues.apache.org/jira/browse/HADOOP-11715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188111#comment-16188111
]
Hadoop QA commented on HADOOP-11715:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HADOOP-11715 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-11715 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12729101/HADOOP-11715.3.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13440/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> azureFs::getFileStatus doesn't check the file system scheme and thus could
> throw a misleading exception.
> -
>
> Key: HADOOP-11715
> URL: https://issues.apache.org/jira/browse/HADOOP-11715
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/azure
>Affects Versions: 2.7.0
>Reporter: Brandon Li
>Assignee: nijel
> Labels: BB2015-05-TBR
> Attachments: HADOOP-11715.1.patch, HADOOP-11715.2.patch,
> HADOOP-11715.3.patch
>
>
> azureFs::getFileStatus doesn't check the file system scheme and thus could
> throw a misleading exception.
> For example, it complains filenotfound instead of wrong-fs for an hdfs path:
> Caused by: java.io.FileNotFoundException:
> hdfs://headnode0:9000/hive/scratch/hadoopqa/a7d34a22-57eb-4678-84b4-43d84027d45f/hive_2015-03-02_23-13-04_713_5722627238053417441-1/hadoopqa/_tez_scratch_dir/_tez_scratch_dir/split_Map_1/job.split:
> No such file or directory.
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1625)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11715) azureFs::getFileStatus doesn't check the file system scheme and thus could throw a misleading exception.
[
https://issues.apache.org/jira/browse/HADOOP-11715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188104#comment-16188104
]
Steve Loughran commented on HADOOP-11715:
-
Not seen this before.
I think just calling {{Path p2 = qualify(path)}} would be enough to do the
check as part of making references absolute
> azureFs::getFileStatus doesn't check the file system scheme and thus could
> throw a misleading exception.
> -
>
> Key: HADOOP-11715
> URL: https://issues.apache.org/jira/browse/HADOOP-11715
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/azure
>Affects Versions: 2.7.0
>Reporter: Brandon Li
>Assignee: nijel
> Labels: BB2015-05-TBR
> Attachments: HADOOP-11715.1.patch, HADOOP-11715.2.patch,
> HADOOP-11715.3.patch
>
>
> azureFs::getFileStatus doesn't check the file system scheme and thus could
> throw a misleading exception.
> For example, it complains filenotfound instead of wrong-fs for an hdfs path:
> Caused by: java.io.FileNotFoundException:
> hdfs://headnode0:9000/hive/scratch/hadoopqa/a7d34a22-57eb-4678-84b4-43d84027d45f/hive_2015-03-02_23-13-04_713_5722627238053417441-1/hadoopqa/_tez_scratch_dir/_tez_scratch_dir/split_Map_1/job.split:
> No such file or directory.
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1625)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11715) azureFs::getFileStatus doesn't check the file system scheme and thus could throw a misleading exception.
[ https://issues.apache.org/jira/browse/HADOOP-11715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-11715: Issue Type: Sub-task (was: Bug) Parent: HADOOP-14552 > azureFs::getFileStatus doesn't check the file system scheme and thus could > throw a misleading exception. > - > > Key: HADOOP-11715 > URL: https://issues.apache.org/jira/browse/HADOOP-11715 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/azure >Affects Versions: 2.7.0 >Reporter: Brandon Li >Assignee: nijel > Labels: BB2015-05-TBR > Attachments: HADOOP-11715.1.patch, HADOOP-11715.2.patch, > HADOOP-11715.3.patch > > > azureFs::getFileStatus doesn't check the file system scheme and thus could > throw a misleading exception. > For example, it complains filenotfound instead of wrong-fs for an hdfs path: > Caused by: java.io.FileNotFoundException: > hdfs://headnode0:9000/hive/scratch/hadoopqa/a7d34a22-57eb-4678-84b4-43d84027d45f/hive_2015-03-02_23-13-04_713_5722627238053417441-1/hadoopqa/_tez_scratch_dir/_tez_scratch_dir/split_Map_1/job.split: > No such file or directory. > at > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1625) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11715) azureFs::getFileStatus doesn't check the file system scheme and thus could throw a misleading exception.
[ https://issues.apache.org/jira/browse/HADOOP-11715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-11715: Component/s: fs/azure > azureFs::getFileStatus doesn't check the file system scheme and thus could > throw a misleading exception. > - > > Key: HADOOP-11715 > URL: https://issues.apache.org/jira/browse/HADOOP-11715 > Project: Hadoop Common > Issue Type: Bug > Components: fs, fs/azure >Affects Versions: 2.7.0 >Reporter: Brandon Li >Assignee: nijel > Labels: BB2015-05-TBR > Attachments: HADOOP-11715.1.patch, HADOOP-11715.2.patch, > HADOOP-11715.3.patch > > > azureFs::getFileStatus doesn't check the file system scheme and thus could > throw a misleading exception. > For example, it complains filenotfound instead of wrong-fs for an hdfs path: > Caused by: java.io.FileNotFoundException: > hdfs://headnode0:9000/hive/scratch/hadoopqa/a7d34a22-57eb-4678-84b4-43d84027d45f/hive_2015-03-02_23-13-04_713_5722627238053417441-1/hadoopqa/_tez_scratch_dir/_tez_scratch_dir/split_Map_1/job.split: > No such file or directory. > at > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1625) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14845) Azure wasb: getFileStatus not making any auth checks
[
https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188056#comment-16188056
]
Steve Loughran edited comment on HADOOP-14845 at 10/2/17 1:53 PM:
--
its in branch-2, it's the forward port to trunk that's problematic...I'm
awaiting a submission of a new patch there & test results
was (Author: ste...@apache.org):
its in branch-2, it's the forward port to trunk that's problematic
> Azure wasb: getFileStatus not making any auth checks
>
>
> Key: HADOOP-14845
> URL: https://issues.apache.org/jira/browse/HADOOP-14845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure, security
>Affects Versions: 2.8.0, 2.7.4
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Fix For: 2.9.0
>
> Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch,
> HADOOP-14845.003.patch, HADOOP-14845-branch-2-001.patch.txt,
> HADOOP-14845-branch-2-002.patch, HADOOP-14845-branch-2-003.patch
>
>
> The HDFS spec requires only traverse checks for any file accessed via
> getFileStatus ... and since WASB does not support traverse checks, removing
> this call effectively removed all protections for the getFileStatus call. The
> reasoning at that time was that doing a performAuthCheck was the wrong thing
> to do, since it was going against the specand that the correct fix to the
> getFileStatus issue was to implement traverse checks rather than go against
> the spec by calling performAuthCheck. The side-effects of such a change were
> not fully clear at that time, but the thinking was that it was safer to
> remain true to the spec, as far as possible.
> The reasoning remains correct even today. But in view of the security hole
> introduced by this change (that anyone can load up any other user's data in
> hive), and keeping in mind that WASB does not intend to implement traverse
> checks, we propose a compromise.
> We propose (re)introducing a read-access check to getFileStatus(), that would
> check the existing ancestor for read-access whenever invoked. Although not
> perfect (in that it is a departure from the spec), we believe that it is a
> good compromise between having no checks at all; and implementing full-blown
> traverse checks.
> For scenarios that deal with intermediate folders like mkdirs, the call would
> check for read access against an existing ancestor (when invoked from shell)
> for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/"
> exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar"
> }}. This can be thought of, as being a close-enough substitute for the
> traverse checks that hdfs does.
> For other scenarios that don't deal with non-existent intermediate folders –
> like read, delete etc, the check will happen against the parent. Once again,
> we can think of the read-check against the parent as a substitute for the
> traverse check, which can be customized for various users with ranger
> policies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14845) Azure wasb: getFileStatus not making any auth checks
[
https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14845:
Fix Version/s: 2.9.0
> Azure wasb: getFileStatus not making any auth checks
>
>
> Key: HADOOP-14845
> URL: https://issues.apache.org/jira/browse/HADOOP-14845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure, security
>Affects Versions: 2.8.0, 2.7.4
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Fix For: 2.9.0
>
> Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch,
> HADOOP-14845.003.patch, HADOOP-14845-branch-2-001.patch.txt,
> HADOOP-14845-branch-2-002.patch, HADOOP-14845-branch-2-003.patch
>
>
> The HDFS spec requires only traverse checks for any file accessed via
> getFileStatus ... and since WASB does not support traverse checks, removing
> this call effectively removed all protections for the getFileStatus call. The
> reasoning at that time was that doing a performAuthCheck was the wrong thing
> to do, since it was going against the specand that the correct fix to the
> getFileStatus issue was to implement traverse checks rather than go against
> the spec by calling performAuthCheck. The side-effects of such a change were
> not fully clear at that time, but the thinking was that it was safer to
> remain true to the spec, as far as possible.
> The reasoning remains correct even today. But in view of the security hole
> introduced by this change (that anyone can load up any other user's data in
> hive), and keeping in mind that WASB does not intend to implement traverse
> checks, we propose a compromise.
> We propose (re)introducing a read-access check to getFileStatus(), that would
> check the existing ancestor for read-access whenever invoked. Although not
> perfect (in that it is a departure from the spec), we believe that it is a
> good compromise between having no checks at all; and implementing full-blown
> traverse checks.
> For scenarios that deal with intermediate folders like mkdirs, the call would
> check for read access against an existing ancestor (when invoked from shell)
> for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/"
> exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar"
> }}. This can be thought of, as being a close-enough substitute for the
> traverse checks that hdfs does.
> For other scenarios that don't deal with non-existent intermediate folders –
> like read, delete etc, the check will happen against the parent. Once again,
> we can think of the read-check against the parent as a substitute for the
> traverse check, which can be customized for various users with ranger
> policies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14845) Azure wasb: getFileStatus not making any auth checks
[
https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188056#comment-16188056
]
Steve Loughran commented on HADOOP-14845:
-
its in branch-2, it's the forward port to trunk that's problematic
> Azure wasb: getFileStatus not making any auth checks
>
>
> Key: HADOOP-14845
> URL: https://issues.apache.org/jira/browse/HADOOP-14845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure, security
>Affects Versions: 2.8.0, 2.7.4
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch,
> HADOOP-14845.003.patch, HADOOP-14845-branch-2-001.patch.txt,
> HADOOP-14845-branch-2-002.patch, HADOOP-14845-branch-2-003.patch
>
>
> The HDFS spec requires only traverse checks for any file accessed via
> getFileStatus ... and since WASB does not support traverse checks, removing
> this call effectively removed all protections for the getFileStatus call. The
> reasoning at that time was that doing a performAuthCheck was the wrong thing
> to do, since it was going against the specand that the correct fix to the
> getFileStatus issue was to implement traverse checks rather than go against
> the spec by calling performAuthCheck. The side-effects of such a change were
> not fully clear at that time, but the thinking was that it was safer to
> remain true to the spec, as far as possible.
> The reasoning remains correct even today. But in view of the security hole
> introduced by this change (that anyone can load up any other user's data in
> hive), and keeping in mind that WASB does not intend to implement traverse
> checks, we propose a compromise.
> We propose (re)introducing a read-access check to getFileStatus(), that would
> check the existing ancestor for read-access whenever invoked. Although not
> perfect (in that it is a departure from the spec), we believe that it is a
> good compromise between having no checks at all; and implementing full-blown
> traverse checks.
> For scenarios that deal with intermediate folders like mkdirs, the call would
> check for read access against an existing ancestor (when invoked from shell)
> for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/"
> exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar"
> }}. This can be thought of, as being a close-enough substitute for the
> traverse checks that hdfs does.
> For other scenarios that don't deal with non-existent intermediate folders –
> like read, delete etc, the check will happen against the parent. Once again,
> we can think of the read-check against the parent as a substitute for the
> traverse check, which can be customized for various users with ranger
> policies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-14768. - Resolution: Fixed Fix Version/s: 2.9.0 Target Version/s: 2.9.0, 3.1.0 (was: 2.9.0, 3.0.0) Committed to 3.1 & cherry picked into branch-2 If you need it in 3.0, # reopen the JIRA, # apply the trunk patch to branch-3 # rerun all the hadoop-azure tests # if the tests pass, add a patch with the branch-3-011.patch style of marker, resubmit, with a declaration of endpoint. thanks > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.8.1 >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Fix For: 2.9.0 > > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch, > HADOOP-14768.004.patch, HADOOP-14768.005.patch, HADOOP-14768.006.patch, > HADOOP-14768.007.patch, HADOOP-14768-branch-2-008.patch, > HADOOP-14768-branch-2-009.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14768: Status: Open (was: Patch Available) > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.8.1 >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch, > HADOOP-14768.004.patch, HADOOP-14768.005.patch, HADOOP-14768.006.patch, > HADOOP-14768.007.patch, HADOOP-14768-branch-2-008.patch, > HADOOP-14768-branch-2-009.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16187990#comment-16187990 ] Steve Loughran commented on HADOOP-14768: - yes, though I think there's been some toe-stamping between different bits of code along with confusion as to who got to the backport > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.8.1 >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch, > HADOOP-14768.004.patch, HADOOP-14768.005.patch, HADOOP-14768.006.patch, > HADOOP-14768.007.patch, HADOOP-14768-branch-2-008.patch, > HADOOP-14768-branch-2-009.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
