[jira] [Updated] (HDFS-15674) TestBPOfferService#testMissBlocksWhenReregister fails on trunk
[ https://issues.apache.org/jira/browse/HDFS-15674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HDFS-15674: Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) > TestBPOfferService#testMissBlocksWhenReregister fails on trunk > -- > > Key: HDFS-15674 > URL: https://issues.apache.org/jira/browse/HDFS-15674 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ahmed Hussein >Assignee: Masatake Iwasaki >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h > Remaining Estimate: 0h > > qbt report (Nov 8, 2020, 11:28 AM) shows failures timing out in > testMissBlocksWhenReregister -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15674) TestBPOfferService#testMissBlocksWhenReregister fails on trunk
[ https://issues.apache.org/jira/browse/HDFS-15674?focusedWorklogId=513399=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-513399 ] ASF GitHub Bot logged work on HDFS-15674: - Author: ASF GitHub Bot Created on: 18/Nov/20 07:11 Start Date: 18/Nov/20 07:11 Worklog Time Spent: 10m Work Description: iwasakims merged pull request #2467: URL: https://github.com/apache/hadoop/pull/2467 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 513399) Time Spent: 50m (was: 40m) > TestBPOfferService#testMissBlocksWhenReregister fails on trunk > -- > > Key: HDFS-15674 > URL: https://issues.apache.org/jira/browse/HDFS-15674 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ahmed Hussein >Assignee: Masatake Iwasaki >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > qbt report (Nov 8, 2020, 11:28 AM) shows failures timing out in > testMissBlocksWhenReregister -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15674) TestBPOfferService#testMissBlocksWhenReregister fails on trunk
[ https://issues.apache.org/jira/browse/HDFS-15674?focusedWorklogId=513400=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-513400 ] ASF GitHub Bot logged work on HDFS-15674: - Author: ASF GitHub Bot Created on: 18/Nov/20 07:11 Start Date: 18/Nov/20 07:11 Worklog Time Spent: 10m Work Description: iwasakims commented on pull request #2467: URL: https://github.com/apache/hadoop/pull/2467#issuecomment-729483923 I merged this. Thanks, @Hexiaoqiao and @goiri . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 513400) Time Spent: 1h (was: 50m) > TestBPOfferService#testMissBlocksWhenReregister fails on trunk > -- > > Key: HDFS-15674 > URL: https://issues.apache.org/jira/browse/HDFS-15674 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ahmed Hussein >Assignee: Masatake Iwasaki >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > qbt report (Nov 8, 2020, 11:28 AM) shows failures timing out in > testMissBlocksWhenReregister -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15674) TestBPOfferService#testMissBlocksWhenReregister fails on trunk
[ https://issues.apache.org/jira/browse/HDFS-15674?focusedWorklogId=513390=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-513390 ] ASF GitHub Bot logged work on HDFS-15674: - Author: ASF GitHub Bot Created on: 18/Nov/20 06:56 Start Date: 18/Nov/20 06:56 Worklog Time Spent: 10m Work Description: Hexiaoqiao commented on pull request #2467: URL: https://github.com/apache/hadoop/pull/2467#issuecomment-729475534 Thanks @iwasakims for your works. +1 from my side. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 513390) Time Spent: 40m (was: 0.5h) > TestBPOfferService#testMissBlocksWhenReregister fails on trunk > -- > > Key: HDFS-15674 > URL: https://issues.apache.org/jira/browse/HDFS-15674 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ahmed Hussein >Assignee: Masatake Iwasaki >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > qbt report (Nov 8, 2020, 11:28 AM) shows failures timing out in > testMissBlocksWhenReregister -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15672) TestBalancerWithMultipleNameNodes#testBalancingBlockpoolsWithBlockPoolPolicy fails on trunk
[
https://issues.apache.org/jira/browse/HDFS-15672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki reassigned HDFS-15672:
---
Assignee: Masatake Iwasaki
> TestBalancerWithMultipleNameNodes#testBalancingBlockpoolsWithBlockPoolPolicy
> fails on trunk
> ---
>
> Key: HDFS-15672
> URL: https://issues.apache.org/jira/browse/HDFS-15672
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ahmed Hussein
>Assignee: Masatake Iwasaki
>Priority: Major
>
> qbt report shows the following error:
> {code:bash}
> org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes.testBalancingBlockpoolsWithBlockPoolPolicy
> Failing for the past 1 build (Since Failed#317 )
> Took 10 min.
> Error Message
> test timed out after 60 milliseconds
> Stacktrace
> org.junit.runners.model.TestTimedOutException: test timed out after 60
> milliseconds
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes.sleep(TestBalancerWithMultipleNameNodes.java:353)
> at
> org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes.wait(TestBalancerWithMultipleNameNodes.java:159)
> at
> org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes.runBalancer(TestBalancerWithMultipleNameNodes.java:175)
> at
> org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes.runTest(TestBalancerWithMultipleNameNodes.java:550)
> at
> org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes.testBalancingBlockpoolsWithBlockPoolPolicy(TestBalancerWithMultipleNameNodes.java:609)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15674) TestBPOfferService#testMissBlocksWhenReregister fails on trunk
[
https://issues.apache.org/jira/browse/HDFS-15674?focusedWorklogId=513038=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-513038
]
ASF GitHub Bot logged work on HDFS-15674:
-
Author: ASF GitHub Bot
Created on: 17/Nov/20 17:04
Start Date: 17/Nov/20 17:04
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2467:
URL: https://github.com/apache/hadoop/pull/2467#issuecomment-729065972
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 26m 14s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 48s | | trunk passed |
| +1 :green_heart: | compile | 1m 19s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 1m 13s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 49s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 18s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 35s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 59s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 3m 6s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 4s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| +1 :green_heart: | compile | 1m 8s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 1m 8s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 40s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 11s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 49s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 1m 21s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 3m 1s | | the patch passed |
_ Other Tests _ |
| -1 :x: | unit | 100m 16s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2467/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 219m 57s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate |
| | hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR |
| | hadoop.hdfs.TestDecommissionWithBackoffMonitor |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2467/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2467 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux f861655c6c83 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b57f04cd5bf |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
[jira] [Created] (HDFS-15687) allowSnapshot fails when directory already has a Trash sub directory
Shashikant Banerjee created HDFS-15687:
--
Summary: allowSnapshot fails when directory already has a Trash
sub directory
Key: HDFS-15687
URL: https://issues.apache.org/jira/browse/HDFS-15687
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Nilotpal Nandi
Assignee: Shashikant Banerjee
Steps :
1. Create an encryption zone , Trash directory would be created inside EZ
directory.
/opt/cloudera/parcels/CDH/bin/hdfs crypto -createZone -keyName
testkeysnapshot1605613314 -path /user/hrt_6/test_dir1/snap_encrypt_dir1605613504
2. Try to make the EZ directory snapshottable.
/opt/cloudera/parcels/CDH/bin/hdfs dfsadmin -allowSnapshot
/user/hrt_6/test_dir1/snap_encrypt_dir1605613504
It fails with error :
{noformat}
/opt/cloudera/parcels/CDH/bin/hdfs dfsadmin -allowSnapshot
/user/hrt_6/test_dir1/snap_encrypt_dir1605613504
2020-11-17 11:45:16,598|INFO|MainThread|machine.py:180 -
run()||GUID=b35fc918-ed08-4c5d-92c1-c5aab449fb10|allowSnapshot: Can't provision
trash for snapshottable directory
/user/hrt_6/test_dir1/snap_encrypt_dir1605613504 because trash path
/user/hrt_6/test_dir1/snap_encrypt_dir1605613504/.Trash already exists.
2020-11-17 11:45:16,956|INFO|MainThread|machine.py:209 -
run()||GUID=b35fc918-ed08-4c5d-92c1-c5aab449fb10|Exit Code: 255{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15674) TestBPOfferService#testMissBlocksWhenReregister fails on trunk
[ https://issues.apache.org/jira/browse/HDFS-15674?focusedWorklogId=512891=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-512891 ] ASF GitHub Bot logged work on HDFS-15674: - Author: ASF GitHub Bot Created on: 17/Nov/20 13:47 Start Date: 17/Nov/20 13:47 Worklog Time Spent: 10m Work Description: iwasakims commented on pull request #2467: URL: https://github.com/apache/hadoop/pull/2467#issuecomment-728938518 The failure could be reproduced within 10 tries on my local environment. I checked that pending IBRs are cleared before sending heartbeat in failed case by adding debug logging to `IncrementalBlockReportManager#clearIBRs`. (The logging is not including the PR.) ``` $ cd hadoop-hdfs-project/hadoop-hdfs $ for i in `seq 100` ; do echo $i && mvn test -Dtest=TestBPOfferService#testMissBlocksWhenReregister -DignoreTestFailure=false || break ; done ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 512891) Time Spent: 20m (was: 10m) > TestBPOfferService#testMissBlocksWhenReregister fails on trunk > -- > > Key: HDFS-15674 > URL: https://issues.apache.org/jira/browse/HDFS-15674 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ahmed Hussein >Assignee: Masatake Iwasaki >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > qbt report (Nov 8, 2020, 11:28 AM) shows failures timing out in > testMissBlocksWhenReregister -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15674) TestBPOfferService#testMissBlocksWhenReregister fails on trunk
[
https://issues.apache.org/jira/browse/HDFS-15674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233582#comment-17233582
]
Masatake Iwasaki commented on HDFS-15674:
-
The test intermittently fails if count of reported block in IBRs does not
reached to expected value(4000).
{noformat}
[ERROR] Failures:
[ERROR] TestBPOfferService.testMissBlocksWhenReregister:341 Timed out wait
for IBR counts FBRCount = 2441, IBRCount = 3999; expected = 4000. Exception:
Timed out waiting for condition. Thread diagnostics:
Timestamp: 2020-11-16 12:45:57,425
{noformat}
This waiting condition does not make sense because pending IBRs are possible to
be deleted by {{clearIBRs()}} in {{BPServiceActor#reRegister}}. It is ok
because succeeding full block report covers them based on the discussion of
HDFS-15113.
The tests should check that all blocks are reported rather than counting number
of reported blocks in FBR/IBR independently.
I submitted [PR #2467|https://github.com/apache/hadoop/pull/2467] for this.
> TestBPOfferService#testMissBlocksWhenReregister fails on trunk
> --
>
> Key: HDFS-15674
> URL: https://issues.apache.org/jira/browse/HDFS-15674
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ahmed Hussein
>Assignee: Masatake Iwasaki
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> qbt report (Nov 8, 2020, 11:28 AM) shows failures timing out in
> testMissBlocksWhenReregister
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15674) TestBPOfferService#testMissBlocksWhenReregister fails on trunk
[ https://issues.apache.org/jira/browse/HDFS-15674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HDFS-15674: Status: Patch Available (was: Open) > TestBPOfferService#testMissBlocksWhenReregister fails on trunk > -- > > Key: HDFS-15674 > URL: https://issues.apache.org/jira/browse/HDFS-15674 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ahmed Hussein >Assignee: Masatake Iwasaki >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > qbt report (Nov 8, 2020, 11:28 AM) shows failures timing out in > testMissBlocksWhenReregister -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15674) TestBPOfferService#testMissBlocksWhenReregister fails on trunk
[ https://issues.apache.org/jira/browse/HDFS-15674?focusedWorklogId=512886=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-512886 ] ASF GitHub Bot logged work on HDFS-15674: - Author: ASF GitHub Bot Created on: 17/Nov/20 13:22 Start Date: 17/Nov/20 13:22 Worklog Time Spent: 10m Work Description: iwasakims opened a new pull request #2467: URL: https://github.com/apache/hadoop/pull/2467 https://issues.apache.org/jira/browse/HDFS-15674 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 512886) Remaining Estimate: 0h Time Spent: 10m > TestBPOfferService#testMissBlocksWhenReregister fails on trunk > -- > > Key: HDFS-15674 > URL: https://issues.apache.org/jira/browse/HDFS-15674 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ahmed Hussein >Assignee: Masatake Iwasaki >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > qbt report (Nov 8, 2020, 11:28 AM) shows failures timing out in > testMissBlocksWhenReregister -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15674) TestBPOfferService#testMissBlocksWhenReregister fails on trunk
[ https://issues.apache.org/jira/browse/HDFS-15674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-15674: -- Labels: pull-request-available (was: ) > TestBPOfferService#testMissBlocksWhenReregister fails on trunk > -- > > Key: HDFS-15674 > URL: https://issues.apache.org/jira/browse/HDFS-15674 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ahmed Hussein >Assignee: Masatake Iwasaki >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > qbt report (Nov 8, 2020, 11:28 AM) shows failures timing out in > testMissBlocksWhenReregister -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15635) ViewFileSystemOverloadScheme support specifying mount table loader imp through conf
[
https://issues.apache.org/jira/browse/HDFS-15635?focusedWorklogId=512856=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-512856
]
ASF GitHub Bot logged work on HDFS-15635:
-
Author: ASF GitHub Bot
Created on: 17/Nov/20 11:49
Start Date: 17/Nov/20 11:49
Worklog Time Spent: 10m
Work Description: zuston commented on pull request #2389:
URL: https://github.com/apache/hadoop/pull/2389#issuecomment-728876032
> @zuston Could you please check the checkstyle warnings seems related?
Sorry I will do it soon
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 512856)
Time Spent: 1.5h (was: 1h 20m)
> ViewFileSystemOverloadScheme support specifying mount table loader imp
> through conf
> ---
>
> Key: HDFS-15635
> URL: https://issues.apache.org/jira/browse/HDFS-15635
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: viewfsOverloadScheme
>Reporter: Junfan Zhang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> According to HDFS-15289, the default mountable loader is
> {{[HCFSMountTableConfigLoader|https://github.com/apache/hadoop/blob/4734c77b4b64b7c6432da4cc32881aba85f94ea1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/HCFSMountTableConfigLoader.java#L35]}}.
> In some scenarios, users want to implement the mount table loader by
> themselves, so it is necessary to dynamically configure the loader.
>
> cc [~shv], [~abhishekd], [~hexiaoqiao]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15635) ViewFileSystemOverloadScheme support specifying mount table loader imp through conf
[
https://issues.apache.org/jira/browse/HDFS-15635?focusedWorklogId=512854=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-512854
]
ASF GitHub Bot logged work on HDFS-15635:
-
Author: ASF GitHub Bot
Created on: 17/Nov/20 11:49
Start Date: 17/Nov/20 11:49
Worklog Time Spent: 10m
Work Description: zuston commented on pull request #2389:
URL: https://github.com/apache/hadoop/pull/2389#issuecomment-728875824
Sorry I will do it soon
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 512854)
Time Spent: 1h 10m (was: 1h)
> ViewFileSystemOverloadScheme support specifying mount table loader imp
> through conf
> ---
>
> Key: HDFS-15635
> URL: https://issues.apache.org/jira/browse/HDFS-15635
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: viewfsOverloadScheme
>Reporter: Junfan Zhang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> According to HDFS-15289, the default mountable loader is
> {{[HCFSMountTableConfigLoader|https://github.com/apache/hadoop/blob/4734c77b4b64b7c6432da4cc32881aba85f94ea1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/HCFSMountTableConfigLoader.java#L35]}}.
> In some scenarios, users want to implement the mount table loader by
> themselves, so it is necessary to dynamically configure the loader.
>
> cc [~shv], [~abhishekd], [~hexiaoqiao]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15635) ViewFileSystemOverloadScheme support specifying mount table loader imp through conf
[
https://issues.apache.org/jira/browse/HDFS-15635?focusedWorklogId=512855=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-512855
]
ASF GitHub Bot logged work on HDFS-15635:
-
Author: ASF GitHub Bot
Created on: 17/Nov/20 11:49
Start Date: 17/Nov/20 11:49
Worklog Time Spent: 10m
Work Description: zuston removed a comment on pull request #2389:
URL: https://github.com/apache/hadoop/pull/2389#issuecomment-728875824
Sorry I will do it soon
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 512855)
Time Spent: 1h 20m (was: 1h 10m)
> ViewFileSystemOverloadScheme support specifying mount table loader imp
> through conf
> ---
>
> Key: HDFS-15635
> URL: https://issues.apache.org/jira/browse/HDFS-15635
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: viewfsOverloadScheme
>Reporter: Junfan Zhang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> According to HDFS-15289, the default mountable loader is
> {{[HCFSMountTableConfigLoader|https://github.com/apache/hadoop/blob/4734c77b4b64b7c6432da4cc32881aba85f94ea1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/HCFSMountTableConfigLoader.java#L35]}}.
> In some scenarios, users want to implement the mount table loader by
> themselves, so it is necessary to dynamically configure the loader.
>
> cc [~shv], [~abhishekd], [~hexiaoqiao]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13254) RBF: Cannot mv/cp file cross namespace
[ https://issues.apache.org/jira/browse/HDFS-13254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233481#comment-17233481 ] JiangHua Zhu commented on HDFS-13254: - Hi, all. The content of your discussion is wonderful. Here I want to add, whether we can consider moving data directly through cp/mv, and it may be more efficient to move data through the bottom layer. However, the content of ns, namenode, and datanode will be changed here. > RBF: Cannot mv/cp file cross namespace > -- > > Key: HDFS-13254 > URL: https://issues.apache.org/jira/browse/HDFS-13254 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wu Weiwei >Assignee: Hemanth Boyina >Priority: Major > > When I try to mv a file from a namespace to another, the client return an > error. > > Do we have any plan to support cp/mv file cross namespace? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13965) hadoop.security.kerberos.ticket.cache.path setting is not honored when KMS encryption is enabled.
[
https://issues.apache.org/jira/browse/HDFS-13965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233467#comment-17233467
]
Arun Prabu commented on HDFS-13965:
---
Our application has to run as root. So, we temporarily change the effective
user id of our process to configured OS user and run kinit as that user
kinit halsys -kt /etc/krb5.keytab -c /tmp/krb5cc_1099
After kinit, hdfs connection is created as user halsys using libhdfs C apis
After running kinit the effective user id of the process is again elevated to
root
When the auto-renewal thread in java layer detects its time to renew, the
ticker gets renewed and owner of /tmp/krb5cc_1099 is changed to root
lets say arbitrary ticket cache is used for user halsys with id 1099. So, as
per above suggestion if arbitrary ticket path like /custom_path/krb5cc_1009 is
used,
kinit halsys -kt /etc/krb5.keytab -c /custom_path/krb5cc_1099
will auto-renewal thread stop renewing the ticket? or is that ownership will
not be changed? What is the expected behavior.
> hadoop.security.kerberos.ticket.cache.path setting is not honored when KMS
> encryption is enabled.
> -
>
> Key: HDFS-13965
> URL: https://issues.apache.org/jira/browse/HDFS-13965
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, kms
>Affects Versions: 2.7.3, 2.7.7
>Reporter: LOKESKUMAR VIJAYAKUMAR
>Assignee: Kitti Nanasi
>Priority: Major
>
> _We use the *+hadoop.security.kerberos.ticket.cache.path+* setting to provide
> a custom kerberos cache path for all hadoop operations to be run as specified
> user. But this setting is not honored when KMS encryption is enabled._
> _The below program to read a file works when KMS encryption is not enabled,
> but it fails when the KMS encryption is enabled._
> _Looks like *hadoop.security.kerberos.ticket.cache.path* setting is not
> honored by *createConnection on KMSClientProvider.java.*_
>
> HadoopTest.java (CLASSPATH needs to be set to compile and run)
>
> import java.io.InputStream;
> import java.net.URI;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.Path;
>
> public class HadoopTest {
> public static int runRead(String[] args) throws Exception{
> if (args.length < 3) {
> System.err.println("HadoopTest hadoop_file_path
> hadoop_user kerberos_cache");
> return 1;
> }
> Path inputPath = new Path(args[0]);
> Configuration conf = new Configuration();
> URI defaultURI = FileSystem.getDefaultUri(conf);
>
> conf.set("hadoop.security.kerberos.ticket.cache.path",args[2]);
> FileSystem fs =
> FileSystem.newInstance(defaultURI,conf,args[1]);
> InputStream is = fs.open(inputPath);
> byte[] buffer = new byte[4096];
> int nr = is.read(buffer);
> while (nr != -1)
> {
> System.out.write(buffer, 0, nr);
> nr = is.read(buffer);
> }
> return 0;
> }
> public static void main( String[] args ) throws Exception {
> int returnCode = HadoopTest.runRead(args);
> System.exit(returnCode);
> }
> }
>
>
>
> [root@lstrost3 testhadoop]# pwd
> /testhadoop
>
> [root@lstrost3 testhadoop]# ls
> HadoopTest.java
>
> [root@lstrost3 testhadoop]# export CLASSPATH=`hadoop classpath --glob`:.
>
> [root@lstrost3 testhadoop]# javac HadoopTest.java
>
> [root@lstrost3 testhadoop]# java HadoopTest
> HadoopTest hadoop_file_path hadoop_user kerberos_cache
>
> [root@lstrost3 testhadoop]# java HadoopTest /loki/loki.file loki
> /tmp/krb5cc_1006
> 18/09/27 23:23:20 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 18/09/27 23:23:21 WARN shortcircuit.DomainSocketFactory: The short-circuit
> local reads feature cannot be used because libhadoop cannot be loaded.
> Exception in thread "main" java.io.IOException:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> GSSException: *{color:#FF}No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt){color}*
> at
> {color:#FF}*org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:551)*{color}
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:831)
> at
>
