[jira] [Commented] (HDFS-15240) Erasure Coding: dirty buffer causes reconstruction block error
[
https://issues.apache.org/jira/browse/HDFS-15240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17240532#comment-17240532
]
Hadoop QA commented on HDFS-15240:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
20s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
51s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
36s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
15s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
0s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
42s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
51s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
22m 48s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
57s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
58s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m
18s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
30s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
3s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
40s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m
40s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
1s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 18m
1s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
47s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
24s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 9s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
|
[jira] [Work logged] (HDFS-15648) TestFileChecksum should be parameterized with a boolean flag
[
https://issues.apache.org/jira/browse/HDFS-15648?focusedWorklogId=517761=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-517761
]
ASF GitHub Bot logged work on HDFS-15648:
-
Author: ASF GitHub Bot
Created on: 30/Nov/20 06:47
Start Date: 30/Nov/20 06:47
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2501:
URL: https://github.com/apache/hadoop/pull/2501#issuecomment-735587353
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 30m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 41s | | trunk passed |
| +1 :green_heart: | compile | 1m 19s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 1m 11s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 50s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 20s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 53s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 1m 25s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 3m 8s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 6s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 12s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 4s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 1m 4s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 39s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 12s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 52s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 49s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 1m 23s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 3m 6s | | the patch passed |
_ Other Tests _ |
| -1 :x: | unit | 121m 45s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2501/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 240m 38s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestDecommissionWithBackoffMonitor |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2501/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2501 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 2772aa502e61 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4d2ae5b3989 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2501/1/testReport/ |
| Max. process+thread count | 3437 (vs. ulimit of 5500) |
| modules | C:
[jira] [Commented] (HDFS-14437) Exception happened when rollEditLog expects empty EditsDoubleBuffer.bufCurrent but not
[
https://issues.apache.org/jira/browse/HDFS-14437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17240493#comment-17240493
]
Ravuri Sushma sree commented on HDFS-14437:
---
Thank you everyone for the discussion here, I have tried applying the patch and
the solution on PR. [~hexiaoqiao] can you please let me know if anymore updates
are available on this JIRA.
> Exception happened when rollEditLog expects empty
> EditsDoubleBuffer.bufCurrent but not
> -
>
> Key: HDFS-14437
> URL: https://issues.apache.org/jira/browse/HDFS-14437
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, namenode, qjm
>Reporter: angerszhu
>Priority: Major
> Attachments: HDFS-14437.reproduction.patch,
> HDFS-14437.reproductionwithlog.patch, screenshot-1.png
>
>
> For the problem mentioned in https://issues.apache.org/jira/browse/HDFS-10943
> , I have sort the process of write and flush EditLog and some important
> function, I found the in the class FSEditLog class, the close() function
> will call such process like below:
>
> {code:java}
> waitForSyncToFinish();
> endCurrentLogSegment(true);{code}
> since we have gain the object lock in the function close(), so when
> waitForSyncToFish() method return, it mean all logSync job has done and all
> data in bufReady has been flushed out, and since current thread has the lock
> of this object, when call endCurrentLogSegment(), no other thread will gain
> the lock so they can't write new editlog into currentBuf.
> But when we don't call waitForSyncToFish() before endCurrentLogSegment(),
> there may be some autoScheduled logSync()'s flush process is doing, since
> this process don't need
> synchronization since it has mention in the comment of logSync() method :
>
> {code:java}
> /**
> * Sync all modifications done by this thread.
> *
> * The internal concurrency design of this class is as follows:
> * - Log items are written synchronized into an in-memory buffer,
> * and each assigned a transaction ID.
> * - When a thread (client) would like to sync all of its edits, logSync()
> * uses a ThreadLocal transaction ID to determine what edit number must
> * be synced to.
> * - The isSyncRunning volatile boolean tracks whether a sync is currently
> * under progress.
> *
> * The data is double-buffered within each edit log implementation so that
> * in-memory writing can occur in parallel with the on-disk writing.
> *
> * Each sync occurs in three steps:
> * 1. synchronized, it swaps the double buffer and sets the isSyncRunning
> * flag.
> * 2. unsynchronized, it flushes the data to storage
> * 3. synchronized, it resets the flag and notifies anyone waiting on the
> * sync.
> *
> * The lack of synchronization on step 2 allows other threads to continue
> * to write into the memory buffer while the sync is in progress.
> * Because this step is unsynchronized, actions that need to avoid
> * concurrency with sync() should be synchronized and also call
> * waitForSyncToFinish() before assuming they are running alone.
> */
> public void logSync() {
> long syncStart = 0;
> // Fetch the transactionId of this thread.
> long mytxid = myTransactionId.get().txid;
>
> boolean sync = false;
> try {
> EditLogOutputStream logStream = null;
> synchronized (this) {
> try {
> printStatistics(false);
> // if somebody is already syncing, then wait
> while (mytxid > synctxid && isSyncRunning) {
> try {
> wait(1000);
> } catch (InterruptedException ie) {
> }
> }
> //
> // If this transaction was already flushed, then nothing to do
> //
> if (mytxid <= synctxid) {
> numTransactionsBatchedInSync++;
> if (metrics != null) {
> // Metrics is non-null only when used inside name node
> metrics.incrTransactionsBatchedInSync();
> }
> return;
> }
>
> // now, this thread will do the sync
> syncStart = txid;
> isSyncRunning = true;
> sync = true;
> // swap buffers
> try {
> if (journalSet.isEmpty()) {
> throw new IOException("No journals available to flush");
> }
> editLogStream.setReadyToFlush();
> } catch (IOException e) {
> final String msg =
> "Could not sync enough journals to persistent storage " +
> "due to " + e.getMessage() + ". " +
> "Unsynced transactions: " + (txid - synctxid);
> LOG.fatal(msg, new Exception());
> synchronized(journalSetLock) {
> IOUtils.cleanup(LOG, journalSet);
>
[jira] [Commented] (HDFS-15422) Reported IBR is partially replaced with stored info when queuing.
[ https://issues.apache.org/jira/browse/HDFS-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17240491#comment-17240491 ] Ravuri Sushma sree commented on HDFS-15422: --- Thank you everyone for the discussion here. Can anyone let me know if this issue is reproducible in UT? > Reported IBR is partially replaced with stored info when queuing. > - > > Key: HDFS-15422 > URL: https://issues.apache.org/jira/browse/HDFS-15422 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Kihwal Lee >Priority: Critical > Attachments: HDFS-15422-branch-2.10.001.patch > > > When queueing an IBR (incremental block report) on a standby namenode, some > of the reported information is being replaced with the existing stored > information. This can lead to false block corruption. > We had a namenode, after transitioning to active, started reporting missing > blocks with "SIZE_MISMATCH" as corrupt reason. These were blocks that were > appended and the sizes were actually correct on the datanodes. Upon further > investigation, it was determined that the namenode was queueing IBRs with > altered information. > Although it sounds bad, I am not making it blocker -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15676) TestRouterRpcMultiDestination#testNamenodeMetrics fails on trunk
[ https://issues.apache.org/jira/browse/HDFS-15676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki reassigned HDFS-15676: --- Assignee: Masatake Iwasaki > TestRouterRpcMultiDestination#testNamenodeMetrics fails on trunk > > > Key: HDFS-15676 > URL: https://issues.apache.org/jira/browse/HDFS-15676 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ahmed Hussein >Assignee: Masatake Iwasaki >Priority: Major > > qbt report (Nov 8, 2020, 11:28 AM) shows failures in testNamenodeMetrics -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15648) TestFileChecksum should be parameterized with a boolean flag
[
https://issues.apache.org/jira/browse/HDFS-15648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated HDFS-15648:
Status: Patch Available (was: Open)
> TestFileChecksum should be parameterized with a boolean flag
>
>
> Key: HDFS-15648
> URL: https://issues.apache.org/jira/browse/HDFS-15648
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ahmed Hussein
>Assignee: Masatake Iwasaki
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{TestFileChecksumCompositeCrc}} extends {{TestFileChecksum}} overriding 3
> methods that return a constant flag True/False.
> The class is useless and it causes confusion with two different jiras, while
> the main bug should be in TestFileChecksum.
> The {{TestFileChecksum}} should be parameterized
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15648) TestFileChecksum should be parameterized with a boolean flag
[
https://issues.apache.org/jira/browse/HDFS-15648?focusedWorklogId=517717=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-517717
]
ASF GitHub Bot logged work on HDFS-15648:
-
Author: ASF GitHub Bot
Created on: 30/Nov/20 02:45
Start Date: 30/Nov/20 02:45
Worklog Time Spent: 10m
Work Description: iwasakims opened a new pull request #2501:
URL: https://github.com/apache/hadoop/pull/2501
https://issues.apache.org/jira/browse/HDFS-15648
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 517717)
Remaining Estimate: 0h
Time Spent: 10m
> TestFileChecksum should be parameterized with a boolean flag
>
>
> Key: HDFS-15648
> URL: https://issues.apache.org/jira/browse/HDFS-15648
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ahmed Hussein
>Assignee: Masatake Iwasaki
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{TestFileChecksumCompositeCrc}} extends {{TestFileChecksum}} overriding 3
> methods that return a constant flag True/False.
> The class is useless and it causes confusion with two different jiras, while
> the main bug should be in TestFileChecksum.
> The {{TestFileChecksum}} should be parameterized
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15648) TestFileChecksum should be parameterized with a boolean flag
[
https://issues.apache.org/jira/browse/HDFS-15648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDFS-15648:
--
Labels: pull-request-available (was: )
> TestFileChecksum should be parameterized with a boolean flag
>
>
> Key: HDFS-15648
> URL: https://issues.apache.org/jira/browse/HDFS-15648
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ahmed Hussein
>Assignee: Masatake Iwasaki
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{TestFileChecksumCompositeCrc}} extends {{TestFileChecksum}} overriding 3
> methods that return a constant flag True/False.
> The class is useless and it causes confusion with two different jiras, while
> the main bug should be in TestFileChecksum.
> The {{TestFileChecksum}} should be parameterized
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15240) Erasure Coding: dirty buffer causes reconstruction block error
[
https://issues.apache.org/jira/browse/HDFS-15240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17240439#comment-17240439
]
HuangTao commented on HDFS-15240:
-
Upload [^HDFS-15240.012.patch] to fix flaky test case.
The test cases can't pass with XOR-2-1 erasure coding, because these two cases
need to allow at least two blocks to be broken.
> Erasure Coding: dirty buffer causes reconstruction block error
> --
>
> Key: HDFS-15240
> URL: https://issues.apache.org/jira/browse/HDFS-15240
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, erasure-coding
>Reporter: HuangTao
>Assignee: HuangTao
>Priority: Major
> Attachments: HDFS-15240.001.patch, HDFS-15240.002.patch,
> HDFS-15240.003.patch, HDFS-15240.004.patch, HDFS-15240.005.patch,
> HDFS-15240.006.patch, HDFS-15240.007.patch, HDFS-15240.008.patch,
> HDFS-15240.009.patch, HDFS-15240.010.patch, HDFS-15240.011.patch,
> HDFS-15240.012.patch, image-2020-07-16-15-56-38-608.png,
> org.apache.hadoop.hdfs.TestReconstructStripedFile-output.txt,
> org.apache.hadoop.hdfs.TestReconstructStripedFile.txt,
> test-HDFS-15240.006.patch
>
>
> When read some lzo files we found some blocks were broken.
> I read back all internal blocks(b0-b8) of the block group(RS-6-3-1024k) from
> DN directly, and choose 6(b0-b5) blocks to decode other 3(b6', b7', b8')
> blocks. And find the longest common sequenece(LCS) between b6'(decoded) and
> b6(read from DN)(b7'/b7 and b8'/b8).
> After selecting 6 blocks of the block group in combinations one time and
> iterating through all cases, I find one case that the length of LCS is the
> block length - 64KB, 64KB is just the length of ByteBuffer used by
> StripedBlockReader. So the corrupt reconstruction block is made by a dirty
> buffer.
> The following log snippet(only show 2 of 28 cases) is my check program
> output. In my case, I known the 3th block is corrupt, so need other 5 blocks
> to decode another 3 blocks, then find the 1th block's LCS substring is block
> length - 64kb.
> It means (0,1,2,4,5,6)th blocks were used to reconstruct 3th block, and the
> dirty buffer was used before read the 1th block.
> Must be noted that StripedBlockReader read from the offset 0 of the 1th block
> after used the dirty buffer.
> EDITED for readability.
> {code:java}
> decode from block[0, 2, 3, 4, 5, 7] to generate block[1', 6', 8']
> Check the first 131072 bytes between block[1] and block[1'], the longest
> common substring length is 4
> Check the first 131072 bytes between block[6] and block[6'], the longest
> common substring length is 4
> Check the first 131072 bytes between block[8] and block[8'], the longest
> common substring length is 4
> decode from block[0, 2, 3, 4, 5, 6] to generate block[1', 7', 8']
> Check the first 131072 bytes between block[1] and block[1'], the longest
> common substring length is 65536
> CHECK AGAIN: all 27262976 bytes between block[1] and block[1'], the longest
> common substring length is 27197440 # this one
> Check the first 131072 bytes between block[7] and block[7'], the longest
> common substring length is 4
> Check the first 131072 bytes between block[8] and block[8'], the longest
> common substring length is 4{code}
> Now I know the dirty buffer causes reconstruction block error, but how does
> the dirty buffer come about?
> After digging into the code and DN log, I found this following DN log is the
> root reason.
> {code:java}
> [INFO] [stripedRead-1017] : Interrupted while waiting for IO on channel
> java.nio.channels.SocketChannel[connected local=/:52586
> remote=/:50010]. 18 millis timeout left.
> [WARN] [StripedBlockReconstruction-199] : Failed to reconstruct striped
> block: BP-714356632--1519726836856:blk_-YY_3472979393
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil.getNextCompletedStripedRead(StripedBlockUtil.java:314)
> at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.doReadMinimumSources(StripedReader.java:308)
> at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.readMinimumSources(StripedReader.java:269)
> at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReconstructor.reconstruct(StripedBlockReconstructor.java:94)
> at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReconstructor.run(StripedBlockReconstructor.java:60)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
>
[jira] [Updated] (HDFS-15240) Erasure Coding: dirty buffer causes reconstruction block error
[
https://issues.apache.org/jira/browse/HDFS-15240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
HuangTao updated HDFS-15240:
Attachment: HDFS-15240.012.patch
> Erasure Coding: dirty buffer causes reconstruction block error
> --
>
> Key: HDFS-15240
> URL: https://issues.apache.org/jira/browse/HDFS-15240
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, erasure-coding
>Reporter: HuangTao
>Assignee: HuangTao
>Priority: Major
> Attachments: HDFS-15240.001.patch, HDFS-15240.002.patch,
> HDFS-15240.003.patch, HDFS-15240.004.patch, HDFS-15240.005.patch,
> HDFS-15240.006.patch, HDFS-15240.007.patch, HDFS-15240.008.patch,
> HDFS-15240.009.patch, HDFS-15240.010.patch, HDFS-15240.011.patch,
> HDFS-15240.012.patch, image-2020-07-16-15-56-38-608.png,
> org.apache.hadoop.hdfs.TestReconstructStripedFile-output.txt,
> org.apache.hadoop.hdfs.TestReconstructStripedFile.txt,
> test-HDFS-15240.006.patch
>
>
> When read some lzo files we found some blocks were broken.
> I read back all internal blocks(b0-b8) of the block group(RS-6-3-1024k) from
> DN directly, and choose 6(b0-b5) blocks to decode other 3(b6', b7', b8')
> blocks. And find the longest common sequenece(LCS) between b6'(decoded) and
> b6(read from DN)(b7'/b7 and b8'/b8).
> After selecting 6 blocks of the block group in combinations one time and
> iterating through all cases, I find one case that the length of LCS is the
> block length - 64KB, 64KB is just the length of ByteBuffer used by
> StripedBlockReader. So the corrupt reconstruction block is made by a dirty
> buffer.
> The following log snippet(only show 2 of 28 cases) is my check program
> output. In my case, I known the 3th block is corrupt, so need other 5 blocks
> to decode another 3 blocks, then find the 1th block's LCS substring is block
> length - 64kb.
> It means (0,1,2,4,5,6)th blocks were used to reconstruct 3th block, and the
> dirty buffer was used before read the 1th block.
> Must be noted that StripedBlockReader read from the offset 0 of the 1th block
> after used the dirty buffer.
> EDITED for readability.
> {code:java}
> decode from block[0, 2, 3, 4, 5, 7] to generate block[1', 6', 8']
> Check the first 131072 bytes between block[1] and block[1'], the longest
> common substring length is 4
> Check the first 131072 bytes between block[6] and block[6'], the longest
> common substring length is 4
> Check the first 131072 bytes between block[8] and block[8'], the longest
> common substring length is 4
> decode from block[0, 2, 3, 4, 5, 6] to generate block[1', 7', 8']
> Check the first 131072 bytes between block[1] and block[1'], the longest
> common substring length is 65536
> CHECK AGAIN: all 27262976 bytes between block[1] and block[1'], the longest
> common substring length is 27197440 # this one
> Check the first 131072 bytes between block[7] and block[7'], the longest
> common substring length is 4
> Check the first 131072 bytes between block[8] and block[8'], the longest
> common substring length is 4{code}
> Now I know the dirty buffer causes reconstruction block error, but how does
> the dirty buffer come about?
> After digging into the code and DN log, I found this following DN log is the
> root reason.
> {code:java}
> [INFO] [stripedRead-1017] : Interrupted while waiting for IO on channel
> java.nio.channels.SocketChannel[connected local=/:52586
> remote=/:50010]. 18 millis timeout left.
> [WARN] [StripedBlockReconstruction-199] : Failed to reconstruct striped
> block: BP-714356632--1519726836856:blk_-YY_3472979393
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil.getNextCompletedStripedRead(StripedBlockUtil.java:314)
> at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.doReadMinimumSources(StripedReader.java:308)
> at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.readMinimumSources(StripedReader.java:269)
> at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReconstructor.reconstruct(StripedBlockReconstructor.java:94)
> at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReconstructor.run(StripedBlockReconstructor.java:60)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834) {code}
> Reading from DN may timeout(hold by a future(F)) and output the INFO log, but
> the
[jira] [Assigned] (HDFS-15648) TestFileChecksum should be parameterized with a boolean flag
[
https://issues.apache.org/jira/browse/HDFS-15648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki reassigned HDFS-15648:
---
Assignee: Masatake Iwasaki
> TestFileChecksum should be parameterized with a boolean flag
>
>
> Key: HDFS-15648
> URL: https://issues.apache.org/jira/browse/HDFS-15648
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ahmed Hussein
>Assignee: Masatake Iwasaki
>Priority: Major
>
> {{TestFileChecksumCompositeCrc}} extends {{TestFileChecksum}} overriding 3
> methods that return a constant flag True/False.
> The class is useless and it causes confusion with two different jiras, while
> the main bug should be in TestFileChecksum.
> The {{TestFileChecksum}} should be parameterized
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15240) Erasure Coding: dirty buffer causes reconstruction block error
[
https://issues.apache.org/jira/browse/HDFS-15240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17240416#comment-17240416
]
Hui Fei commented on HDFS-15240:
[~umamaheswararao] Could you please take a look again ? Thanks!
> Erasure Coding: dirty buffer causes reconstruction block error
> --
>
> Key: HDFS-15240
> URL: https://issues.apache.org/jira/browse/HDFS-15240
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, erasure-coding
>Reporter: HuangTao
>Assignee: HuangTao
>Priority: Major
> Attachments: HDFS-15240.001.patch, HDFS-15240.002.patch,
> HDFS-15240.003.patch, HDFS-15240.004.patch, HDFS-15240.005.patch,
> HDFS-15240.006.patch, HDFS-15240.007.patch, HDFS-15240.008.patch,
> HDFS-15240.009.patch, HDFS-15240.010.patch, HDFS-15240.011.patch,
> image-2020-07-16-15-56-38-608.png,
> org.apache.hadoop.hdfs.TestReconstructStripedFile-output.txt,
> org.apache.hadoop.hdfs.TestReconstructStripedFile.txt,
> test-HDFS-15240.006.patch
>
>
> When read some lzo files we found some blocks were broken.
> I read back all internal blocks(b0-b8) of the block group(RS-6-3-1024k) from
> DN directly, and choose 6(b0-b5) blocks to decode other 3(b6', b7', b8')
> blocks. And find the longest common sequenece(LCS) between b6'(decoded) and
> b6(read from DN)(b7'/b7 and b8'/b8).
> After selecting 6 blocks of the block group in combinations one time and
> iterating through all cases, I find one case that the length of LCS is the
> block length - 64KB, 64KB is just the length of ByteBuffer used by
> StripedBlockReader. So the corrupt reconstruction block is made by a dirty
> buffer.
> The following log snippet(only show 2 of 28 cases) is my check program
> output. In my case, I known the 3th block is corrupt, so need other 5 blocks
> to decode another 3 blocks, then find the 1th block's LCS substring is block
> length - 64kb.
> It means (0,1,2,4,5,6)th blocks were used to reconstruct 3th block, and the
> dirty buffer was used before read the 1th block.
> Must be noted that StripedBlockReader read from the offset 0 of the 1th block
> after used the dirty buffer.
> EDITED for readability.
> {code:java}
> decode from block[0, 2, 3, 4, 5, 7] to generate block[1', 6', 8']
> Check the first 131072 bytes between block[1] and block[1'], the longest
> common substring length is 4
> Check the first 131072 bytes between block[6] and block[6'], the longest
> common substring length is 4
> Check the first 131072 bytes between block[8] and block[8'], the longest
> common substring length is 4
> decode from block[0, 2, 3, 4, 5, 6] to generate block[1', 7', 8']
> Check the first 131072 bytes between block[1] and block[1'], the longest
> common substring length is 65536
> CHECK AGAIN: all 27262976 bytes between block[1] and block[1'], the longest
> common substring length is 27197440 # this one
> Check the first 131072 bytes between block[7] and block[7'], the longest
> common substring length is 4
> Check the first 131072 bytes between block[8] and block[8'], the longest
> common substring length is 4{code}
> Now I know the dirty buffer causes reconstruction block error, but how does
> the dirty buffer come about?
> After digging into the code and DN log, I found this following DN log is the
> root reason.
> {code:java}
> [INFO] [stripedRead-1017] : Interrupted while waiting for IO on channel
> java.nio.channels.SocketChannel[connected local=/:52586
> remote=/:50010]. 18 millis timeout left.
> [WARN] [StripedBlockReconstruction-199] : Failed to reconstruct striped
> block: BP-714356632--1519726836856:blk_-YY_3472979393
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil.getNextCompletedStripedRead(StripedBlockUtil.java:314)
> at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.doReadMinimumSources(StripedReader.java:308)
> at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.readMinimumSources(StripedReader.java:269)
> at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReconstructor.reconstruct(StripedBlockReconstructor.java:94)
> at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReconstructor.run(StripedBlockReconstructor.java:60)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834) {code}
> Reading from DN may timeout(hold by a
[jira] [Work logged] (HDFS-15699) lz4 sources missing for native Visual Studio project
[ https://issues.apache.org/jira/browse/HDFS-15699?focusedWorklogId=517704=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-517704 ] ASF GitHub Bot logged work on HDFS-15699: - Author: ASF GitHub Bot Created on: 30/Nov/20 00:00 Start Date: 30/Nov/20 00:00 Worklog Time Spent: 10m Work Description: sunchao commented on pull request #2498: URL: https://github.com/apache/hadoop/pull/2498#issuecomment-735478935 @GauthamBanasandra can you convert this into a HADOOP common JIRA? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 517704) Remaining Estimate: 23h 20m (was: 23.5h) Time Spent: 40m (was: 0.5h) > lz4 sources missing for native Visual Studio project > > > Key: HDFS-15699 > URL: https://issues.apache.org/jira/browse/HDFS-15699 > Project: Hadoop HDFS > Issue Type: Bug > Components: native >Affects Versions: 3.3.0 > Environment: Windows >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: pull-request-available > Fix For: 3.3.0 > > Original Estimate: 24h > Time Spent: 40m > Remaining Estimate: 23h 20m > > lz4 sources are missing for the *native.vcxproj* Visual Studio project for > Windows. This is causing compilation failure on Windows. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15691) Fix flaky test TestServerWebApp.getHomeDir
[
https://issues.apache.org/jira/browse/HDFS-15691?focusedWorklogId=517692=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-517692
]
ASF GitHub Bot logged work on HDFS-15691:
-
Author: ASF GitHub Bot
Created on: 29/Nov/20 23:00
Start Date: 29/Nov/20 23:00
Worklog Time Spent: 10m
Work Description: lzx404243 commented on pull request #2482:
URL: https://github.com/apache/hadoop/pull/2482#issuecomment-735471505
Closing this PR per discussion above.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 517692)
Time Spent: 50m (was: 40m)
> Fix flaky test TestServerWebApp.getHomeDir
> --
>
> Key: HDFS-15691
> URL: https://issues.apache.org/jira/browse/HDFS-15691
> Project: Hadoop HDFS
> Issue Type: Bug
> Environment: macOS 10.15.6
> java version "1.8.0_151"
> Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)
> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
>
>Reporter: Zhengxi Li
>Assignee: Zhengxi Li
>Priority: Minor
> Labels: pull-request-available
> Attachments: HDFS-15691.001.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The test '{{org.apache.hadoop.lib.servlet.TestServerWebApp.getHomeDir'}} is
> not idempotent and fails if run twice in the same JVM, because it pollutes
> state shared among tests. It may be good to clean this state pollution so
> that some other tests do not fail in the future due to the shared state
> polluted by this test.
>
> PR link: https://github.com/apache/hadoop/pull/2482
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15691) Fix flaky test TestServerWebApp.getHomeDir
[
https://issues.apache.org/jira/browse/HDFS-15691?focusedWorklogId=517691=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-517691
]
ASF GitHub Bot logged work on HDFS-15691:
-
Author: ASF GitHub Bot
Created on: 29/Nov/20 23:00
Start Date: 29/Nov/20 23:00
Worklog Time Spent: 10m
Work Description: lzx404243 closed pull request #2482:
URL: https://github.com/apache/hadoop/pull/2482
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 517691)
Time Spent: 40m (was: 0.5h)
> Fix flaky test TestServerWebApp.getHomeDir
> --
>
> Key: HDFS-15691
> URL: https://issues.apache.org/jira/browse/HDFS-15691
> Project: Hadoop HDFS
> Issue Type: Bug
> Environment: macOS 10.15.6
> java version "1.8.0_151"
> Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)
> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
>
>Reporter: Zhengxi Li
>Assignee: Zhengxi Li
>Priority: Minor
> Labels: pull-request-available
> Attachments: HDFS-15691.001.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The test '{{org.apache.hadoop.lib.servlet.TestServerWebApp.getHomeDir'}} is
> not idempotent and fails if run twice in the same JVM, because it pollutes
> state shared among tests. It may be good to clean this state pollution so
> that some other tests do not fail in the future due to the shared state
> polluted by this test.
>
> PR link: https://github.com/apache/hadoop/pull/2482
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15691) Fix flaky test TestServerWebApp.getHomeDir
[
https://issues.apache.org/jira/browse/HDFS-15691?focusedWorklogId=517690=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-517690
]
ASF GitHub Bot logged work on HDFS-15691:
-
Author: ASF GitHub Bot
Created on: 29/Nov/20 22:59
Start Date: 29/Nov/20 22:59
Worklog Time Spent: 10m
Work Description: lzx404243 commented on a change in pull request #2482:
URL: https://github.com/apache/hadoop/pull/2482#discussion_r532282135
##
File path:
hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/lib/servlet/TestServerWebApp.java
##
@@ -43,6 +43,8 @@ public void getHomeDir() {
assertEquals(ServerWebApp.getDir("TestServerWebApp0", ".log.dir",
"/tmp/log"), "/tmp/log");
System.setProperty("TestServerWebApp0.log.dir", "/tmplog");
assertEquals(ServerWebApp.getDir("TestServerWebApp0", ".log.dir",
"/tmp/log"), "/tmplog");
+System.clearProperty("TestServerWebApp0.home.dir");
+System.clearProperty("TestServerWebApp0.log.dir");
Review comment:
Hi @aajisaka,
thanks for your review! You raised some good points.
Per further discussion with Xiaoqiao He in [jira
link](https://issues.apache.org/jira/browse/HDFS-15691?focusedCommentId=17237118=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17237118),
it looks like setting the system properties can be harmless as system
properties are not reused in different tests.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 517690)
Time Spent: 0.5h (was: 20m)
> Fix flaky test TestServerWebApp.getHomeDir
> --
>
> Key: HDFS-15691
> URL: https://issues.apache.org/jira/browse/HDFS-15691
> Project: Hadoop HDFS
> Issue Type: Bug
> Environment: macOS 10.15.6
> java version "1.8.0_151"
> Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)
> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
>
>Reporter: Zhengxi Li
>Assignee: Zhengxi Li
>Priority: Minor
> Labels: pull-request-available
> Attachments: HDFS-15691.001.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The test '{{org.apache.hadoop.lib.servlet.TestServerWebApp.getHomeDir'}} is
> not idempotent and fails if run twice in the same JVM, because it pollutes
> state shared among tests. It may be good to clean this state pollution so
> that some other tests do not fail in the future due to the shared state
> polluted by this test.
>
> PR link: https://github.com/apache/hadoop/pull/2482
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15691) Fix flaky test TestServerWebApp.getHomeDir
[
https://issues.apache.org/jira/browse/HDFS-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17240386#comment-17240386
]
Zhengxi Li commented on HDFS-15691:
---
Marking this as closed per the discussion above.
> Fix flaky test TestServerWebApp.getHomeDir
> --
>
> Key: HDFS-15691
> URL: https://issues.apache.org/jira/browse/HDFS-15691
> Project: Hadoop HDFS
> Issue Type: Bug
> Environment: macOS 10.15.6
> java version "1.8.0_151"
> Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)
> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
>
>Reporter: Zhengxi Li
>Assignee: Zhengxi Li
>Priority: Minor
> Labels: pull-request-available
> Attachments: HDFS-15691.001.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The test '{{org.apache.hadoop.lib.servlet.TestServerWebApp.getHomeDir'}} is
> not idempotent and fails if run twice in the same JVM, because it pollutes
> state shared among tests. It may be good to clean this state pollution so
> that some other tests do not fail in the future due to the shared state
> polluted by this test.
>
> PR link: https://github.com/apache/hadoop/pull/2482
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15691) Fix flaky test TestServerWebApp.getHomeDir
[
https://issues.apache.org/jira/browse/HDFS-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhengxi Li resolved HDFS-15691.
---
Resolution: Not A Bug
> Fix flaky test TestServerWebApp.getHomeDir
> --
>
> Key: HDFS-15691
> URL: https://issues.apache.org/jira/browse/HDFS-15691
> Project: Hadoop HDFS
> Issue Type: Bug
> Environment: macOS 10.15.6
> java version "1.8.0_151"
> Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)
> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
>
>Reporter: Zhengxi Li
>Assignee: Zhengxi Li
>Priority: Minor
> Labels: pull-request-available
> Attachments: HDFS-15691.001.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The test '{{org.apache.hadoop.lib.servlet.TestServerWebApp.getHomeDir'}} is
> not idempotent and fails if run twice in the same JVM, because it pollutes
> state shared among tests. It may be good to clean this state pollution so
> that some other tests do not fail in the future due to the shared state
> polluted by this test.
>
> PR link: https://github.com/apache/hadoop/pull/2482
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15691) Fix flaky test TestServerWebApp.getHomeDir
[
https://issues.apache.org/jira/browse/HDFS-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhengxi Li updated HDFS-15691:
--
Status: Open (was: Patch Available)
> Fix flaky test TestServerWebApp.getHomeDir
> --
>
> Key: HDFS-15691
> URL: https://issues.apache.org/jira/browse/HDFS-15691
> Project: Hadoop HDFS
> Issue Type: Bug
> Environment: macOS 10.15.6
> java version "1.8.0_151"
> Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)
> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
>
>Reporter: Zhengxi Li
>Assignee: Zhengxi Li
>Priority: Minor
> Labels: pull-request-available
> Attachments: HDFS-15691.001.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The test '{{org.apache.hadoop.lib.servlet.TestServerWebApp.getHomeDir'}} is
> not idempotent and fails if run twice in the same JVM, because it pollutes
> state shared among tests. It may be good to clean this state pollution so
> that some other tests do not fail in the future due to the shared state
> polluted by this test.
>
> PR link: https://github.com/apache/hadoop/pull/2482
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15691) Fix flaky test TestServerWebApp.getHomeDir
[
https://issues.apache.org/jira/browse/HDFS-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17240382#comment-17240382
]
Zhengxi Li commented on HDFS-15691:
---
Hi [~hexiaoqiao],
although each test creates some system properties, you are right that each of
those new system properties has very specific names that aren't reused between
tests. If we keep it this way(using different system properties in different
tests), we should be okay.
> Fix flaky test TestServerWebApp.getHomeDir
> --
>
> Key: HDFS-15691
> URL: https://issues.apache.org/jira/browse/HDFS-15691
> Project: Hadoop HDFS
> Issue Type: Bug
> Environment: macOS 10.15.6
> java version "1.8.0_151"
> Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)
> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
>
>Reporter: Zhengxi Li
>Assignee: Zhengxi Li
>Priority: Minor
> Labels: pull-request-available
> Attachments: HDFS-15691.001.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The test '{{org.apache.hadoop.lib.servlet.TestServerWebApp.getHomeDir'}} is
> not idempotent and fails if run twice in the same JVM, because it pollutes
> state shared among tests. It may be good to clean this state pollution so
> that some other tests do not fail in the future due to the shared state
> polluted by this test.
>
> PR link: https://github.com/apache/hadoop/pull/2482
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15699) lz4 sources missing for native Visual Studio project
[ https://issues.apache.org/jira/browse/HDFS-15699?focusedWorklogId=517600=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-517600 ] ASF GitHub Bot logged work on HDFS-15699: - Author: ASF GitHub Bot Created on: 29/Nov/20 09:38 Start Date: 29/Nov/20 09:38 Worklog Time Spent: 10m Work Description: GauthamBanasandra commented on pull request #2498: URL: https://github.com/apache/hadoop/pull/2498#issuecomment-735367083 This PR only changes the build related code, which can't be tested. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 517600) Remaining Estimate: 23.5h (was: 23h 40m) Time Spent: 0.5h (was: 20m) > lz4 sources missing for native Visual Studio project > > > Key: HDFS-15699 > URL: https://issues.apache.org/jira/browse/HDFS-15699 > Project: Hadoop HDFS > Issue Type: Bug > Components: native >Affects Versions: 3.3.0 > Environment: Windows >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: pull-request-available > Fix For: 3.3.0 > > Original Estimate: 24h > Time Spent: 0.5h > Remaining Estimate: 23.5h > > lz4 sources are missing for the *native.vcxproj* Visual Studio project for > Windows. This is causing compilation failure on Windows. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15699) lz4 sources missing for native Visual Studio project
[
https://issues.apache.org/jira/browse/HDFS-15699?focusedWorklogId=517589=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-517589
]
ASF GitHub Bot logged work on HDFS-15699:
-
Author: ASF GitHub Bot
Created on: 29/Nov/20 08:00
Start Date: 29/Nov/20 08:00
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2498:
URL: https://github.com/apache/hadoop/pull/2498#issuecomment-735358033
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 56s | | trunk passed |
| +1 :green_heart: | compile | 21m 50s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 18m 36s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | mvnsite | 1m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 92m 43s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 1m 40s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 0s | | the patch passed |
| +1 :green_heart: | compile | 22m 31s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 22m 31s | | the patch passed |
| +1 :green_heart: | compile | 21m 8s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 21m 8s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 39s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 18m 29s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 10s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 1m 45s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
_ Other Tests _ |
| +1 :green_heart: | unit | 10m 34s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not
generate ASF License warnings. |
| | | 179m 51s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2498/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2498 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient |
| uname | Linux 011a18195cf3 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 68442b48c8a |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2498/1/testReport/ |
| Max. process+thread count | 2922 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2498/1/console |
| versions | git=2.17.1 maven=3.6.0 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries
