[jira] [Commented] (HDDS-726) Ozone Client should update SCM to move the container out of allocation path in case a write transaction fails
[ https://issues.apache.org/jira/browse/HDDS-726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770836#comment-16770836 ] Mukul Kumar Singh commented on HDDS-726: Thanks for working on this [~shashikant]. The patch generally looks good to me. Please find me comments as following. 1) BlockManagerImpl.java:202, the implementation streams all the allocated container and then finds the one not matching the size. This can result in allocate block to fail even when other container are present from which the blocks can be allocated. 2) PipelineStateMap:221, the excludedn and exclude pipeline are both calling the same function, discardPipeline:230 3) SCMPipelineManager:198, lets rename pipelineIds to excludePipelines or similar. > Ozone Client should update SCM to move the container out of allocation path > in case a write transaction fails > - > > Key: HDDS-726 > URL: https://issues.apache.org/jira/browse/HDDS-726 > Project: Hadoop Distributed Data Store > Issue Type: Test >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDDS-726.000.patch, HDDS-726.001.patch, > HDDS-726.002.patch, HDDS-726.003.patch, HDDS-726.004.patch > > > Once an container write transaction fails, it will be marked corrupted. Once > Ozone client gets an exception in such case it should tell SCM to move the > container out of allocation path. SCM will eventually close the container. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1101) SCM CA: Write Certificate information to SCM Metadata
[
https://issues.apache.org/jira/browse/HDDS-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770833#comment-16770833
]
Anu Engineer commented on HDDS-1101:
{quote}can we add api to get a stored certificates based on serial id
{quote}
We have this call in this patch, do we need anything else?
{quote}public X509Certificate getCertificateByID(BigInteger serialID, CertType
certType)
{quote}
> SCM CA: Write Certificate information to SCM Metadata
> -
>
> Key: HDDS-1101
> URL: https://issues.apache.org/jira/browse/HDDS-1101
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: SCM
>Reporter: Anu Engineer
>Assignee: Anu Engineer
>Priority: Major
> Attachments: HDDS-1101.000.patch, HDDS-1101.001.patch
>
>
> Make SCM CA write to the Metadata layer of SCM.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1121) Key read failure when data is written parallel in to Ozone
[
https://issues.apache.org/jira/browse/HDDS-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770814#comment-16770814
]
Bharat Viswanadham commented on HDDS-1121:
--
Thank You [~linyiqun] for the review. Addressed your review comments.
Uploaded the rebased patch.
> Key read failure when data is written parallel in to Ozone
> --
>
> Key: HDDS-1121
> URL: https://issues.apache.org/jira/browse/HDDS-1121
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HDDS-1121.00.patch, HDDS-1121.01.patch,
> HDDS-1121.02.patch
>
>
> When hive is run with multiple threads for data ingestion to ozone. After
> ingestion is done, during read we see this below error.
> This issue is found during hive testing, and found by [~t3rmin4t0r]
> {code:java}
> caused by: org.apache.hadoop.ozone.common.OzoneChecksumException: Checksum
> mismatch at index 0
> at
> org.apache.hadoop.ozone.common.ChecksumData.verifyChecksumDataMatches(ChecksumData.java:143)
> at org.apache.hadoop.ozone.common.Checksum.verifyChecksum(Checksum.java:239)
> at org.apache.hadoop.ozone.common.Checksum.verifyChecksum(Checksum.java:217)
> at
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.readChunkFromContainer(BlockInputStream.java:227)
> at
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.seek(BlockInputStream.java:259)
> at
> org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.seek(KeyInputStream.java:249)
> at
> org.apache.hadoop.ozone.client.io.KeyInputStream.seek(KeyInputStream.java:180)
> at
> org.apache.hadoop.fs.ozone.OzoneFSInputStream.seek(OzoneFSInputStream.java:62)
> at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:82)
> at org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121)
> at
> org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:111)
> at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:555)
> at org.apache.orc.impl.ReaderImpl.(ReaderImpl.java:370)
> at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.(ReaderImpl.java:61)
> at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:105)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.populateAndCacheStripeDetails(OrcInputFormat.java:1647)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.callInternal(OrcInputFormat.java:1533)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.access$2700(OrcInputFormat.java:1329)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1513)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1510)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1510)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1329)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1121) Key read failure when data is written parallel in to Ozone
[
https://issues.apache.org/jira/browse/HDDS-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-1121:
-
Attachment: HDDS-1121.02.patch
> Key read failure when data is written parallel in to Ozone
> --
>
> Key: HDDS-1121
> URL: https://issues.apache.org/jira/browse/HDDS-1121
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HDDS-1121.00.patch, HDDS-1121.01.patch,
> HDDS-1121.02.patch
>
>
> When hive is run with multiple threads for data ingestion to ozone. After
> ingestion is done, during read we see this below error.
> This issue is found during hive testing, and found by [~t3rmin4t0r]
> {code:java}
> caused by: org.apache.hadoop.ozone.common.OzoneChecksumException: Checksum
> mismatch at index 0
> at
> org.apache.hadoop.ozone.common.ChecksumData.verifyChecksumDataMatches(ChecksumData.java:143)
> at org.apache.hadoop.ozone.common.Checksum.verifyChecksum(Checksum.java:239)
> at org.apache.hadoop.ozone.common.Checksum.verifyChecksum(Checksum.java:217)
> at
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.readChunkFromContainer(BlockInputStream.java:227)
> at
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.seek(BlockInputStream.java:259)
> at
> org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.seek(KeyInputStream.java:249)
> at
> org.apache.hadoop.ozone.client.io.KeyInputStream.seek(KeyInputStream.java:180)
> at
> org.apache.hadoop.fs.ozone.OzoneFSInputStream.seek(OzoneFSInputStream.java:62)
> at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:82)
> at org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121)
> at
> org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:111)
> at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:555)
> at org.apache.orc.impl.ReaderImpl.(ReaderImpl.java:370)
> at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.(ReaderImpl.java:61)
> at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:105)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.populateAndCacheStripeDetails(OrcInputFormat.java:1647)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.callInternal(OrcInputFormat.java:1533)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.access$2700(OrcInputFormat.java:1329)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1513)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1510)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1510)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1329)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13158) Fix Spelling Mistakes - DECOMISSIONED
[ https://issues.apache.org/jira/browse/HDFS-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770805#comment-16770805 ] Akira Ajisaka commented on HDFS-13158: -- +1 > Fix Spelling Mistakes - DECOMISSIONED > - > > Key: HDFS-13158 > URL: https://issues.apache.org/jira/browse/HDFS-13158 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: HDFS-13158.001.patch, HDFS-13158.2.patch, > HDFS-13158.3.patch, HDFS-13158.4.patch, HDFS-13158.5.patch, > HDFS-13158.6.patch, HDFS-13158.7.patch > > > https://github.com/apache/hadoop/search?l=Java=DECOMISSIONED > There are references in the code to _DECOMISSIONED_ but the correct spelling > is _DECOMMISSIONED_. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14176) Replace incorrect use of system property user.name
[
https://issues.apache.org/jira/browse/HDFS-14176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770802#comment-16770802
]
Dinesh Chitlangia commented on HDFS-14176:
--
[~jojochuang] - Let me know if patch 03 looks good to you. Thanks.
> Replace incorrect use of system property user.name
> --
>
> Key: HDFS-14176
> URL: https://issues.apache.org/jira/browse/HDFS-14176
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.3.0
> Environment: Kerberized
>Reporter: Wei-Chiu Chuang
>Assignee: Dinesh Chitlangia
>Priority: Major
> Attachments: HDFS-14176.01.patch, HDFS-14176.02.patch,
> HDFS-14176.03.patch
>
>
> Looking at the Hadoop source code, there are a few places where the code
> assumes the user name can be acquired from Java's system property
> {{user.name}}.
> For example,
> {code:java|title=FileSystem}
> /** Return the current user's home directory in this FileSystem.
>* The default implementation returns {@code "/user/$USER/"}.
>*/
> public Path getHomeDirectory() {
> return this.makeQualified(
> new Path(USER_HOME_PREFIX + "/" + System.getProperty("user.name")));
> }
> {code}
> This is incorrect, as in a Kerberized environment, a user may login as a user
> principal different from its system login account.
> It would be better to use
> {{UserGroupInformation.getCurrentUser().getShortUserName()}}, similar to
> HDFS-12485.
> Unfortunately, I am seeing this improper use in Yarn, HDFS federation
> SFTPFilesystem and Ozone code (tests are ignored)
> The impact should be small, since it only affects the case where system is
> Kerberized and that the user principal is different from system login account.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14219) ConcurrentModificationException occurs in datanode occasionally
[
https://issues.apache.org/jira/browse/HDFS-14219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770801#comment-16770801
]
Hadoop QA commented on HDFS-14219:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HDFS-14219 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-14219 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959048/HDFS-14219.001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26238/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> ConcurrentModificationException occurs in datanode occasionally
> ---
>
> Key: HDFS-14219
> URL: https://issues.apache.org/jira/browse/HDFS-14219
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.2
>Reporter: Tao Jie
>Assignee: Tao Jie
>Priority: Minor
> Attachments: HDFS-14219.001.patch
>
>
> ERROR occasionally occurs in datanode log:
> ERROR BPServiceActor.java:792 - Exception in BPOfferService for Block pool
> BP-1687106048-10.14.9.17-1535259994856 (Datanode Uuid
> c17e635a-912f-4488-b4e8-93a58a27b5db) service to osscmh-9-21/10.14.9.21:8070
> java.util.ConcurrentModificationException: modification=62852685 !=
> iterModification = 62852684
> at
> org.apache.hadoop.util.LightWeightGSet$SetIterator.ensureNext(LightWeightGSet.java:305)
> at
> org.apache.hadoop.util.LightWeightGSet$SetIterator.hasNext(LightWeightGSet.java:322)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getBlockReports(FsDatasetImpl.java:1872)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport(BPServiceActor.java:349)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:648)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:790)
> at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14219) ConcurrentModificationException occurs in datanode occasionally
[ https://issues.apache.org/jira/browse/HDFS-14219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Jie updated HDFS-14219: --- Status: Patch Available (was: Open) > ConcurrentModificationException occurs in datanode occasionally > --- > > Key: HDFS-14219 > URL: https://issues.apache.org/jira/browse/HDFS-14219 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.2 >Reporter: Tao Jie >Assignee: Tao Jie >Priority: Minor > Attachments: HDFS-14219.001.patch > > > ERROR occasionally occurs in datanode log: > ERROR BPServiceActor.java:792 - Exception in BPOfferService for Block pool > BP-1687106048-10.14.9.17-1535259994856 (Datanode Uuid > c17e635a-912f-4488-b4e8-93a58a27b5db) service to osscmh-9-21/10.14.9.21:8070 > java.util.ConcurrentModificationException: modification=62852685 != > iterModification = 62852684 > at > org.apache.hadoop.util.LightWeightGSet$SetIterator.ensureNext(LightWeightGSet.java:305) > at > org.apache.hadoop.util.LightWeightGSet$SetIterator.hasNext(LightWeightGSet.java:322) > at > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getBlockReports(FsDatasetImpl.java:1872) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport(BPServiceActor.java:349) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:648) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:790) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14219) ConcurrentModificationException occurs in datanode occasionally
[ https://issues.apache.org/jira/browse/HDFS-14219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Jie reassigned HDFS-14219: -- Assignee: Tao Jie > ConcurrentModificationException occurs in datanode occasionally > --- > > Key: HDFS-14219 > URL: https://issues.apache.org/jira/browse/HDFS-14219 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.2 >Reporter: Tao Jie >Assignee: Tao Jie >Priority: Minor > Attachments: HDFS-14219.001.patch > > > ERROR occasionally occurs in datanode log: > ERROR BPServiceActor.java:792 - Exception in BPOfferService for Block pool > BP-1687106048-10.14.9.17-1535259994856 (Datanode Uuid > c17e635a-912f-4488-b4e8-93a58a27b5db) service to osscmh-9-21/10.14.9.21:8070 > java.util.ConcurrentModificationException: modification=62852685 != > iterModification = 62852684 > at > org.apache.hadoop.util.LightWeightGSet$SetIterator.ensureNext(LightWeightGSet.java:305) > at > org.apache.hadoop.util.LightWeightGSet$SetIterator.hasNext(LightWeightGSet.java:322) > at > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getBlockReports(FsDatasetImpl.java:1872) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport(BPServiceActor.java:349) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:648) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:790) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14219) ConcurrentModificationException occurs in datanode occasionally
[
https://issues.apache.org/jira/browse/HDFS-14219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770796#comment-16770796
]
Tao Jie commented on HDFS-14219:
I checked code in FsDatasetImpl.java, it seems missed one line change in
HDFS-10682 when it was merged into branch-2.8.x
{code}
List curVolumes = null;
-synchronized(this) {
+try (AutoCloseableLock lock = datasetLock.acquire()) {
curVolumes = volumes.getVolumes();
for (FsVolumeSpi v : curVolumes) {
builders.put(v.getStorageID(),
BlockListAsLongs.builder(maxDataLength));
{code}
[~vagarychen], [~arpitagarwal] would you have a look at this patch?
> ConcurrentModificationException occurs in datanode occasionally
> ---
>
> Key: HDFS-14219
> URL: https://issues.apache.org/jira/browse/HDFS-14219
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.2
>Reporter: Tao Jie
>Priority: Minor
> Attachments: HDFS-14219.001.patch
>
>
> ERROR occasionally occurs in datanode log:
> ERROR BPServiceActor.java:792 - Exception in BPOfferService for Block pool
> BP-1687106048-10.14.9.17-1535259994856 (Datanode Uuid
> c17e635a-912f-4488-b4e8-93a58a27b5db) service to osscmh-9-21/10.14.9.21:8070
> java.util.ConcurrentModificationException: modification=62852685 !=
> iterModification = 62852684
> at
> org.apache.hadoop.util.LightWeightGSet$SetIterator.ensureNext(LightWeightGSet.java:305)
> at
> org.apache.hadoop.util.LightWeightGSet$SetIterator.hasNext(LightWeightGSet.java:322)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getBlockReports(FsDatasetImpl.java:1872)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport(BPServiceActor.java:349)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:648)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:790)
> at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14219) ConcurrentModificationException occurs in datanode occasionally
[ https://issues.apache.org/jira/browse/HDFS-14219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Jie updated HDFS-14219: --- Attachment: HDFS-14219.001.patch > ConcurrentModificationException occurs in datanode occasionally > --- > > Key: HDFS-14219 > URL: https://issues.apache.org/jira/browse/HDFS-14219 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.2 >Reporter: Tao Jie >Priority: Minor > Attachments: HDFS-14219.001.patch > > > ERROR occasionally occurs in datanode log: > ERROR BPServiceActor.java:792 - Exception in BPOfferService for Block pool > BP-1687106048-10.14.9.17-1535259994856 (Datanode Uuid > c17e635a-912f-4488-b4e8-93a58a27b5db) service to osscmh-9-21/10.14.9.21:8070 > java.util.ConcurrentModificationException: modification=62852685 != > iterModification = 62852684 > at > org.apache.hadoop.util.LightWeightGSet$SetIterator.ensureNext(LightWeightGSet.java:305) > at > org.apache.hadoop.util.LightWeightGSet$SetIterator.hasNext(LightWeightGSet.java:322) > at > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getBlockReports(FsDatasetImpl.java:1872) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport(BPServiceActor.java:349) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:648) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:790) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1121) Key read failure when data is written parallel in to Ozone
[
https://issues.apache.org/jira/browse/HDDS-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770777#comment-16770777
]
Shashikant Banerjee commented on HDDS-1121:
---
Thanks [~bharatviswa] for reporting and working on this. Can you please rebase
the patch on trunk as its not applying anymore?
> Key read failure when data is written parallel in to Ozone
> --
>
> Key: HDDS-1121
> URL: https://issues.apache.org/jira/browse/HDDS-1121
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HDDS-1121.00.patch, HDDS-1121.01.patch
>
>
> When hive is run with multiple threads for data ingestion to ozone. After
> ingestion is done, during read we see this below error.
> This issue is found during hive testing, and found by [~t3rmin4t0r]
> {code:java}
> caused by: org.apache.hadoop.ozone.common.OzoneChecksumException: Checksum
> mismatch at index 0
> at
> org.apache.hadoop.ozone.common.ChecksumData.verifyChecksumDataMatches(ChecksumData.java:143)
> at org.apache.hadoop.ozone.common.Checksum.verifyChecksum(Checksum.java:239)
> at org.apache.hadoop.ozone.common.Checksum.verifyChecksum(Checksum.java:217)
> at
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.readChunkFromContainer(BlockInputStream.java:227)
> at
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.seek(BlockInputStream.java:259)
> at
> org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.seek(KeyInputStream.java:249)
> at
> org.apache.hadoop.ozone.client.io.KeyInputStream.seek(KeyInputStream.java:180)
> at
> org.apache.hadoop.fs.ozone.OzoneFSInputStream.seek(OzoneFSInputStream.java:62)
> at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:82)
> at org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121)
> at
> org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:111)
> at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:555)
> at org.apache.orc.impl.ReaderImpl.(ReaderImpl.java:370)
> at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.(ReaderImpl.java:61)
> at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:105)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.populateAndCacheStripeDetails(OrcInputFormat.java:1647)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.callInternal(OrcInputFormat.java:1533)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.access$2700(OrcInputFormat.java:1329)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1513)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1510)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1510)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1329)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-726) Ozone Client should update SCM to move the container out of allocation path in case a write transaction fails
[ https://issues.apache.org/jira/browse/HDDS-726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770775#comment-16770775 ] Shashikant Banerjee commented on HDDS-726: -- Thanks [~msingh]. Patch v4 is rebased. > Ozone Client should update SCM to move the container out of allocation path > in case a write transaction fails > - > > Key: HDDS-726 > URL: https://issues.apache.org/jira/browse/HDDS-726 > Project: Hadoop Distributed Data Store > Issue Type: Test >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDDS-726.000.patch, HDDS-726.001.patch, > HDDS-726.002.patch, HDDS-726.003.patch, HDDS-726.004.patch > > > Once an container write transaction fails, it will be marked corrupted. Once > Ozone client gets an exception in such case it should tell SCM to move the > container out of allocation path. SCM will eventually close the container. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-726) Ozone Client should update SCM to move the container out of allocation path in case a write transaction fails
[ https://issues.apache.org/jira/browse/HDDS-726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDDS-726: - Attachment: HDDS-726.004.patch > Ozone Client should update SCM to move the container out of allocation path > in case a write transaction fails > - > > Key: HDDS-726 > URL: https://issues.apache.org/jira/browse/HDDS-726 > Project: Hadoop Distributed Data Store > Issue Type: Test >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDDS-726.000.patch, HDDS-726.001.patch, > HDDS-726.002.patch, HDDS-726.003.patch, HDDS-726.004.patch > > > Once an container write transaction fails, it will be marked corrupted. Once > Ozone client gets an exception in such case it should tell SCM to move the > container out of allocation path. SCM will eventually close the container. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-726) Ozone Client should update SCM to move the container out of allocation path in case a write transaction fails
[ https://issues.apache.org/jira/browse/HDDS-726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770764#comment-16770764 ] Mukul Kumar Singh commented on HDDS-726: Thanks for updating the patch [~shashikant]. The latest patch does not apply. Can you please rebase the patch on top of latest trunk. > Ozone Client should update SCM to move the container out of allocation path > in case a write transaction fails > - > > Key: HDDS-726 > URL: https://issues.apache.org/jira/browse/HDDS-726 > Project: Hadoop Distributed Data Store > Issue Type: Test >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDDS-726.000.patch, HDDS-726.001.patch, > HDDS-726.002.patch, HDDS-726.003.patch > > > Once an container write transaction fails, it will be marked corrupted. Once > Ozone client gets an exception in such case it should tell SCM to move the > container out of allocation path. SCM will eventually close the container. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10419) Building HDFS on top of new storage layer (HDDS)
[
https://issues.apache.org/jira/browse/HDFS-10419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770763#comment-16770763
]
Brahma Reddy Battula commented on HDFS-10419:
-
{quote}bq. Konstantine has proposed other alternatives and we would evaluate
(a) or (b) for his alternative. I am not locked into any particular path or
how we would do it.
{quote}
Any conclusion on this idea? Looks it [~shv] idea is better. but I have
following doubts.Please do correct me.
* I think,BlockTokenSecretManger need to move out of Namenode.
* will we introduce two RPC's to Storagepolices,EC, Snapshots and fsck..etc
which associate with src and blockID
* Still working set can be keep in memory?
> Building HDFS on top of new storage layer (HDDS)
>
>
> Key: HDFS-10419
> URL: https://issues.apache.org/jira/browse/HDFS-10419
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Jing Zhao
>Assignee: Jing Zhao
>Priority: Major
> Attachments: Evolving NN using new block-container layer.pdf
>
>
> In HDFS-7240, Ozone defines storage containers to store both the data and the
> metadata. The storage container layer provides an object storage interface
> and aims to manage data/metadata in a distributed manner. More details about
> storage containers can be found in the design doc in HDFS-7240.
> HDFS can adopt the storage containers to store and manage blocks. The general
> idea is:
> # Each block can be treated as an object and the block ID is the object's key.
> # Blocks will still be stored in DataNodes but as objects in storage
> containers.
> # The block management work can be separated out of the NameNode and will be
> handled by the storage container layer in a more distributed way. The
> NameNode will only manage the namespace (i.e., files and directories).
> # For each file, the NameNode only needs to record a list of block IDs which
> are used as keys to obtain real data from storage containers.
> # A new DFSClient implementation talks to both NameNode and the storage
> container layer to read/write.
> HDFS, especially the NameNode, can get much better scalability from this
> design. Currently the NameNode's heaviest workload comes from the block
> management, which includes maintaining the block-DataNode mapping, receiving
> full/incremental block reports, tracking block states (under/over/miss
> replicated), and joining every writing pipeline protocol to guarantee the
> data consistency. These work bring high memory footprint and make NameNode
> suffer from GC. HDFS-5477 already proposes to convert BlockManager as a
> service. If we can build HDFS on top of the storage container layer, we not
> only separate out the BlockManager from the NameNode, but also replace it
> with a new distributed management scheme.
> The storage container work is currently in progress in HDFS-7240, and the
> work proposed here is still in an experimental/exploring stage. We can do
> this experiment in a feature branch so that people with interests can be
> involved.
> A design doc will be uploaded later explaining more details.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14290) Unexpected message type: PooledUnsafeDirectByteBuf when get datanode info by DatanodeWebHdfsMethods
[ https://issues.apache.org/jira/browse/HDFS-14290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14290: --- Description: The issue is there is no HttpRequestDecoder in InboundHandler of netty, appear unexpected message type when read message. !webhdfs show.png! DEBUG org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer: Proxy failed. Cause: com.xiaomi.infra.thirdparty.io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: PooledUnsafeDirectByteBuf at com.xiaomi.infra.thirdparty.io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106) at com.xiaomi.infra.thirdparty.io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) at com.xiaomi.infra.thirdparty.io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:304) at com.xiaomi.infra.thirdparty.io.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:137) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) at com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1051) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300) at org.apache.hadoop.hdfs.server.datanode.web.SimpleHttpProxyHandler$Forwarder.channelRead(SimpleHttpProxyHandler.java:80) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1414) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:945) at com.xiaomi.infra.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:146) at com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) at com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) at com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) at com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at com.xiaomi.infra.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886) at com.xiaomi.infra.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalStateException: unexpected message type: PooledUnsafeDirectByteBuf at com.xiaomi.infra.thirdparty.io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:123) at com.xiaomi.infra.thirdparty.io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88) ... 30 more 2018-12-04,14:23:28,690 DEBUG org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer: Proxy failed. Cause: java.nio.channels.ClosedChannelException at
[jira] [Commented] (HDDS-1092) Use Java 11 JRE to run Ozone in containers
[ https://issues.apache.org/jira/browse/HDDS-1092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770751#comment-16770751 ] Akira Ajisaka commented on HDDS-1092: - Thanks [~anu]! > Use Java 11 JRE to run Ozone in containers > -- > > Key: HDDS-1092 > URL: https://issues.apache.org/jira/browse/HDDS-1092 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Attachments: HDDS-1092-docker-hadoop-runner.002.patch, > HDDS-1092.001.patch > > Time Spent: 40m > Remaining Estimate: 0h > > As of now we use opendk 1.8.0 in the Ozone containers. > Java 9 and Java 10 introduces advanced support for the resource management of > the containers and not all of them are available from the latest release of > 1.8.0. (see this blog for more details: > https://medium.com/adorsys/jvm-memory-settings-in-a-container-environment-64b0840e1d9e) > I propose to switch to use Java 11 in the containers and test everything with > Java 11 at runtime. > Note: this issue is just about the runtime jdk not about the compile time JDK. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14290) Unexpected message type: PooledUnsafeDirectByteBuf when get datanode info by DatanodeWebHdfsMethods
[ https://issues.apache.org/jira/browse/HDFS-14290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14290: --- Attachment: image-2019-02-18-11-06-15-100.png > Unexpected message type: PooledUnsafeDirectByteBuf when get datanode info by > DatanodeWebHdfsMethods > --- > > Key: HDFS-14290 > URL: https://issues.apache.org/jira/browse/HDFS-14290 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, webhdfs >Affects Versions: 2.7.0, 2.7.1 >Reporter: Lisheng Sun >Priority: Major > Attachments: image-2019-02-18-11-06-15-100.png > > > The issue is there is no HttpRequestDecoder in InboundHandler of netty, > appear unexpected message type when read message. > > > > DEBUG org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer: Proxy > failed. Cause: > com.xiaomi.infra.thirdparty.io.netty.handler.codec.EncoderException: > java.lang.IllegalStateException: unexpected message type: > PooledUnsafeDirectByteBuf > at > com.xiaomi.infra.thirdparty.io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106) > at > com.xiaomi.infra.thirdparty.io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) > at > com.xiaomi.infra.thirdparty.io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:304) > at > com.xiaomi.infra.thirdparty.io.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:137) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) > at > com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1051) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300) > at > org.apache.hadoop.hdfs.server.datanode.web.SimpleHttpProxyHandler$Forwarder.channelRead(SimpleHttpProxyHandler.java:80) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) > at > com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1414) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) > at > com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:945) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:146) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) > at > com.xiaomi.infra.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886) > at >
[jira] [Updated] (HDFS-14290) Unexpected message type: PooledUnsafeDirectByteBuf when get datanode info by DatanodeWebHdfsMethods
[ https://issues.apache.org/jira/browse/HDFS-14290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14290: --- Attachment: (was: image-2019-02-18-11-06-15-100.png) > Unexpected message type: PooledUnsafeDirectByteBuf when get datanode info by > DatanodeWebHdfsMethods > --- > > Key: HDFS-14290 > URL: https://issues.apache.org/jira/browse/HDFS-14290 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, webhdfs >Affects Versions: 2.7.0, 2.7.1 >Reporter: Lisheng Sun >Priority: Major > > The issue is there is no HttpRequestDecoder in InboundHandler of netty, > appear unexpected message type when read message. > !image-2019-02-18-11-06-15-100.png! > > DEBUG org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer: Proxy > failed. Cause: > com.xiaomi.infra.thirdparty.io.netty.handler.codec.EncoderException: > java.lang.IllegalStateException: unexpected message type: > PooledUnsafeDirectByteBuf > at > com.xiaomi.infra.thirdparty.io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106) > at > com.xiaomi.infra.thirdparty.io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) > at > com.xiaomi.infra.thirdparty.io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:304) > at > com.xiaomi.infra.thirdparty.io.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:137) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) > at > com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1051) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300) > at > org.apache.hadoop.hdfs.server.datanode.web.SimpleHttpProxyHandler$Forwarder.channelRead(SimpleHttpProxyHandler.java:80) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) > at > com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1414) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) > at > com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:945) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:146) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) > at > com.xiaomi.infra.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886) > at >
[jira] [Updated] (HDFS-14290) Unexpected message type: PooledUnsafeDirectByteBuf when get datanode info by DatanodeWebHdfsMethods
[ https://issues.apache.org/jira/browse/HDFS-14290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14290: --- Attachment: webhdfs show.png > Unexpected message type: PooledUnsafeDirectByteBuf when get datanode info by > DatanodeWebHdfsMethods > --- > > Key: HDFS-14290 > URL: https://issues.apache.org/jira/browse/HDFS-14290 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, webhdfs >Affects Versions: 2.7.0, 2.7.1 >Reporter: Lisheng Sun >Priority: Major > Attachments: webhdfs show.png > > > The issue is there is no HttpRequestDecoder in InboundHandler of netty, > appear unexpected message type when read message. > !image-2019-02-18-11-06-15-100.png! > > DEBUG org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer: Proxy > failed. Cause: > com.xiaomi.infra.thirdparty.io.netty.handler.codec.EncoderException: > java.lang.IllegalStateException: unexpected message type: > PooledUnsafeDirectByteBuf > at > com.xiaomi.infra.thirdparty.io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106) > at > com.xiaomi.infra.thirdparty.io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) > at > com.xiaomi.infra.thirdparty.io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:304) > at > com.xiaomi.infra.thirdparty.io.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:137) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) > at > com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1051) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300) > at > org.apache.hadoop.hdfs.server.datanode.web.SimpleHttpProxyHandler$Forwarder.channelRead(SimpleHttpProxyHandler.java:80) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) > at > com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1414) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) > at > com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:945) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:146) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) > at > com.xiaomi.infra.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886) > at >
[jira] [Commented] (HDFS-14290) Unexpected message type: PooledUnsafeDirectByteBuf when get datanode info by DatanodeWebHdfsMethods
[ https://issues.apache.org/jira/browse/HDFS-14290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770749#comment-16770749 ] Ayush Saxena commented on HDFS-14290: - Seems same as HDFS-14289? > Unexpected message type: PooledUnsafeDirectByteBuf when get datanode info by > DatanodeWebHdfsMethods > --- > > Key: HDFS-14290 > URL: https://issues.apache.org/jira/browse/HDFS-14290 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, webhdfs >Affects Versions: 2.7.0, 2.7.1 >Reporter: Lisheng Sun >Priority: Major > Attachments: webhdfs show.png > > > The issue is there is no HttpRequestDecoder in InboundHandler of netty, > appear unexpected message type when read message. > > !webhdfs show.png! > DEBUG org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer: Proxy > failed. Cause: > com.xiaomi.infra.thirdparty.io.netty.handler.codec.EncoderException: > java.lang.IllegalStateException: unexpected message type: > PooledUnsafeDirectByteBuf > at > com.xiaomi.infra.thirdparty.io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106) > at > com.xiaomi.infra.thirdparty.io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) > at > com.xiaomi.infra.thirdparty.io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:304) > at > com.xiaomi.infra.thirdparty.io.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:137) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) > at > com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1051) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300) > at > org.apache.hadoop.hdfs.server.datanode.web.SimpleHttpProxyHandler$Forwarder.channelRead(SimpleHttpProxyHandler.java:80) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) > at > com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1414) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) > at > com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:945) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:146) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) > at > com.xiaomi.infra.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886) > at
[jira] [Updated] (HDFS-14290) Unexpected message type: PooledUnsafeDirectByteBuf when get datanode info by DatanodeWebHdfsMethods
[ https://issues.apache.org/jira/browse/HDFS-14290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14290: --- Description: The issue is there is no HttpRequestDecoder in InboundHandler of netty, appear unexpected message type when read message. !image-2019-02-18-11-06-15-100.png! DEBUG org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer: Proxy failed. Cause: com.xiaomi.infra.thirdparty.io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: PooledUnsafeDirectByteBuf at com.xiaomi.infra.thirdparty.io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106) at com.xiaomi.infra.thirdparty.io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) at com.xiaomi.infra.thirdparty.io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:304) at com.xiaomi.infra.thirdparty.io.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:137) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) at com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1051) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300) at org.apache.hadoop.hdfs.server.datanode.web.SimpleHttpProxyHandler$Forwarder.channelRead(SimpleHttpProxyHandler.java:80) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1414) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:945) at com.xiaomi.infra.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:146) at com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) at com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) at com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) at com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at com.xiaomi.infra.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886) at com.xiaomi.infra.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalStateException: unexpected message type: PooledUnsafeDirectByteBuf at com.xiaomi.infra.thirdparty.io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:123) at com.xiaomi.infra.thirdparty.io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88) ... 30 more 2018-12-04,14:23:28,690 DEBUG org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer: Proxy failed. Cause: java.nio.channels.ClosedChannelException at
[jira] [Commented] (HDFS-3246) pRead equivalent for direct read path
[ https://issues.apache.org/jira/browse/HDFS-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770748#comment-16770748 ] Zheng Hu commented on HDFS-3246: Discussed with [~jojochuang] in HBASE-21879 (thanks for his helpful comment), so this feature should go into Hadoop2.x, seems we done part of work in HDFS-8901(the internal bytebuffer pread implementation), while it has not been backported to our Hadoop2.x. I mean we should make both this issue and HDFS-8901 go into Hadoop2.x. Thanks. > pRead equivalent for direct read path > - > > Key: HDFS-3246 > URL: https://issues.apache.org/jira/browse/HDFS-3246 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, performance >Affects Versions: 3.0.0-alpha1 >Reporter: Henry Robinson >Assignee: Chen Zhang >Priority: Major > > There is no pread equivalent in ByteBufferReadable. We should consider adding > one. It would be relatively easy to implement for the distributed case > (certainly compared to HDFS-2834), since DFSInputStream does most of the > heavy lifting. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14290) Unexpected message type: PooledUnsafeDirectByteBuf when get datanode info by DatanodeWebHdfsMethods
[ https://issues.apache.org/jira/browse/HDFS-14290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14290: --- Affects Version/s: 2.7.1 > Unexpected message type: PooledUnsafeDirectByteBuf when get datanode info by > DatanodeWebHdfsMethods > --- > > Key: HDFS-14290 > URL: https://issues.apache.org/jira/browse/HDFS-14290 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, webhdfs >Affects Versions: 2.7.0, 2.7.1 >Reporter: Lisheng Sun >Priority: Major > > The issue is there is no HttpRequestDecoder in InboundHandler of netty, > appear unexpected message type when read message. > > > > DEBUG org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer: Proxy > failed. Cause: > com.xiaomi.infra.thirdparty.io.netty.handler.codec.EncoderException: > java.lang.IllegalStateException: unexpected message type: > PooledUnsafeDirectByteBuf > at > com.xiaomi.infra.thirdparty.io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106) > at > com.xiaomi.infra.thirdparty.io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) > at > com.xiaomi.infra.thirdparty.io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:304) > at > com.xiaomi.infra.thirdparty.io.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:137) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) > at > com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1051) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300) > at > org.apache.hadoop.hdfs.server.datanode.web.SimpleHttpProxyHandler$Forwarder.channelRead(SimpleHttpProxyHandler.java:80) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) > at > com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1414) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) > at > com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) > at > com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:945) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:146) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) > at > com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) > at > com.xiaomi.infra.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886) > at > com.xiaomi.infra.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at
[jira] [Created] (HDFS-14290) Unexpected message type: PooledUnsafeDirectByteBuf when get datanode info by DatanodeWebHdfsMethods
Lisheng Sun created HDFS-14290: -- Summary: Unexpected message type: PooledUnsafeDirectByteBuf when get datanode info by DatanodeWebHdfsMethods Key: HDFS-14290 URL: https://issues.apache.org/jira/browse/HDFS-14290 Project: Hadoop HDFS Issue Type: Bug Components: datanode, webhdfs Affects Versions: 2.7.0 Reporter: Lisheng Sun The issue is there is no HttpRequestDecoder in InboundHandler of netty, appear unexpected message type when read message. DEBUG org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer: Proxy failed. Cause: com.xiaomi.infra.thirdparty.io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: PooledUnsafeDirectByteBuf at com.xiaomi.infra.thirdparty.io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106) at com.xiaomi.infra.thirdparty.io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) at com.xiaomi.infra.thirdparty.io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:304) at com.xiaomi.infra.thirdparty.io.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:137) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) at com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1051) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300) at org.apache.hadoop.hdfs.server.datanode.web.SimpleHttpProxyHandler$Forwarder.channelRead(SimpleHttpProxyHandler.java:80) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1414) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:945) at com.xiaomi.infra.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:146) at com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) at com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) at com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) at com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at com.xiaomi.infra.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886) at com.xiaomi.infra.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalStateException: unexpected message type: PooledUnsafeDirectByteBuf at com.xiaomi.infra.thirdparty.io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:123) at com.xiaomi.infra.thirdparty.io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88) ... 30 more 2018-12-04,14:23:28,690 DEBUG
[jira] [Commented] (HDFS-3246) pRead equivalent for direct read path
[ https://issues.apache.org/jira/browse/HDFS-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770727#comment-16770727 ] Zheng Hu commented on HDFS-3246: [~jojochuang], Mind you help to grant the JIRA permission to [~stakiar], he is working on this feature, and out HBase will benifit a lot from his work. Thanks [~stakiar]. > pRead equivalent for direct read path > - > > Key: HDFS-3246 > URL: https://issues.apache.org/jira/browse/HDFS-3246 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, performance >Affects Versions: 3.0.0-alpha1 >Reporter: Henry Robinson >Assignee: Chen Zhang >Priority: Major > > There is no pread equivalent in ByteBufferReadable. We should consider adding > one. It would be relatively easy to implement for the distributed case > (certainly compared to HDFS-2834), since DFSInputStream does most of the > heavy lifting. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14289) Unexpected message type: PooledUnsafeDirectByteBuf when get datanode info by DatanodeWebHdfsMethods
Lisheng Sun created HDFS-14289: -- Summary: Unexpected message type: PooledUnsafeDirectByteBuf when get datanode info by DatanodeWebHdfsMethods Key: HDFS-14289 URL: https://issues.apache.org/jira/browse/HDFS-14289 Project: Hadoop HDFS Issue Type: Bug Components: datanode, webhdfs Affects Versions: 2.7.0 Reporter: Lisheng Sun DEBUG org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer: Proxy failed. Cause: com.xiaomi.infra.thirdparty.io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: PooledUnsafeDirectByteBuf at com.xiaomi.infra.thirdparty.io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106) at com.xiaomi.infra.thirdparty.io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) at com.xiaomi.infra.thirdparty.io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:304) at com.xiaomi.infra.thirdparty.io.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:137) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) at com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1051) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300) at org.apache.hadoop.hdfs.server.datanode.web.SimpleHttpProxyHandler$Forwarder.channelRead(SimpleHttpProxyHandler.java:80) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1414) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:945) at com.xiaomi.infra.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:146) at com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) at com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) at com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) at com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at com.xiaomi.infra.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886) at com.xiaomi.infra.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalStateException: unexpected message type: PooledUnsafeDirectByteBuf at com.xiaomi.infra.thirdparty.io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:123) at com.xiaomi.infra.thirdparty.io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88) ... 30 more 2018-12-04,14:23:28,690 DEBUG org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer: Proxy failed. Cause: java.nio.channels.ClosedChannelException at
[jira] [Commented] (HDFS-14130) Make ZKFC ObserverNode aware
[ https://issues.apache.org/jira/browse/HDFS-14130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770708#comment-16770708 ] xiangheng commented on HDFS-14130: -- Thanks [~csun] and [~shv],**Because one of my long holidays(The Spring Festival), delayed the progress of this patch,Thanks all people. > Make ZKFC ObserverNode aware > > > Key: HDFS-14130 > URL: https://issues.apache.org/jira/browse/HDFS-14130 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HDFS-12943 >Reporter: Konstantin Shvachko >Assignee: xiangheng >Priority: Major > Attachments: HDFS-14130-HDFS-12943.001.patch, > HDFS-14130-HDFS-12943.003.patch, HDFS-14130-HDFS-12943.004.patch, > HDFS-14130-HDFS-12943.005.patch, HDFS-14130-HDFS-12943.006.patch, > HDFS-14130-HDFS-12943.007.patch, HDFS-14130.008.patch > > > Need to fix automatic failover with ZKFC. Currently it does not know about > ObserverNodes trying to convert them to SBNs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14130) Make ZKFC ObserverNode aware
[
https://issues.apache.org/jira/browse/HDFS-14130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770703#comment-16770703
]
Hadoop QA commented on HDFS-14130:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
40s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 14s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
1s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 24s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
11s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 99m 13s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
48s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}214m 14s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14130 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959031/HDFS-14130.008.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux f1a3096e4443 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ba56bc2 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
[jira] [Commented] (HDFS-14279) [SBN Read] Race condition in ObserverReadProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770635#comment-16770635
]
Konstantin Shvachko commented on HDFS-14279:
I think we can simplify it mere:
{code}
private NNProxyInfo getCurrentProxy() {
return changeProxy(null);
}
{code}
> [SBN Read] Race condition in ObserverReadProxyProvider
> --
>
> Key: HDFS-14279
> URL: https://issues.apache.org/jira/browse/HDFS-14279
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, namenode
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-14279.000.patch
>
>
> There is a race condition in {{ObserverReadProxyProvider#getCurrentProxy()}}:
> {code}
> private NNProxyInfo getCurrentProxy() {
> if (currentProxy == null) {
> changeProxy(null);
> }
> return currentProxy;
> }
> {code}
> {{currentProxy}} is a {{volatile}}. Another {{changeProxy()}} could occur
> after the {{changeProxy()}} and before the {{return}}, thus making the return
> value incorrect. I have seen this result in an NPE.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14288) [SBN read] Observer should not allow write throw webhdfs commands
Brahma Reddy Battula created HDFS-14288: --- Summary: [SBN read] Observer should not allow write throw webhdfs commands Key: HDFS-14288 URL: https://issues.apache.org/jira/browse/HDFS-14288 Project: Hadoop HDFS Issue Type: Bug Reporter: Brahma Reddy Battula Assignee: Brahma Reddy Battula Currently Observer node allows writes through webhdfs commands ( even through UI). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14130) Make ZKFC ObserverNode aware
[ https://issues.apache.org/jira/browse/HDFS-14130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770633#comment-16770633 ] Konstantin Shvachko commented on HDFS-14130: Updated the patch to fix the tests and made some corrections in comments. Also made it against trunk rather than the branch, which has been already merged. > Make ZKFC ObserverNode aware > > > Key: HDFS-14130 > URL: https://issues.apache.org/jira/browse/HDFS-14130 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HDFS-12943 >Reporter: Konstantin Shvachko >Assignee: xiangheng >Priority: Major > Attachments: HDFS-14130-HDFS-12943.001.patch, > HDFS-14130-HDFS-12943.003.patch, HDFS-14130-HDFS-12943.004.patch, > HDFS-14130-HDFS-12943.005.patch, HDFS-14130-HDFS-12943.006.patch, > HDFS-14130-HDFS-12943.007.patch, HDFS-14130.008.patch > > > Need to fix automatic failover with ZKFC. Currently it does not know about > ObserverNodes trying to convert them to SBNs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14130) Make ZKFC ObserverNode aware
[ https://issues.apache.org/jira/browse/HDFS-14130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-14130: --- Attachment: HDFS-14130.008.patch > Make ZKFC ObserverNode aware > > > Key: HDFS-14130 > URL: https://issues.apache.org/jira/browse/HDFS-14130 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HDFS-12943 >Reporter: Konstantin Shvachko >Assignee: xiangheng >Priority: Major > Attachments: HDFS-14130-HDFS-12943.001.patch, > HDFS-14130-HDFS-12943.003.patch, HDFS-14130-HDFS-12943.004.patch, > HDFS-14130-HDFS-12943.005.patch, HDFS-14130-HDFS-12943.006.patch, > HDFS-14130-HDFS-12943.007.patch, HDFS-14130.008.patch > > > Need to fix automatic failover with ZKFC. Currently it does not know about > ObserverNodes trying to convert them to SBNs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1088) Add blockade Tests to test Replica Manager
[
https://issues.apache.org/jira/browse/HDDS-1088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770612#comment-16770612
]
Hadoop QA commented on HDDS-1088:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 12s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
20s{color} | {color:red} dist in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} pylint {color} | {color:orange} 0m
12s{color} | {color:orange} The patch generated 90 new + 356 unchanged - 11
fixed = 446 total (was 367) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 15s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
20s{color} | {color:green} dist in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-1088 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959027/HDDS-1088.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient pylint |
| uname | Linux 59862f1ef62e 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ba56bc2 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDDS-Build/2297/artifact/out/patch-mvninstall-hadoop-ozone_dist.txt
|
| pylint | v1.9.2 |
| pylint |
https://builds.apache.org/job/PreCommit-HDDS-Build/2297/artifact/out/diff-patch-pylint.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDDS-Build/2297/testReport/ |
| Max. process+thread count | 412 (vs. ulimit of 1) |
| modules | C: hadoop-ozone/dist U: hadoop-ozone/dist |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/2297/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add blockade Tests to test Replica Manager
> --
>
> Key: HDDS-1088
> URL: https://issues.apache.org/jira/browse/HDDS-1088
> Project:
[jira] [Created] (HDDS-1125) java.lang.InterruptedException seen in datanode logs
Nilotpal Nandi created HDDS-1125:
Summary: java.lang.InterruptedException seen in datanode logs
Key: HDDS-1125
URL: https://issues.apache.org/jira/browse/HDDS-1125
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Nilotpal Nandi
steps taken :
# created 12 datanodes cluster and running workload on all the nodes
exception seen :
-
{noformat}
2019-02-15 10:16:48,713 ERROR org.apache.ratis.server.impl.LogAppender:
943007c8-4fdd-4926-89e2-2c8c52c05073: Failed readStateMachineData for (t:3,
i:3084), STATEMACHINELOGENTRY, client-632E77ADA885, cid=6232
java.lang.InterruptedException
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:347)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
at
org.apache.ratis.server.storage.RaftLog$EntryWithData.getEntry(RaftLog.java:433)
at org.apache.ratis.util.DataQueue.pollList(DataQueue.java:133)
at org.apache.ratis.server.impl.LogAppender.createRequest(LogAppender.java:171)
at
org.apache.ratis.grpc.server.GrpcLogAppender.appendLog(GrpcLogAppender.java:152)
at
org.apache.ratis.grpc.server.GrpcLogAppender.runAppenderImpl(GrpcLogAppender.java:96)
at org.apache.ratis.server.impl.LogAppender.runAppender(LogAppender.java:101)
at java.lang.Thread.run(Thread.java:748)
2019-02-15 10:16:48,714 ERROR org.apache.ratis.server.impl.LogAppender:
GrpcLogAppender(943007c8-4fdd-4926-89e2-2c8c52c05073 ->
8c77b16b-8054-49e3-b669-1ff759cfd271) hit IOException while loading raft log
org.apache.ratis.server.storage.RaftLogIOException:
943007c8-4fdd-4926-89e2-2c8c52c05073: Failed readStateMachineData for (t:3,
i:3084), STATEMACHINELOGENTRY, client-632E77ADA885, cid=6232
at
org.apache.ratis.server.storage.RaftLog$EntryWithData.getEntry(RaftLog.java:440)
at org.apache.ratis.util.DataQueue.pollList(DataQueue.java:133)
at org.apache.ratis.server.impl.LogAppender.createRequest(LogAppender.java:171)
at
org.apache.ratis.grpc.server.GrpcLogAppender.appendLog(GrpcLogAppender.java:152)
at
org.apache.ratis.grpc.server.GrpcLogAppender.runAppenderImpl(GrpcLogAppender.java:96)
at org.apache.ratis.server.impl.LogAppender.runAppender(LogAppender.java:101)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.InterruptedException
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:347)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
at
org.apache.ratis.server.storage.RaftLog$EntryWithData.getEntry(RaftLog.java:433)
... 6 more
2019-02-15 10:16:48,715 ERROR org.apache.ratis.server.impl.LogAppender:
943007c8-4fdd-4926-89e2-2c8c52c05073: Failed readStateMachineData for (t:3,
i:3084), STATEMACHINELOGENTRY, client-632E77ADA885, cid=6232
java.lang.InterruptedException
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:347)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
at
org.apache.ratis.server.storage.RaftLog$EntryWithData.getEntry(RaftLog.java:433)
at org.apache.ratis.util.DataQueue.pollList(DataQueue.java:133)
at org.apache.ratis.server.impl.LogAppender.createRequest(LogAppender.java:171)
at
org.apache.ratis.grpc.server.GrpcLogAppender.appendLog(GrpcLogAppender.java:152)
at
org.apache.ratis.grpc.server.GrpcLogAppender.runAppenderImpl(GrpcLogAppender.java:96)
at org.apache.ratis.server.impl.LogAppender.runAppender(LogAppender.java:101)
at java.lang.Thread.run(Thread.java:748)
2019-02-15 10:16:48,715 ERROR org.apache.ratis.server.impl.LogAppender:
GrpcLogAppender(943007c8-4fdd-4926-89e2-2c8c52c05073 ->
a40a7b01-a30b-469c-b373-9fcb20a126ed) hit IOException while loading raft log
org.apache.ratis.server.storage.RaftLogIOException:
943007c8-4fdd-4926-89e2-2c8c52c05073: Failed readStateMachineData for (t:3,
i:3084), STATEMACHINELOGENTRY, client-632E77ADA885, cid=6232
at
org.apache.ratis.server.storage.RaftLog$EntryWithData.getEntry(RaftLog.java:440)
at org.apache.ratis.util.DataQueue.pollList(DataQueue.java:133)
at org.apache.ratis.server.impl.LogAppender.createRequest(LogAppender.java:171)
at
org.apache.ratis.grpc.server.GrpcLogAppender.appendLog(GrpcLogAppender.java:152)
at
org.apache.ratis.grpc.server.GrpcLogAppender.runAppenderImpl(GrpcLogAppender.java:96)
at org.apache.ratis.server.impl.LogAppender.runAppender(LogAppender.java:101)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.InterruptedException
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:347)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
at
org.apache.ratis.server.storage.RaftLog$EntryWithData.getEntry(RaftLog.java:433)
... 6 more
2019-02-15 10:16:48,723 WARN
org.apache.ratis.grpc.client.GrpcClientProtocolService:
943007c8-4fdd-4926-89e2-2c8c52c05073-5: onError:
[jira] [Created] (HDDS-1124) java.lang.IllegalStateException exception in datanode log
Nilotpal Nandi created HDDS-1124:
Summary: java.lang.IllegalStateException exception in datanode log
Key: HDDS-1124
URL: https://issues.apache.org/jira/browse/HDDS-1124
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Nilotpal Nandi
steps taken :
# created 12 datanodes cluster and running workload on all the nodes
exception seen :
---
{noformat}
2019-02-15 10:15:53,355 INFO org.apache.ratis.server.storage.RaftLogWorker:
943007c8-4fdd-4926-89e2-2c8c52c05073-RaftLogWorker: Rolled log segment from
/data/disk1/ozone/meta/ratis/01d3ef2a-912c-4fc0-80b6-012343d76adb/current/log_inprogress_3036
to
/data/disk1/ozone/meta/ratis/01d3ef2a-912c-4fc0-80b6-012343d76adb/current/log_3036-3047
2019-02-15 10:15:53,367 INFO org.apache.ratis.server.impl.RaftServerImpl:
943007c8-4fdd-4926-89e2-2c8c52c05073: set configuration 3048:
[a40a7b01-a30b-469c-b373-9fcb20a126ed:172.27.54.212:9858,
8c77b16b-8054-49e3-b669-1ff759cfd271:172.27.23.196:9858,
943007c8-4fdd-4926-89e2-2c8c52c05073:172.27.76.72:9858], old=null at 3048
2019-02-15 10:15:53,523 INFO org.apache.ratis.server.storage.RaftLogWorker:
943007c8-4fdd-4926-89e2-2c8c52c05073-RaftLogWorker: created new log segment
/data/disk1/ozone/meta/ratis/01d3ef2a-912c-4fc0-80b6-012343d76adb/current/log_inprogress_3048
2019-02-15 10:15:53,580 ERROR org.apache.ratis.grpc.server.GrpcLogAppender:
Failed onNext serverReply {
requestorId: "943007c8-4fdd-4926-89e2-2c8c52c05073"
replyId: "a40a7b01-a30b-469c-b373-9fcb20a126ed"
raftGroupId {
id: "\001\323\357*\221,O\300\200\266\001#C\327j\333"
}
success: true
}
term: 3
nextIndex: 3049
followerCommit: 3047
java.lang.IllegalStateException: reply's next index is 3049, request's previous
is term: 1
index: 3047
at org.apache.ratis.util.Preconditions.assertTrue(Preconditions.java:60)
at
org.apache.ratis.grpc.server.GrpcLogAppender.onSuccess(GrpcLogAppender.java:285)
at
org.apache.ratis.grpc.server.GrpcLogAppender$AppendLogResponseHandler.onNextImpl(GrpcLogAppender.java:230)
at
org.apache.ratis.grpc.server.GrpcLogAppender$AppendLogResponseHandler.onNext(GrpcLogAppender.java:215)
at
org.apache.ratis.grpc.server.GrpcLogAppender$AppendLogResponseHandler.onNext(GrpcLogAppender.java:197)
at
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:421)
at
org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
at
org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:519)
at
org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2019-02-15 10:15:56,442 INFO org.apache.ratis.server.storage.RaftLogWorker:
943007c8-4fdd-4926-89e2-2c8c52c05073-RaftLogWorker: Rolling segment
log-3048_3066 to index:3066
2019-02-15 10:15:56,442 INFO org.apache.ratis.server.storage.RaftLogWorker:
943007c8-4fdd-4926-89e2-2c8c52c05073-RaftLogWorker: Rolled log segment from
/data/disk1/ozone/meta/ratis/01d3ef2a-912c-4fc0-80b6-012343d76adb/current/log_inprogress_3048
to
/data/disk1/ozone/meta/ratis/01d3ef2a-912c-4fc0-80b6-012343d76adb/current/log_3048-3066
2019-02-15 10:15:56,564 INFO org.apache.ratis.server.storage.RaftLogWorker:
943007c8-4fdd-4926-89e2-2c8c52c05073-RaftLogWorker: created new log segment
/data/disk1/ozone/meta/ratis/01d3ef2a-912c-4fc0-80b6-012343d76adb/current/log_inprogress_3067
2019-02-15 10:16:45,420 INFO org.apache.ratis.server.storage.RaftLogWorker:
943007c8-4fdd-4926-89e2-2c8c52c05073-RaftLogWorker: Rolling segment
log-3067_3077 to index:3077
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1088) Add blockade Tests to test Replica Manager
[ https://issues.apache.org/jira/browse/HDDS-1088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nilotpal Nandi updated HDDS-1088: - Attachment: HDDS-1088.001.patch > Add blockade Tests to test Replica Manager > -- > > Key: HDDS-1088 > URL: https://issues.apache.org/jira/browse/HDDS-1088 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Nilotpal Nandi >Assignee: Nilotpal Nandi >Priority: Major > Attachments: HDDS-1088.001.patch > > > We need to add tests for testing Replica Manager for scenarios like loss of > node, adding new nodes, under-replicated containers -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1088) Add blockade Tests to test Replica Manager
[ https://issues.apache.org/jira/browse/HDDS-1088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nilotpal Nandi updated HDDS-1088: - Status: Patch Available (was: Open) > Add blockade Tests to test Replica Manager > -- > > Key: HDDS-1088 > URL: https://issues.apache.org/jira/browse/HDDS-1088 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Nilotpal Nandi >Assignee: Nilotpal Nandi >Priority: Major > Attachments: HDDS-1088.001.patch > > > We need to add tests for testing Replica Manager for scenarios like loss of > node, adding new nodes, under-replicated containers -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1088) Add blockade Tests to test Replica Manager
[ https://issues.apache.org/jira/browse/HDDS-1088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nilotpal Nandi reassigned HDDS-1088: Assignee: Nilotpal Nandi > Add blockade Tests to test Replica Manager > -- > > Key: HDDS-1088 > URL: https://issues.apache.org/jira/browse/HDDS-1088 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Nilotpal Nandi >Assignee: Nilotpal Nandi >Priority: Major > > We need to add tests for testing Replica Manager for scenarios like loss of > node, adding new nodes, under-replicated containers -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1102) docker datanode stopped when new datanodes are added to the cluster
[
https://issues.apache.org/jira/browse/HDDS-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nilotpal Nandi updated HDDS-1102:
-
Attachment: allnode.log
> docker datanode stopped when new datanodes are added to the cluster
>
>
> Key: HDDS-1102
> URL: https://issues.apache.org/jira/browse/HDDS-1102
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Nilotpal Nandi
>Priority: Major
> Attachments: allnode.log, datanode.log
>
>
> steps taken:
>
> # created 5 datanode cluster.
> # shutdown 2 datanodes
> # started the datanodes again.
> One of the datanodes was shut down.
> exception seen :
>
> {noformat}
> 2019-02-14 07:37:26 INFO LeaderElection:230 -
> 6a0522ba-019e-4b77-ac1f-a9322cd525b8 got exception when requesting votes: {}
> java.util.concurrent.ExecutionException:
> org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL:
> a3d1dd2d-554e-4e87-a2cf-076a229af352: group-FD6FA533F1FB not found.
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:214)
> at
> org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:146)
> at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:102)
> Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException:
> INTERNAL: a3d1dd2d-554e-4e87-a2cf-076a229af352: group-FD6FA533F1FB not found.
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233)
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214)
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139)
> at
> org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265)
> at
> org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:83)
> at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:187)
> at
> org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$0(LeaderElection.java:188)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2019-02-14 07:37:26 INFO LeaderElection:46 -
> 6a0522ba-019e-4b77-ac1f-a9322cd525b8: Election PASSED; received 1 response(s)
> [6a0522ba-019e-4b77-ac1f-a9322cd525b8<-61ad3bf3-e9b1-48e5-90e3-3b78c8b5bba5#0:OK-t7]
> and 1 exception(s); 6a0522ba-019e-4b77-ac1f-a9322cd525b8:t7, leader=null,
> voted=6a0522ba-019e-4b77-ac1f-a9322cd525b8,
> raftlog=6a0522ba-019e-4b77-ac1f-a9322cd525b8-SegmentedRaftLog:OPENED, conf=3:
> [61ad3bf3-e9b1-48e5-90e3-3b78c8b5bba5:172.20.0.8:9858,
> 6a0522ba-019e-4b77-ac1f-a9322cd525b8:172.20.0.6:9858,
> 0f377918-aafa-4d8a-972a-6ead54048fba:172.20.0.3:9858], old=null
> 2019-02-14 07:37:26 INFO LeaderElection:52 - 0:
> java.util.concurrent.ExecutionException:
> org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL:
> a3d1dd2d-554e-4e87-a2cf-076a229af352: group-FD6FA533F1FB not found.
> 2019-02-14 07:37:26 INFO RoleInfo:130 - 6a0522ba-019e-4b77-ac1f-a9322cd525b8:
> shutdown LeaderElection
> 2019-02-14 07:37:26 INFO RaftServerImpl:161 -
> 6a0522ba-019e-4b77-ac1f-a9322cd525b8 changes role from CANDIDATE to LEADER at
> term 7 for changeToLeader
> 2019-02-14 07:37:26 INFO RaftServerImpl:258 -
> 6a0522ba-019e-4b77-ac1f-a9322cd525b8: change Leader from null to
> 6a0522ba-019e-4b77-ac1f-a9322cd525b8 at term 7 for becomeLeader, leader
> elected after 1066ms
> 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 -
> raft.server.staging.catchup.gap = 1000 (default)
> 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 - raft.server.rpc.sleep.time
> = 25ms (default)
> 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 - raft.server.watch.timeout
> = 10s (default)
> 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 -
> raft.server.watch.timeout.denomination = 1s (default)
> 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 -
> raft.server.log.appender.snapshot.chunk.size.max = 16MB (=16777216) (default)
> 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 -
> raft.server.log.appender.buffer.byte-limit = 33554432 (custom)
> 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 -
>
[jira] [Commented] (HDDS-1102) docker datanode stopped when new datanodes are added to the cluster
[
https://issues.apache.org/jira/browse/HDDS-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770480#comment-16770480
]
Nilotpal Nandi commented on HDDS-1102:
--
Here is all node logs for a different run :
[^allnode.log]
> docker datanode stopped when new datanodes are added to the cluster
>
>
> Key: HDDS-1102
> URL: https://issues.apache.org/jira/browse/HDDS-1102
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Nilotpal Nandi
>Priority: Major
> Attachments: allnode.log, datanode.log
>
>
> steps taken:
>
> # created 5 datanode cluster.
> # shutdown 2 datanodes
> # started the datanodes again.
> One of the datanodes was shut down.
> exception seen :
>
> {noformat}
> 2019-02-14 07:37:26 INFO LeaderElection:230 -
> 6a0522ba-019e-4b77-ac1f-a9322cd525b8 got exception when requesting votes: {}
> java.util.concurrent.ExecutionException:
> org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL:
> a3d1dd2d-554e-4e87-a2cf-076a229af352: group-FD6FA533F1FB not found.
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:214)
> at
> org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:146)
> at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:102)
> Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException:
> INTERNAL: a3d1dd2d-554e-4e87-a2cf-076a229af352: group-FD6FA533F1FB not found.
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233)
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214)
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139)
> at
> org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265)
> at
> org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:83)
> at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:187)
> at
> org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$0(LeaderElection.java:188)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2019-02-14 07:37:26 INFO LeaderElection:46 -
> 6a0522ba-019e-4b77-ac1f-a9322cd525b8: Election PASSED; received 1 response(s)
> [6a0522ba-019e-4b77-ac1f-a9322cd525b8<-61ad3bf3-e9b1-48e5-90e3-3b78c8b5bba5#0:OK-t7]
> and 1 exception(s); 6a0522ba-019e-4b77-ac1f-a9322cd525b8:t7, leader=null,
> voted=6a0522ba-019e-4b77-ac1f-a9322cd525b8,
> raftlog=6a0522ba-019e-4b77-ac1f-a9322cd525b8-SegmentedRaftLog:OPENED, conf=3:
> [61ad3bf3-e9b1-48e5-90e3-3b78c8b5bba5:172.20.0.8:9858,
> 6a0522ba-019e-4b77-ac1f-a9322cd525b8:172.20.0.6:9858,
> 0f377918-aafa-4d8a-972a-6ead54048fba:172.20.0.3:9858], old=null
> 2019-02-14 07:37:26 INFO LeaderElection:52 - 0:
> java.util.concurrent.ExecutionException:
> org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL:
> a3d1dd2d-554e-4e87-a2cf-076a229af352: group-FD6FA533F1FB not found.
> 2019-02-14 07:37:26 INFO RoleInfo:130 - 6a0522ba-019e-4b77-ac1f-a9322cd525b8:
> shutdown LeaderElection
> 2019-02-14 07:37:26 INFO RaftServerImpl:161 -
> 6a0522ba-019e-4b77-ac1f-a9322cd525b8 changes role from CANDIDATE to LEADER at
> term 7 for changeToLeader
> 2019-02-14 07:37:26 INFO RaftServerImpl:258 -
> 6a0522ba-019e-4b77-ac1f-a9322cd525b8: change Leader from null to
> 6a0522ba-019e-4b77-ac1f-a9322cd525b8 at term 7 for becomeLeader, leader
> elected after 1066ms
> 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 -
> raft.server.staging.catchup.gap = 1000 (default)
> 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 - raft.server.rpc.sleep.time
> = 25ms (default)
> 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 - raft.server.watch.timeout
> = 10s (default)
> 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 -
> raft.server.watch.timeout.denomination = 1s (default)
> 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 -
> raft.server.log.appender.snapshot.chunk.size.max = 16MB (=16777216) (default)
> 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 -
> raft.server.log.appender.buffer.byte-limit = 33554432 (custom)
> 2019-02-14
[jira] [Commented] (HDFS-13118) SnapshotDiffReport should provide the INode type
[
https://issues.apache.org/jira/browse/HDFS-13118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770411#comment-16770411
]
Hadoop QA commented on HDFS-13118:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 47s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 3m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
11s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 5s{color} | {color:orange} hadoop-hdfs-project: The patch generated 26 new
+ 384 unchanged - 6 fixed = 410 total (was 390) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 25s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
42s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client generated 1 new
+ 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
44s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 30s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}169m 21s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs-client |
| | Null passed for non-null parameter of
org.apache.hadoop.hdfs.protocol.proto.HdfsProtos$SnapshotDiffReportListingEntryProto$Builder.setFileType(HdfsProtos$HdfsFileStatusProto$FileType)
in
org.apache.hadoop.hdfs.protocolPB.PBHelperClient.convert(SnapshotDiffReportListing$DiffReportListingEntry)
Method invoked at PBHelperClient.java:of
org.apache.hadoop.hdfs.protocol.proto.HdfsProtos$SnapshotDiffReportListingEntryProto$Builder.setFileType(HdfsProtos$HdfsFileStatusProto$FileType)
in
[jira] [Commented] (HDDS-1120) Add a config to disable checksum verification during read even though checksum data is present in the persisted data
[ https://issues.apache.org/jira/browse/HDDS-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770393#comment-16770393 ] Yiqun Lin commented on HDDS-1120: - The patch almost looks good to me, [~bharatviswa]. Would you mind adding an unit test for the case of JIRA description mentioned? Will like following behaviour: With checksum persisted, then corrupt the data * If enabled the checksum verification (by default way), the read operation will fail. * If disabled, the read operation should be successful. > Add a config to disable checksum verification during read even though > checksum data is present in the persisted data > > > Key: HDDS-1120 > URL: https://issues.apache.org/jira/browse/HDDS-1120 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Affects Versions: 0.4.0 >Reporter: Shashikant Banerjee >Assignee: Bharat Viswanadham >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-1120.00.patch > > > Currently, if the checksum is computed during data write and persisted in the > disk, we will always end up verifying it while reading. This Jira aims to > selectively disable checksum verification during reads even though checksum > info is present in the data stored. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12478) [PROVIDED Phase 2] Command line tools for managing Provided Storage Backup mounts
[
https://issues.apache.org/jira/browse/HDFS-12478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770391#comment-16770391
]
Hadoop QA commented on HDFS-12478:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-12090 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
32s{color} | {color:green} HDFS-12090 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
59s{color} | {color:green} HDFS-12090 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} HDFS-12090 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
25s{color} | {color:green} HDFS-12090 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
26s{color} | {color:green} HDFS-12090 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
59s{color} | {color:green} HDFS-12090 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
30s{color} | {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
29s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
19s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
30s{color} | {color:red} hadoop-hdfs-project in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 30s{color} |
{color:red} hadoop-hdfs-project in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 30s{color}
| {color:red} hadoop-hdfs-project in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 7s{color} | {color:orange} hadoop-hdfs-project: The patch generated 14 new
+ 817 unchanged - 4 fixed = 831 total (was 821) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
32s{color} | {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
32s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
20s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 2m
10s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
28s{color} | {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
32s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
18s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
21s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs-client generated 27
new + 0 unchanged - 0 fixed = 27 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
45s{color} | {color:red}
[jira] [Comment Edited] (HDDS-1121) Key read failure when data is written parallel in to Ozone
[
https://issues.apache.org/jira/browse/HDDS-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770383#comment-16770383
]
Yiqun Lin edited comment on HDDS-1121 at 2/17/19 12:29 PM:
---
Thanks [~bharatviswa] for explanation! Looks like we need to rebase the patch
as I saw related class files was updated by other commit, :). I have a minor
suggestion: Can we document the point that checksum is not thread-safe to use
in its javadoc? It will be noticed by developers to use this.
was (Author: linyiqun):
Thanks [~bharatviswa] for explanation! Looks like we need to rebase the patch
as I saw related class files was updated by other commit, :).
> Key read failure when data is written parallel in to Ozone
> --
>
> Key: HDDS-1121
> URL: https://issues.apache.org/jira/browse/HDDS-1121
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HDDS-1121.00.patch, HDDS-1121.01.patch
>
>
> When hive is run with multiple threads for data ingestion to ozone. After
> ingestion is done, during read we see this below error.
> This issue is found during hive testing, and found by [~t3rmin4t0r]
> {code:java}
> caused by: org.apache.hadoop.ozone.common.OzoneChecksumException: Checksum
> mismatch at index 0
> at
> org.apache.hadoop.ozone.common.ChecksumData.verifyChecksumDataMatches(ChecksumData.java:143)
> at org.apache.hadoop.ozone.common.Checksum.verifyChecksum(Checksum.java:239)
> at org.apache.hadoop.ozone.common.Checksum.verifyChecksum(Checksum.java:217)
> at
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.readChunkFromContainer(BlockInputStream.java:227)
> at
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.seek(BlockInputStream.java:259)
> at
> org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.seek(KeyInputStream.java:249)
> at
> org.apache.hadoop.ozone.client.io.KeyInputStream.seek(KeyInputStream.java:180)
> at
> org.apache.hadoop.fs.ozone.OzoneFSInputStream.seek(OzoneFSInputStream.java:62)
> at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:82)
> at org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121)
> at
> org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:111)
> at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:555)
> at org.apache.orc.impl.ReaderImpl.(ReaderImpl.java:370)
> at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.(ReaderImpl.java:61)
> at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:105)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.populateAndCacheStripeDetails(OrcInputFormat.java:1647)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.callInternal(OrcInputFormat.java:1533)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.access$2700(OrcInputFormat.java:1329)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1513)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1510)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1510)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1329)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-1121) Key read failure when data is written parallel in to Ozone
[
https://issues.apache.org/jira/browse/HDDS-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770383#comment-16770383
]
Yiqun Lin edited comment on HDDS-1121 at 2/17/19 12:21 PM:
---
Thanks [~bharatviswa] for explanation! Looks like we need to rebase the patch
as I saw related class files was updated by other commit, :).
was (Author: linyiqun):
Thanks [~bharatviswa] for explanation! Looks like we need to rebase the patch
as I saw related class was updated by other commit, :).
> Key read failure when data is written parallel in to Ozone
> --
>
> Key: HDDS-1121
> URL: https://issues.apache.org/jira/browse/HDDS-1121
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HDDS-1121.00.patch, HDDS-1121.01.patch
>
>
> When hive is run with multiple threads for data ingestion to ozone. After
> ingestion is done, during read we see this below error.
> This issue is found during hive testing, and found by [~t3rmin4t0r]
> {code:java}
> caused by: org.apache.hadoop.ozone.common.OzoneChecksumException: Checksum
> mismatch at index 0
> at
> org.apache.hadoop.ozone.common.ChecksumData.verifyChecksumDataMatches(ChecksumData.java:143)
> at org.apache.hadoop.ozone.common.Checksum.verifyChecksum(Checksum.java:239)
> at org.apache.hadoop.ozone.common.Checksum.verifyChecksum(Checksum.java:217)
> at
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.readChunkFromContainer(BlockInputStream.java:227)
> at
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.seek(BlockInputStream.java:259)
> at
> org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.seek(KeyInputStream.java:249)
> at
> org.apache.hadoop.ozone.client.io.KeyInputStream.seek(KeyInputStream.java:180)
> at
> org.apache.hadoop.fs.ozone.OzoneFSInputStream.seek(OzoneFSInputStream.java:62)
> at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:82)
> at org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121)
> at
> org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:111)
> at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:555)
> at org.apache.orc.impl.ReaderImpl.(ReaderImpl.java:370)
> at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.(ReaderImpl.java:61)
> at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:105)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.populateAndCacheStripeDetails(OrcInputFormat.java:1647)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.callInternal(OrcInputFormat.java:1533)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.access$2700(OrcInputFormat.java:1329)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1513)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1510)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1510)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1329)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1121) Key read failure when data is written parallel in to Ozone
[
https://issues.apache.org/jira/browse/HDDS-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770383#comment-16770383
]
Yiqun Lin commented on HDDS-1121:
-
Thanks [~bharatviswa] for explanation! Looks like we need to rebase the patch
as I saw related class was updated by other commit, :).
> Key read failure when data is written parallel in to Ozone
> --
>
> Key: HDDS-1121
> URL: https://issues.apache.org/jira/browse/HDDS-1121
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HDDS-1121.00.patch, HDDS-1121.01.patch
>
>
> When hive is run with multiple threads for data ingestion to ozone. After
> ingestion is done, during read we see this below error.
> This issue is found during hive testing, and found by [~t3rmin4t0r]
> {code:java}
> caused by: org.apache.hadoop.ozone.common.OzoneChecksumException: Checksum
> mismatch at index 0
> at
> org.apache.hadoop.ozone.common.ChecksumData.verifyChecksumDataMatches(ChecksumData.java:143)
> at org.apache.hadoop.ozone.common.Checksum.verifyChecksum(Checksum.java:239)
> at org.apache.hadoop.ozone.common.Checksum.verifyChecksum(Checksum.java:217)
> at
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.readChunkFromContainer(BlockInputStream.java:227)
> at
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.seek(BlockInputStream.java:259)
> at
> org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.seek(KeyInputStream.java:249)
> at
> org.apache.hadoop.ozone.client.io.KeyInputStream.seek(KeyInputStream.java:180)
> at
> org.apache.hadoop.fs.ozone.OzoneFSInputStream.seek(OzoneFSInputStream.java:62)
> at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:82)
> at org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121)
> at
> org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:111)
> at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:555)
> at org.apache.orc.impl.ReaderImpl.(ReaderImpl.java:370)
> at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.(ReaderImpl.java:61)
> at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:105)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.populateAndCacheStripeDetails(OrcInputFormat.java:1647)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.callInternal(OrcInputFormat.java:1533)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.access$2700(OrcInputFormat.java:1329)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1513)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1510)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1510)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1329)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12478) [PROVIDED Phase 2] Command line tools for managing Provided Storage Backup mounts
[
https://issues.apache.org/jira/browse/HDFS-12478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HDFS-12478:
--
Status: Patch Available (was: Open)
007 - Rebased onto rebased HDFS-12090.
This patch depends on HDFS-13118.005.patch
> [PROVIDED Phase 2] Command line tools for managing Provided Storage Backup
> mounts
> -
>
> Key: HDFS-12478
> URL: https://issues.apache.org/jira/browse/HDFS-12478
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ewan Higgs
>Assignee: Ewan Higgs
>Priority: Minor
> Attachments: HDFS-12478-HDFS-12090.004.patch,
> HDFS-12478-HDFS-12090.005.patch, HDFS-12478-HDFS-12090.006.patch,
> HDFS-12478-HDFS-12090.007.patch, HDFS-12478-HDFS-9806.001.patch,
> HDFS-12478-HDFS-9806.002.patch, HDFS-12478-HDFS-9806.003.patch
>
>
> This is a task for implementing the command line interface for attaching a
> PROVIDED storage backup system (see HDFS-9806, HDFS-12090).
> # The administrator should be able to mount a PROVIDED storage volume from
> the command line.
> {code}hdfs attach -create [-name ] path (external)>{code}
> # Whitelist of users who are able to manage mounts (create, attach, detach).
> # Be able to interrogate the status of the attached storage (last time a
> snapshot was taken, files being backed up).
> # The administrator should be able to remove an attached PROVIDED storage
> volume from the command line. This simply means that the synchronization
> process no longer runs. If the administrator has configured their setup to no
> longer have local copies of the data, the blocks in the subtree are simply no
> longer accessible as the external file store system is currently inaccessible.
> {code}hdfs attach -remove [-force | -flush]{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13118) SnapshotDiffReport should provide the INode type
[ https://issues.apache.org/jira/browse/HDFS-13118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770372#comment-16770372 ] Ewan Higgs commented on HDFS-13118: --- 005 - rebased onto rebased HDFS-12090 branch. > SnapshotDiffReport should provide the INode type > > > Key: HDFS-13118 > URL: https://issues.apache.org/jira/browse/HDFS-13118 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Affects Versions: 3.0.0 >Reporter: Ewan Higgs >Assignee: Ewan Higgs >Priority: Major > Attachments: HDFS-13118.001.patch, HDFS-13118.002.patch, > HDFS-13118.003.patch, HDFS-13118.004.patch, HDFS-13118.005.patch > > > Currently the snapshot diff report will list which inodes were added, > removed, renamed, etc. But to see what the INode actually is, we need to > actually access the underlying snapshot - and this is cumbersome to do > programmatically when the snapshot diff already has the information. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12478) [PROVIDED Phase 2] Command line tools for managing Provided Storage Backup mounts
[
https://issues.apache.org/jira/browse/HDFS-12478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HDFS-12478:
--
Attachment: HDFS-12478-HDFS-12090.007.patch
> [PROVIDED Phase 2] Command line tools for managing Provided Storage Backup
> mounts
> -
>
> Key: HDFS-12478
> URL: https://issues.apache.org/jira/browse/HDFS-12478
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ewan Higgs
>Assignee: Ewan Higgs
>Priority: Minor
> Attachments: HDFS-12478-HDFS-12090.004.patch,
> HDFS-12478-HDFS-12090.005.patch, HDFS-12478-HDFS-12090.006.patch,
> HDFS-12478-HDFS-12090.007.patch, HDFS-12478-HDFS-9806.001.patch,
> HDFS-12478-HDFS-9806.002.patch, HDFS-12478-HDFS-9806.003.patch
>
>
> This is a task for implementing the command line interface for attaching a
> PROVIDED storage backup system (see HDFS-9806, HDFS-12090).
> # The administrator should be able to mount a PROVIDED storage volume from
> the command line.
> {code}hdfs attach -create [-name ] path (external)>{code}
> # Whitelist of users who are able to manage mounts (create, attach, detach).
> # Be able to interrogate the status of the attached storage (last time a
> snapshot was taken, files being backed up).
> # The administrator should be able to remove an attached PROVIDED storage
> volume from the command line. This simply means that the synchronization
> process no longer runs. If the administrator has configured their setup to no
> longer have local copies of the data, the blocks in the subtree are simply no
> longer accessible as the external file store system is currently inaccessible.
> {code}hdfs attach -remove [-force | -flush]{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12478) [PROVIDED Phase 2] Command line tools for managing Provided Storage Backup mounts
[
https://issues.apache.org/jira/browse/HDFS-12478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HDFS-12478:
--
Status: Open (was: Patch Available)
> [PROVIDED Phase 2] Command line tools for managing Provided Storage Backup
> mounts
> -
>
> Key: HDFS-12478
> URL: https://issues.apache.org/jira/browse/HDFS-12478
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ewan Higgs
>Assignee: Ewan Higgs
>Priority: Minor
> Attachments: HDFS-12478-HDFS-12090.004.patch,
> HDFS-12478-HDFS-12090.005.patch, HDFS-12478-HDFS-12090.006.patch,
> HDFS-12478-HDFS-9806.001.patch, HDFS-12478-HDFS-9806.002.patch,
> HDFS-12478-HDFS-9806.003.patch
>
>
> This is a task for implementing the command line interface for attaching a
> PROVIDED storage backup system (see HDFS-9806, HDFS-12090).
> # The administrator should be able to mount a PROVIDED storage volume from
> the command line.
> {code}hdfs attach -create [-name ] path (external)>{code}
> # Whitelist of users who are able to manage mounts (create, attach, detach).
> # Be able to interrogate the status of the attached storage (last time a
> snapshot was taken, files being backed up).
> # The administrator should be able to remove an attached PROVIDED storage
> volume from the command line. This simply means that the synchronization
> process no longer runs. If the administrator has configured their setup to no
> longer have local copies of the data, the blocks in the subtree are simply no
> longer accessible as the external file store system is currently inaccessible.
> {code}hdfs attach -remove [-force | -flush]{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13118) SnapshotDiffReport should provide the INode type
[ https://issues.apache.org/jira/browse/HDFS-13118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HDFS-13118: -- Status: Patch Available (was: Open) > SnapshotDiffReport should provide the INode type > > > Key: HDFS-13118 > URL: https://issues.apache.org/jira/browse/HDFS-13118 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Affects Versions: 3.0.0 >Reporter: Ewan Higgs >Assignee: Ewan Higgs >Priority: Major > Attachments: HDFS-13118.001.patch, HDFS-13118.002.patch, > HDFS-13118.003.patch, HDFS-13118.004.patch, HDFS-13118.005.patch > > > Currently the snapshot diff report will list which inodes were added, > removed, renamed, etc. But to see what the INode actually is, we need to > actually access the underlying snapshot - and this is cumbersome to do > programmatically when the snapshot diff already has the information. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13118) SnapshotDiffReport should provide the INode type
[ https://issues.apache.org/jira/browse/HDFS-13118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HDFS-13118: -- Attachment: HDFS-13118.005.patch > SnapshotDiffReport should provide the INode type > > > Key: HDFS-13118 > URL: https://issues.apache.org/jira/browse/HDFS-13118 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Affects Versions: 3.0.0 >Reporter: Ewan Higgs >Assignee: Ewan Higgs >Priority: Major > Attachments: HDFS-13118.001.patch, HDFS-13118.002.patch, > HDFS-13118.003.patch, HDFS-13118.004.patch, HDFS-13118.005.patch > > > Currently the snapshot diff report will list which inodes were added, > removed, renamed, etc. But to see what the INode actually is, we need to > actually access the underlying snapshot - and this is cumbersome to do > programmatically when the snapshot diff already has the information. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13118) SnapshotDiffReport should provide the INode type
[ https://issues.apache.org/jira/browse/HDFS-13118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HDFS-13118: -- Status: Open (was: Patch Available) > SnapshotDiffReport should provide the INode type > > > Key: HDFS-13118 > URL: https://issues.apache.org/jira/browse/HDFS-13118 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Affects Versions: 3.0.0 >Reporter: Ewan Higgs >Assignee: Ewan Higgs >Priority: Major > Attachments: HDFS-13118.001.patch, HDFS-13118.002.patch, > HDFS-13118.003.patch, HDFS-13118.004.patch, HDFS-13118.005.patch > > > Currently the snapshot diff report will list which inodes were added, > removed, renamed, etc. But to see what the INode actually is, we need to > actually access the underlying snapshot - and this is cumbersome to do > programmatically when the snapshot diff already has the information. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12571) Ozone: remove spaces from the beginning of the hdfs script
[
https://issues.apache.org/jira/browse/HDFS-12571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770338#comment-16770338
]
Mindy commented on HDFS-12571:
--
I am getting the same error on my mac
bogon:sbin chengruijiao$ sudo ./start-all.sh
Password:
WARNING: HADOOP_SECURE_DN_USER has been replaced by HDFS_DATANODE_SECURE_USER.
Using value of HADOOP_SECURE_DN_USER.
Starting namenodes on [localhost]
/usr/local/Cellar/hadoop/3.1.0/libexec/bin/../libexec/hadoop-functions.sh: line
398: syntax error near unexpected token `<'
/usr/local/Cellar/hadoop/3.1.0/libexec/bin/../libexec/hadoop-functions.sh: line
398: ` done < <(for text in "${input[@]}"; do'
/usr/local/Cellar/hadoop/3.1.0/libexec/bin/../libexec/hadoop-config.sh: line
70: hadoop_deprecate_envvar: command not found
/usr/local/Cellar/hadoop/3.1.0/libexec/bin/../libexec/hadoop-config.sh: line
87: hadoop_bootstrap: command not found
/usr/local/Cellar/hadoop/3.1.0/libexec/bin/../libexec/hadoop-config.sh: line
104: hadoop_parse_args: command not found
/usr/local/Cellar/hadoop/3.1.0/libexec/bin/../libexec/hadoop-config.sh: line
105: shift: : numeric argument required
/usr/local/Cellar/hadoop/3.1.0/libexec/bin/../libexec/hadoop-config.sh: line
244: hadoop_need_reexec: command not found
/usr/local/Cellar/hadoop/3.1.0/libexec/bin/../libexec/hadoop-config.sh: line
252: hadoop_verify_user_perm: command not found
/usr/local/Cellar/hadoop/3.1.0/libexec/bin/hdfs: line 213:
hadoop_validate_classname: command not found
/usr/local/Cellar/hadoop/3.1.0/libexec/bin/hdfs: line 214:
hadoop_exit_with_usage: command not found
/usr/local/Cellar/hadoop/3.1.0/libexec/bin/../libexec/hadoop-config.sh: line
263: hadoop_add_client_opts: command not found
/usr/local/Cellar/hadoop/3.1.0/libexec/bin/../libexec/hadoop-config.sh: line
270: hadoop_subcommand_opts: command not found
/usr/local/Cellar/hadoop/3.1.0/libexec/bin/../libexec/hadoop-config.sh: line
273: hadoop_generic_java_subcmd_handler: command not found
Starting datanodes
/usr/local/Cellar/hadoop/3.1.0/libexec/bin/../libexec/hadoop-functions.sh: line
398: syntax error near unexpected token `<'
/usr/local/Cellar/hadoop/3.1.0/libexec/bin/../libexec/hadoop-functions.sh: line
398: ` done < <(for text in "${input[@]}"; do'
/usr/local/Cellar/hadoop/3.1.0/libexec/bin/../libexec/hadoop-config.sh: line
70: hadoop_deprecate_envvar: command not found
/usr/local/Cellar/hadoop/3.1.0/libexec/bin/../libexec/hadoop-config.sh: line
87: hadoop_bootstrap: command not found
/usr/local/Cellar/hadoop/3.1.0/libexec/bin/../libexec/hadoop-config.sh: line
104: hadoop_parse_args: command not found
/usr/local/Cellar/hadoop/3.1.0/libexec/bin/../libexec/hadoop-config.sh: line
105: shift: : numeric argument required
/usr/local/Cellar/hadoop/3.1.0/libexec/bin/../libexec/hadoop-config.sh: line
244: hadoop_need_reexec: command not found
/usr/local/Cellar/hadoop/3.1.0/libexec/bin/../libexec/hadoop-config.sh: line
252: hadoop_verify_user_perm: command not found
/usr/local/Cellar/hadoop/3.1.0/libexec/bin/hdfs: line 213:
hadoop_validate_classname: command not found
/usr/local/Cellar/hadoop/3.1.0/libexec/bin/hdfs: line 214:
hadoop_exit_with_usage: command not found
/usr/local/Cellar/hadoop/3.1.0/libexec/bin/../libexec/hadoop-config.sh: line
263: hadoop_add_client_opts: command not found
/usr/local/Cellar/hadoop/3.1.0/libexec/bin/../libexec/hadoop-config.sh: line
270: hadoop_subcommand_opts: command not found
/usr/local/Cellar/hadoop/3.1.0/libexec/bin/../libexec/hadoop-config.sh: line
273: hadoop_generic_java_subcmd_handler: command not found
Starting secondary namenodes [account.jetbrains.com]
> Ozone: remove spaces from the beginning of the hdfs script
>
>
> Key: HDFS-12571
> URL: https://issues.apache.org/jira/browse/HDFS-12571
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Critical
> Labels: ozoneMerge
> Fix For: HDFS-7240
>
> Attachments: HDFS-12571-HDFS-7240.001.patch
>
>
> It seems that during one of the previous merge some unnecessary spaces has
> been added to the hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs file.
> After a dist build I can not start server with the hdfs command:
> {code}
> /tmp/hadoop-3.1.0-SNAPSHOT/bin/../libexec/hadoop-functions.sh: line 398:
> syntax error near unexpected token `<'
> /tmp/hadoop-3.1.0-SNAPSHOT/bin/../libexec/hadoop-functions.sh: line 398: `
> done < <(for text in "${input[@]}"; do'
> /tmp/hadoop-3.1.0-SNAPSHOT/bin/../libexec/hadoop-config.sh: line 70:
> hadoop_deprecate_envvar: command not found
> /tmp/hadoop-3.1.0-SNAPSHOT/bin/../libexec/hadoop-config.sh: line 87:
> hadoop_bootstrap: command not found
>
