[jira] [Created] (HDFS-17255) There should be mechanism between client and NN to eliminate stale nodes from current pipeline sooner.
Uma Maheswara Rao G created HDFS-17255: -- Summary: There should be mechanism between client and NN to eliminate stale nodes from current pipeline sooner. Key: HDFS-17255 URL: https://issues.apache.org/jira/browse/HDFS-17255 Project: Hadoop HDFS Issue Type: Bug Reporter: Uma Maheswara Rao G In one of users cluster, they hit an issue similar to HDFS-2891. Client is always seeing first node as failed even though 2nd node is the problematic one( timeouts due to pulling out for NW). When pipeline failure happens, client will ask for another new node and replace it in pipeline. But actual bad mode still be in pipeline as client detected wrong node ( actually a good node) as bad. So, pipeline failure continues until it detects the real wrong node in random shuffling. NN actully detected wrong node as stale. But pipeline reconstruction will only bother about client detected failed node and it will be replaced with new node. I don't have best solution in hand, but we can discuss. I think it may be a good idea if client pass all current pipeline node to recheck in first pipeline failure. So, NN can give some hints back to client which other nodes are not good and provide additional backup replacement nodes in a single call. It looks over designing to me, but I don't really have any other best ideas in my mind. Changing protocol API is painful due to compatibility problems and testing needed. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-1765) Block Replication should respect under-replication block priority
[ https://issues.apache.org/jira/browse/HDFS-1765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-1765: -- Attachment: HDFS-1765-Implementation-Proposal.pdf > Block Replication should respect under-replication block priority > - > > Key: HDFS-1765 > URL: https://issues.apache.org/jira/browse/HDFS-1765 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 0.23.0 >Reporter: Hairong Kuang >Assignee: Uma Maheswara Rao G >Priority: Major > Fix For: 2.0.0-alpha, 0.23.7 > > Attachments: HDFS-1765-Implementation-Proposal.pdf, HDFS-1765.patch, > HDFS-1765.patch, HDFS-1765.patch, HDFS-1765.patch, HDFS-1765.pdf, > underReplicatedQueue.pdf > > Time Spent: 0.5h > Remaining Estimate: 0h > > Currently under-replicated blocks are assigned different priorities depending > on how many replicas a block has. However the replication monitor works on > blocks in a round-robin fashion. So the newly added high priority blocks > won't get replicated until all low-priority blocks are done. One example is > that on decommissioning datanode WebUI we often observe that "blocks with > only decommissioning replicas" do not get scheduled to replicate before other > blocks, so risking data availability if the node is shutdown for repair > before decommission completes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-13146) Ozone: Fix TestBlockDeletingService
[
https://issues.apache.org/jira/browse/HDFS-13146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G resolved HDFS-13146.
Assignee: Uma Maheswara Rao G
Resolution: Won't Fix
Closing this Jira it is not relevant anymore.
> Ozone: Fix TestBlockDeletingService
> ---
>
> Key: HDFS-13146
> URL: https://issues.apache.org/jira/browse/HDFS-13146
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: HDFS-7240
>Reporter: Xiaoyu Yao
>Assignee: Uma Maheswara Rao G
>Priority: Major
>
> The unit tests in this class failed to shutdown individual ContainerManager
> created in teach test, and the failure stack looks like:
> {code}
> org.apache.hadoop.metrics2.MetricsException:
> org.apache.hadoop.metrics2.MetricsException:
> Hadoop:service=OzoneDataNode,name=ContainerLocationManager already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(DefaultMetricsSystem.java:135)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newMBeanName(DefaultMetricsSystem.java:110)
> at org.apache.hadoop.metrics2.util.MBeans.getMBeanName(MBeans.java:155)
> at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:87)
> at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:67)
> at
> org.apache.hadoop.ozone.container.common.impl.ContainerLocationManagerImpl.(ContainerLocationManagerImpl.java:74)
> at
> org.apache.hadoop.ozone.container.common.impl.ContainerManagerImpl.init(ContainerManagerImpl.java:188)
> at
> org.apache.hadoop.ozone.container.common.TestBlockDeletingService.createContainerManager(TestBlockDeletingService.java:117)
> at
> org.apache.hadoop.ozone.container.common.TestBlockDeletingService.testBlockDeletionTimeout(TestBlockDeletingService.java:254)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
> at
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> Caused by: org.apache.hadoop.metrics2.MetricsException:
> Hadoop:service=OzoneDataNode,name=ContainerLocationManager already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(DefaultMetricsSystem.java:131)
> ... 33 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16056) Can't start by resouceManager
[ https://issues.apache.org/jira/browse/HDFS-16056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G resolved HDFS-16056. Assignee: Uma Maheswara Rao G Resolution: Invalid Please file in YARN project if your are still seeing this issue. I think you may want to check permissions on your machines. Resolving this as invalid as it's not correct project. > Can't start by resouceManager > - > > Key: HDFS-16056 > URL: https://issues.apache.org/jira/browse/HDFS-16056 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.3.0 > Environment: windows 10 >Reporter: JYXL >Assignee: Uma Maheswara Rao G >Priority: Major > > When I use start-all.cmd, it can start namenode, datanode, nodemanager > successfully, but cannot start resoucemanager. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16289) Hadoop HA checkpointer issue
[ https://issues.apache.org/jira/browse/HDFS-16289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G resolved HDFS-16289. Assignee: Uma Maheswara Rao G Resolution: Incomplete Not sure we have enough details here. Are you still seeing this issue? Feel free to reopen if you are seeing it. > Hadoop HA checkpointer issue > - > > Key: HDFS-16289 > URL: https://issues.apache.org/jira/browse/HDFS-16289 > Project: Hadoop HDFS > Issue Type: Bug > Components: dfs >Affects Versions: 3.2.2 >Reporter: Boris Bondarenko >Assignee: Uma Maheswara Rao G >Priority: Minor > > In HA setup active namenode will reject fsimage sync from one of the two > standby namenodes all the time. This maybe an edge case, in our environment > it primarily affect standby cluster. What we experienced was memory problem > on standby namenodes in the scenario when the standby node was not able to > complete sync cycle for a long time. > It is my understanding that the break out from the loop will only happen when > doCheckpoint call succeeds otherwise it throws an exception and continues. > I can provide more details on my findings with code references if necessary. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-4190) Read complete block into memory once in BlockScanning and reduce concurrent disk access
[ https://issues.apache.org/jira/browse/HDFS-4190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G resolved HDFS-4190. --- Assignee: Uma Maheswara Rao G Resolution: Won't Fix We have HDFS caching feature in-place, if one wants to cache, they can just use that feature. Resolving this now. > Read complete block into memory once in BlockScanning and reduce concurrent > disk access > --- > > Key: HDFS-4190 > URL: https://issues.apache.org/jira/browse/HDFS-4190 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0-alpha1 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > > When we perform bulk write operations to DFS we observed that block scan is > one bottleneck for concurrent disk access. > To see real load on disks, keep single data node and local client flushing > data to DFS. > When we switch off block scanning we have seen >10% improvement. I will > update real figures in comment. > Even though I am doing only write operation, implicitly there will be a read > operation for each block due to block scanning. Next scan will happen only > after 21 days, but once scan will happen after adding the block. This will be > the concurrent access to disks. > Other point to note is that, we will read the block, packet by packet in > block scanning as well. We know that, we have to read complete block, > so, it may be correct to load complete block once and do checksums > verification for that data? > I tried with MemoryMappedBuffers: > mapped the complete block once in blockScanning and does the checksum > verification with that. Seen good improvement in that bulk write scenario. > But we don't have any API to clean the mapped buffer immediately. With my > experiment I just used, Cleaner class from sun package. That will not be > correct to use in production. So, we have to write JNI call to clean that > mmapped buffer. > I am not sure I missed something here. please correct me If i missed some > points. > Thoughts? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-3399) BookKeeper option support for NN HA
[ https://issues.apache.org/jira/browse/HDFS-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G resolved HDFS-3399. --- Resolution: Fixed This option we provided as experimental, however we have journal node based shared storage and pretty stable now. Just resolving this JIRA. > BookKeeper option support for NN HA > --- > > Key: HDFS-3399 > URL: https://issues.apache.org/jira/browse/HDFS-3399 > Project: Hadoop HDFS > Issue Type: New Feature > Components: ha >Affects Versions: 2.0.0-alpha, 3.0.0-alpha1 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Attachments: BKTestDoc.pdf > > > Here is the JIRA to BookKeeper support issues with NN HA. We can file all the > BookKeeperJournalManager issues under this JIRA for more easy tracking. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-3399) BookKeeper option support for NN HA
[ https://issues.apache.org/jira/browse/HDFS-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G reassigned HDFS-3399: - Assignee: Uma Maheswara Rao G > BookKeeper option support for NN HA > --- > > Key: HDFS-3399 > URL: https://issues.apache.org/jira/browse/HDFS-3399 > Project: Hadoop HDFS > Issue Type: New Feature > Components: ha >Affects Versions: 2.0.0-alpha, 3.0.0-alpha1 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Attachments: BKTestDoc.pdf > > > Here is the JIRA to BookKeeper support issues with NN HA. We can file all the > BookKeeperJournalManager issues under this JIRA for more easy tracking. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-3085) Local data node may need to reconsider for read, when reading a very big file as that local DN may get recover in some time.
[ https://issues.apache.org/jira/browse/HDFS-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G resolved HDFS-3085. --- Target Version/s: (was: ) Resolution: Won't Fix Closing this as client failed DN refreshing functionality already added in latest code IIRC. > Local data node may need to reconsider for read, when reading a very big file > as that local DN may get recover in some time. > > > Key: HDFS-3085 > URL: https://issues.apache.org/jira/browse/HDFS-3085 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, hdfs-client >Affects Versions: 2.0.0-alpha >Reporter: Uma Maheswara Rao G >Priority: Major > > While reading the file, we will add the DN to deadNodes list and will skip > from reads. > If we are reading very huge file (may take hours), and failed read from local > datanode, then this will be added to deadnode list and will be excluded for > the further reads for that file. > If the local node recovered immediately,but that will not used for further > read. Read may continue with the remote nodes. It will effect the read > performance. > It will be good if we reconsider the local node after certain period based on > some factors. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-2891) Some times first DataNode detected as bad when we power off for the second DataNode.
[ https://issues.apache.org/jira/browse/HDFS-2891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G resolved HDFS-2891. --- Resolution: Won't Fix We don't see any issues so far and it is quite old issue. Just resolving it. Feel free to open, if you are seeing it. > Some times first DataNode detected as bad when we power off for the second > DataNode. > > > Key: HDFS-2891 > URL: https://issues.apache.org/jira/browse/HDFS-2891 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs-client >Affects Versions: 1.1.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > > In one of my clusters, observed this situation. > This issue looks to be due to time out in ResponseProcesser at client side, > it is marking first DataNode as bad. > This happens in 20.2 version. This can be there in branch-1 as well and will > check for trunk. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-2891) Some times first DataNode detected as bad when we power off for the second DataNode.
[ https://issues.apache.org/jira/browse/HDFS-2891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G reassigned HDFS-2891: - Assignee: Uma Maheswara Rao G > Some times first DataNode detected as bad when we power off for the second > DataNode. > > > Key: HDFS-2891 > URL: https://issues.apache.org/jira/browse/HDFS-2891 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs-client >Affects Versions: 1.1.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > > In one of my clusters, observed this situation. > This issue looks to be due to time out in ResponseProcesser at client side, > it is marking first DataNode as bad. > This happens in 20.2 version. This can be there in branch-1 as well and will > check for trunk. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-1944) Reading of the previously synced data will fail if the last block got corrupted.
[ https://issues.apache.org/jira/browse/HDFS-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G resolved HDFS-1944. --- Resolution: Works for Me Closing this as it is very old issue and I don't think it's an issue anymore. > Reading of the previously synced data will fail if the last block got > corrupted. > - > > Key: HDFS-1944 > URL: https://issues.apache.org/jira/browse/HDFS-1944 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs-client, namenode >Affects Versions: 0.20-append >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > > For example a file can comprise 5 blcoks . In that we have written 4.5 blocks > and invoked sync. >Reading of this 4.5 blocks will be successful at this point of time. > Now when writing the remaining 0.5 block write has failed due to checksum > errors, Then the reading of the previously synced 4.5 blocks also fails. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-1944) Reading of the previously synced data will fail if the last block got corrupted.
[ https://issues.apache.org/jira/browse/HDFS-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G reassigned HDFS-1944: - Assignee: Uma Maheswara Rao G > Reading of the previously synced data will fail if the last block got > corrupted. > - > > Key: HDFS-1944 > URL: https://issues.apache.org/jira/browse/HDFS-1944 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs-client, namenode >Affects Versions: 0.20-append >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > > For example a file can comprise 5 blcoks . In that we have written 4.5 blocks > and invoked sync. >Reading of this 4.5 blocks will be successful at this point of time. > Now when writing the remaining 0.5 block write has failed due to checksum > errors, Then the reading of the previously synced 4.5 blocks also fails. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16924) Add libhdfs APIs for createFile
[ https://issues.apache.org/jira/browse/HDFS-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G reassigned HDFS-16924: -- Assignee: Uma Maheswara Rao G > Add libhdfs APIs for createFile > --- > > Key: HDFS-16924 > URL: https://issues.apache.org/jira/browse/HDFS-16924 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs >Reporter: Zoltán Borók-Nagy >Assignee: Uma Maheswara Rao G >Priority: Major > > HDFS-14478 introduces builder-based APIs for openFile() based on HADOOP-15229. > We should also add builder-based APIs for createFile() based on HADOOP-14365. > This would be especially useful for object stores to tune performance of file > writes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16924) Add libhdfs APIs for createFile
[ https://issues.apache.org/jira/browse/HDFS-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17754269#comment-17754269 ] Uma Maheswara Rao G commented on HDFS-16924: [~boroknagyz] looking at your above comment your reqquirement is satisfied with hdfsOpenFile API? In that case, we can close this ? > Add libhdfs APIs for createFile > --- > > Key: HDFS-16924 > URL: https://issues.apache.org/jira/browse/HDFS-16924 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs >Reporter: Zoltán Borók-Nagy >Priority: Major > > HDFS-14478 introduces builder-based APIs for openFile() based on HADOOP-15229. > We should also add builder-based APIs for createFile() based on HADOOP-14365. > This would be especially useful for object stores to tune performance of file > writes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16996) TestFileCreation failed with ClassCastException
Uma Maheswara Rao G created HDFS-16996:
--
Summary: TestFileCreation failed with ClassCastException
Key: HDFS-16996
URL: https://issues.apache.org/jira/browse/HDFS-16996
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Uma Maheswara Rao G
{code:java}
[ERROR]
testFsCloseAfterClusterShutdown(org.apache.hadoop.hdfs.TestFileCreation) Time
elapsed: 1.725 s <<< FAILURE! java.lang.AssertionError: Test resulted in an
unexpected exit: 1: Block report processor encountered fatal exception:
java.lang.ClassCastException: org.apache.hadoop.fs.FsServerDefaults cannot be
cast to java.lang.Boolean at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:2166) at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:2152) at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:2145) at
org.apache.hadoop.hdfs.TestFileCreation.testFsCloseAfterClusterShutdown(TestFileCreation.java:1198)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at
org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at
org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at
org.junit.runners.ParentRunner.run(ParentRunner.java:413) at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: 1: Block report processor encountered fatal exception:
java.lang.ClassCastException: org.apache.hadoop.fs.FsServerDefaults cannot be
cast to java.lang.Boolean at
org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:381) at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.run(BlockManager.java:5451){code}
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5532/10/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16911) Distcp with snapshot diff to support Ozone filesystem.
[
https://issues.apache.org/jira/browse/HDFS-16911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G reassigned HDFS-16911:
--
Assignee: Sadanand Shenoy
> Distcp with snapshot diff to support Ozone filesystem.
> --
>
> Key: HDFS-16911
> URL: https://issues.apache.org/jira/browse/HDFS-16911
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp
>Reporter: Sadanand Shenoy
>Assignee: Sadanand Shenoy
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Currently in DistcpSync i.e the step which applies the diff b/w 2 provided
> snapshots as arguments to the distcp job with -diff option, only
> DistributedFilesystem and WebHDFS filesytems are supported.
>
> {code:java}
> // currently we require both the source and the target file system are
> // DistributedFileSystem or (S)WebHdfsFileSystem.
> if (!(srcFs instanceof DistributedFileSystem
> || srcFs instanceof WebHdfsFileSystem)) {
> throw new IllegalArgumentException("Unsupported source file system: "
> + srcFs.getScheme() + "://. " +
> "Supported file systems: hdfs://, webhdfs:// and swebhdfs://.");
> }
> if (!(tgtFs instanceof DistributedFileSystem
> || tgtFs instanceof WebHdfsFileSystem)) {
> throw new IllegalArgumentException("Unsupported target file system: "
> + tgtFs.getScheme() + "://. " +
> "Supported file systems: hdfs://, webhdfs:// and swebhdfs://.");
> }{code}
> As Ozone now supports snapshot feature after HDDS-6517, add support to use
> distcp with ozone snapshots.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16911) Distcp with snapshot diff to support Ozone filesystem.
[
https://issues.apache.org/jira/browse/HDFS-16911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G resolved HDFS-16911.
Fix Version/s: 3.4.0
Resolution: Fixed
Thanks [~sadanand_shenoy] for the contribution. I have just merged this to
trunk.
> Distcp with snapshot diff to support Ozone filesystem.
> --
>
> Key: HDFS-16911
> URL: https://issues.apache.org/jira/browse/HDFS-16911
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp
>Reporter: Sadanand Shenoy
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Currently in DistcpSync i.e the step which applies the diff b/w 2 provided
> snapshots as arguments to the distcp job with -diff option, only
> DistributedFilesystem and WebHDFS filesytems are supported.
>
> {code:java}
> // currently we require both the source and the target file system are
> // DistributedFileSystem or (S)WebHdfsFileSystem.
> if (!(srcFs instanceof DistributedFileSystem
> || srcFs instanceof WebHdfsFileSystem)) {
> throw new IllegalArgumentException("Unsupported source file system: "
> + srcFs.getScheme() + "://. " +
> "Supported file systems: hdfs://, webhdfs:// and swebhdfs://.");
> }
> if (!(tgtFs instanceof DistributedFileSystem
> || tgtFs instanceof WebHdfsFileSystem)) {
> throw new IllegalArgumentException("Unsupported target file system: "
> + tgtFs.getScheme() + "://. " +
> "Supported file systems: hdfs://, webhdfs:// and swebhdfs://.");
> }{code}
> As Ozone now supports snapshot feature after HDDS-6517, add support to use
> distcp with ozone snapshots.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13369) FSCK Report broken with RequestHedgingProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-13369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17584557#comment-17584557
]
Uma Maheswara Rao G commented on HDFS-13369:
Any one looking to rebase this?
> FSCK Report broken with RequestHedgingProxyProvider
>
>
> Key: HDFS-13369
> URL: https://issues.apache.org/jira/browse/HDFS-13369
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.8.3, 3.0.0, 3.1.0
>Reporter: Harshakiran Reddy
>Assignee: Ranith Sardar
>Priority: Major
> Attachments: HDFS-13369.001.patch, HDFS-13369.002.patch,
> HDFS-13369.003.patch, HDFS-13369.004.patch, HDFS-13369.005.patch,
> HDFS-13369.006.patch, HDFS-13369.007.patch
>
>
> Scenario:-
> 1.Configure the RequestHedgingProxy
> 2. write some files in file system
> 3. Take FSCK report for the above files
>
> {noformat}
> bin> hdfs fsck /file1 -locations -files -blocks
> Exception in thread "main" java.lang.ClassCastException:
> org.apache.hadoop.hdfs.server.namenode.ha.RequestHedgingProxyProvider$RequestHedgingInvocationHandler
> cannot be cast to org.apache.hadoop.ipc.RpcInvocationHandler
> at org.apache.hadoop.ipc.RPC.getConnectionIdForProxy(RPC.java:626)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.getConnectionId(RetryInvocationHandler.java:438)
> at org.apache.hadoop.ipc.RPC.getConnectionIdForProxy(RPC.java:628)
> at org.apache.hadoop.ipc.RPC.getServerAddress(RPC.java:611)
> at org.apache.hadoop.hdfs.HAUtil.getAddressOfActive(HAUtil.java:263)
> at
> org.apache.hadoop.hdfs.tools.DFSck.getCurrentNamenodeAddress(DFSck.java:257)
> at org.apache.hadoop.hdfs.tools.DFSck.doWork(DFSck.java:319)
> at org.apache.hadoop.hdfs.tools.DFSck.access$000(DFSck.java:72)
> at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:156)
> at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:153)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
> at org.apache.hadoop.hdfs.tools.DFSck.run(DFSck.java:152)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.hdfs.tools.DFSck.main(DFSck.java:385){noformat}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16192) ViewDistributedFileSystem#rename wrongly using src in the place of dst.
Uma Maheswara Rao G created HDFS-16192: -- Summary: ViewDistributedFileSystem#rename wrongly using src in the place of dst. Key: HDFS-16192 URL: https://issues.apache.org/jira/browse/HDFS-16192 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G In ViewDistributedFileSystem, we are mistakenly used src path in the place of dst path when finding mount path info. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15760) Validate the target indices in ErasureCoding worker in reconstruction process
[ https://issues.apache.org/jira/browse/HDFS-15760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G reassigned HDFS-15760: -- Assignee: Uma Maheswara Rao G > Validate the target indices in ErasureCoding worker in reconstruction process > - > > Key: HDFS-15760 > URL: https://issues.apache.org/jira/browse/HDFS-15760 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ec >Affects Versions: 3.4.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > > As we have seen issues like > # HDFS-15186 > # HDFS-14768 > It is a good idea to validate the indices at the ECWorker side and skip the > unintended in indices from target list. > Both of the issues triggered because, NN accidentally scheduled for > reconstruction in decom process due to busy node. We have fixed to make sure > NN considers busy nodes as live replicas. However, it may be good idea to > safe gaud the condition at ECWorker also in case if any other condition > triggers and that leads ECWroker to calculate the indices similar the above > issues, then EC function returns wrong o/p. I think it's ok to recover only > the missing indices from the given src indices. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15453) Implement ViewFsAdmin to list the mount points, target fs for path etc.
[ https://issues.apache.org/jira/browse/HDFS-15453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G reassigned HDFS-15453: -- Assignee: Uma Maheswara Rao G > Implement ViewFsAdmin to list the mount points, target fs for path etc. > --- > > Key: HDFS-15453 > URL: https://issues.apache.org/jira/browse/HDFS-15453 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: viewfs, viewfsOverloadScheme >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Minor > > I think, it may be a good idea to have some admin commands to list mount > points and getting target fs type for the given path etc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15760) Validate the target indices in ErasureCoding worker in reconstruction process
Uma Maheswara Rao G created HDFS-15760: -- Summary: Validate the target indices in ErasureCoding worker in reconstruction process Key: HDFS-15760 URL: https://issues.apache.org/jira/browse/HDFS-15760 Project: Hadoop HDFS Issue Type: Improvement Components: ec Affects Versions: 3.4.0 Reporter: Uma Maheswara Rao G As we have seen issues like # HDFS-15186 # HDFS-14768 It is a good idea to validate the indices at the ECWorker side and skip the unintended in indices from target list. Both of the issues triggered because, NN accidentally scheduled for reconstruction in decom process due to busy node. We have fixed to make sure NN considers busy nodes as live replicas. However, it may be good idea to safe gaud the condition at ECWorker also in case if any other condition triggers and that leads ECWroker to calculate the indices similar the above issues, then EC function returns wrong o/p. I think it's ok to recover only the missing indices from the given src indices. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15240) Erasure Coding: dirty buffer causes reconstruction block error
[
https://issues.apache.org/jira/browse/HDFS-15240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17242747#comment-17242747
]
Uma Maheswara Rao G commented on HDFS-15240:
+1 latest patch looks good to me.
> Erasure Coding: dirty buffer causes reconstruction block error
> --
>
> Key: HDFS-15240
> URL: https://issues.apache.org/jira/browse/HDFS-15240
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, erasure-coding
>Reporter: HuangTao
>Assignee: HuangTao
>Priority: Major
> Attachments: HDFS-15240.001.patch, HDFS-15240.002.patch,
> HDFS-15240.003.patch, HDFS-15240.004.patch, HDFS-15240.005.patch,
> HDFS-15240.006.patch, HDFS-15240.007.patch, HDFS-15240.008.patch,
> HDFS-15240.009.patch, HDFS-15240.010.patch, HDFS-15240.011.patch,
> HDFS-15240.012.patch, HDFS-15240.013.patch,
> image-2020-07-16-15-56-38-608.png,
> org.apache.hadoop.hdfs.TestReconstructStripedFile-output.txt,
> org.apache.hadoop.hdfs.TestReconstructStripedFile.txt,
> test-HDFS-15240.006.patch
>
>
> When read some lzo files we found some blocks were broken.
> I read back all internal blocks(b0-b8) of the block group(RS-6-3-1024k) from
> DN directly, and choose 6(b0-b5) blocks to decode other 3(b6', b7', b8')
> blocks. And find the longest common sequenece(LCS) between b6'(decoded) and
> b6(read from DN)(b7'/b7 and b8'/b8).
> After selecting 6 blocks of the block group in combinations one time and
> iterating through all cases, I find one case that the length of LCS is the
> block length - 64KB, 64KB is just the length of ByteBuffer used by
> StripedBlockReader. So the corrupt reconstruction block is made by a dirty
> buffer.
> The following log snippet(only show 2 of 28 cases) is my check program
> output. In my case, I known the 3th block is corrupt, so need other 5 blocks
> to decode another 3 blocks, then find the 1th block's LCS substring is block
> length - 64kb.
> It means (0,1,2,4,5,6)th blocks were used to reconstruct 3th block, and the
> dirty buffer was used before read the 1th block.
> Must be noted that StripedBlockReader read from the offset 0 of the 1th block
> after used the dirty buffer.
> EDITED for readability.
> {code:java}
> decode from block[0, 2, 3, 4, 5, 7] to generate block[1', 6', 8']
> Check the first 131072 bytes between block[1] and block[1'], the longest
> common substring length is 4
> Check the first 131072 bytes between block[6] and block[6'], the longest
> common substring length is 4
> Check the first 131072 bytes between block[8] and block[8'], the longest
> common substring length is 4
> decode from block[0, 2, 3, 4, 5, 6] to generate block[1', 7', 8']
> Check the first 131072 bytes between block[1] and block[1'], the longest
> common substring length is 65536
> CHECK AGAIN: all 27262976 bytes between block[1] and block[1'], the longest
> common substring length is 27197440 # this one
> Check the first 131072 bytes between block[7] and block[7'], the longest
> common substring length is 4
> Check the first 131072 bytes between block[8] and block[8'], the longest
> common substring length is 4{code}
> Now I know the dirty buffer causes reconstruction block error, but how does
> the dirty buffer come about?
> After digging into the code and DN log, I found this following DN log is the
> root reason.
> {code:java}
> [INFO] [stripedRead-1017] : Interrupted while waiting for IO on channel
> java.nio.channels.SocketChannel[connected local=/:52586
> remote=/:50010]. 18 millis timeout left.
> [WARN] [StripedBlockReconstruction-199] : Failed to reconstruct striped
> block: BP-714356632--1519726836856:blk_-YY_3472979393
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil.getNextCompletedStripedRead(StripedBlockUtil.java:314)
> at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.doReadMinimumSources(StripedReader.java:308)
> at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.readMinimumSources(StripedReader.java:269)
> at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReconstructor.reconstruct(StripedBlockReconstructor.java:94)
> at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReconstructor.run(StripedBlockReconstructor.java:60)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834) {code}
>
[jira] [Created] (HDFS-15701) Add resolveMountPath API in FileSystem
Uma Maheswara Rao G created HDFS-15701: -- Summary: Add resolveMountPath API in FileSystem Key: HDFS-15701 URL: https://issues.apache.org/jira/browse/HDFS-15701 Project: Hadoop HDFS Issue Type: Sub-task Components: fs, ViewHDFS Affects Versions: 3.4.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Currently FileSystem has an API resolvePath. To know where the path has mounted, the applications can use that API as the retuned path is from actual target path in the case of mount file systems like ViewFS, ViewFSOverloadScheme or ViewDistributedFileSystem. However, resolvePath does more than what is needed by Apps when they want to know where the path has mounted. It's because resolvePath internally calls "getFileStatus". This additional call is unnecessary when apps just want to where the path mounted. Since we have mounted filesystems available in FS, I think it's good to add resolveMountPath API, which will just do the following. If the fs is mounted fs, then it will resolve it's mount tables and return the actual target path. If the fs is non mounted, then it will simply return the same path. Currently Applications like Hive, Ranger using resolvePath API. ( this is forcing to do additional RPC internally) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15240) Erasure Coding: dirty buffer causes reconstruction block error
[
https://issues.apache.org/jira/browse/HDFS-15240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238943#comment-17238943
]
Uma Maheswara Rao G commented on HDFS-15240:
I think the latest patch mostly looks good to me.
I think [~ferhui] asked why do we need to interchange below line:
{code}
reader.closeBlockReader();
441 reconstructor.freeBuffer(reader.getReadBuffer());
442 reconstructor.freeBuffer(reader.getReadBuffer());
442 reader.freeReadBuffer();
443 reader.freeReadBuffer();
{code}
Could you please clarify?
> Erasure Coding: dirty buffer causes reconstruction block error
> --
>
> Key: HDFS-15240
> URL: https://issues.apache.org/jira/browse/HDFS-15240
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, erasure-coding
>Reporter: HuangTao
>Assignee: HuangTao
>Priority: Major
> Attachments: HDFS-15240.001.patch, HDFS-15240.002.patch,
> HDFS-15240.003.patch, HDFS-15240.004.patch, HDFS-15240.005.patch,
> HDFS-15240.006.patch, HDFS-15240.007.patch, HDFS-15240.008.patch,
> image-2020-07-16-15-56-38-608.png,
> org.apache.hadoop.hdfs.TestReconstructStripedFile-output.txt,
> org.apache.hadoop.hdfs.TestReconstructStripedFile.txt,
> test-HDFS-15240.006.patch
>
>
> When read some lzo files we found some blocks were broken.
> I read back all internal blocks(b0-b8) of the block group(RS-6-3-1024k) from
> DN directly, and choose 6(b0-b5) blocks to decode other 3(b6', b7', b8')
> blocks. And find the longest common sequenece(LCS) between b6'(decoded) and
> b6(read from DN)(b7'/b7 and b8'/b8).
> After selecting 6 blocks of the block group in combinations one time and
> iterating through all cases, I find one case that the length of LCS is the
> block length - 64KB, 64KB is just the length of ByteBuffer used by
> StripedBlockReader. So the corrupt reconstruction block is made by a dirty
> buffer.
> The following log snippet(only show 2 of 28 cases) is my check program
> output. In my case, I known the 3th block is corrupt, so need other 5 blocks
> to decode another 3 blocks, then find the 1th block's LCS substring is block
> length - 64kb.
> It means (0,1,2,4,5,6)th blocks were used to reconstruct 3th block, and the
> dirty buffer was used before read the 1th block.
> Must be noted that StripedBlockReader read from the offset 0 of the 1th block
> after used the dirty buffer.
> EDITED for readability.
> {code:java}
> decode from block[0, 2, 3, 4, 5, 7] to generate block[1', 6', 8']
> Check the first 131072 bytes between block[1] and block[1'], the longest
> common substring length is 4
> Check the first 131072 bytes between block[6] and block[6'], the longest
> common substring length is 4
> Check the first 131072 bytes between block[8] and block[8'], the longest
> common substring length is 4
> decode from block[0, 2, 3, 4, 5, 6] to generate block[1', 7', 8']
> Check the first 131072 bytes between block[1] and block[1'], the longest
> common substring length is 65536
> CHECK AGAIN: all 27262976 bytes between block[1] and block[1'], the longest
> common substring length is 27197440 # this one
> Check the first 131072 bytes between block[7] and block[7'], the longest
> common substring length is 4
> Check the first 131072 bytes between block[8] and block[8'], the longest
> common substring length is 4{code}
> Now I know the dirty buffer causes reconstruction block error, but how does
> the dirty buffer come about?
> After digging into the code and DN log, I found this following DN log is the
> root reason.
> {code:java}
> [INFO] [stripedRead-1017] : Interrupted while waiting for IO on channel
> java.nio.channels.SocketChannel[connected local=/:52586
> remote=/:50010]. 18 millis timeout left.
> [WARN] [StripedBlockReconstruction-199] : Failed to reconstruct striped
> block: BP-714356632--1519726836856:blk_-YY_3472979393
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil.getNextCompletedStripedRead(StripedBlockUtil.java:314)
> at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.doReadMinimumSources(StripedReader.java:308)
> at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.readMinimumSources(StripedReader.java:269)
> at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReconstructor.reconstruct(StripedBlockReconstructor.java:94)
> at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReconstructor.run(StripedBlockReconstructor.java:60)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
>
[jira] [Resolved] (HDFS-15635) ViewFileSystemOverloadScheme support specifying mount table loader imp through conf
[
https://issues.apache.org/jira/browse/HDFS-15635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G resolved HDFS-15635.
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
Thank you [~zuston] for the contribution. I have committed it to trunk!
> ViewFileSystemOverloadScheme support specifying mount table loader imp
> through conf
> ---
>
> Key: HDFS-15635
> URL: https://issues.apache.org/jira/browse/HDFS-15635
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: viewfsOverloadScheme
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> According to HDFS-15289, the default mountable loader is
> {{[HCFSMountTableConfigLoader|https://github.com/apache/hadoop/blob/4734c77b4b64b7c6432da4cc32881aba85f94ea1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/HCFSMountTableConfigLoader.java#L35]}}.
> In some scenarios, users want to implement the mount table loader by
> themselves, so it is necessary to dynamically configure the loader.
>
> cc [~shv], [~abhishekd], [~hexiaoqiao]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15635) ViewFileSystemOverloadScheme support specifying mount table loader imp through conf
[
https://issues.apache.org/jira/browse/HDFS-15635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G reassigned HDFS-15635:
--
Assignee: Junfan Zhang
> ViewFileSystemOverloadScheme support specifying mount table loader imp
> through conf
> ---
>
> Key: HDFS-15635
> URL: https://issues.apache.org/jira/browse/HDFS-15635
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: viewfsOverloadScheme
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> According to HDFS-15289, the default mountable loader is
> {{[HCFSMountTableConfigLoader|https://github.com/apache/hadoop/blob/4734c77b4b64b7c6432da4cc32881aba85f94ea1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/HCFSMountTableConfigLoader.java#L35]}}.
> In some scenarios, users want to implement the mount table loader by
> themselves, so it is necessary to dynamically configure the loader.
>
> cc [~shv], [~abhishekd], [~hexiaoqiao]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15686) Provide documentation for ViewHDFS
Uma Maheswara Rao G created HDFS-15686: -- Summary: Provide documentation for ViewHDFS Key: HDFS-15686 URL: https://issues.apache.org/jira/browse/HDFS-15686 Project: Hadoop HDFS Issue Type: Sub-task Components: viewfs, viewfsOverloadScheme, ViewHDFS Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15240) Erasure Coding: dirty buffer causes reconstruction block error
[
https://issues.apache.org/jira/browse/HDFS-15240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230489#comment-17230489
]
Uma Maheswara Rao G commented on HDFS-15240:
HI [~marvelrock] Thanks a lot for working on this issue and thanks [~ferhui]
for reviewing this issue.
This seems to be very tricky to find and debug. NPE is causing because of stale
future coming from readService#poll. This future we already cleared as part-of
previous iterations due to timeouts. Either we have to make sure all stale
futures removed from service when clearing the futures. Otherwise in the
current code we are assuming service#poll will give one of the submitted
current future only.
Anyway your fix looks good to me and it can handle the case of stale future and
avoid NPE.
Other than the above comments from [~ferhui], I have the follwoing.
I think we can avoid making StripedReconstructor public. The suggestion is:
{code:java}
@InterfaceAudience.Private
100 public abstract class StripedReconstructor {
{code}
You can create ErasureCodingTestHelper in tests with the package
*.hadoop.hdfs.server.datanode.erasurecode and add a small static util method
to access StripedReconstructor methods. This way you can avoid this src class
public.
> Erasure Coding: dirty buffer causes reconstruction block error
> --
>
> Key: HDFS-15240
> URL: https://issues.apache.org/jira/browse/HDFS-15240
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, erasure-coding
>Reporter: HuangTao
>Assignee: HuangTao
>Priority: Major
> Attachments: HDFS-15240.001.patch, HDFS-15240.002.patch,
> HDFS-15240.003.patch, HDFS-15240.004.patch, HDFS-15240.005.patch,
> HDFS-15240.006.patch, image-2020-07-16-15-56-38-608.png,
> org.apache.hadoop.hdfs.TestReconstructStripedFile-output.txt,
> org.apache.hadoop.hdfs.TestReconstructStripedFile.txt,
> test-HDFS-15240.006.patch
>
>
> When read some lzo files we found some blocks were broken.
> I read back all internal blocks(b0-b8) of the block group(RS-6-3-1024k) from
> DN directly, and choose 6(b0-b5) blocks to decode other 3(b6', b7', b8')
> blocks. And find the longest common sequenece(LCS) between b6'(decoded) and
> b6(read from DN)(b7'/b7 and b8'/b8).
> After selecting 6 blocks of the block group in combinations one time and
> iterating through all cases, I find one case that the length of LCS is the
> block length - 64KB, 64KB is just the length of ByteBuffer used by
> StripedBlockReader. So the corrupt reconstruction block is made by a dirty
> buffer.
> The following log snippet(only show 2 of 28 cases) is my check program
> output. In my case, I known the 3th block is corrupt, so need other 5 blocks
> to decode another 3 blocks, then find the 1th block's LCS substring is block
> length - 64kb.
> It means (0,1,2,4,5,6)th blocks were used to reconstruct 3th block, and the
> dirty buffer was used before read the 1th block.
> Must be noted that StripedBlockReader read from the offset 0 of the 1th block
> after used the dirty buffer.
> EDITED for readability.
> {code:java}
> decode from block[0, 2, 3, 4, 5, 7] to generate block[1', 6', 8']
> Check the first 131072 bytes between block[1] and block[1'], the longest
> common substring length is 4
> Check the first 131072 bytes between block[6] and block[6'], the longest
> common substring length is 4
> Check the first 131072 bytes between block[8] and block[8'], the longest
> common substring length is 4
> decode from block[0, 2, 3, 4, 5, 6] to generate block[1', 7', 8']
> Check the first 131072 bytes between block[1] and block[1'], the longest
> common substring length is 65536
> CHECK AGAIN: all 27262976 bytes between block[1] and block[1'], the longest
> common substring length is 27197440 # this one
> Check the first 131072 bytes between block[7] and block[7'], the longest
> common substring length is 4
> Check the first 131072 bytes between block[8] and block[8'], the longest
> common substring length is 4{code}
> Now I know the dirty buffer causes reconstruction block error, but how does
> the dirty buffer come about?
> After digging into the code and DN log, I found this following DN log is the
> root reason.
> {code:java}
> [INFO] [stripedRead-1017] : Interrupted while waiting for IO on channel
> java.nio.channels.SocketChannel[connected local=/:52586
> remote=/:50010]. 18 millis timeout left.
> [WARN] [StripedBlockReconstruction-199] : Failed to reconstruct striped
> block: BP-714356632--1519726836856:blk_-YY_3472979393
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil.getNextCompletedStripedRead(StripedBlockUtil.java:314)
> at
>
[jira] [Commented] (HDFS-15289) Allow viewfs mounts with HDFS/HCFS scheme and centralized mount table
[ https://issues.apache.org/jira/browse/HDFS-15289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17221114#comment-17221114 ] Uma Maheswara Rao G commented on HDFS-15289: [~zuston] Yes, it will support. For specific to HDFS existing clusters, we introduced ViewDistributedFileSystem(HDFS-15533), where users can configure fs.hdfs.impl with ViewDistributedFileSystem, so that users can add mount points with respective to HDFS paths. And fallBack cluster would act as default cluster. > Allow viewfs mounts with HDFS/HCFS scheme and centralized mount table > - > > Key: HDFS-15289 > URL: https://issues.apache.org/jira/browse/HDFS-15289 > Project: Hadoop HDFS > Issue Type: New Feature > Components: fs >Affects Versions: 3.2.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Attachments: ViewFSOverloadScheme - V1.0.pdf, ViewFSOverloadScheme.png > > > ViewFS provides flexibility to mount different filesystem types with mount > points configuration table. This approach is solving the scalability > problems, but users need to reconfigure the filesystem to ViewFS and to its > scheme. This will be problematic in the case of paths persisted in meta > stores, ex: Hive. In systems like Hive, it will store uris in meta store. So, > changing the file system scheme will create a burden to upgrade/recreate meta > stores. In our experience many users are not ready to change that. > Router based federation is another implementation to provide coordinated > mount points for HDFS federation clusters. Even though this provides > flexibility to handle mount points easily, this will not allow > other(non-HDFS) file systems to mount. So, this does not solve the purpose > when users want to mount external(non-HDFS) filesystems. > So, the problem here is: Even though many users want to adapt to the scalable > fs options available, technical challenges of changing schemes (ex: in meta > stores) in deployments are obstructing them. > So, we propose to allow hdfs scheme in ViewFS like client side mount system > and provision user to create mount links without changing URI paths. > I will upload detailed design doc shortly. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15635) ViewFileSystemOverloadScheme support specifying mount table loader imp through conf
[
https://issues.apache.org/jira/browse/HDFS-15635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-15635:
---
Parent: HDFS-15289
Issue Type: Sub-task (was: Improvement)
> ViewFileSystemOverloadScheme support specifying mount table loader imp
> through conf
> ---
>
> Key: HDFS-15635
> URL: https://issues.apache.org/jira/browse/HDFS-15635
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: viewfsOverloadScheme
>Reporter: Junfan Zhang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> According to HDFS-15289, the default mountable loader is
> {{[HCFSMountTableConfigLoader|https://github.com/apache/hadoop/blob/4734c77b4b64b7c6432da4cc32881aba85f94ea1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/HCFSMountTableConfigLoader.java#L35]}}.
> In some scenarios, users want to implement the mount table loader by
> themselves, so it is necessary to dynamically configure the loader.
>
> cc [~shv], [~abhishekd], [~hexiaoqiao]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15637) Viewfs should make mount-table to read from central place
[ https://issues.apache.org/jira/browse/HDFS-15637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219172#comment-17219172 ] Uma Maheswara Rao G commented on HDFS-15637: [~zuston], Thank you for thinking about this and filing JIRA. We have already worked on this as part of HDFS-15289 This JIRA aimed mainly two key tasks, 1. make viewfs extended to support any scheme and 2. reading mountable from central place. You may want to look at this specific JIRA HDFS-15306 under HDFS-15289. If you want to make any improvements, please feel free to file subtasks under HDFS-15289. Recently I talked about these improvements at [SDC |https://www.youtube.com/watch?v=pTyklxxEDyY=youtu.be] for more details, please watch this. > Viewfs should make mount-table to read from central place > - > > Key: HDFS-15637 > URL: https://issues.apache.org/jira/browse/HDFS-15637 > Project: Hadoop HDFS > Issue Type: Improvement > Components: viewfs >Reporter: Junfan Zhang >Priority: Major > > Like [HDFS-15637|https://issues.apache.org/jira/browse/HDFS-15637]. > Viewfs should make mount-table to read from central place to solve the > problem of difficult mount-table conf update -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15625) Namenode trashEmptier should not init ViewFs on startup
[ https://issues.apache.org/jira/browse/HDFS-15625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-15625: --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) Failures are unrelated and they are due to OOME. Thanks [~weichiu] for the review! > Namenode trashEmptier should not init ViewFs on startup > --- > > Key: HDFS-15625 > URL: https://issues.apache.org/jira/browse/HDFS-15625 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode, viewfs >Affects Versions: 3.4.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Currently on NN startup, Namenode starts trash emptier. > As part of HDFS-15450 we already fixed it by setting fs.hdfs.impl config in > NN. > But as part of other JIRAs we seem to be reverted that change. > If I remember correctly, the issue in HDFS-15450 was about NN can't init > ViewFsOverloadScheme because, NN always sets it's RPC address in > fs.defaultFS. So, in HA case, user configured uri would always be a logical > uri. So, NN could not start it as it can't init because it can't find any > mount points with that changed fs.defaultFS authority. However that issues > sorted out as for HDFS we recommend to use ViewHDFS and it will auto set a > fallback mountpoint if no mount points configured. > However, we still need that changes needed back as initing > ViewHDFS/ViewFSOverloadScheme would be costly when it has more mount points. > Initing this mount based fs in trashEmptier is unnecessary. > With this JIRA I will just bring back similar functionality back. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15625) Namenode trashEmptier should not init ViewFs on startup
[ https://issues.apache.org/jira/browse/HDFS-15625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-15625: --- Status: Patch Available (was: Open) > Namenode trashEmptier should not init ViewFs on startup > --- > > Key: HDFS-15625 > URL: https://issues.apache.org/jira/browse/HDFS-15625 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode, viewfs >Affects Versions: 3.4.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Currently on NN startup, Namenode starts trash emptier. > As part of HDFS-15450 we already fixed it by setting fs.hdfs.impl config in > NN. > But as part of other JIRAs we seem to be reverted that change. > If I remember correctly, the issue in HDFS-15450 was about NN can't init > ViewFsOverloadScheme because, NN always sets it's RPC address in > fs.defaultFS. So, in HA case, user configured uri would always be a logical > uri. So, NN could not start it as it can't init because it can't find any > mount points with that changed fs.defaultFS authority. However that issues > sorted out as for HDFS we recommend to use ViewHDFS and it will auto set a > fallback mountpoint if no mount points configured. > However, we still need that changes needed back as initing > ViewHDFS/ViewFSOverloadScheme would be costly when it has more mount points. > Initing this mount based fs in trashEmptier is unnecessary. > With this JIRA I will just bring back similar functionality back. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15625) Namenode trashEmptier should not init ViewFs on startup
Uma Maheswara Rao G created HDFS-15625: -- Summary: Namenode trashEmptier should not init ViewFs on startup Key: HDFS-15625 URL: https://issues.apache.org/jira/browse/HDFS-15625 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode, viewfs Affects Versions: 3.4.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Currently on NN startup, Namenode starts trash emptier. As part of HDFS-15450 we already fixed it by setting fs.hdfs.impl config in NN. But as part of other JIRAs we seem to be reverted that change. If I remember correctly, the issue in HDFS-15450 was about NN can't init ViewFsOverloadScheme because, NN always sets it's RPC address in fs.defaultFS. So, in HA case, user configured uri would always be a logical uri. So, NN could not start it as it can't init because it can't find any mount points with that changed fs.defaultFS authority. However that issues sorted out as for HDFS we recommend to use ViewHDFS and it will auto set a fallback mountpoint if no mount points configured. However, we still need that changes needed back as initing ViewHDFS/ViewFSOverloadScheme would be costly when it has more mount points. Initing this mount based fs in trashEmptier is unnecessary. With this JIRA I will just bring back similar functionality back. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15598) ViewHDFS#canonicalizeUri should not be restricted to DFS only API.
[ https://issues.apache.org/jira/browse/HDFS-15598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G resolved HDFS-15598. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed Committed to trunk! > ViewHDFS#canonicalizeUri should not be restricted to DFS only API. > -- > > Key: HDFS-15598 > URL: https://issues.apache.org/jira/browse/HDFS-15598 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.4.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > As part of HIve Partitions verification, insert failed due to canonicalizeUri > restricted to DFS only. This can be relaxed and delegate to > vfs#canonicalizeUri -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15598) ViewHDFS#canonicalizeUri should not be restricted to DFS only API.
[
https://issues.apache.org/jira/browse/HDFS-15598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201757#comment-17201757
]
Uma Maheswara Rao G commented on HDFS-15598:
Hive Insert to a partition is failing.
{code:java}
INFO : TaskCounter_Reducer_2_OUTPUT_out_Reducer_2:
INFO : OUTPUT_RECORDS: 0
ERROR : Job Commit failed with exception
'java.lang.UnsupportedOperationException(This API:canonicalizeUri is specific
to DFS. Can't run on other fs:o3fs://bucket.vol.ozone1)'
java.lang.UnsupportedOperationException: This API:canonicalizeUri is specific
to DFS. Can't run on other fs:o3fs://bucket.vol.ozone1
at
org.apache.hadoop.hdfs.ViewDistributedFileSystem.checkDFS(ViewDistributedFileSystem.java:402)
at
org.apache.hadoop.hdfs.ViewDistributedFileSystem.canonicalizeUri(ViewDistributedFileSystem.java:1086)
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:761)
at org.apache.hadoop.fs.FileSystem.makeQualified(FileSystem.java:629)
at
org.apache.hadoop.hive.ql.exec.Utilities.handleDirectInsertTableFinalPath(Utilities.java:4602)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.jobCloseOp(FileSinkOperator.java:1470)
at org.apache.hadoop.hive.ql.exec.Operator.jobClose(Operator.java:798)
at org.apache.hadoop.hive.ql.exec.Operator.jobClose(Operator.java:803)
at org.apache.hadoop.hive.ql.exec.Operator.jobClose(Operator.java:803)
at org.apache.hadoop.hive.ql.exec.Operator.jobClose(Operator.java:803)
at org.apache.hadoop.hive.ql.exec.Operator.jobClose(Operator.java:803)
at org.apache.hadoop.hive.ql.exec.tez.TezTask.close(TezTask.java:627)
at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:342)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:357)
at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330)
at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:730)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:490)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:484)
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:225)
at
org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)
at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:322)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:340)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
ERROR : FAILED: Execution Error, return code 3 from
org.apache.hadoop.hive.ql.exec.tez.TezTask
INFO : Completed executing
command(queryId=hive_20200924190718_d1092cd8-6b63-4374-ba1b-1f6df8212f30); Time
taken: 14.705 seconds
INFO : OK
{code}
> ViewHDFS#canonicalizeUri should not be restricted to DFS only API.
> --
>
> Key: HDFS-15598
> URL: https://issues.apache.org/jira/browse/HDFS-15598
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
>
> As part of HIve Partitions verification, insert failed due to canonicalizeUri
> restricted to DFS only. This can be relaxed and delegate to
> vfs#canonicalizeUri
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15598) ViewHDFS#canonicalizeUri should not be restricted to DFS only API.
Uma Maheswara Rao G created HDFS-15598: -- Summary: ViewHDFS#canonicalizeUri should not be restricted to DFS only API. Key: HDFS-15598 URL: https://issues.apache.org/jira/browse/HDFS-15598 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: 3.4.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G As part of HIve Partitions verification, insert failed due to canonicalizeUri restricted to DFS only. This can be relaxed and delegate to vfs#canonicalizeUri -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15596) ViewHDFS#create(f, permission, cflags, bufferSize, replication, blockSize, progress, checksumOpt) should not be restricted to DFS only.
[ https://issues.apache.org/jira/browse/HDFS-15596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G resolved HDFS-15596. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Target Version/s: 3.3.1 Resolution: Fixed Thanks [~ayushtkn] for the review! Committed. > ViewHDFS#create(f, permission, cflags, bufferSize, replication, blockSize, > progress, checksumOpt) should not be restricted to DFS only. > --- > > Key: HDFS-15596 > URL: https://issues.apache.org/jira/browse/HDFS-15596 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > The ViewHDFS#create(f, permission, cflags, bufferSize, replication, > blockSize, progress, checksumOpt) API already available in FileSystem. It > will use other overloaded API and finally can go to ViewFileSystem. This case > works in regular ViewFileSystem also. With ViewHDFS, we restricted this to > DFS only which cause discp to fail when target is non hdfs as it's using this > API. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15596) ViewHDFS#create(f, permission, cflags, bufferSize, replication, blockSize, progress, checksumOpt) should not be restricted to DFS only.
[
https://issues.apache.org/jira/browse/HDFS-15596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201340#comment-17201340
]
Uma Maheswara Rao G commented on HDFS-15596:
After the fix distCp ran successfully:
{code:java}
[root@uma-1 /]# sudo -u hdfs hadoop distcp /test /OzoneTest
20/09/24 06:56:09 INFO tools.DistCp: Input Options:
DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
ignoreFailures=false, overwrite=false, append=false, useDiff=false,
useRdiff=false, fromSnapshot=null, toSnapshot=null, skipCRC=false,
blocking=true, numListstatusThreads=0, maxMaps=20, mapBandwidth=0.0,
copyStrategy='uniformsize', preserveStatus=[], atomicWorkPath=null,
logPath=null, sourceFileListing=null, sourcePaths=[/test],
targetPath=/OzoneTest, filtersFile='null', blocksPerChunk=0,
copyBufferSize=8192, verboseLog=false, directWrite=false}, sourcePaths=[/test],
targetPathExists=true, preserveRawXattrsfalse
20/09/24 06:56:10 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 2;
dirCnt = 1
20/09/24 06:56:10 INFO tools.SimpleCopyListing: Build file listing completed.
20/09/24 06:56:10 INFO Configuration.deprecation: io.sort.mb is deprecated.
Instead, use mapreduce.task.io.sort.mb
20/09/24 06:56:10 INFO Configuration.deprecation: io.sort.factor is deprecated.
Instead, use mapreduce.task.io.sort.factor
20/09/24 06:56:10 INFO tools.DistCp: Number of paths in the copy list: 2
20/09/24 06:56:10 INFO tools.DistCp: Number of paths in the copy list: 2
20/09/24 06:56:10 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
to rm23
20/09/24 06:56:10 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding
for path: /user/hdfs/.staging/job_1600930164279_0009
20/09/24 06:56:10 INFO mapreduce.JobSubmitter: number of splits:2
20/09/24 06:56:11 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1600930164279_0009
20/09/24 06:56:11 INFO mapreduce.JobSubmitter: Executing with tokens: []
20/09/24 06:56:11 INFO conf.Configuration: resource-types.xml not found
20/09/24 06:56:11 INFO resource.ResourceUtils: Unable to find
'resource-types.xml'.
20/09/24 06:56:11 INFO impl.YarnClientImpl: Submitted application
application_1600930164279_0009
20/09/24 06:56:11 INFO mapreduce.Job: The url to track the job:
http://uma-2.uma.xxx.xxx.xxx:8088/proxy/application_1600930164279_0009/
20/09/24 06:56:11 INFO tools.DistCp: DistCp job-id: job_1600930164279_0009
20/09/24 06:56:11 INFO mapreduce.Job: Running job: job_1600930164279_0009
20/09/24 06:56:22 INFO mapreduce.Job: Job job_1600930164279_0009 running in
uber mode : false
20/09/24 06:56:22 INFO mapreduce.Job: map 0% reduce 0%
20/09/24 06:56:28 INFO mapreduce.Job: map 50% reduce 0%
20/09/24 06:56:34 INFO mapreduce.Job: map 100% reduce 0%
20/09/24 06:56:34 INFO mapreduce.Job: Job job_1600930164279_0009 completed
successfully
20/09/24 06:56:35 INFO mapreduce.Job: Counters: 43
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=583482
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=865
HDFS: Number of bytes written=0
HDFS: Number of read operations=20
HDFS: Number of large read operations=0
HDFS: Number of write operations=4
HDFS: Number of bytes read erasure-coded=0
O3FS: Number of bytes read=0
O3FS: Number of bytes written=17
O3FS: Number of read operations=12
O3FS: Number of large read operations=0
O3FS: Number of write operations=3
Job Counters
Launched map tasks=2
Other local map tasks=2
Total time spent by all maps in occupied slots (ms)=13070
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=13070
Total vcore-milliseconds taken by all map tasks=13070
Total megabyte-milliseconds taken by all map tasks=13383680
Map-Reduce Framework
Map input records=2
Map output records=0
Input split bytes=228
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=285
CPU time spent (ms)=3030
Physical memory (bytes) snapshot=804298752
Virtual memory (bytes) snapshot=5346963456
Total committed heap usage (bytes)=729284608
Peak Map Physical memory (bytes)=428199936
Peak Map Virtual memory (bytes)=2674827264
File Input Format Counters
Bytes Read=620
File Output Format Counters
Bytes Written=0
DistCp Counters
Bandwidth in Btyes=17
Bytes Copied=17
Bytes Expected=17
Files Copied=1
DIR_COPY=1
{code}
> ViewHDFS#create(f, permission, cflags, bufferSize, replication, blockSize,
> progress, checksumOpt) should not be restricted to DFS only.
> ---
>
> Key: HDFS-15596
> URL: https://issues.apache.org/jira/browse/HDFS-15596
> Project: Hadoop HDFS
> Issue Type:
[jira] [Commented] (HDFS-15596) ViewHDFS#create(f, permission, cflags, bufferSize, replication, blockSize, progress, checksumOpt) should not be restricted to DFS only.
[
https://issues.apache.org/jira/browse/HDFS-15596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201336#comment-17201336
]
Uma Maheswara Rao G commented on HDFS-15596:
distcp fails with the following error:
{code:java}
Error: java.io.IOException: File copy failed: hdfs://ns1/test/test.txt -->
hdfs://ns1/OzoneTest/test/test.txt at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:219) at
org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:48) at
org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at
org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) Caused by:
java.io.IOException: Couldn't run retriable-command: Copying
hdfs://ns1/test/test.txt to hdfs://ns1/OzoneTest/test/test.txt at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
... 10 more Caused by: java.lang.UnsupportedOperationException: This
API:create is specific to DFS. Can't run on other fs:o3fs://bucket.vol.ozone1
at
org.apache.hadoop.hdfs.ViewDistributedFileSystem.checkDFS(ViewDistributedFileSystem.java:402)
at
org.apache.hadoop.hdfs.ViewDistributedFileSystem.create(ViewDistributedFileSystem.java:391)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToFile(RetriableFileCopyCommand.java:201)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:143)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
... 11 more{code}
> ViewHDFS#create(f, permission, cflags, bufferSize, replication, blockSize,
> progress, checksumOpt) should not be restricted to DFS only.
> ---
>
> Key: HDFS-15596
> URL: https://issues.apache.org/jira/browse/HDFS-15596
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The ViewHDFS#create(f, permission, cflags, bufferSize, replication,
> blockSize, progress, checksumOpt) API already available in FileSystem. It
> will use other overloaded API and finally can go to ViewFileSystem. This case
> works in regular ViewFileSystem also. With ViewHDFS, we restricted this to
> DFS only which cause discp to fail when target is non hdfs as it's using this
> API.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15596) ViewHDFS#create(f, permission, cflags, bufferSize, replication, blockSize, progress, checksumOpt) should not be restricted to DFS only.
[ https://issues.apache.org/jira/browse/HDFS-15596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-15596: --- Environment: (was: The ViewHDFS#create(f, permission, cflags, bufferSize, replication, blockSize, progress, checksumOpt) API already available in FileSystem. It will use other overloaded API and finally can go to ViewFileSystem. This case works in regular ViewFileSystem also. With ViewHDFS, we restricted this to DFS only which cause discp to fail when target is non hdfs as it's using this API.) > ViewHDFS#create(f, permission, cflags, bufferSize, replication, blockSize, > progress, checksumOpt) should not be restricted to DFS only. > --- > > Key: HDFS-15596 > URL: https://issues.apache.org/jira/browse/HDFS-15596 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15596) ViewHDFS#create(f, permission, cflags, bufferSize, replication, blockSize, progress, checksumOpt) should not be restricted to DFS only.
[ https://issues.apache.org/jira/browse/HDFS-15596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-15596: --- Description: The ViewHDFS#create(f, permission, cflags, bufferSize, replication, blockSize, progress, checksumOpt) API already available in FileSystem. It will use other overloaded API and finally can go to ViewFileSystem. This case works in regular ViewFileSystem also. With ViewHDFS, we restricted this to DFS only which cause discp to fail when target is non hdfs as it's using this API. > ViewHDFS#create(f, permission, cflags, bufferSize, replication, blockSize, > progress, checksumOpt) should not be restricted to DFS only. > --- > > Key: HDFS-15596 > URL: https://issues.apache.org/jira/browse/HDFS-15596 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > > The ViewHDFS#create(f, permission, cflags, bufferSize, replication, > blockSize, progress, checksumOpt) API already available in FileSystem. It > will use other overloaded API and finally can go to ViewFileSystem. This case > works in regular ViewFileSystem also. With ViewHDFS, we restricted this to > DFS only which cause discp to fail when target is non hdfs as it's using this > API. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15596) ViewHDFS#create(f, permission, cflags, bufferSize, replication, blockSize, progress, checksumOpt) should not be restricted to DFS only.
Uma Maheswara Rao G created HDFS-15596: -- Summary: ViewHDFS#create(f, permission, cflags, bufferSize, replication, blockSize, progress, checksumOpt) should not be restricted to DFS only. Key: HDFS-15596 URL: https://issues.apache.org/jira/browse/HDFS-15596 Project: Hadoop HDFS Issue Type: Sub-task Environment: The ViewHDFS#create(f, permission, cflags, bufferSize, replication, blockSize, progress, checksumOpt) API already available in FileSystem. It will use other overloaded API and finally can go to ViewFileSystem. This case works in regular ViewFileSystem also. With ViewHDFS, we restricted this to DFS only which cause discp to fail when target is non hdfs as it's using this API. Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15592) DistCP fails with ViewHDFS and preserveEC options if the actual target path is non HDFS
[
https://issues.apache.org/jira/browse/HDFS-15592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-15592:
---
Summary: DistCP fails with ViewHDFS and preserveEC options if the actual
target path is non HDFS (was: DistCP fails with ViewHDFS if the actual target
path is non HDFS)
> DistCP fails with ViewHDFS and preserveEC options if the actual target path
> is non HDFS
> ---
>
> Key: HDFS-15592
> URL: https://issues.apache.org/jira/browse/HDFS-15592
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: viewfs, ViewHDFS
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
>
> When we configure target path mount point with Ozone (or any other fs),
> distcp will fail.
> The reason is, if the src path having ec policy enabled, it will try to
> retain that properties.SO, in this case it is using DFS specific createFile
> API.
> But here we have to ensure, tareget path can from non hdfs in ViewHDFS case.
> In RetriayableFIleCopyCommand#copyToFile, we should fix the following piece
> of code.
>
> {code:java}
> if (preserveEC && sourceStatus.isErasureCoded()
> && sourceStatus instanceof HdfsFileStatus
> && targetFS instanceof DistributedFileSystem) {
> ecPolicy = ((HdfsFileStatus) sourceStatus).getErasureCodingPolicy();
> }{code}
>
> Here it's just checking targetFs instanceof DistributedFileSystem, but in
> ViewHDFS case, fs will be DFS only but actual target can point to mounted fs.
> So, to handle this case, we should use resolvePath API and check the resolved
> target path scheme is dfs or or not.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDFS-15592) DistCP fails with ViewHDFS if the actual target path is non HDFS
[
https://issues.apache.org/jira/browse/HDFS-15592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-15592:
---
Comment: was deleted
(was:
{code:java}
Error: java.io.IOException: File copy failed: hdfs://ns1/test/test.txt -->
hdfs://ns1/OzoneTest/test/test.txt at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:219) at
org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:48) at
org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at
org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) Caused by:
java.io.IOException: Couldn't run retriable-command: Copying
hdfs://ns1/test/test.txt to hdfs://ns1/OzoneTest/test/test.txt at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
... 10 more Caused by: java.lang.UnsupportedOperationException: This
API:create is specific to DFS. Can't run on other fs:o3fs://bucket.vol.ozone1
at
org.apache.hadoop.hdfs.ViewDistributedFileSystem.checkDFS(ViewDistributedFileSystem.java:402)
at
org.apache.hadoop.hdfs.ViewDistributedFileSystem.create(ViewDistributedFileSystem.java:391)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToFile(RetriableFileCopyCommand.java:201)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:143)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
... 11 more
{code}
)
> DistCP fails with ViewHDFS if the actual target path is non HDFS
>
>
> Key: HDFS-15592
> URL: https://issues.apache.org/jira/browse/HDFS-15592
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: viewfs, ViewHDFS
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
>
> When we configure target path mount point with Ozone (or any other fs),
> distcp will fail.
> The reason is, if the src path having ec policy enabled, it will try to
> retain that properties.SO, in this case it is using DFS specific createFile
> API.
> But here we have to ensure, tareget path can from non hdfs in ViewHDFS case.
> In RetriayableFIleCopyCommand#copyToFile, we should fix the following piece
> of code.
>
> {code:java}
> if (preserveEC && sourceStatus.isErasureCoded()
> && sourceStatus instanceof HdfsFileStatus
> && targetFS instanceof DistributedFileSystem) {
> ecPolicy = ((HdfsFileStatus) sourceStatus).getErasureCodingPolicy();
> }{code}
>
> Here it's just checking targetFs instanceof DistributedFileSystem, but in
> ViewHDFS case, fs will be DFS only but actual target can point to mounted fs.
> So, to handle this case, we should use resolvePath API and check the resolved
> target path scheme is dfs or or not.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15592) DistCP fails with ViewHDFS if the actual target path is non HDFS
[
https://issues.apache.org/jira/browse/HDFS-15592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200597#comment-17200597
]
Uma Maheswara Rao G edited comment on HDFS-15592 at 9/23/20, 6:38 AM:
--
{code:java}
Error: java.io.IOException: File copy failed: hdfs://ns1/test/test.txt -->
hdfs://ns1/OzoneTest/test/test.txt at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:219) at
org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:48) at
org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at
org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) Caused by:
java.io.IOException: Couldn't run retriable-command: Copying
hdfs://ns1/test/test.txt to hdfs://ns1/OzoneTest/test/test.txt at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
... 10 more Caused by: java.lang.UnsupportedOperationException: This
API:create is specific to DFS. Can't run on other fs:o3fs://bucket.vol.ozone1
at
org.apache.hadoop.hdfs.ViewDistributedFileSystem.checkDFS(ViewDistributedFileSystem.java:402)
at
org.apache.hadoop.hdfs.ViewDistributedFileSystem.create(ViewDistributedFileSystem.java:391)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToFile(RetriableFileCopyCommand.java:201)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:143)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
... 11 more
{code}
was (Author: umamaheswararao):
{noformat}
Error: java.io.IOException: File copy failed: hdfs://ns1/test/test.txt -->
hdfs://ns1/OzoneTest/test/test.txt at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:219) at
org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:48) at
org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at
org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) Caused by:
java.io.IOException: Couldn't run retriable-command: Copying
hdfs://ns1/test/test.txt to hdfs://ns1/OzoneTest/test/test.txt at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
... 10 more Caused by: java.lang.UnsupportedOperationException: This
API:create is specific to DFS. Can't run on other fs:o3fs://bucket.vol.ozone1
at
org.apache.hadoop.hdfs.ViewDistributedFileSystem.checkDFS(ViewDistributedFileSystem.java:402)
at
org.apache.hadoop.hdfs.ViewDistributedFileSystem.create(ViewDistributedFileSystem.java:391)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToFile(RetriableFileCopyCommand.java:201)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:143)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
... 11 more
{noformat}
> DistCP fails with ViewHDFS if the actual target path is non HDFS
>
>
> Key: HDFS-15592
> URL: https://issues.apache.org/jira/browse/HDFS-15592
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: viewfs, ViewHDFS
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
>
> When we configure target path mount point with Ozone (or any other fs),
> distcp will fail.
> The reason is, if the src path having ec policy enabled, it will try to
> retain that properties.SO, in this case it is using DFS specific createFile
> API.
> But
[jira] [Commented] (HDFS-15592) DistCP fails with ViewHDFS if the actual target path is non HDFS
[
https://issues.apache.org/jira/browse/HDFS-15592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200597#comment-17200597
]
Uma Maheswara Rao G commented on HDFS-15592:
{noformat}
Error: java.io.IOException: File copy failed: hdfs://ns1/test/test.txt -->
hdfs://ns1/OzoneTest/test/test.txt at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:219) at
org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:48) at
org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at
org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) Caused by:
java.io.IOException: Couldn't run retriable-command: Copying
hdfs://ns1/test/test.txt to hdfs://ns1/OzoneTest/test/test.txt at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
... 10 more Caused by: java.lang.UnsupportedOperationException: This
API:create is specific to DFS. Can't run on other fs:o3fs://bucket.vol.ozone1
at
org.apache.hadoop.hdfs.ViewDistributedFileSystem.checkDFS(ViewDistributedFileSystem.java:402)
at
org.apache.hadoop.hdfs.ViewDistributedFileSystem.create(ViewDistributedFileSystem.java:391)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToFile(RetriableFileCopyCommand.java:201)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:143)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
... 11 more
{noformat}
> DistCP fails with ViewHDFS if the actual target path is non HDFS
>
>
> Key: HDFS-15592
> URL: https://issues.apache.org/jira/browse/HDFS-15592
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: viewfs, ViewHDFS
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
>
> When we configure target path mount point with Ozone (or any other fs),
> distcp will fail.
> The reason is, if the src path having ec policy enabled, it will try to
> retain that properties.SO, in this case it is using DFS specific createFile
> API.
> But here we have to ensure, tareget path can from non hdfs in ViewHDFS case.
> In RetriayableFIleCopyCommand#copyToFile, we should fix the following piece
> of code.
>
> {code:java}
> if (preserveEC && sourceStatus.isErasureCoded()
> && sourceStatus instanceof HdfsFileStatus
> && targetFS instanceof DistributedFileSystem) {
> ecPolicy = ((HdfsFileStatus) sourceStatus).getErasureCodingPolicy();
> }{code}
>
> Here it's just checking targetFs instanceof DistributedFileSystem, but in
> ViewHDFS case, fs will be DFS only but actual target can point to mounted fs.
> So, to handle this case, we should use resolvePath API and check the resolved
> target path scheme is dfs or or not.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15592) DistCP fails with ViewHDFS if the actual target path is non HDFS
Uma Maheswara Rao G created HDFS-15592:
--
Summary: DistCP fails with ViewHDFS if the actual target path is
non HDFS
Key: HDFS-15592
URL: https://issues.apache.org/jira/browse/HDFS-15592
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ViewHDFS, viewfs
Affects Versions: 3.4.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
When we configure target path mount point with Ozone (or any other fs), distcp
will fail.
The reason is, if the src path having ec policy enabled, it will try to retain
that properties.SO, in this case it is using DFS specific createFile API.
But here we have to ensure, tareget path can from non hdfs in ViewHDFS case.
In RetriayableFIleCopyCommand#copyToFile, we should fix the following piece of
code.
{code:java}
if (preserveEC && sourceStatus.isErasureCoded()
&& sourceStatus instanceof HdfsFileStatus
&& targetFS instanceof DistributedFileSystem) {
ecPolicy = ((HdfsFileStatus) sourceStatus).getErasureCodingPolicy();
}{code}
Here it's just checking targetFs instanceof DistributedFileSystem, but in
ViewHDFS case, fs will be DFS only but actual target can point to mounted fs.
So, to handle this case, we should use resolvePath API and check the resolved
target path scheme is dfs or or not.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15578) Fix the rename issues with fallback fs enabled
[
https://issues.apache.org/jira/browse/HDFS-15578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-15578:
---
Fix Version/s: 3.4.0
3.3.1
Hadoop Flags: Reviewed
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to trunk and 3.3.1
> Fix the rename issues with fallback fs enabled
> --
>
> Key: HDFS-15578
> URL: https://issues.apache.org/jira/browse/HDFS-15578
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: viewfs, viewfsOverloadScheme
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.1, 3.4.0
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> When we enable fallback, rename should success if the src.parent or
> dst.parent on inernalDir.
> {noformat}
> org.apache.hadoop.security.AccessControlException: InternalDir of
> ViewFileSystem is readonly, operation rename not permitted on path
> /newFileOnRoot.org.apache.hadoop.security.AccessControlException: InternalDir
> of ViewFileSystem is readonly, operation rename not permitted on path
> /newFileOnRoot.
> at
> org.apache.hadoop.fs.viewfs.ViewFileSystem.readOnlyMountTable(ViewFileSystem.java:95)
> at
> org.apache.hadoop.fs.viewfs.ViewFileSystem.readOnlyMountTable(ViewFileSystem.java:101)
> at
> org.apache.hadoop.fs.viewfs.ViewFileSystem.rename(ViewFileSystem.java:683) at
> org.apache.hadoop.hdfs.ViewDistributedFileSystem.rename(ViewDistributedFileSystem.java:533)
> at
> org.apache.hadoop.hdfs.TestViewDistributedFileSystemWithMountLinks.verifyRename(TestViewDistributedFileSystemWithMountLinks.java:114)
> at
> org.apache.hadoop.hdfs.TestViewDistributedFileSystemWithMountLinks.testRenameOnInternalDirWithFallback(TestViewDistributedFileSystemWithMountLinks.java:90)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at
> org.junit.runner.JUnitCore.run(JUnitCore.java:137) at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
> at
> com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
> at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58){noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15585) ViewDFS#getDelegationToken should not throw UnsupportedOperationException.
Uma Maheswara Rao G created HDFS-15585: -- Summary: ViewDFS#getDelegationToken should not throw UnsupportedOperationException. Key: HDFS-15585 URL: https://issues.apache.org/jira/browse/HDFS-15585 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: 3.4.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G When starting Hive in secure environment, it is throwing UnsupportedOprationException from ViewDFS. at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:736) ~[hive-service-3.1.3000.7.2.3.0-54.jar:3.1.3000.7.2.3.0-54] at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1077) ~[hive-service-3.1.3000.7.2.3.0-54.jar:3.1.3000.7.2.3.0-54] ... 9 more Caused by: java.lang.UnsupportedOperationException at org.apache.hadoop.hdfs.ViewDistributedFileSystem.getDelegationToken(ViewDistributedFileSystem.java:1042) ~[hadoop-hdfs-client-3.1.1.7.2.3.0-54.jar:?] at org.apache.hadoop.security.token.DelegationTokenIssuer.collectDelegationTokens(DelegationTokenIssuer.java:95) ~[hadoop-common-3.1.1.7.2.3.0-54.jar:?] at org.apache.hadoop.security.token.DelegationTokenIssuer.addDelegationTokens(DelegationTokenIssuer.java:76) ~[hadoop-common-3.1.1.7.2.3.0-54.jar:?] at org.apache.tez.common.security.TokenCache.obtainTokensForFileSystemsInternal(TokenCache.java:140) ~[tez-api-0.9.1.7.2.3.0-54.jar:0.9.1.7.2.3.0-54] at org.apache.tez.common.security.TokenCache.obtainTokensForFileSystemsInternal(TokenCache.java:101) ~[tez-api-0.9.1.7.2.3.0-54.jar:0.9.1.7.2.3.0-54] at org.apache.tez.common.security.TokenCache.obtainTokensForFileSystems(TokenCache.java:77) ~[tez-api-0.9.1.7.2.3.0-54.jar:0.9.1.7.2.3.0-54] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.createLlapCredentials(TezSessionState.java:443) ~[hive-exec-3.1.3000.7.2.3.0-54.jar:3.1.3000.7.2.3.0-54] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:354) ~[hive-exec-3.1.3000.7.2.3.0-54.jar:3.1.3000.7.2.3.0-54] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:313) ~[hive-exec-3.1.3000.7.2.3.0-54.jar:3.1.3000.7.2.3.0-54] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15289) Allow viewfs mounts with HDFS/HCFS scheme and centralized mount table
[ https://issues.apache.org/jira/browse/HDFS-15289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-15289: --- Target Version/s: 3.3.1, 3.4.0, 3.1.5, 3.2.3 (was: 3.2.2, 3.3.1, 3.4.0, 3.1.5) > Allow viewfs mounts with HDFS/HCFS scheme and centralized mount table > - > > Key: HDFS-15289 > URL: https://issues.apache.org/jira/browse/HDFS-15289 > Project: Hadoop HDFS > Issue Type: New Feature > Components: fs >Affects Versions: 3.2.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Attachments: ViewFSOverloadScheme - V1.0.pdf, ViewFSOverloadScheme.png > > > ViewFS provides flexibility to mount different filesystem types with mount > points configuration table. This approach is solving the scalability > problems, but users need to reconfigure the filesystem to ViewFS and to its > scheme. This will be problematic in the case of paths persisted in meta > stores, ex: Hive. In systems like Hive, it will store uris in meta store. So, > changing the file system scheme will create a burden to upgrade/recreate meta > stores. In our experience many users are not ready to change that. > Router based federation is another implementation to provide coordinated > mount points for HDFS federation clusters. Even though this provides > flexibility to handle mount points easily, this will not allow > other(non-HDFS) file systems to mount. So, this does not solve the purpose > when users want to mount external(non-HDFS) filesystems. > So, the problem here is: Even though many users want to adapt to the scalable > fs options available, technical challenges of changing schemes (ex: in meta > stores) in deployments are obstructing them. > So, we propose to allow hdfs scheme in ViewFS like client side mount system > and provision user to create mount links without changing URI paths. > I will upload detailed design doc shortly. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15289) Allow viewfs mounts with HDFS/HCFS scheme and centralized mount table
[ https://issues.apache.org/jira/browse/HDFS-15289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17196589#comment-17196589 ] Uma Maheswara Rao G commented on HDFS-15289: [~hexiaoqiao], Thanks for ping. Sure we can set to 3.2.3. > Allow viewfs mounts with HDFS/HCFS scheme and centralized mount table > - > > Key: HDFS-15289 > URL: https://issues.apache.org/jira/browse/HDFS-15289 > Project: Hadoop HDFS > Issue Type: New Feature > Components: fs >Affects Versions: 3.2.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Attachments: ViewFSOverloadScheme - V1.0.pdf, ViewFSOverloadScheme.png > > > ViewFS provides flexibility to mount different filesystem types with mount > points configuration table. This approach is solving the scalability > problems, but users need to reconfigure the filesystem to ViewFS and to its > scheme. This will be problematic in the case of paths persisted in meta > stores, ex: Hive. In systems like Hive, it will store uris in meta store. So, > changing the file system scheme will create a burden to upgrade/recreate meta > stores. In our experience many users are not ready to change that. > Router based federation is another implementation to provide coordinated > mount points for HDFS federation clusters. Even though this provides > flexibility to handle mount points easily, this will not allow > other(non-HDFS) file systems to mount. So, this does not solve the purpose > when users want to mount external(non-HDFS) filesystems. > So, the problem here is: Even though many users want to adapt to the scalable > fs options available, technical challenges of changing schemes (ex: in meta > stores) in deployments are obstructing them. > So, we propose to allow hdfs scheme in ViewFS like client side mount system > and provision user to create mount links without changing URI paths. > I will upload detailed design doc shortly. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15578) Fix the rename issues with fallback fs enabled
[
https://issues.apache.org/jira/browse/HDFS-15578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-15578:
---
Status: Patch Available (was: Open)
> Fix the rename issues with fallback fs enabled
> --
>
> Key: HDFS-15578
> URL: https://issues.apache.org/jira/browse/HDFS-15578
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: viewfs, viewfsOverloadScheme
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
>
> When we enable fallback, rename should success if the src.parent or
> dst.parent on inernalDir.
> {noformat}
> org.apache.hadoop.security.AccessControlException: InternalDir of
> ViewFileSystem is readonly, operation rename not permitted on path
> /newFileOnRoot.org.apache.hadoop.security.AccessControlException: InternalDir
> of ViewFileSystem is readonly, operation rename not permitted on path
> /newFileOnRoot.
> at
> org.apache.hadoop.fs.viewfs.ViewFileSystem.readOnlyMountTable(ViewFileSystem.java:95)
> at
> org.apache.hadoop.fs.viewfs.ViewFileSystem.readOnlyMountTable(ViewFileSystem.java:101)
> at
> org.apache.hadoop.fs.viewfs.ViewFileSystem.rename(ViewFileSystem.java:683) at
> org.apache.hadoop.hdfs.ViewDistributedFileSystem.rename(ViewDistributedFileSystem.java:533)
> at
> org.apache.hadoop.hdfs.TestViewDistributedFileSystemWithMountLinks.verifyRename(TestViewDistributedFileSystemWithMountLinks.java:114)
> at
> org.apache.hadoop.hdfs.TestViewDistributedFileSystemWithMountLinks.testRenameOnInternalDirWithFallback(TestViewDistributedFileSystemWithMountLinks.java:90)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at
> org.junit.runner.JUnitCore.run(JUnitCore.java:137) at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
> at
> com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
> at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58){noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15578) Fix the rename issues with fallback fs enabled
[
https://issues.apache.org/jira/browse/HDFS-15578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-15578:
---
Description:
When we enable fallback, rename should success if the src.parent or dst.parent
on inernalDir.
{noformat}
org.apache.hadoop.security.AccessControlException: InternalDir of
ViewFileSystem is readonly, operation rename not permitted on path
/newFileOnRoot.org.apache.hadoop.security.AccessControlException: InternalDir
of ViewFileSystem is readonly, operation rename not permitted on path
/newFileOnRoot.
at
org.apache.hadoop.fs.viewfs.ViewFileSystem.readOnlyMountTable(ViewFileSystem.java:95)
at
org.apache.hadoop.fs.viewfs.ViewFileSystem.readOnlyMountTable(ViewFileSystem.java:101)
at org.apache.hadoop.fs.viewfs.ViewFileSystem.rename(ViewFileSystem.java:683)
at
org.apache.hadoop.hdfs.ViewDistributedFileSystem.rename(ViewDistributedFileSystem.java:533)
at
org.apache.hadoop.hdfs.TestViewDistributedFileSystemWithMountLinks.verifyRename(TestViewDistributedFileSystemWithMountLinks.java:114)
at
org.apache.hadoop.hdfs.TestViewDistributedFileSystemWithMountLinks.testRenameOnInternalDirWithFallback(TestViewDistributedFileSystemWithMountLinks.java:90)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at
org.junit.runner.JUnitCore.run(JUnitCore.java:137) at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at
com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
at
com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58){noformat}
was:
When we enabled fallback, rename should success if the src.parent or dst.parent
on inernalDir.
{noformat}
org.apache.hadoop.security.AccessControlException: InternalDir of
ViewFileSystem is readonly, operation rename not permitted on path
/newFileOnRoot.org.apache.hadoop.security.AccessControlException: InternalDir
of ViewFileSystem is readonly, operation rename not permitted on path
/newFileOnRoot.
at
org.apache.hadoop.fs.viewfs.ViewFileSystem.readOnlyMountTable(ViewFileSystem.java:95)
at
org.apache.hadoop.fs.viewfs.ViewFileSystem.readOnlyMountTable(ViewFileSystem.java:101)
at org.apache.hadoop.fs.viewfs.ViewFileSystem.rename(ViewFileSystem.java:683)
at
org.apache.hadoop.hdfs.ViewDistributedFileSystem.rename(ViewDistributedFileSystem.java:533)
at
org.apache.hadoop.hdfs.TestViewDistributedFileSystemWithMountLinks.verifyRename(TestViewDistributedFileSystemWithMountLinks.java:114)
at
org.apache.hadoop.hdfs.TestViewDistributedFileSystemWithMountLinks.testRenameOnInternalDirWithFallback(TestViewDistributedFileSystemWithMountLinks.java:90)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
[jira] [Created] (HDFS-15578) Fix the rename issues with fallback fs enabled
Uma Maheswara Rao G created HDFS-15578:
--
Summary: Fix the rename issues with fallback fs enabled
Key: HDFS-15578
URL: https://issues.apache.org/jira/browse/HDFS-15578
Project: Hadoop HDFS
Issue Type: Sub-task
Components: viewfs, viewfsOverloadScheme
Affects Versions: 3.4.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
When we enabled fallback, rename should success if the src.parent or dst.parent
on inernalDir.
{noformat}
org.apache.hadoop.security.AccessControlException: InternalDir of
ViewFileSystem is readonly, operation rename not permitted on path
/newFileOnRoot.org.apache.hadoop.security.AccessControlException: InternalDir
of ViewFileSystem is readonly, operation rename not permitted on path
/newFileOnRoot.
at
org.apache.hadoop.fs.viewfs.ViewFileSystem.readOnlyMountTable(ViewFileSystem.java:95)
at
org.apache.hadoop.fs.viewfs.ViewFileSystem.readOnlyMountTable(ViewFileSystem.java:101)
at org.apache.hadoop.fs.viewfs.ViewFileSystem.rename(ViewFileSystem.java:683)
at
org.apache.hadoop.hdfs.ViewDistributedFileSystem.rename(ViewDistributedFileSystem.java:533)
at
org.apache.hadoop.hdfs.TestViewDistributedFileSystemWithMountLinks.verifyRename(TestViewDistributedFileSystemWithMountLinks.java:114)
at
org.apache.hadoop.hdfs.TestViewDistributedFileSystemWithMountLinks.testRenameOnInternalDirWithFallback(TestViewDistributedFileSystemWithMountLinks.java:90)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at
org.junit.runner.JUnitCore.run(JUnitCore.java:137) at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at
com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
at
com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58){noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15532) listFiles on root/InternalDir will fail if fallback root has file
[
https://issues.apache.org/jira/browse/HDFS-15532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-15532:
---
Fix Version/s: 3.4.0
3.3.1
Hadoop Flags: Reviewed
Resolution: Fixed
Status: Resolved (was: Patch Available)
> listFiles on root/InternalDir will fail if fallback root has file
> -
>
> Key: HDFS-15532
> URL: https://issues.apache.org/jira/browse/HDFS-15532
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.1, 3.4.0
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> listFiles implementation gets the RemoteIterator created in
> InternalViewFSDirFs as the root is an InternalViewFSDir.
> If there is a fallback and a file exist at root level, it would have
> collected when collecting locatedStatuses.
> When its iterating over to that fallbacks file from RemoteIterator (which
> was returned from InternalViewFSDirFs ), iterator's next will will call
> getFileBlockLocations if it's a file.
> {code:java}
> @Override
> public LocatedFileStatus next() throws IOException {
> System.out.println(this);
> if (!hasNext()) {
> throw new NoSuchElementException("No more entries in " + f);
> }
> FileStatus result = stats[i++];
> // for files, use getBlockLocations(FileStatus, int, int) to avoid
> // calling getFileStatus(Path) to load the FileStatus again
> BlockLocation[] locs = result.isFile() ?
> getFileBlockLocations(result, 0, result.getLen()) :
> null;
> return new LocatedFileStatus(result, locs);
> }{code}
>
> this getFileBlockLocations will be made on InternalViewFSDirFs, as that
> Iterator created originally from that fs.
> InternalViewFSDirFs#getFileBlockLocations does not handle fallback cases.
> It's always expecting "/", this means it always assuming the dir.
> But with the fallback and returning Iterator from InternalViewFSDirFs, will
> create problems.
> Probably we need to handle fallback case in getFileBlockLocations as well.(
> Fallback only should be the reason for call coming to InternalViewFSDirFs
> with other than "/")
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15529) getChildFilesystems should include fallback fs as well
[ https://issues.apache.org/jira/browse/HDFS-15529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-15529: --- Fix Version/s: 3.3.1 > getChildFilesystems should include fallback fs as well > -- > > Key: HDFS-15529 > URL: https://issues.apache.org/jira/browse/HDFS-15529 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: viewfs, viewfsOverloadScheme >Affects Versions: 3.4.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Critical > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Currently getChildSystems API used by many other APIs, like > getAdditionalTokenIssuers, getTrashRoots etc. > If fallBack filesystem not included in child filesystems, Application like > YARN who uses getAdditionalTokenIssuers, would not get delegation tokens for > fallback fs. This would be a critical bug for secure clusters. > Similarly, trashRoots. when applications tried to use trashRoots, it will not > considers trash folders from fallback. So, it will leak from cleanup logics. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15558) ViewDistributedFileSystem#recoverLease should call super.recoverLease when there are no mounts configured
[ https://issues.apache.org/jira/browse/HDFS-15558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-15558: --- Fix Version/s: 3.3.1 > ViewDistributedFileSystem#recoverLease should call super.recoverLease when > there are no mounts configured > - > > Key: HDFS-15558 > URL: https://issues.apache.org/jira/browse/HDFS-15558 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15478) When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs.
[ https://issues.apache.org/jira/browse/HDFS-15478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-15478: --- Fix Version/s: 3.3.1 > When Empty mount points, we are assigning fallback link to self. But it > should not use full URI for target fs. > -- > > Key: HDFS-15478 > URL: https://issues.apache.org/jira/browse/HDFS-15478 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Fix For: 3.3.1, 3.4.0 > > > On empty mount tables detection, we will automatically assign fallback with > the same initialized uri fs. Currently we are using given uri for creating > target fs. > When creating target fs, we use Chrooted fs where it will set the path from > uri as base directory. So, this can make path wrong in the case of fs > initialized with path. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15532) listFiles on root/InternalDir will fail if fallback root has file
[
https://issues.apache.org/jira/browse/HDFS-15532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-15532:
---
Affects Version/s: 3.4.0
Status: Patch Available (was: Open)
> listFiles on root/InternalDir will fail if fallback root has file
> -
>
> Key: HDFS-15532
> URL: https://issues.apache.org/jira/browse/HDFS-15532
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> listFiles implementation gets the RemoteIterator created in
> InternalViewFSDirFs as the root is an InternalViewFSDir.
> If there is a fallback and a file exist at root level, it would have
> collected when collecting locatedStatuses.
> When its iterating over to that fallbacks file from RemoteIterator (which
> was returned from InternalViewFSDirFs ), iterator's next will will call
> getFileBlockLocations if it's a file.
> {code:java}
> @Override
> public LocatedFileStatus next() throws IOException {
> System.out.println(this);
> if (!hasNext()) {
> throw new NoSuchElementException("No more entries in " + f);
> }
> FileStatus result = stats[i++];
> // for files, use getBlockLocations(FileStatus, int, int) to avoid
> // calling getFileStatus(Path) to load the FileStatus again
> BlockLocation[] locs = result.isFile() ?
> getFileBlockLocations(result, 0, result.getLen()) :
> null;
> return new LocatedFileStatus(result, locs);
> }{code}
>
> this getFileBlockLocations will be made on InternalViewFSDirFs, as that
> Iterator created originally from that fs.
> InternalViewFSDirFs#getFileBlockLocations does not handle fallback cases.
> It's always expecting "/", this means it always assuming the dir.
> But with the fallback and returning Iterator from InternalViewFSDirFs, will
> create problems.
> Probably we need to handle fallback case in getFileBlockLocations as well.(
> Fallback only should be the reason for call coming to InternalViewFSDirFs
> with other than "/")
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15532) listFiles on root/InternalDir will fail if fallback root has file
[
https://issues.apache.org/jira/browse/HDFS-15532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-15532:
---
Summary: listFiles on root/InternalDir will fail if fallback root has file
(was: listFiles on root will fail if fallback root has file)
> listFiles on root/InternalDir will fail if fallback root has file
> -
>
> Key: HDFS-15532
> URL: https://issues.apache.org/jira/browse/HDFS-15532
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
>
> listFiles implementation gets the RemoteIterator created in
> InternalViewFSDirFs as the root is an InternalViewFSDir.
> If there is a fallback and a file exist at root level, it would have
> collected when collecting locatedStatuses.
> When its iterating over to that fallbacks file from RemoteIterator (which
> was returned from InternalViewFSDirFs ), iterator's next will will call
> getFileBlockLocations if it's a file.
> {code:java}
> @Override
> public LocatedFileStatus next() throws IOException {
> System.out.println(this);
> if (!hasNext()) {
> throw new NoSuchElementException("No more entries in " + f);
> }
> FileStatus result = stats[i++];
> // for files, use getBlockLocations(FileStatus, int, int) to avoid
> // calling getFileStatus(Path) to load the FileStatus again
> BlockLocation[] locs = result.isFile() ?
> getFileBlockLocations(result, 0, result.getLen()) :
> null;
> return new LocatedFileStatus(result, locs);
> }{code}
>
> this getFileBlockLocations will be made on InternalViewFSDirFs, as that
> Iterator created originally from that fs.
> InternalViewFSDirFs#getFileBlockLocations does not handle fallback cases.
> It's always expecting "/", this means it always assuming the dir.
> But with the fallback and returning Iterator from InternalViewFSDirFs, will
> create problems.
> Probably we need to handle fallback case in getFileBlockLocations as well.(
> Fallback only should be the reason for call coming to InternalViewFSDirFs
> with other than "/")
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15533) Provide DFS API compatible class(ViewDistributedFileSystem), but use ViewFileSystemOverloadScheme inside
[
https://issues.apache.org/jira/browse/HDFS-15533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-15533:
---
Target Version/s: 3.2.2, 3.4.0 (was: 3.4.0)
> Provide DFS API compatible class(ViewDistributedFileSystem), but use
> ViewFileSystemOverloadScheme inside
>
>
> Key: HDFS-15533
> URL: https://issues.apache.org/jira/browse/HDFS-15533
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: dfs, viewfs
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Fix For: 3.3.1, 3.4.0
>
>
> I have been working on a thought from last week is that, we wanted to provide
> DFS compatible APIs with mount functionality. So, that existing DFS
> applications can work with out class cast issues.
> When we tested with other components like Hive and HBase, I noticed some
> classcast issues.
> {code:java}
> HBase example:
> java.lang.ClassCastException:
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
> org.apache.hadoop.hdfs.DistributedFileSystemjava.lang.ClassCastException:
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
> org.apache.hadoop.hdfs.DistributedFileSystem at
> org.apache.hadoop.hbase.util.FSUtils.getDFSHedgedReadMetrics(FSUtils.java:1748)
> at
> org.apache.hadoop.hbase.regionserver.MetricsRegionServerWrapperImpl.(MetricsRegionServerWrapperImpl.java:146)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1594)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1001)
> at java.lang.Thread.run(Thread.java:748){code}
> {code:java}
> Hive:
> |io.AcidUtils|: Failed to get files with ID; using regular API: Only
> supported for DFS; got class
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme{code}
> SO, here the implementation details are like follows:
> We extended DistributedFileSystem and created a class called "
> ViewDistributedFileSystem"
> This vfs=ViewFirstibutedFileSystem, try to initialize
> ViewFileSystemOverloadScheme. If success call will delegate to vfs. If fails
> to initialize due to no mount points, or other errors, it will just fallback
> to regular DFS init. If users does not configure any mount, system will
> behave exactly like today's DFS. If there are mount points, vfs functionality
> will come under DFS.
> I have a patch and will post it in some time.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15558) ViewDistributedFileSystem#recoverLease should call super.recoverLease when there are no mounts configured
[ https://issues.apache.org/jira/browse/HDFS-15558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-15558: --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) Thanks [~bharat] for the review! > ViewDistributedFileSystem#recoverLease should call super.recoverLease when > there are no mounts configured > - > > Key: HDFS-15558 > URL: https://issues.apache.org/jira/browse/HDFS-15558 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15558) ViewDistributedFileSystem#recoverLease should call super.recoverLease when there are no mounts configured
[ https://issues.apache.org/jira/browse/HDFS-15558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-15558: --- Status: Patch Available (was: Open) > ViewDistributedFileSystem#recoverLease should call super.recoverLease when > there are no mounts configured > - > > Key: HDFS-15558 > URL: https://issues.apache.org/jira/browse/HDFS-15558 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15558) ViewDistributedFileSystem#recoverLease should call super.recoverLease when there are no mounts configured
Uma Maheswara Rao G created HDFS-15558: -- Summary: ViewDistributedFileSystem#recoverLease should call super.recoverLease when there are no mounts configured Key: HDFS-15558 URL: https://issues.apache.org/jira/browse/HDFS-15558 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15529) getChildFilesystems should include fallback fs as well
[ https://issues.apache.org/jira/browse/HDFS-15529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G resolved HDFS-15529. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed Thanks [~ayushsaxena] for review. I have committed this to trunk. > getChildFilesystems should include fallback fs as well > -- > > Key: HDFS-15529 > URL: https://issues.apache.org/jira/browse/HDFS-15529 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: viewfs, viewfsOverloadScheme >Affects Versions: 3.4.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Critical > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Currently getChildSystems API used by many other APIs, like > getAdditionalTokenIssuers, getTrashRoots etc. > If fallBack filesystem not included in child filesystems, Application like > YARN who uses getAdditionalTokenIssuers, would not get delegation tokens for > fallback fs. This would be a critical bug for secure clusters. > Similarly, trashRoots. when applications tried to use trashRoots, it will not > considers trash folders from fallback. So, it will leak from cleanup logics. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15533) Provide DFS API compatible class(ViewDistributedFileSystem), but use ViewFileSystemOverloadScheme inside
[
https://issues.apache.org/jira/browse/HDFS-15533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17186902#comment-17186902
]
Uma Maheswara Rao G commented on HDFS-15533:
[~abhishekd] FYI, if this can help in any of use cases.
> Provide DFS API compatible class(ViewDistributedFileSystem), but use
> ViewFileSystemOverloadScheme inside
>
>
> Key: HDFS-15533
> URL: https://issues.apache.org/jira/browse/HDFS-15533
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: dfs, viewfs
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Fix For: 3.3.1, 3.4.0
>
>
> I have been working on a thought from last week is that, we wanted to provide
> DFS compatible APIs with mount functionality. So, that existing DFS
> applications can work with out class cast issues.
> When we tested with other components like Hive and HBase, I noticed some
> classcast issues.
> {code:java}
> HBase example:
> java.lang.ClassCastException:
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
> org.apache.hadoop.hdfs.DistributedFileSystemjava.lang.ClassCastException:
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
> org.apache.hadoop.hdfs.DistributedFileSystem at
> org.apache.hadoop.hbase.util.FSUtils.getDFSHedgedReadMetrics(FSUtils.java:1748)
> at
> org.apache.hadoop.hbase.regionserver.MetricsRegionServerWrapperImpl.(MetricsRegionServerWrapperImpl.java:146)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1594)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1001)
> at java.lang.Thread.run(Thread.java:748){code}
> {code:java}
> Hive:
> |io.AcidUtils|: Failed to get files with ID; using regular API: Only
> supported for DFS; got class
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme{code}
> SO, here the implementation details are like follows:
> We extended DistributedFileSystem and created a class called "
> ViewDistributedFileSystem"
> This vfs=ViewFirstibutedFileSystem, try to initialize
> ViewFileSystemOverloadScheme. If success call will delegate to vfs. If fails
> to initialize due to no mount points, or other errors, it will just fallback
> to regular DFS init. If users does not configure any mount, system will
> behave exactly like today's DFS. If there are mount points, vfs functionality
> will come under DFS.
> I have a patch and will post it in some time.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15329) Provide FileContext based ViewFSOverloadScheme implementation
[ https://issues.apache.org/jira/browse/HDFS-15329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17186899#comment-17186899 ] Uma Maheswara Rao G commented on HDFS-15329: I just converted it as top level issue to see whether I can assign. But no luck. > Provide FileContext based ViewFSOverloadScheme implementation > - > > Key: HDFS-15329 > URL: https://issues.apache.org/jira/browse/HDFS-15329 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs, hdfs, viewfs, viewfsOverloadScheme >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This Jira to track for FileContext based ViewFSOverloadScheme implementation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15329) Provide FileContext based ViewFSOverloadScheme implementation
[ https://issues.apache.org/jira/browse/HDFS-15329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-15329: --- Parent: (was: HDFS-15289) Issue Type: Bug (was: Sub-task) > Provide FileContext based ViewFSOverloadScheme implementation > - > > Key: HDFS-15329 > URL: https://issues.apache.org/jira/browse/HDFS-15329 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs, hdfs, viewfs, viewfsOverloadScheme >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This Jira to track for FileContext based ViewFSOverloadScheme implementation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14096) [SPS] : Add Support for Storage Policy Satisfier in ViewFs
[ https://issues.apache.org/jira/browse/HDFS-14096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-14096: --- Fix Version/s: 3.2.2 > [SPS] : Add Support for Storage Policy Satisfier in ViewFs > -- > > Key: HDFS-14096 > URL: https://issues.apache.org/jira/browse/HDFS-14096 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: federation >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Fix For: 3.3.0, 3.2.2 > > Attachments: HDFS-14096-01.patch, HDFS-14096-02.patch, > HDFS-14096-03.patch, HDFS-14096-04.patch > > > Add support for SPS in ViewFileSystem. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15117) EC: Add getECTopologyResultForPolicies to DistributedFileSystem
[
https://issues.apache.org/jira/browse/HDFS-15117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184957#comment-17184957
]
Uma Maheswara Rao G commented on HDFS-15117:
[~ayushtkn], Thanks for checking.
I think we can't merge this patch to 3.2 because of protoBuff version
compatibility issues?
{code:java}
@Override
public GetECTopologyResultForPoliciesResponseProto
getECTopologyResultForPolicies(
RpcController controller, GetECTopologyResultForPoliciesRequestProto
req)
throws ServiceException {
try {
ProtocolStringList policies = req.getPoliciesList(); {code}
Looks like we used ProtocolStringList class which seems to be added in proto
2.6.
Since 3.2 was pointing to 2.5, this can't be compiled there.
[https://abi-laboratory.pro/?view=compat_report=java=protobuf-java=2.5.0=2.6.0=be7b9=src]
> EC: Add getECTopologyResultForPolicies to DistributedFileSystem
> ---
>
> Key: HDFS-15117
> URL: https://issues.apache.org/jira/browse/HDFS-15117
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: ec
> Fix For: 3.3.0
>
> Attachments: HDFS-15117-01.patch, HDFS-15117-02.patch,
> HDFS-15117-03.patch, HDFS-15117-04.patch, HDFS-15117-05.patch,
> HDFS-15117-06.patch, HDFS-15117-07.patch, HDFS-15117-08.patch
>
>
> Add getECTopologyResultForPolicies API to distributed filesystem.
> It is as of now only present as part of ECAdmin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15515) mkdirs on fallback should throw IOE out instead of suppressing and returning false
[
https://issues.apache.org/jira/browse/HDFS-15515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-15515:
---
Fix Version/s: 3.1.5
3.2.2
> mkdirs on fallback should throw IOE out instead of suppressing and returning
> false
> --
>
> Key: HDFS-15515
> URL: https://issues.apache.org/jira/browse/HDFS-15515
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Fix For: 3.2.2, 3.3.1, 3.1.5
>
>
> Currently when doing mkdirs on fallback dir, we catching IOE and returning
> false.
> I think we should just throw IOE out as the fs#mkdirs throws IOE out.
> I noticed a case when we attempt to create .reserved dirs, NN throws
> HadoopIAE.
> But we will catch and return false. Here exception should be thrown out.
> {code:java}
> try {
> return linkedFallbackFs.mkdirs(dirToCreate, permission);
> } catch (IOException e) {
> if (LOG.isDebugEnabled()) {
> StringBuilder msg =
> new StringBuilder("Failed to create ").append(dirToCreate)
> .append(" at fallback : ")
> .append(linkedFallbackFs.getUri());
> LOG.debug(msg.toString(), e);
> }
> return false;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15533) Provide DFS API compatible class(ViewDistributedFileSystem), but use ViewFileSystemOverloadScheme inside
[
https://issues.apache.org/jira/browse/HDFS-15533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-15533:
---
Fix Version/s: 3.3.1
> Provide DFS API compatible class(ViewDistributedFileSystem), but use
> ViewFileSystemOverloadScheme inside
>
>
> Key: HDFS-15533
> URL: https://issues.apache.org/jira/browse/HDFS-15533
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: dfs, viewfs
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Fix For: 3.3.1, 3.4.0
>
>
> I have been working on a thought from last week is that, we wanted to provide
> DFS compatible APIs with mount functionality. So, that existing DFS
> applications can work with out class cast issues.
> When we tested with other components like Hive and HBase, I noticed some
> classcast issues.
> {code:java}
> HBase example:
> java.lang.ClassCastException:
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
> org.apache.hadoop.hdfs.DistributedFileSystemjava.lang.ClassCastException:
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
> org.apache.hadoop.hdfs.DistributedFileSystem at
> org.apache.hadoop.hbase.util.FSUtils.getDFSHedgedReadMetrics(FSUtils.java:1748)
> at
> org.apache.hadoop.hbase.regionserver.MetricsRegionServerWrapperImpl.(MetricsRegionServerWrapperImpl.java:146)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1594)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1001)
> at java.lang.Thread.run(Thread.java:748){code}
> {code:java}
> Hive:
> |io.AcidUtils|: Failed to get files with ID; using regular API: Only
> supported for DFS; got class
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme{code}
> SO, here the implementation details are like follows:
> We extended DistributedFileSystem and created a class called "
> ViewDistributedFileSystem"
> This vfs=ViewFirstibutedFileSystem, try to initialize
> ViewFileSystemOverloadScheme. If success call will delegate to vfs. If fails
> to initialize due to no mount points, or other errors, it will just fallback
> to regular DFS init. If users does not configure any mount, system will
> behave exactly like today's DFS. If there are mount points, vfs functionality
> will come under DFS.
> I have a patch and will post it in some time.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15515) mkdirs on fallback should throw IOE out instead of suppressing and returning false
[
https://issues.apache.org/jira/browse/HDFS-15515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184692#comment-17184692
]
Uma Maheswara Rao G commented on HDFS-15515:
Thank you [~ste...@apache.org] :)
> mkdirs on fallback should throw IOE out instead of suppressing and returning
> false
> --
>
> Key: HDFS-15515
> URL: https://issues.apache.org/jira/browse/HDFS-15515
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Fix For: 3.3.1
>
>
> Currently when doing mkdirs on fallback dir, we catching IOE and returning
> false.
> I think we should just throw IOE out as the fs#mkdirs throws IOE out.
> I noticed a case when we attempt to create .reserved dirs, NN throws
> HadoopIAE.
> But we will catch and return false. Here exception should be thrown out.
> {code:java}
> try {
> return linkedFallbackFs.mkdirs(dirToCreate, permission);
> } catch (IOException e) {
> if (LOG.isDebugEnabled()) {
> StringBuilder msg =
> new StringBuilder("Failed to create ").append(dirToCreate)
> .append(" at fallback : ")
> .append(linkedFallbackFs.getUri());
> LOG.debug(msg.toString(), e);
> }
> return false;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8631) WebHDFS : Support setQuota
[ https://issues.apache.org/jira/browse/HDFS-8631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184678#comment-17184678 ] Uma Maheswara Rao G commented on HDFS-8631: --- [~ayushtkn], yes patch is same. Thanks for re-opening it. Let's see the Jenkins results. > WebHDFS : Support setQuota > -- > > Key: HDFS-8631 > URL: https://issues.apache.org/jira/browse/HDFS-8631 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.2 >Reporter: nijel >Assignee: Chao Sun >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-8631-001.patch, HDFS-8631-002.patch, > HDFS-8631-003.patch, HDFS-8631-004.patch, HDFS-8631-005.patch, > HDFS-8631-006.patch, HDFS-8631-007.patch, HDFS-8631-008.patch, > HDFS-8631-009.patch, HDFS-8631-010.patch, HDFS-8631-011.patch, > HDFS-8631-branch-3.2.001.patch > > > User is able do quota management from filesystem object. Same operation can > be allowed trough REST API. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8631) WebHDFS : Support setQuota
[ https://issues.apache.org/jira/browse/HDFS-8631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184199#comment-17184199 ] Uma Maheswara Rao G commented on HDFS-8631: --- [~surendrasingh] , [~ayushtkn] Could you please check above? Thanks > WebHDFS : Support setQuota > -- > > Key: HDFS-8631 > URL: https://issues.apache.org/jira/browse/HDFS-8631 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.2 >Reporter: nijel >Assignee: Chao Sun >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-8631-001.patch, HDFS-8631-002.patch, > HDFS-8631-003.patch, HDFS-8631-004.patch, HDFS-8631-005.patch, > HDFS-8631-006.patch, HDFS-8631-007.patch, HDFS-8631-008.patch, > HDFS-8631-009.patch, HDFS-8631-010.patch, HDFS-8631-011.patch, > HDFS-8631-branch-3.2.001.patch > > > User is able do quota management from filesystem object. Same operation can > be allowed trough REST API. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15117) EC: Add getECTopologyResultForPolicies to DistributedFileSystem
[ https://issues.apache.org/jira/browse/HDFS-15117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183789#comment-17183789 ] Uma Maheswara Rao G commented on HDFS-15117: [~ayushtkn] Can we merge this to 3.2 as well? I realized ViewDistributedFileSystem provided one API which was introduced in this class. If we want to back-port to 3.2, we may need this in 3.2 to keep clean merge. Thanks > EC: Add getECTopologyResultForPolicies to DistributedFileSystem > --- > > Key: HDFS-15117 > URL: https://issues.apache.org/jira/browse/HDFS-15117 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: ec > Fix For: 3.3.0 > > Attachments: HDFS-15117-01.patch, HDFS-15117-02.patch, > HDFS-15117-03.patch, HDFS-15117-04.patch, HDFS-15117-05.patch, > HDFS-15117-06.patch, HDFS-15117-07.patch, HDFS-15117-08.patch > > > Add getECTopologyResultForPolicies API to distributed filesystem. > It is as of now only present as part of ECAdmin. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8631) WebHDFS : Support setQuota
[ https://issues.apache.org/jira/browse/HDFS-8631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183680#comment-17183680 ] Uma Maheswara Rao G commented on HDFS-8631: --- Updated back-port patch to 3.2. Same patch to trigger Jenkins. Cherry-pick has conflicts, but HDFS-8631-011.patch can be applied successfully. > WebHDFS : Support setQuota > -- > > Key: HDFS-8631 > URL: https://issues.apache.org/jira/browse/HDFS-8631 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.2 >Reporter: nijel >Assignee: Chao Sun >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-8631-001.patch, HDFS-8631-002.patch, > HDFS-8631-003.patch, HDFS-8631-004.patch, HDFS-8631-005.patch, > HDFS-8631-006.patch, HDFS-8631-007.patch, HDFS-8631-008.patch, > HDFS-8631-009.patch, HDFS-8631-010.patch, HDFS-8631-011.patch, > HDFS-8631-branch-3.2.001.patch > > > User is able do quota management from filesystem object. Same operation can > be allowed trough REST API. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8631) WebHDFS : Support setQuota
[ https://issues.apache.org/jira/browse/HDFS-8631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-8631: -- Attachment: HDFS-8631-branch-3.2.001.patch > WebHDFS : Support setQuota > -- > > Key: HDFS-8631 > URL: https://issues.apache.org/jira/browse/HDFS-8631 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.2 >Reporter: nijel >Assignee: Chao Sun >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-8631-001.patch, HDFS-8631-002.patch, > HDFS-8631-003.patch, HDFS-8631-004.patch, HDFS-8631-005.patch, > HDFS-8631-006.patch, HDFS-8631-007.patch, HDFS-8631-008.patch, > HDFS-8631-009.patch, HDFS-8631-010.patch, HDFS-8631-011.patch, > HDFS-8631-branch-3.2.001.patch > > > User is able do quota management from filesystem object. Same operation can > be allowed trough REST API. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15533) Provide DFS API compatible class(ViewDistributedFileSystem), but use ViewFileSystemOverloadScheme inside
[
https://issues.apache.org/jira/browse/HDFS-15533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-15533:
---
Summary: Provide DFS API compatible class(ViewDistributedFileSystem), but
use ViewFileSystemOverloadScheme inside (was: Provide DFS API compatible
class, but use ViewFileSystemOverloadScheme inside)
> Provide DFS API compatible class(ViewDistributedFileSystem), but use
> ViewFileSystemOverloadScheme inside
>
>
> Key: HDFS-15533
> URL: https://issues.apache.org/jira/browse/HDFS-15533
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: dfs, viewfs
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Fix For: 3.4.0
>
>
> I have been working on a thought from last week is that, we wanted to provide
> DFS compatible APIs with mount functionality. So, that existing DFS
> applications can work with out class cast issues.
> When we tested with other components like Hive and HBase, I noticed some
> classcast issues.
> {code:java}
> HBase example:
> java.lang.ClassCastException:
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
> org.apache.hadoop.hdfs.DistributedFileSystemjava.lang.ClassCastException:
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
> org.apache.hadoop.hdfs.DistributedFileSystem at
> org.apache.hadoop.hbase.util.FSUtils.getDFSHedgedReadMetrics(FSUtils.java:1748)
> at
> org.apache.hadoop.hbase.regionserver.MetricsRegionServerWrapperImpl.(MetricsRegionServerWrapperImpl.java:146)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1594)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1001)
> at java.lang.Thread.run(Thread.java:748){code}
> {code:java}
> Hive:
> |io.AcidUtils|: Failed to get files with ID; using regular API: Only
> supported for DFS; got class
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme{code}
> SO, here the implementation details are like follows:
> We extended DistributedFileSystem and created a class called "
> ViewDistributedFileSystem"
> This vfs=ViewFirstibutedFileSystem, try to initialize
> ViewFileSystemOverloadScheme. If success call will delegate to vfs. If fails
> to initialize due to no mount points, or other errors, it will just fallback
> to regular DFS init. If users does not configure any mount, system will
> behave exactly like today's DFS. If there are mount points, vfs functionality
> will come under DFS.
> I have a patch and will post it in some time.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15533) Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside
[
https://issues.apache.org/jira/browse/HDFS-15533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G resolved HDFS-15533.
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
I have just committed this to trunk. Thanks [~ayushtkn] for reviews!
> Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside
> -
>
> Key: HDFS-15533
> URL: https://issues.apache.org/jira/browse/HDFS-15533
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: dfs, viewfs
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Fix For: 3.4.0
>
>
> I have been working on a thought from last week is that, we wanted to provide
> DFS compatible APIs with mount functionality. So, that existing DFS
> applications can work with out class cast issues.
> When we tested with other components like Hive and HBase, I noticed some
> classcast issues.
> {code:java}
> HBase example:
> java.lang.ClassCastException:
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
> org.apache.hadoop.hdfs.DistributedFileSystemjava.lang.ClassCastException:
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
> org.apache.hadoop.hdfs.DistributedFileSystem at
> org.apache.hadoop.hbase.util.FSUtils.getDFSHedgedReadMetrics(FSUtils.java:1748)
> at
> org.apache.hadoop.hbase.regionserver.MetricsRegionServerWrapperImpl.(MetricsRegionServerWrapperImpl.java:146)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1594)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1001)
> at java.lang.Thread.run(Thread.java:748){code}
> {code:java}
> Hive:
> |io.AcidUtils|: Failed to get files with ID; using regular API: Only
> supported for DFS; got class
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme{code}
> SO, here the implementation details are like follows:
> We extended DistributedFileSystem and created a class called "
> ViewDistributedFileSystem"
> This vfs=ViewFirstibutedFileSystem, try to initialize
> ViewFileSystemOverloadScheme. If success call will delegate to vfs. If fails
> to initialize due to no mount points, or other errors, it will just fallback
> to regular DFS init. If users does not configure any mount, system will
> behave exactly like today's DFS. If there are mount points, vfs functionality
> will come under DFS.
> I have a patch and will post it in some time.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15533) Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside
[
https://issues.apache.org/jira/browse/HDFS-15533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-15533:
---
Description:
I have been working on a thought from last week is that, we wanted to provide
DFS compatible APIs with mount functionality. So, that existing DFS
applications can work with out class cast issues.
When we tested with other components like Hive and HBase, I noticed some
classcast issues.
{code:java}
HBase example:
java.lang.ClassCastException:
org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
org.apache.hadoop.hdfs.DistributedFileSystemjava.lang.ClassCastException:
org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
org.apache.hadoop.hdfs.DistributedFileSystem at
org.apache.hadoop.hbase.util.FSUtils.getDFSHedgedReadMetrics(FSUtils.java:1748)
at
org.apache.hadoop.hbase.regionserver.MetricsRegionServerWrapperImpl.(MetricsRegionServerWrapperImpl.java:146)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1594)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1001)
at java.lang.Thread.run(Thread.java:748){code}
{code:java}
Hive:
|io.AcidUtils|: Failed to get files with ID; using regular API: Only supported
for DFS; got class
org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme{code}
SO, here the implementation details are like follows:
We extended DistributedFileSystem and created a class called "
ViewDistributedFileSystem"
This vfs=ViewFirstibutedFileSystem, try to initialize
ViewFileSystemOverloadScheme. If success call will delegate to vfs. If fails
to initialize due to no mount points, or other errors, it will just fallback to
regular DFS init. If users does not configure any mount, system will behave
exactly like today's DFS. If there are mount points, vfs functionality will
come under DFS.
I have a patch and will post it in some time.
was:
I have been working on a thought from last week is that, we wanted to provide
DFS compatible APIs with mount functionality. So, that existing DFS
applications can work with out class cast issues.
When we tested with other components like Hive and HBase, I noticed some
classcast issues.
{code:java}
HBase example:
java.lang.ClassCastException:
org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
org.apache.hadoop.hdfs.DistributedFileSystemjava.lang.ClassCastException:
org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
org.apache.hadoop.hdfs.DistributedFileSystem at
org.apache.hadoop.hbase.util.FSUtils.getDFSHedgedReadMetrics(FSUtils.java:1748)
at
org.apache.hadoop.hbase.regionserver.MetricsRegionServerWrapperImpl.(MetricsRegionServerWrapperImpl.java:146)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1594)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1001)
at java.lang.Thread.run(Thread.java:748){code}
{code:java}
Hive:
|io.AcidUtils|: Failed to get files with ID; using regular API: Only supported
for DFS; got class
org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme{code}
SO, here the implementation details are like follows:
We extended DistributedFileSystem and created a class called "
ViewDistributedFileSystem"
This vfs=ViewFirstibutedFileSystem, try to initialize
ViewFileSystemOverloadScheme. If success call will delegate to vfs. If fails
to initialize due to no mount points, or other errors, it will just fallback to
regular DFS init. If users does not configure any mount, system will behave
exactly like today's DFS. If there are mount points, vfs functionality will
come under DFS.
I will a patch and will post it in some time.
> Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside
> -
>
> Key: HDFS-15533
> URL: https://issues.apache.org/jira/browse/HDFS-15533
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: dfs, viewfs
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
>
> I have been working on a thought from last week is that, we wanted to provide
> DFS compatible APIs with mount functionality. So, that existing DFS
> applications can work with out class cast issues.
> When we tested with other components like Hive and HBase, I noticed some
> classcast issues.
> {code:java}
> HBase example:
> java.lang.ClassCastException:
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
> org.apache.hadoop.hdfs.DistributedFileSystemjava.lang.ClassCastException:
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme
[jira] [Commented] (HDFS-15533) Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside
[
https://issues.apache.org/jira/browse/HDFS-15533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17178124#comment-17178124
]
Uma Maheswara Rao G commented on HDFS-15533:
[~ayushtkn], This is specifically for HDFS users as HDFS specific APIs are used
directly in components like Hive/Hbase. Example, encryption APIs are specific
to HDFS only and application used them. So, when HDFS want to use
ViewFileSystemOverloadScheme, we can not avoid class cast or API compat issues
easily. This is an attempt for HDFS users to use ViewFileSystemOverloadScheme
via ViewDistributedFileSystem to get API compatibility. For other fs users can
continue to use ViewFileSystemOverloadScheme, as they might not have many
local implementations like in HDFS. Currently we have two key
features(Encryption, EC) in HDFS, which APIs were not added in FileSystem
interface.
Eventually we can bring such key feature APIs into FileSystem if there is an
interest. However, that will take some time to make upstream projects try and
integrate such newly added APIs ( they might need to mess-up with reflection to
check API availability).
I have updated PR to demonstrate the idea. Please take a look at it.
> Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside
> -
>
> Key: HDFS-15533
> URL: https://issues.apache.org/jira/browse/HDFS-15533
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: dfs, viewfs
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
>
> I have been working on a thought from last week is that, we wanted to provide
> DFS compatible APIs with mount functionality. So, that existing DFS
> applications can work with out class cast issues.
> When we tested with other components like Hive and HBase, I noticed some
> classcast issues.
> {code:java}
> HBase example:
> java.lang.ClassCastException:
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
> org.apache.hadoop.hdfs.DistributedFileSystemjava.lang.ClassCastException:
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
> org.apache.hadoop.hdfs.DistributedFileSystem at
> org.apache.hadoop.hbase.util.FSUtils.getDFSHedgedReadMetrics(FSUtils.java:1748)
> at
> org.apache.hadoop.hbase.regionserver.MetricsRegionServerWrapperImpl.(MetricsRegionServerWrapperImpl.java:146)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1594)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1001)
> at java.lang.Thread.run(Thread.java:748){code}
> {code:java}
> Hive:
> |io.AcidUtils|: Failed to get files with ID; using regular API: Only
> supported for DFS; got class
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme{code}
> SO, here the implementation details are like follows:
> We extended DistributedFileSystem and created a class called "
> ViewDistributedFileSystem"
> This vfs=ViewFirstibutedFileSystem, try to initialize
> ViewFileSystemOverloadScheme. If success call will delegate to vfs. If fails
> to initialize due to no mount points, or other errors, it will just fallback
> to regular DFS init. If users does not configure any mount, system will
> behave exactly like today's DFS. If there are mount points, vfs functionality
> will come under DFS.
> I will a patch and will post it in some time.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15329) Provide FileContext based ViewFSOverloadScheme implementation
[ https://issues.apache.org/jira/browse/HDFS-15329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177370#comment-17177370 ] Uma Maheswara Rao G edited comment on HDFS-15329 at 8/14/20, 8:23 AM: -- Thanks [~abhishekd] I was trying to assign this issue to you, but somehow I am not seeing ur name getting listed there. If you are getting "Assign to me" option, feel free to assign it urself. was (Author: umamaheswararao): Thanks [~abhishekd] I was trying to assign this issue to you, but somehow I am seeing ur name getting listed there. If you are getting "Assign to me" option, feel free to assign it urself. > Provide FileContext based ViewFSOverloadScheme implementation > - > > Key: HDFS-15329 > URL: https://issues.apache.org/jira/browse/HDFS-15329 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs, hdfs, viewfs, viewfsOverloadScheme >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Priority: Major > > This Jira to track for FileContext based ViewFSOverloadScheme implementation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15533) Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside
[
https://issues.apache.org/jira/browse/HDFS-15533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-15533:
---
Summary: Provide DFS API compatible class, but use
ViewFileSystemOverloadScheme inside (was: Provide DFS API compatible call, but
use ViewFileSystemOverloadScheme inside)
> Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside
> -
>
> Key: HDFS-15533
> URL: https://issues.apache.org/jira/browse/HDFS-15533
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: dfs, viewfs
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
>
> I have been working on a thought from last week is that, we wanted to provide
> DFS compatible APIs with mount functionality. So, that existing DFS
> applications can work with out class cast issues.
> When we tested with other components like Hive and HBase, I noticed some
> classcast issues.
> {code:java}
> HBase example:
> java.lang.ClassCastException:
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
> org.apache.hadoop.hdfs.DistributedFileSystemjava.lang.ClassCastException:
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
> org.apache.hadoop.hdfs.DistributedFileSystem at
> org.apache.hadoop.hbase.util.FSUtils.getDFSHedgedReadMetrics(FSUtils.java:1748)
> at
> org.apache.hadoop.hbase.regionserver.MetricsRegionServerWrapperImpl.(MetricsRegionServerWrapperImpl.java:146)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1594)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1001)
> at java.lang.Thread.run(Thread.java:748){code}
> {code:java}
> Hive:
> |io.AcidUtils|: Failed to get files with ID; using regular API: Only
> supported for DFS; got class
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme{code}
> SO, here the implementation details are like follows:
> We extended DistributedFileSystem and created a class called "
> ViewDistributedFileSystem"
> This vfs=ViewFirstibutedFileSystem, try to initialize
> ViewFileSystemOverloadScheme. If success call will delegate to vfs. If fails
> to initialize due to no mount points, or other errors, it will just fallback
> to regular DFS init. If users does not configure any mount, system will
> behave exactly like today's DFS. If there are mount points, vfs functionality
> will come under DFS.
> I will a patch and will post it in some time.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15533) Provide DFS API compatible call, but use ViewFileSystemOverloadScheme inside
Uma Maheswara Rao G created HDFS-15533:
--
Summary: Provide DFS API compatible call, but use
ViewFileSystemOverloadScheme inside
Key: HDFS-15533
URL: https://issues.apache.org/jira/browse/HDFS-15533
Project: Hadoop HDFS
Issue Type: Sub-task
Components: dfs, viewfs
Affects Versions: 3.4.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
I have been working on a thought from last week is that, we wanted to provide
DFS compatible APIs with mount functionality. So, that existing DFS
applications can work with out class cast issues.
When we tested with other components like Hive and HBase, I noticed some
classcast issues.
{code:java}
HBase example:
java.lang.ClassCastException:
org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
org.apache.hadoop.hdfs.DistributedFileSystemjava.lang.ClassCastException:
org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
org.apache.hadoop.hdfs.DistributedFileSystem at
org.apache.hadoop.hbase.util.FSUtils.getDFSHedgedReadMetrics(FSUtils.java:1748)
at
org.apache.hadoop.hbase.regionserver.MetricsRegionServerWrapperImpl.(MetricsRegionServerWrapperImpl.java:146)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1594)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1001)
at java.lang.Thread.run(Thread.java:748){code}
{code:java}
Hive:
|io.AcidUtils|: Failed to get files with ID; using regular API: Only supported
for DFS; got class
org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme{code}
SO, here the implementation details are like follows:
We extended DistributedFileSystem and created a class called "
ViewDistributedFileSystem"
This vfs=ViewFirstibutedFileSystem, try to initialize
ViewFileSystemOverloadScheme. If success call will delegate to vfs. If fails
to initialize due to no mount points, or other errors, it will just fallback to
regular DFS init. If users does not configure any mount, system will behave
exactly like today's DFS. If there are mount points, vfs functionality will
come under DFS.
I will a patch and will post it in some time.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15329) Provide FileContext based ViewFSOverloadScheme implementation
[ https://issues.apache.org/jira/browse/HDFS-15329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177370#comment-17177370 ] Uma Maheswara Rao G commented on HDFS-15329: Thanks [~abhishekd] I was trying to assign this issue to you, but somehow I am seeing ur name getting listed there. If you are getting "Assign to me" option, feel free to assign it urself. > Provide FileContext based ViewFSOverloadScheme implementation > - > > Key: HDFS-15329 > URL: https://issues.apache.org/jira/browse/HDFS-15329 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs, hdfs, viewfs, viewfsOverloadScheme >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Priority: Major > > This Jira to track for FileContext based ViewFSOverloadScheme implementation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15329) Provide FileContext based ViewFSOverloadScheme implementation
[ https://issues.apache.org/jira/browse/HDFS-15329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G reassigned HDFS-15329: -- Assignee: (was: Uma Maheswara Rao G) > Provide FileContext based ViewFSOverloadScheme implementation > - > > Key: HDFS-15329 > URL: https://issues.apache.org/jira/browse/HDFS-15329 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs, hdfs, viewfs, viewfsOverloadScheme >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Priority: Major > > This Jira to track for FileContext based ViewFSOverloadScheme implementation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15329) Provide FileContext based ViewFSOverloadScheme implementation
[ https://issues.apache.org/jira/browse/HDFS-15329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G reassigned HDFS-15329: -- Assignee: Uma Maheswara Rao G > Provide FileContext based ViewFSOverloadScheme implementation > - > > Key: HDFS-15329 > URL: https://issues.apache.org/jira/browse/HDFS-15329 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs, hdfs, viewfs, viewfsOverloadScheme >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > > This Jira to track for FileContext based ViewFSOverloadScheme implementation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15329) Provide FileContext based ViewFSOverloadScheme implementation
[ https://issues.apache.org/jira/browse/HDFS-15329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G reassigned HDFS-15329: -- Assignee: (was: Uma Maheswara Rao G) > Provide FileContext based ViewFSOverloadScheme implementation > - > > Key: HDFS-15329 > URL: https://issues.apache.org/jira/browse/HDFS-15329 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs, hdfs, viewfs, viewfsOverloadScheme >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Priority: Major > > This Jira to track for FileContext based ViewFSOverloadScheme implementation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15532) listFiles on root will fail if fallback root has file
Uma Maheswara Rao G created HDFS-15532:
--
Summary: listFiles on root will fail if fallback root has file
Key: HDFS-15532
URL: https://issues.apache.org/jira/browse/HDFS-15532
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
listFiles implementation gets the RemoteIterator created in InternalViewFSDirFs
as the root is an InternalViewFSDir.
If there is a fallback and a file exist at root level, it would have collected
when collecting locatedStatuses.
When its iterating over to that fallbacks file from RemoteIterator (which was
returned from InternalViewFSDirFs ), iterator's next will will call
getFileBlockLocations if it's a file.
{code:java}
@Override
public LocatedFileStatus next() throws IOException {
System.out.println(this);
if (!hasNext()) {
throw new NoSuchElementException("No more entries in " + f);
}
FileStatus result = stats[i++];
// for files, use getBlockLocations(FileStatus, int, int) to avoid
// calling getFileStatus(Path) to load the FileStatus again
BlockLocation[] locs = result.isFile() ?
getFileBlockLocations(result, 0, result.getLen()) :
null;
return new LocatedFileStatus(result, locs);
}{code}
this getFileBlockLocations will be made on InternalViewFSDirFs, as that
Iterator created originally from that fs.
InternalViewFSDirFs#getFileBlockLocations does not handle fallback cases. It's
always expecting "/", this means it always assuming the dir.
But with the fallback and returning Iterator from InternalViewFSDirFs, will
create problems.
Probably we need to handle fallback case in getFileBlockLocations as well.(
Fallback only should be the reason for call coming to InternalViewFSDirFs with
other than "/")
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15529) getChildFilesystems should include fallback fs as well
Uma Maheswara Rao G created HDFS-15529: -- Summary: getChildFilesystems should include fallback fs as well Key: HDFS-15529 URL: https://issues.apache.org/jira/browse/HDFS-15529 Project: Hadoop HDFS Issue Type: Sub-task Components: viewfs, viewfsOverloadScheme Affects Versions: 3.4.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Currently getChildSystems API used by many other APIs, like getAdditionalTokenIssuers, getTrashRoots etc. If fallBack filesystem not included in child filesystems, Application like YARN who uses getAdditionalTokenIssuers, would not get delegation tokens for fallback fs. This would be a critical bug for secure clusters. Similarly, trashRoots. when applications tried to use trashRoots, it will not considers trash folders from fallback. So, it will leak from cleanup logics. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15515) mkdirs on fallback should throw IOE out instead of suppressing and returning false
[
https://issues.apache.org/jira/browse/HDFS-15515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G resolved HDFS-15515.
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Target Version/s: 3.4.0
Resolution: Fixed
> mkdirs on fallback should throw IOE out instead of suppressing and returning
> false
> --
>
> Key: HDFS-15515
> URL: https://issues.apache.org/jira/browse/HDFS-15515
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Fix For: 3.4.0
>
>
> Currently when doing mkdirs on fallback dir, we catching IOE and returning
> false.
> I think we should just throw IOE out as the fs#mkdirs throws IOE out.
> I noticed a case when we attempt to create .reserved dirs, NN throws
> HadoopIAE.
> But we will catch and return false. Here exception should be thrown out.
> {code:java}
> try {
> return linkedFallbackFs.mkdirs(dirToCreate, permission);
> } catch (IOException e) {
> if (LOG.isDebugEnabled()) {
> StringBuilder msg =
> new StringBuilder("Failed to create ").append(dirToCreate)
> .append(" at fallback : ")
> .append(linkedFallbackFs.getUri());
> LOG.debug(msg.toString(), e);
> }
> return false;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15515) mkdirs on fallback should throw IOE out instead of suppressing and returning false
[
https://issues.apache.org/jira/browse/HDFS-15515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17174953#comment-17174953
]
Uma Maheswara Rao G commented on HDFS-15515:
[~ste...@apache.org] , I noticed ur comments above. Thank you for the comments.
[~ayushtkn] , [~ste...@apache.org] , [~Hemanth Boyina] , thanks for review!.
> mkdirs on fallback should throw IOE out instead of suppressing and returning
> false
> --
>
> Key: HDFS-15515
> URL: https://issues.apache.org/jira/browse/HDFS-15515
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
>
> Currently when doing mkdirs on fallback dir, we catching IOE and returning
> false.
> I think we should just throw IOE out as the fs#mkdirs throws IOE out.
> I noticed a case when we attempt to create .reserved dirs, NN throws
> HadoopIAE.
> But we will catch and return false. Here exception should be thrown out.
> {code:java}
> try {
> return linkedFallbackFs.mkdirs(dirToCreate, permission);
> } catch (IOException e) {
> if (LOG.isDebugEnabled()) {
> StringBuilder msg =
> new StringBuilder("Failed to create ").append(dirToCreate)
> .append(" at fallback : ")
> .append(linkedFallbackFs.getUri());
> LOG.debug(msg.toString(), e);
> }
> return false;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15515) mkdirs on fallback should throw IOE out instead of suppressing and returning false
Uma Maheswara Rao G created HDFS-15515:
--
Summary: mkdirs on fallback should throw IOE out instead of
suppressing and returning false
Key: HDFS-15515
URL: https://issues.apache.org/jira/browse/HDFS-15515
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Currently when doing mkdirs on fallback dir, we catching IOE and returning
false.
I think we should just throw IOE out as the fs#mkdirs throws IOE out.
I noticed a case when we attempt to create .reserved dirs, NN throws HadoopIAE.
But we will catch and return false. Here exception should be thrown out.
{code:java}
try {
return linkedFallbackFs.mkdirs(dirToCreate, permission);
} catch (IOException e) {
if (LOG.isDebugEnabled()) {
StringBuilder msg =
new StringBuilder("Failed to create ").append(dirToCreate)
.append(" at fallback : ")
.append(linkedFallbackFs.getUri());
LOG.debug(msg.toString(), e);
}
return false;
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15329) Provide FileContext based ViewFSOverloadScheme implementation
[ https://issues.apache.org/jira/browse/HDFS-15329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167508#comment-17167508 ] Uma Maheswara Rao G commented on HDFS-15329: HI [~abhishekd], sorry for late reply. I missed ur comment. Please take it up if you wanted to work on. I can help you on reviews! Thanks a lot, for offering help. I was spending some time in integrating ViewFileSystemOverloadScheme with Hive. So far, I figured out issues that: with encryption zones enables, we/Hive may need some handlings as they are creating HDFS specific shims. Looks like ORC also using fileIds specific APIs from HDFS. Except that basic queries passed. How about in your case integration stuff? any components enabled with ViewFileSystemOverloadScheme? Just curious to know :-) > Provide FileContext based ViewFSOverloadScheme implementation > - > > Key: HDFS-15329 > URL: https://issues.apache.org/jira/browse/HDFS-15329 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs, hdfs, viewfs, viewfsOverloadScheme >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > > This Jira to track for FileContext based ViewFSOverloadScheme implementation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15444) mkdir should not create dir in fallback if the dir already in mount Path
[ https://issues.apache.org/jira/browse/HDFS-15444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167506#comment-17167506 ] Uma Maheswara Rao G commented on HDFS-15444: [~jianghuazhu], thanks you for the question. It's already covered in mkdirs of ViewFileSystem specific [implementation | https://github.com/apache/hadoop/blob/5d8600e80ad7864b332b60d5a01585fdf00848ee/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java#L1429]. It's should be an issue with mkdir of ViewFs.java. The scenario is: mount link /a/b/c --> hdfs://nn1/a/b/c fallback --> hdfs://nn1/ now if you try to create hdfs://nn1/a/b, it might end up creating a dir in hdfs://nn1/a/b, It should just return saying dir already exist in mount. because it will check the parent dir first, that is hdfs://nn1/a/. Since /a is an internal dir, it will go to InternalViewFS#mkdir with path as /b to create. Here it should check if internalViewFS /a existing children matching with the path /b. Yes, there is ia path already in mount link /a/b, so it should not create this path in fallback. We did not have that children check in [ViewFs.java|https://github.com/apache/hadoop/blob/5d8600e80ad7864b332b60d5a01585fdf00848ee/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java#L1185] > mkdir should not create dir in fallback if the dir already in mount Path > > > Key: HDFS-15444 > URL: https://issues.apache.org/jira/browse/HDFS-15444 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15478) When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs.
[ https://issues.apache.org/jira/browse/HDFS-15478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-15478: --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) Thank you [~ayushtkn] for review! > When Empty mount points, we are assigning fallback link to self. But it > should not use full URI for target fs. > -- > > Key: HDFS-15478 > URL: https://issues.apache.org/jira/browse/HDFS-15478 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Fix For: 3.4.0 > > > On empty mount tables detection, we will automatically assign fallback with > the same initialized uri fs. Currently we are using given uri for creating > target fs. > When creating target fs, we use Chrooted fs where it will set the path from > uri as base directory. So, this can make path wrong in the case of fs > initialized with path. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
