[jira] [Created] (HDFS-13827) Distcp job is failing due to Invalid arguments
Sudhansu Bhuyan created HDFS-13827: -- Summary: Distcp job is failing due to Invalid arguments Key: HDFS-13827 URL: https://issues.apache.org/jira/browse/HDFS-13827 Project: Hadoop HDFS Issue Type: Bug Components: distcp Affects Versions: 2.7.1 Reporter: Sudhansu Bhuyan 18/08/14 18:35:59 ERROR tools.DistCp: Invalid arguments: java.lang.IllegalArgumentException: Neither source file listing nor source paths present at org.apache.hadoop.tools.OptionsParser.parseSourceAndTargetPaths(OptionsParser.java:348) at org.apache.hadoop.tools.OptionsParser.parse(OptionsParser.java:89) at org.apache.hadoop.tools.DistCp.run(DistCp.java:117) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.tools.DistCp.main(DistCp.java:462) Invalid arguments: Neither source file listing nor source paths present usage: distcp OPTIONS [source_path...] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13826) Add a hidden configuration for NameNode to generate fake block locations
Todd Lipcon created HDFS-13826: -- Summary: Add a hidden configuration for NameNode to generate fake block locations Key: HDFS-13826 URL: https://issues.apache.org/jira/browse/HDFS-13826 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Todd Lipcon Assignee: Todd Lipcon In doing testing and benchmarking of the NameNode and dependent systems, it's often useful to be able to use an fsimage provided by some production system in a controlled environment without actually having access to any of the data. For example, while doing some recent work on Apache Impala I was trying to optimize the transmission and storage of block locations and tokens and measure the results based on metadata from a production user. In order to achieve this, it would be useful for the NN to expose a developer-only (undocumented) configuration to generate fake block locations and return them to callers. The "fake" locations should be randomly distributed across a fixed set of fake datanodes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: Hadoop 3.2 Release Plan proposal
Hi All, Thanks for the feedbacks. Inline with earlier communication dated 17th July 2018, I would like to provide some updates. We are approaching previously proposed feature freeze date (Aug 21, about 7 days from today). If any features in branch which are targeted to 3.2.0, please reply to this email thread. Steve has mentioned about the s3 features which will come close to Code Freeze Date (Aug 31st). *Here's an updated 3.2.0 feature status:* 1. Merged & Completed features: - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads Initial cut. - (Uma) HDFS-10285: HDFS Storage Policy Satisfier 2. Features close to finish: - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major patches are all in, only one last patch is in review state. - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler. Close to commit. - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API and CLI. 2 patches are pending which will be closed by Feature freeze date. - (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2. Patch in progress. - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works. - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage. In progress. 3. Tentative features: - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be done before Aug 2018. - (Eric) YARN-7129: Application Catalog for YARN applications. Challenging as more discussions are on-going. *Summary of 3.2.0 issues status:* 39 Blocker and Critical issues [1] are open, I am checking with owners to get status on each of them to get in by Code Freeze date. [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker, Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER BY priority DESC Thanks, Sunil On Fri, Jul 20, 2018 at 8:03 AM Sunil G wrote: > Thanks Subru for the thoughts. > One of the main reason for a major release is to push out critical > features with a faster cadence to the users. If we are pulling more and > more different types of features to a minor release, that branch will > become more destabilized and it may be tough to say that 3.1.2 is stable > that 3.1.1 for eg. We always tend to improve and stabilize features in > subsequent minor release. > For few companies, it makes sense to push out these new features faster to > make a reach to the users. Adding to the point to the backporting issues, I > agree that its a pain and we can workaround that with some git scripts. If > we can make such scripts available to committers, backport will be > seem-less across branches and we can achieve the faster release cadence > also. > > Thoughts? > > - Sunil > > > On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan wrote: > >> Thanks Sunil for volunteering to lead the release effort. I am generally >> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we >> already >> have too many branches to be maintained. I already see many commits are in >> different branches with no apparent rationale, for e.g: 3.1 has commits >> which are absent in 3.0 etc. >> >> Additionally AFAIK 3.x has not been deployed in any major production >> setting so the cost of adding features should be minimal. >> >> Thoughts? >> >> -Subru >> >> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G wrote: >> >> > Thanks Steve, Aaron, Wangda for sharing thoughts. >> > >> > Yes, important changes and features are much needed, hence we will be >> > keeping the door open for them as possible. Also considering few more >> > offline requests from other folks, I think extending the timeframe by >> > couple of weeks makes sense (including a second RC buffer) and this >> should >> > ideally help us to ship this by September itself. >> > >> > Revised dates (I will be updating same in Roadmap wiki as well) >> > >> > - Feature freeze date : all features to merge by August 21, 2018. >> > >> > - Code freeze date : blockers/critical only, no improvements and non >> > blocker/critical >> > >> > bug-fixes August 31, 2018. >> > >> > - Release date: September 15, 2018 >> > >> > Thank Eric and Zian, I think Wangda has already answered your questions. >> > >> > Thanks >> > Sunil >> > >> > >> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan >> wrote: >> > >> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for that. >> > > >> > > To concerns from Steve, >> > > >> > > It is a good idea to keep the door open to get important changes / >> > > features in before cutoff. I would prefer to keep the proposed release >> > date >> > > to make sure things can happen earlier instead of last minute and we >> all >> > > know that releases are always get delayed :). I'm also fine if we want >> > get >> > > another several weeks time. >> > > >> > > Regarding of 3.3 release, I would suggest doing that before >> thanksgiving. >> > > Do you think is it good or too early / late? >> > > >> > > Eric, >> > > >> > > The YARN-8220 will be
[jira] [Created] (HDFS-13825) HDFS Uses very outdated okhttp library
Ben Parker created HDFS-13825: - Summary: HDFS Uses very outdated okhttp library Key: HDFS-13825 URL: https://issues.apache.org/jira/browse/HDFS-13825 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 3.0.3 Reporter: Ben Parker HDFS Client uses okHttp library version 2.7.4 which is two years out of date. [https://mvnrepository.com/artifact/com.squareup.okhttp/okhttp] The updates for this library have been moved to a new package here: [https://mvnrepository.com/artifact/com.squareup.okhttp3/okhttp] This causes dependancy management problems for services that use HDFS. For example trying to use okHttp in code that runs on Amazon EMR gives you Method not found errors due to the new version being kicked out in favour of the one used by HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/868/ [Aug 13, 2018 7:17:52 AM] (msingh) HDDS-308. SCM should identify a container with pending deletes using [Aug 13, 2018 8:52:55 AM] (sunilg) YARN-8561. [Submarine] Initial implementation: Training job submission [Aug 13, 2018 9:32:56 AM] (drankye) HDFS-13668. FSPermissionChecker may throws AIOOE when check inode [Aug 13, 2018 10:57:45 AM] (ewan.higgs) HADOOP-15645. ITestS3GuardToolLocal.testDiffCommand fails if bucket has [Aug 13, 2018 5:08:58 PM] (aengineer) HDDS-346. ozoneShell show the new volume info after updateVolume command [Aug 13, 2018 5:40:31 PM] (xiao) HADOOP-15638. KMS Accept Queue Size default changed from 500 to 128 in [Aug 13, 2018 6:35:19 PM] (arp) HDFS-13823. NameNode UI : "Utilities -> Browse the file system -> open a [Aug 13, 2018 7:47:49 PM] (xyao) HDDS-324. Use pipeline name as Ratis groupID to allow datanode to report [Aug 13, 2018 8:50:00 PM] (eyang) YARN-7417. Remove duplicated code from IndexedFileAggregatedLogsBlock [Aug 13, 2018 11:12:37 PM] (weichiu) HDFS-13813. Exit NameNode if dangling child inode is detected when [Aug 14, 2018 12:36:13 AM] (weichiu) HDFS-13738. fsck -list-corruptfileblocks has infinite loop if user is -1 overall The following subsystems voted -1: asflicense findbugs pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine Unread field:FSBasedSubmarineStorageImpl.java:[line 39] Found reliance on default encoding in org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceJobSubmitter.generateCommandLaunchScript(RunJobParameters, TaskType, Component):in org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceJobSubmitter.generateCommandLaunchScript(RunJobParameters, TaskType, Component): new java.io.FileWriter(File) At YarnServiceJobSubmitter.java:[line 192] org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceJobSubmitter.generateCommandLaunchScript(RunJobParameters, TaskType, Component) may fail to clean up java.io.Writer on checked exception Obligation to clean up resource created at YarnServiceJobSubmitter.java:to clean up java.io.Writer on checked exception Obligation to clean up resource created at YarnServiceJobSubmitter.java:[line 192] is not discharged org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceUtils.getComponentArrayJson(String, int, String) concatenates strings using + in a loop At YarnServiceUtils.java:using + in a loop At YarnServiceUtils.java:[line 72] Failed CTEST tests : test_test_libhdfs_threaded_hdfs_static test_libhdfs_threaded_hdfspp_test_shim_static Failed junit tests : hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapred.TestMRTimelineEventHandling hadoop.tools.util.TestProducerConsumer hadoop.yarn.sls.TestSLSRunner cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/868/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/868/artifact/out/diff-compile-javac-root.txt [328K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/868/artifact/out/diff-checkstyle-root.txt [4.0K] pathlen: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/868/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/868/artifact/out/diff-patch-pylint.txt [24K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/868/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/868/artifact/out/diff-patch-shelldocs.txt [16K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/868/artifact/out/whitespace-eol.txt [9.4M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/868/artifact/out/whitespace-tabs.txt [1.1M] xml: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/868/artifact/out/xml.txt [4.0K] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/868/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-submarine-warnings.html [12K]
[jira] [Resolved] (HDFS-13031) To detect fsimage corruption on the spot
[ https://issues.apache.org/jira/browse/HDFS-13031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Antal resolved HDFS-13031. --- Resolution: Won't Fix Created HDFS-13818 as follow-up Jira for the issue. > To detect fsimage corruption on the spot > > > Key: HDFS-13031 > URL: https://issues.apache.org/jira/browse/HDFS-13031 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs > Environment: >Reporter: Yongjun Zhang >Assignee: Adam Antal >Priority: Major > > Since we fixed HDFS-9406, there are new cases reported from the field that > similar fsimage corruption happens. We need good fsimage + editlogs to replay > to reproduce the corruption. However, usually when the corruption is detected > (at later NN restart), the good fsimage is already deleted. > We need to have a way to detect fsimage corruption on the spot. Currently > what I think we could do is: > # after SNN creates a new fsimage, it spawn a new modified NN process (NN > with some new command line args) to just load the fsimage and do nothing > else. > # If the process failed, the currently running SNN will do either a) backup > the fsimage + editlogs or b) no longer do checkpointing. And it need to > somehow raise a flag to user that the fsimage is corrupt. > In step 2, if we do a, we need to introduce new NN->JN API to backup > editlogs; if we do b, it changes SNN's behavior, and kind of not compatible. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-295) TestCloseContainerByPipeline is failing because of timeout
[ https://issues.apache.org/jira/browse/HDDS-295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee resolved HDDS-295. -- Resolution: Duplicate Fix Version/s: 0.2.1 > TestCloseContainerByPipeline is failing because of timeout > -- > > Key: HDDS-295 > URL: https://issues.apache.org/jira/browse/HDDS-295 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Reporter: Mukul Kumar Singh >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.2.1 > > > The test is failing because the test is timing out waiting for the container > to be closed. > The details are logged at > https://builds.apache.org/job/PreCommit-HDDS-Build/627/testReport/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13824) Number of Dead nodes is not showing in the Overview and Subclusters pages. However Live nodes are relecting properly
Soumyapn created HDFS-13824: --- Summary: Number of Dead nodes is not showing in the Overview and Subclusters pages. However Live nodes are relecting properly Key: HDFS-13824 URL: https://issues.apache.org/jira/browse/HDFS-13824 Project: Hadoop HDFS Issue Type: Bug Components: federation Affects Versions: 3.1.0 Reporter: Soumyapn Attachments: image-2018-08-14-11-47-05-025.png Scenario: Suppose we have 2 nameservices with 3 Datanodes each. If we make 2 DN's down, then the Datanodes page, Live nodes field in Overview and Live in Subclusters page is reflected to 4. But the Deadnodes field in Overview and Subclusters page is showing as 0. It is not reflected. !image-2018-08-14-11-47-05-025.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
