[jira] [Created] (HDFS-11385) DistCp CopyCommitter should issue warning instead of throw exception when failed to delete file.
Yongjun Zhang created HDFS-11385:
Summary: DistCp CopyCommitter should issue warning instead of
throw exception when failed to delete file.
Key: HDFS-11385
URL: https://issues.apache.org/jira/browse/HDFS-11385
Project: Hadoop HDFS
Issue Type: Bug
Components: distcp
Reporter: Yongjun Zhang
CopyCommiter#deleteMissing(Configuration conf) does the following in a loop for
all files that need to be deleted:
{code}
if (result) {
LOG.info("Deleted " + trgtFileStatus.getPath() + " - Missing at
source");
deletedEntries++;
} else {
throw new IOException("Unable to delete " + trgtFileStatus.getPath());
}
{code}
If for some reason there is a failure to delete a file, then an exception is
thrown and no other files will be deleted.
It seems more reasonable to issue a warning here instead of throw an exception.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/216/ [Jan 12, 2017 3:37:49 PM] (Arun Suresh) YARN-5646. Addendum patch to include missing changes to site.xml. [Jan 13, 2017 12:52:29 PM] (sunilg) YARN-6081. LeafQueue#getTotalPendingResourcesConsideringUserLimit should [Jan 13, 2017 7:46:30 PM] (weichiu) HDFS-11306. Print remaining edit logs from buffer if edit log can't be [Jan 14, 2017 6:38:46 AM] (aengineer) HDFS-11344. The default value of the setting [Jan 15, 2017 6:02:11 AM] (lei) HADOOP-13928. TestAdlFileContextMainOperationsLive.testGetFileContext1 [Jan 16, 2017 7:11:53 AM] (lei) Revert "HDFS-11259. Update fsck to display maintenance state info. [Jan 16, 2017 9:45:22 PM] (arp) HDFS-11342. Fix FileInputStream leak in loadLastPartialChunkChecksum. [Jan 16, 2017 10:43:29 PM] (arp) HDFS-11339. Support File IO sampling for Datanode IO profiling hooks. [Jan 16, 2017 10:53:53 PM] (jitendra) HDFS-11307. The rpc to portmap service for NFS has hardcoded timeout. [Jan 17, 2017 12:20:24 AM] (junping_du) YARN-6011. Add a new web service to list the files on a container in [Jan 17, 2017 1:10:23 AM] (aajisaka) HADOOP-13933. Add haadmin -getAllServiceState option to get the HA state [Jan 17, 2017 7:05:28 AM] (arp) HDFS-11339. Fix import. [Jan 17, 2017 8:23:51 AM] (aajisaka) HADOOP-13989. Fix typo in hadoop-client shade configuration. Contributed [Jan 17, 2017 8:46:00 AM] (bibinchundatt) YARN-6057. yarn.scheduler.minimum-allocation-* descriptions are [Jan 17, 2017 8:56:10 AM] (aajisaka) MAPREDUCE-6819. Replace UTF8 with Text in MRBench. Contributed by Peter [Jan 17, 2017 4:01:42 PM] (jlowe) MAPREDUCE-6831. Flaky test TestJobImpl.testKilledDuringKillAbort. [Jan 17, 2017 8:55:47 PM] (templedf) YARN-6071. Fix incompatible API change on AM-RM protocol due to [Jan 17, 2017 9:10:24 PM] (kihwal) HADOOP-13976. Path globbing does not match newlines. Contributed by Eric [Jan 17, 2017 9:56:06 PM] (wang) HADOOP-13978. Update project release notes for 3.0.0-alpha2. Contributed [Jan 17, 2017 10:33:26 PM] (xyao) HDFS-11209. SNN can't checkpoint when rolling upgrade is not finalized. [Jan 17, 2017 10:48:03 PM] (subru) YARN-6016. Fix minor bugs in handling of local AMRMToken in AMRMProxy. [Jan 18, 2017 12:23:38 AM] (wang) Revert "HADOOP-13989. Fix typo in hadoop-client shade configuration. [Jan 18, 2017 1:01:31 AM] (kasha) YARN-5831. FairScheduler: Propagate allowPreemptionFrom flag all the way [Jan 18, 2017 8:54:33 AM] (aajisaka) HDFS-11290. TestFSNameSystemMBean should wait until JMX cache is [Jan 18, 2017 9:11:33 AM] (aajisaka) HADOOP-13955. Replace deprecated HttpServer2 and SSLFactory constants. [Jan 18, 2017 8:46:32 PM] (shv) HDFS-10733. NameNode terminated after full GC thinking QJM is [Jan 18, 2017 9:31:05 PM] (wang) HDFS-10759. Change fsimage bool isStriped from boolean to an enum. [Jan 18, 2017 9:31:33 PM] (wangda) YARN-5556. CapacityScheduler: Support deleting queues without requiring [Jan 18, 2017 10:35:51 PM] (aw) HADOOP-13964. Remove vestigal templates directories [Jan 18, 2017 10:39:05 PM] (aw) HADOOP-13673. Update scripts to be smarter when running with privilege [Jan 18, 2017 11:18:48 PM] (wang) HADOOP-13996. Fix some release build issues. [Jan 19, 2017 1:42:56 AM] (yqlin) HADOOP-13965. Groups should be consistent in using default group mapping [Jan 19, 2017 4:05:13 AM] (yqlin) HDFS-11316. [Jan 19, 2017 7:38:20 AM] (lei) HDFS-11287. Storage class member storageDirs should be private to avoid [Jan 19, 2017 8:24:11 AM] (wang) HDFS-110268. Correctly reconstruct erasure coding file from FSImage. [Jan 19, 2017 8:28:31 AM] (lei) HDFS-11259. Update fsck to display maintenance state info. (Manoj [Jan 19, 2017 8:56:39 AM] (aajisaka) HADOOP-14001. Improve delegation token validity checking. [Jan 19, 2017 11:25:40 AM] (stevel) HADOOP-13496. Include file lengths in Mismatch in length error for [Jan 19, 2017 11:49:40 AM] (stevel) HADOOP-13999 Add -DskipShade maven profile to disable jar shading to [Jan 19, 2017 2:14:28 PM] (Arun Suresh) YARN-6110. Fix opportunistic containers documentation. (Akira Ajisaka [Jan 19, 2017 6:18:59 PM] (jianhe) YARN-6104. RegistrySecurity overrides zookeeper sasl system properties. [Jan 19, 2017 8:45:17 PM] (templedf) YARN-6028. Add document for container metrics (Contributed by Weiwei [Jan 19, 2017 11:50:07 PM] (wang) Preparing for 3.0.0-alpha3 development [Jan 20, 2017 1:32:25 AM] (arp) HADOOP-14002. Document -DskipShade property in BUILDING.txt. Contributed [Jan 20, 2017 4:52:55 AM] (sjlee) YARN-5928. Move ATSv2 HBase backend code into a new module that is only [Jan 20, 2017 6:31:15 AM] (mingma) HDFS-11296. Maintenance state expiry should be an epoch time and not jvm [Jan 20, 2017 6:48:04 AM] (lei) HADOOP-13956. Read ADLS credentials from Credential Provider. (John [Jan 20, 2017 6:36:31 PM] (wang) HADOOP-14004. Missing hadoop-cloud-storage-project module in pom.xml. [Jan 20, 2017 6:39:38 PM] (wang) Update straggler 3
Re: Problems compiling hadoop trunk on windows
Glad to hear that it's worked for you.. I think problem with VS2015 Or .NET framework.But I never tried with VS2015. Compilation can be success with VS2010 and SDK7.1 (.NET 4.0). Thanks And Regards Brahma Reddy Battula From: Arun Suresh Sent: Wednesday, February 1, 2017 9:37 AM To: Brahma Reddy Battula Cc: Hadoop Common; yarn-...@hadoop.apache.org; Hdfs-dev Subject: Re: Problems compiling hadoop trunk on windows Thanks Brahma.. That worked.. unfortunately, now the compile complains that I do not have the right sdk, even though I have VS 2015, win SDK 8.1 and SDK 7.1 installed. Cheers -Arun On Sun, Jan 29, 2017 at 7:31 PM, Brahma Reddy Battula < brahmareddy.batt...@huawei.com> wrote: > It might be problem with maven repository, What's your maven repository > path..? > > May be you can change to shorter path and check. > > > > Regards > Brahma Reddy Battula > > -Original Message- > From: Arun Suresh [mailto:asur...@apache.org] > Sent: 29 January 2017 03:15 > To: Hadoop Common; yarn-...@hadoop.apache.org; Hdfs-dev > Subject: Problems compiling hadoop trunk on windows > > Hi > > I was wondering if folks who compile Hadoop trunk on windows regularly > have hit this issue: > > [INFO] --- native-maven-plugin:1.0-alpha-8:javah (default) @ hadoop-common > --- > [INFO] cmd.exe /X /C ""C:\Program Files\Java\jdk1.8.0_121\bin\javah" -d > C:\Users\arsuresh\stuff\hadoop\hadoop-common-project\ > hadoop-common\target\native\javah > -classpath org.apache.hadoop.util.NativeCrc32" > > > *The command line is too long.* > [INFO] > > [INFO] Reactor Summary: > [INFO] > [INFO] Apache Hadoop Main . SUCCESS [ > 1.232 s] > . > . > INFO] Apache Hadoop Common ... FAILURE [ > 20.245 s] . > . > [ERROR] Failed to execute goal > org.codehaus.mojo:native-maven-plugin:1.0-alpha-8:javah (default) on > project hadoop-common: Error running javah command: Error executing command > line. Exit code:1 -> [Help 1] > > It looks like the arguments passed to 'javah' causes the command line to > grow too long. I tried moving the root dir to C:\hadoopo, but that doesnt > work either. Is there some config I need to set before compiling ? > > Cheers > -Arun >
[jira] [Created] (HDFS-11384) Add option for balancer to disperse getBlocks calls to avoid NameNode's rpc.CallQueueLength spike
yunjiong zhao created HDFS-11384: Summary: Add option for balancer to disperse getBlocks calls to avoid NameNode's rpc.CallQueueLength spike Key: HDFS-11384 URL: https://issues.apache.org/jira/browse/HDFS-11384 Project: Hadoop HDFS Issue Type: Improvement Components: balancer & mover Affects Versions: 2.7.3 Reporter: yunjiong zhao Assignee: yunjiong zhao When running balancer on hadoop cluster which have more than 3000 Datanodes will cause NameNode's rpc.CallQueueLength spike. We observed this situation could cause Hbase cluster failure due to RegionServer's WAL timeout. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/304/ [Jan 31, 2017 8:48:31 AM] (sunilg) YARN-5866. Fix few issues reported by jshint in new YARN UI. Contributed [Jan 31, 2017 7:38:34 PM] (wang) HADOOP-14018. shaded jars of hadoop-client modules are missing hadoop's [Feb 1, 2017 12:25:33 AM] (templedf) YARN-4658. Typo in [Feb 1, 2017 6:42:51 AM] (lei) HDFS-11335. Remove -1 overall The following subsystems voted -1: asflicense unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.security.TestKDiag hadoop.hdfs.TestEncryptionZones hadoop.yarn.server.timeline.webapp.TestTimelineWebServices hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.TestMiniYarnClusterNodeUtilization cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/304/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/304/artifact/out/diff-compile-javac-root.txt [168K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/304/artifact/out/diff-checkstyle-root.txt [16M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/304/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/304/artifact/out/diff-patch-shellcheck.txt [24K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/304/artifact/out/diff-patch-shelldocs.txt [16K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/304/artifact/out/whitespace-eol.txt [11M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/304/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/304/artifact/out/diff-javadoc-javadoc-root.txt [2.2M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/304/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [132K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/304/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [244K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/304/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/304/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [60K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/304/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [324K] asflicense: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/304/artifact/out/patch-asflicense-problems.txt [4.0K] Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
