[
https://issues.apache.org/jira/browse/HADOOP-5916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan updated HADOOP-5916:
--------------------------------
Description:
Currently when the test.build.data property is not set, as happens when run
through some configurations of Eclipse, the fall-back value varies wildly.
Most calls set to /tmp, which is not good as it is beyond the scope of the ant
clean task and thus will not be deleted. Others default to "." which can drop
the test files right in the current directory. Speaking with Konstanin, it
seems the correct location should be "build/test/data" to ensure any files that
are created are within the scope of Ant's clean command.
Attached is the current variation in this setting.
was:
Currently when the test.build.data property is not set, as happens when run
through some configurations of Eclipse, the fall-back value varies wildly.
Most calls set to /tmp, which is not good as it is beyond the scope of the ant
clean task and thus will not be deleted. Others default to "." which can drop
the test files right in the current directory. Speaking with Konstanin, it
seems the correct location should be "build/test/data" to ensure any files that
are created are within the scope of Ant's clean command.
Current variation of setting the test.build.data:
{noformat}
./contrib/fairscheduler/src/test/org/apache/hadoop/mapred/TestFairScheduler.java:
final static String TEST_DIR = new File(System.getProperty("test.build.data",
./contrib/sqoop/src/java/org/apache/hadoop/sqoop/ImportOptions.java: String
tmpDir = System.getProperty("test.build.data", "/tmp/");
./contrib/sqoop/src/test/org/apache/hadoop/sqoop/testutil/ImportJobTestCase.java:
String tmpDir = System.getProperty("test.build.data", "/tmp/");
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestAutoInputFormat.java:
Path dir = new Path(System.getProperty("test.build.data", ".") + "/mapred");
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestMultipleArchiveFiles.java:
"-jobconf", "stream.tmpdir=" +
System.getProperty("test.build.data","/tmp")
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestMultipleCachefiles.java:
"-jobconf",
"stream.tmpdir="+System.getProperty("test.build.data","/tmp"),
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestRawBytesStreaming.java:
"-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp"),
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamAggregate.java:
"-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp")
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamDataProtocol.java:
"-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp")
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamedMerge.java:
"-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp")
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreaming.java:
"-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp")
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingAsDifferentUser.java:
"stream.tmpdir=" + System.getProperty("test.build.data", "/tmp") };
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingBadRecords.java:
"-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp")
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingBadRecords.java:
"-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp")
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingCombiner.java:
"-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp")
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingEmptyInpNonemptyOut.java:
"-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp")
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingExitStatus.java:
"-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp")
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingFailure.java:
"-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp")
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingKeyValue.java:
"-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp")
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingSeparator.java:
"-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp"),
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingStderr.java:
"-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp")
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamReduceNone.java:
"-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp")
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestSymLink.java:
"-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp"),
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestTypedBytesStreaming.java:
"-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp"),
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestUlimit.java:
System.getProperty("test.build.data","/tmp")
./contrib/streaming/src/test/org/apache/hadoop/streaming/UtilTest.java:
antTestDir_ = System.getProperty("test.build.data", userDir_);
./contrib/streaming/src/test/org/apache/hadoop/typedbytes/TestIO.java:
this.tmpfile = new File(System.getProperty("test.build.data", "/tmp"),
./examples/org/apache/hadoop/examples/DBCountPageView.java:
System.getProperty("test.build.data",".") + "/URLAccess");
./test/core/org/apache/hadoop/filecache/TestDistributedCache.java: new
Path(System.getProperty("test.build.data","/tmp/cachebasedir"))
./test/core/org/apache/hadoop/filecache/TestDistributedCache.java:
System.getProperty("test.build.data", "/tmp/distributedcache");
./test/core/org/apache/hadoop/fs/kfs/TestKosmosFileSystem.java: baseDir
= new Path(System.getProperty("test.build.data", "/tmp" ) +
./test/core/org/apache/hadoop/fs/TestChecksumFileSystem.java: =
System.getProperty("test.build.data","build/test/data/work-dir/localfs");
./test/core/org/apache/hadoop/fs/TestDFVariations.java: super(new
File(System.getProperty("test.build.data","/tmp")), 0L);
./test/core/org/apache/hadoop/fs/TestDU.java:
System.getProperty("test.build.data","/tmp"), "dutmp");
./test/core/org/apache/hadoop/fs/TestGetFileBlockLocations.java:
System.getProperty("test.build.data", "/tmp/testGetFileBlockLocations");
./test/core/org/apache/hadoop/fs/TestLocalFileSystem.java: =
System.getProperty("test.build.data","build/test/data/work-dir/localfs");
./test/core/org/apache/hadoop/fs/TestTrash.java: new Path(new
File(System.getProperty("test.build.data","/tmp")
./test/core/org/apache/hadoop/fs/TestTruncatedInputBug.java: new
Path(System.getProperty("test.build.data","/tmp"))
./test/core/org/apache/hadoop/io/TestArrayFile.java:
System.getProperty("test.build.data",".") + "/test.array";
./test/core/org/apache/hadoop/io/TestBloomMapFile.java: Path dirName = new
Path(System.getProperty("test.build.data",".") +
./test/core/org/apache/hadoop/io/TestMapFile.java: Path dirName = new
Path(System.getProperty("test.build.data",".") +
./test/core/org/apache/hadoop/io/TestMapFile.java: Path dirName = new
Path(System.getProperty("test.build.data",".") +
./test/core/org/apache/hadoop/io/TestMapFile.java: Path dirName = new
Path(System.getProperty("test.build.data",".") +
./test/core/org/apache/hadoop/io/TestSequenceFileSerialization.java: Path
file = new Path(System.getProperty("test.build.data",".") +
./test/core/org/apache/hadoop/io/TestSetFile.java:
System.getProperty("test.build.data",".") + "/test.set";
./test/hdfs/org/apache/hadoop/fs/loadGenerator/TestLoadGenerator.java:
System.getProperty("test.build.data","build/test/data");
./test/hdfs/org/apache/hadoop/fs/loadGenerator/TestLoadGenerator.java: final
String SCRIPT_TEST_DIR = new File(System.getProperty("test.build.data",
./test/hdfs/org/apache/hadoop/hdfs/MiniDFSCluster.java: return
System.getProperty("test.build.data", "build/test/data") + "/dfs/";
./test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestFsck.java: File
baseDir = new File(System.getProperty("test.build.data",
./test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestFsck.java: File
baseDir = new File(System.getProperty("test.build.data",
./test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestNameEditsConfigs.java:
System.getProperty("test.build.data", "build/test/data"), "dfs/");
./test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestStartup.java: String
baseDir = System.getProperty("test.build.data", "/tmp");
./test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestStorageRestore.java:
String baseDir = System.getProperty("test.build.data", "build/test/data");
./test/hdfs/org/apache/hadoop/hdfs/TestDFSShell.java: new
Path(System.getProperty("test.build.data","/tmp"))
./test/hdfs/org/apache/hadoop/hdfs/TestDFSStartupVersions.java:
System.getProperty("test.build.data","/tmp").toString().replace(' ', '+'));
./test/hdfs/org/apache/hadoop/hdfs/TestDFSUpgradeFromImage.java: String
dataDir = System.getProperty("test.build.data", "build/test/data");
./test/hdfs/org/apache/hadoop/hdfs/TestFileStatus.java: new
Path(System.getProperty("test.build.data","/tmp"))
./test/hdfs/org/apache/hadoop/hdfs/TestFSInputChecker.java: String dir =
System.getProperty("test.build.data", ".");
./test/hdfs/org/apache/hadoop/hdfs/TestHDFSServerPorts.java: return
System.getProperty("test.build.data", "build/test/data");
./test/hdfs/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestDelimitedImageVisitor.java:
private static String ROOT = System.getProperty("test.build.data","/tmp");
./test/hdfs/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java:
private static String ROOT = System.getProperty("test.build.data","/tmp");
./test/hdfs/org/apache/hadoop/hdfs/UpgradeUtilities.java:
System.getProperty("test.build.data","/tmp").replace(' ', '+'));
./test/hdfs-with-mr/org/apache/hadoop/fs/DFSCIOTest.java: private static
String TEST_ROOT_DIR =
System.getProperty("test.build.data","/benchmarks/DFSCIOTest");
./test/hdfs-with-mr/org/apache/hadoop/fs/DistributedFSCheck.java: private
static Path TEST_ROOT_DIR = new
Path(System.getProperty("test.build.data","/benchmarks/DistributedFSCheck"));
./test/hdfs-with-mr/org/apache/hadoop/fs/TestCopyFiles.java: new
Path(System.getProperty("test.build.data","/tmp"))
./test/hdfs-with-mr/org/apache/hadoop/fs/TestDFSIO.java: private static String
TEST_ROOT_DIR = System.getProperty("test.build.data","/benchmarks/TestDFSIO");
./test/hdfs-with-mr/org/apache/hadoop/fs/TestFileSystem.java: private static
String ROOT = System.getProperty("test.build.data","fs_test");
./test/hdfs-with-mr/org/apache/hadoop/io/TestSequenceFileMergeProgress.java:
Path dir = new Path(System.getProperty("test.build.data",".") + "/mapred");
./test/hdfs-with-mr/org/apache/hadoop/tools/TestDistCh.java: new
Path(System.getProperty("test.build.data","/tmp")
./test/mapred/org/apache/hadoop/io/TestSequenceFile.java: Path file = new
Path(System.getProperty("test.build.data",".")+"/test.seq");
./test/mapred/org/apache/hadoop/io/TestSequenceFile.java: new
Path(System.getProperty("test.build.data",".")+"/test.rc.seq");
./test/mapred/org/apache/hadoop/io/TestSequenceFile.java: new
Path(System.getProperty("test.build.data",".")+"/test.bc.seq");
./test/mapred/org/apache/hadoop/io/TestSequenceFile.java: Path file = new
Path(System.getProperty("test.build.data",".")+"/test.seq.metadata");
./test/mapred/org/apache/hadoop/io/TestSequenceFile.java: new
Path(System.getProperty("test.build.data",".")+"/test.sorted.seq.metadata");
./test/mapred/org/apache/hadoop/io/TestSequenceFile.java: new
Path(System.getProperty("test.build.data",".")+"/test.rc.seq.metadata");
./test/mapred/org/apache/hadoop/io/TestSequenceFile.java: new
Path(System.getProperty("test.build.data",".")+"/test.bc.seq.metadata");
./test/mapred/org/apache/hadoop/io/TestSequenceFile.java: Path path1 = new
Path(System.getProperty("test.build.data",".")+"/test1.seq");
./test/mapred/org/apache/hadoop/io/TestSequenceFile.java: Path path2 = new
Path(System.getProperty("test.build.data",".")+"/test2.seq");
./test/mapred/org/apache/hadoop/io/TestSequenceFile.java: Path path = new
Path(System.getProperty("test.build.data",".")+"/broken.seq");
./test/mapred/org/apache/hadoop/mapred/jobcontrol/TestJobControl.java: Path
rootDataDir = new Path(System.getProperty("test.build.data", "."),
"TestJobControlData");
./test/mapred/org/apache/hadoop/mapred/jobcontrol/TestLocalJobControl.java:
Path rootDataDir = new Path(System.getProperty("test.build.data", "."),
./test/mapred/org/apache/hadoop/mapred/join/TestWrappedRecordReaderClassloader.java:
Path testdir = new Path(System.getProperty("test.build.data", "/tmp"))
./test/mapred/org/apache/hadoop/mapred/lib/TestChainMapReduce.java: String
localPathRoot = System.getProperty("test.build.data", "/tmp")
./test/mapred/org/apache/hadoop/mapred/lib/TestChainMapReduce.java: String
localPathRoot = System.getProperty("test.build.data", "/tmp")
./test/mapred/org/apache/hadoop/mapred/lib/TestLineInputFormat.java: new
Path(new Path(System.getProperty("test.build.data", "."), "data"),
./test/mapred/org/apache/hadoop/mapred/lib/TestMultipleOutputs.java:
String localPathRoot = System.getProperty("test.build.data", "/tmp")
./test/mapred/org/apache/hadoop/mapred/lib/TestMultithreadedMapRunner.java:
String localPathRoot = System.getProperty("test.build.data", "/tmp")
./test/mapred/org/apache/hadoop/mapred/MRBench.java: new
Path(System.getProperty("test.build.data","/benchmarks/MRBench"));
./test/mapred/org/apache/hadoop/mapred/MRCaching.java: String TEST_ROOT_DIR
= new Path(System.getProperty("test.build.data","/tmp"))
./test/mapred/org/apache/hadoop/mapred/NotificationTestCase.java: String
localPathRoot = System.getProperty("test.build.data","/tmp")
./test/mapred/org/apache/hadoop/mapred/NotificationTestCase.java: String
localPathRoot = System.getProperty("test.build.data","/tmp")
./test/mapred/org/apache/hadoop/mapred/TestCommandLineJobSubmission.java: File
buildDir = new File(System.getProperty("test.build.data", "/tmp"));
./test/mapred/org/apache/hadoop/mapred/TestEmptyJob.java: new
File(System.getProperty("test.build.data", "/tmp")).toURI()
./test/mapred/org/apache/hadoop/mapred/TestFileInputFormatPathFilter.java:
new Path(new Path(System.getProperty("test.build.data", "."), "data"),
./test/mapred/org/apache/hadoop/mapred/TestFileOutputCommitter.java:
System.getProperty("test.build.data", "."), "output");
./test/mapred/org/apache/hadoop/mapred/TestFileOutputFormat.java: String
localPathRoot = System.getProperty("test.build.data", "/tmp")
./test/mapred/org/apache/hadoop/mapred/TestIndexCache.java: Path p = new
Path(System.getProperty("test.build.data", "/tmp"),
./test/mapred/org/apache/hadoop/mapred/TestIndexCache.java: Path p = new
Path(System.getProperty("test.build.data", "/tmp"),
./test/mapred/org/apache/hadoop/mapred/TestJobHistoryParsing.java: Path
historyDir = new Path(System.getProperty("test.build.data", "."),
./test/mapred/org/apache/hadoop/mapred/TestJobHistoryVersion.java: new
Path(System.getProperty("test.build.data", "/tmp"),
./test/mapred/org/apache/hadoop/mapred/TestJobInProgress.java: new
Path(System.getProperty("test.build.data","/tmp"), "jip-testing");
./test/mapred/org/apache/hadoop/mapred/TestJobQueueInformation.java: new
Path(System.getProperty("test.build.data","/tmp"),
./test/mapred/org/apache/hadoop/mapred/TestJobTrackerRestart.java: new
Path(System.getProperty("test.build.data","/tmp"),
./test/mapred/org/apache/hadoop/mapred/TestJobTrackerSafeMode.java: new
Path(System.getProperty("test.build.data", "/tmp"), "jt-safemode");
./test/mapred/org/apache/hadoop/mapred/TestKeyValueTextInputFormat.java: new
Path(new Path(System.getProperty("test.build.data", "."), "data"),
./test/mapred/org/apache/hadoop/mapred/TestMapRed.java:
System.getProperty("test.build.data","/tmp")).makeQualified(fs);
./test/mapred/org/apache/hadoop/mapred/TestMapRed.java:
System.getProperty("test.build.data","/tmp")).makeQualified(fs);
./test/mapred/org/apache/hadoop/mapred/TestMiniMRLocalFS.java: new
File(System.getProperty("test.build.data","/tmp"))
./test/mapred/org/apache/hadoop/mapred/TestMiniMRMapRedDebugScript.java:
String TEST_ROOT_DIR = new Path(System.getProperty("test.build.data",
./test/mapred/org/apache/hadoop/mapred/TestMiniMRTaskTempDir.java: String
TEST_ROOT_DIR = new Path(System.getProperty("test.build.data",
./test/mapred/org/apache/hadoop/mapred/TestMiniMRWithDFS.java: new
File(System.getProperty("test.build.data","/tmp"))
./test/mapred/org/apache/hadoop/mapred/TestMultiFileInputFormat.java: Path
dir = new Path(System.getProperty("test.build.data",".") + "/mapred");
./test/mapred/org/apache/hadoop/mapred/TestMultipleTextOutputFormat.java:
new Path(System.getProperty("test.build.data", "."),
./test/mapred/org/apache/hadoop/mapred/TestRecoveryManager.java: new
Path(System.getProperty("test.build.data", "/tmp"),
./test/mapred/org/apache/hadoop/mapred/TestSequenceFileAsBinaryInputFormat.java:
Path dir = new Path(System.getProperty("test.build.data",".") + "/mapred");
./test/mapred/org/apache/hadoop/mapred/TestSequenceFileAsBinaryOutputFormat.java:
new Path(new Path(new Path(System.getProperty("test.build.data",".")),
./test/mapred/org/apache/hadoop/mapred/TestSequenceFileAsBinaryOutputFormat.java:
Path dir = new Path(System.getProperty("test.build.data",".") + "/mapred");
./test/mapred/org/apache/hadoop/mapred/TestSequenceFileAsBinaryOutputFormat.java:
Path outputdir = new Path(System.getProperty("test.build.data",".")
./test/mapred/org/apache/hadoop/mapred/TestSequenceFileAsTextInputFormat.java:
Path dir = new Path(System.getProperty("test.build.data",".") + "/mapred");
./test/mapred/org/apache/hadoop/mapred/TestSequenceFileInputFilter.java:
private static final Path inDir = new
Path(System.getProperty("test.build.data",".") + "/mapred");
./test/mapred/org/apache/hadoop/mapred/TestSequenceFileInputFormat.java:
Path dir = new Path(System.getProperty("test.build.data",".") + "/mapred");
./test/mapred/org/apache/hadoop/mapred/TestSpilledRecordsCounter.java:
String TEST_ROOT_DIR = new Path(System.getProperty("test.build.data",
./test/mapred/org/apache/hadoop/mapred/TestTaskFail.java: String
TEST_ROOT_DIR = new Path(System.getProperty("test.build.data",
./test/mapred/org/apache/hadoop/mapred/TestTextInputFormat.java: new
Path(new Path(System.getProperty("test.build.data", "."), "data"),
./test/mapred/org/apache/hadoop/mapred/TestTextOutputFormat.java:
new Path(System.getProperty("test.build.data", "."),
./test/mapred/org/apache/hadoop/mapred/ThreadedMapBenchmark.java: new
Path(System.getProperty("test.build.data",
./test/mapred/org/apache/hadoop/mapreduce/lib/partition/TestTotalOrderPartitioner.java:
final Path testdir = new Path(System.getProperty("test.build.data", "/tmp")
./test/mapred/org/apache/hadoop/mapreduce/TestMapReduceLocal.java: new
Path(System.getProperty("test.build.data","/tmp"));
./test/mapred/org/apache/hadoop/mapreduce/TestValueIterReset.java: new
Path(System.getProperty("test.build.data","/tmp"));
./test/mapred/org/apache/hadoop/record/TestRecordWritable.java: Path dir =
new Path(System.getProperty("test.build.data",".") + "/mapred");
{noformat}
Deleted overly long list of offending statements. Will attach as a file.
> Standardize fall-back value of test.build.data for testing directories
> ----------------------------------------------------------------------
>
> Key: HADOOP-5916
> URL: https://issues.apache.org/jira/browse/HADOOP-5916
> Project: Hadoop Core
> Issue Type: Bug
> Components: test
> Reporter: Jakob Homan
> Attachments: test.build.data.txt
>
>
> Currently when the test.build.data property is not set, as happens when run
> through some configurations of Eclipse, the fall-back value varies wildly.
> Most calls set to /tmp, which is not good as it is beyond the scope of the
> ant clean task and thus will not be deleted. Others default to "." which can
> drop the test files right in the current directory. Speaking with Konstanin,
> it seems the correct location should be "build/test/data" to ensure any files
> that are created are within the scope of Ant's clean command.
> Attached is the current variation in this setting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
