[
https://issues.apache.org/jira/browse/IGNITE-4341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722519#comment-15722519
]
Ivan Veselovsky commented on IGNITE-4341:
-
2 problems were solved while incorporating this test:
1) distributed cache mechanism was not working if the Hadoop job submitted with
grid().hadoop().submit(); This is related to the fact that real Hadoop client
sets job property {code}MRJobConfig.MAPREDUCE_JOB_DIR{code} , while test job
submission does not. As a result,
org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2JobResourceManager#prepareJobEnvironment
method did not copy the distributed resources to the local directories, while
it should: the "mapreduce.job.dir" property only defines the staging directory,
it is not directly related to distributed caching mechanism.
2) Special local file system implementation
(org.apache.ignite.internal.processors.hadoop.impl.fs.HadoopLocalFileSystemV1)
used in most of Hadoop tests, while original
org.apache.hadoop.fs.LocalFileSystem is used in terasort test. But since the
instance of file system is cached in FileSystem.CACHE, an instance from
previous tests were actually used. This is fixed by adding
org.apache.ignite.internal.processors.hadoop.impl.fs.HadoopFileSystemsUtils#clearFileSystemCache
method and invoking it from #beforeTestsStarted and #beforeTest in Hadoop test
suites.
> Add TeraSort example as a unit test to Ignite
> -
>
> Key: IGNITE-4341
> URL: https://issues.apache.org/jira/browse/IGNITE-4341
> Project: Ignite
> Issue Type: Test
> Components: hadoop
>Affects Versions: 1.7
>Reporter: Ivan Veselovsky
>Assignee: Ivan Veselovsky
> Fix For: 2.0
>
>
> Add canonical TeraSort example as a unit test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
