[
https://issues.apache.org/jira/browse/SAMZA-508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Riccomini updated SAMZA-508:
----------------------------------
Description:
SAMZA-468 added support for single-node 'localhost' execution of integration
tests (YARN, Kafka, ZK, and Samza jobs are all deployed locally). Deploying
tests to a single non-local host machine don't work right now because of the
way we're deploying job tarballs. We're using yarn.package.path=file:/... URI
schemes, and SCP'ing the job tarballs to all machines. This is a problem when
executing on a non-local node because YARN stats the local job tarball when
run-job.sh is run, and compares the tarball's
[ctime/mtime/atime|http://www.linux-faqs.info/general/difference-between-mtime-ctime-and-atime]
fields against the remote machines. They don't match since we're SCP'ing them,
so YARN fails the job's deployment.
We'll have to either modify the remote timestamps, or use some other filesystem
(HDFS, HTTP, etc) to deploy the job tarballs to YARN NMs.
This bug can be triggered simply by replacing all instances of `localhost` in
samza-test/src/main/python/config/*json with a different machine's hostname.
was:
SAMZA-468 added support for single-node execution of integration tests.
Multi-node integration tests don't work right now because of the way we're
deploying job tarballs. We're using yarn.package.path=file:/... URI schemes,
and SCP'ing the job tarballs to all machines. This is a problem when executing
on multiple nodes because YARN stats the local job tarball when run-job.sh is
run, and compares the tarball's
[ctime/mtime/atime|http://www.linux-faqs.info/general/difference-between-mtime-ctime-and-atime]
fields against the remote machines. They don't match since we're SCP'ing them,
so YARN fails the job's deployment.
We'll have to either modify the remote timestamps, or use some other filesystem
(HDFS, HTTP, etc) to deploy the job tarballs to YARN NMs.
> Supporting integration test execution on remote machines
> --------------------------------------------------------
>
> Key: SAMZA-508
> URL: https://issues.apache.org/jira/browse/SAMZA-508
> Project: Samza
> Issue Type: Bug
> Components: test
> Reporter: Chris Riccomini
>
> SAMZA-468 added support for single-node 'localhost' execution of integration
> tests (YARN, Kafka, ZK, and Samza jobs are all deployed locally). Deploying
> tests to a single non-local host machine don't work right now because of the
> way we're deploying job tarballs. We're using yarn.package.path=file:/... URI
> schemes, and SCP'ing the job tarballs to all machines. This is a problem when
> executing on a non-local node because YARN stats the local job tarball when
> run-job.sh is run, and compares the tarball's
> [ctime/mtime/atime|http://www.linux-faqs.info/general/difference-between-mtime-ctime-and-atime]
> fields against the remote machines. They don't match since we're SCP'ing
> them, so YARN fails the job's deployment.
> We'll have to either modify the remote timestamps, or use some other
> filesystem (HDFS, HTTP, etc) to deploy the job tarballs to YARN NMs.
> This bug can be triggered simply by replacing all instances of `localhost` in
> samza-test/src/main/python/config/*json with a different machine's hostname.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
