[
https://issues.apache.org/jira/browse/SAMZA-875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215298#comment-15215298
]
Alexey Raga commented on SAMZA-875:
-----------------------------------
Getting exceptions is not the only issue.
It complicates deployments significantly. I can only put my tarball to the
locations that are accessible from within the job itself.
For example, if I use an S3 bucket (and this is what I actually want to do). I
am currently automatically deploying jobs from CI or semi-
manually from some jumphost in my network.
I now need to setup permissions on my Yarn nodes so this bucket is accessible,
I also need to configure my network to be able to reach outside. All this for
getting a modified date of a tarball which is already unpacked and is not
needed otherwise.
It also makes the deployment process hard to comprehend and to explain to
system admins.
Look at that:
- I need to download a tarball to extract a properties file from it and to
unpack deployment scripts
- The properties file within that tarball points to the original location which
is used when the deployment scripts are run
- Scripts are run and the tarball gets downloaded again (perhaps on another
machine?), YARN containers are created and tasks are distributed
- Tasks are reaching to the tarball again (!), this time only to get the
modified time
If at any point the tarball (that I already have) is not reachable - then yes,
I get exceptions.
> Don't try to access the package during containerStart
> -----------------------------------------------------
>
> Key: SAMZA-875
> URL: https://issues.apache.org/jira/browse/SAMZA-875
> Project: Samza
> Issue Type: Bug
> Components: container
> Affects Versions: 0.10.1
> Reporter: Alexey Raga
>
> When the job is submitted using {{run-job.sh}} the package file is given to
> {{YARN}}. The job is the accepted, the container is created, the package is
> unpacked and is ready to execute.
> However, the {{startContainer}} method ({{ContainerUtil:159}}) then tries to
> access the original package file.
> {code}
> try {
> fileStatus =
> packagePath.getFileSystem(yarnConfiguration).getFileStatus(packagePath);
> } catch (IOException ioe) {
> log.error("IO Exception when accessing the package status from the
> filesystem", ioe);
> throw new SamzaException("IO Exception when accessing the package
> status from the filesystem");
> }
> {code}
> It wants to do it just to set the length of the file and the modification
> time to the resource:
> {code}
> packageResource.setSize(fileStatus.getLen());
> packageResource.setTimestamp(fileStatus.getModificationTime());
> {code}
> If these attributes (length and timestamp) are really needed then I think
> they could be captured and submitted by {{run-job.sh}} which would allow to
> avoid this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
