[jira] [Updated] (YARN-3791) FSDownload
[
https://issues.apache.org/jira/browse/YARN-3791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
HuanWang updated YARN-3791:
---
Description:
Inadvertently,we set two source ftp path:
{code}
{ { ftp://10.27.178.207:21/home/cbt/1213/jxf.sql, 143322551, FILE, null
},pending,[(container_20150608111420_41540_1213_1503_)],4237640867118938,DOWNLOADING}
ftp://10.27.89.13:21/home/cbt/common/2/sql.jar, 1433225415000, FILE, null
},pending,[(container_20150608111420_41540_1213_1503_)],4237640866988089,DOWNLOADING}
the first one is a wrong path,only one source was set this;but Follow the log,i
saw Starting from the first path source download,All next jobs sources were
downloaded from ftp://10.27.178.207 by default.
the log is :
code
2015-06-09 11:14:34,653 INFO [AsyncDispatcher event handler]
localizer.ResourceLocalizationService
(ResourceLocalizationService.java:addResource(544)) - Downloading public rsrc:{
ftp://10.27.89.13:21/home/cbt/common/2/sql.jar, 1433225415000, FILE, null }
2015-06-09 11:14:34,653 INFO [AsyncDispatcher event handler]
localizer.ResourceLocalizationService
(ResourceLocalizationService.java:addResource(544)) - Downloading public rsrc:{
ftp://10.27.178.207:21/home/cbt/1213/jxf.sql, 143322551, FILE, null }
2015-06-09 11:14:37,883 INFO [Public Localizer]
localizer.ResourceLocalizationService
(ResourceLocalizationService.java:run(672)) - Failed to download rsrc { {
ftp://10.27.178.207:21/home/cbt/1213/jxf.sql, 143322551, FILE, null
},pending,[(container_20150608111420_41540_1213_1503_)],4237640867118938,DOWNLOADING}
java.io.IOException: Login failed on server - 10.27.178.207, port - 21
at
org.apache.hadoop.fs.ftp.FTPFileSystem.connect(FTPFileSystem.java:133)
at
org.apache.hadoop.fs.ftp.FTPFileSystem.getFileStatus(FTPFileSystem.java:390)
at
com.suning.cybertron.superion.util.FSDownload.copy(FSDownload.java:172)
at
com.suning.cybertron.superion.util.FSDownload.call(FSDownload.java:279)
at
com.suning.cybertron.superion.util.FSDownload.call(FSDownload.java:52)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2015-06-09 11:14:37,885 INFO [Public Localizer] localizer.LocalizedResource
(LocalizedResource.java:handle(203)) - Resource
ftp://10.27.178.207:21/home/cbt/1213/jxf.sql transitioned from DOWNLOADING to
FAILED
2015-06-09 11:14:37,886 INFO [Public Localizer]
localizer.ResourceLocalizationService
(ResourceLocalizationService.java:run(672)) - Failed to download rsrc { {
ftp://10.27.89.13:21/home/cbt/common/2/sql.jar, 1433225415000, FILE, null
},pending,[(container_20150608111420_41540_1213_1503_)],4237640866988089,DOWNLOADING}
java.io.IOException: Login failed on server - 10.27.178.207, port - 21
at
org.apache.hadoop.fs.ftp.FTPFileSystem.connect(FTPFileSystem.java:133)
at
org.apache.hadoop.fs.ftp.FTPFileSystem.getFileStatus(FTPFileSystem.java:390)
at
com.suning.cybertron.superion.util.FSDownload.copy(FSDownload.java:172)
at
com.suning.cybertron.superion.util.FSDownload.call(FSDownload.java:279)
at
com.suning.cybertron.superion.util.FSDownload.call(FSDownload.java:52)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2015-06-09 11:14:37,886 INFO [AsyncDispatcher event handler]
container.Container (ContainerImpl.java:handle(853)) - Container
container_20150608111420_41540_1213_1503_ transitioned from LOCALIZING to
LOCALIZATION_FAILED
2015-06-09 11:14:37,887 INFO [Public Localizer] localizer.LocalizedResource
(LocalizedResource.java:handle(203)) - Resource
ftp://10.27.89.13:21/home/cbt/common/2/sql.jar transitioned from DOWNLOADING to
FAILED
2015-06-09 11:14:37,887 INFO [AsyncDispatcher event handler]
localizer.LocalResourcesTrackerImpl
(LocalResourcesTrackerImpl.java:handle(133)) - Container
container_20150608111420_41540_1213_1503_ sent RELEASE event on a resource
request { ftp://10.27.89.13:21/home/cbt/common/2/sql.jar, 1433225415000, FILE,
null } not present
{code}
I debug the code of yarn.I found the piont is
org.apache.hadoop.fs.FileSystem#cache
the code source is here:
{code}
private FileSystem
[jira] [Updated] (YARN-3791) FSDownload
[
https://issues.apache.org/jira/browse/YARN-3791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
HuanWang updated YARN-3791:
---
Description:
Inadvertently,we set two source ftp path:
{code}
{ { ftp://10.27.178.207:21/home/cbt/1213/jxf.sql, 143322551, FILE, null
},pending,[(container_20150608111420_41540_1213_1503_)],4237640867118938,DOWNLOADING}
ftp://10.27.89.13:21/home/cbt/common/2/sql.jar, 1433225415000, FILE, null
},pending,[(container_20150608111420_41540_1213_1503_)],4237640866988089,DOWNLOADING}
the first one is a wrong path,only one source was set this;but Follow the log,i
saw Starting from the first path source download,All next jobs sources were
downloaded from ftp://10.27.178.207 by default.
{code}
the log is :
{code}
2015-06-09 11:14:34,653 INFO [AsyncDispatcher event handler]
localizer.ResourceLocalizationService
(ResourceLocalizationService.java:addResource(544)) - Downloading public rsrc:{
ftp://10.27.89.13:21/home/cbt/common/2/sql.jar, 1433225415000, FILE, null }
2015-06-09 11:14:34,653 INFO [AsyncDispatcher event handler]
localizer.ResourceLocalizationService
(ResourceLocalizationService.java:addResource(544)) - Downloading public rsrc:{
ftp://10.27.178.207:21/home/cbt/1213/jxf.sql, 143322551, FILE, null }
2015-06-09 11:14:37,883 INFO [Public Localizer]
localizer.ResourceLocalizationService
(ResourceLocalizationService.java:run(672)) - Failed to download rsrc { {
ftp://10.27.178.207:21/home/cbt/1213/jxf.sql, 143322551, FILE, null
},pending,[(container_20150608111420_41540_1213_1503_)],4237640867118938,DOWNLOADING}
java.io.IOException: Login failed on server - 10.27.178.207, port - 21
at
org.apache.hadoop.fs.ftp.FTPFileSystem.connect(FTPFileSystem.java:133)
at
org.apache.hadoop.fs.ftp.FTPFileSystem.getFileStatus(FTPFileSystem.java:390)
at
com.suning.cybertron.superion.util.FSDownload.copy(FSDownload.java:172)
at
com.suning.cybertron.superion.util.FSDownload.call(FSDownload.java:279)
at
com.suning.cybertron.superion.util.FSDownload.call(FSDownload.java:52)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2015-06-09 11:14:37,885 INFO [Public Localizer] localizer.LocalizedResource
(LocalizedResource.java:handle(203)) - Resource
ftp://10.27.178.207:21/home/cbt/1213/jxf.sql transitioned from DOWNLOADING to
FAILED
2015-06-09 11:14:37,886 INFO [Public Localizer]
localizer.ResourceLocalizationService
(ResourceLocalizationService.java:run(672)) - Failed to download rsrc { {
ftp://10.27.89.13:21/home/cbt/common/2/sql.jar, 1433225415000, FILE, null
},pending,[(container_20150608111420_41540_1213_1503_)],4237640866988089,DOWNLOADING}
java.io.IOException: Login failed on server - 10.27.178.207, port - 21
at
org.apache.hadoop.fs.ftp.FTPFileSystem.connect(FTPFileSystem.java:133)
at
org.apache.hadoop.fs.ftp.FTPFileSystem.getFileStatus(FTPFileSystem.java:390)
at
com.suning.cybertron.superion.util.FSDownload.copy(FSDownload.java:172)
at
com.suning.cybertron.superion.util.FSDownload.call(FSDownload.java:279)
at
com.suning.cybertron.superion.util.FSDownload.call(FSDownload.java:52)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2015-06-09 11:14:37,886 INFO [AsyncDispatcher event handler]
container.Container (ContainerImpl.java:handle(853)) - Container
container_20150608111420_41540_1213_1503_ transitioned from LOCALIZING to
LOCALIZATION_FAILED
2015-06-09 11:14:37,887 INFO [Public Localizer] localizer.LocalizedResource
(LocalizedResource.java:handle(203)) - Resource
ftp://10.27.89.13:21/home/cbt/common/2/sql.jar transitioned from DOWNLOADING to
FAILED
2015-06-09 11:14:37,887 INFO [AsyncDispatcher event handler]
localizer.LocalResourcesTrackerImpl
(LocalResourcesTrackerImpl.java:handle(133)) - Container
container_20150608111420_41540_1213_1503_ sent RELEASE event on a resource
request { ftp://10.27.89.13:21/home/cbt/common/2/sql.jar, 1433225415000, FILE,
null } not present
{code}
I debug the code of yarn.I found the piont is
org.apache.hadoop.fs.FileSystem#cache
the code source is here:
{code}
private
