[
https://issues.apache.org/jira/browse/IMPALA-7668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
bharath v updated IMPALA-7668:
------------------------------
Description:
There are a few places in the code that uses URLClassLoaders to load some java
classes at runtime. One example is when loading Java UDFs at startup.
{code:java}
public static List<Function> extractFunctions(String db,
...........
URL[] classLoaderUrls = new URL[] {new URL(localJarPath.toString())};
URLClassLoader urlClassLoader = new URLClassLoader(classLoaderUrls);
{code}
Starting JDK7, URLClassloader lets the caller close all the closeables opened
by it, avoiding bugs like FD leaks etc.
[https://docs.oracle.com/javase/7/docs/api/java/net/URLClassLoader.html#close(])
We have seen issues like lingering FDs from this code using certain versions of
JDKs where the FDs of temporary jars (copied to /tmp) by this code are not
closed and hence their space from disk is not claimed causing disk space issues.
Usually, error message looks like the following.
{noformat}
I1101 12:29:09.738245 3067 jni-util.cc:230] org.apache.hadoop.fs.FSError:
java.io.IOException: No space left on device
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:256)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:417)
at
org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunks(FSOutputSummer.java:213)
at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:164)
at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:145)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.close(ChecksumFileSystem.java:405)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:246)
at org.apache.hadoop.io.IOUtils.closeStream(IOUtils.java:285)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:70)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:121)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:369)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:341)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2123)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2092)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2068)
at org.apache.impala.common.FileSystemUtil.copyToLocal(FileSystemUtil.java:418)
at
org.apache.impala.catalog.CatalogServiceCatalog.extractFunctions(CatalogServiceCatalog.java:851)
at
org.apache.impala.catalog.CatalogServiceCatalog.loadJavaFunctions(CatalogServiceCatalog.java:944)
at
org.apache.impala.catalog.CatalogServiceCatalog.refreshFunctions(CatalogServiceCatalog.java:978)
at
org.apache.impala.service.CatalogOpExecutor.execResetMetadata(CatalogOpExecutor.java:3091)
at org.apache.impala.service.JniCatalog.resetMetadata(JniCatalog.java:160)
Caused by: java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:326)
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:254)
... 27 more
I1101 12:29:09.755692 3067 status.cc:125] FSError: java.io.IOException: No
space left on device
CAUSED BY: IOException: No space left on device
@ 0x96628a impala::Status::Status()
@ 0xced58d impala::JniUtil::GetJniExceptionMsg()
@ 0x9578e0 impala::Catalog::ResetMetadata()
@ 0x94a68f CatalogServiceThriftIf::ResetMetadata()
@ 0x9fc997 impala::CatalogServiceProcessor::process_ResetMetadata()
@ 0x9faa6f impala::CatalogServiceProcessor::dispatchCall()
@ 0x933f2a apache::thrift::TDispatchProcessor::process()
@ 0xb17c19 apache::thrift::server::TAcceptQueueServer::Task::run()
@ 0xb10399 impala::ThriftThread::RunRunnable()
@ 0xb11192 boost::detail::function::void_function_obj_invoker0<>::invoke()
@ 0xd602cf impala::Thread::SuperviseThread()
@ 0xd60aca boost::detail::thread_data<>::run()
@ 0x12d758a (unknown)
@ 0x3430807aa1 (unknown)
@ 0x34304e8c4d (unknown)
E1101 12:29:09.756640 3067 catalog-server.cc:84] FSError: java.io.IOException:
No space left on device
CAUSED BY: IOException: No space left on device
{noformat}
/tmp looks filled up
{noformat}
catalogd 3634 impala 275r REG 253,5 11961993 44
/tmp/7c1acc9e-2b82-4668-a099-4fb3b96b328f.jar (deleted)
catalogd 3634 impala 277r REG 253,5 11961993 46
/tmp/f4a49e7d-f5a9-4ae5-9291-62ce39233d0b.jar (deleted)
catalogd 3634 impala 279r REG 253,5 11961993 67
/tmp/ceaa27a6-37a9-4139-afb0-694949f8d5ad.jar (deleted)
catalogd 3634 impala 281r REG 253,5 11961993 47
/tmp/fbc12d6d-aeab-47c7-944c-f4580927f24f.jar (deleted)
catalogd 3634 impala 282r REG 253,5 11961993 48
/tmp/793815e9-c0a2-46c6-a4f8-64e3ff3e3942.jar (deleted)
catalogd 3634 impala 284r REG 253,5 49661 49
/tmp/256be23b-5853-42cc-8922-44e84d31f84c.jar (deleted)
catalogd 3634 impala 285r REG 253,5 11961993 69
/tmp/1136b8dd-aeff-4f5d-bf6e-99c26b0e7980.jar (deleted)
catalogd 3634 impala 286r REG 253,5 49661 50
/tmp/7e1555a7-e9d3-4136-ad22-5cb41e51fdf4.jar (deleted)
catalogd 3634 impala 287r REG 253,5 11961988 88
/tmp/62659aa9-b313-4670-ad04-06336e35ae3e.jar (deleted)
catalogd 3634 impala 288r REG 253,5 11961993 51
/tmp/64c267d8-7c66-4c7a-b66d-0bd32743d290.jar (deleted)
catalogd 3634 impala 289r REG 253,5 11961993 70
/tmp/119ddf11-6617-4691-8856-56387bdd218d.jar (deleted)
catalogd 3634 impala 290r REG 253,5 49661 52
/tmp/114f90b9-c40e-429d-a1ee-d62ad75247f4.jar (deleted)
catalogd 3634 impala 291r REG 253,5 49661 53
/tmp/1e74761b-09ee-4916-9ebe-05c78c93fa20.jar (deleted)
catalogd 3634 impala 292r REG 253,5 49661 54
/tmp/54b722b3-d995-48f4-a5f9-6e29084a3a18.jar (deleted)
catalogd 3634 impala 293r REG 253,5 49661 55
/tmp/3458c0ea-cf95-4f2e-b47d-ce5573a30843.jar (deleted)
catalogd 3634 impala 303r REG 253,5 11961993 56
/tmp/d08e5ec7-407a-4036-8a1e-f59c9c5a56ef.jar (deleted)
catalogd 3634 impala 304r REG 253,5 11961993 60
/tmp/0b06d239-9025-4ac4-9d42-9a172a6fd9a7.jar (deleted)
catalogd 3634 impala 305r REG 253,5 11961988 89
/tmp/9f510691-1cd9-4e6d-80e7-2d56a939e283.jar (deleted)
catalogd 3634 impala 306r REG 253,5 11961993 63
/tmp/e664a87e-0734-458b-9e54-f61dbdf30e1b.jar (deleted)
catalogd 3634 impala 307r REG 253,5 11961993 66
/tmp/29720d48-4b1a-476b-a8ce-f9e3c42006e6.jar (deleted)
catalogd 3634 impala 308r REG 253,5 11961993 71
/tmp/27f98197-80fb-4704-a887-d7d3458dfd11.jar (deleted)
catalogd 3634 impala 309r REG 253,5 49660 72
/tmp/2fb57d8b-5962-4927-8482-58a2d16fa6ca.jar (deleted)
catalogd 3634 impala 310r REG 253,5 11961988 102
/tmp/ea7aa558-dd70-4e4f-b438-8a145d187044.jar (deleted)
catalogd 3634 impala 311r REG 253,5 49660 73
/tmp/fd499d2b-c042-4f5e-ab04-0508444061dc.jar (deleted)
catalogd 3634 impala 312r REG 253,5 11961988 90
/tmp/b92b2beb-9fd3-49ba-b999-b1053877e4cf.jar (deleted)
catalogd 3634 impala 313r REG 253,5 11961993 74
/tmp/f76647c5-c8d6-494c-bb1f-a6b346d6636e.jar (deleted)
catalogd 3634 impala 314r REG 253,5 49660 80
/tmp/d8796b89-1bf7-44e1-80b1-4c91acb695f7.jar (deleted)
catalogd 3634 impala 315r REG 253,5 49660 81
/tmp/97685f6a-40ca-449c-9f45-ca5c2ff65f8b.jar (deleted)
catalogd 3634 impala 316r REG 253,5 49660 82
/tmp/174c33ad-ab2f-4150-b147-273c3f9e9c3d.jar (deleted)
catalogd 3634 impala 317r REG 253,5 49660 83
/tmp/97592f9c-7f9b-4857-908a-639bc588009d.jar (deleted)
catalogd 3634 impala 318r REG 253,5 11961988 91
/tmp/a3a6dffc-8aee-424a-877c-09701f751905.jar (deleted)
catalogd 3634 impala 319r REG 253,5 11961993 84
/tmp/313d1b3d-a8ba-4211-8d8b-ef63c0a6085d.jar (deleted)
catalogd 3634 impala 320r REG 253,5 11961993 85
/tmp/f579a717-c10d-44c0-ac03-375a1cb94395.jar (deleted)
catalogd 3634 impala 321r REG 253,5 11961993 86
/tmp/8b8465a3-62e0-45f6-989e-59c08ebc8535.jar (deleted)
{noformat}
was:
There are a few places in the code that uses URLClassLoaders to load some java
classes at runtime. One example is when loading Java UDFs at startup.
{code:java}
public static List<Function> extractFunctions(String db,
...........
URL[] classLoaderUrls = new URL[] {new URL(localJarPath.toString())};
URLClassLoader urlClassLoader = new URLClassLoader(classLoaderUrls);
{code}
Starting JDK7, URLClassloader lets the caller close all the closeables opened
by it, avoiding bugs like FD leaks etc.
https://docs.oracle.com/javase/7/docs/api/java/net/URLClassLoader.html#close()
We have seen issues like lingering FDs from this code using certain versions of
JDKs where the FDs of temporary jars (copied to /tmp) by this code are not
closed and hence their space from disk is not claimed causing disk space issues.
> close() URLClassLoaders after usage.
> ------------------------------------
>
> Key: IMPALA-7668
> URL: https://issues.apache.org/jira/browse/IMPALA-7668
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Affects Versions: Impala 2.12.0, Impala 3.1.0
> Reporter: bharath v
> Assignee: bharath v
> Priority: Major
> Fix For: Impala 3.1.0
>
>
> There are a few places in the code that uses URLClassLoaders to load some
> java classes at runtime. One example is when loading Java UDFs at startup.
> {code:java}
> public static List<Function> extractFunctions(String db,
> ...........
> URL[] classLoaderUrls = new URL[] {new URL(localJarPath.toString())};
> URLClassLoader urlClassLoader = new URLClassLoader(classLoaderUrls);
> {code}
> Starting JDK7, URLClassloader lets the caller close all the closeables opened
> by it, avoiding bugs like FD leaks etc.
> [https://docs.oracle.com/javase/7/docs/api/java/net/URLClassLoader.html#close(])
> We have seen issues like lingering FDs from this code using certain versions
> of JDKs where the FDs of temporary jars (copied to /tmp) by this code are not
> closed and hence their space from disk is not claimed causing disk space
> issues.
> Usually, error message looks like the following.
> {noformat}
> I1101 12:29:09.738245 3067 jni-util.cc:230] org.apache.hadoop.fs.FSError:
> java.io.IOException: No space left on device
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:256)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:417)
> at
> org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunks(FSOutputSummer.java:213)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:164)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:145)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.close(ChecksumFileSystem.java:405)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
> at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:246)
> at org.apache.hadoop.io.IOUtils.closeStream(IOUtils.java:285)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:70)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:121)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:369)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:341)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
> at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2123)
> at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2092)
> at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2068)
> at
> org.apache.impala.common.FileSystemUtil.copyToLocal(FileSystemUtil.java:418)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.extractFunctions(CatalogServiceCatalog.java:851)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.loadJavaFunctions(CatalogServiceCatalog.java:944)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.refreshFunctions(CatalogServiceCatalog.java:978)
> at
> org.apache.impala.service.CatalogOpExecutor.execResetMetadata(CatalogOpExecutor.java:3091)
> at org.apache.impala.service.JniCatalog.resetMetadata(JniCatalog.java:160)
> Caused by: java.io.IOException: No space left on device
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:326)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:254)
> ... 27 more
> I1101 12:29:09.755692 3067 status.cc:125] FSError: java.io.IOException: No
> space left on device
> CAUSED BY: IOException: No space left on device
> @ 0x96628a impala::Status::Status()
> @ 0xced58d impala::JniUtil::GetJniExceptionMsg()
> @ 0x9578e0 impala::Catalog::ResetMetadata()
> @ 0x94a68f CatalogServiceThriftIf::ResetMetadata()
> @ 0x9fc997 impala::CatalogServiceProcessor::process_ResetMetadata()
> @ 0x9faa6f impala::CatalogServiceProcessor::dispatchCall()
> @ 0x933f2a apache::thrift::TDispatchProcessor::process()
> @ 0xb17c19 apache::thrift::server::TAcceptQueueServer::Task::run()
> @ 0xb10399 impala::ThriftThread::RunRunnable()
> @ 0xb11192 boost::detail::function::void_function_obj_invoker0<>::invoke()
> @ 0xd602cf impala::Thread::SuperviseThread()
> @ 0xd60aca boost::detail::thread_data<>::run()
> @ 0x12d758a (unknown)
> @ 0x3430807aa1 (unknown)
> @ 0x34304e8c4d (unknown)
> E1101 12:29:09.756640 3067 catalog-server.cc:84] FSError:
> java.io.IOException: No space left on device
> CAUSED BY: IOException: No space left on device
> {noformat}
> /tmp looks filled up
> {noformat}
> catalogd 3634 impala 275r REG 253,5 11961993 44
> /tmp/7c1acc9e-2b82-4668-a099-4fb3b96b328f.jar (deleted)
> catalogd 3634 impala 277r REG 253,5 11961993 46
> /tmp/f4a49e7d-f5a9-4ae5-9291-62ce39233d0b.jar (deleted)
> catalogd 3634 impala 279r REG 253,5 11961993 67
> /tmp/ceaa27a6-37a9-4139-afb0-694949f8d5ad.jar (deleted)
> catalogd 3634 impala 281r REG 253,5 11961993 47
> /tmp/fbc12d6d-aeab-47c7-944c-f4580927f24f.jar (deleted)
> catalogd 3634 impala 282r REG 253,5 11961993 48
> /tmp/793815e9-c0a2-46c6-a4f8-64e3ff3e3942.jar (deleted)
> catalogd 3634 impala 284r REG 253,5 49661 49
> /tmp/256be23b-5853-42cc-8922-44e84d31f84c.jar (deleted)
> catalogd 3634 impala 285r REG 253,5 11961993 69
> /tmp/1136b8dd-aeff-4f5d-bf6e-99c26b0e7980.jar (deleted)
> catalogd 3634 impala 286r REG 253,5 49661 50
> /tmp/7e1555a7-e9d3-4136-ad22-5cb41e51fdf4.jar (deleted)
> catalogd 3634 impala 287r REG 253,5 11961988 88
> /tmp/62659aa9-b313-4670-ad04-06336e35ae3e.jar (deleted)
> catalogd 3634 impala 288r REG 253,5 11961993 51
> /tmp/64c267d8-7c66-4c7a-b66d-0bd32743d290.jar (deleted)
> catalogd 3634 impala 289r REG 253,5 11961993 70
> /tmp/119ddf11-6617-4691-8856-56387bdd218d.jar (deleted)
> catalogd 3634 impala 290r REG 253,5 49661 52
> /tmp/114f90b9-c40e-429d-a1ee-d62ad75247f4.jar (deleted)
> catalogd 3634 impala 291r REG 253,5 49661 53
> /tmp/1e74761b-09ee-4916-9ebe-05c78c93fa20.jar (deleted)
> catalogd 3634 impala 292r REG 253,5 49661 54
> /tmp/54b722b3-d995-48f4-a5f9-6e29084a3a18.jar (deleted)
> catalogd 3634 impala 293r REG 253,5 49661 55
> /tmp/3458c0ea-cf95-4f2e-b47d-ce5573a30843.jar (deleted)
> catalogd 3634 impala 303r REG 253,5 11961993 56
> /tmp/d08e5ec7-407a-4036-8a1e-f59c9c5a56ef.jar (deleted)
> catalogd 3634 impala 304r REG 253,5 11961993 60
> /tmp/0b06d239-9025-4ac4-9d42-9a172a6fd9a7.jar (deleted)
> catalogd 3634 impala 305r REG 253,5 11961988 89
> /tmp/9f510691-1cd9-4e6d-80e7-2d56a939e283.jar (deleted)
> catalogd 3634 impala 306r REG 253,5 11961993 63
> /tmp/e664a87e-0734-458b-9e54-f61dbdf30e1b.jar (deleted)
> catalogd 3634 impala 307r REG 253,5 11961993 66
> /tmp/29720d48-4b1a-476b-a8ce-f9e3c42006e6.jar (deleted)
> catalogd 3634 impala 308r REG 253,5 11961993 71
> /tmp/27f98197-80fb-4704-a887-d7d3458dfd11.jar (deleted)
> catalogd 3634 impala 309r REG 253,5 49660 72
> /tmp/2fb57d8b-5962-4927-8482-58a2d16fa6ca.jar (deleted)
> catalogd 3634 impala 310r REG 253,5 11961988 102
> /tmp/ea7aa558-dd70-4e4f-b438-8a145d187044.jar (deleted)
> catalogd 3634 impala 311r REG 253,5 49660 73
> /tmp/fd499d2b-c042-4f5e-ab04-0508444061dc.jar (deleted)
> catalogd 3634 impala 312r REG 253,5 11961988 90
> /tmp/b92b2beb-9fd3-49ba-b999-b1053877e4cf.jar (deleted)
> catalogd 3634 impala 313r REG 253,5 11961993 74
> /tmp/f76647c5-c8d6-494c-bb1f-a6b346d6636e.jar (deleted)
> catalogd 3634 impala 314r REG 253,5 49660 80
> /tmp/d8796b89-1bf7-44e1-80b1-4c91acb695f7.jar (deleted)
> catalogd 3634 impala 315r REG 253,5 49660 81
> /tmp/97685f6a-40ca-449c-9f45-ca5c2ff65f8b.jar (deleted)
> catalogd 3634 impala 316r REG 253,5 49660 82
> /tmp/174c33ad-ab2f-4150-b147-273c3f9e9c3d.jar (deleted)
> catalogd 3634 impala 317r REG 253,5 49660 83
> /tmp/97592f9c-7f9b-4857-908a-639bc588009d.jar (deleted)
> catalogd 3634 impala 318r REG 253,5 11961988 91
> /tmp/a3a6dffc-8aee-424a-877c-09701f751905.jar (deleted)
> catalogd 3634 impala 319r REG 253,5 11961993 84
> /tmp/313d1b3d-a8ba-4211-8d8b-ef63c0a6085d.jar (deleted)
> catalogd 3634 impala 320r REG 253,5 11961993 85
> /tmp/f579a717-c10d-44c0-ac03-375a1cb94395.jar (deleted)
> catalogd 3634 impala 321r REG 253,5 11961993 86
> /tmp/8b8465a3-62e0-45f6-989e-59c08ebc8535.jar (deleted)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
