[
https://issues.apache.org/jira/browse/IMPALA-11260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17719565#comment-17719565
]
Michael Smith commented on IMPALA-11260:
----------------------------------------
The CatalogdMetaProvider stores lots of different objects, including Java
standard library containers. Implementing size calculation without reflection
looks challenging, and approximating size would probably not be sufficient.
There are alternative implementations, but none of them are compelling due to
similar problems with Java 9+ or incompatible licenses.
I think our best bet for the moment is to work around this. At some point we
may still need to fork ehcache if it continues to be effectively unmaintained;
at some point in the future https://openjdk.org/jeps/8249196 may also become an
option.
> Catalog cache item sizes of CatalogdMetaProvider are underestimated on Java9+
> -----------------------------------------------------------------------------
>
> Key: IMPALA-11260
> URL: https://issues.apache.org/jira/browse/IMPALA-11260
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 3.1.0, Impala 3.2.0, Impala 4.0.0, Impala 3.3.0,
> Impala 3.4.0, Impala 3.4.1, Impala 4.1.0
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
>
> When running local catalog mode on Java11, the Ehcache sizeof lib complains
> that cache sizes may be underestimated:
> {code:java}
> W0421 20:50:44.238312 9819 ObjectGraphWalker.java:251]
> 744e548159a57cb5:879ee74c00000000] The JVM is preventing Ehcache from
> accessing the subgraph beneath 'final jdk.internal.loader.URLClassPath
> jdk.internal.loader.ClassLoaders$AppClassLoader.ucp' - cache sizes may be
> underestimated as a result
> Java exception follows:
> java.lang.reflect.InaccessibleObjectException: Unable to make field final
> jdk.internal.loader.URLClassPath
> jdk.internal.loader.ClassLoaders$AppClassLoader.ucp accessible: module
> java.base does not "opens jdk.internal.loader" to unnamed module @6ba7383d
> at
> java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:340)
> at
> java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:280)
> at
> java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:176)
> at java.base/java.lang.reflect.Field.setAccessible(Field.java:170)
> at
> org.ehcache.sizeof.ObjectGraphWalker.getAllFields(ObjectGraphWalker.java:245)
> at
> org.ehcache.sizeof.ObjectGraphWalker.getFilteredFields(ObjectGraphWalker.java:204)
> at
> org.ehcache.sizeof.ObjectGraphWalker.walk(ObjectGraphWalker.java:159)
> at org.ehcache.sizeof.SizeOf.deepSizeOf(SizeOf.java:74)
> at
> org.apache.impala.catalog.local.CatalogdMetaProvider$SizeOfWeigher.weigh(CatalogdMetaProvider.java:1999)
> at
> com.google.common.cache.LocalCache$Segment.setValue(LocalCache.java:2010)
> at
> com.google.common.cache.LocalCache$Segment.replace(LocalCache.java:2956)
> at com.google.common.cache.LocalCache.replace(LocalCache.java:4258)
> at
> org.apache.impala.catalog.local.CatalogdMetaProvider.loadWithCaching(CatalogdMetaProvider.java:540)
> at
> org.apache.impala.catalog.local.CatalogdMetaProvider.loadIcebergApiTable(CatalogdMetaProvider.java:1056)
> at
> org.apache.impala.catalog.local.LocalIcebergTable.loadIcebergTableViaMetaProvider(LocalIcebergTable.java:87)
> at
> org.apache.impala.catalog.local.LocalTable.load(LocalTable.java:107)
> at org.apache.impala.catalog.local.LocalDb.getTable(LocalDb.java:127)
> at
> org.apache.impala.analysis.StmtMetadataLoader.getMissingTables(StmtMetadataLoader.java:310)
> at
> org.apache.impala.analysis.StmtMetadataLoader.loadTables(StmtMetadataLoader.java:165)
> at
> org.apache.impala.analysis.StmtMetadataLoader.loadTables(StmtMetadataLoader.java:141)
> at
> org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:2014)
> at
> org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1926)
> at
> org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1750)
> at
> org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:164){code}
> Similar errors on other classes:
> {code}
> The JVM is preventing Ehcache from accessing the subgraph beneath 'final
> jdk.internal.loader.AbstractClassLoaderValue
> jdk.internal.loader.AbstractClassLoaderValue$Sub.this$0' - cache sizes may be
> underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'final
> jdk.internal.loader.URLClassPath
> jdk.internal.loader.ClassLoaders$AppClassLoader.ucp' - cache sizes may be
> underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.lang.Object jdk.internal.loader.AbstractClassLoaderValue$Sub.key'
> - cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.lang.String java.lang.module.Configuration.targetPlatform' - cache
> sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.lang.String java.lang.module.ModuleDescriptor.mainClass' - cache
> sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.lang.String java.lang.module.ModuleDescriptor.name' - cache sizes
> may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.lang.String java.lang.module.ModuleDescriptor.rawVersionString' -
> cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.lang.module.ModuleDescriptor$Version
> java.lang.module.ModuleDescriptor.version' - cache sizes may be
> underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.util.List java.lang.module.Configuration.parents' - cache sizes
> may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.util.Map java.lang.module.Configuration.graph' - cache sizes may
> be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.util.Map java.lang.module.Configuration.nameToModule' - cache
> sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.util.Map jdk.internal.loader.BuiltinClassLoader.moduleToReader' -
> cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.util.Map jdk.internal.loader.BuiltinClassLoader.nameToModule' -
> cache sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.util.Map jdk.internal.module.ServicesCatalog.map' - cache sizes
> may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.util.Set java.lang.module.Configuration.modules' - cache sizes may
> be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.util.Set java.lang.module.ModuleDescriptor.exports' - cache sizes
> may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.util.Set java.lang.module.ModuleDescriptor.modifiers' - cache
> sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.util.Set java.lang.module.ModuleDescriptor.opens' - cache sizes
> may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.util.Set java.lang.module.ModuleDescriptor.packages' - cache sizes
> may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.util.Set java.lang.module.ModuleDescriptor.provides' - cache sizes
> may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.util.Set java.lang.module.ModuleDescriptor.requires' - cache sizes
> may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final java.util.Set java.lang.module.ModuleDescriptor.uses' - cache sizes may
> be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final jdk.internal.loader.BuiltinClassLoader
> jdk.internal.loader.BuiltinClassLoader.parent' - cache sizes may be
> underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> final jdk.internal.loader.URLClassPath
> jdk.internal.loader.BuiltinClassLoader.ucp' - cache sizes may be
> underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> jdk.internal.reflect.ConstructorAccessorImpl
> jdk.internal.reflect.DelegatingConstructorAccessorImpl.delegate' - cache
> sizes may be underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> volatile java.lang.ref.SoftReference
> jdk.internal.loader.BuiltinClassLoader.resourceCache' - cache sizes may be
> underestimated as a result
> The JVM is preventing Ehcache from accessing the subgraph beneath 'private
> volatile java.util.List java.lang.module.Configuration.allConfigurations' -
> cache sizes may be underestimated as a result
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
