曹勇 created KYLIN-5087:
-------------------------
Summary: java.lang.UnsupportedOperationException: Spark version
2.0.0 not supported
Key: KYLIN-5087
URL: https://issues.apache.org/jira/browse/KYLIN-5087
Project: Kylin
Issue Type: Bug
Components: Environment , Integration, Query Engine, Spark Engine,
Web
Affects Versions: v4.0.0
Environment: hadoop-3.2.2
hive-2.3.9
kylin-4.0.0
scala-2.12.14
spark-3.1.2-bin-hadoop3.2
zookeeper-3.7.0
Reporter: 曹勇
Attachments: image-2021-09-09-20-11-08-870.png
When I query example sql `select part_dt, sum(price) as total_selled,
count(distinct seller_id) as sellers from kylin_sales group by part_dt order by
part_dt` for kylin_sales_cube in the document ,the issue happened:
{code:java}
2021-09-09 20:06:10,063 WARN [Thread-7] config.package:422 : Can not load the
default value of `spark.yarn.isHadoopProvided` from
`org/apache/spark/deploy/yarn/config.properties` with error,
java.lang.NullPointerException. Using `false` as a default value.
2021-09-09 20:06:10,482 INFO [Thread-7] client.AHSProxy:42 : Connecting to
Application History server at node1/192.168.111.49:10200
2021-09-09 20:06:10,601 INFO [Thread-7] yarn.Client:57 : Requesting a new
application from cluster with 3 NodeManagers
2021-09-09 20:06:11,337 INFO [Thread-7] conf.Configuration:2795 :
resource-types.xml not found
2021-09-09 20:06:11,338 INFO [Thread-7] resource.ResourceUtils:442 : Unable to
find 'resource-types.xml'.
2021-09-09 20:06:11,350 INFO [Thread-7] yarn.Client:57 : Verifying our
application has not requested more than the maximum memory capability of the
cluster (8192 MB per container)
2021-09-09 20:06:11,350 INFO [Thread-7] yarn.Client:57 : Will allocate AM
container, with 896 MB memory including 384 MB overhead
2021-09-09 20:06:11,351 INFO [Thread-7] yarn.Client:57 : Setting up container
launch context for our AM
2021-09-09 20:06:11,351 INFO [Thread-7] yarn.Client:57 : Setting up the launch
environment for our AM container
2021-09-09 20:06:11,356 INFO [Thread-7] yarn.Client:57 : Preparing resources
for our AM container
2021-09-09 20:06:11,434 WARN [Thread-7] yarn.Client:69 : Neither
spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading
libraries under SPARK_HOME.
2021-09-09 20:06:13,436 INFO [Thread-7] yarn.Client:57 : Uploading resource
file:/home/hadoop/kylin-4.0.0/tomcat/temp/spark-d250f1d5-51aa-47d1-9aa5-b36d3b60d336/__spark_libs__3169881654091659892.zip
->
hdfs://ns1/user/hadoop/.sparkStaging/application_1631008247268_0018/__spark_libs__3169881654091659892.zip
2021-09-09 20:06:14,657 INFO [Thread-7] yarn.Client:57 : Uploading resource
file:/home/hadoop/kylin-4.0.0/lib/kylin-parquet-job-4.0.0.jar ->
hdfs://ns1/user/hadoop/.sparkStaging/application_1631008247268_0018/kylin-parquet-job-4.0.0.jar
2021-09-09 20:06:15,058 INFO [Thread-7] yarn.Client:57 : Uploading resource
file:/home/hadoop/kylin-4.0.0/conf/spark-executor-log4j.properties ->
hdfs://ns1/user/hadoop/.sparkStaging/application_1631008247268_0018/spark-executor-log4j.properties
2021-09-09 20:06:15,238 INFO [Thread-7] yarn.Client:57 : Uploading resource
file:/home/hadoop/kylin-4.0.0/tomcat/temp/spark-d250f1d5-51aa-47d1-9aa5-b36d3b60d336/__spark_conf__4477603375905676389.zip
->
hdfs://ns1/user/hadoop/.sparkStaging/application_1631008247268_0018/__spark_conf__.zip
2021-09-09 20:06:15,354 INFO [Thread-7] spark.SecurityManager:57 : Changing
view acls to: hadoop
2021-09-09 20:06:15,354 INFO [Thread-7] spark.SecurityManager:57 : Changing
modify acls to: hadoop
2021-09-09 20:06:15,354 INFO [Thread-7] spark.SecurityManager:57 : Changing
view acls groups to:
2021-09-09 20:06:15,354 INFO [Thread-7] spark.SecurityManager:57 : Changing
modify acls groups to:
2021-09-09 20:06:15,354 INFO [Thread-7] spark.SecurityManager:57 :
SecurityManager: authentication disabled; ui acls disabled; users with view
permissions: Set(hadoop); groups with view permissions: Set(); users with
modify permissions: Set(hadoop); groups with modify permissions: Set()
2021-09-09 20:06:15,378 INFO [Thread-7] yarn.Client:57 : Submitting
application application_1631008247268_0018 to ResourceManager
2021-09-09 20:06:15,637 INFO [Thread-7] impl.YarnClientImpl:329 : Submitted
application application_1631008247268_0018
2021-09-09 20:06:16,646 INFO [Thread-7] yarn.Client:57 : Application report
for application_1631008247268_0018 (state: ACCEPTED)
2021-09-09 20:06:16,656 INFO [Thread-7] yarn.Client:57 :
client token: N/A
diagnostics: AM container is launched, waiting for AM container to
Register with RM
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1631189175391
final status: UNDEFINED
tracking URL: http://node1:8088/proxy/application_1631008247268_0018/
user: hadoop
2021-09-09 20:06:17,659 INFO [Thread-7] yarn.Client:57 : Application report
for application_1631008247268_0018 (state: ACCEPTED)
2021-09-09 20:06:18,662 INFO [Thread-7] yarn.Client:57 : Application report
for application_1631008247268_0018 (state: ACCEPTED)
2021-09-09 20:06:19,665 INFO [Thread-7] yarn.Client:57 : Application report
for application_1631008247268_0018 (state: ACCEPTED)
2021-09-09 20:06:19,947 INFO [dispatcher-event-loop-7]
cluster.YarnClientSchedulerBackend:57 : Add WebUI Filter.
org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS ->
node1,node2, PROXY_URI_BASES ->
http://node1:8088/proxy/application_1631008247268_0018,http://node2:8088/proxy/application_1631008247268_0018,
RM_HA_URLS -> node1:8088,node2:8088), /proxy/application_1631008247268_0018
2021-09-09 20:06:20,668 INFO [Thread-7] yarn.Client:57 : Application report
for application_1631008247268_0018 (state: RUNNING)
2021-09-09 20:06:20,669 INFO [Thread-7] yarn.Client:57 :
client token: N/A
diagnostics: N/A
ApplicationMaster host: 192.168.111.25
ApplicationMaster RPC port: -1
queue: default
start time: 1631189175391
final status: UNDEFINED
tracking URL: http://node1:8088/proxy/application_1631008247268_0018/
user: hadoop
2021-09-09 20:06:20,672 INFO [Thread-7] cluster.YarnClientSchedulerBackend:57
: Application application_1631008247268_0018 has started running.
2021-09-09 20:06:20,693 INFO [Thread-7] util.Utils:57 : Successfully started
service 'org.apache.spark.network.netty.NettyBlockTransferService' on port
37707.
2021-09-09 20:06:20,693 INFO [Thread-7] netty.NettyBlockTransferService:81 :
Server created on node3:37707
2021-09-09 20:06:20,696 INFO [Thread-7] storage.BlockManager:57 : Using
org.apache.spark.storage.RandomBlockReplicationPolicy for block replication
policy
2021-09-09 20:06:20,706 INFO [Thread-7] storage.BlockManagerMaster:57 :
Registering BlockManager BlockManagerId(driver, node3, 37707, None)
2021-09-09 20:06:20,713 INFO [dispatcher-BlockManagerMaster]
storage.BlockManagerMasterEndpoint:57 : Registering block manager node3:37707
with 2004.6 MiB RAM, BlockManagerId(driver, node3, 37707, None)
2021-09-09 20:06:20,719 INFO [Thread-7] storage.BlockManagerMaster:57 :
Registered BlockManager BlockManagerId(driver, node3, 37707, None)
2021-09-09 20:06:20,721 INFO [Thread-7] storage.BlockManager:57 : Initialized
BlockManager: BlockManagerId(driver, node3, 37707, None)
2021-09-09 20:06:20,832 INFO [dispatcher-event-loop-2]
cluster.YarnSchedulerBackend$YarnSchedulerEndpoint:57 : ApplicationMaster
registered as NettyRpcEndpointRef(spark-client://YarnAM)
2021-09-09 20:06:20,922 INFO [Thread-7] ui.ServerInfo:57 : Adding filter to
/metrics/json: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
2021-09-09 20:06:20,924 INFO [Thread-7] handler.ContextHandler:916 : Started
o.s.j.s.ServletContextHandler@71ddd2af{/metrics/json,null,AVAILABLE,@Spark}
2021-09-09 20:06:25,825 INFO [dispatcher-CoarseGrainedScheduler]
cluster.YarnSchedulerBackend$YarnDriverEndpoint:57 : Registered executor
NettyRpcEndpointRef(spark-client://Executor) (192.168.111.50:36340) with ID 1,
ResourceProfileId 0
2021-09-09 20:06:25,872 INFO [Thread-7] cluster.YarnClientSchedulerBackend:57
: SchedulerBackend is ready for scheduling beginning after reached
minRegisteredResourcesRatio: 0.8
2021-09-09 20:06:25,949 ERROR [Thread-7] sql.SparderContext:94 : Error for
initializing spark
java.lang.UnsupportedOperationException: Spark version 2.0.0 not supported
at
org.apache.spark.utils.KylinReflectUtils$.getSessionState(KylinReflectUtils.scala:42)
at
org.apache.spark.sql.KylinSession.sessionState$lzycompute(KylinSession.scala:47)
at org.apache.spark.sql.KylinSession.sessionState(KylinSession.scala:46)
at
org.apache.kylin.query.UdfManager.$anonfun$registerBuiltInFunc$1(UdfManager.scala:35)
at
org.apache.kylin.query.UdfManager.$anonfun$registerBuiltInFunc$1$adapted(UdfManager.scala:34)
at scala.collection.immutable.List.foreach(List.scala:392)
at
org.apache.kylin.query.UdfManager.org$apache$kylin$query$UdfManager$$registerBuiltInFunc(UdfManager.scala:34)
at org.apache.kylin.query.UdfManager$.create(UdfManager.scala:85)
at
org.apache.spark.sql.KylinSession$KylinBuilder.getOrCreateKylinSession(KylinSession.scala:116)
at
org.apache.spark.sql.SparderContext$$anon$1.run(SparderContext.scala:146)
at java.lang.Thread.run(Thread.java:748)
2021-09-09 20:06:25,951 INFO [Thread-7] sql.SparderContext:57 : Setting
initializing Spark thread to null.
2021-09-09 20:06:25,952 INFO [Query 71e4c4e7-df0a-2b6b-ee48-1b0a8c1d6384-35]
monitor.SparderContextCanary:68 : Start monitoring Sparder
2021-09-09 20:06:25,953 INFO [Query 71e4c4e7-df0a-2b6b-ee48-1b0a8c1d6384-35]
sql.SparderContext:57 : Init spark.
2021-09-09 20:06:25,954 INFO [Query 71e4c4e7-df0a-2b6b-ee48-1b0a8c1d6384-35]
sql.SparderContext:57 : Initializing Spark thread starting.
2021-09-09 20:06:25,954 INFO [Query 71e4c4e7-df0a-2b6b-ee48-1b0a8c1d6384-35]
sql.SparderContext:57 : Initializing Spark, waiting for done.
2021-09-09 20:06:25,955 INFO [Thread-37] sql.SparderContext:57 :
SparderContext deploy with spark master: yarn
2021-09-09 20:06:25,964 INFO [Thread-37] sql.SparderContext:57 : Spark
application id is application_1631008247268_0018
2021-09-09 20:06:25,965 INFO [Thread-37] sql.SparderContext:57 : Spark context
started successfully with stack trace:
2021-09-09 20:06:25,965 INFO [Thread-37] sql.SparderContext:57 :
java.lang.Thread.getStackTrace(Thread.java:1559)
org.apache.spark.sql.SparderContext$$anon$1.$anonfun$run$8(SparderContext.scala:171)
org.apache.spark.internal.Logging.logInfo(Logging.scala:57)
org.apache.spark.internal.Logging.logInfo$(Logging.scala:56)
org.apache.spark.sql.SparderContext$.logInfo(SparderContext.scala:45)
org.apache.spark.sql.SparderContext$$anon$1.run(SparderContext.scala:171)
java.lang.Thread.run(Thread.java:748)
2021-09-09 20:06:25,966 INFO [Thread-37] sql.SparderContext:57 : Class loader:
org.apache.kylin.spark.classloader.SparkClassLoader@3187093f
2021-09-09 20:06:25,969 INFO [Thread-37] memory.MonitorEnv:57 : create driver
monitor env
2021-09-09 20:06:25,976 INFO [dispatcher-BlockManagerMaster]
storage.BlockManagerMasterEndpoint:57 : Registering block manager node2:44050
with 2004.6 MiB RAM, BlockManagerId(1, node2, 44050, None)
2021-09-09 20:06:25,988 INFO [Thread-37] sql.SparderContext:57 : setup master
endpoint finished.hostPort:node3:40116
2021-09-09 20:06:26,025 INFO [Thread-37] client.AHSProxy:42 : Connecting to
Application History server at node1/192.168.111.49:10200
2021-09-09 20:06:26,029 INFO [Thread-37] sql.SparderContext:57 : Setting
initializing Spark thread to null.
2021-09-09 20:06:26,037 ERROR [Query 71e4c4e7-df0a-2b6b-ee48-1b0a8c1d6384-35]
service.QueryService:576 : Exception while executing query
java.sql.SQLException: Error while executing SQL "select * from (select
part_dt, sum(price) as total_selled, count(distinct seller_id) as sellers from
kylin_sales group by part_dt order by part_dt) limit 50000":
java.lang.UnsupportedOperationException: Spark version 2.0.0 not supported
at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
at
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:163)
at
org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
at
org.apache.kylin.rest.service.QueryService.executeRequest(QueryService.java:1026)
at
org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:710)
at
org.apache.kylin.rest.service.QueryService.query(QueryService.java:221)
at
org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:514)
at
org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:469)
at
org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:405)
at
org.apache.kylin.rest.controller.QueryController.query(QueryController.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:854)
at
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:765)
at
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
at
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
at
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
at
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:215)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
at
org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
at
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
at
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:209)
at
com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:244)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:165)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:1025)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:452)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1195)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:654)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:317)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: java.lang.UnsupportedOperationException:
Spark version 2.0.0 not supported
at
org.apache.kylin.query.exec.SparkExec.collectToEnumerable(SparkExec.java:50)
at Baz.bind(Unknown Source)
at
org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:365)
at
org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:301)
at
org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:559)
at
org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:550)
at
org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:182)
at
org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
at
org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:44)
at
org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:667)
at
org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:619)
at
org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
at
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
... 84 more
Caused by: java.lang.UnsupportedOperationException: Spark version 2.0.0 not
supported
at
org.apache.spark.utils.KylinReflectUtils$.getSessionState(KylinReflectUtils.scala:42)
at
org.apache.spark.sql.KylinSession.sessionState$lzycompute(KylinSession.scala:47)
at org.apache.spark.sql.KylinSession.sessionState(KylinSession.scala:46)
at org.apache.spark.sql.SQLContext.sessionState(SQLContext.scala:78)
at org.apache.spark.sql.SQLContext.conf(SQLContext.scala:80)
at
org.apache.spark.sql.execution.datasources.FileStatusCache$.getOrCreate(FileStatusCache.scala:44)
at
org.apache.spark.sql.execution.datasource.ShardFileStatusCache$.getFileStatusCache(ShardFileStatusCache.scala:29)
at
org.apache.spark.sql.SparderContext$.$anonfun$initSpark$1(SparderContext.scala:214)
at
org.apache.spark.sql.SparderContext$.withClassLoad(SparderContext.scala:257)
at
org.apache.spark.sql.SparderContext$.initSpark(SparderContext.scala:129)
at
org.apache.spark.sql.SparderContext$.$anonfun$getOriginalSparkSession$1(SparderContext.scala:58)
at
org.apache.spark.sql.SparderContext$.withClassLoad(SparderContext.scala:257)
at
org.apache.spark.sql.SparderContext$.getOriginalSparkSession(SparderContext.scala:55)
at
org.apache.spark.sql.SparderContext$.$anonfun$initSpark$1(SparderContext.scala:214)
at
org.apache.spark.sql.SparderContext$.withClassLoad(SparderContext.scala:257)
at
org.apache.spark.sql.SparderContext$.initSpark(SparderContext.scala:129)
at
org.apache.spark.sql.SparderContext$.$anonfun$getOriginalSparkSession$1(SparderContext.scala:58)
at
org.apache.spark.sql.SparderContext$.withClassLoad(SparderContext.scala:257)
at
org.apache.spark.sql.SparderContext$.getOriginalSparkSession(SparderContext.scala:55)
at
org.apache.spark.sql.SparderContextFacade$.current(SparderContextFacade.scala:34)
at
org.apache.spark.sql.SparderContext$.getSparkSession(SparderContext.scala:65)
at
org.apache.kylin.query.runtime.plans.TableScanPlan$.$anonfun$createOLAPTable$1(TableScanPlan.scala:78)
at org.apache.spark.utils.LogEx.logTime(LogEx.scala:40)
at org.apache.spark.utils.LogEx.logTime$(LogEx.scala:38)
at
org.apache.kylin.query.runtime.plans.TableScanPlan$.logTime(TableScanPlan.scala:43)
at
org.apache.kylin.query.runtime.plans.TableScanPlan$.createOLAPTable(TableScanPlan.scala:51)
at
org.apache.kylin.query.runtime.CalciteToSparkPlaner.$anonfun$visit$2(CalciteToSparkPlaner.scala:58)
at
org.apache.kylin.query.runtime.CalciteToSparkPlaner.logTime(CalciteToSparkPlaner.scala:120)
at
org.apache.kylin.query.runtime.CalciteToSparkPlaner.visit(CalciteToSparkPlaner.scala:58)
at org.apache.calcite.rel.SingleRel.childrenAccept(SingleRel.java:72)
at
org.apache.kylin.query.runtime.CalciteToSparkPlaner.visit(CalciteToSparkPlaner.scala:45)
at org.apache.calcite.rel.SingleRel.childrenAccept(SingleRel.java:72)
at
org.apache.kylin.query.runtime.CalciteToSparkPlaner.visit(CalciteToSparkPlaner.scala:45)
at org.apache.calcite.rel.SingleRel.childrenAccept(SingleRel.java:72)
at
org.apache.kylin.query.runtime.CalciteToSparkPlaner.visit(CalciteToSparkPlaner.scala:45)
at org.apache.calcite.rel.SingleRel.childrenAccept(SingleRel.java:72)
at
org.apache.kylin.query.runtime.CalciteToSparkPlaner.visit(CalciteToSparkPlaner.scala:45)
at org.apache.calcite.rel.RelVisitor.go(RelVisitor.java:61)
at
org.apache.kylin.query.runtime.SparkEngine.toSparkPlan(SparkEngine.java:60)
at
org.apache.kylin.query.runtime.SparkEngine.compute(SparkEngine.java:49)
at
org.apache.kylin.query.exec.QueryEngineFactory.compute(QueryEngineFactory.java:47)
at
org.apache.kylin.query.exec.SparkExec.collectToEnumerable(SparkExec.java:41)
... 96 more
2021-09-09 20:06:26,041 INFO [Query 71e4c4e7-df0a-2b6b-ee48-1b0a8c1d6384-35]
service.QueryService:1222 : Processed rows for each storageContext: 0
2021-09-09 20:06:26,048 INFO [Query 71e4c4e7-df0a-2b6b-ee48-1b0a8c1d6384-35]
service.QueryService:391 :
==========================[QUERY]===============================
Query Id: 71e4c4e7-df0a-2b6b-ee48-1b0a8c1d6384
SQL: select part_dt, sum(price) as total_selled, count(distinct seller_id) as
sellers from kylin_sales group by part_dt order by part_dt
User: ADMIN
Success: false
Duration: 21.519
Project: learn_kylin
Realization Names: [CUBE[name=kylin_sales_cube]]
Cuboid Ids: [16384]
Is Exactly Matched: [false]
Total scan count: 0
Total scan files: 0
Total metadata time: 0ms
Total spark scan time: 0ms
Total scan bytes: -1
Result row count: 0
Storage cache used: false
Is Query Push-Down: false
Is Prepare: false
Used Spark pool: null
Trace URL: null
Message: java.lang.UnsupportedOperationException: Spark version 2.0.0 not
supported
while executing SQL: "select * from (select part_dt, sum(price) as
total_selled, count(distinct seller_id) as sellers from kylin_sales group by
part_dt order by part_dt) limit 50000"
==========================[QUERY]===============================2021-09-09
20:06:26,052 ERROR [http-bio-7070-exec-1] controller.BasicController:65 :
org.apache.kylin.rest.exception.InternalErrorException:
java.lang.UnsupportedOperationException: Spark version 2.0.0 not supported
while executing SQL: "select * from (select part_dt, sum(price) as
total_selled, count(distinct seller_id) as sellers from kylin_sales group by
part_dt order by part_dt) limit 50000"
at
org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:486)
at
org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:405)
at
org.apache.kylin.rest.controller.QueryController.query(QueryController.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:854)
at
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:765)
at
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
at
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
at
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
at
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:215)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
at
org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
at
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
at
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:209)
at
com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:244)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:165)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:1025)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:452)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1195)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:654)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:317)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.UnsupportedOperationException: Spark version 2.0.0 not
supported
at
org.apache.spark.utils.KylinReflectUtils$.getSessionState(KylinReflectUtils.scala:42)
at
org.apache.spark.sql.KylinSession.sessionState$lzycompute(KylinSession.scala:47)
at org.apache.spark.sql.KylinSession.sessionState(KylinSession.scala:46)
at org.apache.spark.sql.SQLContext.sessionState(SQLContext.scala:78)
at org.apache.spark.sql.SQLContext.conf(SQLContext.scala:80)
at
org.apache.spark.sql.execution.datasources.FileStatusCache$.getOrCreate(FileStatusCache.scala:44)
at
org.apache.spark.sql.execution.datasource.ShardFileStatusCache$.getFileStatusCache(ShardFileStatusCache.scala:29)
at
org.apache.spark.sql.SparderContext$.$anonfun$initSpark$1(SparderContext.scala:214)
at
org.apache.spark.sql.SparderContext$.withClassLoad(SparderContext.scala:257)
at
org.apache.spark.sql.SparderContext$.initSpark(SparderContext.scala:129)
at
org.apache.spark.sql.SparderContext$.$anonfun$getOriginalSparkSession$1(SparderContext.scala:58)
at
org.apache.spark.sql.SparderContext$.withClassLoad(SparderContext.scala:257)
at
org.apache.spark.sql.SparderContext$.getOriginalSparkSession(SparderContext.scala:55)
at
org.apache.spark.sql.SparderContext$.$anonfun$initSpark$1(SparderContext.scala:214)
at
org.apache.spark.sql.SparderContext$.withClassLoad(SparderContext.scala:257)
at
org.apache.spark.sql.SparderContext$.initSpark(SparderContext.scala:129)
at
org.apache.spark.sql.SparderContext$.$anonfun$getOriginalSparkSession$1(SparderContext.scala:58)
at
org.apache.spark.sql.SparderContext$.withClassLoad(SparderContext.scala:257)
at
org.apache.spark.sql.SparderContext$.getOriginalSparkSession(SparderContext.scala:55)
at
org.apache.spark.sql.SparderContextFacade$.current(SparderContextFacade.scala:34)
at
org.apache.spark.sql.SparderContext$.getSparkSession(SparderContext.scala:65)
at
org.apache.kylin.query.runtime.plans.TableScanPlan$.$anonfun$createOLAPTable$1(TableScanPlan.scala:78)
at org.apache.spark.utils.LogEx.logTime(LogEx.scala:40)
at org.apache.spark.utils.LogEx.logTime$(LogEx.scala:38)
at
org.apache.kylin.query.runtime.plans.TableScanPlan$.logTime(TableScanPlan.scala:43)
at
org.apache.kylin.query.runtime.plans.TableScanPlan$.createOLAPTable(TableScanPlan.scala:51)
at
org.apache.kylin.query.runtime.CalciteToSparkPlaner.$anonfun$visit$2(CalciteToSparkPlaner.scala:58)
at
org.apache.kylin.query.runtime.CalciteToSparkPlaner.logTime(CalciteToSparkPlaner.scala:120)
at
org.apache.kylin.query.runtime.CalciteToSparkPlaner.visit(CalciteToSparkPlaner.scala:58)
at org.apache.calcite.rel.SingleRel.childrenAccept(SingleRel.java:72)
at
org.apache.kylin.query.runtime.CalciteToSparkPlaner.visit(CalciteToSparkPlaner.scala:45)
at org.apache.calcite.rel.SingleRel.childrenAccept(SingleRel.java:72)
at
org.apache.kylin.query.runtime.CalciteToSparkPlaner.visit(CalciteToSparkPlaner.scala:45)
at org.apache.calcite.rel.SingleRel.childrenAccept(SingleRel.java:72)
at
org.apache.kylin.query.runtime.CalciteToSparkPlaner.visit(CalciteToSparkPlaner.scala:45)
at org.apache.calcite.rel.SingleRel.childrenAccept(SingleRel.java:72)
at
org.apache.kylin.query.runtime.CalciteToSparkPlaner.visit(CalciteToSparkPlaner.scala:45)
at org.apache.calcite.rel.RelVisitor.go(RelVisitor.java:61)
at
org.apache.kylin.query.runtime.SparkEngine.toSparkPlan(SparkEngine.java:60)
at
org.apache.kylin.query.runtime.SparkEngine.compute(SparkEngine.java:49)
at
org.apache.kylin.query.exec.QueryEngineFactory.compute(QueryEngineFactory.java:47)
at
org.apache.kylin.query.exec.SparkExec.collectToEnumerable(SparkExec.java:41)
at Baz.bind(Unknown Source)
at
org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:365)
at
org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:301)
at
org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:559)
at
org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:550)
at
org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:182)
at
org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
at
org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:44)
at
org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:667)
at
org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:619)
at
org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
at
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
at
org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
at
org.apache.kylin.rest.service.QueryService.executeRequest(QueryService.java:1026)
at
org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:710)
at
org.apache.kylin.rest.service.QueryService.query(QueryService.java:221)
at
org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:514)
at
org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:469)
... 78 more
2021-09-09 20:06:35,790 INFO [FetcherRunner 1341171673-29]
threadpool.DefaultFetcherRunner:117 : Job Fetcher: 0 should running, 0 actual
running, 0 stopped, 0 ready, 2 already succeed, 0 error, 0 discarded, 0 others
{code}
web ui as follow:
!image-2021-09-09-20-11-08-870.png!
Obviously,my spark version is spark-3.1.2-bin-hadoop3.2 in the environment. But
why kylin call spark version 2.0.0 ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
