[
https://issues.apache.org/jira/browse/SEDONA-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17482163#comment-17482163
]
Jean-Denis Giguère commented on SEDONA-79:
------------------------------------------
Based on a propositon of Paweł Kociński @Imbruced on
[https://gitter.im/apache/sedona?,] I find an alternative way to create the
thrift server. The reference proposed by Paweł is
[https://geoatscale.com/article?id=b5b1fd23-87b5-4369-8089-3ea1c371a829]
(redaction in progress).
This is the scala code I used:
{code:java}
import org.apache.sedona.sql.utils.SedonaSQLRegistrator
import org.apache.sedona.viz.core.Serde.SedonaVizKryoRegistrator
import org.apache.spark.serializer.KryoSerializer
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.hive.thriftserver.HiveThriftServer2
object SedonaThrift {
def main(args: Array[String]): Unit = {
val sparkSession: SparkSession =
SparkSession
.builder()
.master("local[2]")
.config("spark.sql.hive.thriftServer.singleSession", "true")
.config("hive.server2.thrift.bind.host", "0.0.0.0")
.config("hive.server2.thrift.port", "10001")
.config("spark.serializer", classOf[KryoSerializer].getName)
.config("spark.kryo.registrator",
classOf[SedonaVizKryoRegistrator].getName)
.appName("SedonaThrift")
.enableHiveSupport()
.getOrCreate()
SedonaSQLRegistrator.registerAll(sparkSession)
HiveThriftServer2.startWithContext(sparkSession.sqlContext)
}
}
{code}
> Pure SQL does not work with Thrift Server
> -----------------------------------------
>
> Key: SEDONA-79
> URL: https://issues.apache.org/jira/browse/SEDONA-79
> Project: Apache Sedona
> Issue Type: Bug
> Reporter: Jean-Denis Giguère
> Priority: Minor
> Labels: sql
>
> h2. Current situation
> When I add Sedona SQL support to the Thrift Server on a local Spark Cluster,
> a {{scala.ScalaReflectionException}} associated with the class
> {{org.locationtech.jts.geom.Geometry}} is triggered.
> {code:bash}
> /opt/spark/bin/spark-submit --class
> org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 \
> --master $SPARK_MASTER \
> --hiveconf hive.server2.thrift.port=10001 --hiveconf
> hive.server2.thrift.bind.host=0.0.0.0 \
> --packages
> org.apache.sedona:sedona-python-adapter-3.0_2.12:1.1.1-incubating,org.apache.sedona:sedona-viz-3.0_2.12:1.1.1-incubating,org.datasyslab:geotools-wrapper:geotools-24.0
> \
> --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
> --conf
> spark.kryo.registrator=org.apache.sedona.viz.core.Serde.SedonaVizKryoRegistrator
> \
> --conf
> spark.sql.extensions=org.apache.sedona.viz.sql.SedonaVizExtensions,org.apache.sedona.sql.SedonaSqlExtensions
> {code}
> The error is not triggered when I launch the server, but when I attempt to
> connect to it with the Apache Hive JDBC driver.
> This error is not present when I use {{spark-shell}} with the same parameter.
> h2. Expected situation
> I would expect to be able to start the thrift server to expose sedona SQL on
> a standalone cluster.
> h2. Logs
> {noformat}
> docker-compose up spark-thrift
> WARNING: The Docker Engine you're using is running in swarm mode.
> Compose does not use swarm mode to deploy services to multiple nodes in a
> swarm. All containers will be scheduled on the current node.
> To deploy your application across the swarm, use `docker stack deploy`.
> spark-dbt_spark-master_1 is up-to-date
> spark-dbt_spark-worker-b_1 is up-to-date
> spark-dbt_spark-worker-a_1 is up-to-date
> Starting spark-dbt_spark-thrift_1 ... done
> Attaching to spark-dbt_spark-thrift_1
> spark-thrift_1 | WARNING: An illegal reflective access operation has
> occurred
> spark-thrift_1 | WARNING: Illegal reflective access by
> org.apache.spark.unsafe.Platform
> (file:/opt/spark/jars/spark-unsafe_2.12-3.0.2.jar) to constructor
> java.nio.DirectByteBuffer(long,int)
> spark-thrift_1 | WARNING: Please consider reporting this to the
> maintainers of org.apache.spark.unsafe.Platform
> spark-thrift_1 | WARNING: Use --illegal-access=warn to enable warnings of
> further illegal reflective access operations
> spark-thrift_1 | WARNING: All illegal access operations will be denied in
> a future release
> spark-thrift_1 | Ivy Default Cache set to: /root/.ivy2/cache
> spark-thrift_1 | The jars for the packages stored in: /root/.ivy2/jars
> spark-thrift_1 | :: loading settings :: url =
> jar:file:/opt/spark/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
> spark-thrift_1 | org.apache.sedona#sedona-python-adapter-3.0_2.12 added as
> a dependency
> spark-thrift_1 | org.apache.sedona#sedona-viz-3.0_2.12 added as a
> dependency
> spark-thrift_1 | org.datasyslab#geotools-wrapper added as a dependency
> spark-thrift_1 | :: resolving dependencies ::
> org.apache.spark#spark-submit-parent-56248f53-61ad-4c72-bd1f-c13739ab99c0;1.0
> spark-thrift_1 | confs: [default]
> spark-thrift_1 | found
> org.apache.sedona#sedona-python-adapter-3.0_2.12;1.1.1-incubating in central
> spark-thrift_1 | found org.locationtech.jts#jts-core;1.18.0 in central
> spark-thrift_1 | found org.wololo#jts2geojson;0.16.1 in central
> spark-thrift_1 | found
> com.fasterxml.jackson.core#jackson-databind;2.12.2 in central
> spark-thrift_1 | found
> com.fasterxml.jackson.core#jackson-annotations;2.12.2 in central
> spark-thrift_1 | found com.fasterxml.jackson.core#jackson-core;2.12.2 in
> central
> spark-thrift_1 | found
> org.apache.sedona#sedona-core-3.0_2.12;1.1.1-incubating in central
> spark-thrift_1 | found
> org.scala-lang.modules#scala-collection-compat_2.12;2.5.0 in central
> spark-thrift_1 | found
> org.apache.sedona#sedona-sql-3.0_2.12;1.1.1-incubating in central
> spark-thrift_1 | found
> org.apache.sedona#sedona-viz-3.0_2.12;1.1.1-incubating in central
> spark-thrift_1 | found org.beryx#awt-color-factory;1.0.0 in central
> spark-thrift_1 | found org.datasyslab#geotools-wrapper;geotools-24.0 in
> central
> spark-thrift_1 | :: resolution report :: resolve 533ms :: artifacts dl 15ms
> spark-thrift_1 | :: modules in use:
> spark-thrift_1 | com.fasterxml.jackson.core#jackson-annotations;2.12.2
> from central in [default]
> spark-thrift_1 | com.fasterxml.jackson.core#jackson-core;2.12.2 from
> central in [default]
> spark-thrift_1 | com.fasterxml.jackson.core#jackson-databind;2.12.2 from
> central in [default]
> spark-thrift_1 | org.apache.sedona#sedona-core-3.0_2.12;1.1.1-incubating
> from central in [default]
> spark-thrift_1 |
> org.apache.sedona#sedona-python-adapter-3.0_2.12;1.1.1-incubating from
> central in [default]
> spark-thrift_1 | org.apache.sedona#sedona-sql-3.0_2.12;1.1.1-incubating
> from central in [default]
> spark-thrift_1 | org.apache.sedona#sedona-viz-3.0_2.12;1.1.1-incubating
> from central in [default]
> spark-thrift_1 | org.beryx#awt-color-factory;1.0.0 from central in
> [default]
> spark-thrift_1 | org.datasyslab#geotools-wrapper;geotools-24.0 from
> central in [default]
> spark-thrift_1 | org.locationtech.jts#jts-core;1.18.0 from central in
> [default]
> spark-thrift_1 |
> org.scala-lang.modules#scala-collection-compat_2.12;2.5.0 from central in
> [default]
> spark-thrift_1 | org.wololo#jts2geojson;0.16.1 from central in [default]
> spark-thrift_1 | :: evicted modules:
> spark-thrift_1 | org.locationtech.jts#jts-core;1.18.1 by
> [org.locationtech.jts#jts-core;1.18.0] in [default]
> spark-thrift_1 |
> ---------------------------------------------------------------------
> spark-thrift_1 | | | modules ||
> artifacts |
> spark-thrift_1 | | conf | number| search|dwnlded|evicted||
> number|dwnlded|
> spark-thrift_1 |
> ---------------------------------------------------------------------
> spark-thrift_1 | | default | 13 | 0 | 0 | 1 ||
> 12 | 0 |
> spark-thrift_1 |
> ---------------------------------------------------------------------
> spark-thrift_1 | :: retrieving ::
> org.apache.spark#spark-submit-parent-56248f53-61ad-4c72-bd1f-c13739ab99c0
> spark-thrift_1 | confs: [default]
> spark-thrift_1 | 0 artifacts copied, 12 already retrieved (0kB/13ms)
> spark-thrift_1 | 22/01/24 20:55:06 WARN NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> spark-thrift_1 | Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
> spark-thrift_1 | 22/01/24 20:55:06 INFO HiveThriftServer2: Started daemon
> with process name: 7@9d3fb159061a
> spark-thrift_1 | 22/01/24 20:55:06 INFO SignalUtils: Registered signal
> handler for TERM
> spark-thrift_1 | 22/01/24 20:55:06 INFO SignalUtils: Registered signal
> handler for HUP
> spark-thrift_1 | 22/01/24 20:55:06 INFO SignalUtils: Registered signal
> handler for INT
> spark-thrift_1 | 22/01/24 20:55:06 INFO HiveThriftServer2: Starting
> SparkContext
> spark-thrift_1 | 22/01/24 20:55:06 INFO HiveConf: Found configuration file
> null
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Running Spark
> version 3.0.2
> spark-thrift_1 | 22/01/24 20:55:07 INFO ResourceUtils:
> ==============================================================
> spark-thrift_1 | 22/01/24 20:55:07 INFO ResourceUtils: Resources for
> spark.driver:
> spark-thrift_1 |
> spark-thrift_1 | 22/01/24 20:55:07 INFO ResourceUtils:
> ==============================================================
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Submitted
> application: SparkSQL::172.28.0.5
> spark-thrift_1 | 22/01/24 20:55:07 INFO SecurityManager: Changing view
> acls to: root
> spark-thrift_1 | 22/01/24 20:55:07 INFO SecurityManager: Changing modify
> acls to: root
> spark-thrift_1 | 22/01/24 20:55:07 INFO SecurityManager: Changing view
> acls groups to:
> spark-thrift_1 | 22/01/24 20:55:07 INFO SecurityManager: Changing modify
> acls groups to:
> spark-thrift_1 | 22/01/24 20:55:07 INFO SecurityManager: SecurityManager:
> authentication disabled; ui acls disabled; users with view permissions:
> Set(root); groups with view permissions: Set(); users with modify
> permissions: Set(root); groups with modify permissions: Set()
> spark-thrift_1 | 22/01/24 20:55:07 INFO Utils: Successfully started
> service 'sparkDriver' on port 34759.
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkEnv: Registering
> MapOutputTracker
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkEnv: Registering
> BlockManagerMaster
> spark-thrift_1 | 22/01/24 20:55:07 INFO BlockManagerMasterEndpoint: Using
> org.apache.spark.storage.DefaultTopologyMapper for getting topology
> information
> spark-thrift_1 | 22/01/24 20:55:07 INFO BlockManagerMasterEndpoint:
> BlockManagerMasterEndpoint up
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkEnv: Registering
> BlockManagerMasterHeartbeat
> spark-thrift_1 | 22/01/24 20:55:07 INFO DiskBlockManager: Created local
> directory at /tmp/blockmgr-2f049248-da56-420b-b1ec-2c5fe1e924fc
> spark-thrift_1 | 22/01/24 20:55:07 INFO MemoryStore: MemoryStore started
> with capacity 434.4 MiB
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkEnv: Registering
> OutputCommitCoordinator
> spark-thrift_1 | 22/01/24 20:55:07 INFO Utils: Successfully started
> service 'SparkUI' on port 4040.
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkUI: Bound SparkUI to
> spark-thrift, and started at http://9d3fb159061a:4040
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR
> file:///root/.ivy2/jars/org.apache.sedona_sedona-python-adapter-3.0_2.12-1.1.1-incubating.jar
> at
> spark://9d3fb159061a:34759/jars/org.apache.sedona_sedona-python-adapter-3.0_2.12-1.1.1-incubating.jar
> with timestamp 1643057707087
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR
> file:///root/.ivy2/jars/org.apache.sedona_sedona-viz-3.0_2.12-1.1.1-incubating.jar
> at
> spark://9d3fb159061a:34759/jars/org.apache.sedona_sedona-viz-3.0_2.12-1.1.1-incubating.jar
> with timestamp 1643057707087
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR
> file:///root/.ivy2/jars/org.datasyslab_geotools-wrapper-geotools-24.0.jar at
> spark://9d3fb159061a:34759/jars/org.datasyslab_geotools-wrapper-geotools-24.0.jar
> with timestamp 1643057707087
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR
> file:///root/.ivy2/jars/org.locationtech.jts_jts-core-1.18.0.jar at
> spark://9d3fb159061a:34759/jars/org.locationtech.jts_jts-core-1.18.0.jar with
> timestamp 1643057707087
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR
> file:///root/.ivy2/jars/org.wololo_jts2geojson-0.16.1.jar at
> spark://9d3fb159061a:34759/jars/org.wololo_jts2geojson-0.16.1.jar with
> timestamp 1643057707087
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR
> file:///root/.ivy2/jars/org.apache.sedona_sedona-core-3.0_2.12-1.1.1-incubating.jar
> at
> spark://9d3fb159061a:34759/jars/org.apache.sedona_sedona-core-3.0_2.12-1.1.1-incubating.jar
> with timestamp 1643057707087
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR
> file:///root/.ivy2/jars/org.apache.sedona_sedona-sql-3.0_2.12-1.1.1-incubating.jar
> at
> spark://9d3fb159061a:34759/jars/org.apache.sedona_sedona-sql-3.0_2.12-1.1.1-incubating.jar
> with timestamp 1643057707087
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR
> file:///root/.ivy2/jars/org.scala-lang.modules_scala-collection-compat_2.12-2.5.0.jar
> at
> spark://9d3fb159061a:34759/jars/org.scala-lang.modules_scala-collection-compat_2.12-2.5.0.jar
> with timestamp 1643057707087
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR
> file:///root/.ivy2/jars/com.fasterxml.jackson.core_jackson-databind-2.12.2.jar
> at
> spark://9d3fb159061a:34759/jars/com.fasterxml.jackson.core_jackson-databind-2.12.2.jar
> with timestamp 1643057707087
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR
> file:///root/.ivy2/jars/com.fasterxml.jackson.core_jackson-annotations-2.12.2.jar
> at
> spark://9d3fb159061a:34759/jars/com.fasterxml.jackson.core_jackson-annotations-2.12.2.jar
> with timestamp 1643057707087
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR
> file:///root/.ivy2/jars/com.fasterxml.jackson.core_jackson-core-2.12.2.jar at
> spark://9d3fb159061a:34759/jars/com.fasterxml.jackson.core_jackson-core-2.12.2.jar
> with timestamp 1643057707087
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR
> file:///root/.ivy2/jars/org.beryx_awt-color-factory-1.0.0.jar at
> spark://9d3fb159061a:34759/jars/org.beryx_awt-color-factory-1.0.0.jar with
> timestamp 1643057707087
> spark-thrift_1 | 22/01/24 20:55:07 WARN SparkContext: Please ensure that
> the number of slots available on your executors is limited by the number of
> cores to task cpus and not another custom resource. If cores is not the
> limiting resource then dynamic allocation will not work properly!
> spark-thrift_1 | 22/01/24 20:55:07 INFO
> StandaloneAppClient$ClientEndpoint: Connecting to master
> spark://spark-master:7077...
> spark-thrift_1 | 22/01/24 20:55:07 INFO TransportClientFactory:
> Successfully created connection to spark-master/172.28.0.2:7077 after 24 ms
> (0 ms spent in bootstraps)
> spark-thrift_1 | 22/01/24 20:55:08 INFO StandaloneSchedulerBackend:
> Connected to Spark cluster with app ID app-20220124205508-0004
> spark-thrift_1 | 22/01/24 20:55:08 INFO
> StandaloneAppClient$ClientEndpoint: Executor added: app-20220124205508-0004/0
> on worker-20220124201503-172.28.0.3-7000 (172.28.0.3:7000) with 2 core(s)
> spark-thrift_1 | 22/01/24 20:55:08 INFO StandaloneSchedulerBackend:
> Granted executor ID app-20220124205508-0004/0 on hostPort 172.28.0.3:7000
> with 2 core(s), 1024.0 MiB RAM
> spark-thrift_1 | 22/01/24 20:55:08 INFO
> StandaloneAppClient$ClientEndpoint: Executor added: app-20220124205508-0004/1
> on worker-20220124201513-172.28.0.4-7000 (172.28.0.4:7000) with 2 core(s)
> spark-thrift_1 | 22/01/24 20:55:08 INFO StandaloneSchedulerBackend:
> Granted executor ID app-20220124205508-0004/1 on hostPort 172.28.0.4:7000
> with 2 core(s), 1024.0 MiB RAM
> spark-thrift_1 | 22/01/24 20:55:08 INFO Utils: Successfully started
> service 'org.apache.spark.network.netty.NettyBlockTransferService' on port
> 39909.
> spark-thrift_1 | 22/01/24 20:55:08 INFO NettyBlockTransferService: Server
> created on 9d3fb159061a:39909
> spark-thrift_1 | 22/01/24 20:55:08 INFO BlockManager: Using
> org.apache.spark.storage.RandomBlockReplicationPolicy for block replication
> policy
> spark-thrift_1 | 22/01/24 20:55:08 INFO BlockManagerMaster: Registering
> BlockManager BlockManagerId(driver, 9d3fb159061a, 39909, None)
> spark-thrift_1 | 22/01/24 20:55:08 INFO BlockManagerMasterEndpoint:
> Registering block manager 9d3fb159061a:39909 with 434.4 MiB RAM,
> BlockManagerId(driver, 9d3fb159061a, 39909, None)
> spark-thrift_1 | 22/01/24 20:55:08 INFO BlockManagerMaster: Registered
> BlockManager BlockManagerId(driver, 9d3fb159061a, 39909, None)
> spark-thrift_1 | 22/01/24 20:55:08 INFO BlockManager: Initialized
> BlockManager: BlockManagerId(driver, 9d3fb159061a, 39909, None)
> spark-thrift_1 | 22/01/24 20:55:08 INFO
> StandaloneAppClient$ClientEndpoint: Executor updated:
> app-20220124205508-0004/1 is now RUNNING
> spark-thrift_1 | 22/01/24 20:55:08 INFO
> StandaloneAppClient$ClientEndpoint: Executor updated:
> app-20220124205508-0004/0 is now RUNNING
> spark-thrift_1 | 22/01/24 20:55:08 INFO StandaloneSchedulerBackend:
> SchedulerBackend is ready for scheduling beginning after reached
> minRegisteredResourcesRatio: 0.0
> spark-thrift_1 | 22/01/24 20:55:08 INFO SharedState: Setting
> hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir
> ('file:/opt/spark/spark-warehouse').
> spark-thrift_1 | 22/01/24 20:55:08 INFO SharedState: Warehouse path is
> 'file:/opt/spark/spark-warehouse'.
> spark-thrift_1 | 22/01/24 20:55:08 INFO HiveUtils: Initializing
> HiveMetastoreConnection version 2.3.7 using Spark classes.
> spark-thrift_1 | 22/01/24 20:55:09 INFO SessionState: Created HDFS
> directory: /tmp/hive/root/9a5cefa7-7ee5-46cc-a7ff-2b8bb4eb7ac2
> spark-thrift_1 | 22/01/24 20:55:09 INFO SessionState: Created local
> directory: /tmp/root/9a5cefa7-7ee5-46cc-a7ff-2b8bb4eb7ac2
> spark-thrift_1 | 22/01/24 20:55:09 INFO SessionState: Created HDFS
> directory: /tmp/hive/root/9a5cefa7-7ee5-46cc-a7ff-2b8bb4eb7ac2/_tmp_space.db
> spark-thrift_1 | 22/01/24 20:55:09 INFO HiveClientImpl: Warehouse location
> for Hive client (version 2.3.7) is file:/opt/spark/spark-warehouse
> spark-thrift_1 | 22/01/24 20:55:10 WARN HiveConf: HiveConf of name
> hive.stats.jdbc.timeout does not exist
> spark-thrift_1 | 22/01/24 20:55:10 WARN HiveConf: HiveConf of name
> hive.stats.retries.wait does not exist
> spark-thrift_1 | 22/01/24 20:55:10 INFO HiveMetaStore: 0: Opening raw
> store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
> spark-thrift_1 | 22/01/24 20:55:10 INFO ObjectStore: ObjectStore,
> initialize called
> spark-thrift_1 | 22/01/24 20:55:10 INFO Persistence: Property
> hive.metastore.integral.jdo.pushdown unknown - will be ignored
> spark-thrift_1 | 22/01/24 20:55:10 INFO Persistence: Property
> datanucleus.cache.level2 unknown - will be ignored
> spark-thrift_1 | 22/01/24 20:55:10 INFO ResourceProfile: Default
> ResourceProfile created, executor resources: Map(cores -> name: cores,
> amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script:
> , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
> spark-thrift_1 | 22/01/24 20:55:11 INFO
> CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor
> NettyRpcEndpointRef(spark-client://Executor) (172.28.0.4:46986) with ID 1
> spark-thrift_1 | 22/01/24 20:55:11 INFO
> CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor
> NettyRpcEndpointRef(spark-client://Executor) (172.28.0.3:48170) with ID 0
> spark-thrift_1 | 22/01/24 20:55:11 INFO BlockManagerMasterEndpoint:
> Registering block manager 172.28.0.4:34799 with 434.4 MiB RAM,
> BlockManagerId(1, 172.28.0.4, 34799, None)
> spark-thrift_1 | 22/01/24 20:55:11 INFO BlockManagerMasterEndpoint:
> Registering block manager 172.28.0.3:34961 with 434.4 MiB RAM,
> BlockManagerId(0, 172.28.0.3, 34961, None)
> spark-thrift_1 | 22/01/24 20:55:12 INFO ObjectStore: Setting MetaStore
> object pin classes with
> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
> spark-thrift_1 | 22/01/24 20:55:14 INFO MetaStoreDirectSql: Using direct
> SQL, underlying DB is DERBY
> spark-thrift_1 | 22/01/24 20:55:14 INFO ObjectStore: Initialized
> ObjectStore
> spark-thrift_1 | 22/01/24 20:55:14 WARN ObjectStore: Version information
> not found in metastore. hive.metastore.schema.verification is not enabled so
> recording the schema version 2.3.0
> spark-thrift_1 | 22/01/24 20:55:14 WARN ObjectStore:
> setMetaStoreSchemaVersion called but recording version is disabled: version =
> 2.3.0, comment = Set by MetaStore [email protected]
> spark-thrift_1 | 22/01/24 20:55:14 INFO HiveMetaStore: Added admin role in
> metastore
> spark-thrift_1 | 22/01/24 20:55:14 INFO HiveMetaStore: Added public role
> in metastore
> spark-thrift_1 | 22/01/24 20:55:14 INFO HiveMetaStore: No user is added in
> admin role, since config is empty
> spark-thrift_1 | 22/01/24 20:55:14 INFO HiveMetaStore: 0: get_all_functions
> spark-thrift_1 | 22/01/24 20:55:14 INFO audit: ugi=root
> ip=unknown-ip-addr cmd=get_all_functions
> spark-thrift_1 | 22/01/24 20:55:14 INFO HiveMetaStore: 0: get_database:
> default
> spark-thrift_1 | 22/01/24 20:55:14 INFO audit: ugi=root
> ip=unknown-ip-addr cmd=get_database: default
> spark-thrift_1 | 22/01/24 20:55:14 INFO HiveUtils: Initializing execution
> hive, version 2.3.7
> spark-thrift_1 | 22/01/24 20:55:14 INFO HiveClientImpl: Warehouse location
> for Hive client (version 2.3.7) is file:/opt/spark/spark-warehouse
> spark-thrift_1 | 22/01/24 20:55:14 INFO SessionManager: Operation log root
> directory is created: /tmp/root/operation_logs
> spark-thrift_1 | 22/01/24 20:55:14 INFO SessionManager: HiveServer2:
> Background operation thread pool size: 100
> spark-thrift_1 | 22/01/24 20:55:14 INFO SessionManager: HiveServer2:
> Background operation thread wait queue size: 100
> spark-thrift_1 | 22/01/24 20:55:14 INFO SessionManager: HiveServer2:
> Background operation thread keepalive time: 10 seconds
> spark-thrift_1 | 22/01/24 20:55:14 INFO AbstractService:
> Service:OperationManager is inited.
> spark-thrift_1 | 22/01/24 20:55:14 INFO AbstractService:
> Service:SessionManager is inited.
> spark-thrift_1 | 22/01/24 20:55:14 INFO AbstractService: Service:
> CLIService is inited.
> spark-thrift_1 | 22/01/24 20:55:14 INFO AbstractService:
> Service:ThriftBinaryCLIService is inited.
> spark-thrift_1 | 22/01/24 20:55:14 INFO AbstractService: Service:
> HiveServer2 is inited.
> spark-thrift_1 | 22/01/24 20:55:14 INFO AbstractService:
> Service:OperationManager is started.
> spark-thrift_1 | 22/01/24 20:55:14 INFO AbstractService:
> Service:SessionManager is started.
> spark-thrift_1 | 22/01/24 20:55:14 INFO AbstractService:
> Service:CLIService is started.
> spark-thrift_1 | 22/01/24 20:55:14 INFO ObjectStore: ObjectStore,
> initialize called
> spark-thrift_1 | 22/01/24 20:55:14 INFO MetaStoreDirectSql: Using direct
> SQL, underlying DB is DERBY
> spark-thrift_1 | 22/01/24 20:55:14 INFO ObjectStore: Initialized
> ObjectStore
> spark-thrift_1 | 22/01/24 20:55:14 INFO HiveMetaStore: 0: get_databases:
> default
> spark-thrift_1 | 22/01/24 20:55:14 INFO audit: ugi=root
> ip=unknown-ip-addr cmd=get_databases: default
> spark-thrift_1 | 22/01/24 20:55:14 INFO HiveMetaStore: 0: Cleaning up
> thread local RawStore...
> spark-thrift_1 | 22/01/24 20:55:14 INFO audit: ugi=root
> ip=unknown-ip-addr cmd=Cleaning up thread local RawStore...
> spark-thrift_1 | 22/01/24 20:55:14 INFO HiveMetaStore: 0: Done cleaning up
> thread local RawStore
> spark-thrift_1 | 22/01/24 20:55:14 INFO audit: ugi=root
> ip=unknown-ip-addr cmd=Done cleaning up thread local RawStore
> spark-thrift_1 | 22/01/24 20:55:14 INFO AbstractService:
> Service:ThriftBinaryCLIService is started.
> spark-thrift_1 | 22/01/24 20:55:14 INFO AbstractService:
> Service:HiveServer2 is started.
> spark-thrift_1 | 22/01/24 20:55:14 INFO HiveThriftServer2:
> HiveThriftServer2 started
> spark-thrift_1 | 22/01/24 20:55:14 INFO ThriftCLIService: Starting
> ThriftBinaryCLIService on port 10001 with 5...500 worker threads
> spark-thrift_1 | 22/01/24 20:56:24 INFO ThriftCLIService: Client protocol
> version: HIVE_CLI_SERVICE_PROTOCOL_V8
> spark-thrift_1 | 22/01/24 20:56:24 INFO SessionState: Created HDFS
> directory: /tmp/hive/anonymous/77e79a4b-6753-4d2a-8d46-b1af590f3ab2
> spark-thrift_1 | 22/01/24 20:56:24 INFO SessionState: Created local
> directory: /tmp/root/77e79a4b-6753-4d2a-8d46-b1af590f3ab2
> spark-thrift_1 | 22/01/24 20:56:24 INFO SessionState: Created HDFS
> directory:
> /tmp/hive/anonymous/77e79a4b-6753-4d2a-8d46-b1af590f3ab2/_tmp_space.db
> spark-thrift_1 | 22/01/24 20:56:24 INFO HiveSessionImpl: Operation log
> session directory is created:
> /tmp/root/operation_logs/77e79a4b-6753-4d2a-8d46-b1af590f3ab2
> spark-thrift_1 | Exception in thread "HiveServer2-Handler-Pool: Thread-87"
> java.lang.ExceptionInInitializerError
> spark-thrift_1 | at
> org.apache.sedona.sql.UDF.UdfRegistrator$.registerAll(UdfRegistrator.scala:32)
> spark-thrift_1 | at
> org.apache.sedona.sql.utils.SedonaSQLRegistrator$.registerAll(SedonaSQLRegistrator.scala:34)
> spark-thrift_1 | at
> org.apache.sedona.sql.SedonaSqlExtensions.$anonfun$apply$1(SedonaSqlExtensions.scala:28)
> spark-thrift_1 | at
> org.apache.spark.sql.SparkSessionExtensions.$anonfun$buildCheckRules$1(SparkSessionExtensions.scala:174)
> spark-thrift_1 | at
> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
> spark-thrift_1 | at
> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> spark-thrift_1 | at
> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> spark-thrift_1 | at
> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
> spark-thrift_1 | at
> scala.collection.TraversableLike.map(TraversableLike.scala:238)
> spark-thrift_1 | at
> scala.collection.TraversableLike.map$(TraversableLike.scala:231)
> spark-thrift_1 | at
> scala.collection.AbstractTraversable.map(Traversable.scala:108)
> spark-thrift_1 | at
> org.apache.spark.sql.SparkSessionExtensions.buildCheckRules(SparkSessionExtensions.scala:174)
> spark-thrift_1 | at
> org.apache.spark.sql.internal.BaseSessionStateBuilder.customCheckRules(BaseSessionStateBuilder.scala:227)
> spark-thrift_1 | at
> org.apache.spark.sql.hive.HiveSessionStateBuilder$$anon$1.<init>(HiveSessionStateBuilder.scala:100)
> spark-thrift_1 | at
> org.apache.spark.sql.hive.HiveSessionStateBuilder.analyzer(HiveSessionStateBuilder.scala:74)
> spark-thrift_1 | at
> org.apache.spark.sql.internal.BaseSessionStateBuilder.$anonfun$build$2(BaseSessionStateBuilder.scala:336)
> spark-thrift_1 | at
> org.apache.spark.sql.internal.SessionState.analyzer$lzycompute(SessionState.scala:83)
> spark-thrift_1 | at
> org.apache.spark.sql.internal.SessionState.analyzer(SessionState.scala:83)
> spark-thrift_1 | at
> org.apache.spark.sql.execution.QueryExecution.$anonfun$analyzed$1(QueryExecution.scala:68)
> spark-thrift_1 | at
> org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
> spark-thrift_1 | at
> org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:138)
> spark-thrift_1 | at
> org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
> spark-thrift_1 | at
> org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:138)
> spark-thrift_1 | at
> org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:68)
> spark-thrift_1 | at
> org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:66)
> spark-thrift_1 | at
> org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:58)
> spark-thrift_1 | at
> org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:99)
> spark-thrift_1 | at
> org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
> spark-thrift_1 | at
> org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:97)
> spark-thrift_1 | at
> org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:607)
> spark-thrift_1 | at
> org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
> spark-thrift_1 | at
> org.apache.spark.sql.SparkSession.sql(SparkSession.scala:602)
> spark-thrift_1 | at
> org.apache.spark.sql.SQLContext.sql(SQLContext.scala:650)
> spark-thrift_1 | at
> org.apache.spark.sql.hive.thriftserver.SparkSQLSessionManager.openSession(SparkSQLSessionManager.scala:70)
> spark-thrift_1 | at
> org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:203)
> spark-thrift_1 | at
> org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:353)
> spark-thrift_1 | at
> org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:247)
> spark-thrift_1 | at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1377)
> spark-thrift_1 | at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1362)
> spark-thrift_1 | at
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
> spark-thrift_1 | at
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> spark-thrift_1 | at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:53)
> spark-thrift_1 | at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310)
> spark-thrift_1 | at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> spark-thrift_1 | at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> spark-thrift_1 | at java.base/java.lang.Thread.run(Unknown Source)
> spark-thrift_1 | Caused by: scala.ScalaReflectionException: class
> org.locationtech.jts.geom.Geometry in JavaMirror with
> org.apache.hadoop.hive.ql.exec.UDFClassLoader@5a71c867 of type class
> org.apache.hadoop.hive.ql.exec.UDFClassLoader with classpath [] and parent
> being jdk.internal.loader.ClassLoaders$AppClassLoader@3c153a1 of type class
> jdk.internal.loader.ClassLoaders$AppClassLoader with classpath [<unknown>]
> and parent being
> jdk.internal.loader.ClassLoaders$PlatformClassLoader@435dacd9 of type class
> jdk.internal.loader.ClassLoaders$PlatformClassLoader with classpath
> [<unknown>] and parent being primordial classloader with boot classpath
> [<unknown>] not found.
> spark-thrift_1 | at
> scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:146)
> spark-thrift_1 | at
> scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:29)
> spark-thrift_1 | at
> org.apache.spark.sql.sedona_sql.expressions.TraitSTAggregateExec$$typecreator1$1.apply(AggregateFunctions.scala:42)
> spark-thrift_1 | at
> scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:237)
> spark-thrift_1 | at
> scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:237)
> spark-thrift_1 | at
> org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:54)
> spark-thrift_1 | at
> org.apache.spark.sql.sedona_sql.expressions.TraitSTAggregateExec.$init$(AggregateFunctions.scala:42)
> spark-thrift_1 | at
> org.apache.spark.sql.sedona_sql.expressions.ST_Union_Aggr.<init>(AggregateFunctions.scala:56)
> spark-thrift_1 | at
> org.apache.sedona.sql.UDF.Catalog$.<init>(Catalog.scala:126)
> spark-thrift_1 | at
> org.apache.sedona.sql.UDF.Catalog$.<clinit>(Catalog.scala)
> spark-thrift_1 | ... 46 more
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)