[ 
https://issues.apache.org/jira/browse/SEDONA-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17482163#comment-17482163
 ] 

Jean-Denis Giguère commented on SEDONA-79:
------------------------------------------

Based on a propositon of Paweł Kociński @Imbruced on 
[https://gitter.im/apache/sedona?,] I find an alternative way to create the 
thrift server. The reference proposed by  Paweł is 
[https://geoatscale.com/article?id=b5b1fd23-87b5-4369-8089-3ea1c371a829] 
(redaction in progress).

This is the scala code I used:


{code:java}
import org.apache.sedona.sql.utils.SedonaSQLRegistrator
import org.apache.sedona.viz.core.Serde.SedonaVizKryoRegistrator
import org.apache.spark.serializer.KryoSerializer
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.hive.thriftserver.HiveThriftServer2

object SedonaThrift {

  def main(args: Array[String]): Unit = {
    val sparkSession: SparkSession =
      SparkSession
        .builder()
        .master("local[2]")
        .config("spark.sql.hive.thriftServer.singleSession", "true")
        .config("hive.server2.thrift.bind.host", "0.0.0.0")
        .config("hive.server2.thrift.port", "10001")
        .config("spark.serializer", classOf[KryoSerializer].getName)
        .config("spark.kryo.registrator", 
classOf[SedonaVizKryoRegistrator].getName)
        .appName("SedonaThrift")
        .enableHiveSupport()
        .getOrCreate()

    SedonaSQLRegistrator.registerAll(sparkSession)

    HiveThriftServer2.startWithContext(sparkSession.sqlContext)

  }

}
{code}
 

 

> Pure SQL does not work with Thrift Server
> -----------------------------------------
>
>                 Key: SEDONA-79
>                 URL: https://issues.apache.org/jira/browse/SEDONA-79
>             Project: Apache Sedona
>          Issue Type: Bug
>            Reporter: Jean-Denis Giguère
>            Priority: Minor
>              Labels: sql
>
> h2. Current situation
> When I add Sedona SQL support to the Thrift Server on a local Spark Cluster, 
> a {{scala.ScalaReflectionException}} associated with the class 
> {{org.locationtech.jts.geom.Geometry}} is triggered.
> {code:bash}
> /opt/spark/bin/spark-submit --class 
> org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 \
>     --master $SPARK_MASTER  \
>     --hiveconf hive.server2.thrift.port=10001 --hiveconf 
> hive.server2.thrift.bind.host=0.0.0.0 \
>     --packages 
> org.apache.sedona:sedona-python-adapter-3.0_2.12:1.1.1-incubating,org.apache.sedona:sedona-viz-3.0_2.12:1.1.1-incubating,org.datasyslab:geotools-wrapper:geotools-24.0
>  \
>     --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
>     --conf 
> spark.kryo.registrator=org.apache.sedona.viz.core.Serde.SedonaVizKryoRegistrator
>  \
>     --conf 
> spark.sql.extensions=org.apache.sedona.viz.sql.SedonaVizExtensions,org.apache.sedona.sql.SedonaSqlExtensions
> {code}
> The error is not triggered when I launch the server, but when I attempt to 
> connect to it with the  Apache Hive JDBC driver. 
> This error is not present when I use {{spark-shell}} with the same parameter.
> h2.  Expected situation
> I would expect to be able to start the thrift server to expose sedona SQL on 
> a standalone cluster.
> h2. Logs
> {noformat}
> docker-compose up spark-thrift
> WARNING: The Docker Engine you're using is running in swarm mode.
> Compose does not use swarm mode to deploy services to multiple nodes in a 
> swarm. All containers will be scheduled on the current node.
> To deploy your application across the swarm, use `docker stack deploy`.
> spark-dbt_spark-master_1 is up-to-date
> spark-dbt_spark-worker-b_1 is up-to-date
> spark-dbt_spark-worker-a_1 is up-to-date
> Starting spark-dbt_spark-thrift_1 ... done
> Attaching to spark-dbt_spark-thrift_1
> spark-thrift_1    | WARNING: An illegal reflective access operation has 
> occurred
> spark-thrift_1    | WARNING: Illegal reflective access by 
> org.apache.spark.unsafe.Platform 
> (file:/opt/spark/jars/spark-unsafe_2.12-3.0.2.jar) to constructor 
> java.nio.DirectByteBuffer(long,int)
> spark-thrift_1    | WARNING: Please consider reporting this to the 
> maintainers of org.apache.spark.unsafe.Platform
> spark-thrift_1    | WARNING: Use --illegal-access=warn to enable warnings of 
> further illegal reflective access operations
> spark-thrift_1    | WARNING: All illegal access operations will be denied in 
> a future release
> spark-thrift_1    | Ivy Default Cache set to: /root/.ivy2/cache
> spark-thrift_1    | The jars for the packages stored in: /root/.ivy2/jars
> spark-thrift_1    | :: loading settings :: url = 
> jar:file:/opt/spark/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
> spark-thrift_1    | org.apache.sedona#sedona-python-adapter-3.0_2.12 added as 
> a dependency
> spark-thrift_1    | org.apache.sedona#sedona-viz-3.0_2.12 added as a 
> dependency
> spark-thrift_1    | org.datasyslab#geotools-wrapper added as a dependency
> spark-thrift_1    | :: resolving dependencies :: 
> org.apache.spark#spark-submit-parent-56248f53-61ad-4c72-bd1f-c13739ab99c0;1.0
> spark-thrift_1    |   confs: [default]
> spark-thrift_1    |   found 
> org.apache.sedona#sedona-python-adapter-3.0_2.12;1.1.1-incubating in central
> spark-thrift_1    |   found org.locationtech.jts#jts-core;1.18.0 in central
> spark-thrift_1    |   found org.wololo#jts2geojson;0.16.1 in central
> spark-thrift_1    |   found 
> com.fasterxml.jackson.core#jackson-databind;2.12.2 in central
> spark-thrift_1    |   found 
> com.fasterxml.jackson.core#jackson-annotations;2.12.2 in central
> spark-thrift_1    |   found com.fasterxml.jackson.core#jackson-core;2.12.2 in 
> central
> spark-thrift_1    |   found 
> org.apache.sedona#sedona-core-3.0_2.12;1.1.1-incubating in central
> spark-thrift_1    |   found 
> org.scala-lang.modules#scala-collection-compat_2.12;2.5.0 in central
> spark-thrift_1    |   found 
> org.apache.sedona#sedona-sql-3.0_2.12;1.1.1-incubating in central
> spark-thrift_1    |   found 
> org.apache.sedona#sedona-viz-3.0_2.12;1.1.1-incubating in central
> spark-thrift_1    |   found org.beryx#awt-color-factory;1.0.0 in central
> spark-thrift_1    |   found org.datasyslab#geotools-wrapper;geotools-24.0 in 
> central
> spark-thrift_1    | :: resolution report :: resolve 533ms :: artifacts dl 15ms
> spark-thrift_1    |   :: modules in use:
> spark-thrift_1    |   com.fasterxml.jackson.core#jackson-annotations;2.12.2 
> from central in [default]
> spark-thrift_1    |   com.fasterxml.jackson.core#jackson-core;2.12.2 from 
> central in [default]
> spark-thrift_1    |   com.fasterxml.jackson.core#jackson-databind;2.12.2 from 
> central in [default]
> spark-thrift_1    |   org.apache.sedona#sedona-core-3.0_2.12;1.1.1-incubating 
> from central in [default]
> spark-thrift_1    |   
> org.apache.sedona#sedona-python-adapter-3.0_2.12;1.1.1-incubating from 
> central in [default]
> spark-thrift_1    |   org.apache.sedona#sedona-sql-3.0_2.12;1.1.1-incubating 
> from central in [default]
> spark-thrift_1    |   org.apache.sedona#sedona-viz-3.0_2.12;1.1.1-incubating 
> from central in [default]
> spark-thrift_1    |   org.beryx#awt-color-factory;1.0.0 from central in 
> [default]
> spark-thrift_1    |   org.datasyslab#geotools-wrapper;geotools-24.0 from 
> central in [default]
> spark-thrift_1    |   org.locationtech.jts#jts-core;1.18.0 from central in 
> [default]
> spark-thrift_1    |   
> org.scala-lang.modules#scala-collection-compat_2.12;2.5.0 from central in 
> [default]
> spark-thrift_1    |   org.wololo#jts2geojson;0.16.1 from central in [default]
> spark-thrift_1    |   :: evicted modules:
> spark-thrift_1    |   org.locationtech.jts#jts-core;1.18.1 by 
> [org.locationtech.jts#jts-core;1.18.0] in [default]
> spark-thrift_1    |   
> ---------------------------------------------------------------------
> spark-thrift_1    |   |                  |            modules            ||   
> artifacts   |
> spark-thrift_1    |   |       conf       | number| search|dwnlded|evicted|| 
> number|dwnlded|
> spark-thrift_1    |   
> ---------------------------------------------------------------------
> spark-thrift_1    |   |      default     |   13  |   0   |   0   |   1   ||   
> 12  |   0   |
> spark-thrift_1    |   
> ---------------------------------------------------------------------
> spark-thrift_1    | :: retrieving :: 
> org.apache.spark#spark-submit-parent-56248f53-61ad-4c72-bd1f-c13739ab99c0
> spark-thrift_1    |   confs: [default]
> spark-thrift_1    |   0 artifacts copied, 12 already retrieved (0kB/13ms)
> spark-thrift_1    | 22/01/24 20:55:06 WARN NativeCodeLoader: Unable to load 
> native-hadoop library for your platform... using builtin-java classes where 
> applicable
> spark-thrift_1    | Using Spark's default log4j profile: 
> org/apache/spark/log4j-defaults.properties
> spark-thrift_1    | 22/01/24 20:55:06 INFO HiveThriftServer2: Started daemon 
> with process name: 7@9d3fb159061a
> spark-thrift_1    | 22/01/24 20:55:06 INFO SignalUtils: Registered signal 
> handler for TERM
> spark-thrift_1    | 22/01/24 20:55:06 INFO SignalUtils: Registered signal 
> handler for HUP
> spark-thrift_1    | 22/01/24 20:55:06 INFO SignalUtils: Registered signal 
> handler for INT
> spark-thrift_1    | 22/01/24 20:55:06 INFO HiveThriftServer2: Starting 
> SparkContext
> spark-thrift_1    | 22/01/24 20:55:06 INFO HiveConf: Found configuration file 
> null
> spark-thrift_1    | 22/01/24 20:55:07 INFO SparkContext: Running Spark 
> version 3.0.2
> spark-thrift_1    | 22/01/24 20:55:07 INFO ResourceUtils: 
> ==============================================================
> spark-thrift_1    | 22/01/24 20:55:07 INFO ResourceUtils: Resources for 
> spark.driver:
> spark-thrift_1    | 
> spark-thrift_1    | 22/01/24 20:55:07 INFO ResourceUtils: 
> ==============================================================
> spark-thrift_1    | 22/01/24 20:55:07 INFO SparkContext: Submitted 
> application: SparkSQL::172.28.0.5
> spark-thrift_1    | 22/01/24 20:55:07 INFO SecurityManager: Changing view 
> acls to: root
> spark-thrift_1    | 22/01/24 20:55:07 INFO SecurityManager: Changing modify 
> acls to: root
> spark-thrift_1    | 22/01/24 20:55:07 INFO SecurityManager: Changing view 
> acls groups to: 
> spark-thrift_1    | 22/01/24 20:55:07 INFO SecurityManager: Changing modify 
> acls groups to: 
> spark-thrift_1    | 22/01/24 20:55:07 INFO SecurityManager: SecurityManager: 
> authentication disabled; ui acls disabled; users  with view permissions: 
> Set(root); groups with view permissions: Set(); users  with modify 
> permissions: Set(root); groups with modify permissions: Set()
> spark-thrift_1    | 22/01/24 20:55:07 INFO Utils: Successfully started 
> service 'sparkDriver' on port 34759.
> spark-thrift_1    | 22/01/24 20:55:07 INFO SparkEnv: Registering 
> MapOutputTracker
> spark-thrift_1    | 22/01/24 20:55:07 INFO SparkEnv: Registering 
> BlockManagerMaster
> spark-thrift_1    | 22/01/24 20:55:07 INFO BlockManagerMasterEndpoint: Using 
> org.apache.spark.storage.DefaultTopologyMapper for getting topology 
> information
> spark-thrift_1    | 22/01/24 20:55:07 INFO BlockManagerMasterEndpoint: 
> BlockManagerMasterEndpoint up
> spark-thrift_1    | 22/01/24 20:55:07 INFO SparkEnv: Registering 
> BlockManagerMasterHeartbeat
> spark-thrift_1    | 22/01/24 20:55:07 INFO DiskBlockManager: Created local 
> directory at /tmp/blockmgr-2f049248-da56-420b-b1ec-2c5fe1e924fc
> spark-thrift_1    | 22/01/24 20:55:07 INFO MemoryStore: MemoryStore started 
> with capacity 434.4 MiB
> spark-thrift_1    | 22/01/24 20:55:07 INFO SparkEnv: Registering 
> OutputCommitCoordinator
> spark-thrift_1    | 22/01/24 20:55:07 INFO Utils: Successfully started 
> service 'SparkUI' on port 4040.
> spark-thrift_1    | 22/01/24 20:55:07 INFO SparkUI: Bound SparkUI to 
> spark-thrift, and started at http://9d3fb159061a:4040
> spark-thrift_1    | 22/01/24 20:55:07 INFO SparkContext: Added JAR 
> file:///root/.ivy2/jars/org.apache.sedona_sedona-python-adapter-3.0_2.12-1.1.1-incubating.jar
>  at 
> spark://9d3fb159061a:34759/jars/org.apache.sedona_sedona-python-adapter-3.0_2.12-1.1.1-incubating.jar
>  with timestamp 1643057707087
> spark-thrift_1    | 22/01/24 20:55:07 INFO SparkContext: Added JAR 
> file:///root/.ivy2/jars/org.apache.sedona_sedona-viz-3.0_2.12-1.1.1-incubating.jar
>  at 
> spark://9d3fb159061a:34759/jars/org.apache.sedona_sedona-viz-3.0_2.12-1.1.1-incubating.jar
>  with timestamp 1643057707087
> spark-thrift_1    | 22/01/24 20:55:07 INFO SparkContext: Added JAR 
> file:///root/.ivy2/jars/org.datasyslab_geotools-wrapper-geotools-24.0.jar at 
> spark://9d3fb159061a:34759/jars/org.datasyslab_geotools-wrapper-geotools-24.0.jar
>  with timestamp 1643057707087
> spark-thrift_1    | 22/01/24 20:55:07 INFO SparkContext: Added JAR 
> file:///root/.ivy2/jars/org.locationtech.jts_jts-core-1.18.0.jar at 
> spark://9d3fb159061a:34759/jars/org.locationtech.jts_jts-core-1.18.0.jar with 
> timestamp 1643057707087
> spark-thrift_1    | 22/01/24 20:55:07 INFO SparkContext: Added JAR 
> file:///root/.ivy2/jars/org.wololo_jts2geojson-0.16.1.jar at 
> spark://9d3fb159061a:34759/jars/org.wololo_jts2geojson-0.16.1.jar with 
> timestamp 1643057707087
> spark-thrift_1    | 22/01/24 20:55:07 INFO SparkContext: Added JAR 
> file:///root/.ivy2/jars/org.apache.sedona_sedona-core-3.0_2.12-1.1.1-incubating.jar
>  at 
> spark://9d3fb159061a:34759/jars/org.apache.sedona_sedona-core-3.0_2.12-1.1.1-incubating.jar
>  with timestamp 1643057707087
> spark-thrift_1    | 22/01/24 20:55:07 INFO SparkContext: Added JAR 
> file:///root/.ivy2/jars/org.apache.sedona_sedona-sql-3.0_2.12-1.1.1-incubating.jar
>  at 
> spark://9d3fb159061a:34759/jars/org.apache.sedona_sedona-sql-3.0_2.12-1.1.1-incubating.jar
>  with timestamp 1643057707087
> spark-thrift_1    | 22/01/24 20:55:07 INFO SparkContext: Added JAR 
> file:///root/.ivy2/jars/org.scala-lang.modules_scala-collection-compat_2.12-2.5.0.jar
>  at 
> spark://9d3fb159061a:34759/jars/org.scala-lang.modules_scala-collection-compat_2.12-2.5.0.jar
>  with timestamp 1643057707087
> spark-thrift_1    | 22/01/24 20:55:07 INFO SparkContext: Added JAR 
> file:///root/.ivy2/jars/com.fasterxml.jackson.core_jackson-databind-2.12.2.jar
>  at 
> spark://9d3fb159061a:34759/jars/com.fasterxml.jackson.core_jackson-databind-2.12.2.jar
>  with timestamp 1643057707087
> spark-thrift_1    | 22/01/24 20:55:07 INFO SparkContext: Added JAR 
> file:///root/.ivy2/jars/com.fasterxml.jackson.core_jackson-annotations-2.12.2.jar
>  at 
> spark://9d3fb159061a:34759/jars/com.fasterxml.jackson.core_jackson-annotations-2.12.2.jar
>  with timestamp 1643057707087
> spark-thrift_1    | 22/01/24 20:55:07 INFO SparkContext: Added JAR 
> file:///root/.ivy2/jars/com.fasterxml.jackson.core_jackson-core-2.12.2.jar at 
> spark://9d3fb159061a:34759/jars/com.fasterxml.jackson.core_jackson-core-2.12.2.jar
>  with timestamp 1643057707087
> spark-thrift_1    | 22/01/24 20:55:07 INFO SparkContext: Added JAR 
> file:///root/.ivy2/jars/org.beryx_awt-color-factory-1.0.0.jar at 
> spark://9d3fb159061a:34759/jars/org.beryx_awt-color-factory-1.0.0.jar with 
> timestamp 1643057707087
> spark-thrift_1    | 22/01/24 20:55:07 WARN SparkContext: Please ensure that 
> the number of slots available on your executors is limited by the number of 
> cores to task cpus and not another custom resource. If cores is not the 
> limiting resource then dynamic allocation will not work properly!
> spark-thrift_1    | 22/01/24 20:55:07 INFO 
> StandaloneAppClient$ClientEndpoint: Connecting to master 
> spark://spark-master:7077...
> spark-thrift_1    | 22/01/24 20:55:07 INFO TransportClientFactory: 
> Successfully created connection to spark-master/172.28.0.2:7077 after 24 ms 
> (0 ms spent in bootstraps)
> spark-thrift_1    | 22/01/24 20:55:08 INFO StandaloneSchedulerBackend: 
> Connected to Spark cluster with app ID app-20220124205508-0004
> spark-thrift_1    | 22/01/24 20:55:08 INFO 
> StandaloneAppClient$ClientEndpoint: Executor added: app-20220124205508-0004/0 
> on worker-20220124201503-172.28.0.3-7000 (172.28.0.3:7000) with 2 core(s)
> spark-thrift_1    | 22/01/24 20:55:08 INFO StandaloneSchedulerBackend: 
> Granted executor ID app-20220124205508-0004/0 on hostPort 172.28.0.3:7000 
> with 2 core(s), 1024.0 MiB RAM
> spark-thrift_1    | 22/01/24 20:55:08 INFO 
> StandaloneAppClient$ClientEndpoint: Executor added: app-20220124205508-0004/1 
> on worker-20220124201513-172.28.0.4-7000 (172.28.0.4:7000) with 2 core(s)
> spark-thrift_1    | 22/01/24 20:55:08 INFO StandaloneSchedulerBackend: 
> Granted executor ID app-20220124205508-0004/1 on hostPort 172.28.0.4:7000 
> with 2 core(s), 1024.0 MiB RAM
> spark-thrift_1    | 22/01/24 20:55:08 INFO Utils: Successfully started 
> service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 
> 39909.
> spark-thrift_1    | 22/01/24 20:55:08 INFO NettyBlockTransferService: Server 
> created on 9d3fb159061a:39909
> spark-thrift_1    | 22/01/24 20:55:08 INFO BlockManager: Using 
> org.apache.spark.storage.RandomBlockReplicationPolicy for block replication 
> policy
> spark-thrift_1    | 22/01/24 20:55:08 INFO BlockManagerMaster: Registering 
> BlockManager BlockManagerId(driver, 9d3fb159061a, 39909, None)
> spark-thrift_1    | 22/01/24 20:55:08 INFO BlockManagerMasterEndpoint: 
> Registering block manager 9d3fb159061a:39909 with 434.4 MiB RAM, 
> BlockManagerId(driver, 9d3fb159061a, 39909, None)
> spark-thrift_1    | 22/01/24 20:55:08 INFO BlockManagerMaster: Registered 
> BlockManager BlockManagerId(driver, 9d3fb159061a, 39909, None)
> spark-thrift_1    | 22/01/24 20:55:08 INFO BlockManager: Initialized 
> BlockManager: BlockManagerId(driver, 9d3fb159061a, 39909, None)
> spark-thrift_1    | 22/01/24 20:55:08 INFO 
> StandaloneAppClient$ClientEndpoint: Executor updated: 
> app-20220124205508-0004/1 is now RUNNING
> spark-thrift_1    | 22/01/24 20:55:08 INFO 
> StandaloneAppClient$ClientEndpoint: Executor updated: 
> app-20220124205508-0004/0 is now RUNNING
> spark-thrift_1    | 22/01/24 20:55:08 INFO StandaloneSchedulerBackend: 
> SchedulerBackend is ready for scheduling beginning after reached 
> minRegisteredResourcesRatio: 0.0
> spark-thrift_1    | 22/01/24 20:55:08 INFO SharedState: Setting 
> hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir 
> ('file:/opt/spark/spark-warehouse').
> spark-thrift_1    | 22/01/24 20:55:08 INFO SharedState: Warehouse path is 
> 'file:/opt/spark/spark-warehouse'.
> spark-thrift_1    | 22/01/24 20:55:08 INFO HiveUtils: Initializing 
> HiveMetastoreConnection version 2.3.7 using Spark classes.
> spark-thrift_1    | 22/01/24 20:55:09 INFO SessionState: Created HDFS 
> directory: /tmp/hive/root/9a5cefa7-7ee5-46cc-a7ff-2b8bb4eb7ac2
> spark-thrift_1    | 22/01/24 20:55:09 INFO SessionState: Created local 
> directory: /tmp/root/9a5cefa7-7ee5-46cc-a7ff-2b8bb4eb7ac2
> spark-thrift_1    | 22/01/24 20:55:09 INFO SessionState: Created HDFS 
> directory: /tmp/hive/root/9a5cefa7-7ee5-46cc-a7ff-2b8bb4eb7ac2/_tmp_space.db
> spark-thrift_1    | 22/01/24 20:55:09 INFO HiveClientImpl: Warehouse location 
> for Hive client (version 2.3.7) is file:/opt/spark/spark-warehouse
> spark-thrift_1    | 22/01/24 20:55:10 WARN HiveConf: HiveConf of name 
> hive.stats.jdbc.timeout does not exist
> spark-thrift_1    | 22/01/24 20:55:10 WARN HiveConf: HiveConf of name 
> hive.stats.retries.wait does not exist
> spark-thrift_1    | 22/01/24 20:55:10 INFO HiveMetaStore: 0: Opening raw 
> store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
> spark-thrift_1    | 22/01/24 20:55:10 INFO ObjectStore: ObjectStore, 
> initialize called
> spark-thrift_1    | 22/01/24 20:55:10 INFO Persistence: Property 
> hive.metastore.integral.jdo.pushdown unknown - will be ignored
> spark-thrift_1    | 22/01/24 20:55:10 INFO Persistence: Property 
> datanucleus.cache.level2 unknown - will be ignored
> spark-thrift_1    | 22/01/24 20:55:10 INFO ResourceProfile: Default 
> ResourceProfile created, executor resources: Map(cores -> name: cores, 
> amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: 
> , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
> spark-thrift_1    | 22/01/24 20:55:11 INFO 
> CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor 
> NettyRpcEndpointRef(spark-client://Executor) (172.28.0.4:46986) with ID 1
> spark-thrift_1    | 22/01/24 20:55:11 INFO 
> CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor 
> NettyRpcEndpointRef(spark-client://Executor) (172.28.0.3:48170) with ID 0
> spark-thrift_1    | 22/01/24 20:55:11 INFO BlockManagerMasterEndpoint: 
> Registering block manager 172.28.0.4:34799 with 434.4 MiB RAM, 
> BlockManagerId(1, 172.28.0.4, 34799, None)
> spark-thrift_1    | 22/01/24 20:55:11 INFO BlockManagerMasterEndpoint: 
> Registering block manager 172.28.0.3:34961 with 434.4 MiB RAM, 
> BlockManagerId(0, 172.28.0.3, 34961, None)
> spark-thrift_1    | 22/01/24 20:55:12 INFO ObjectStore: Setting MetaStore 
> object pin classes with 
> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
> spark-thrift_1    | 22/01/24 20:55:14 INFO MetaStoreDirectSql: Using direct 
> SQL, underlying DB is DERBY
> spark-thrift_1    | 22/01/24 20:55:14 INFO ObjectStore: Initialized 
> ObjectStore
> spark-thrift_1    | 22/01/24 20:55:14 WARN ObjectStore: Version information 
> not found in metastore. hive.metastore.schema.verification is not enabled so 
> recording the schema version 2.3.0
> spark-thrift_1    | 22/01/24 20:55:14 WARN ObjectStore: 
> setMetaStoreSchemaVersion called but recording version is disabled: version = 
> 2.3.0, comment = Set by MetaStore [email protected]
> spark-thrift_1    | 22/01/24 20:55:14 INFO HiveMetaStore: Added admin role in 
> metastore
> spark-thrift_1    | 22/01/24 20:55:14 INFO HiveMetaStore: Added public role 
> in metastore
> spark-thrift_1    | 22/01/24 20:55:14 INFO HiveMetaStore: No user is added in 
> admin role, since config is empty
> spark-thrift_1    | 22/01/24 20:55:14 INFO HiveMetaStore: 0: get_all_functions
> spark-thrift_1    | 22/01/24 20:55:14 INFO audit: ugi=root    
> ip=unknown-ip-addr      cmd=get_all_functions   
> spark-thrift_1    | 22/01/24 20:55:14 INFO HiveMetaStore: 0: get_database: 
> default
> spark-thrift_1    | 22/01/24 20:55:14 INFO audit: ugi=root    
> ip=unknown-ip-addr      cmd=get_database: default       
> spark-thrift_1    | 22/01/24 20:55:14 INFO HiveUtils: Initializing execution 
> hive, version 2.3.7
> spark-thrift_1    | 22/01/24 20:55:14 INFO HiveClientImpl: Warehouse location 
> for Hive client (version 2.3.7) is file:/opt/spark/spark-warehouse
> spark-thrift_1    | 22/01/24 20:55:14 INFO SessionManager: Operation log root 
> directory is created: /tmp/root/operation_logs
> spark-thrift_1    | 22/01/24 20:55:14 INFO SessionManager: HiveServer2: 
> Background operation thread pool size: 100
> spark-thrift_1    | 22/01/24 20:55:14 INFO SessionManager: HiveServer2: 
> Background operation thread wait queue size: 100
> spark-thrift_1    | 22/01/24 20:55:14 INFO SessionManager: HiveServer2: 
> Background operation thread keepalive time: 10 seconds
> spark-thrift_1    | 22/01/24 20:55:14 INFO AbstractService: 
> Service:OperationManager is inited.
> spark-thrift_1    | 22/01/24 20:55:14 INFO AbstractService: 
> Service:SessionManager is inited.
> spark-thrift_1    | 22/01/24 20:55:14 INFO AbstractService: Service: 
> CLIService is inited.
> spark-thrift_1    | 22/01/24 20:55:14 INFO AbstractService: 
> Service:ThriftBinaryCLIService is inited.
> spark-thrift_1    | 22/01/24 20:55:14 INFO AbstractService: Service: 
> HiveServer2 is inited.
> spark-thrift_1    | 22/01/24 20:55:14 INFO AbstractService: 
> Service:OperationManager is started.
> spark-thrift_1    | 22/01/24 20:55:14 INFO AbstractService: 
> Service:SessionManager is started.
> spark-thrift_1    | 22/01/24 20:55:14 INFO AbstractService: 
> Service:CLIService is started.
> spark-thrift_1    | 22/01/24 20:55:14 INFO ObjectStore: ObjectStore, 
> initialize called
> spark-thrift_1    | 22/01/24 20:55:14 INFO MetaStoreDirectSql: Using direct 
> SQL, underlying DB is DERBY
> spark-thrift_1    | 22/01/24 20:55:14 INFO ObjectStore: Initialized 
> ObjectStore
> spark-thrift_1    | 22/01/24 20:55:14 INFO HiveMetaStore: 0: get_databases: 
> default
> spark-thrift_1    | 22/01/24 20:55:14 INFO audit: ugi=root    
> ip=unknown-ip-addr      cmd=get_databases: default      
> spark-thrift_1    | 22/01/24 20:55:14 INFO HiveMetaStore: 0: Cleaning up 
> thread local RawStore...
> spark-thrift_1    | 22/01/24 20:55:14 INFO audit: ugi=root    
> ip=unknown-ip-addr      cmd=Cleaning up thread local RawStore...        
> spark-thrift_1    | 22/01/24 20:55:14 INFO HiveMetaStore: 0: Done cleaning up 
> thread local RawStore
> spark-thrift_1    | 22/01/24 20:55:14 INFO audit: ugi=root    
> ip=unknown-ip-addr      cmd=Done cleaning up thread local RawStore      
> spark-thrift_1    | 22/01/24 20:55:14 INFO AbstractService: 
> Service:ThriftBinaryCLIService is started.
> spark-thrift_1    | 22/01/24 20:55:14 INFO AbstractService: 
> Service:HiveServer2 is started.
> spark-thrift_1    | 22/01/24 20:55:14 INFO HiveThriftServer2: 
> HiveThriftServer2 started
> spark-thrift_1    | 22/01/24 20:55:14 INFO ThriftCLIService: Starting 
> ThriftBinaryCLIService on port 10001 with 5...500 worker threads
> spark-thrift_1    | 22/01/24 20:56:24 INFO ThriftCLIService: Client protocol 
> version: HIVE_CLI_SERVICE_PROTOCOL_V8
> spark-thrift_1    | 22/01/24 20:56:24 INFO SessionState: Created HDFS 
> directory: /tmp/hive/anonymous/77e79a4b-6753-4d2a-8d46-b1af590f3ab2
> spark-thrift_1    | 22/01/24 20:56:24 INFO SessionState: Created local 
> directory: /tmp/root/77e79a4b-6753-4d2a-8d46-b1af590f3ab2
> spark-thrift_1    | 22/01/24 20:56:24 INFO SessionState: Created HDFS 
> directory: 
> /tmp/hive/anonymous/77e79a4b-6753-4d2a-8d46-b1af590f3ab2/_tmp_space.db
> spark-thrift_1    | 22/01/24 20:56:24 INFO HiveSessionImpl: Operation log 
> session directory is created: 
> /tmp/root/operation_logs/77e79a4b-6753-4d2a-8d46-b1af590f3ab2
> spark-thrift_1    | Exception in thread "HiveServer2-Handler-Pool: Thread-87" 
> java.lang.ExceptionInInitializerError
> spark-thrift_1    |   at 
> org.apache.sedona.sql.UDF.UdfRegistrator$.registerAll(UdfRegistrator.scala:32)
> spark-thrift_1    |   at 
> org.apache.sedona.sql.utils.SedonaSQLRegistrator$.registerAll(SedonaSQLRegistrator.scala:34)
> spark-thrift_1    |   at 
> org.apache.sedona.sql.SedonaSqlExtensions.$anonfun$apply$1(SedonaSqlExtensions.scala:28)
> spark-thrift_1    |   at 
> org.apache.spark.sql.SparkSessionExtensions.$anonfun$buildCheckRules$1(SparkSessionExtensions.scala:174)
> spark-thrift_1    |   at 
> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
> spark-thrift_1    |   at 
> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> spark-thrift_1    |   at 
> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> spark-thrift_1    |   at 
> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
> spark-thrift_1    |   at 
> scala.collection.TraversableLike.map(TraversableLike.scala:238)
> spark-thrift_1    |   at 
> scala.collection.TraversableLike.map$(TraversableLike.scala:231)
> spark-thrift_1    |   at 
> scala.collection.AbstractTraversable.map(Traversable.scala:108)
> spark-thrift_1    |   at 
> org.apache.spark.sql.SparkSessionExtensions.buildCheckRules(SparkSessionExtensions.scala:174)
> spark-thrift_1    |   at 
> org.apache.spark.sql.internal.BaseSessionStateBuilder.customCheckRules(BaseSessionStateBuilder.scala:227)
> spark-thrift_1    |   at 
> org.apache.spark.sql.hive.HiveSessionStateBuilder$$anon$1.<init>(HiveSessionStateBuilder.scala:100)
> spark-thrift_1    |   at 
> org.apache.spark.sql.hive.HiveSessionStateBuilder.analyzer(HiveSessionStateBuilder.scala:74)
> spark-thrift_1    |   at 
> org.apache.spark.sql.internal.BaseSessionStateBuilder.$anonfun$build$2(BaseSessionStateBuilder.scala:336)
> spark-thrift_1    |   at 
> org.apache.spark.sql.internal.SessionState.analyzer$lzycompute(SessionState.scala:83)
> spark-thrift_1    |   at 
> org.apache.spark.sql.internal.SessionState.analyzer(SessionState.scala:83)
> spark-thrift_1    |   at 
> org.apache.spark.sql.execution.QueryExecution.$anonfun$analyzed$1(QueryExecution.scala:68)
> spark-thrift_1    |   at 
> org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
> spark-thrift_1    |   at 
> org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:138)
> spark-thrift_1    |   at 
> org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
> spark-thrift_1    |   at 
> org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:138)
> spark-thrift_1    |   at 
> org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:68)
> spark-thrift_1    |   at 
> org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:66)
> spark-thrift_1    |   at 
> org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:58)
> spark-thrift_1    |   at 
> org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:99)
> spark-thrift_1    |   at 
> org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
> spark-thrift_1    |   at 
> org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:97)
> spark-thrift_1    |   at 
> org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:607)
> spark-thrift_1    |   at 
> org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
> spark-thrift_1    |   at 
> org.apache.spark.sql.SparkSession.sql(SparkSession.scala:602)
> spark-thrift_1    |   at 
> org.apache.spark.sql.SQLContext.sql(SQLContext.scala:650)
> spark-thrift_1    |   at 
> org.apache.spark.sql.hive.thriftserver.SparkSQLSessionManager.openSession(SparkSQLSessionManager.scala:70)
> spark-thrift_1    |   at 
> org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:203)
> spark-thrift_1    |   at 
> org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:353)
> spark-thrift_1    |   at 
> org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:247)
> spark-thrift_1    |   at 
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1377)
> spark-thrift_1    |   at 
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1362)
> spark-thrift_1    |   at 
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
> spark-thrift_1    |   at 
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> spark-thrift_1    |   at 
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:53)
> spark-thrift_1    |   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310)
> spark-thrift_1    |   at 
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> spark-thrift_1    |   at 
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> spark-thrift_1    |   at java.base/java.lang.Thread.run(Unknown Source)
> spark-thrift_1    | Caused by: scala.ScalaReflectionException: class 
> org.locationtech.jts.geom.Geometry in JavaMirror with 
> org.apache.hadoop.hive.ql.exec.UDFClassLoader@5a71c867 of type class 
> org.apache.hadoop.hive.ql.exec.UDFClassLoader with classpath [] and parent 
> being jdk.internal.loader.ClassLoaders$AppClassLoader@3c153a1 of type class 
> jdk.internal.loader.ClassLoaders$AppClassLoader with classpath [<unknown>] 
> and parent being 
> jdk.internal.loader.ClassLoaders$PlatformClassLoader@435dacd9 of type class 
> jdk.internal.loader.ClassLoaders$PlatformClassLoader with classpath 
> [<unknown>] and parent being primordial classloader with boot classpath 
> [<unknown>] not found.
> spark-thrift_1    |   at 
> scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:146)
> spark-thrift_1    |   at 
> scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:29)
> spark-thrift_1    |   at 
> org.apache.spark.sql.sedona_sql.expressions.TraitSTAggregateExec$$typecreator1$1.apply(AggregateFunctions.scala:42)
> spark-thrift_1    |   at 
> scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:237)
> spark-thrift_1    |   at 
> scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:237)
> spark-thrift_1    |   at 
> org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:54)
> spark-thrift_1    |   at 
> org.apache.spark.sql.sedona_sql.expressions.TraitSTAggregateExec.$init$(AggregateFunctions.scala:42)
> spark-thrift_1    |   at 
> org.apache.spark.sql.sedona_sql.expressions.ST_Union_Aggr.<init>(AggregateFunctions.scala:56)
> spark-thrift_1    |   at 
> org.apache.sedona.sql.UDF.Catalog$.<init>(Catalog.scala:126)
> spark-thrift_1    |   at 
> org.apache.sedona.sql.UDF.Catalog$.<clinit>(Catalog.scala)
> spark-thrift_1    |   ... 46 more
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Reply via email to