Currently my pyspark code able to connect to hive metastore at port 9083. However using this approach i can't put in-place any security mechanism like LDAP and sql authentication control. Is there anyway to connect from pyspark to spark thrift server on port 10000 without exposing hive metastore url to the pyspark ? I would like to authenticate the user before allow to execute spark sql, and user should only allow to query from databases,tables that they have the access.
Thank you,comet
