lordk911 opened a new issue #1256: URL: https://github.com/apache/incubator-linkis/issues/1256
I'm using Spark3.1 , I want to integration with apache atlas and ranger, to do data governance. I know there is a project https://github.com/hortonworks-spark/spark-atlas-connector , but it not support spark3.x finally I make it, what I do I will show bellow: 1、first you need spark-atlas-connector_2.12-XXX.jar , this can download from maven 2、mkdir a dir named sac on spark client server 3、in the dir sac we make in step2 , put some jars and config file: atlas-application.properties atlas-common-2.1.0.jar atlas-intg-2.1.0.jar atlas-notification-2.1.0.jar commons-configuration-1.10.jar kafka-clients-2.0.0.3.1.4.0-315.jar spark-atlas-connector_2.12-3.1.1.3.1.7270.0-253.jar 4、config spark-defaults.conf, add bellow configuration item: spark.driver.extraClassPath /{your dir prefix}/sac/* spark.extraListeners com.hortonworks.spark.atlas.SparkAtlasEventTracker spark.sql.queryExecutionListeners com.hortonworks.spark.atlas.SparkAtlasEventTracker 5、use atlas 2.1.0. That's all. 6、if your atlas version prior to 2.1.0 you need to copy spark_model.json from atlas 2.1.0 and put it to <YOUR_ATLAS_HOME>/models/1000-Hadoop 7、also atlas version prior to 2.1.0 may not display spark information on the web-site, replace <YOUR_ATLAS_HOME>/server/webapp/atlas/WEB-INF/lib directory with atlas 2.1.0's lib directory. about data security , I found a apache project kyuubi , it have a spark-security model, doc is here : https://submarine.apache.org/docs/userDocs/submarine-security/spark-security/README/ just follow it. now it not support spark3.2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
