rameeshm opened a new pull request, #885: URL: https://github.com/apache/ranger/pull/885
… dispatching functionality ## What changes were proposed in this pull request? **audit-ingestor** - This would have the functionality of audit log inbounding from various plugin and committing into Kafka - This will be refactored audit-server microservice application in the current audit-server. **audit-dispatcher.** - This should be micro-services for subscribing the audits from kafka ranger audit topic and send it into Solr / hdfs / opensearch and other destination. - This should be configurable to take the form of Solr Audit dispatcher, HDFS audit dispatcher etc based on the destination type. - Each dispatcher for destination type should be a separate service dedicated to commit the audits to respective destination ## How was this patch tested? Tested in Docker setup ``` // Ranger Docker docker compose -f docker-compose.ranger.yml -f docker-compose.ranger-usersync.yml -f docker-compose.ranger-tagsync.yml up -d // Hadoop + Hive + Kafka docker compose -f docker-compose.ranger.yml -f docker-compose.ranger-hadoop.yml -f docker-compose.ranger-hive.yml -f docker-compose.ranger-kafka.yml up -d // start ranger audit server - Single one docker compose -f docker-compose.ranger.yml -f docker-compose.ranger-hadoop.yml -f docker-compose.ranger-kafka.yml -f docker-compose.ranger-audit-server.yml up -d ``` Commands run: ``` # HDFS load docker exec -it ranger-hadoop bash -c ' export KRB5CCNAME=/tmp/krb5cc_hdfs_custom kinit -kt /etc/keytabs/testuser1.keytab testuser1/[email protected] for i in {1..80}; do echo "Iteration $i" hdfs dfs -ls /user/ sleep 1 done' # Hive load docker exec -it ranger-hive bash -c ' export KRB5CCNAME=/tmp/krb5cc_hive_custom kinit -kt /etc/keytabs/testuser1.keytab testuser1/[email protected] for i in {1..40}; do echo "Iteration $i" beeline -u "jdbc:hive2://localhost:10000/default;principal=hive/[email protected]" -e "SHOW DATABASES;" sleep 2 done' ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
