Sailaja Polavarapu created RANGER-2804:
------------------------------------------
Summary: Ranger Ozone plugin unable to write to solr audits in SSL
enabled cluster
Key: RANGER-2804
URL: https://issues.apache.org/jira/browse/RANGER-2804
Project: Ranger
Issue Type: Bug
Components: Ranger
Reporter: Sailaja Polavarapu
In a secure environment with Auto-TLS enabled, ranger-ozone plugin is unable to
write to solr audits. The following exception is thrown:
2020-04-06 22:32:44,581 WARN
[ozone.async.summary.batch_ozone.async.summary.batch.solr_destWriter]
org.apache.ranger.audit.provider.BaseAuditHandler (BaseAuditHandler.java:403) -
Log failure count: 1 in past 01:00.021 minutes; 9 during process lifetime
2020-04-06 22:32:44,581 ERROR
[ozone.async.summary.batch_ozone.async.summary.batch.solr_destWriter]
org.apache.ranger.audit.queue.AuditFileSpool (AuditFileSpool.java:709) - Error
sending logs to consumer. provider=ozone.async.summary.batch,
consumer=ozone.async.summary.batch.solr
2020-04-06 22:32:44,582 INFO
[ozone.async.summary.batch_ozone.async.summary.batch.solr_destWriter]
org.apache.ranger.audit.queue.AuditFileSpool (AuditFileSpool.java:769) -
Destination is down. sleeping for 30000 milli seconds. indexQueue=0,
queueName=ozone.async.summary.batch, consumer=ozone.async.summary.batch.solr
2020-04-06 22:33:44,583 INFO
[ozone.async.summary.batch_ozone.async.summary.batch.solr_destWriter]
org.apache.ranger.audit.provider.BaseAuditHandler (BaseAuditHandler.java:332) -
Audit Status Log: name=ozone.async.summary.batch.solr, interval=01:00.022
minutes, events=1, failedCount=1, totalEvents=9, totalFailedCount=9
2020-04-06 22:33:44,597 WARN
[ozone.async.summary.batch_ozone.async.summary.batch.solr_destWriter]
org.apache.ranger.audit.provider.BaseAuditHandler (BaseAuditHandler.java:394) -
failed to log audit event:
{"repoType":201,"repo":"cm_ozone","reqUser":"om","evtTime":"2020-04-06
18:02:51.063","access":"create","resource":"vol1/bucket1","resType":"bucket","action":"create","result":1,"agent":"ozone","policy":17,"enforcer":"ranger-acl","cliIP":"172.27.91.205","reqData":"/vol1/bucket1","agentHost":"vs-ozrgr-1.vs-ozrgr.root.hwx.site","logType":"RangerAudit","id":"7ffd1e19-cfea-4d4f-8077-538bb1232c3f-0","seq_num":1,"event_count":1,"event_dur_ms":1,"tags":[],"additional_info":"{\"remote-ip-address\":172.27.91.205,
\"forwarded-ip-addresses\":[]","cluster_name":"Cluster 1","policy_version":1}
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from
server at
https://vs-ozrgr-2.vs-ozrgr.root.hwx.site:8985/solr/ranger_audits_shard1_replica_n1:
Expected mime type application/octet-stream but got text/html. <html><head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 401 Unauthorized access</title>
</head>
<body><h2>HTTP ERROR 401 Unauthorized access</h2>
<table>
<tr><th>URI:</th><td>/solr/ranger_audits_shard1_replica_n1/update</td></tr>
<tr><th>STATUS:</th><td>401</td></tr>
<tr><th>MESSAGE:</th><td>Unauthorized access</td></tr>
<tr><th>SERVLET:</th><td>default</td></tr>
</table>
</body>
</html>
at
org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:551)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1019)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:106)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:71)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:85)
at org.apache.ranger.audit.utils.SolrAppUtil$1.run(SolrAppUtil.java:35)
at org.apache.ranger.audit.utils.SolrAppUtil$1.run(SolrAppUtil.java:32)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
at
org.apache.ranger.audit.provider.MiscUtil.executePrivilegedAction(MiscUtil.java:516)
at
org.apache.ranger.audit.utils.SolrAppUtil.addDocsToSolr(SolrAppUtil.java:32)
at
org.apache.ranger.audit.destination.SolrAuditDestination.log(SolrAuditDestination.java:258)
at
org.apache.ranger.audit.provider.BaseAuditHandler.logJSON(BaseAuditHandler.java:192)
at
org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:879)
at
org.apache.ranger.audit.queue.AuditFileSpool.runLogAudit(AuditFileSpool.java:827)
at
org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:757)
at java.lang.Thread.run(Thread.java:748)
Caused by:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
from server at
https://vs-ozrgr-2.vs-ozrgr.root.hwx.site:8985/solr/ranger_audits_shard1_replica_n1:
Expected mime type application/octet-stream but got text/html. <html><head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 401 Unauthorized access</title>
</head>
<body><h2>HTTP ERROR 401 Unauthorized access</h2>
<table>
<tr><th>URI:</th><td>/solr/ranger_audits_shard1_replica_n1/update</td></tr>
<tr><th>STATUS:</th><td>401</td></tr>
<tr><th>MESSAGE:</th><td>Unauthorized access</td></tr>
<tr><th>SERVLET:</th><td>default</td></tr>
</table>
</body>
</html>
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:484)
at
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:414)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:528)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
2020-04-06 22:33:44,598 WARN
[ozone.async.summary.batch_ozone.async.summary.batch.solr_destWriter]
org.apache.ranger.audit.provider.BaseAuditHandler (BaseAuditHandler.java:403) -
Log failure count: 1 in past 01:00.018 minutes; 10 during process lifetime
2020-04-06 22:33:44,598 ERROR
[ozone.async.summary.batch_ozone.async.summary.batch.solr_destWriter]
org.apache.ranger.audit.queue.AuditFileSpool (AuditFileSpool.java:709) - Error
sending logs to consumer. provider=ozone.async.summary.batch,
consumer=ozone.async.summary.batch.solr
2020-04-06 22:33:44,599 INFO
[ozone.async.summary.batch_ozone.async.summary.batch.solr_destWriter]
org.apache.ranger.audit.queue.AuditFileSpool (AuditFileSpool.java:769) -
Destination is down. sleeping for 30000 milli seconds. indexQueue=0,
queueName=ozone.async.summary.batch, consumer=ozone.async.summary.batch.solr
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
