[jira] [Commented] (RANGER-2926) Issue in setting up Audit Log with ElasticSearch
[
https://issues.apache.org/jira/browse/RANGER-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17386191#comment-17386191
]
Bhavik Patel commented on RANGER-2926:
--
[~bdasari] I think you have to update your ES version to *7.6.0*.
If your issue got resolved then can you please close this Jira?
> Issue in setting up Audit Log with ElasticSearch
> -
>
> Key: RANGER-2926
> URL: https://issues.apache.org/jira/browse/RANGER-2926
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 2.0.0
>Reporter: Bhanu
>Priority: Major
>
> Hi,
> We are using Ranger 2.1.0.
> Trying to setup AuditLog with ElasticSearch Server having version 7.0.1
> We have configured the Ranger with all details but there is an error that is
> keep on coming as below. Please let me know where we are going wrong here. We
> have tried recreating the index multiple times with all below parameters
> 2020-07-27T13:08:35.233Z ERROR org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler Error sending message to
> ElasticSearch
> org.elasticsearch.action.ActionRequestValidationException: Validation Failed:
> 1: type is missing;2: type is missing;
> at org.elasticsearch.action.bulk.BulkRequest.validate(BulkRequest.java:393)
> at
> org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1480)
> at
> org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1454)
> at
> org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:497)
> at
> org.apache.ranger.audit.destination.ElasticSearchAuditDestination.log(ElasticSearchAuditDestination.java:125)
> at
> org.apache.ranger.audit.queue.AuditBatchQueue.runLogAudit(AuditBatchQueue.java:309)
> at
> org.apache.ranger.audit.queue.AuditBatchQueue.run(AuditBatchQueue.java:215)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2020-07-27T13:08:35.233Z WARN org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler failed to log audit event:
> \{"repoType":17,"repo":"prestostg-tkg","reqUser":"bdasari","evtTime":"2020-07-27
>
> 13:08:35.102","resource":"hive_stg/ref_maritz","resType":"schema","action":"select","result":1,"agent":"presto","policy":21,"enforcer":"ranger-acl","agentHost":"coordinator2-694c5dbbb6-msh58","logType":"RangerAudit","id":"f733c835-c9ee-4507-b917-9eb822303d2b-792211","seq_num":1584423,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"","policy_version":148},
> errorMessage=
> 2020-07-27T13:08:35.233Z WARN org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler Log failure count: 4 in
> past 01:30.003 minutes; 792212 during process lifetime
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-2926) Issue in setting up Audit Log with ElasticSearch
[
https://issues.apache.org/jira/browse/RANGER-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17165888#comment-17165888
]
Bhanu commented on RANGER-2926:
---
Yes, I am able to insert, below is the configuration that I have used as per
logs i framed it. Will try the default mapping as provided by Bhavin and see.
PUT _index_name/_mapping
{
"properties": {
"repoType":{
"type": "integer"
},
"repo":{
"type": "text"
},
"reqUser":{
"type": "text"
},
"evtTime": {
"type": "date"
},
"resource":{
"type": "text"
},
"resType":{
"type": "text"
},
"action":{
"type": "text"
},
"result":{
"type": "integer"
},
"agent":{
"type": "text"
},
"policy":{
"type": "integer"
},
"enforcer":{
"type": "text"
},
"agentHost":{
"type": "text"
},
"logType":{
"type": "text"
},
"id":{
"type": "text"
},
"seq_num":{
"type": "integer"
},
"event_count":{
"type": "integer"
},
"event_dur_ms":{
"type": "integer"
},
"tags":{
"type": "text"
},
"cluster_name":{
"type": "text"
},
"policy_version":{
"type": "integer"
}
}
}
> Issue in setting up Audit Log with ElasticSearch
> -
>
> Key: RANGER-2926
> URL: https://issues.apache.org/jira/browse/RANGER-2926
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 2.0.0
>Reporter: Bhanu
>Priority: Major
>
> Hi,
> We are using Ranger 2.1.0.
> Trying to setup AuditLog with ElasticSearch Server having version 7.0.1
> We have configured the Ranger with all details but there is an error that is
> keep on coming as below. Please let me know where we are going wrong here. We
> have tried recreating the index multiple times with all below parameters
> 2020-07-27T13:08:35.233Z ERROR org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler Error sending message to
> ElasticSearch
> org.elasticsearch.action.ActionRequestValidationException: Validation Failed:
> 1: type is missing;2: type is missing;
> at org.elasticsearch.action.bulk.BulkRequest.validate(BulkRequest.java:393)
> at
> org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1480)
> at
> org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1454)
> at
> org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:497)
> at
> org.apache.ranger.audit.destination.ElasticSearchAuditDestination.log(ElasticSearchAuditDestination.java:125)
> at
> org.apache.ranger.audit.queue.AuditBatchQueue.runLogAudit(AuditBatchQueue.java:309)
> at
> org.apache.ranger.audit.queue.AuditBatchQueue.run(AuditBatchQueue.java:215)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2020-07-27T13:08:35.233Z WARN org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler failed to log audit event:
> \{"repoType":17,"repo":"prestostg-tkg","reqUser":"bdasari","evtTime":"2020-07-27
>
> 13:08:35.102","resource":"hive_stg/ref_maritz","resType":"schema","action":"select","result":1,"agent":"presto","policy":21,"enforcer":"ranger-acl","agentHost":"coordinator2-694c5dbbb6-msh58","logType":"RangerAudit","id":"f733c835-c9ee-4507-b917-9eb822303d2b-792211","seq_num":1584423,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"","policy_version":148},
> errorMessage=
> 2020-07-27T13:08:35.233Z WARN org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler Log failure count: 4 in
> past 01:30.003 minutes; 792212 during process lifetime
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-2926) Issue in setting up Audit Log with ElasticSearch
[
https://issues.apache.org/jira/browse/RANGER-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17165748#comment-17165748
]
Bhavik Patel commented on RANGER-2926:
--
The [default index
properties|https://github.com/apache/ranger/blob/master/security-admin/contrib/elasticsearch_for_audit_setup/conf/ranger_es_schema.json]
we have used. Any specific/private configuration we had not done.
> Issue in setting up Audit Log with ElasticSearch
> -
>
> Key: RANGER-2926
> URL: https://issues.apache.org/jira/browse/RANGER-2926
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 2.0.0
>Reporter: Bhanu
>Priority: Major
>
> Hi,
> We are using Ranger 2.1.0.
> Trying to setup AuditLog with ElasticSearch Server having version 7.0.1
> We have configured the Ranger with all details but there is an error that is
> keep on coming as below. Please let me know where we are going wrong here. We
> have tried recreating the index multiple times with all below parameters
> 2020-07-27T13:08:35.233Z ERROR org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler Error sending message to
> ElasticSearch
> org.elasticsearch.action.ActionRequestValidationException: Validation Failed:
> 1: type is missing;2: type is missing;
> at org.elasticsearch.action.bulk.BulkRequest.validate(BulkRequest.java:393)
> at
> org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1480)
> at
> org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1454)
> at
> org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:497)
> at
> org.apache.ranger.audit.destination.ElasticSearchAuditDestination.log(ElasticSearchAuditDestination.java:125)
> at
> org.apache.ranger.audit.queue.AuditBatchQueue.runLogAudit(AuditBatchQueue.java:309)
> at
> org.apache.ranger.audit.queue.AuditBatchQueue.run(AuditBatchQueue.java:215)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2020-07-27T13:08:35.233Z WARN org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler failed to log audit event:
> \{"repoType":17,"repo":"prestostg-tkg","reqUser":"bdasari","evtTime":"2020-07-27
>
> 13:08:35.102","resource":"hive_stg/ref_maritz","resType":"schema","action":"select","result":1,"agent":"presto","policy":21,"enforcer":"ranger-acl","agentHost":"coordinator2-694c5dbbb6-msh58","logType":"RangerAudit","id":"f733c835-c9ee-4507-b917-9eb822303d2b-792211","seq_num":1584423,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"","policy_version":148},
> errorMessage=
> 2020-07-27T13:08:35.233Z WARN org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler Log failure count: 4 in
> past 01:30.003 minutes; 792212 during process lifetime
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-2926) Issue in setting up Audit Log with ElasticSearch
[
https://issues.apache.org/jira/browse/RANGER-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17165746#comment-17165746
]
Pradeep Agrawal commented on RANGER-2926:
-
[~bdasari] : Are you able to insert the same data manually via curl request to
your ES index ?
> Issue in setting up Audit Log with ElasticSearch
> -
>
> Key: RANGER-2926
> URL: https://issues.apache.org/jira/browse/RANGER-2926
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 2.0.0
>Reporter: Bhanu
>Priority: Major
>
> Hi,
> We are using Ranger 2.1.0.
> Trying to setup AuditLog with ElasticSearch Server having version 7.0.1
> We have configured the Ranger with all details but there is an error that is
> keep on coming as below. Please let me know where we are going wrong here. We
> have tried recreating the index multiple times with all below parameters
> 2020-07-27T13:08:35.233Z ERROR org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler Error sending message to
> ElasticSearch
> org.elasticsearch.action.ActionRequestValidationException: Validation Failed:
> 1: type is missing;2: type is missing;
> at org.elasticsearch.action.bulk.BulkRequest.validate(BulkRequest.java:393)
> at
> org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1480)
> at
> org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1454)
> at
> org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:497)
> at
> org.apache.ranger.audit.destination.ElasticSearchAuditDestination.log(ElasticSearchAuditDestination.java:125)
> at
> org.apache.ranger.audit.queue.AuditBatchQueue.runLogAudit(AuditBatchQueue.java:309)
> at
> org.apache.ranger.audit.queue.AuditBatchQueue.run(AuditBatchQueue.java:215)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2020-07-27T13:08:35.233Z WARN org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler failed to log audit event:
> \{"repoType":17,"repo":"prestostg-tkg","reqUser":"bdasari","evtTime":"2020-07-27
>
> 13:08:35.102","resource":"hive_stg/ref_maritz","resType":"schema","action":"select","result":1,"agent":"presto","policy":21,"enforcer":"ranger-acl","agentHost":"coordinator2-694c5dbbb6-msh58","logType":"RangerAudit","id":"f733c835-c9ee-4507-b917-9eb822303d2b-792211","seq_num":1584423,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"","policy_version":148},
> errorMessage=
> 2020-07-27T13:08:35.233Z WARN org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler Log failure count: 4 in
> past 01:30.003 minutes; 792212 during process lifetime
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-2926) Issue in setting up Audit Log with ElasticSearch
[
https://issues.apache.org/jira/browse/RANGER-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17165741#comment-17165741
]
Bhanu commented on RANGER-2926:
---
May I know the index properties that you have used , also any other
configurations made at ElasticSearch side or Ranger Side
> Issue in setting up Audit Log with ElasticSearch
> -
>
> Key: RANGER-2926
> URL: https://issues.apache.org/jira/browse/RANGER-2926
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 2.0.0
>Reporter: Bhanu
>Priority: Major
>
> Hi,
> We are using Ranger 2.1.0.
> Trying to setup AuditLog with ElasticSearch Server having version 7.0.1
> We have configured the Ranger with all details but there is an error that is
> keep on coming as below. Please let me know where we are going wrong here. We
> have tried recreating the index multiple times with all below parameters
> 2020-07-27T13:08:35.233Z ERROR org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler Error sending message to
> ElasticSearch
> org.elasticsearch.action.ActionRequestValidationException: Validation Failed:
> 1: type is missing;2: type is missing;
> at org.elasticsearch.action.bulk.BulkRequest.validate(BulkRequest.java:393)
> at
> org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1480)
> at
> org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1454)
> at
> org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:497)
> at
> org.apache.ranger.audit.destination.ElasticSearchAuditDestination.log(ElasticSearchAuditDestination.java:125)
> at
> org.apache.ranger.audit.queue.AuditBatchQueue.runLogAudit(AuditBatchQueue.java:309)
> at
> org.apache.ranger.audit.queue.AuditBatchQueue.run(AuditBatchQueue.java:215)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2020-07-27T13:08:35.233Z WARN org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler failed to log audit event:
> \{"repoType":17,"repo":"prestostg-tkg","reqUser":"bdasari","evtTime":"2020-07-27
>
> 13:08:35.102","resource":"hive_stg/ref_maritz","resType":"schema","action":"select","result":1,"agent":"presto","policy":21,"enforcer":"ranger-acl","agentHost":"coordinator2-694c5dbbb6-msh58","logType":"RangerAudit","id":"f733c835-c9ee-4507-b917-9eb822303d2b-792211","seq_num":1584423,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"","policy_version":148},
> errorMessage=
> 2020-07-27T13:08:35.233Z WARN org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler Log failure count: 4 in
> past 01:30.003 minutes; 792212 during process lifetime
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-2926) Issue in setting up Audit Log with ElasticSearch
[
https://issues.apache.org/jira/browse/RANGER-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17165740#comment-17165740
]
Bhavik Patel commented on RANGER-2926:
--
We are already using it in the production environment and we are not observing
any such error but, our Elasticsearch version is 7.6.0.
You can update your Elasticsearch version to 7.6.0.
> Issue in setting up Audit Log with ElasticSearch
> -
>
> Key: RANGER-2926
> URL: https://issues.apache.org/jira/browse/RANGER-2926
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 2.0.0
>Reporter: Bhanu
>Priority: Major
>
> Hi,
> We are using Ranger 2.1.0.
> Trying to setup AuditLog with ElasticSearch Server having version 7.0.1
> We have configured the Ranger with all details but there is an error that is
> keep on coming as below. Please let me know where we are going wrong here. We
> have tried recreating the index multiple times with all below parameters
> 2020-07-27T13:08:35.233Z ERROR org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler Error sending message to
> ElasticSearch
> org.elasticsearch.action.ActionRequestValidationException: Validation Failed:
> 1: type is missing;2: type is missing;
> at org.elasticsearch.action.bulk.BulkRequest.validate(BulkRequest.java:393)
> at
> org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1480)
> at
> org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1454)
> at
> org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:497)
> at
> org.apache.ranger.audit.destination.ElasticSearchAuditDestination.log(ElasticSearchAuditDestination.java:125)
> at
> org.apache.ranger.audit.queue.AuditBatchQueue.runLogAudit(AuditBatchQueue.java:309)
> at
> org.apache.ranger.audit.queue.AuditBatchQueue.run(AuditBatchQueue.java:215)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2020-07-27T13:08:35.233Z WARN org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler failed to log audit event:
> \{"repoType":17,"repo":"prestostg-tkg","reqUser":"bdasari","evtTime":"2020-07-27
>
> 13:08:35.102","resource":"hive_stg/ref_maritz","resType":"schema","action":"select","result":1,"agent":"presto","policy":21,"enforcer":"ranger-acl","agentHost":"coordinator2-694c5dbbb6-msh58","logType":"RangerAudit","id":"f733c835-c9ee-4507-b917-9eb822303d2b-792211","seq_num":1584423,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"","policy_version":148},
> errorMessage=
> 2020-07-27T13:08:35.233Z WARN org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler Log failure count: 4 in
> past 01:30.003 minutes; 792212 during process lifetime
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
