[jira] [Updated] (RANGER-2927) Ranger Client Libraries (Python) For REST Services
[ https://issues.apache.org/jira/browse/RANGER-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Kumar updated RANGER-2927: --- Attachment: 0001-RANGER-2927-Python-client-Policy-Changes-1.patch > Ranger Client Libraries (Python) For REST Services > -- > > Key: RANGER-2927 > URL: https://issues.apache.org/jira/browse/RANGER-2927 > Project: Ranger > Issue Type: New Feature > Components: Ranger >Affects Versions: 2.1 >Reporter: Abhishek Kumar >Assignee: Abhishek Kumar >Priority: Major > Attachments: 0001-RANGER-2927-Python-client-Policy-Changes-1.patch > > Original Estimate: 336h > Remaining Estimate: 336h > > Build client libraries in Python to be used by client applications in > managing ranger policies and other entities. It will abstract communication > with the Ranger REST API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-2927) Ranger Client Libraries (Python) For REST Services
[ https://issues.apache.org/jira/browse/RANGER-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17165948#comment-17165948 ] Abhishek Kumar commented on RANGER-2927: Review Link: [https://reviews.apache.org/r/72710] > Ranger Client Libraries (Python) For REST Services > -- > > Key: RANGER-2927 > URL: https://issues.apache.org/jira/browse/RANGER-2927 > Project: Ranger > Issue Type: New Feature > Components: Ranger >Affects Versions: 2.1 >Reporter: Abhishek Kumar >Assignee: Abhishek Kumar >Priority: Major > Original Estimate: 336h > Remaining Estimate: 336h > > Build client libraries in Python to be used by client applications in > managing ranger policies and other entities. It will abstract communication > with the Ranger REST API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (RANGER-2927) Ranger Client Libraries (Python) For REST Services
[ https://issues.apache.org/jira/browse/RANGER-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Kumar updated RANGER-2927: --- Description: Build client libraries in Python to be used by client applications in managing ranger policies and other entities. It will abstract communication with the Ranger REST API. (was: Build client libraries in Python.) > Ranger Client Libraries (Python) For REST Services > -- > > Key: RANGER-2927 > URL: https://issues.apache.org/jira/browse/RANGER-2927 > Project: Ranger > Issue Type: New Feature > Components: Ranger >Affects Versions: 2.1 >Reporter: Abhishek Kumar >Assignee: Abhishek Kumar >Priority: Major > Original Estimate: 336h > Remaining Estimate: 336h > > Build client libraries in Python to be used by client applications in > managing ranger policies and other entities. It will abstract communication > with the Ranger REST API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (RANGER-2927) Ranger Client Libraries (Python) For REST Services
Abhishek Kumar created RANGER-2927: -- Summary: Ranger Client Libraries (Python) For REST Services Key: RANGER-2927 URL: https://issues.apache.org/jira/browse/RANGER-2927 Project: Ranger Issue Type: New Feature Components: Ranger Affects Versions: 2.1 Reporter: Abhishek Kumar Assignee: Abhishek Kumar Build client libraries in Python. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (RANGER-2862) Ranger Client Libraries (Java) For REST Services
[ https://issues.apache.org/jira/browse/RANGER-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Kumar updated RANGER-2862: --- Description: Build client libraries in Java to be used by client applications in managing ranger policies and other entities. It will abstract communication with the Ranger REST API. (was: Build client libraries in Java and Python to be used by client applications in managing ranger policies and other entities. It will abstract communication with the Ranger REST API.) > Ranger Client Libraries (Java) For REST Services > > > Key: RANGER-2862 > URL: https://issues.apache.org/jira/browse/RANGER-2862 > Project: Ranger > Issue Type: New Feature > Components: Ranger >Reporter: Abhishek Kumar >Assignee: Abhishek Kumar >Priority: Major > Fix For: 2.1.0 > > Attachments: 0001-RANGER-2862-Final-Commit.patch, > 0001-RANGER-2862-Introducing-Client-Libraries.patch, > 0001-RANGER-2862-Introducing-tests-and-including-all-publ.patch, > 0001-RANGER-2862-Refactoring1-on-comments-1.patch > > > Build client libraries in Java to be used by client applications in managing > ranger policies and other entities. It will abstract communication with the > Ranger REST API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (RANGER-2862) Ranger Client Libraries (Java) For REST Services
[ https://issues.apache.org/jira/browse/RANGER-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Kumar updated RANGER-2862: --- Summary: Ranger Client Libraries (Java) For REST Services (was: Ranger Client Libraries For REST Services) > Ranger Client Libraries (Java) For REST Services > > > Key: RANGER-2862 > URL: https://issues.apache.org/jira/browse/RANGER-2862 > Project: Ranger > Issue Type: New Feature > Components: Ranger >Reporter: Abhishek Kumar >Assignee: Abhishek Kumar >Priority: Major > Fix For: 2.1.0 > > Attachments: 0001-RANGER-2862-Final-Commit.patch, > 0001-RANGER-2862-Introducing-Client-Libraries.patch, > 0001-RANGER-2862-Introducing-tests-and-including-all-publ.patch, > 0001-RANGER-2862-Refactoring1-on-comments-1.patch > > > Build client libraries in Java and Python to be used by client applications > in managing ranger policies and other entities. It will abstract > communication with the Ranger REST API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (RANGER-2926) Issue in setting up Audit Log with ElasticSearch
[
https://issues.apache.org/jira/browse/RANGER-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17165888#comment-17165888
]
Bhanu edited comment on RANGER-2926 at 7/27/20, 6:06 PM:
-
Yes, I am able to insert, below is the configuration that I have used as per
logs. Will try the default mapping as provided by Bhavin and see if it resolves.
PUT _index_name/_mapping
{
"properties": {
"repoType":
{ "type": "integer" }
,
"repo":
{ "type": "text" }
,
"reqUser":
{ "type": "text" }
,
"evtTime":
{ "type": "date" }
,
"resource":
{ "type": "text" }
,
"resType":
{ "type": "text" }
,
"action":
{ "type": "text" }
,
"result":
{ "type": "integer" }
,
"agent":
{ "type": "text" }
,
"policy":
{ "type": "integer" }
,
"enforcer":
{ "type": "text" }
,
"agentHost":
{ "type": "text" }
,
"logType":
{ "type": "text" }
,
"id":
{ "type": "text" }
,
"seq_num":
{ "type": "integer" }
,
"event_count":
{ "type": "integer" }
,
"event_dur_ms":
{ "type": "integer" }
,
"tags":
{ "type": "text" }
,
"cluster_name":
{ "type": "text" }
,
"policy_version":
{ "type": "integer" }
}
}
was (Author: bdasari):
Yes, I am able to insert, below is the configuration that I have used as per
logs i framed it. Will try the default mapping as provided by Bhavin and see.
PUT _index_name/_mapping
{
"properties": {
"repoType":{
"type": "integer"
},
"repo":{
"type": "text"
},
"reqUser":{
"type": "text"
},
"evtTime": {
"type": "date"
},
"resource":{
"type": "text"
},
"resType":{
"type": "text"
},
"action":{
"type": "text"
},
"result":{
"type": "integer"
},
"agent":{
"type": "text"
},
"policy":{
"type": "integer"
},
"enforcer":{
"type": "text"
},
"agentHost":{
"type": "text"
},
"logType":{
"type": "text"
},
"id":{
"type": "text"
},
"seq_num":{
"type": "integer"
},
"event_count":{
"type": "integer"
},
"event_dur_ms":{
"type": "integer"
},
"tags":{
"type": "text"
},
"cluster_name":{
"type": "text"
},
"policy_version":{
"type": "integer"
}
}
}
> Issue in setting up Audit Log with ElasticSearch
> -
>
> Key: RANGER-2926
> URL: https://issues.apache.org/jira/browse/RANGER-2926
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 2.0.0
>Reporter: Bhanu
>Priority: Major
>
> Hi,
> We are using Ranger 2.1.0.
> Trying to setup AuditLog with ElasticSearch Server having version 7.0.1
> We have configured the Ranger with all details but there is an error that is
> keep on coming as below. Please let me know where we are going wrong here. We
> have tried recreating the index multiple times with all below parameters
> 2020-07-27T13:08:35.233Z ERROR org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler Error sending message to
> ElasticSearch
> org.elasticsearch.action.ActionRequestValidationException: Validation Failed:
> 1: type is missing;2: type is missing;
> at org.elasticsearch.action.bulk.BulkRequest.validate(BulkRequest.java:393)
> at
> org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1480)
> at
> org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1454)
> at
> org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:497)
> at
> org.apache.ranger.audit.destination.ElasticSearchAuditDestination.log(ElasticSearchAuditDestination.java:125)
> at
> org.apache.ranger.audit.queue.AuditBatchQueue.runLogAudit(AuditBatchQueue.java:309)
> at
> org.apache.ranger.audit.queue.AuditBatchQueue.run(AuditBatchQueue.java:215)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2020-07-27T13:08:35.233Z WARN org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler failed to log audit event:
> \{"repoType":17,"repo":"prestostg-tkg","reqUser":"bdasari","evtTime":"2020-07-27
>
> 13:08:35.102","resource":"hive_stg/ref_maritz","resType":"schema","action":"select","result":1,"agent":"presto","policy":21,"enforcer":"ranger-acl","agentHost":"coordinator2-694c5dbbb6-msh58","logType":"RangerAudit","id":"f733c835-c9ee-4507-b917-9eb822303d2b-792211","seq_num":1584423,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"","policy_version":148},
> errorMessage=
> 2020-07-27T13:08:35.233Z WARN org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler Log failure count: 4 in
> past 01:30.003 minutes; 792212 during process lifetime
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-2926) Issue in setting up Audit Log with ElasticSearch
[
https://issues.apache.org/jira/browse/RANGER-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17165888#comment-17165888
]
Bhanu commented on RANGER-2926:
---
Yes, I am able to insert, below is the configuration that I have used as per
logs i framed it. Will try the default mapping as provided by Bhavin and see.
PUT _index_name/_mapping
{
"properties": {
"repoType":{
"type": "integer"
},
"repo":{
"type": "text"
},
"reqUser":{
"type": "text"
},
"evtTime": {
"type": "date"
},
"resource":{
"type": "text"
},
"resType":{
"type": "text"
},
"action":{
"type": "text"
},
"result":{
"type": "integer"
},
"agent":{
"type": "text"
},
"policy":{
"type": "integer"
},
"enforcer":{
"type": "text"
},
"agentHost":{
"type": "text"
},
"logType":{
"type": "text"
},
"id":{
"type": "text"
},
"seq_num":{
"type": "integer"
},
"event_count":{
"type": "integer"
},
"event_dur_ms":{
"type": "integer"
},
"tags":{
"type": "text"
},
"cluster_name":{
"type": "text"
},
"policy_version":{
"type": "integer"
}
}
}
> Issue in setting up Audit Log with ElasticSearch
> -
>
> Key: RANGER-2926
> URL: https://issues.apache.org/jira/browse/RANGER-2926
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 2.0.0
>Reporter: Bhanu
>Priority: Major
>
> Hi,
> We are using Ranger 2.1.0.
> Trying to setup AuditLog with ElasticSearch Server having version 7.0.1
> We have configured the Ranger with all details but there is an error that is
> keep on coming as below. Please let me know where we are going wrong here. We
> have tried recreating the index multiple times with all below parameters
> 2020-07-27T13:08:35.233Z ERROR org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler Error sending message to
> ElasticSearch
> org.elasticsearch.action.ActionRequestValidationException: Validation Failed:
> 1: type is missing;2: type is missing;
> at org.elasticsearch.action.bulk.BulkRequest.validate(BulkRequest.java:393)
> at
> org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1480)
> at
> org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1454)
> at
> org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:497)
> at
> org.apache.ranger.audit.destination.ElasticSearchAuditDestination.log(ElasticSearchAuditDestination.java:125)
> at
> org.apache.ranger.audit.queue.AuditBatchQueue.runLogAudit(AuditBatchQueue.java:309)
> at
> org.apache.ranger.audit.queue.AuditBatchQueue.run(AuditBatchQueue.java:215)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2020-07-27T13:08:35.233Z WARN org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler failed to log audit event:
> \{"repoType":17,"repo":"prestostg-tkg","reqUser":"bdasari","evtTime":"2020-07-27
>
> 13:08:35.102","resource":"hive_stg/ref_maritz","resType":"schema","action":"select","result":1,"agent":"presto","policy":21,"enforcer":"ranger-acl","agentHost":"coordinator2-694c5dbbb6-msh58","logType":"RangerAudit","id":"f733c835-c9ee-4507-b917-9eb822303d2b-792211","seq_num":1584423,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"","policy_version":148},
> errorMessage=
> 2020-07-27T13:08:35.233Z WARN org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler Log failure count: 4 in
> past 01:30.003 minutes; 792212 during process lifetime
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-2926) Issue in setting up Audit Log with ElasticSearch
[
https://issues.apache.org/jira/browse/RANGER-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17165748#comment-17165748
]
Bhavik Patel commented on RANGER-2926:
--
The [default index
properties|https://github.com/apache/ranger/blob/master/security-admin/contrib/elasticsearch_for_audit_setup/conf/ranger_es_schema.json]
we have used. Any specific/private configuration we had not done.
> Issue in setting up Audit Log with ElasticSearch
> -
>
> Key: RANGER-2926
> URL: https://issues.apache.org/jira/browse/RANGER-2926
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 2.0.0
>Reporter: Bhanu
>Priority: Major
>
> Hi,
> We are using Ranger 2.1.0.
> Trying to setup AuditLog with ElasticSearch Server having version 7.0.1
> We have configured the Ranger with all details but there is an error that is
> keep on coming as below. Please let me know where we are going wrong here. We
> have tried recreating the index multiple times with all below parameters
> 2020-07-27T13:08:35.233Z ERROR org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler Error sending message to
> ElasticSearch
> org.elasticsearch.action.ActionRequestValidationException: Validation Failed:
> 1: type is missing;2: type is missing;
> at org.elasticsearch.action.bulk.BulkRequest.validate(BulkRequest.java:393)
> at
> org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1480)
> at
> org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1454)
> at
> org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:497)
> at
> org.apache.ranger.audit.destination.ElasticSearchAuditDestination.log(ElasticSearchAuditDestination.java:125)
> at
> org.apache.ranger.audit.queue.AuditBatchQueue.runLogAudit(AuditBatchQueue.java:309)
> at
> org.apache.ranger.audit.queue.AuditBatchQueue.run(AuditBatchQueue.java:215)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2020-07-27T13:08:35.233Z WARN org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler failed to log audit event:
> \{"repoType":17,"repo":"prestostg-tkg","reqUser":"bdasari","evtTime":"2020-07-27
>
> 13:08:35.102","resource":"hive_stg/ref_maritz","resType":"schema","action":"select","result":1,"agent":"presto","policy":21,"enforcer":"ranger-acl","agentHost":"coordinator2-694c5dbbb6-msh58","logType":"RangerAudit","id":"f733c835-c9ee-4507-b917-9eb822303d2b-792211","seq_num":1584423,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"","policy_version":148},
> errorMessage=
> 2020-07-27T13:08:35.233Z WARN org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler Log failure count: 4 in
> past 01:30.003 minutes; 792212 during process lifetime
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-2926) Issue in setting up Audit Log with ElasticSearch
[
https://issues.apache.org/jira/browse/RANGER-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17165746#comment-17165746
]
Pradeep Agrawal commented on RANGER-2926:
-
[~bdasari] : Are you able to insert the same data manually via curl request to
your ES index ?
> Issue in setting up Audit Log with ElasticSearch
> -
>
> Key: RANGER-2926
> URL: https://issues.apache.org/jira/browse/RANGER-2926
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 2.0.0
>Reporter: Bhanu
>Priority: Major
>
> Hi,
> We are using Ranger 2.1.0.
> Trying to setup AuditLog with ElasticSearch Server having version 7.0.1
> We have configured the Ranger with all details but there is an error that is
> keep on coming as below. Please let me know where we are going wrong here. We
> have tried recreating the index multiple times with all below parameters
> 2020-07-27T13:08:35.233Z ERROR org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler Error sending message to
> ElasticSearch
> org.elasticsearch.action.ActionRequestValidationException: Validation Failed:
> 1: type is missing;2: type is missing;
> at org.elasticsearch.action.bulk.BulkRequest.validate(BulkRequest.java:393)
> at
> org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1480)
> at
> org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1454)
> at
> org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:497)
> at
> org.apache.ranger.audit.destination.ElasticSearchAuditDestination.log(ElasticSearchAuditDestination.java:125)
> at
> org.apache.ranger.audit.queue.AuditBatchQueue.runLogAudit(AuditBatchQueue.java:309)
> at
> org.apache.ranger.audit.queue.AuditBatchQueue.run(AuditBatchQueue.java:215)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2020-07-27T13:08:35.233Z WARN org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler failed to log audit event:
> \{"repoType":17,"repo":"prestostg-tkg","reqUser":"bdasari","evtTime":"2020-07-27
>
> 13:08:35.102","resource":"hive_stg/ref_maritz","resType":"schema","action":"select","result":1,"agent":"presto","policy":21,"enforcer":"ranger-acl","agentHost":"coordinator2-694c5dbbb6-msh58","logType":"RangerAudit","id":"f733c835-c9ee-4507-b917-9eb822303d2b-792211","seq_num":1584423,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"","policy_version":148},
> errorMessage=
> 2020-07-27T13:08:35.233Z WARN org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler Log failure count: 4 in
> past 01:30.003 minutes; 792212 during process lifetime
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-2926) Issue in setting up Audit Log with ElasticSearch
[
https://issues.apache.org/jira/browse/RANGER-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17165741#comment-17165741
]
Bhanu commented on RANGER-2926:
---
May I know the index properties that you have used , also any other
configurations made at ElasticSearch side or Ranger Side
> Issue in setting up Audit Log with ElasticSearch
> -
>
> Key: RANGER-2926
> URL: https://issues.apache.org/jira/browse/RANGER-2926
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 2.0.0
>Reporter: Bhanu
>Priority: Major
>
> Hi,
> We are using Ranger 2.1.0.
> Trying to setup AuditLog with ElasticSearch Server having version 7.0.1
> We have configured the Ranger with all details but there is an error that is
> keep on coming as below. Please let me know where we are going wrong here. We
> have tried recreating the index multiple times with all below parameters
> 2020-07-27T13:08:35.233Z ERROR org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler Error sending message to
> ElasticSearch
> org.elasticsearch.action.ActionRequestValidationException: Validation Failed:
> 1: type is missing;2: type is missing;
> at org.elasticsearch.action.bulk.BulkRequest.validate(BulkRequest.java:393)
> at
> org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1480)
> at
> org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1454)
> at
> org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:497)
> at
> org.apache.ranger.audit.destination.ElasticSearchAuditDestination.log(ElasticSearchAuditDestination.java:125)
> at
> org.apache.ranger.audit.queue.AuditBatchQueue.runLogAudit(AuditBatchQueue.java:309)
> at
> org.apache.ranger.audit.queue.AuditBatchQueue.run(AuditBatchQueue.java:215)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2020-07-27T13:08:35.233Z WARN org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler failed to log audit event:
> \{"repoType":17,"repo":"prestostg-tkg","reqUser":"bdasari","evtTime":"2020-07-27
>
> 13:08:35.102","resource":"hive_stg/ref_maritz","resType":"schema","action":"select","result":1,"agent":"presto","policy":21,"enforcer":"ranger-acl","agentHost":"coordinator2-694c5dbbb6-msh58","logType":"RangerAudit","id":"f733c835-c9ee-4507-b917-9eb822303d2b-792211","seq_num":1584423,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"","policy_version":148},
> errorMessage=
> 2020-07-27T13:08:35.233Z WARN org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler Log failure count: 4 in
> past 01:30.003 minutes; 792212 during process lifetime
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-2926) Issue in setting up Audit Log with ElasticSearch
[
https://issues.apache.org/jira/browse/RANGER-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17165740#comment-17165740
]
Bhavik Patel commented on RANGER-2926:
--
We are already using it in the production environment and we are not observing
any such error but, our Elasticsearch version is 7.6.0.
You can update your Elasticsearch version to 7.6.0.
> Issue in setting up Audit Log with ElasticSearch
> -
>
> Key: RANGER-2926
> URL: https://issues.apache.org/jira/browse/RANGER-2926
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 2.0.0
>Reporter: Bhanu
>Priority: Major
>
> Hi,
> We are using Ranger 2.1.0.
> Trying to setup AuditLog with ElasticSearch Server having version 7.0.1
> We have configured the Ranger with all details but there is an error that is
> keep on coming as below. Please let me know where we are going wrong here. We
> have tried recreating the index multiple times with all below parameters
> 2020-07-27T13:08:35.233Z ERROR org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler Error sending message to
> ElasticSearch
> org.elasticsearch.action.ActionRequestValidationException: Validation Failed:
> 1: type is missing;2: type is missing;
> at org.elasticsearch.action.bulk.BulkRequest.validate(BulkRequest.java:393)
> at
> org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1480)
> at
> org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1454)
> at
> org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:497)
> at
> org.apache.ranger.audit.destination.ElasticSearchAuditDestination.log(ElasticSearchAuditDestination.java:125)
> at
> org.apache.ranger.audit.queue.AuditBatchQueue.runLogAudit(AuditBatchQueue.java:309)
> at
> org.apache.ranger.audit.queue.AuditBatchQueue.run(AuditBatchQueue.java:215)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2020-07-27T13:08:35.233Z WARN org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler failed to log audit event:
> \{"repoType":17,"repo":"prestostg-tkg","reqUser":"bdasari","evtTime":"2020-07-27
>
> 13:08:35.102","resource":"hive_stg/ref_maritz","resType":"schema","action":"select","result":1,"agent":"presto","policy":21,"enforcer":"ranger-acl","agentHost":"coordinator2-694c5dbbb6-msh58","logType":"RangerAudit","id":"f733c835-c9ee-4507-b917-9eb822303d2b-792211","seq_num":1584423,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"","policy_version":148},
> errorMessage=
> 2020-07-27T13:08:35.233Z WARN org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler Log failure count: 4 in
> past 01:30.003 minutes; 792212 during process lifetime
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (RANGER-2926) Issue in setting up Audit Log with ElasticSearch
[
https://issues.apache.org/jira/browse/RANGER-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bhanu updated RANGER-2926:
--
Summary: Issue in setting up Audit Log with ElasticSearch (was: Issue is
setting up Audit Log with ElasticSearch )
> Issue in setting up Audit Log with ElasticSearch
> -
>
> Key: RANGER-2926
> URL: https://issues.apache.org/jira/browse/RANGER-2926
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 2.0.0
>Reporter: Bhanu
>Priority: Major
>
> Hi,
> We are using Ranger 2.1.0.
> Trying to setup AuditLog with ElasticSearch Server having version 7.0.1
> We have configured the Ranger with all details but there is an error that is
> keep on coming as below. Please let me know where we are going wrong here. We
> have tried recreating the index multiple times with all below parameters
> 2020-07-27T13:08:35.233Z ERROR org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler Error sending message to
> ElasticSearch
> org.elasticsearch.action.ActionRequestValidationException: Validation Failed:
> 1: type is missing;2: type is missing;
> at org.elasticsearch.action.bulk.BulkRequest.validate(BulkRequest.java:393)
> at
> org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1480)
> at
> org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1454)
> at
> org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:497)
> at
> org.apache.ranger.audit.destination.ElasticSearchAuditDestination.log(ElasticSearchAuditDestination.java:125)
> at
> org.apache.ranger.audit.queue.AuditBatchQueue.runLogAudit(AuditBatchQueue.java:309)
> at
> org.apache.ranger.audit.queue.AuditBatchQueue.run(AuditBatchQueue.java:215)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2020-07-27T13:08:35.233Z WARN org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler failed to log audit event:
> \{"repoType":17,"repo":"prestostg-tkg","reqUser":"bdasari","evtTime":"2020-07-27
>
> 13:08:35.102","resource":"hive_stg/ref_maritz","resType":"schema","action":"select","result":1,"agent":"presto","policy":21,"enforcer":"ranger-acl","agentHost":"coordinator2-694c5dbbb6-msh58","logType":"RangerAudit","id":"f733c835-c9ee-4507-b917-9eb822303d2b-792211","seq_num":1584423,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"","policy_version":148},
> errorMessage=
> 2020-07-27T13:08:35.233Z WARN org.apache.ranger.audit.queue.AuditBatchQueue0
> org.apache.ranger.audit.provider.BaseAuditHandler Log failure count: 4 in
> past 01:30.003 minutes; 792212 during process lifetime
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (RANGER-2926) Issue is setting up Audit Log with ElasticSearch
Bhanu created RANGER-2926:
-
Summary: Issue is setting up Audit Log with ElasticSearch
Key: RANGER-2926
URL: https://issues.apache.org/jira/browse/RANGER-2926
Project: Ranger
Issue Type: Bug
Components: Ranger
Affects Versions: 2.0.0
Reporter: Bhanu
Hi,
We are using Ranger 2.1.0.
Trying to setup AuditLog with ElasticSearch Server having version 7.0.1
We have configured the Ranger with all details but there is an error that is
keep on coming as below. Please let me know where we are going wrong here. We
have tried recreating the index multiple times with all below parameters
2020-07-27T13:08:35.233Z ERROR org.apache.ranger.audit.queue.AuditBatchQueue0
org.apache.ranger.audit.provider.BaseAuditHandler Error sending message to
ElasticSearch
org.elasticsearch.action.ActionRequestValidationException: Validation Failed:
1: type is missing;2: type is missing;
at org.elasticsearch.action.bulk.BulkRequest.validate(BulkRequest.java:393)
at
org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1480)
at
org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1454)
at
org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:497)
at
org.apache.ranger.audit.destination.ElasticSearchAuditDestination.log(ElasticSearchAuditDestination.java:125)
at
org.apache.ranger.audit.queue.AuditBatchQueue.runLogAudit(AuditBatchQueue.java:309)
at org.apache.ranger.audit.queue.AuditBatchQueue.run(AuditBatchQueue.java:215)
at java.base/java.lang.Thread.run(Thread.java:834)
2020-07-27T13:08:35.233Z WARN org.apache.ranger.audit.queue.AuditBatchQueue0
org.apache.ranger.audit.provider.BaseAuditHandler failed to log audit event:
\{"repoType":17,"repo":"prestostg-tkg","reqUser":"bdasari","evtTime":"2020-07-27
13:08:35.102","resource":"hive_stg/ref_maritz","resType":"schema","action":"select","result":1,"agent":"presto","policy":21,"enforcer":"ranger-acl","agentHost":"coordinator2-694c5dbbb6-msh58","logType":"RangerAudit","id":"f733c835-c9ee-4507-b917-9eb822303d2b-792211","seq_num":1584423,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"","policy_version":148},
errorMessage=
2020-07-27T13:08:35.233Z WARN org.apache.ranger.audit.queue.AuditBatchQueue0
org.apache.ranger.audit.provider.BaseAuditHandler Log failure count: 4 in past
01:30.003 minutes; 792212 during process lifetime
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Review Request 72702: RANGER-2923: Changing data type of sync_source_info column to accommodate more characters
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72702/ --- (Updated July 27, 2020, 7:41 a.m.) Review request for ranger, Ankita Sinha, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy. Changes --- Addressed review comments Bugs: RANGER-2923 https://issues.apache.org/jira/browse/RANGER-2923 Repository: ranger Description --- Insert values for the column "sync_source_info" under table "x_ugsync_audit_info" can go beyond the maximum length of column. (currently limit is VARCHAR 4000). Now changing data type to different db flavours as follows : mysql : VARCHAR => MEDIUMTEXT oracle : VARCHAR => CLOB postgres: VARCHAR => TEXT sqlanywhere : VARCHAR => TEXT sqlserver : VARCHAR => nvarchar(max) Diffs (updated) - security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql 832d650cf security-admin/db/mysql/patches/048-alter-sync-source-info-in-x-ugsync-audit-info.sql PRE-CREATION security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql 4576e9677 security-admin/db/oracle/patches/048-alter-sync-source-info-in-x-ugsync-audit-info.sql PRE-CREATION security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql 6a0941ff6 security-admin/db/postgres/patches/048-alter-sync-source-info-in-x-ugsync-audit-info.sql PRE-CREATION security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql b663f3b02 security-admin/db/sqlanywhere/patches/048-alter-sync-source-info-in-x-ugsync-audit-info.sql PRE-CREATION security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql ab370ffb1 security-admin/db/sqlserver/patches/048-alter-sync-source-info-in-x-ugsync-audit-info.sql PRE-CREATION Diff: https://reviews.apache.org/r/72702/diff/2/ Changes: https://reviews.apache.org/r/72702/diff/1-2/ Testing --- Inserted large values of more than 4000 characters in column "sync_source_info" under table "x_ugsync_audit_info". Insert query succeeds. Thanks, Mahesh Bandal
