> On Dec. 21, 2021, 7:09 p.m., Yao Zhou wrote:
> > security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
> > Lines 136-166 (patched)
> > <https://reviews.apache.org/r/73756/diff/4/?file=2257093#file2257093line136>
> >
> > Fetching all streams from a given log group could be quite slow and can
> > easily trigger throttling given every AWS account has API limits.
> >
> > 1/ Have you verified that filterLogEvents() does not work if we don't
> > provide a log stream? I was able to do it through AWS CLI but haven't tried
> > SDK yet.
> > ```
> > aws logs filter-log-events --log-group-name "xxx"
> >
> > ....(all events across streams)
> > ```
> > 2/ I would suggest we change the log stream to log stream prefix, which
> > matches the client-side config and also gives you the benefit of searching
> > events across streams (instead of withLogStreamNames, you do
> > withLogStreamPrefix see
> > https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/logs/model/FilterLogEventsRequest.html#setLogStreamNamePrefix-java.lang.String-)
> >
> > ```
> > logStreamName = PropertiesUtil.getProperty(CONFIG_PREFIX + "." +
> > "log_stream");
> > ```
>
> Pradeep Agrawal wrote:
> I tried with below given diff and its not returning any records. I added
> debug logs also and its not returning anything.
>
> ========
>
> diff --git
> a/security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
>
> b/security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
> index 896ed05e1..cc300eacb 100644
> ---
> a/security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
> +++
> b/security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
> @@ -21,6 +21,7 @@ package org.apache.ranger.amazon.cloudwatch;
>
> import static
> org.apache.ranger.audit.destination.AmazonCloudWatchAuditDestination.CONFIG_PREFIX;
> import static
> org.apache.ranger.audit.destination.AmazonCloudWatchAuditDestination.PROP_LOG_GROUP_NAME;
> +import static
> org.apache.ranger.audit.destination.AmazonCloudWatchAuditDestination.PROP_LOG_STREAM_PREFIX;
>
> import java.text.SimpleDateFormat;
> import java.util.ArrayList;
> @@ -45,12 +46,9 @@ import
> org.springframework.beans.factory.annotation.Autowired;
> import org.springframework.stereotype.Component;
>
> import com.amazonaws.services.logs.AWSLogs;
> -import com.amazonaws.services.logs.model.DescribeLogStreamsRequest;
> -import com.amazonaws.services.logs.model.DescribeLogStreamsResult;
> import com.amazonaws.services.logs.model.FilterLogEventsRequest;
> import com.amazonaws.services.logs.model.FilterLogEventsResult;
> import com.amazonaws.services.logs.model.FilteredLogEvent;
> -import com.amazonaws.services.logs.model.LogStream;
>
> @Component
> public class CloudWatchUtil {
> @@ -66,7 +64,7 @@ public class CloudWatchUtil {
>
> public CloudWatchUtil() {
> logGroupName = PropertiesUtil.getProperty(CONFIG_PREFIX +
> "." + PROP_LOG_GROUP_NAME, "ranger_audits");
> - logStreamName = PropertiesUtil.getProperty(CONFIG_PREFIX
> + "." + "log_stream");
> + logStreamName = PropertiesUtil.getProperty(CONFIG_PREFIX
> + "." + PROP_LOG_STREAM_PREFIX, "ranger");
> String timeZone =
> PropertiesUtil.getProperty("ranger.cloudwatch.timezone");
> if (timeZone != null) {
> LOGGER.info("Setting timezone to " + timeZone);
> @@ -131,7 +129,7 @@ public class CloudWatchUtil {
> Date fromDate = null;
> Date toDate = null;
>
> - String nextToken = null;
> + /*String nextToken = null;
> boolean done = false;
> // load log stream names from cloudwatch if logStreamName
> is not provided
> List<String> logStreamNames = new ArrayList<String>();
> @@ -163,7 +161,7 @@ public class CloudWatchUtil {
> } while (!done);
> } else {
> logStreamNames.add(logStreamName);
> - }
> + }*/
>
> if (searchCriteria.getParamList() != null) {
> List<String> filterExpr = new ArrayList<String>();
> @@ -238,7 +236,7 @@ public class CloudWatchUtil {
> // Add FilterPattern which will only fetch logs required
> filterLogEventsRequest = new FilterLogEventsRequest()
> .withLogGroupName(logGroupName)
> - .withLogStreamNames(logStreamNames)
> + .withLogStreamNamePrefix(logStreamName)
> .withStartTime(fromDate.getTime())
> .withEndTime(toDate.getTime())
>
> .withFilterPattern(filterPattern.toString());
>
> Pradeep Agrawal wrote:
> Can you confirm the same at your end.
I tested with below code and I was able to see events from different log
streams. The code is written in Scala but it's using the latest AWS JAVA SDK
(1.12.122). I suspect that you are seeing empty result because your filters
(e.g. startTime/endTime/filterPattern) are excluding all the events.
```
val logs: AWSLogs = AWSLogsClientBuilder
.standard()
.withCredentials(credProvider)
.withRegion("us-east-1")
.build()
val req = new FilterLogEventsRequest()
req.setLogGroupName("xxx")
val events: List[FilteredLogEvent] =
logs.filterLogEvents(req).getEvents.asScala.toList
events.foreach(println)
```
- Yao
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73756/#review223893
-----------------------------------------------------------
On Dec. 10, 2021, 8:08 p.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73756/
> -----------------------------------------------------------
>
> (Updated Dec. 10, 2021, 8:08 p.m.)
>
>
> Review request for ranger, Dineshkumar Yadav, Kishor Gollapalliwar, Abhay
> Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Sailaja Polavarapu,
> and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3540
> https://issues.apache.org/jira/browse/RANGER-3540
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement: ** This is related to RANGER-2967 which includes changes
> only at the plugin end. Access audit logs should be accessible and appear at
> Ranger admin UI end as well.
>
> **Proposed Solution: ** Proposed patch make use of AWS API's to read access
> audit logs from cloudwatch loggroup.
>
> **Known issue:** Cloudwatch APIs does not provide sorting of recording in
> descending order of timestamp, hence read operation will be slow. Hence its
> recommended to use the filter to minimise the resultset which shall reduce
> the response time and access audit page will load faster.
> Due to this issue as of now maximum 10k records will be loaded at a time to
> handle out of memory issue.
>
>
> Diffs
> -----
>
> agents-audit/pom.xml 5d031cca1
>
> agents-audit/src/main/java/org/apache/ranger/audit/destination/AmazonCloudWatchAuditDestination.java
> b236a2653
> agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java
> f58b813f8
> hbase-agent/conf/ranger-hbase-audit-changes.cfg a6c7ffd41
> hbase-agent/scripts/install.properties 87a24819e
> hdfs-agent/conf/ranger-hdfs-audit-changes.cfg 92d2a4b08
> hdfs-agent/scripts/install.properties 323b878cf
> hive-agent/conf/ranger-hive-audit-changes.cfg 52c715ef5
> hive-agent/scripts/install.properties 3720b66c8
> kms/scripts/install.properties 6b6b66270
> knox-agent/conf/ranger-knox-audit-changes.cfg 52c715ef5
> knox-agent/scripts/install.properties 470400499
> plugin-atlas/conf/ranger-atlas-audit-changes.cfg 2d8251b5f
> plugin-atlas/scripts/install.properties 3b777bd6a
> plugin-elasticsearch/conf/ranger-elasticsearch-audit-changes.cfg 52c715ef5
> plugin-elasticsearch/scripts/install.properties 4111afe3f
> plugin-kafka/conf/ranger-kafka-audit-changes.cfg bc5a0890d
> plugin-kafka/scripts/install.properties 1e325e0ec
> plugin-kms/conf/ranger-kms-audit-changes.cfg e5e9ae489
> plugin-kylin/conf/ranger-kylin-audit-changes.cfg 52c715ef5
> plugin-kylin/scripts/install.properties 013433837
> plugin-ozone/conf/ranger-ozone-audit-changes.cfg 0eace6d29
> plugin-ozone/scripts/install.properties 1891d565f
> plugin-presto/conf/ranger-presto-audit-changes.cfg bc5a0890d
> plugin-presto/scripts/install.properties ce162a2bd
> plugin-solr/conf/ranger-solr-audit-changes.cfg ffa0a7696
> plugin-solr/scripts/install.properties d1852e695
> plugin-sqoop/conf/ranger-sqoop-audit-changes.cfg 52c715ef5
> plugin-sqoop/scripts/install.properties 81b4526a6
> plugin-yarn/conf/ranger-yarn-audit-changes.cfg 52c715ef5
> plugin-yarn/scripts/install.properties e73ab8b14
> pom.xml f9c46f669
> security-admin/pom.xml e9e9a537b
> security-admin/scripts/install.properties 5a8b00c13
> security-admin/scripts/ranger-admin-site-template.xml 72ff66eaf
> security-admin/scripts/setup.sh c3f51a03a
> security-admin/scripts/upgrade_admin.py 10fa485bd
> security-admin/src/main/java/org/apache/ranger/AccessAuditsService.java
> e902e65d0
>
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchAccessAuditsService.java
> PRE-CREATION
>
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchMgr.java
> PRE-CREATION
>
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
> PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/biz/AssetMgr.java d3ce25158
> security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java
> 75ebae6f5
> security-admin/src/main/java/org/apache/ranger/biz/XAuditMgr.java 4e5410e8b
>
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchAccessAuditsService.java
> 0b2e7df7f
>
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchUtil.java
> 9bee640a5
>
> security-admin/src/main/java/org/apache/ranger/solr/SolrAccessAuditsService.java
> 0aea46d1b
> security-admin/src/main/java/org/apache/ranger/solr/SolrUtil.java 239698f3f
> security-admin/src/main/resources/conf.dist/ranger-admin-site.xml d32a324ec
> storm-agent/conf/ranger-storm-audit-changes.cfg 52c715ef5
> storm-agent/scripts/install.properties d219abf59
>
>
> Diff: https://reviews.apache.org/r/73756/diff/4/
>
>
> Testing
> -------
>
> Tested by creating IAM user in AWS and provided required configuration in the
> install.properties.
>
> **Note:** AWS region name, access key and secret key should be provided in
> the environment.
>
>
> Thanks,
>
> Pradeep Agrawal
>
>
