Hi Jochen!!,
The AWS plugin is not reading any logs, though it is reaching to the AWS
cloudtrail successfully. I am sharing the payload sample and the
configuration I have done in both Graylog and AWS. Please tell me if any
thing wrong I have done doing it.
this is one of the value that mesage.body() from
"CloudtrailSNSNotificationParser" is returing :-
{"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/11/924399563845_CloudTrail_us-west-1_20160511T0715Z_FCFmYjMFmTZFthnu.json.gz"]}
this exists at the AWS.
Following is the content which I got after manually extracting the above
mentioned
924399563845_CloudTrail_us-west-1_20160511T0715Z_FCFmYjMFmTZFthnu.json.gz
:-
{"Records":[{"eventVersion":"1.03","userIdentity":{"type":"IAMUser","principalId":"AIDAIZDWJD4XSD7OPQYUW",
"arn":"arn:aws:iam::924399563845:user/nileshk","accountId":"924399563845","userName":"nileshk",
"sessionContext":{"attributes":{"mfaAuthenticated":"false","creationDate":"2016-05-11T07:11:24Z"}},
"invokedBy":"signin.amazonaws.com"},"eventTime":"2016-05-11T07:12:45Z","eventSource":"s3.amazonaws.com",
"eventName":"GetBucketVersioning","awsRegion":"us-west-1","sourceIPAddress":"14.140.226.238",
"userAgent":"signin.amazonaws.com","requestParameters":{"bucketName":"fluidcmlogs"},
"responseElements":null,"requestID":"66745CC932793217","eventID":"08f194cd-52c8-41eb-9619-beee1af30074",
"eventType":"AwsApiCall","recipientAccountId":"924399563845"},{"eventVersion":"1.03",
"userIdentity":{"type":"IAMUser","principalId":"AIDAIZDWJD4XSD7OPQYUW","arn":"arn:aws:iam::924399563845:user/nileshk",
"accountId":"924399563845","userName":"nileshk","sessionContext":{"attributes":{"mfaAuthenticated":"false",
"creationDate":"2016-05-11T07:11:24Z"}},"invokedBy":"signin.amazonaws.com"},"eventTime":"2016-05-11T07:12:46Z",
"eventSource":"s3.amazonaws.com","eventName":"GetBucketVersioning","awsRegion":"us-west-1","sourceIPAddress":"14.140.226.238",
"userAgent":"signin.amazonaws.com","requestParameters":{"bucketName":"fluidcmlogs"},"responseElements":null,"requestID":"E1360FC36112F5A8",
"eventID":"cab707ed-f20b-40ea-ac3f-6e208fc6eb4a","eventType":"AwsApiCall","recipientAccountId":"924399563845"}]}
Setting/configuration at the AWS account(for doing this followed the following
"https://marketplace.graylog.org/addons/3f132fab-50f0-4c88-b63d-9ac99aa6c20e";).
Step 1: Enabling CloudTrail for an AWS region
Trail name:Fluidcm-CloudTrail
S3 bucket: fluidcmlogs
Log file prefix: fluidcm
SNS notification for every log file delivery: Yes
SNS topic:cloudtrail-log-write
Step 2: Set up SQS for CloudTrail write notifications
Queue Name: fluidcm-notifications
Kept Default setting as it is..
Step 3: Created Aws Policy for s3 bucket and SQS for full access user.
The region where your new queue will be created:US West (N. California)
Seetin/configuration at Graylog.
Title :- Aws CloudTrail
AWS Region :- US_WEST_1
AWS access key :- auto generated
SQS queue name :- fluidcm-notifications
AWS secret key :- auto generated
On Wednesday, 11 May 2016 11:32:07 UTC+5:30, Anant Sawant wrote:
>
> HI Jochen!!
>
> Thanks for the reply. The error has been resolved. The plugin is able to
> connect to AWS and also get the message body as per the
> "CloudtrailSNSNotificationParser" class from the plugin but no logs are
> visible in Graylog. I inserted my own logs to see if the plugin is reading
> events from AWS or not, and found that it receiving the message's body,
> "message.getBody()" methid is returning the following
>
>
> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/10/924399563845_CloudTrail_us-west-1_20160510T1220Z_Dd8u8fCREYcu0Bd8.json.gz"]}
>
> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/10/924399563845_CloudTrail_us-west-1_20160510T1750Z_ePOdk3E0lg1KL5vt.json.gz"]}
>
> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/ap-northeast-1/2016/05/06/924399563845_CloudTrail_ap-northeast-1_20160506T1010Z_LE3fKktT1wVK1vA5.json.gz"]}
>
> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/10/924399563845_CloudTrail_us-west-1_20160510T1315Z_aKJCNFF9np7FC0Gg.json.gz"]}
>
> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/10/924399563845_CloudTrail_us-west-1_20160510T2250Z_a2TsampYHKq5baC8.json.gz"]}
>
> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/ap-northeast-2/2016/05/06/924399563845_CloudTrail_ap-northeast-2_20160506T0935Z_g7rcYdIFmA4ymndh.json.gz"]}
>
> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/06/924399563845_CloudTrail_us-west-1_20160506T1010Z_IzqYaYzIcsBdcOBu.json.gz"]}
>
> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/10/924399563845_CloudTrail_us-west-1_20160510T1750Z_ePOdk3E0lg1KL5vt.json.gz"]}
>
> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/ap-northeast-1/2016/05/06/924399563845_CloudTrail_ap-northeast-1_20160506T1010Z_LE3fKktT1wVK1vA5.json.gz"]}
>
> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/10/924399563845_CloudTrail_us-west-1_20160510T1640Z_39iM51Yqif0mMMRJ.json.gz"]}
>
> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/10/924399563845_CloudTrail_us-west-1_20160510T1315Z_aKJCNFF9np7FC0Gg.json.gz"]}
>
> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/10/924399563845_CloudTrail_us-west-1_20160510T2250Z_a2TsampYHKq5baC8.json.gz"]}
> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-east-1/2016/05/06/924399563845_CloudTrail_us-east-1_20160506T1255Z_YloXCOB1lCnODpp4.json.gz"]}
>
> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/10/924399563845_CloudTrail_us-west-1_20160510T1900Z_GIsgmOpucjjQQ6Pm.json.gz"]}
>
> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/10/924399563845_CloudTrail_us-west-1_20160510T1405Z_39oVS7OoNWENcRaQ.json.gz"]}
>
> and this exists on AWS.
>
> but call to "envelope.message" in the same class
> "CloudtrailSNSNotificationParser" returns "null". Is this the reason why we
> are not getting any events in Graylog. Please give me some advice to
> overcome this situation.
>
> Thanks in advance.
> Anant
>
>
> On Monday, 9 May 2016 18:02:27 UTC+5:30, Jochen Schalanda wrote:
>>
>> Hi Anant,
>>
>> it looks like the plugin is expecting some value in the payload to be a
>> boolean type but received a string.
>>
>> Do you still have access to the raw message payload? The string "hi"
>> suggests that this was simply a test message.
>>
>> Cheers,
>> Jochen
>>
>> On Friday, 6 May 2016 13:02:38 UTC+2, Anant Sawant wrote:
>>>
>>> Hi Everyone!!,
>>>
>>> I went through the documentation for setting up the Cloudtrail plugin
>>> for US-WEST-1, but I am getting the following error.
>>> I have done all the AWS settings/configuration as described at "
>>> https://marketplace.graylog.org/addons/3f132fab-50f0-4c88-b63d-9ac99aa6c20e";,
>>>
>>> the only diffrence is I have set the Queue name to fluidcm-notifiaction
>>> insted of cloudtrail-notification.
>>>
>>>
>>> 2016-04-16 21:11:25,899 ERROR:
>>> com.graylog2.input.cloudtrail.CloudTrailSubscriber - Could not read
>>> messages from SNS. This is most likely a misconfiguration of the plugin.
>>> Going into sleep loop and retrying.
>>> java.lang.RuntimeException: Could not parse SNS notification: hi
>>> at
>>> com.graylog2.input.cloudtrail.notifications.CloudtrailSNSNotificationParser.parse(CloudtrailSNSNotificationParser.java:36)
>>> at
>>> com.graylog2.input.cloudtrail.notifications.CloudtrailSQSClient.getNotifications(CloudtrailSQSClient.java:48)
>>> at
>>> com.graylog2.input.cloudtrail.CloudTrailSubscriber.run(CloudTrailSubscriber.java:80)
>>> Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized
>>> token 'hi': was expecting ('true', 'false' or 'null')
>>> at [Source: hi; line: 1, column: 5]
>>> at
>>> com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1487)
>>> at
>>> com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:518)
>>> at
>>> com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:2299)
>>> at
>>> com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1458)
>>> at
>>> com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:683)
>>> at
>>> com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3602)
>>> at
>>> com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3547)
>>> at
>>> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2578)
>>> at
>>> com.graylog2.input.cloudtrail.notifications.CloudtrailSNSNotificationParser.parse(CloudtrailSNSNotificationParser.java:24)
>>> ... 2 more
>>>
>>> I'm not that knowledgeable about AWS, but I can't see how it's not
>>> working. It's dumping to the S3 bucket correctly within the AWS console. I
>>> have given full access to the user.
>>> Can anyone please tell me what wrong I have done?
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/edc29764-7336-4379-890c-fe1b409ed991%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
