Hi Jochen!!,
The AWS plugin is not reading any logs, though it is reaching to the AWS
cloudtrail successfully. I am sharing the payload sample and the
configuration I have done in both Graylog and AWS. Please tell me if any
thing wrong I have done doing it.
this is one of the value that mesage.body() from
"CloudtrailSNSNotificationParser" is returing :-
{"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/11/924399563845_CloudTrail_us-west-1_20160511T0715Z_FCFmYjMFmTZFthnu.json.gz"]}
this exists at the AWS.
Following is the content which I got after manually extracting the above
mentioned
924399563845_CloudTrail_us-west-1_20160511T0715Z_FCFmYjMFmTZFthnu.json.gz
:-
{"Records":[{"eventVersion":"1.03","userIdentity":{"type":"IAMUser","principalId":"AIDAIZDWJD4XSD7OPQYUW",
"arn":"arn:aws:iam::924399563845:user/nileshk","accountId":"924399563845","userName":"nileshk",
"sessionContext":{"attributes":{"mfaAuthenticated":"false","creationDate":"2016-05-11T07:11:24Z"}},
"invokedBy":"signin.amazonaws.com"},"eventTime":"2016-05-11T07:12:45Z","eventSource":"s3.amazonaws.com",
"eventName":"GetBucketVersioning","awsRegion":"us-west-1","sourceIPAddress":"14.140.226.238",
"userAgent":"signin.amazonaws.com","requestParameters":{"bucketName":"fluidcmlogs"},
"responseElements":null,"requestID":"66745CC932793217","eventID":"08f194cd-52c8-41eb-9619-beee1af30074",
"eventType":"AwsApiCall","recipientAccountId":"924399563845"},{"eventVersion":"1.03",
"userIdentity":{"type":"IAMUser","principalId":"AIDAIZDWJD4XSD7OPQYUW","arn":"arn:aws:iam::924399563845:user/nileshk",
"accountId":"924399563845","userName":"nileshk","sessionContext":{"attributes":{"mfaAuthenticated":"false",
"creationDate":"2016-05-11T07:11:24Z"}},"invokedBy":"signin.amazonaws.com"},"eventTime":"2016-05-11T07:12:46Z",
"eventSource":"s3.amazonaws.com","eventName":"GetBucketVersioning","awsRegion":"us-west-1","sourceIPAddress":"14.140.226.238",
"userAgent":"signin.amazonaws.com","requestParameters":{"bucketName":"fluidcmlogs"},"responseElements":null,"requestID":"E1360FC36112F5A8",
"eventID":"cab707ed-f20b-40ea-ac3f-6e208fc6eb4a","eventType":"AwsApiCall","recipientAccountId":"924399563845"}]}
Setting/configuration at the AWS account(for doing this followed the following
"https://marketplace.graylog.org/addons/3f132fab-50f0-4c88-b63d-9ac99aa6c20e";).
Step 1: Enabling CloudTrail for an AWS region
Trail name:Fluidcm-CloudTrail
S3 bucket: fluidcmlogs
Log file prefix: fluidcm
SNS notification for every log file delivery: Yes
SNS topic:cloudtrail-log-write
Step 2: Set up SQS for CloudTrail write notifications
Queue Name: fluidcm-notifications
Kept Default setting as it is..
Step 3: Created Aws Policy for s3 bucket and SQS for full access user.
The region where your new queue will be created:US West (N. California)
Seetin/configuration at Graylog.
Title :- Aws CloudTrail
AWS Region :- US_WEST_1
AWS access key :- auto generated
SQS queue name :- fluidcm-notifications
AWS secret key :- auto generated
On Wednesday, 11 May 2016 11:32:07 UTC+5:30, Anant Sawant wrote:
>
> HI Jochen!!
>
> Thanks for the reply. The error has been resolved. The plugin is able to
> connect to AWS and also get the message body as per the
> "CloudtrailSNSNotificationParser" class from the plugin but no logs are
> visible in Graylog. I inserted my own logs to see if the plugin is reading
> events from AWS or not, and found that it receiving the message's body,
> "message.getBody()" methid is returning the following
>
>
> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/10/924399563845_CloudTrail_us-west-1_20160510T1220Z_Dd8u8fCREYcu0Bd8.json.gz"]}
>
> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/10/924399563845_CloudTrail_us-west-1_20160510T1750Z_ePOdk3E0lg1KL5vt.json.gz"]}
>
> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/ap-northeast-1/2016/05/06/924399563845_CloudTrail_ap-northeast-1_20160506T1010Z_LE3fKktT1wVK1vA5.json.gz"]}
>
> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/10/924399563845_CloudTrail_us-west-1_20160510T1315Z_aKJCNFF9np7FC0Gg.json.gz"]}
>
> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/10/924399563845_CloudTrail_us-west-1_20160510T2250Z_a2TsampYHKq5baC8.json.gz"]}
>
> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/ap-northeast-2/2016/05/06/924399563845_CloudTrail_ap-northeast-2_20160506T0935Z_g7rcYdIFmA4ymndh.json.gz"]}
>
> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/06/924399563845_CloudTrail_us-west-1_20160506T1010Z_IzqYaYzIcsBdcOBu.json.gz"]}
>
> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/10/924399563845_CloudTrail_us-west-1_20160510T1750Z_ePOdk3E0lg1KL5vt.json.gz"]}
>
> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/ap-northeast-1/2016/05/06/924399563845_CloudTrail_ap-northeast-1_20160506T1010Z_LE3fKktT1wVK1vA5.json.gz"]}
>
> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/10/924399563845_CloudTrail_us-west-1_20160510T1640Z_39iM51Yqif0mMMRJ.json.gz"]}
>
> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/10/924399563845_CloudTrail_us-west-1_20160510T1315Z_aKJCNFF9np7FC0Gg.json.gz"]}
>
> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/10/924399563845_CloudTrail_us-west-1_20160510T2250Z_a2TsampYHKq5baC8.json.gz"]}
> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-east-1/2016/05/06/924399563845_CloudTrail_us-east-1_20160506T1255Z_YloXCOB1lCnODpp4.json.gz"]}
>
> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/10/924399563845_CloudTrail_us-west-1_20160510T1900Z_GIsgmOpucjjQQ6Pm.json.gz"]}
>
> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/10/924399563845_CloudTrail_us-west-1_20160510T1405Z_39oVS7OoNWENcRaQ.json.gz"]}
>
> and this exists on AWS.
>
> but call to "envelope.message" in the same class
> "CloudtrailSNSNotificationParser" returns "null". Is this the reason why we
> are not getting any events in Graylog. Please give me some advice to
> overcome this situation.
>
> Thanks in advance.
> Anant
>
>
> On Monday, 9 May 2016 18:02:27 UTC+5:30, Jochen Schalanda wrote:
>>
>> Hi Anant,
>>
>> it looks like the plugin is expecting some value in the payload to be a
>> boolean type but received a string.
>>
>> Do you still have access to the raw message payload? The string "hi"
>> suggests that this was simply a test message.
>>
>> Cheers,
>> Jochen
>>
>> On Friday, 6 May 2016 13:02:38 UTC+2, Anant Sawant wrote:
>>>
>>> Hi Everyone!!,
>>>
>>> I went through the documentation for setting up the Cloudtrail plugin
>>> for US-WEST-1, but I am getting the following error.
>>> I have done all the AWS settings/configuration as described at "
>>> https://marketplace.graylog.org/addons/3f132fab-50f0-4c88-b63d-9ac99aa6c20e";,
>>>
>>> the only diffrence is I have set the Queue name to fluidcm-notifiaction
>>> insted of cloudtrail-notification.
>>>
>>>
>>> 2016-04-16 21:11:25,899 ERROR:
>>> com.graylog2.input.cloudtrail.CloudTrailSubscriber - Could not read
>>> messages from SNS. This is most likely a misconfiguration of the plugin.
>>> Going into sleep loop and retrying.
>>> java.lang.RuntimeException: Could not parse SNS notification: hi
>>> at
>>> com.graylog2.input.cloudtrail.notifications.CloudtrailSNSNotificationParser.parse(CloudtrailSNSNotificationParser.java:36)
>>> at
>>> com.graylog2.input.cloudtrail.notifications.CloudtrailSQSClient.getNotifications(CloudtrailSQSClient.java:48)
>>> at
>>> com.graylog2.input.cloudtrail.CloudTrailSubscriber.run(CloudTrailSubscriber.java:80)
>>> Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized
>>> token 'hi': was expecting ('true', 'false' or 'null')
>>> at [Source: hi; line: 1, column: 5]
>>> at
>>> com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1487)
>>> at
>>> com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:518)
>>> at
>>> com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:2299)
>>> at
>>> com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1458)
>>> at
>>> com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:683)
>>> at
>>> com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3602)
>>> at
>>> com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3547)
>>> at
>>> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2578)
>>> at
>>> com.graylog2.input.cloudtrail.notifications.CloudtrailSNSNotificationParser.parse(CloudtrailSNSNotificationParser.java:24)
>>> ... 2 more
>>>
>>> I'm not that knowledgeable about AWS, but I can't see how it's not
>>> working. It's dumping to the S3 bucket correctly within the AWS console. I
>>> have given full access to the user.
>>> Can anyone please tell me what wrong I have done?
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/edc29764-7336-4379-890c-fe1b409ed991%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
