Hi,
The sample payload from the AWS account created at
{"s3Bucket":"fluidcmlogs","
s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/11/924399563845_CloudTrail_us-west-1_20160511T0715Z_FCFmYjMFmTZFthnu.json.gz"]}
is as follows
{"Records":[{"eventVersion":"1.03","userIdentity":{"type":"IAMUser","principalId":"AIDAIZDWJD4XSD7OPQYUW",
"arn":"arn:aws:iam::924399563845:user/nileshk","accountId":"924399563845","userName":"nileshk",
"sessionContext":{"attributes":{"mfaAuthenticated":"false","creationDate":"2016-05-11T07:11:24Z"}},
"invokedBy":"signin.amazonaws.com"},"eventTime":"2016-05-11T07:12:45Z","eventSource":"s3.amazonaws.com",
"eventName":"GetBucketVersioning","awsRegion":"us-west-1","sourceIPAddress":"14.140.226.238",
"userAgent":"signin.amazonaws.com","requestParameters":{"bucketName":"fluidcmlogs"},
"responseElements":null,"requestID":"66745CC932793217","eventID":"08f194cd-52c8-41eb-9619-beee1af30074",
"eventType":"AwsApiCall","recipientAccountId":"924399563845"},{"eventVersion":"1.03",
"userIdentity":{"type":"IAMUser","principalId":"AIDAIZDWJD4XSD7OPQYUW","arn":"arn:aws:iam::924399563845:user/nileshk",
"accountId":"924399563845","userName":"nileshk","sessionContext":{"attributes":{"mfaAuthenticated":"false",
"creationDate":"2016-05-11T07:11:24Z"}},"invokedBy":"signin.amazonaws.com"},"eventTime":"2016-05-11T07:12:46Z",
"eventSource":"s3.amazonaws.com","eventName":"GetBucketVersioning","awsRegion":"us-west-1","sourceIPAddress":"14.140.226.238",
"userAgent":"signin.amazonaws.com","requestParameters":{"bucketName":"fluidcmlogs"},"responseElements":null,"requestID":"E1360FC36112F5A8",
"eventID":"cab707ed-f20b-40ea-ac3f-6e208fc6eb4a","eventType":"AwsApiCall","recipientAccountId":"924399563845"}]}
is this format not supported by the plugin or is there something elase which I
am missing, I have shared the AWS and graylog configuration in the previous
post.
please point in some direction to resolve this issue.
Thanks in advance!!
Anant.
On Thursday, 12 May 2016 13:25:06 UTC+5:30, Anant Sawant wrote:
>
> Hi Jochen!!,
>
> The AWS plugin is not reading any logs, though it is reaching to the AWS
> cloudtrail successfully. I am sharing the payload sample and the
> configuration I have done in both Graylog and AWS. Please tell me if any
> thing wrong I have done doing it.
>
> this is one of the value that mesage.body() from
> "CloudtrailSNSNotificationParser" is returing :-
> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/11/924399563845_CloudTrail_us-west-1_20160511T0715Z_FCFmYjMFmTZFthnu.json.gz"]}
>
> this exists at the AWS.
>
> Following is the content which I got after manually extracting the above
> mentioned
> 924399563845_CloudTrail_us-west-1_20160511T0715Z_FCFmYjMFmTZFthnu.json.gz
> :-
>
> {"Records":[{"eventVersion":"1.03","userIdentity":{"type":"IAMUser","principalId":"AIDAIZDWJD4XSD7OPQYUW",
> "arn":"arn:aws:iam::924399563845:user/nileshk","accountId":"924399563845","userName":"nileshk",
> "sessionContext":{"attributes":{"mfaAuthenticated":"false","creationDate":"2016-05-11T07:11:24Z"}},
> "invokedBy":"signin.amazonaws.com"},"eventTime":"2016-05-11T07:12:45Z","eventSource":"s3.amazonaws.com",
> "eventName":"GetBucketVersioning","awsRegion":"us-west-1","sourceIPAddress":"14.140.226.238",
> "userAgent":"signin.amazonaws.com","requestParameters":{"bucketName":"fluidcmlogs"},
> "responseElements":null,"requestID":"66745CC932793217","eventID":"08f194cd-52c8-41eb-9619-beee1af30074",
> "eventType":"AwsApiCall","recipientAccountId":"924399563845"},{"eventVersion":"1.03",
> "userIdentity":{"type":"IAMUser","principalId":"AIDAIZDWJD4XSD7OPQYUW","arn":"arn:aws:iam::924399563845:user/nileshk",
> "accountId":"924399563845","userName":"nileshk","sessionContext":{"attributes":{"mfaAuthenticated":"false",
> "creationDate":"2016-05-11T07:11:24Z"}},"invokedBy":"signin.amazonaws.com"},"eventTime":"2016-05-11T07:12:46Z",
> "eventSource":"s3.amazonaws.com","eventName":"GetBucketVersioning","awsRegion":"us-west-1","sourceIPAddress":"14.140.226.238",
> "userAgent":"signin.amazonaws.com","requestParameters":{"bucketName":"fluidcmlogs"},"responseElements":null,"requestID":"E1360FC36112F5A8",
> "eventID":"cab707ed-f20b-40ea-ac3f-6e208fc6eb4a","eventType":"AwsApiCall","recipientAccountId":"924399563845"}]}
>
> Setting/configuration at the AWS account(for doing this followed the
> following
> "https://marketplace.graylog.org/addons/3f132fab-50f0-4c88-b63d-9ac99aa6c20e";).
>
>
> Step 1: Enabling CloudTrail for an AWS region
>
> Trail name:Fluidcm-CloudTrail
> S3 bucket: fluidcmlogs
> Log file prefix: fluidcm
> SNS notification for every log file delivery: Yes
> SNS topic:cloudtrail-log-write
>
> Step 2: Set up SQS for CloudTrail write notifications
>
> Queue Name: fluidcm-notifications
> Kept Default setting as it is..
>
> Step 3: Created Aws Policy for s3 bucket and SQS for full access user.
>
> The region where your new queue will be created:US West (N. California)
>
> Seetin/configuration at Graylog.
>
> Title :- Aws CloudTrail
> AWS Region :- US_WEST_1
> AWS access key :- auto generated
> SQS queue name :- fluidcm-notifications
> AWS secret key :- auto generated
>
> On Wednesday, 11 May 2016 11:32:07 UTC+5:30, Anant Sawant wrote:
>>
>> HI Jochen!!
>>
>> Thanks for the reply. The error has been resolved. The plugin is able to
>> connect to AWS and also get the message body as per the
>> "CloudtrailSNSNotificationParser" class from the plugin but no logs are
>> visible in Graylog. I inserted my own logs to see if the plugin is reading
>> events from AWS or not, and found that it receiving the message's body,
>> "message.getBody()" methid is returning the following
>>
>>
>> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/10/924399563845_CloudTrail_us-west-1_20160510T1220Z_Dd8u8fCREYcu0Bd8.json.gz"]}
>>
>> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/10/924399563845_CloudTrail_us-west-1_20160510T1750Z_ePOdk3E0lg1KL5vt.json.gz"]}
>>
>> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/ap-northeast-1/2016/05/06/924399563845_CloudTrail_ap-northeast-1_20160506T1010Z_LE3fKktT1wVK1vA5.json.gz"]}
>>
>> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/10/924399563845_CloudTrail_us-west-1_20160510T1315Z_aKJCNFF9np7FC0Gg.json.gz"]}
>>
>> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/10/924399563845_CloudTrail_us-west-1_20160510T2250Z_a2TsampYHKq5baC8.json.gz"]}
>>
>> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/ap-northeast-2/2016/05/06/924399563845_CloudTrail_ap-northeast-2_20160506T0935Z_g7rcYdIFmA4ymndh.json.gz"]}
>>
>> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/06/924399563845_CloudTrail_us-west-1_20160506T1010Z_IzqYaYzIcsBdcOBu.json.gz"]}
>>
>> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/10/924399563845_CloudTrail_us-west-1_20160510T1750Z_ePOdk3E0lg1KL5vt.json.gz"]}
>>
>> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/ap-northeast-1/2016/05/06/924399563845_CloudTrail_ap-northeast-1_20160506T1010Z_LE3fKktT1wVK1vA5.json.gz"]}
>>
>> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/10/924399563845_CloudTrail_us-west-1_20160510T1640Z_39iM51Yqif0mMMRJ.json.gz"]}
>>
>> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/10/924399563845_CloudTrail_us-west-1_20160510T1315Z_aKJCNFF9np7FC0Gg.json.gz"]}
>>
>> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/10/924399563845_CloudTrail_us-west-1_20160510T2250Z_a2TsampYHKq5baC8.json.gz"]}
>> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-east-1/2016/05/06/924399563845_CloudTrail_us-east-1_20160506T1255Z_YloXCOB1lCnODpp4.json.gz"]}
>>
>> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/10/924399563845_CloudTrail_us-west-1_20160510T1900Z_GIsgmOpucjjQQ6Pm.json.gz"]}
>>
>> {"s3Bucket":"fluidcmlogs","s3ObjectKey":["fluidcm/AWSLogs/924399563845/CloudTrail/us-west-1/2016/05/10/924399563845_CloudTrail_us-west-1_20160510T1405Z_39oVS7OoNWENcRaQ.json.gz"]}
>>
>> and this exists on AWS.
>>
>> but call to "envelope.message" in the same class
>> "CloudtrailSNSNotificationParser" returns "null". Is this the reason why we
>> are not getting any events in Graylog. Please give me some advice to
>> overcome this situation.
>>
>> Thanks in advance.
>> Anant
>>
>>
>> On Monday, 9 May 2016 18:02:27 UTC+5:30, Jochen Schalanda wrote:
>>>
>>> Hi Anant,
>>>
>>> it looks like the plugin is expecting some value in the payload to be a
>>> boolean type but received a string.
>>>
>>> Do you still have access to the raw message payload? The string "hi"
>>> suggests that this was simply a test message.
>>>
>>> Cheers,
>>> Jochen
>>>
>>> On Friday, 6 May 2016 13:02:38 UTC+2, Anant Sawant wrote:
>>>>
>>>> Hi Everyone!!,
>>>>
>>>> I went through the documentation for setting up the Cloudtrail plugin
>>>> for US-WEST-1, but I am getting the following error.
>>>> I have done all the AWS settings/configuration as described at "
>>>> https://marketplace.graylog.org/addons/3f132fab-50f0-4c88-b63d-9ac99aa6c20e";,
>>>>
>>>> the only diffrence is I have set the Queue name to fluidcm-notifiaction
>>>> insted of cloudtrail-notification.
>>>>
>>>>
>>>> 2016-04-16 21:11:25,899 ERROR:
>>>> com.graylog2.input.cloudtrail.CloudTrailSubscriber - Could not read
>>>> messages from SNS. This is most likely a misconfiguration of the plugin.
>>>> Going into sleep loop and retrying.
>>>> java.lang.RuntimeException: Could not parse SNS notification: hi
>>>> at
>>>> com.graylog2.input.cloudtrail.notifications.CloudtrailSNSNotificationParser.parse(CloudtrailSNSNotificationParser.java:36)
>>>> at
>>>> com.graylog2.input.cloudtrail.notifications.CloudtrailSQSClient.getNotifications(CloudtrailSQSClient.java:48)
>>>> at
>>>> com.graylog2.input.cloudtrail.CloudTrailSubscriber.run(CloudTrailSubscriber.java:80)
>>>> Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized
>>>> token 'hi': was expecting ('true', 'false' or 'null')
>>>> at [Source: hi; line: 1, column: 5]
>>>> at
>>>> com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1487)
>>>> at
>>>> com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:518)
>>>> at
>>>> com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:2299)
>>>> at
>>>> com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1458)
>>>> at
>>>> com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:683)
>>>> at
>>>> com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3602)
>>>> at
>>>> com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3547)
>>>> at
>>>> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2578)
>>>> at
>>>> com.graylog2.input.cloudtrail.notifications.CloudtrailSNSNotificationParser.parse(CloudtrailSNSNotificationParser.java:24)
>>>> ... 2 more
>>>>
>>>> I'm not that knowledgeable about AWS, but I can't see how it's not
>>>> working. It's dumping to the S3 bucket correctly within the AWS console. I
>>>> have given full access to the user.
>>>> Can anyone please tell me what wrong I have done?
>>>>
>>>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/43eff8e6-365d-411b-aae2-0df88fbfb966%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
