MX created FLUME-2380:
-------------------------
Summary: Encrypted files from spooldir to S3 using file channel
Key: FLUME-2380
URL: https://issues.apache.org/jira/browse/FLUME-2380
Project: Flume
Issue Type: Question
Components: File Channel
Environment: Linux \ S3
Reporter: MX
Priority: Blocker
0down votefavorite
I am a newbie to flume tring to move data from spooldir to S3, I am
successfully able to move the data and roll the files into a bucket. I need to
encrypt the data loaded into S3. I tried the following configuration in flume ,
log files tells me that "Encryption is enabled" but the files are written in S3
without encryption. Can someone tell me if flume encryption works with the
config?
flume.conf
-----------
agent.sources = eventlog
agent.channels = file_channel
agent.sinks = s3sink
# Define / Configure source
agent.sources.eventlog.type = spooldir
agent.sources.eventlog.spoolDir = /test/data
agent.sources.eventlog.restart = true
agent.sources.eventlog.batchSize = 1000
#agent.sources.eventlog.type = seq
# HDFS sinks
agent.sinks.s3sink.type = hdfs
agent.sinks.s3sink.hdfs.path = s3n://<ACCESSKEY>:<SECRETKEY>@<BUCKET>
agent.sinks.s3sink.hdfs.fileType = DataStream
agent.sinks.s3sink.hdfs.filePrefix = event
agent.sinks.s3sink.hdfs.fileSuffix = .log
agent.sinks.s3sink.hdfs.writeFormat = Text
agent.sinks.s3sink.hdfs.batchSize = 1000
agent.sinks.s3sink.hdfs.rollSize = 10485760
agent.sinks.s3sink.hdfs.rollCount = 0
agent.sinks.s3sink.hdfs.rollInterval = 60
agent.sinks.s3sink.hdfs.threadsPoolSize = 100
# Use a channel which buffers events in memory
agent.channels.file_channel.type = file
#25GB
agent.channels.fileChannel.maxFileSize = 25000000000
agent.channels.file_channel.checkpointDir = /var/flume/checkpoint
agent.channels.file_channel.dataDirs = /<directory>/data
agent.channels.file_channel.capacity = 50000000
agent.channels.file_channel.transactionCapacity = 10000
agent.channels.file_channel.encryption.activeKey = key-1
agent.channels.file_channel.encryption.cipherProvider = AESCTRNOPADDING
#agent.channels.file_channel.encryption.keyProvider = key-provider-1
agent.channels.file_channel.encryption.keyProvider = JCEKSFILE
agent.channels.file_channel.encryption.keyProvider.keys = key-1
agent.channels.file_channel.encryption.keyProvider.keyStoreFile =
/<directory>/my.keystore
agent.channels.file_channel.encryption.keyProvider.keyStorePasswordFile =
/<directory>/my.keystore.password
# Bind the source and sink to the channel
agent.sources.eventlog.channels = file_channel
agent.sinks.s3sink.channel = file_channel
--
This message was sent by Atlassian JIRA
(v6.2#6252)
