[
https://issues.apache.org/jira/browse/FLUME-3352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17046255#comment-17046255
]
ASF subversion and git services commented on FLUME-3352:
--------------------------------------------------------
Commit 5c9bfe6dc745d85f2459958b767e6e05550d8a81 in flume's branch
refs/heads/trunk from Yang.Tao
[ https://gitbox.apache.org/repos/asf?p=flume.git;h=5c9bfe6 ]
FLUME-3352: Skip spooldir canary when tracker dir is used (#314)
* Unnecessary canary test will block on readonly spooldir while another
trackerdir is set.
* Add unit tests to ensure TrackingPolicy.RENAME / TrackingPolicy.TRACKER_DIR
effective on readonly spooldir
> Unnecessary canary test will block on readonly spooldir while another
> trackerdir is set.
> ----------------------------------------------------------------------------------------
>
> Key: FLUME-3352
> URL: https://issues.apache.org/jira/browse/FLUME-3352
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: 1.9.0
> Reporter: taoyang
> Priority: Blocker
> Labels: patch, pull-request-available
> Fix For: 1.10.0
>
> Attachments: FLUME-3352-0.patch
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> *Phenomenon*
> In many cases, we have just read permission on spoolDir and write permission
> on trackerDir.
> However whenever flume starts for spooldir source , it will always try to
> create a '.canary' file in the spooling directory.
> Then it leads to the failure of some processing unnecessarily.
> *Recur*
> (Usually spooldir is mounted readonly from a nas in production and this time
> we create a readonly directory instead.)
> First we create the spoolDir by root user and create a file in it and it is
> readonly for others by default.
> {code:java}
> su root
> mkdir /home/hadoop/testspooldir
> echo 'foo' > /home/hadoop/testspooldir/bar{code}
> Then switch to another user (hadoop) who runs flume and make sure it has read
> permission for the spooldir.
> {code:java}
> su hadoop
> mkdir /home/hadoop/testtrackerdir
> ll /home/hadoop/testspooldir
> >> total 4-rw-r--r-- 1 root root 4 Jan 16 19:15 bar{code}
> now create the example.conf:
> {code:java}
> a1.sources = r1
> a1.sinks = k1
> a1.channels = c1
> a1.sources.r1.type = spooldir
> a1.sources.r1.deletePolicy = never
> a1.sources.r1.spoolDir = /home/hadoop/testspooldir
> a1.sources.r1.trackerDir = /home/hadoop/testtrackerdir
> a1.sources.r1.trackingPolicy = tracker_dir
> a1.sinks.k1.type = logger
> a1.channels.c1.type = memory
> a1.channels.c1.capacity = 1000
> a1.channels.c1.transactionCapacity = 100
> a1.sources.r1.channels = c1
> a1.sinks.k1.channel = c1
> {code}
> and start flume with it
> {code:java}
> bin/flume-ng agent --conf conf --conf-file conf/example.conf --name a1
> -Dflume.root.logger=INFO,console{code}
> then the IOException is thrown.
> {code:java}
> 2020-01-16 19:16:12,777 (lifecycleSupervisor-1-0) [ERROR -
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)]
> Unabl2020-01-16 19:16:12,777 (lifecycleSupervisor-1-0) [ERROR -
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)]
> Unable to start EventDrivenSourceRunner: { source:Spool Directory source r1:
> { spoolDir: /home/hadoop/testspooldir } } - Exception
> follows.org.apache.flume.FlumeException: Unable to read and modify files in
> the spooling directory: /home/hadoop/testspooldir at
> org.apache.flume.client.avro.ReliableSpoolingFileEventReader.<init>(ReliableSpoolingFileEventReader.java:195)
> at
> org.apache.flume.client.avro.ReliableSpoolingFileEventReader.<init>(ReliableSpoolingFileEventReader.java:89)
> at
> org.apache.flume.client.avro.ReliableSpoolingFileEventReader$Builder.build(ReliableSpoolingFileEventReader.java:882)
> at
> org.apache.flume.source.SpoolDirectorySource.start(SpoolDirectorySource.java:111)
> at
> org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
> at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:249)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)Caused by: java.io.IOException:
> Permission denied at java.io.UnixFileSystem.createFileExclusively(Native
> Method) at java.io.File.createTempFile(File.java:2024) at
> org.apache.flume.client.avro.ReliableSpoolingFileEventReader.<init>(ReliableSpoolingFileEventReader.java:185)
> ... 12 more{code}
> *Fix*
> We just add the condition where this trick is necessary.
> The pr/patch will be submitted as as shown below.
> Or let it still exist and using warning log instead of exception thrown is
> better?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
