> On 七月 25, 2016, 1:45 p.m., Attila Simon wrote:
> > Hi,
> >
> > Your change interfears with a caching mechanism. I believe the best way to
> > describe this interference is this test below. The problem in a nutshell is
> > when a wildcard is specified then the parent directory used by file
> > matching is not the immediate parent but the directory high above. Thus its
> > last modification time won't be updated when a new file was added. Only
> > mtime of dir1 is updated when file2.txt is added but caching is initialized
> > by fg1 directory so it only monitors that for changes. I think either a
> > documentation should state that when using wildcards then caching won't
> > work or even better an assertation should check this combination at
> > startup. (Making cache work with wildcard would be the best).
> >
> >
> > @Test
> > public void testWildcardsDirFilteringCache() throws IOException,
> > InterruptedException {
> > //first iteration everything is working as expected
> > File f1 = new File(tmpDir.getAbsolutePath() + "/fg1/dir1/file1.txt");
> > Files.createParentDirs(f1);
> > Files.write("file1\n", f1, Charsets.UTF_8);
> >
> > Context context = new Context();
> > context.put(POSITION_FILE, posFilePath);
> > context.put(FILE_GROUPS, "fg1");
> > context.put(FILE_GROUPS_PREFIX + "fg1", tmpDir.getAbsolutePath() +
> > "/fg1/*/file.*");
> >
> > Configurables.configure(source, context);
> > source.start();
> > source.process();
> > Transaction txn = channel.getTransaction();
> > txn.begin();
> > List<String> out = Lists.newArrayList();
> > for (int i = 0; i < 2; i++) {
> > Event e = channel.take();
> > if (e != null) {
> > out.add(TestTaildirEventReader.bodyAsString(e));
> > }
> > }
> > txn.commit();
> > txn.close();
> >
> > // empty iterations simulating that time is passing by
> > Thread.sleep(1000);
> > source.process();
> > Thread.sleep(1000);
> >
> > //file was created after a while it should be picked up as well
> > File f2 = new File(tmpDir.getAbsolutePath() + "/fg1/dir1/file2.txt");
> > Files.write("file2\n", f2, Charsets.UTF_8);
> >
> > source.process();
> > txn = channel.getTransaction();
> > txn.begin();
> > for (int i = 0; i < 2; i++) {
> > Event e = channel.take();
> > if (e != null) {
> > out.add(TestTaildirEventReader.bodyAsString(e));
> > }
> > }
> > txn.commit();
> > txn.close();
> >
> > assertEquals(2, out.size()); //fails as file2.txt won't appear in the
> > channel ever
> > assertTrue(out.contains("file1"));
> > assertTrue(out.contains("file2"));
> > }
Thanks for your review. I will modify the code asap.
- qiao
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50378/#review143374
-----------------------------------------------------------
On 七月 24, 2016, 10:37 a.m., qiao wen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50378/
> -----------------------------------------------------------
>
> (Updated 七月 24, 2016, 10:37 a.m.)
>
>
> Review request for Flume.
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> In our log management project, we wan't to track many log files like this:
> /app/dir1/log.*
> /app/dir2/log.*
> ...
> /app/dirn/log.*
> But TaildirSource can't support wildcards in filegroup directory name. The
> following config is expected:
> a1.sources.r1.filegroups.fg = /app/*/log.*
>
>
> Diffs
> -----
>
> flume-ng-doc/sphinx/FlumeUserGuide.rst 1334500
>
> flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/TaildirMatcher.java
> ad9f720
>
> flume-ng-sources/flume-taildir-source/src/test/java/org/apache/flume/source/taildir/TestTaildirSource.java
> 097ee0b
>
> Diff: https://reviews.apache.org/r/50378/diff/
>
>
> Testing
> -------
>
> All tests in TestTaildirSource passed.
>
>
> Thanks,
>
> qiao wen
>
>
