Unit tests
Hi guys, I know that there are a few tests that are problematic, i.e. may fail arbitrarily. Please reply here with all tests that are known to be flaky. I will create a jira issue for fixing that. Obviously, we won't fix all tests in the project at this moment, but we can definitely fix that in the future. Currently, the goal I think we should achieve is stabilizing the tests and make sure that we don't have false build failures. Thanks
[jira] [Commented] (FLUME-2935) Bump java target version to 1.7
[ https://issues.apache.org/jira/browse/FLUME-2935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15343684#comment-15343684 ] Lior Zeno commented on FLUME-2935: -- This includes pom modification, documentation update, making sure that Jenkins is configured correctly with the right environment and jvm version, and finally, making sure that the project build succeeds and that all tests pass. > Bump java target version to 1.7 > --- > > Key: FLUME-2935 > URL: https://issues.apache.org/jira/browse/FLUME-2935 > Project: Flume > Issue Type: Task >Affects Versions: v1.7.0 >Reporter: Lior Zeno > Fix For: v1.7.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLUME-2762) Improve HDFS Sink performance
[ https://issues.apache.org/jira/browse/FLUME-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik resolved FLUME-2762. Resolution: Invalid Release Note: my thoughts didn't pan out on prototyping > Improve HDFS Sink performance > - > > Key: FLUME-2762 > URL: https://issues.apache.org/jira/browse/FLUME-2762 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Reporter: Roshan Naik >Assignee: Roshan Naik > Fix For: v1.7.0 > > > Have some thoughts around improving HDFS sink's performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2807) Add a simple split interceptor
[ https://issues.apache.org/jira/browse/FLUME-2807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2807: - Fix Version/s: (was: v1.6.0) > Add a simple split interceptor > --- > > Key: FLUME-2807 > URL: https://issues.apache.org/jira/browse/FLUME-2807 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.6.0, v1.7.0 >Reporter: seekerak > Labels: features, patch > Fix For: v1.7.0 > > Attachments: FLUME-2807.patch > > > a simple split interceptor , aims to deal with the situation as follows: > the source data like this: > “ > 1,tom,boy,13 > 2,lili,girl,14 > 3,jack,boy,10 > ... > ” > and i hope to sink source data into two different hdfs directories named by > boy and girl like this: > “ > hdfs:///sink/boy/ > hdfs:///sink/girl/ > ” > we can use this interceptor to accomplish this goal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2132) Exception while syncing from Flume to HDFS
[
https://issues.apache.org/jira/browse/FLUME-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lior Zeno updated FLUME-2132:
-
Fix Version/s: (was: v1.3.0)
v1.7.0
> Exception while syncing from Flume to HDFS
> --
>
> Key: FLUME-2132
> URL: https://issues.apache.org/jira/browse/FLUME-2132
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: v1.3.0
> Environment: Flume 1.3.0, Hadoop 1.2.0, 8GB RAM, Intel Pentium core 2
> duo
>Reporter: Divya R
> Labels: flume, hadoop
> Fix For: v1.7.0
>
>
> I'm running hadoop 1.2.0 and flume 1.3.0. Every thing works fine if its
> independently run. When I start my tomcat I get the below exception after
> some time.
> {quote}2013-07-17 12:40:35,640 (ResponseProcessor for block
> blk_5249456272858461891_436734) [WARN -
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:3015)]
> DFSOutputStream ResponseProcessor exception for block
> blk_5249456272858461891_436734java.net.SocketTimeoutException: 63000 millis
> timeout while waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/127.0.0.1:24433
> remote=/127.0.0.1:50010]
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> at java.io.DataInputStream.readFully(DataInputStream.java:195)
> at java.io.DataInputStream.readLong(DataInputStream.java:416)
> at
> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:124)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2967){quote}
> {quote}2013-07-17 12:40:35,800 (hdfs-hdfs-write-roll-timer-0) [WARN -
> org.apache.flume.sink.hdfs.BucketWriter.doClose(BucketWriter.java:277)]
> failed to close() HDFSWriter for file
> (hdfs://localhost:9000/flume/Broadsoft_App2/20130717/jboss/Broadsoft_App2.1374044838498.tmp).
> Exception follows.
> java.io.IOException: All datanodes 127.0.0.1:50010 are bad. Aborting...
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3096)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2100(DFSClient.java:2589)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2793){quote}
> Java snippet for Configuraion
> {quote}configuration.set("fs.default.name", "hdfs://localhost:9000");
> configuration.set("mapred.job.tracker", "hdfs://localhost:9000");{quote}
> I'm using a single datanode to read the files that where written to hdfs by
> flume, my java program just reads the files from hdfs to show it on the
> screen nothing much.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLUME-2882) Add Generic configuration provider
[ https://issues.apache.org/jira/browse/FLUME-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2882: - Fix Version/s: (was: v1.6.0) v1.7.0 > Add Generic configuration provider > -- > > Key: FLUME-2882 > URL: https://issues.apache.org/jira/browse/FLUME-2882 > Project: Flume > Issue Type: Improvement > Components: Configuration >Reporter: Enrique Ruiz Garcia > Fix For: v1.7.0 > > Attachments: FLUME-2882.patch > > > Add the ability to specify custom configuration provider to flume node (use > new optional '-confprovider' option to a specify class name that you can > implement from GenericConfigurationProvider class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2717) Add commons-io dependency into hadoop-2 profile to enable Flume 1.5 to support Hadoop 2.7
[ https://issues.apache.org/jira/browse/FLUME-2717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2717: - Fix Version/s: (was: v1.5.0) v1.7.0 > Add commons-io dependency into hadoop-2 profile to enable Flume 1.5 to > support Hadoop 2.7 > - > > Key: FLUME-2717 > URL: https://issues.apache.org/jira/browse/FLUME-2717 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.5.0 >Reporter: sam liu >Assignee: li xiang > Fix For: v1.7.0 > > Attachments: FLUME-2717-001.patch > > > By default, for branch origin/flume-1.5, the hadoop2.version is 2.4.0. > However, if we want to use hadoop-2.7.0 in flume-1.5, some hadoop regarding > tests will fail with exception 'java.lang.NoClassDefFoundError: > org/apache/commons/io/Charsets'. > The missed class Charsets is a new class in commons-io 2.4 jar file which is > invoked by hadoop-2.7.0, however flume-1.5 depends on commons-io 2.1 which > does not include the class 'Charsets'. > Therefore the solution to enable flume-1.5 to support hadoop-2.7.0 is to add > commons-io 2.4 as a dependency into the hadoop-2 profile. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2461) memoryChannel bytesRemaining counting error
[
https://issues.apache.org/jira/browse/FLUME-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lior Zeno updated FLUME-2461:
-
Fix Version/s: (was: v1.5.0.1)
v1.7.0
> memoryChannel bytesRemaining counting error
> ---
>
> Key: FLUME-2461
> URL: https://issues.apache.org/jira/browse/FLUME-2461
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: v1.5.0.1
>Reporter: yangwei
>Priority: Minor
> Labels: patch
> Fix For: v1.7.0
>
> Attachments: FLUME-2461.patch
>
>
> In doRollback function putByteCounter permits are released by bytesRemaining.
> This is wrong for the below cases:
> In the doCommit function:
> 1)
> if(!bytesRemaining.tryAcquire(putByteCounter, keepAlive,
> TimeUnit.SECONDS)) {
> throw new ChannelException("Cannot commit transaction. Heap space "
> +
> "limit of " + byteCapacity + "reached. Please increase heap
> space" +
> " allocated to the channel as the sinks may not be keeping up " +
> "with the sources");
> }
> 2)
> if(!queueRemaining.tryAcquire(-remainingChange, keepAlive,
> TimeUnit.SECONDS)) {
> bytesRemaining.release(putByteCounter);
> throw new ChannelFullException("Space for commit to queue couldn't
> be acquired." +
> " Sinks are likely not keeping up with sources, or the buffer
> size is too tight");
> }
> When they throw ChannelException, bytesRemaining should not release any
> permits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLUME-2318) SpoolingDirectory is unable to handle empty files
[
https://issues.apache.org/jira/browse/FLUME-2318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lior Zeno updated FLUME-2318:
-
Fix Version/s: (was: v1.4.0)
v1.7.0
> SpoolingDirectory is unable to handle empty files
> -
>
> Key: FLUME-2318
> URL: https://issues.apache.org/jira/browse/FLUME-2318
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: v1.4.0
>Reporter: Muhammad Ehsan ul Haque
>Priority: Minor
> Labels: easytest, patch
> Fix For: v1.7.0
>
> Attachments: FLUME-2318-0.patch, FLUME-2318-1.patch,
> FLUME-2318-2.patch
>
>
> Empty files should be returned as an empty event instead of no event.
> h4. Scenario
> From the start consume files in this order
> # f1: File with data or empty file
> # f2: Empty File
> # No file in spooling directory
> h4. Expected Outcome
> # channel.take() should return event with f1 data.
> # channel.take() should return event with f2 data (empty data).
> # channel.take() should return null.
> h4. What happens
> # channel.take() returns event with f1 data.
> # channel.take() returns null.
> # Exception is raised when the SpoolDirectorySource thread tries to read
> events from the ReliableSpoolingFileEventReader. Snippet of trace is
> 2014-02-09 15:46:35,832 (pool-1-thread-1) [INFO -
> org.apache.flume.client.avro.ReliableSpoolingFileEventReader.rollCurrentFile(ReliableSpoolingFileEventReader.java:346)]
> Preparing to move file /tmp/1391957195572-0/file1 to
> /tmp/1391957195572-0/file1.COMPLETED
> 2014-02-09 15:46:36,334 (pool-1-thread-1) [INFO -
> org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:228)]
> Last read was never committed - resetting mark position.
> 2014-02-09 15:46:36,335 (pool-1-thread-1) [INFO -
> org.apache.flume.client.avro.ReliableSpoolingFileEventReader.rollCurrentFile(ReliableSpoolingFileEventReader.java:346)]
> Preparing to move file /tmp/1391957195572-0/file2 to
> /tmp/1391957195572-0/file2.COMPLETED
> 2014-02-09 15:46:36,839 (pool-1-thread-1) [ERROR -
> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:252)]
> FATAL: Spool Directory source null: { spoolDir: /tmp/1391957195572-0 }:
> Uncaught exception in SpoolDirectorySource thread. Restart or reconfigure
> Flume to continue processing.
> java.lang.IllegalStateException: File should not roll when commit is
> outstanding.
> at
> org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:225)
> at
> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:224)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> h4. Unit Test
> In TestSpoolDirectorySource
> {code}
> @Test
> public void testWithEmptyFile2()
> throws InterruptedException, IOException {
> Context context = new Context();
> File f1 = new File(tmpDir.getAbsolutePath() + "/file1");
> Files.write("some data".getBytes(), f1);
> File f2 = new File(tmpDir.getAbsolutePath() + "/file2");
> Files.write(new byte[0], f2);
> context.put(SpoolDirectorySourceConfigurationConstants.SPOOL_DIRECTORY,
> tmpDir.getAbsolutePath());
> Configurables.configure(source, context);
> source.start();
> Thread.sleep(10);
> for (int i=0; i<2; i++) {
> Transaction txn = channel.getTransaction();
> txn.begin();
> Event e = channel.take();
> txn.commit();
> txn.close();
> }
> Transaction txn = channel.getTransaction();
> txn.begin();
> Assert.assertNull(channel.take());
> txn.commit();
> txn.close();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLUME-2689) reloading conf file leads syslogTcpSource not receives any event
[ https://issues.apache.org/jira/browse/FLUME-2689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2689: - Fix Version/s: (was: v1.5.2) v1.7.0 > reloading conf file leads syslogTcpSource not receives any event > > > Key: FLUME-2689 > URL: https://issues.apache.org/jira/browse/FLUME-2689 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: v1.5.1 > Environment: configuring syslog sending logs to remote flume agent >Reporter: yangwei > Fix For: v1.7.0 > > Attachments: flume-2689-0.patch > > > Reloading conf file will stop old syslog source and start new syslog source. > Stopping syslog tcp source only closes the NioServerSocketChannel, resulting > in the client sends data through the old channel. In that case, the new > source never receives data. The tcpdump shows the events have received but > the new source doesn't and ss shows the client connection stays same with old > one. > The right way to stop syslog source is close both the NioSocketChannel and > NioServerSocketChannel, and shutdown the executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2619) Spooldir source does not log channel exceptions
[ https://issues.apache.org/jira/browse/FLUME-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2619: - Fix Version/s: (was: v1.6.0) v1.7.0 > Spooldir source does not log channel exceptions > --- > > Key: FLUME-2619 > URL: https://issues.apache.org/jira/browse/FLUME-2619 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Reporter: Santiago M. Mola >Assignee: Johny Rufus > Fix For: v1.7.0 > > Attachments: FLUME-2619-1.patch, FLUME-2619.patch > > > Spooldir assumes that any ChannelException means that the channel is full and > it does not log the exception message. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-1092) We should have a consistent shutdown contract with all Flume components
[ https://issues.apache.org/jira/browse/FLUME-1092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-1092: - Fix Version/s: (was: v1.1.0) v1.7.0 > We should have a consistent shutdown contract with all Flume components > --- > > Key: FLUME-1092 > URL: https://issues.apache.org/jira/browse/FLUME-1092 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: v1.1.0 >Reporter: Will McQueen > Fix For: v1.7.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2579) JMS source support durable subscriptions and message listening
[ https://issues.apache.org/jira/browse/FLUME-2579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2579: - Fix Version/s: (was: v1.5.2) v1.7.0 > JMS source support durable subscriptions and message listening > -- > > Key: FLUME-2579 > URL: https://issues.apache.org/jira/browse/FLUME-2579 > Project: Flume > Issue Type: Wish > Components: Sinks+Sources >Affects Versions: v1.5.2 >Reporter: zhangshengbing > Fix For: v1.7.0 > > > best wishes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2762) Improve HDFS Sink performance
[ https://issues.apache.org/jira/browse/FLUME-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2762: - Fix Version/s: (was: v1.6.0) v1.7.0 > Improve HDFS Sink performance > - > > Key: FLUME-2762 > URL: https://issues.apache.org/jira/browse/FLUME-2762 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Reporter: Roshan Naik >Assignee: Roshan Naik > Fix For: v1.7.0 > > > Have some thoughts around improving HDFS sink's performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2871) avro sink reset-connection-interval cause EventDeliveryException
[
https://issues.apache.org/jira/browse/FLUME-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lior Zeno updated FLUME-2871:
-
Fix Version/s: (was: v1.6.0)
v1.7.0
> avro sink reset-connection-interval cause EventDeliveryException
>
>
> Key: FLUME-2871
> URL: https://issues.apache.org/jira/browse/FLUME-2871
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: v1.6.0
>Reporter: chenchunbin
> Fix For: v1.7.0
>
>
> I found avro sink use reset-connection-interval will throw this exception:
> 29 Jan 2016 14:01:45,257 ERROR
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.SinkRunner$PollingRunner.run:160) - Unable to deliver
> event. Exception follows.
> org.apache.flume.EventDeliveryException: Failed to send events
> at
> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:392)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:724)
> Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient {
> host: localhost, port: 58989 }: Failed to send batch
> at
> org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:315)
> at
> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:376)
> ... 3 more
> Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient {
> host: localhost, port: 58989 }: Handshake timed out after 3ms
> at
> org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:359)
> at
> org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:303)
> ... 4 more
> Caused by: java.util.concurrent.TimeoutException
> at java.util.concurrent.FutureTask.get(FutureTask.java:201)
> at
> org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:357)
> ... 5 more
> and then I repace netty-3.5.12.Final.jar to netty-3.10.5.Final.jar, the avro
> sink works well.
> #client.conf
> agent1.channels = c1
> agent1.sources = s1
> agent1.sinks= k1
> agent1.channels = c1
> agent1.channels.c1.type = memory
> agent1.channels.c1.capacity = 1
> agent1.channels.c1.transactionCapacity = 100
> agent1.sources.s1.type = exec
> agent1.sources.s1.command = tail -F /var/log/secure
> agent1.sources.s1.restart = true
> agent1.sources.s1.channels = c1
> agent1.sinks.k1.channel = c1
> agent1.sinks.k1.type = avro
> agent1.sinks.k1.hostname = 127.0.0.1
> agent1.sinks.k1.port = 58989
> agent1.sinks.k1.batch-size = 100
> agent1.sinks.k1.reset-connection-interval = 120
> agent1.sinks.k1.compression-type = deflate
> agent1.sinks.k1.compression-level = 6
> agent1.sinks.k1.connect-timeout = 3
> agent1.sinks.k1.request-timeout = 3
> #center.conf
> agent1.channels = c1
> agent1.sources = s1
> agent1.sinks= k1
> agent1.channels = c1
> agent1.channels.c1.type = memory
> agent1.channels.c1.capacity = 1
> agent1.channels.c1.transactionCapacity = 100
> agent1.sources.s1.type = avro
> agent1.sources.s1.bind = 0.0.0.0
> agent1.sources.s1.port = 58989
> agent1.sources.s1.threads = 1 #fast failed
> agent1.sources.s1.channels = c1
> agent1.sources.s1.compression-type = deflate
> agent1.sinks.k1.type = file_roll
> agent1.sinks.k1.sink.directory = /tmp/center1
> agent1.sinks.k1.channel = c1
> agent1.sinks.k1.sink.rollInterval = 86400
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (FLUME-2847) Dot notation in field names of ElastricSearch sink
[ https://issues.apache.org/jira/browse/FLUME-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno resolved FLUME-2847. -- Resolution: Not A Problem > Dot notation in field names of ElastricSearch sink > -- > > Key: FLUME-2847 > URL: https://issues.apache.org/jira/browse/FLUME-2847 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: v1.6.0 >Reporter: Ilija Vukotic > Fix For: v1.6.0 > > > The new version of ElasticSearch (2.0) does not support field names with dot > "." in the name. Flume sink by default makes fields named "@message.xxx" or > "@field.yyy". While this was simply annoying before, now it breaks it. Would > be great if you could fix it so not everybody has to write it's own sink. > In case there is a way to set fields names it is nowhere documented. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2825) Avro Files are not readable with the converter of HDP2.3
[
https://issues.apache.org/jira/browse/FLUME-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lior Zeno updated FLUME-2825:
-
Fix Version/s: (was: v1.5.2)
v1.7.0
> Avro Files are not readable with the converter of HDP2.3
>
>
> Key: FLUME-2825
> URL: https://issues.apache.org/jira/browse/FLUME-2825
> Project: Flume
> Issue Type: Blog - New Blog Request
> Components: Sinks+Sources
>Affects Versions: v1.5.2
> Environment: HDP2.3
>Reporter: Kettler Karl
> Fix For: v1.7.0
>
>
> Avro Files are not readable with the converter of HDP2.3
> What we can do?
> Obj avro.schema�
> {"type":"record","name":"Doc","doc":"adoc","fields":[{"name":"id","type":"string"},{"name":"user_friends_count","type":["int","null"]},{"name":"user_location","type":["string","null"]},{"name":"user_description","type":["string","null"]},{"name":"user_statuses_count","type":["int","null"]},{"name":"user_followers_count","type":["int","null"]},{"name":"user_name","type":["string","null"]},{"name":"user_screen_name","type":["string","null"]},{"name":"created_at","type":["string","null"]},{"name":"text","type":["string","null"]},{"name":"retweet_count","type":["long","null"]},{"name":"retweeted","type":["boolean","null"]},{"name":"in_reply_to_user_id","type":["long","null"]},{"name":"source","type":["string","null"]},{"name":"in_reply_to_status_id","type":["long","null"]},{"name":"media_url_https","type":["string","null"]},{"name":"expanded_url","type":["string","null"]}]}����}z��]~ /y)��$657453578462875648�,With
> Long Stroke McGee$Inmate ID: 3893978�� ��
> NeeshaYerdMe_CMB(2015-10-23T07:08:42Z0@_ThisIsImani thank you!��ޠ �
> http://twitter.com/download/iphone; rel="nofollow">Twitter for
> iPhoneКߟ����}z��]~ /y)�
> Kind regards,
> Karl
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (FLUME-2824) How does Flume write Json
[ https://issues.apache.org/jira/browse/FLUME-2824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno resolved FLUME-2824. -- Resolution: Not A Problem > How does Flume write Json > - > > Key: FLUME-2824 > URL: https://issues.apache.org/jira/browse/FLUME-2824 > Project: Flume > Issue Type: Blog - New Blog Request > Components: Sinks+Sources >Affects Versions: v1.5.2 > Environment: HDP2.3 >Reporter: Kettler Karl > Fix For: v1.5.2 > > > How does Flume write in Jason? > We get always Twitter data in Avro format. > TwitterAgent.sources = Twitter > TwitterAgent.channels = MemChannel > TwitterAgent.sinks = HDFS > TwitterAgent.sources.Twitter.type = > org.apache.flume.source.twitter.TwitterSource > TwitterAgent.sources.Twitter.channels = MemChannel > TwitterAgent.sources.Twitter.consumerKey = xxx > TwitterAgent.sources.Twitter.consumerSecret = xxx > TwitterAgent.sources.Twitter.accessToken = xxx > TwitterAgent.sources.Twitter.accessTokenSecret = xxx > TwitterAgent.sources.Twitter.maxBatchSize = 10 > TwitterAgent.sources.Twitter.maxBatchDurationMillis = 200 > TwitterAgent.sources.Twitter.keywords = United Nations > TwitterAgent.sources.Twitter.deserializer.schemaType = LITERAL > TwitterAgent.sinks.HDFS.channel = MemChannel > TwitterAgent.sinks.HDFS.type = hdfs > TwitterAgent.sinks.HDFS.hdfs.path = /demo/tweets/stream/%y-%m-%d/%H%M%S > TwitterAgent.sinks.HDFS.hdfs.filePrefix = events > TwitterAgent.sinks.HDFS.hdfs.round = true > TwitterAgent.sinks.HDFS.hdfs.roundValue = 5 > TwitterAgent.sinks.HDFS.hdfs.roundUnit = minute > TwitterAgent.sinks.HDFS.hdfs.useLocalTimeStamp = true > TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream > TwitterAgent.channels.MemChannel.type = memory > TwitterAgent.channels.MemChannel.capacity = 1000 > TwitterAgent.channels.MemChannel.transactionCapacity = 100 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLUME-2120) Capture Metrics to Monitor SyslogUDPSource and SyslogTCPSource
[ https://issues.apache.org/jira/browse/FLUME-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno resolved FLUME-2120. -- Resolution: Duplicate > Capture Metrics to Monitor SyslogUDPSource and SyslogTCPSource > -- > > Key: FLUME-2120 > URL: https://issues.apache.org/jira/browse/FLUME-2120 > Project: Flume > Issue Type: New Feature > Components: Sinks+Sources >Affects Versions: v1.4.0 >Reporter: Venkatesh Sivasubramanian >Assignee: Venkatesh Sivasubramanian > Fix For: v1.4.1 > > Attachments: FLUME-2120.4.patch, FLUME-2120.patch > > > Need ability to track the number of events received and accepted for the > SyslogUDPSource and SyslogTCPSource. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2573) flume-ng --conf parameter is not used when starting a flume agent
[ https://issues.apache.org/jira/browse/FLUME-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biju Nair updated FLUME-2573: - Description: flume-ng script accepts the parameter [conf|https://github.com/apache/flume/blob/trunk/bin/flume-ng#L183] to pass the directory which stores the configuration files and adds it to the FLUME_CLASSPATH. But [org.apache.flume.node.Application|https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/Application.java#L302] expects configuration files passed to have an absolute path. So even if user passes `conf` and `conf-file` option, `application` looks for the configuration file in the directory from which it is executed. It would be good to take the `conf` value into consideration so that users can place the config files anywhere. Currently big-top init.d script joins the [`conf` and `conf-file` values|(https://github.com/apache/flume/blob/trunk/bin/flume-ng#L183] when calling flume-ng. The same can be done in the ` flume-ng ` script. Alternatively the `conf ` parameter can be read in ` application ` and concatenated in the code. The latter is a cleaner approach but will have impact on ` big-top `. was: flume-ng script accepts the parameter [conf|https://github.com/apache/flume/blob/trunk/bin/flume-ng#L183] to pass the directory which stores the configuration files and adds it to the FLUME_CLASSPATH. But [org.apache.flume.node.Application|https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/Application.java#L302] expects configuration files passed to have an absolute path. So even if user passes `conf` and `conf-file` option, `application` looks for the configuration file in the directory from which it is executed. It would be good to take the `conf` value into consideration so that users can place the config files anywhere. Currently big-top init.d script joins the [`conf` and `conf-file` values|(https://github.com/apache/flume/blob/trunk/bin/flume-ng#L183] when calling flume-ng. The same can be done in the ` flume-ng ` script. Alternatively the `conf ` parameter can be read in ` application ` and concatenated in the code. The latter is a cleaner approach but will have impact on ` big-top `. > flume-ng --conf parameter is not used when starting a flume agent > - > > Key: FLUME-2573 > URL: https://issues.apache.org/jira/browse/FLUME-2573 > Project: Flume > Issue Type: Bug > Components: Node >Affects Versions: v1.4.0 > Environment: ubuntu 12 & Flume 1.4.0 >Reporter: Biju Nair > Labels: flume-ng > Fix For: v1.7.0 > > Attachments: FLUME-2573-0.patch > > > flume-ng script accepts the parameter > [conf|https://github.com/apache/flume/blob/trunk/bin/flume-ng#L183] to pass > the directory which stores the configuration files and adds it to the > FLUME_CLASSPATH. But > [org.apache.flume.node.Application|https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/Application.java#L302] > expects configuration files passed to have an absolute path. So even if user > passes `conf` and `conf-file` option, `application` looks for the > configuration file in the directory from which it is executed. It would be > good to take the `conf` value into consideration so that users can place the > config files anywhere. > Currently big-top init.d script joins the [`conf` and `conf-file` > values|(https://github.com/apache/flume/blob/trunk/bin/flume-ng#L183] when > calling flume-ng. The same can be done in the ` flume-ng ` script. > Alternatively the `conf ` parameter can be read in ` application ` and > concatenated in the code. The latter is a cleaner approach but will have > impact on ` big-top `. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2573) flume-ng --conf parameter is not used when starting a flume agent
[ https://issues.apache.org/jira/browse/FLUME-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2573: - Fix Version/s: (was: v1.4.1) v1.7.0 > flume-ng --conf parameter is not used when starting a flume agent > - > > Key: FLUME-2573 > URL: https://issues.apache.org/jira/browse/FLUME-2573 > Project: Flume > Issue Type: Bug > Components: Node >Affects Versions: v1.4.0 > Environment: ubuntu 12 & Flume 1.4.0 >Reporter: Biju Nair > Labels: flume-ng > Fix For: v1.7.0 > > Attachments: FLUME-2573-0.patch > > > flume-ng script accepts the parameter > [conf|https://github.com/apache/flume/blob/trunk/bin/flume-ng#L183] to pass > the directory which stores the configuration files and adds it to the > FLUME_CLASSPATH. > But > [org.apache.flume.node.Application|https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/Application.java#L302] > expects > configuration files passed to have an absolute path. So even if user passes > `conf` and `conf-file` option, `application` looks for the configuration file > in the directory from which it is executed. It would be good to take the > `conf` value into consideration so that users can place the config files > anywhere. > Currently big-top init.d script joins the [`conf` and `conf-file` > values|(https://github.com/apache/flume/blob/trunk/bin/flume-ng#L183] when > calling flume-ng. The same can be done in the ` flume-ng ` script. > Alternatively > the `conf ` parameter can be read in ` application ` and concatenated in the > code. The latter is a cleaner approach but will have impact on ` big-top `. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2143) Flume build occasionally fails with OutOfMemoryError on Windows.
[ https://issues.apache.org/jira/browse/FLUME-2143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2143: - Fix Version/s: (was: v1.4.1) v1.7.0 > Flume build occasionally fails with OutOfMemoryError on Windows. > > > Key: FLUME-2143 > URL: https://issues.apache.org/jira/browse/FLUME-2143 > Project: Flume > Issue Type: Bug > Components: Build, Windows >Reporter: Roshan Naik >Assignee: Roshan Naik > Labels: windows > Fix For: v1.7.0 > > Attachments: FLUME-2143.patch > > > The build sporadically fails in Flume with the following exception: > [exec] The system is out of resources. > [exec] Consult the following stack trace for details. > [exec] java.lang.OutOfMemoryError: PermGen space > [exec] at java.lang.ClassLoader.defineClass1(Native Method) > [exec] at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) > [exec] at java.lang.ClassLoader.defineClass(ClassLoader.java:615) > [exec] at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) > [exec] at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) > [exec] at java.net.URLClassLoader.access$000(URLClassLoader.java:58) > [exec] at java.net.URLClassLoader$1.run(URLClassLoader.java:197) > [exec] at java.security.AccessController.doPrivileged(Native Method) > [exec] at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > [exec] at > org.codehaus.plexus.compiler.javac.IsolatedClassLoader.loadClass(IsolatedClassLoader.java:56) > [exec] at com.sun.tools.javac.comp.Lower.mapForEnum(Lower.java:319) > [exec] at com.sun.tools.javac.comp.Lower.visitEnumSwitch(Lower.java:3075) > [exec] at com.sun.tools.javac.comp.Lower.visitSwitch(Lower.java:3066) > [exec] at > com.sun.tools.javac.tree.JCTree$JCSwitch.accept(JCTree.java:896) > [exec] at > com.sun.tools.javac.tree.TreeTranslator.translate(TreeTranslator.java:44) > [exec] at com.sun.tools.javac.comp.Lower.translate(Lower.java:1899) > [exec] at > com.sun.tools.javac.tree.TreeTranslator.translate(TreeTranslator.java:56) > [exec] at > com.sun.tools.javac.tree.TreeTranslator.visitBlock(TreeTranslator.java:146) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLUME-953) Enable ability to pass arguments to the FlumeNodeDaemon through Windows Service
[ https://issues.apache.org/jira/browse/FLUME-953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno resolved FLUME-953. - Resolution: Not A Problem > Enable ability to pass arguments to the FlumeNodeDaemon through Windows > Service > --- > > Key: FLUME-953 > URL: https://issues.apache.org/jira/browse/FLUME-953 > Project: Flume > Issue Type: New Feature > Components: Windows >Affects Versions: v0.9.5 > Environment: Windows >Reporter: Thomas Rega >Priority: Minor > Labels: WAP, arguments, daemon, flume, jetty, service, windows > Fix For: v0.9.5 > > Original Estimate: 4h > Remaining Estimate: 4h > > Currently, we are unable to pass in agruments to the > FlumeNodeDaemon.WindowsService method other than Start and Stop. Particularly > a useful command would be to start, but with the jetty wap disabled. Really, > we should be able to specify any arguments (such as -s), by placing them > after Start. For example, Start -s. > I have already written this change locally and would like suggestions before > I submit it for review. Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLUME-2651) Connection problem for KAFKA via ZOOKEEPER
[
https://issues.apache.org/jira/browse/FLUME-2651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lior Zeno resolved FLUME-2651.
--
Resolution: Not A Problem
> Connection problem for KAFKA via ZOOKEEPER
> --
>
> Key: FLUME-2651
> URL: https://issues.apache.org/jira/browse/FLUME-2651
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
>Affects Versions: v0.9.4
> Environment: cloudera 5.3
>Reporter: Serdar Mustaoglu
> Fix For: v0.9.5
>
>
> When I try to get messages from kafka into hdfs file via FLUME, I have
> connections problems.
> metadata.broker.list must be used , props.put("zk.connect", "node2:2181")
> not supported.
> How can connect to kafka via zookeper.
> tier1.sources = source1
> tier1.channels = channel1
> tier1.sinks= sink1
> tier1.sources.source1.type = org.apache.flume.source.kafka.KafkaSource
> tier1.sources.source1.zookeeperConnect = quickstart.cloudera:2181
> tier1.sources.source1.topic = topic1
> tier1.sources.source1.groupId = flume
> tier1.sources.source1.channels = channel1
> tier1.sources.source1.interceptors = i1
> tier1.sources.source1.interceptors.i1.type = timestamp
> tier1.sources.source1.kafka.consumer.timeout.ms = 100
> tier1.channels.channel1.type = memory
> tier1.channels.channel1.capacity = 1
> tier1.channels.channel1.transactionCapacity = 1000
> tier1.sinks.sink1.type = hdfs
> tier1.sinks.sink1.hdfs.path= /user/cloudera/kafka/%{topic}/%y-%m-%d
> tier1.sinks.sink1.hdfs.rollInterval = 5
> tier1.sinks.sink1.hdfs.rollSize = 0
> tier1.sinks.sink1.hdfs.rollCount = 0
> tier1.sinks.sink1.hdfs.fileType = DataStream
> tier1.sinks.sink1.channel = channel1
>
>
> actually, how can configure kafka listening port ?
> java
> Properties props = new Properties();
> ///props.put("zk.connect", "192.168.6.129:2181"); - not working
> reqires metadata.broker.list
> props.put("metadata.broker.list", "192.168.6.129:2181");
> props.put("serializer.class", "kafka.serializer.StringEncoder");
>
> ProducerConfig config = new ProducerConfig(props);
> producer = new Producer(config);
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLUME-402) Add a "Known Issues" section to flume documentation.
[ https://issues.apache.org/jira/browse/FLUME-402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-402: Fix Version/s: (was: v0.9.5) v1.7.0 > Add a "Known Issues" section to flume documentation. > > > Key: FLUME-402 > URL: https://issues.apache.org/jira/browse/FLUME-402 > Project: Flume > Issue Type: Documentation > Components: Docs, Technical Debt >Affects Versions: v0.9.4 >Reporter: Jonathan Hsieh > Fix For: v1.7.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLUME-2935) Bump java target version to 1.7
Lior Zeno created FLUME-2935: Summary: Bump java target version to 1.7 Key: FLUME-2935 URL: https://issues.apache.org/jira/browse/FLUME-2935 Project: Flume Issue Type: Task Affects Versions: v1.7.0 Reporter: Lior Zeno Fix For: v1.7.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 49025: FLUME-1899: Make SpoolDir work with Sub-Directories
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49025/#review138910 --- Since Flume 1.7+ will now only support Java 7+, why don't we consider using Java 7's new DirectoryStream API, rather than the much more expensive listFiles API? This probably will result in a huge performance boost plus far simpler code structure. It will require some rewrite of the code, but since we have tests and we expect current behavior, I would suggest doing that. - Hari Shreedharan On June 21, 2016, 3:32 p.m., Balázs Donát Bessenyei wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/49025/ > --- > > (Updated June 21, 2016, 3:32 p.m.) > > > Review request for Flume, Denes Arvay, Mike Percy, and Attila Simon. > > > Repository: flume-git > > > Description > --- > > SpoolrDir currently monitors a directory and can not handle sub-directories. > This JIRA is to make SpoolDir able to walk down a source directory and > monitor new files. > > > Diffs > - > > > flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java > d54f415 > > flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java > 3fe947d > > flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java > 5053697 > > flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java > fe530ff > flume-ng-doc/sphinx/FlumeUserGuide.rst 74d2887 > > Diff: https://reviews.apache.org/r/49025/diff/ > > > Testing > --- > > Ran tests before the patch: > # mvn clean install -DskipTests -Drat.skip=true; mvn -pl flume-ng-core > -Drat.skip=true test > Tests run: 378, Failures: 0, Errors: 0, Skipped: 2 > > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 07:55 min > [INFO] Finished at: 2016-06-21T16:13:46+02:00 > [INFO] Final Memory: 35M/510M > [INFO] > > > After patch: > # mvn clean install -DskipTests -Drat.skip=true; mvn -pl flume-ng-core > -Drat.skip=true test > Tests run: 380, Failures: 0, Errors: 0, Skipped: 2 > > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 06:18 min > [INFO] Finished at: 2016-06-21T17:04:17+02:00 > [INFO] Final Memory: 35M/511M > [INFO] > > > Patch also includes docs > > > Thanks, > > Balázs Donát Bessenyei > >
[jira] [Updated] (FLUME-2934) Create documentation for TaildirSource
[ https://issues.apache.org/jira/browse/FLUME-2934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Simon updated FLUME-2934: Attachment: FLUME-2934.patch > Create documentation for TaildirSource > -- > > Key: FLUME-2934 > URL: https://issues.apache.org/jira/browse/FLUME-2934 > Project: Flume > Issue Type: Documentation > Components: Docs >Affects Versions: v1.6.0 >Reporter: Attila Simon >Assignee: Attila Simon > Labels: documentation > Fix For: v1.7.0 > > Attachments: FLUME-2934.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-1899) Make SpoolDir work with Sub-Directories
[ https://issues.apache.org/jira/browse/FLUME-1899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341988#comment-15341988 ] Bessenyei Balázs Donát commented on FLUME-1899: --- Attached patch based on trunk Please, review at https://reviews.apache.org/r/49025/ > Make SpoolDir work with Sub-Directories > --- > > Key: FLUME-1899 > URL: https://issues.apache.org/jira/browse/FLUME-1899 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.4.0, v1.3.1 >Reporter: Tzur Turkenitz >Assignee: Phil Scala >Priority: Minor > Labels: Sources, SpoolDir > Attachments: FLUM-1899-1.patch, FLUME-1899-2.patch, FLUME-1899-3.patch > > > SpoolrDir currently monitors a directory and can not handle sub-directories. > This JIRA is to make SpoolDir able to walk down a source directory and > monitor new files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-1899) Make SpoolDir work with Sub-Directories
[ https://issues.apache.org/jira/browse/FLUME-1899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bessenyei Balázs Donát updated FLUME-1899: -- Attachment: FLUME-1899-3.patch > Make SpoolDir work with Sub-Directories > --- > > Key: FLUME-1899 > URL: https://issues.apache.org/jira/browse/FLUME-1899 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.4.0, v1.3.1 >Reporter: Tzur Turkenitz >Assignee: Phil Scala >Priority: Minor > Labels: Sources, SpoolDir > Attachments: FLUM-1899-1.patch, FLUME-1899-2.patch, FLUME-1899-3.patch > > > SpoolrDir currently monitors a directory and can not handle sub-directories. > This JIRA is to make SpoolDir able to walk down a source directory and > monitor new files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (FLUME-2919) Upgrade the Solr version to 6.0.1
[ https://issues.apache.org/jira/browse/FLUME-2919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341339#comment-15341339 ] Minoru Osuka edited comment on FLUME-2919 at 6/21/16 9:18 AM: -- Solr 6 requires Java8. http://lucene.apache.org/solr/6_0_0/changes/Changes.html#v6.0.0.system_requirements https://issues.apache.org/jira/browse/LUCENE-5950 https://cwiki.apache.org/confluence/display/solr/Major+Changes+from+Solr+5+to+Solr+6#MajorChangesfromSolr5toSolr6-Java8Required Does Flume have palan to support Java8? was (Author: minoru): Solr 6 requires Java8. https://cwiki.apache.org/confluence/display/solr/Major+Changes+from+Solr+5+to+Solr+6#MajorChangesfromSolr5toSolr6-Java8Required Does Flume have palan to support Java8? > Upgrade the Solr version to 6.0.1 > - > > Key: FLUME-2919 > URL: https://issues.apache.org/jira/browse/FLUME-2919 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.6.0 >Reporter: Minoru Osuka > Fix For: v1.7.0 > > Attachments: FLUME-2919-1.patch, FLUME-2919-2.patch, FLUME-2919.patch > > > Flume morphline-solr-sink is using Solr 4.3.0. Recently, Solr 6.0.1 has been > released. I propose to upgrade to it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (FLUME-2919) Upgrade the Solr version to 6.0.1
[ https://issues.apache.org/jira/browse/FLUME-2919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341339#comment-15341339 ] Minoru Osuka edited comment on FLUME-2919 at 6/21/16 8:42 AM: -- Solr 6 requires Java8. https://cwiki.apache.org/confluence/display/solr/Major+Changes+from+Solr+5+to+Solr+6#MajorChangesfromSolr5toSolr6-Java8Required Does Flume have palan to support Java8? was (Author: minoru): Solr 6 require Java8. https://cwiki.apache.org/confluence/display/solr/Major+Changes+from+Solr+5+to+Solr+6#MajorChangesfromSolr5toSolr6-Java8Required Does Flume have palan to support Java8? > Upgrade the Solr version to 6.0.1 > - > > Key: FLUME-2919 > URL: https://issues.apache.org/jira/browse/FLUME-2919 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.6.0 >Reporter: Minoru Osuka > Fix For: v1.7.0 > > Attachments: FLUME-2919-1.patch, FLUME-2919-2.patch, FLUME-2919.patch > > > Flume morphline-solr-sink is using Solr 4.3.0. Recently, Solr 6.0.1 has been > released. I propose to upgrade to it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2919) Upgrade the Solr version to 6.0.1
[ https://issues.apache.org/jira/browse/FLUME-2919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341339#comment-15341339 ] Minoru Osuka commented on FLUME-2919: - Solr 6 require Java8. https://cwiki.apache.org/confluence/display/solr/Major+Changes+from+Solr+5+to+Solr+6#MajorChangesfromSolr5toSolr6-Java8Required Does Flume have palan to support Java8? > Upgrade the Solr version to 6.0.1 > - > > Key: FLUME-2919 > URL: https://issues.apache.org/jira/browse/FLUME-2919 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.6.0 >Reporter: Minoru Osuka > Fix For: v1.7.0 > > Attachments: FLUME-2919-1.patch, FLUME-2919-2.patch, FLUME-2919.patch > > > Flume morphline-solr-sink is using Solr 4.3.0. Recently, Solr 6.0.1 has been > released. I propose to upgrade to it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLUME-2826) Right Configuration of flume agent for Twitter data
[ https://issues.apache.org/jira/browse/FLUME-2826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno resolved FLUME-2826. -- Resolution: Not A Problem > Right Configuration of flume agent for Twitter data > --- > > Key: FLUME-2826 > URL: https://issues.apache.org/jira/browse/FLUME-2826 > Project: Flume > Issue Type: Blog - New Blog Request > Components: Configuration >Affects Versions: v1.5.2 >Reporter: Kettler Karl > Fix For: v1.5.2 > > > How does you get Keywords and language? > TwitterAgent.sources = Twitter > TwitterAgent.channels = MemChannel > TwitterAgent.sinks = HDFS > TwitterAgent.sources.Twitter.type = > org.apache.flume.source.twitter.TwitterSource > TwitterAgent.sources.Twitter.channels = MemChannel > TwitterAgent.sources.Twitter.consumerKey = xxx > TwitterAgent.sources.Twitter.consumerSecret = xxx > TwitterAgent.sources.Twitter.accessToken = xxx > TwitterAgent.sources.Twitter.accessTokenSecret = xxx > TwitterAgent.sources.Twitter.maxBatchSize = 10 > TwitterAgent.sources.Twitter.maxBatchDurationMillis = 200 > TwitterAgent.sources.Twitter.keywords = United Nations > TwitterAgent.sources.Twitter.deserializer.schemaType = LITERAL > TwitterAgent.sinks.HDFS.channel = MemChannel > TwitterAgent.sinks.HDFS.type = hdfs > TwitterAgent.sinks.HDFS.hdfs.path = /demo/tweets/stream/%y-%m-%d/%H%M%S > TwitterAgent.sinks.HDFS.hdfs.filePrefix = events > TwitterAgent.sinks.HDFS.hdfs.round = true > TwitterAgent.sinks.HDFS.hdfs.roundValue = 5 > TwitterAgent.sinks.HDFS.hdfs.roundUnit = minute > TwitterAgent.sinks.HDFS.hdfs.useLocalTimeStamp = true > TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream > TwitterAgent.channels.MemChannel.type = memory > TwitterAgent.channels.MemChannel.capacity = 1000 > TwitterAgent.channels.MemChannel.transactionCapacity = 100 > Kind regards, > Karl -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2826) Right Configuration of flume agent for Twitter data
[ https://issues.apache.org/jira/browse/FLUME-2826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341303#comment-15341303 ] Lior Zeno commented on FLUME-2826: -- Questions should be submitted to the users mailing list. Anyway, if this issue is still a concern, the TwitterSource does not support keywords or language configuration. > Right Configuration of flume agent for Twitter data > --- > > Key: FLUME-2826 > URL: https://issues.apache.org/jira/browse/FLUME-2826 > Project: Flume > Issue Type: Blog - New Blog Request > Components: Configuration >Affects Versions: v1.5.2 >Reporter: Kettler Karl > Fix For: v1.5.2 > > > How does you get Keywords and language? > TwitterAgent.sources = Twitter > TwitterAgent.channels = MemChannel > TwitterAgent.sinks = HDFS > TwitterAgent.sources.Twitter.type = > org.apache.flume.source.twitter.TwitterSource > TwitterAgent.sources.Twitter.channels = MemChannel > TwitterAgent.sources.Twitter.consumerKey = xxx > TwitterAgent.sources.Twitter.consumerSecret = xxx > TwitterAgent.sources.Twitter.accessToken = xxx > TwitterAgent.sources.Twitter.accessTokenSecret = xxx > TwitterAgent.sources.Twitter.maxBatchSize = 10 > TwitterAgent.sources.Twitter.maxBatchDurationMillis = 200 > TwitterAgent.sources.Twitter.keywords = United Nations > TwitterAgent.sources.Twitter.deserializer.schemaType = LITERAL > TwitterAgent.sinks.HDFS.channel = MemChannel > TwitterAgent.sinks.HDFS.type = hdfs > TwitterAgent.sinks.HDFS.hdfs.path = /demo/tweets/stream/%y-%m-%d/%H%M%S > TwitterAgent.sinks.HDFS.hdfs.filePrefix = events > TwitterAgent.sinks.HDFS.hdfs.round = true > TwitterAgent.sinks.HDFS.hdfs.roundValue = 5 > TwitterAgent.sinks.HDFS.hdfs.roundUnit = minute > TwitterAgent.sinks.HDFS.hdfs.useLocalTimeStamp = true > TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream > TwitterAgent.channels.MemChannel.type = memory > TwitterAgent.channels.MemChannel.capacity = 1000 > TwitterAgent.channels.MemChannel.transactionCapacity = 100 > Kind regards, > Karl -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLUME-2818) Problems with Avro data and not Json and no data in HDFS
[
https://issues.apache.org/jira/browse/FLUME-2818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lior Zeno resolved FLUME-2818.
--
Resolution: Not A Problem
> Problems with Avro data and not Json and no data in HDFS
>
>
> Key: FLUME-2818
> URL: https://issues.apache.org/jira/browse/FLUME-2818
> Project: Flume
> Issue Type: Request
> Components: Sinks+Sources
>Affects Versions: v1.5.2
> Environment: HDP-2.3.0.0-2557 Sandbox
>Reporter: Kettler Karl
>Priority: Critical
> Fix For: v1.5.2
>
>
> Flume supplies twitter data in avro format and not in Json.
> Why?
> Flume Config Agent:
> TwitterAgent.sources = Twitter
> TwitterAgent.channels = MemChannel
> TwitterAgent.sinks = HDFS
> TwitterAgent.sources.Twitter.type =
> org.apache.flume.source.twitter.TwitterSource
> TwitterAgent.sources.Twitter.channels = MemChannel
> TwitterAgent.sources.Twitter.consumerKey = xxx
> TwitterAgent.sources.Twitter.consumerSecret = xxx
> TwitterAgent.sources.Twitter.accessToken = xxx
> TwitterAgent.sources.Twitter.accessTokenSecret = xxx
> TwitterAgent.sources.Twitter.maxBatchSize = 10
> TwitterAgent.sources.Twitter.maxBatchDurationMillis = 200
> TwitterAgent.sources.Twitter.keywords = United Nations
> TwitterAgent.sources.Twitter.deserializer.schemaType = LITERAL
> # HDFS Sink
> TwitterAgent.sinks.HDFS.channel = MemChannel
> TwitterAgent.sinks.HDFS.type = hdfs
> TwitterAgent.sinks.HDFS.hdfs.path = /demo/tweets/stream/%y-%m-%d/%H%M%S
> TwitterAgent.sinks.HDFS.hdfs.filePrefix = events
> TwitterAgent.sinks.HDFS.hdfs.round = true
> TwitterAgent.sinks.HDFS.hdfs.roundValue = 5
> TwitterAgent.sinks.HDFS.hdfs.roundUnit = minute
> TwitterAgent.sinks.HDFS.hdfs.useLocalTimeStamp = true
> TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
> TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
> TwitterAgent.channels.MemChannel.type = memory
> TwitterAgent.channels.MemChannel.capacity = 1000
> TwitterAgent.channels.MemChannel.transactionCapacity = 100
> Twitter Data from Flume:
> Obj avro.schema�
> {"type":"record","name":"Doc","doc":"adoc","fields":[{"name":"id","type":"string"},{"name":"user_friends_count","type":["int","null"]},{"name":"user_location","type":["string","null"]},{"name":"user_description","type":["string","null"]},{"name":"user_statuses_count","type":["int","null"]},{"name":"user_followers_count","type":["int","null"]},{"name":"user_name","type":["string","null"]},{"name":"user_screen_name","type":["string","null"]},{"name":"created_at","type":["string","null"]},{"name":"text","type":["string","null"]},{"name":"retweet_count","type":["long","null"]},{"name":"retweeted","type":["boolean","null"]},{"name":"in_reply_to_user_id","type":["long","null"]},{"name":"source","type":["string","null"]},{"name":"in_reply_to_status_id","type":["long","null"]},{"name":"media_url_https","type":["string","null"]},{"name":"expanded_url","type":["string","null"]}]}�]3hˊى���|$656461386520784896�
> �お絵描きするショタコン/オタクまっしぐら。論破メインに雑食もぐもぐ/成人済み pixiv：323565 隔離：【@yh_u_】�n� ユハズ
> yhzz_(2015-10-20T13:26:05Z� はじめた～リセマラめんどくさいし緑茶来たから普通にこのまま進める
> https://t.co/ZpfDqw4l9g � http://twitter.com;
> rel="nofollow">Twitter Web Client ^
> https://pbs.twimg.com/media/CRw4Js3UAAAGusn.pngthttp://twitter.com/yhzz_/status/656461386520784896/photo/1$656461390677417984�
> no me veais ni noteis mi presencia no quiere decir que no os este observando
> desde las sombras�� � JKP® BakasumaUserSinCausa(2015-10-20T13:26:06Z� RT
> @NaiiVicious: @Lisi_Hattori @UserSinCausa https://t.co/M2LTJWwqae � http://twitter.com/download/android; rel="nofollow">Twitter for Android ^
> https://pbs.twimg.com/media/CRthC1mWUAIFTF-.jpg�
> http://twitter.com/NaiiVicious/status/656224896297529344/photo/1�]3hˊى���|���
> By loading this twitter data into a HDFS table. It is not possible to convert
> with avro-tools-1.7.7.jar. into Json. We get error message: "No data"
> If we want to read this file we get following error message:
> "java -jar avro-tools-1.7.7.jar tojson twitter.avro > twitter.json
> Exception in thread "main" org.apache.avro.AvroRuntimeException:
> java.io.EOFException"
> I hope you could help us.
> Kind regards,
> Karl
>
>
> Details
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2919) Upgrade the Solr version to 6.0.1
[ https://issues.apache.org/jira/browse/FLUME-2919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341196#comment-15341196 ] Lior Zeno commented on FLUME-2919: -- Since 4.8.0 Solr requires Java 7, do you have a source for that? http://lucene.apache.org/solr/news.html > Upgrade the Solr version to 6.0.1 > - > > Key: FLUME-2919 > URL: https://issues.apache.org/jira/browse/FLUME-2919 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.6.0 >Reporter: Minoru Osuka > Fix For: v1.7.0 > > Attachments: FLUME-2919-1.patch, FLUME-2919-2.patch, FLUME-2919.patch > > > Flume morphline-solr-sink is using Solr 4.3.0. Recently, Solr 6.0.1 has been > released. I propose to upgrade to it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2919) Upgrade the Solr version to 6.0.1
[ https://issues.apache.org/jira/browse/FLUME-2919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341170#comment-15341170 ] Hari Shreedharan commented on FLUME-2919: - Doesn't Solr 6 *require* Java 8? > Upgrade the Solr version to 6.0.1 > - > > Key: FLUME-2919 > URL: https://issues.apache.org/jira/browse/FLUME-2919 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.6.0 >Reporter: Minoru Osuka > Fix For: v1.7.0 > > Attachments: FLUME-2919-1.patch, FLUME-2919-2.patch, FLUME-2919.patch > > > Flume morphline-solr-sink is using Solr 4.3.0. Recently, Solr 6.0.1 has been > released. I propose to upgrade to it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
