Might want to checkout logstash
On Nov 2, 2012 3:14 AM, "Simon Monecke" wrote:
> Hi,
>
> i want to use Flume to collect logs in my network (~2 logs/sec).
> But now I'm not really sure, what are the differences between Flume and a
> MOM like ActiveMQ.
>
> I would say the flume agent on the cli
Can you please paste your config? What version of flume and hbase?
On Jun 18, 2013 1:24 AM, "nagesh" wrote:
>
> Hi,
> I am new to flume. I have tried HBaseSink it is working and hbase puts
> also working. But, when I changed it to Async HBase Sink flume not
> putting any info. to HBase. I am not
Ive been meaning to try ZFS with an SSD based SLOG/ZIL (intent log) for
this as it seems like a good use case.
something like:
pool
sdaN - ZIL (enterprise grade ssd with capacitor/battery for persisting
buffers in event of sudden power loss)
mirror
sda1
sda2
mirror
sda3
sda4
, or if say 5 seconds passes and only
250 new events are available in the source will it grab those on some time
based interval?
Thank you,
--
Iain Wright
Cell: (562) 852-5916
<http://www.labctsi.org/>
This email message is confidential, intended only for the recipient(s)
named above a
Congrats and thanks! We will likely adopt the spillover channel
--
Iain Wright
Cell: (562) 852-5916
<http://www.labctsi.org/>
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure
downstream flume aggregate tiers
should be easy to test by touching your flume config and seeing if its
re-initializes and spits out info to log with startup info
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is
Also need to see the config if possible.
RE: Embedded agent not supporting hbase sink, you could avro sink from
embeedded flume to an avro source on a sperate flume "tier", which
aggregates events from your servers and then writes to hbase using hbase
sink
--
Iain Wright
This email
actually want to extract several headers from an epoch (, MM, DD)
and use them independently as part of the key in an S3 URI , im guessing we
are looking at a custom serializer.
Appreciate the suppor and any thoughts on this
Thanks,
--
Iain Wright
This email message is confidential, intended
As a long time user, thanks and congrats from all of us at Telescope!
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the intended
Email this to unsub: user-unsubscr...@flume.apache.org
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the intended recipient, do
not
How often do you poll the metrics?
Have you checked flume logs?
Is flume starting up fine , then at some point not responding on metrics,
then you do something to bring it back up?
Or is it intermitently not responsive but fixes itself?
On Jul 22, 2015 5:49 PM, "George Blazer" wrote:
> I use :565
have GC logs enabled, check if GC could be causing the freeze.
> Nothing else comes is striking as of now, assuming the network is
> good.
>
> On Thu, Jul 23, 2015 at 12:09 AM, George Blazer wrote:
> > We poll metrics once a minute. It's pretty intermittent
>
INK"
},
"SOURCE.source1": {
"AppendAcceptedCount": "0",
"AppendBatchAcceptedCount": "423562",
"AppendBatchReceivedCount": "423562",
"AppendReceivedCount": "0",
spark before running analytics/reports on our data, maybe
something similar is possible for you.
Just curious, what rollsize/count/etc. is configured in your sink? hdfs
cluster specs? metal or cloud provider based?
Cheers,
--
Iain Wright
This email message is confidential, intended only fo
Hi Guyle,
We ran into the same thing.
Please see https://flume.apache.org/FlumeUserGuide.html#line
On the originating source/where the event enters flume for the first time,
increase maxLineLength, ie:
...
agent1.sources.source1.deserializer.maxLineLength = 1048576
...
Best,
--
Iain Wright
I'd expect it to work with any source, ive used it with exec &
spoolingdirsource
Cheers,
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If yo
dont exist
file: path={{install_directory}}/lib/{{item}} state=absent
with_items:
- avro-1.7.3.jar
- avro-ipc-1.7.3.jar
HTH,
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
discl
Hi robin,
rollCount defaults to 10
-- https://flume.apache.org/FlumeUserGuide.html#hdfs-sink
Best,
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If
If you just want to verify to that your source+interceptors/etc. are
correctly assigning header values to events, I believe you could
temporarily switch to logger sink to see the headers+body in the flume log
On Sep 18, 2015 3:51 PM, "Doug McClure" wrote:
> Is there a debug level or other Flume 1
Are you able to telnet to bigdata1 on port 9083?
On Oct 21, 2015 7:27 PM, "lizhenm...@163.com" wrote:
>
> hi all：
> I use flume to import data from syslog to hive,but encount the follow
> errors.
>
>
> 2015-10-22 10:05:05,115 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN
> - org.apache.f
.0.0.0:90830.0.0.0:
> * LISTEN 32119/java
> --
> lizhenm...@163.com
>
>
> *From:* iain wright
> *Date:* 2015-10-22 14:13
> *To:* user
> *Subject:* Re: flume hive sink not work
>
> Are you able to telnet to bigdata1 o
Congrats! Thanks for carrying the torch on one of our favorite tools
@Telescope
On Oct 23, 2015 5:08 PM, "Mike Percy" wrote:
> Congrats Hari!
>
> On Wed, Oct 21, 2015 at 5:50 PM, Arvind Prabhakar
> wrote:
>
>> Dear Flume Users and Developers,
>>
>> I have had the pleasure of serving as the PMC C
have some tooling inbetween, that
polled the API & posted the result to flume.
bind0.0.0.0The hostname or IP address to listen on
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
di
Glad you understand it more now.
How could the documentation be improved to make things more clear?
You are welcome to contribute a change, or suggest one here (
https://cwiki.apache.org/confluence/display/FLUME/How+to+Contribute)
--
Iain Wright
This email message is confidential, intended
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the intended recipient, do
not disclose or disseminate the message to anyone except the
5931243876.javamail.hud...@aegis.apache.org%3E
http://mail-archives.apache.org/mod_mbox/flume-dev/201209.mbox/%3c1657028524.38.1347067986653.javamail.hud...@aegis.apache.org%3E
Thanks so much,
--
Iain Wright
Cell: (562) 852-5916
<http://www.labctsi.org/>
This email message is confidential, intended on
Thanks Mike, i was able to get it installed on FreeBSD 9, Just had trouble
on the box that had FreeBSD 8.2 installed for some reason
All the best,
--
Iain Wright
<http://www.labctsi.org/>
This email message is confidential, intended only for the recipient(s)
named above and may c
java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
... 19 more
Thank you,
--
Iain Wright
<http://www.labctsi.org/>
This email message is confid
#FLUME_CLASSPATH=""
HBASE_HOME="/app/hbase"
HADOOP_HOME="/app/hadoop"
I've tried copying the sink from /flume-ng-sinks/blah/blah/hbase-blah.sink
to /app/flume/lib/ and setting my CLASSPATH & FLUME_CLASSPATH to
/app/flume/lib but same result.
Appreciate a
Thanks very much for the help & advice Hari
Respectfully,
--
Iain Wright
<http://www.labctsi.org/>
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you a
ed session 0x138a0620045bbce with negotiated timeout 18 for
client /10.10.10.64:44993
Thanks for the continued help from the community on getting us going w/flume
Cheers,
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above and may contain information
I don't mean to hijack the thread, but is this tiered approach recommended
over reading from a local queue and having 10 or so nodes write directly to
hbase when using the async hbase sink?
--
Iain Wright
<http://www.labctsi.org/>
This email message is confidential, intended o
prone to data loss in the docs.
Our source app is Java, we were just writing to a file with log4j.
Thank you and have have great day,
Iain wright
On Oct 25, 2012 9:22 AM, "Brock Noland" wrote:
> If you cannot use RPCclient (project is not in java), then writing the
> events to
IIRC the spool dir source is single threaded.
You could move your source files into 10 directories with 100k each, and
have 10 spooldir sources.
You will need to have fast channels, and likely multiple sinks or fan out,
to keep up with the source most likely.
--
Iain Wright
This email
ad of host, since host is a header
used by the host interceptor
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the intended recipient, d
Best,
iain
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the intended recipient, do
not disclose or disseminate the message to anyone
) - Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
A step in the right direction at least, good luck and please let us know if
you sort out whats going on w/the regex!
Cheers,
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above
Hi Sutanu,
This is working out as well:
multi-ale2-station.sources.source1.interceptors.i1.regex =
host=(\\w+-\\d+-\\w+.attwifi.com)
When in doubtescape i guess :p
Cheers,
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above and may contain
Awesome, glad I was able to help!
Cheers,
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the intended recipient, do
not disclose or
Hi,
These look relevant to your goals:
https://github.com/elastic/elasticsearch/issues/14187
https://github.com/lucidfrontier45/ElasticsearchSink2
HTH,
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is
Hi Muby,
Are the ES and lucene-core jars in your flume lib folder?
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the intended
You likely want to pose the ZK questions on the zookeeper list. I know I've
seen folks have problems when receiving >1MB of data in a response, and
definitely problems with > 200k children of a znode
That said I've used it with hbase 0.94-98 with ~20k regions without issue, I
believe region sev
e are rarely changed, but we need to support such use case when
> they do. We plan to cache the formula on heap and poll the primary source
> once in a while for update.
>
> Best regards,
> Thanh Hong.
>
> From: iain wright [mailto:iainw...@gmail.com]
> Sent: Thursday, 28
Hi Manish,
This is expected behavior of the spooling directory source:
https://flume.apache.org/FlumeUserGuide.html#spooling-directory-source
You will need to have the files flushed/completed elsewhere, and moved into
the spooling directory with a unique name.
--
Iain Wright
This email
Can you telnet master60 31002 from the server the flume agent is running on?
Sent from my iPhone
> On Aug 26, 2016, at 1:00 AM, 罗辉 wrote:
>
> 31002
Congrats! Very excited flume is getting attention from good devs 👍
Sent from my iPhone
> On Sep 19, 2016, at 6:11 PM, Johny Rufus John wrote:
>
> Congrats Bessenyei and Jeff !!
>
> Regards,
> Rufus
>
>> On Mon, Sep 19, 2016 at 4:43 PM, Mike Percy wrote:
>> Hi Apache Flume community,
>>
>> I
seems http 204 is exepected response from S3 delete, can you pastebin your
log/cfg (please omit keys if present in cfg)?
Does your sink stop consuming from the channel when that error is
encountered?
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named
Might be worth trying the debug output (I forget exact sink name) to just log
the headers being attached to events after the interceptor to validate the
regex is working correctly, and for all events.
I setup this exact config at previous company so I know it works.
I also remember needing to
ize increasing over N minutes, and successfulsinkcount not
changing) i dont remember the exact names of the metrics it's been awhile
the metric keys seemed to explain it well enough when i was using this in
the past, are there any specific keys in the response from /metrics you
don't u
Congrats! As a user -- thank you!
-iain
Sent from my iPhone
> On Mar 11, 2017, at 2:21 PM, Bessenyei Balázs Donát wrote:
>
> Hi Flume community and Mike,
>
> Thank you very much for this honour and opportunity.
>
> I'd like to say thank you to all who helped me along the way. Extra
> kudos
Can you please drop your config in a reply or pastebin (omitting any
sensitive info)
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the
auxww|grep -i flume after starting flume?
Are events flowing all the way through and flushed to files every 60s or
20k events? Do you see queueing in the channel? Is the OOM after some time?
JAVA_OPTS="-Xms1024m -Xmx3072m" previously worked for me in flume-env.sh
--
Iain Wright
This e
Does the output of JPS or PS show it was started with those parameters?
Maybe the env file is not being sourced in correctly and its starting with
a default heap?
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above and may contain information that
Nice, glad you were able to sort 👍
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the intended recipient, do
not disclose or
You can run as many flume agents as you want, they are shared nothing. So you
can achieve scale by having your application , or a flume sink, distribute
events over the wire to one of many agents downstream. The flume docs are a
good primer if you haven't read them yet
(https://flume.apache.o
o multiple downstream consumers,
route based on contents of the event, etc.
Is there something in particular you are hoping to accomplish? I think with
a more specific question, you will get a better answer
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named
Hi Sri,
Are you using a memory channel? What source/sink?
Can you please paste/link your obfuscated config
What does the metrics endpoint say in terms of channel size, sinkdrainsuccess
etc, for the period leading up to the OOM?
Best,
Iain
Sent from my iPhone
> On Jul 26, 2017, at 8:00 AM, A
u can get some more info about the heap allocation/do a
heap dump also with jvisualvm. It seems most likely those flags aren't
getting to the jvm
wrt. Monitoring, you can add -Dflume.monitoring.type=HTTP
-Dflume.monitoring.port=34548 to expose the metrics endpoint on http://
:34548/metrics
eriod.
How long does it take for the OOM to occur?
Are events making it through both sinks to their destinations?
Anything else interesting in the logs before the OOM?
Best,
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above and may contain inform
Congrats! Thank you on behalf of all of us end users!
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the intended recipient, do
not
Could you please share the IAM policy attached to the role granting permission
to the bucket, as well the bucket policy, if one is present?
Please remove or obfuscate bucket names, account number, etc.
The policy on the role or bucket is most certainly a missing permission, rename
requires a fe
might reveal something
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the intended recipient, do
not disclose or disseminate the messa
Ah, thats a bummer. I was suspect of the s3a as I had the same problem
years ago (and the same solution!) , but figured it had matured/resolved by
now
Thanks for reporting back for future use by the community!
Cheers,
--
Iain Wright
This email message is confidential, intended only for the
63 matches
Mail list logo
