Re: When things change in hdfs, how do we know

[JJ Meyer]

Hello all,

I had created a NiFi processor a long time back that used the inotify API.
One thing I noticed while working with it is that it is marked with the
`Unstable` annotation. It may be worth checking if anymore work is going on
with it and if it will impact this (if it hasn't already been looked into).

Thanks,
JJ

On Mon, Jan 29, 2018 at 7:27 AM, Otto Fowler 
wrote:

> I have updated the jira as well
>
>
> On January 29, 2018 at 08:22:34, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> https://github.com/ottobackwards/hdfs-inotify-zookeeper
>

Re: Reducing Warnings in Build

[JJ Meyer]

Nick,

You're not off base at all. This is exactly the push back I wanted to hear.
I wasn't really sure if what I was proposing was the proper way of going
about it. My understanding of charsets is limited, and I've usually just
defaulted to UTF-8. That being said, right or wrong, my thought process for
including multiple configs was potential use of multiple charsets that may
be compatible (if that's possible?). But as I'm typing this out, I'm
probably over doing it. Especially since currently we usually take system
default, which a lot of times is UTF-8. So, you are right, it is probably
best to start with the path of least resistance and include one
configuration that is set to UTF-8. Then expand if we ever see a need to do
so.

Thanks,
JJ

On Fri, Apr 21, 2017 at 10:45 AM, Ryan Merriman <merrim...@gmail.com> wrote:

> I think Nick brings up some good points.  Would there ever be a reason to
> not use UTF8 as the default from parsing a message on?  All the tools we
> use for analytics work with UTF8 (am I wrong?).
>
> The only case I can see needing a configurable charset would be if a
> message coming from a sensor were encoded in a charset other than UTF8.  In
> that case there would need to be a configurable charset per parser (in
> parser config?) for decoding but the message could be encoded/decoded with
> UTF8 after that.  Or we could just make UTF encoding a prerequisite for
> sending messages to Metron.
>
> On Fri, Apr 21, 2017 at 10:32 AM, Nick Allen <n...@nickallen.org> wrote:
>
> > Per (2), I think it makes sense to make the charset configurable, but
> with
> > the proposal of 3 separate settings, wouldn't things blow up horribly if
> > the Parsers are producing UTF-8, but Enrichment is expecting UTF-16?
> They
> > are not even speaking the same language, no?
> >
> > This makes me think that we need to understand the scenarios under which
> a
> > user would want to change the charset, before we know what kinds of
> levers
> > to bake-in.  What sort of use case would drive someone to change the
> > charset?  Would there be a particular sensor producing telemetry with a
> > different charset from others?  If one source of telemetry is different
> > than the others, would the entire system even work with varying charsets?
> >
> > Without a good understanding of use cases, I think the only mildly safe
> > thing to do right now, is to have a single, global charset setting.  The
> > user would have to ensure that their entire environment and all the JVMs
> > driving it are set to use that charset.
> >
> > Perhaps my questioning comes from a lack of understanding of charsets.  I
> > can't remember ever having to deviate from the default.  Please chime in
> > and educate me, if I am off base.
> >
> >
> >
> >
> >
> >
> > On Fri, Apr 21, 2017 at 8:50 AM, JJ Meyer <jjmey...@gmail.com> wrote:
> >
> > > Hello everybody,
> > >
> > > Currently our build has a significant amount of warnings (most from the
> > > error prone plugin). A lot of this has been documented here:
> > > https://issues.apache.org/jira/browse/METRON-617
> > >
> > > I want to continue to work on this Jira. I really want to reduce the
> > number
> > > of warnings in our build. As the Jira points out, a lot of them are
> > > unchecked casts or the implicit use of default charset.
> > >
> > > When starting this, I want to start with a specific module. I plan on
> > > starting with `metron-interface` as it's fairly self contained and is
> > > pretty new. Below I want to layout what I plan on doing. Please let me
> > know
> > > what you all think:
> > >
> > > 1. Suppress warnings where generics are not supported or checking type
> is
> > > not possible.
> > > 2. For all unit tests that require supplying a charset I'll supply the
> > > UTF-8 charset.
> > > 3. Update the API to have a charset configuration for each resource
> that
> > > needs one.
> > > 4. Remove @SuppressWarnings("ALL") on all unit tests. I found out error
> > > prone doesn't support this level of suppression. Which is probably a
> good
> > > thing. We will need to explicitly state what we want to suppress
> instead.
> > >
> > > Two big questions came up right away when I was thinking about the
> above:
> > >
> > > 1. When suppressing warnings. I see we sometimes cast a JSONObject to
> > > Map<String, Object>. I know it extends Map, but is it really safe to do
> > > something like the following? Should we have a utility that tr