Right. The original idea is to do batch analytics. Kind of difficult to work with data sitting in an ES index. But if we get a better understanding of the type of batch analytics, it might get us closer to the target.
On Tue, Oct 11, 2016 at 1:03 PM, [email protected] <[email protected]> wrote: > I'm somewhat ignorant here, never having used the MaaS stuff yet, but isn't > that the dataset that the models would run against? I understand there > could be additional use cases, I just wanted to be clear. > > Jon > > On Tue, Oct 11, 2016 at 1:01 PM Nick Allen <[email protected]> wrote: > > > I don't think we put much thought into how exactly the data should be > > landed in HDFS and for what use cases. It just has not been a priority. > > > > That being said, this might be a good time to gather everyone's thoughts > on > > how they would use that kind of data and for what purposes. > > > > > > > > On Tue, Oct 11, 2016 at 12:11 PM, Owen O'Malley <[email protected]> > > wrote: > > > > > Be careful of using compressed JSON, since it isn't splittable. JSON is > > > also very slow for reading. > > > > > > .. Owen > > > > > > On Tue, Oct 11, 2016 at 4:31 AM, Casey Stella <[email protected]> > > wrote: > > > > > > > I'd also tack on to this that the configuration for the hdfs writer > > > should > > > > be moved to zookeeper rather than done in flux, IMO > > > > On Tue, Oct 11, 2016 at 07:20 Otto Fowler <[email protected]> > > > wrote: > > > > > > > > > The storage format and retrieval from that format should be > > > configurable, > > > > > that is a ‘boundary’ for Metron so to speak. > > > > > > > > > > On October 10, 2016 at 16:15:12, [email protected] ( > [email protected]) > > > > > wrote: > > > > > > > > > > Is there a specific reason why the JSON files stored in HDFS are > not > > > > > compressed? I looked for some related JIRAs and mail conversations > > but > > > > > couldn't find this already mentioned. I'm wondering if there was a > > good > > > > > enough of an argument to keep things uncompressed, or if the > subject > > > just > > > > > hadn't been broached yet. > > > > > > > > > > Jon > > > > > -- > > > > > > > > > > Jon > > > > > > > > > > > > > > > > > > > > -- > > Nick Allen <[email protected]> > > > -- > > Jon > -- Nick Allen <[email protected]>
