I think standard openstack python project is fine. Before the project creation, anything I can help with now? (
On 22/11/2017, 4:30 PM, "Tristan Cacqueray" <[email protected] on behalf of [email protected]> wrote: On November 21, 2017 5:48 pm, Clark Boylan wrote: > On Tue, Nov 21, 2017, at 09:17 AM, Tristan Cacqueray wrote: >> > > snip > >> Actually the rfc is this thread :-) >> >> Though I forgot to mention the first steps that could use comments before >> we move on: >> * create the openstack-infra/log-classify project, >> * import the log-classify.crm script, >> * wrap the script with a more user friendly interface, and >> * modify the puppet-log_processor to use that new project instead > > This sounds like a great place to start. Considering the interest > already forming around this I would say go ahead and create the project > and start with the import process so that people have a concrete place > to start working on this. I am sure it will evolve from there, but > getting started is often the most difficult step. > > Related to the last step we have temporarily disabled CRM classification > in the log processor pipeline because we treat the whole file path as a > unique file to classify which ended up filling our workers' disks with > classification files. I think one of the things we will want to address > early on is using the basename rather than the whole path to > significantly reduce the total number of data files on disk. This way we > can get it running in the log processor pipeline again for proper > production feedback of changes that are happening. > > Once again let me know if I can help with anything (happy to review new > project creation changes for example). > Excellent, project creation is proposed here: https://review.openstack.org/#/q/topic:log-classify I'm open to suggestion regarding the name and structure of the project. Otherwise I'll create a standard openstack python project with: logclassify.logstash module to interface with the script using the design of the log-gearman-client.py (e.g. a process(event)). logclassify.cmd module to use the script standalone. And then write a first test and implementation of that basename base data files improvement. If that works ok, then a follow-up change will modify the log-gearman-client to import logclassify instead of running the script directly. > Thank you for getting this started, > Clark > Thanks for the quick feedback! -Tristan _______________________________________________ OpenStack-Infra mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra _______________________________________________ OpenStack-Infra mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
