Re: [DISCUSS] - Remove Kibana
I take your point. If another mpack would be that resource intensive, it doesn't make much sense to go down that road. There are plenty of other features the community would get greater value from for the time investment. -Kyle On Wed, Nov 1, 2017 at 10:14 PM, Michael Miklavcic < michael.miklav...@gmail.com> wrote: > I think the suggestion is creative, however I am strongly opposed to 2 > mpacks. It has been a fair amount of work through upgrades to maintain the > one we have and I think splitting them might make it even worse. I'd much > prefer to keep the dep on Kibana as is rather than make a change that > doesn't simplify things in the end. > > On Nov 1, 2017 6:25 PM, "Kyle Richardson" > wrote: > > > I'll second Otto's suggestion. I like the idea of "splitting" the ES and > > Kibana components from the pure Metron components. I suppose that would > > mean having two mpacks to build for a while though. > > > > I agree with others that, at least for now, Kibana is an integral part of > > the Metron user experience. > > > > -Kyle > > > > On Wed, Nov 1, 2017 at 7:37 PM Michael Miklavcic < > > michael.miklav...@gmail.com> wrote: > > > > > It would not be installed with/by Metron. You'd install and manage > Kibana > > > on your own. Some things can be done with the head plugin, but it > > wouldn't > > > be as pretty. > > > > > > From the sounds of it the community still uses and wants Kibana, so > we'll > > > hold off until the UI can manage more of this functionality. Thanks > > > everyone for the input. > > > > > > On Nov 1, 2017 4:18 PM, "Laurens Vets" wrote: > > > > > > > How would I do that without Kibana? Having a SIEM without the ability > > to > > > > see raw processed events (whether they are alerts or not), would be a > > big > > > > issue I think. > > > > > > > > Or would Kibana always be required, just not installed by Metron? > > > > > > > > On 2017-11-01 11:34, Michael Miklavcic wrote: > > > > > > > >> You could absolutely still do it, I'm simply saying it would not be > > > >> managed > > > >> by us. > > > >> > > > >> On Nov 1, 2017 12:20 PM, "Laurens Vets" wrote: > > > >> > > > >> If there's a viable way of looking at raw processed events (not > > > >>> necessarily alerts), then I'm all for removeing Kibana. I use > > Discover > > > a > > > >>> lot to filter and look at events and create new policies from that. > > > >>> > > > >>> Is there currently a simple way to do this without Kibana? > > > >>> > > > >>> On 2017-11-01 09:13, Michael Miklavcic wrote: > > > >>> > > > >>> As part of the ES upgrade, I got to thinking that it makes sense to > > > remove > > > Kibana and the dashboards we're currently bundling in the MPack. > To > > be > > > clear, this would not remove the ability to independently install > > and > > > use > > > Kibana if the user so chooses, it would only remove the > dashboards, > > > and > > > potentially, the Ambari/MPack management support that we ship. > > > > > > *pros* > > > Removes need to support tooling outside our wheelhouse > > > Smaller testing effort for ongoing support and future upgrades > > > Simplifies our base Metron install via Ambari. Also simplifies > full > > > dev > > > setup. > > > > > > *cons* > > > User would need to install and setup their own Kibana instance and > > > dashboards. > > > If any existing users are using this, they'd need to backup and > > manage > > > their own Kibana dashboards going forward. They would also need to > > > handle > > > any upgrade issues with Kibana post-Metron ES 5.x upgrade. > > > > > > Any concerns? > > > > > > Mike > > > > > > >>> > > > > > >
Re: [DISCUSS] - Remove Kibana
I think the suggestion is creative, however I am strongly opposed to 2 mpacks. It has been a fair amount of work through upgrades to maintain the one we have and I think splitting them might make it even worse. I'd much prefer to keep the dep on Kibana as is rather than make a change that doesn't simplify things in the end. On Nov 1, 2017 6:25 PM, "Kyle Richardson" wrote: > I'll second Otto's suggestion. I like the idea of "splitting" the ES and > Kibana components from the pure Metron components. I suppose that would > mean having two mpacks to build for a while though. > > I agree with others that, at least for now, Kibana is an integral part of > the Metron user experience. > > -Kyle > > On Wed, Nov 1, 2017 at 7:37 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > It would not be installed with/by Metron. You'd install and manage Kibana > > on your own. Some things can be done with the head plugin, but it > wouldn't > > be as pretty. > > > > From the sounds of it the community still uses and wants Kibana, so we'll > > hold off until the UI can manage more of this functionality. Thanks > > everyone for the input. > > > > On Nov 1, 2017 4:18 PM, "Laurens Vets" wrote: > > > > > How would I do that without Kibana? Having a SIEM without the ability > to > > > see raw processed events (whether they are alerts or not), would be a > big > > > issue I think. > > > > > > Or would Kibana always be required, just not installed by Metron? > > > > > > On 2017-11-01 11:34, Michael Miklavcic wrote: > > > > > >> You could absolutely still do it, I'm simply saying it would not be > > >> managed > > >> by us. > > >> > > >> On Nov 1, 2017 12:20 PM, "Laurens Vets" wrote: > > >> > > >> If there's a viable way of looking at raw processed events (not > > >>> necessarily alerts), then I'm all for removeing Kibana. I use > Discover > > a > > >>> lot to filter and look at events and create new policies from that. > > >>> > > >>> Is there currently a simple way to do this without Kibana? > > >>> > > >>> On 2017-11-01 09:13, Michael Miklavcic wrote: > > >>> > > >>> As part of the ES upgrade, I got to thinking that it makes sense to > > remove > > Kibana and the dashboards we're currently bundling in the MPack. To > be > > clear, this would not remove the ability to independently install > and > > use > > Kibana if the user so chooses, it would only remove the dashboards, > > and > > potentially, the Ambari/MPack management support that we ship. > > > > *pros* > > Removes need to support tooling outside our wheelhouse > > Smaller testing effort for ongoing support and future upgrades > > Simplifies our base Metron install via Ambari. Also simplifies full > > dev > > setup. > > > > *cons* > > User would need to install and setup their own Kibana instance and > > dashboards. > > If any existing users are using this, they'd need to backup and > manage > > their own Kibana dashboards going forward. They would also need to > > handle > > any upgrade issues with Kibana post-Metron ES 5.x upgrade. > > > > Any concerns? > > > > Mike > > > > >>> > > >
Re: [DISCUSS] - Remove Kibana
I'll second Otto's suggestion. I like the idea of "splitting" the ES and Kibana components from the pure Metron components. I suppose that would mean having two mpacks to build for a while though. I agree with others that, at least for now, Kibana is an integral part of the Metron user experience. -Kyle On Wed, Nov 1, 2017 at 7:37 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > It would not be installed with/by Metron. You'd install and manage Kibana > on your own. Some things can be done with the head plugin, but it wouldn't > be as pretty. > > From the sounds of it the community still uses and wants Kibana, so we'll > hold off until the UI can manage more of this functionality. Thanks > everyone for the input. > > On Nov 1, 2017 4:18 PM, "Laurens Vets" wrote: > > > How would I do that without Kibana? Having a SIEM without the ability to > > see raw processed events (whether they are alerts or not), would be a big > > issue I think. > > > > Or would Kibana always be required, just not installed by Metron? > > > > On 2017-11-01 11:34, Michael Miklavcic wrote: > > > >> You could absolutely still do it, I'm simply saying it would not be > >> managed > >> by us. > >> > >> On Nov 1, 2017 12:20 PM, "Laurens Vets" wrote: > >> > >> If there's a viable way of looking at raw processed events (not > >>> necessarily alerts), then I'm all for removeing Kibana. I use Discover > a > >>> lot to filter and look at events and create new policies from that. > >>> > >>> Is there currently a simple way to do this without Kibana? > >>> > >>> On 2017-11-01 09:13, Michael Miklavcic wrote: > >>> > >>> As part of the ES upgrade, I got to thinking that it makes sense to > remove > Kibana and the dashboards we're currently bundling in the MPack. To be > clear, this would not remove the ability to independently install and > use > Kibana if the user so chooses, it would only remove the dashboards, > and > potentially, the Ambari/MPack management support that we ship. > > *pros* > Removes need to support tooling outside our wheelhouse > Smaller testing effort for ongoing support and future upgrades > Simplifies our base Metron install via Ambari. Also simplifies full > dev > setup. > > *cons* > User would need to install and setup their own Kibana instance and > dashboards. > If any existing users are using this, they'd need to backup and manage > their own Kibana dashboards going forward. They would also need to > handle > any upgrade issues with Kibana post-Metron ES 5.x upgrade. > > Any concerns? > > Mike > > >>> >
Re: [DISCUSS] - Remove Kibana
It would not be installed with/by Metron. You'd install and manage Kibana on your own. Some things can be done with the head plugin, but it wouldn't be as pretty. >From the sounds of it the community still uses and wants Kibana, so we'll hold off until the UI can manage more of this functionality. Thanks everyone for the input. On Nov 1, 2017 4:18 PM, "Laurens Vets" wrote: > How would I do that without Kibana? Having a SIEM without the ability to > see raw processed events (whether they are alerts or not), would be a big > issue I think. > > Or would Kibana always be required, just not installed by Metron? > > On 2017-11-01 11:34, Michael Miklavcic wrote: > >> You could absolutely still do it, I'm simply saying it would not be >> managed >> by us. >> >> On Nov 1, 2017 12:20 PM, "Laurens Vets" wrote: >> >> If there's a viable way of looking at raw processed events (not >>> necessarily alerts), then I'm all for removeing Kibana. I use Discover a >>> lot to filter and look at events and create new policies from that. >>> >>> Is there currently a simple way to do this without Kibana? >>> >>> On 2017-11-01 09:13, Michael Miklavcic wrote: >>> >>> As part of the ES upgrade, I got to thinking that it makes sense to remove Kibana and the dashboards we're currently bundling in the MPack. To be clear, this would not remove the ability to independently install and use Kibana if the user so chooses, it would only remove the dashboards, and potentially, the Ambari/MPack management support that we ship. *pros* Removes need to support tooling outside our wheelhouse Smaller testing effort for ongoing support and future upgrades Simplifies our base Metron install via Ambari. Also simplifies full dev setup. *cons* User would need to install and setup their own Kibana instance and dashboards. If any existing users are using this, they'd need to backup and manage their own Kibana dashboards going forward. They would also need to handle any upgrade issues with Kibana post-Metron ES 5.x upgrade. Any concerns? Mike >>>
Re: [DISCUSS] - Remove Kibana
How would I do that without Kibana? Having a SIEM without the ability to see raw processed events (whether they are alerts or not), would be a big issue I think. Or would Kibana always be required, just not installed by Metron? On 2017-11-01 11:34, Michael Miklavcic wrote: You could absolutely still do it, I'm simply saying it would not be managed by us. On Nov 1, 2017 12:20 PM, "Laurens Vets" wrote: If there's a viable way of looking at raw processed events (not necessarily alerts), then I'm all for removeing Kibana. I use Discover a lot to filter and look at events and create new policies from that. Is there currently a simple way to do this without Kibana? On 2017-11-01 09:13, Michael Miklavcic wrote: As part of the ES upgrade, I got to thinking that it makes sense to remove Kibana and the dashboards we're currently bundling in the MPack. To be clear, this would not remove the ability to independently install and use Kibana if the user so chooses, it would only remove the dashboards, and potentially, the Ambari/MPack management support that we ship. *pros* Removes need to support tooling outside our wheelhouse Smaller testing effort for ongoing support and future upgrades Simplifies our base Metron install via Ambari. Also simplifies full dev setup. *cons* User would need to install and setup their own Kibana instance and dashboards. If any existing users are using this, they'd need to backup and manage their own Kibana dashboards going forward. They would also need to handle any upgrade issues with Kibana post-Metron ES 5.x upgrade. Any concerns? Mike
Re: [DISCUSS] - Remove Kibana
You could absolutely still do it, I'm simply saying it would not be managed by us. On Nov 1, 2017 12:20 PM, "Laurens Vets" wrote: > If there's a viable way of looking at raw processed events (not > necessarily alerts), then I'm all for removeing Kibana. I use Discover a > lot to filter and look at events and create new policies from that. > > Is there currently a simple way to do this without Kibana? > > On 2017-11-01 09:13, Michael Miklavcic wrote: > >> As part of the ES upgrade, I got to thinking that it makes sense to remove >> Kibana and the dashboards we're currently bundling in the MPack. To be >> clear, this would not remove the ability to independently install and use >> Kibana if the user so chooses, it would only remove the dashboards, and >> potentially, the Ambari/MPack management support that we ship. >> >> *pros* >> Removes need to support tooling outside our wheelhouse >> Smaller testing effort for ongoing support and future upgrades >> Simplifies our base Metron install via Ambari. Also simplifies full dev >> setup. >> >> *cons* >> User would need to install and setup their own Kibana instance and >> dashboards. >> If any existing users are using this, they'd need to backup and manage >> their own Kibana dashboards going forward. They would also need to handle >> any upgrade issues with Kibana post-Metron ES 5.x upgrade. >> >> Any concerns? >> >> Mike >> >
Re: [DISCUSS] - Remove Kibana
If there's a viable way of looking at raw processed events (not necessarily alerts), then I'm all for removeing Kibana. I use Discover a lot to filter and look at events and create new policies from that. Is there currently a simple way to do this without Kibana? On 2017-11-01 09:13, Michael Miklavcic wrote: As part of the ES upgrade, I got to thinking that it makes sense to remove Kibana and the dashboards we're currently bundling in the MPack. To be clear, this would not remove the ability to independently install and use Kibana if the user so chooses, it would only remove the dashboards, and potentially, the Ambari/MPack management support that we ship. *pros* Removes need to support tooling outside our wheelhouse Smaller testing effort for ongoing support and future upgrades Simplifies our base Metron install via Ambari. Also simplifies full dev setup. *cons* User would need to install and setup their own Kibana instance and dashboards. If any existing users are using this, they'd need to backup and manage their own Kibana dashboards going forward. They would also need to handle any upgrade issues with Kibana post-Metron ES 5.x upgrade. Any concerns? Mike
Re: [DISCUSS] - Remove Kibana
Maybe what we should do is move the ES + Kibana mpack stuff to contrib, and change the main mpack to support either the contrib install -or- an existing or non-ambari managed install? Down the road. On November 1, 2017 at 13:44:37, zeo...@gmail.com (zeo...@gmail.com) wrote: I'm probably okay with marking it as deprecated in two releases (after moving to 5.x, thus not really helping with the migration), but it depends a lot on increased functionality for the metron alerts UI IMO. Jon On Wed, Nov 1, 2017 at 12:51 PM Otto Fowler wrote: > I don’t think we should remove it until there is a viable alternative for > the capabilities we rely on it for. > Also, the ‘story’ around metron integration with kibana needs to be solid > and well supported at that time. > > > > On November 1, 2017 at 12:13:18, Michael Miklavcic ( > michael.miklav...@gmail.com) wrote: > > As part of the ES upgrade, I got to thinking that it makes sense to remove > Kibana and the dashboards we're currently bundling in the MPack. To be > clear, this would not remove the ability to independently install and use > Kibana if the user so chooses, it would only remove the dashboards, and > potentially, the Ambari/MPack management support that we ship. > > *pros* > Removes need to support tooling outside our wheelhouse > Smaller testing effort for ongoing support and future upgrades > Simplifies our base Metron install via Ambari. Also simplifies full dev > setup. > > *cons* > User would need to install and setup their own Kibana instance and > dashboards. > If any existing users are using this, they'd need to backup and manage > their own Kibana dashboards going forward. They would also need to handle > any upgrade issues with Kibana post-Metron ES 5.x upgrade. > > Any concerns? > > Mike > -- Jon
Re: [DISCUSS] - Remove Kibana
I'm probably okay with marking it as deprecated in two releases (after moving to 5.x, thus not really helping with the migration), but it depends a lot on increased functionality for the metron alerts UI IMO. Jon On Wed, Nov 1, 2017 at 12:51 PM Otto Fowler wrote: > I don’t think we should remove it until there is a viable alternative for > the capabilities we rely on it for. > Also, the ‘story’ around metron integration with kibana needs to be solid > and well supported at that time. > > > > On November 1, 2017 at 12:13:18, Michael Miklavcic ( > michael.miklav...@gmail.com) wrote: > > As part of the ES upgrade, I got to thinking that it makes sense to remove > Kibana and the dashboards we're currently bundling in the MPack. To be > clear, this would not remove the ability to independently install and use > Kibana if the user so chooses, it would only remove the dashboards, and > potentially, the Ambari/MPack management support that we ship. > > *pros* > Removes need to support tooling outside our wheelhouse > Smaller testing effort for ongoing support and future upgrades > Simplifies our base Metron install via Ambari. Also simplifies full dev > setup. > > *cons* > User would need to install and setup their own Kibana instance and > dashboards. > If any existing users are using this, they'd need to backup and manage > their own Kibana dashboards going forward. They would also need to handle > any upgrade issues with Kibana post-Metron ES 5.x upgrade. > > Any concerns? > > Mike > -- Jon
Re: [DISCUSS] - Remove Kibana
I don’t think we should remove it until there is a viable alternative for the capabilities we rely on it for. Also, the ‘story’ around metron integration with kibana needs to be solid and well supported at that time. On November 1, 2017 at 12:13:18, Michael Miklavcic ( michael.miklav...@gmail.com) wrote: As part of the ES upgrade, I got to thinking that it makes sense to remove Kibana and the dashboards we're currently bundling in the MPack. To be clear, this would not remove the ability to independently install and use Kibana if the user so chooses, it would only remove the dashboards, and potentially, the Ambari/MPack management support that we ship. *pros* Removes need to support tooling outside our wheelhouse Smaller testing effort for ongoing support and future upgrades Simplifies our base Metron install via Ambari. Also simplifies full dev setup. *cons* User would need to install and setup their own Kibana instance and dashboards. If any existing users are using this, they'd need to backup and manage their own Kibana dashboards going forward. They would also need to handle any upgrade issues with Kibana post-Metron ES 5.x upgrade. Any concerns? Mike
Re: [DISCUSS] - Remove Kibana
I completely see the advantages that you point out, Mike. But I would be against removing support at this time. With a project like Metron, it is hard for new users to get their head around what it is and what it can do. The Kibana dashboard gives new users an understanding of how they can leverage the normalized, enriched, and triaged data that Metron produces. It is the only interface we currently have that shows-off plots, maps and the like. On Wed, Nov 1, 2017 at 12:13 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > As part of the ES upgrade, I got to thinking that it makes sense to remove > Kibana and the dashboards we're currently bundling in the MPack. To be > clear, this would not remove the ability to independently install and use > Kibana if the user so chooses, it would only remove the dashboards, and > potentially, the Ambari/MPack management support that we ship. > > *pros* > Removes need to support tooling outside our wheelhouse > Smaller testing effort for ongoing support and future upgrades > Simplifies our base Metron install via Ambari. Also simplifies full dev > setup. > > *cons* > User would need to install and setup their own Kibana instance and > dashboards. > If any existing users are using this, they'd need to backup and manage > their own Kibana dashboards going forward. They would also need to handle > any upgrade issues with Kibana post-Metron ES 5.x upgrade. > > Any concerns? > > Mike >
