Re: [Meeting Notes] Kibble Dev Call - 7 Jan 2021
Hi folks, I created a 2.0 branch in apache/kibble (just for sanity sake if we would like to reuse some code) and I restored the main branch to some commit before breaking the refactor. Can someone of you rename apache/kibble -> apache/kibble-old and then create apache/kibble repo (as discussed earlier)? I don't have the right to rename repo. Cheers, Tomek On Sun, Jan 10, 2021 at 6:10 PM Sharan Foga wrote: > Hi Kaxil > > I have copied the meeting notes onto the Kibble Wiki for reference > > > https://cwiki.apache.org/confluence/display/KIBBLE/Kibble+Development+Call%3A+7th+January+2021 > > with a link to this thread. > > Thanks > Sharan > > On 2021/01/08 21:52:32, Kaxil Naik wrote: > > Hi all, > > > > Here are the notes from our Kibble dev call from yesterday. Please feel > > free to add anything I have missed. > > > > *Attendees:* > > > >- Kaxil Naik > >- Tomek Urbaszek > >- Michał Słowikowski > >- Sharan Foga > >- Daniel Gruno > > > > > > Here is the summary of the call. > > > >- *Kibble v2*: > > - Rewrite the implementation from Scratch for v2 to improve code > and > > to cater for new ideas > > - Cherrypick / copy any code from the current Repo > > - New repo allows us to work with a TDD approach. The current repo > > does not have tests. > > - *Supporting more DBs* > > - For now, we would just support Elasticsearch. > > - Supporting just one DB allows us to maintain it in a better way > > - Would be good to find some kind of ORM to talk to Elasticsearch > > which will allow static Schema and use Class attributes instead > > of dealing > > with dicts > > - *Migration* > > - Make migration tool or make whatever we build new as > > backwards-compatible so that we don’t need to re-scan all the > > data for ASF > > Kibble -- which can take aleast a week if started from scratch. > > (contains 50million records) > >- *Scanners* > > - Build Base Scanner for v2 that will allow easily creating new > > Scanners and encourage more contributions > > - Write docs around how to build a new scanner inheriting > > *BaseScanner* > > - Some of the new Scanners that users have requested are: > > - Gitlab > > - Social Media Scanners: Twitter, Discord, Slack > > - Create a Github Issue template for Scanners so users can request > > new Scanners > >- *Dashboard* > > - Do a POC to see if Apache Superset can replace our current UI > >- Make Apache Superset or the existing UI optional as some Users just > > rely on the API > > - That will allow us to focus on the CORE (Scanner, Server and ES) > > - > > - What happens to the PR that introduces breaking changes?? > > - We should not merge that PR until it is a breaking change, > > instead we should make the PR backwards compatible > > - Or wait until next major release (assign appropriate Github > > Milestone) > > > > > > Let me know if anyone has any thoughts on any of the items listed above. > > > > Best regards, > > Kaxil > > >
Re: [Meeting Notes] Kibble Dev Call - 7 Jan 2021
Thanks Sharan On 2021/01/10 17:10:12, Sharan Foga wrote: > Hi Kaxil> > > I have copied the meeting notes onto the Kibble Wiki for reference > > > https://cwiki.apache.org/confluence/display/KIBBLE/Kibble+Development+Call%3A+7th+January+2021 > > > with a link to this thread.> > > Thanks> > Sharan> > > On 2021/01/08 21:52:32, Kaxil Naik wrote: > > > Hi all,> > > > > > Here are the notes from our Kibble dev call from yesterday. Please feel> > > free to add anything I have missed.> > > > > > *Attendees:*> > > > > >- Kaxil Naik> > >- Tomek Urbaszek> > >- Michał Słowikowski> > >- Sharan Foga> > >- Daniel Gruno> > > > > > > > > Here is the summary of the call.> > > > > >- *Kibble v2*:> > > - Rewrite the implementation from Scratch for v2 to improve code and> > > to cater for new ideas> > > - Cherrypick / copy any code from the current Repo> > > - New repo allows us to work with a TDD approach. The current repo> > > does not have tests.> > > - *Supporting more DBs*> > > - For now, we would just support Elasticsearch.> > > - Supporting just one DB allows us to maintain it in a better way> > > - Would be good to find some kind of ORM to talk to Elasticsearch> > > which will allow static Schema and use Class attributes instead> > > of dealing> > > with dicts> > > - *Migration*> > > - Make migration tool or make whatever we build new as> > > backwards-compatible so that we don’t need to re-scan all the> > > data for ASF> > > Kibble -- which can take aleast a week if started from scratch.> > > (contains 50million records)> > >- *Scanners*> > > - Build Base Scanner for v2 that will allow easily creating new> > > Scanners and encourage more contributions> > > - Write docs around how to build a new scanner inheriting> > > *BaseScanner*> > > - Some of the new Scanners that users have requested are:> > > - Gitlab> > > - Social Media Scanners: Twitter, Discord, Slack> > > - Create a Github Issue template for Scanners so users can request> > > new Scanners> > >- *Dashboard*> > > - Do a POC to see if Apache Superset can replace our current UI> > >- Make Apache Superset or the existing UI optional as some Users just> > > rely on the API> > > - That will allow us to focus on the CORE (Scanner, Server and ES)> > > -> > > - What happens to the PR that introduces breaking changes??> > > - We should not merge that PR until it is a breaking change,> > > instead we should make the PR backwards compatible> > > - Or wait until next major release (assign appropriate Github> > > Milestone)> > > > > > > > > Let me know if anyone has any thoughts on any of the items listed above.> > > > > > Best regards,> > > Kaxil> > > > >
Re: [Meeting Notes] Kibble Dev Call - 7 Jan 2021
Hi Kaxil I have copied the meeting notes onto the Kibble Wiki for reference https://cwiki.apache.org/confluence/display/KIBBLE/Kibble+Development+Call%3A+7th+January+2021 with a link to this thread. Thanks Sharan On 2021/01/08 21:52:32, Kaxil Naik wrote: > Hi all, > > Here are the notes from our Kibble dev call from yesterday. Please feel > free to add anything I have missed. > > *Attendees:* > >- Kaxil Naik >- Tomek Urbaszek >- Michał Słowikowski >- Sharan Foga >- Daniel Gruno > > > Here is the summary of the call. > >- *Kibble v2*: > - Rewrite the implementation from Scratch for v2 to improve code and > to cater for new ideas > - Cherrypick / copy any code from the current Repo > - New repo allows us to work with a TDD approach. The current repo > does not have tests. > - *Supporting more DBs* > - For now, we would just support Elasticsearch. > - Supporting just one DB allows us to maintain it in a better way > - Would be good to find some kind of ORM to talk to Elasticsearch > which will allow static Schema and use Class attributes instead > of dealing > with dicts > - *Migration* > - Make migration tool or make whatever we build new as > backwards-compatible so that we don’t need to re-scan all the > data for ASF > Kibble -- which can take aleast a week if started from scratch. > (contains 50million records) >- *Scanners* > - Build Base Scanner for v2 that will allow easily creating new > Scanners and encourage more contributions > - Write docs around how to build a new scanner inheriting > *BaseScanner* > - Some of the new Scanners that users have requested are: > - Gitlab > - Social Media Scanners: Twitter, Discord, Slack > - Create a Github Issue template for Scanners so users can request > new Scanners >- *Dashboard* > - Do a POC to see if Apache Superset can replace our current UI >- Make Apache Superset or the existing UI optional as some Users just > rely on the API > - That will allow us to focus on the CORE (Scanner, Server and ES) > - > - What happens to the PR that introduces breaking changes?? > - We should not merge that PR until it is a breaking change, > instead we should make the PR backwards compatible > - Or wait until next major release (assign appropriate Github > Milestone) > > > Let me know if anyone has any thoughts on any of the items listed above. > > Best regards, > Kaxil >
Re: [Meeting Notes] Kibble Dev Call - 7 Jan 2021
Hi Tomek Good question! Whichever way we start at some point (now or in the future) we will probably want the new repo to be eventually called apache/kibble. No real preference just some thoughts.. if we rename the existing repo to 'old' then at least people know we are working on something new and will go look for it. Having kibble and kibble-2 around at the same time might be confusing especially since there will probably be activity on both. Thanks Sharan On 2021/01/10 13:53:51, Tomasz Urbaszek wrote: > Thanks Kaxil for the notes! > > Should we create a new repo called something like apache/kibble-2 or should > we rename existing one to apache/kibble-old and then create a new, fresh > apache/kibble? > > Best, > Tomek > > On Fri, Jan 8, 2021 at 10:55 PM Kaxil Naik wrote: > > > Please ignore the last bullet around "breaking changes" -- I mixed that up > > with the Airflow Dev call summary ;) > > > > Regards, > > Kaxil > > > > On Fri, Jan 8, 2021 at 9:52 PM Kaxil Naik wrote: > > > > > Hi all, > > > > > > Here are the notes from our Kibble dev call from yesterday. Please feel > > > free to add anything I have missed. > > > > > > *Attendees:* > > > > > >- Kaxil Naik > > >- Tomek Urbaszek > > >- Michał Słowikowski > > >- Sharan Foga > > >- Daniel Gruno > > > > > > > > > Here is the summary of the call. > > > > > >- *Kibble v2*: > > > - Rewrite the implementation from Scratch for v2 to improve code > > > and to cater for new ideas > > > - Cherrypick / copy any code from the current Repo > > > - New repo allows us to work with a TDD approach. The current repo > > > does not have tests. > > > - *Supporting more DBs* > > > - For now, we would just support Elasticsearch. > > > - Supporting just one DB allows us to maintain it in a better way > > > - Would be good to find some kind of ORM to talk to Elasticsearch > > > which will allow static Schema and use Class attributes instead of > > dealing > > > with dicts > > > - *Migration* > > > - Make migration tool or make whatever we build new as > > > backwards-compatible so that we don’t need to re-scan all the data > > for ASF > > > Kibble -- which can take aleast a week if started from scratch. > > > (contains 50million records) > > >- *Scanners* > > > - Build Base Scanner for v2 that will allow easily creating new > > > Scanners and encourage more contributions > > > - Write docs around how to build a new scanner inheriting > > > *BaseScanner* > > > - Some of the new Scanners that users have requested are: > > > - Gitlab > > > - Social Media Scanners: Twitter, Discord, Slack > > > - Create a Github Issue template for Scanners so users can request > > > new Scanners > > >- *Dashboard* > > > - Do a POC to see if Apache Superset can replace our current UI > > >- Make Apache Superset or the existing UI optional as some Users just > > > rely on the API > > > - That will allow us to focus on the CORE (Scanner, Server and ES) > > > - > > > - What happens to the PR that introduces breaking changes?? > > > - We should not merge that PR until it is a breaking change, > > > instead we should make the PR backwards compatible > > > - Or wait until next major release (assign appropriate Github > > > Milestone) > > > > > > > > > Let me know if anyone has any thoughts on any of the items listed above. > > > > > > Best regards, > > > Kaxil > > > > > >
Re: [Meeting Notes] Kibble Dev Call - 7 Jan 2021
Good meeting and thanks for the notes Kaxil! On 2021/01/08 21:55:13, Kaxil Naik wrote: > Please ignore the last bullet around "breaking changes" -- I mixed that up > with the Airflow Dev call summary ;) > > Regards, > Kaxil > > On Fri, Jan 8, 2021 at 9:52 PM Kaxil Naik wrote: > > > Hi all, > > > > Here are the notes from our Kibble dev call from yesterday. Please feel > > free to add anything I have missed. > > > > *Attendees:* > > > >- Kaxil Naik > >- Tomek Urbaszek > >- Michał Słowikowski > >- Sharan Foga > >- Daniel Gruno > > > > > > Here is the summary of the call. > > > >- *Kibble v2*: > > - Rewrite the implementation from Scratch for v2 to improve code > > and to cater for new ideas > > - Cherrypick / copy any code from the current Repo > > - New repo allows us to work with a TDD approach. The current repo > > does not have tests. > > - *Supporting more DBs* > > - For now, we would just support Elasticsearch. > > - Supporting just one DB allows us to maintain it in a better way > > - Would be good to find some kind of ORM to talk to Elasticsearch > > which will allow static Schema and use Class attributes instead of > > dealing > > with dicts > > - *Migration* > > - Make migration tool or make whatever we build new as > > backwards-compatible so that we don’t need to re-scan all the data > > for ASF > > Kibble -- which can take aleast a week if started from scratch. > > (contains 50million records) > >- *Scanners* > > - Build Base Scanner for v2 that will allow easily creating new > > Scanners and encourage more contributions > > - Write docs around how to build a new scanner inheriting > > *BaseScanner* > > - Some of the new Scanners that users have requested are: > > - Gitlab > > - Social Media Scanners: Twitter, Discord, Slack > > - Create a Github Issue template for Scanners so users can request > > new Scanners > >- *Dashboard* > > - Do a POC to see if Apache Superset can replace our current UI > >- Make Apache Superset or the existing UI optional as some Users just > > rely on the API > > - That will allow us to focus on the CORE (Scanner, Server and ES) > > - > > - What happens to the PR that introduces breaking changes?? > > - We should not merge that PR until it is a breaking change, > > instead we should make the PR backwards compatible > > - Or wait until next major release (assign appropriate Github > > Milestone) > > > > > > Let me know if anyone has any thoughts on any of the items listed above. > > > > Best regards, > > Kaxil > > >
Re: [Meeting Notes] Kibble Dev Call - 7 Jan 2021
Thanks Kaxil for the notes! Should we create a new repo called something like apache/kibble-2 or should we rename existing one to apache/kibble-old and then create a new, fresh apache/kibble? Best, Tomek On Fri, Jan 8, 2021 at 10:55 PM Kaxil Naik wrote: > Please ignore the last bullet around "breaking changes" -- I mixed that up > with the Airflow Dev call summary ;) > > Regards, > Kaxil > > On Fri, Jan 8, 2021 at 9:52 PM Kaxil Naik wrote: > > > Hi all, > > > > Here are the notes from our Kibble dev call from yesterday. Please feel > > free to add anything I have missed. > > > > *Attendees:* > > > >- Kaxil Naik > >- Tomek Urbaszek > >- Michał Słowikowski > >- Sharan Foga > >- Daniel Gruno > > > > > > Here is the summary of the call. > > > >- *Kibble v2*: > > - Rewrite the implementation from Scratch for v2 to improve code > > and to cater for new ideas > > - Cherrypick / copy any code from the current Repo > > - New repo allows us to work with a TDD approach. The current repo > > does not have tests. > > - *Supporting more DBs* > > - For now, we would just support Elasticsearch. > > - Supporting just one DB allows us to maintain it in a better way > > - Would be good to find some kind of ORM to talk to Elasticsearch > > which will allow static Schema and use Class attributes instead of > dealing > > with dicts > > - *Migration* > > - Make migration tool or make whatever we build new as > > backwards-compatible so that we don’t need to re-scan all the data > for ASF > > Kibble -- which can take aleast a week if started from scratch. > > (contains 50million records) > >- *Scanners* > > - Build Base Scanner for v2 that will allow easily creating new > > Scanners and encourage more contributions > > - Write docs around how to build a new scanner inheriting > > *BaseScanner* > > - Some of the new Scanners that users have requested are: > > - Gitlab > > - Social Media Scanners: Twitter, Discord, Slack > > - Create a Github Issue template for Scanners so users can request > > new Scanners > >- *Dashboard* > > - Do a POC to see if Apache Superset can replace our current UI > >- Make Apache Superset or the existing UI optional as some Users just > > rely on the API > > - That will allow us to focus on the CORE (Scanner, Server and ES) > > - > > - What happens to the PR that introduces breaking changes?? > > - We should not merge that PR until it is a breaking change, > > instead we should make the PR backwards compatible > > - Or wait until next major release (assign appropriate Github > > Milestone) > > > > > > Let me know if anyone has any thoughts on any of the items listed above. > > > > Best regards, > > Kaxil > > >
Re: [Meeting Notes] Kibble Dev Call - 7 Jan 2021
Please ignore the last bullet around "breaking changes" -- I mixed that up with the Airflow Dev call summary ;) Regards, Kaxil On Fri, Jan 8, 2021 at 9:52 PM Kaxil Naik wrote: > Hi all, > > Here are the notes from our Kibble dev call from yesterday. Please feel > free to add anything I have missed. > > *Attendees:* > >- Kaxil Naik >- Tomek Urbaszek >- Michał Słowikowski >- Sharan Foga >- Daniel Gruno > > > Here is the summary of the call. > >- *Kibble v2*: > - Rewrite the implementation from Scratch for v2 to improve code > and to cater for new ideas > - Cherrypick / copy any code from the current Repo > - New repo allows us to work with a TDD approach. The current repo > does not have tests. > - *Supporting more DBs* > - For now, we would just support Elasticsearch. > - Supporting just one DB allows us to maintain it in a better way > - Would be good to find some kind of ORM to talk to Elasticsearch > which will allow static Schema and use Class attributes instead of > dealing > with dicts > - *Migration* > - Make migration tool or make whatever we build new as > backwards-compatible so that we don’t need to re-scan all the data for > ASF > Kibble -- which can take aleast a week if started from scratch. > (contains 50million records) >- *Scanners* > - Build Base Scanner for v2 that will allow easily creating new > Scanners and encourage more contributions > - Write docs around how to build a new scanner inheriting > *BaseScanner* > - Some of the new Scanners that users have requested are: > - Gitlab > - Social Media Scanners: Twitter, Discord, Slack > - Create a Github Issue template for Scanners so users can request > new Scanners >- *Dashboard* > - Do a POC to see if Apache Superset can replace our current UI >- Make Apache Superset or the existing UI optional as some Users just > rely on the API > - That will allow us to focus on the CORE (Scanner, Server and ES) > - > - What happens to the PR that introduces breaking changes?? > - We should not merge that PR until it is a breaking change, > instead we should make the PR backwards compatible > - Or wait until next major release (assign appropriate Github > Milestone) > > > Let me know if anyone has any thoughts on any of the items listed above. > > Best regards, > Kaxil >
[Meeting Notes] Kibble Dev Call - 7 Jan 2021
Hi all, Here are the notes from our Kibble dev call from yesterday. Please feel free to add anything I have missed. *Attendees:* - Kaxil Naik - Tomek Urbaszek - Michał Słowikowski - Sharan Foga - Daniel Gruno Here is the summary of the call. - *Kibble v2*: - Rewrite the implementation from Scratch for v2 to improve code and to cater for new ideas - Cherrypick / copy any code from the current Repo - New repo allows us to work with a TDD approach. The current repo does not have tests. - *Supporting more DBs* - For now, we would just support Elasticsearch. - Supporting just one DB allows us to maintain it in a better way - Would be good to find some kind of ORM to talk to Elasticsearch which will allow static Schema and use Class attributes instead of dealing with dicts - *Migration* - Make migration tool or make whatever we build new as backwards-compatible so that we don’t need to re-scan all the data for ASF Kibble -- which can take aleast a week if started from scratch. (contains 50million records) - *Scanners* - Build Base Scanner for v2 that will allow easily creating new Scanners and encourage more contributions - Write docs around how to build a new scanner inheriting *BaseScanner* - Some of the new Scanners that users have requested are: - Gitlab - Social Media Scanners: Twitter, Discord, Slack - Create a Github Issue template for Scanners so users can request new Scanners - *Dashboard* - Do a POC to see if Apache Superset can replace our current UI - Make Apache Superset or the existing UI optional as some Users just rely on the API - That will allow us to focus on the CORE (Scanner, Server and ES) - - What happens to the PR that introduces breaking changes?? - We should not merge that PR until it is a breaking change, instead we should make the PR backwards compatible - Or wait until next major release (assign appropriate Github Milestone) Let me know if anyone has any thoughts on any of the items listed above. Best regards, Kaxil
