Hmm I guess as long as there is an actual human approval before every bot/ai answer (ie a triage team member or maintainer has actually verified that the answer fits the question) I am onboard with the idea. It should hopefully reduce some of the load on them.
I guess it's too early to discuss what the workflow we're picturing for it is. But I would like it if a reporter is unable to see the comment until it's approved (which I guess is exactly what you're also suggesting Jarek). -- Regards, Aritra Basu On Thu, Jun 27, 2024, 12:01 PM Jarek Potiuk <[email protected]> wrote: > Julian - anyone has a voice here :) .. We are not yet voting on it - I > already see that lazy consensus will be enough - but even with voting, > non-committers are encouraged to vote anyway (their votes are non-binding, > but included in the results). > > Aritra/Amogh - I very much share your perspective there. Pure "bot" answers > without human involvement are a bad way to build a community and it's > "no-go" for me. > > The current idea we have, how we could still benefit from having AI/Bot > generated answers, is for maintainers and possibly triage teams to be able > to review the proposed answers and "approve" them before they are posted. > Such an answer might then be marked as "bot/automatically/AI generated" but > it will also get a "Maintainer approved" or "Triage team approved" badge, > This way anyone can see that the answer was reviewed by a maintainer/triage > team member. This way we can get way more efficiency in handling issues > (time for reviewing answers will be far less than writing it). Also > maintainers/triage team members will be able to correct the answers if they > are misleading / wrong. Who approved/corrected it will be visible for the > triage/maintainer team in the DoSu interface, but not seen in the Github UI > who actually approved/corrected it. > > This has very interesting side effects: > > * AI-assisted knowledge transfer between maintainers and triage team - > since the generated response will use past knowledge base, the maintainers > / triage team members (if they treat it seriously and will actually spend a > bit time on reading and understanding answers) - will actually learn from > other's past answers this way. Especially if - as it is in Astro - we will > include references to sources. > * When the answer is posted by bot, and there is a follow-up question, the > nice thing is that it's not tied to a particular maintainer - usually when > now someone answers a question, almost by definition all follow-up > questions are "bound" to that person who first answered it - with generic > "a maintainer approved" badge - this binding is not there. > * When maintainers correct an AI generated answer, this is a **perfect** > learning data to improve the AI system - because you see the "bad" and > "good" answer that is of a high quality (you know it was bad and most > likely it's good after corrections) - which is a gold for AI learning > process > * If it works, it will actually raise contributor's trust to AI generated > answers over time (and they will likely get better as well). > > We are **not** implementing it yet though - those are potential features we > discussed with the Dosu team to implement (and happy to hear feedback on it > as well), but one of the common reservations we have is that **just** > posting bot answers is not going to happen - we need to do way better than > that. > > J. > > > On Thu, Jun 27, 2024 at 6:43 AM Amogh Desai <[email protected]> > wrote: > > > Agreed with Aritra here. > > > > I personally would love to have the "auto" labelling portion, which would > > ensure that > > issues get immediate "right" labels, without relying too much on the > > triaging team but > > I personally am not too comfortable with a bot answering my questions for > > two reasons: > > > > 1. Maintainer/Contributor responses are tailored personally to a question > > and the level of the issue reporter > > 2. Bot responses sometimes are disregarded by readers, just because its a > > "bot" > > > > Thanks & Regards, > > Amogh Desai > > > > > > On Thu, Jun 27, 2024 at 8:53 AM Aritra Basu <[email protected]> > > wrote: > > > > > I'm a bit averse to having it answer questions (the similar issues > part) > > as > > > I believe that's a slippery slope to where it could start being used to > > > provide actual answers. I personally would be more comfortable with > > > answering still being restricted to maintainers/contributors and maybe > > > restricting the bot to just tagging similar issues while still > requiring > > a > > > human to provide advice based on that similarity. > > > > > > Just my personal 2 cents. I'm not too strongly opposed to using it. > > > -- > > > Regards, > > > Aritra Basu > > > > > > On Thu, Jun 27, 2024, 2:41 AM Julian LaNeve > <[email protected] > > > > > > wrote: > > > > > > > Not sure if I get an official vote here but we've been working with > > Devin > > > > and the Dosu team for Cosmos ( > > > > https://github.com/astronomer/astronomer-cosmos ) and it's been > > working > > > > great. Excited to see this in Airflow itself! > > > > > > > > Plus they use Airflow to power their data platform behind the scenes > so > > > > they have a vested interest in this one :) > > > > > > > > -- > > > > > > > > *Julian LaNeve* > > > > CTO > > > > > > > > Email: [email protected] > > > > ( [email protected] ) Mobile: 330 509 5792 > > > > > > > > On Wed, Jun 26, 2024 at 4:58 PM, Vikram Koka < > > > [email protected] > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > > > > > > > Love it! > > > > > > > > > > > > > > > > > > > > On Wed, Jun 26, 2024 at 1:23 PM Vincent Beck < vincbeck@ apache. > > org ( > > > > > [email protected] ) > wrote: > > > > > > > > > > > > > > >> > > > > >> > > > > >> Fantastic idea! > > > > >> > > > > >> > > > > >> > > > > >> On 2024/06/26 20:12:43 Jarek Potiuk wrote: > > > > >> > > > > >> > > > > >>> > > > > >>> > > > > >>> Hello everyone, > > > > >>> > > > > >>> > > > > >>> > > > > >>> Together with Elad, Kaxil, and the Dosu team [1], we’ve been > > looking > > > > into > > > > >>> employing AI / Natural Language processing to help us triage > issues > > > for > > > > >>> Apache Airflow. We do not want to go “all-in” into getting a > > chatbot > > > to > > > > >>> respond to all our issues because we believe this is not how the > > > > >>> > > > > >>> > > > > >> > > > > >> > > > > >> > > > > >> community > > > > >> > > > > >> > > > > >>> > > > > >>> > > > > >>> is being built. We looked at various ways we can start exploring > > the > > > > >>> capabilities of the new ML/AI/Natural Language processing > > available. > > > > >>> > > > > >>> > > > > >>> > > > > >>> We worked with the Dosu team. They are approved by the Apache > > > Software > > > > >>> Foundation infrastructure as Github integration and few ASF > > projects > > > > >>> already use it (including our friends at Superset) - they have a > > > > >>> > > > > >>> > > > > >> > > > > >> > > > > >> > > > > >> fantastic > > > > >> > > > > >> > > > > >>> > > > > >>> > > > > >>> offer to provide free service for open-source projects like > > Airflow. > > > > >>> Together we evaluated what we can start with and initially we > have > > a > > > > >>> proposal to use auto-labeling of issues created in the Airflow > > > > >>> > > > > >>> > > > > >> > > > > >> > > > > >> > > > > >> repository. > > > > >> > > > > >> > > > > >>> > > > > >>> > > > > >>> We have a number of rules that are established for the triage > team > > > [2] > > > > >>> > > > > >>> > > > > >> > > > > >> > > > > >> > > > > >> but > > > > >> > > > > >> > > > > >>> > > > > >>> > > > > >>> those rules are mundane and difficult to follow, and generally a > > lot > > > of > > > > >>> > > > > >>> > > > > >> > > > > >> > > > > >> > > > > >> our > > > > >> > > > > >> > > > > >>> > > > > >>> > > > > >>> issues are either not classified or badly classified, and > currently > > > we > > > > >>> cannot rely on the classification. > > > > >>> > > > > >>> > > > > >>> > > > > >>> What we want to start with is to re-classify our issues and apply > > the > > > > >>> labels retro-actively for all past issues as well as start > applying > > > > them > > > > >>> automatically for new issues. > > > > >>> > > > > >>> > > > > >>> > > > > >>> The risk of doing it is low, and it will allow us to explore > > > > integration > > > > >>> and follow up with more elaborated integration. We have some > > options > > > > such > > > > >>> as getting automated proposals for answers for similar questions, > > as > > > > well > > > > >>> as “chat-bot generated/maintainer approved” answers - but we > > > definitely > > > > >>> > > > > >>> > > > > >> > > > > >> > > > > >> > > > > >> do > > > > >> > > > > >> > > > > >>> > > > > >>> > > > > >>> not want to have bots starting to answer automatically on PRs and > > > > issues. > > > > >>> > > > > >>> > > > > >>> > > > > >>> We think that this will allow us to explore more ways how we can > > make > > > > >>> maintainers and triagers time more efficient - and help us while > we > > > are > > > > >>> focusing also on Airflow 3 development soon. > > > > >>> > > > > >>> > > > > >>> > > > > >>> The Dosu founder - Devin, will send some more information soon > and > > is > > > > >>> available for questions here and in the #triage-team channel on > > > Slack. > > > > >>> > > > > >>> > > > > >>> > > > > >>> Unless we hear some complaints, we will apply labelling changes > in > > a > > > > few > > > > >>> days, I think this stage is not really controversial, and we will > > > run a > > > > >>> LAZY CONSENSUS in a few days. > > > > >>> > > > > >>> > > > > >>> > > > > >>> J. E. K. (and the Dosu team). > > > > >>> > > > > >>> > > > > >>> > > > > >>> [1] https:/ / dosu. dev/ ( https://dosu.dev/ ) > > > > >>> > > > > >>> > > > > >>> > > > > >>> [2] > > > > >>> > > > > >>> > > > > >> > > > > >> > > > > >> > > > > >> https:/ / github. com/ apache/ airflow/ blob/ main/ > > > > ISSUE_TRIAGE_PROCESS. rst#labels > > > > >> ( > > > > >> > > > > > > > > > > https://github.com/apache/airflow/blob/main/ISSUE_TRIAGE_PROCESS.rst#labels > > > > >> ) > > > > >> > > > > >> > > > > >> > > > > >> > > --------------------------------------------------------------------- > > > To > > > > >> unsubscribe, e-mail: dev-unsubscribe@ airflow. apache. org ( > > > > >> [email protected] ) For additional commands, > > e-mail: > > > > dev-help@ > > > > >> airflow. apache. org ( [email protected] ) > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > >
