Hello community (specifically Milan as well), Kindly request wider community support in solving both - temporary issue we have with "edge" provider naming, but also with the proposed long term solution (PEP 752) that I am going to be co-author of.
More context and update on "edge" provider name in PyPI and a bit more general on possible improvements in Packaging scheme that we might have in the future to avoid similar issues: It took me a while - as I've been busy with restructuring providers and other parts of airflow code (still ongoing) but I discussed it with Ofek (author of PEP 752) and used the restructuring of providers to be able to explain more clearly what the problem is and here are the things i put in motion: * I just created an issue in PyPI support to claim "apache-airflow-providers-edge" for us https://github.com/pypi/support/issues/5829 (no matter what's the root cause for not being available to reserve it and who had reserved it - we have no idea who that someone could be - let's see how PyPI support will handle it) * with this PR https://github.com/python/peps/pull/4292 and as agreed with Ofek, I will become a co-author of *PEP-752 Implicit namespaces for package repositories* https://peps.python.org/pep-0752/ . I also used the opportunity of the "edge" case to strengthen arguments for that PEP being needed and asked bot ASF infrastructure team to support it and Dustin Ingram (who was Airflow Summit speaker and is a PEP-delegate for that PEP) - so hopefully, if that PEP gets approved we will avoid similar problems in the future. I would really appreciate, likes, support, maybe some comments and clarifications in those discussions and issues. There are people in the Python/Packaging community who think that such implicit namespaces are not needed and that they can be replaced by upcoming cryptographic provenance, but together with Ofek, we think that both mechanisms are needed - for slightly different reasons. One of the things that was mentioned in the discussion was that we need some more community voices to support our quest, so If people reading it could support our case, that would be fantastic. J. On Mon, Dec 23, 2024 at 7:59 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > As for your invitation to collaborate on PEP, I am honored and grateful > for the opportunity. Please let me know how I can get involved and what the > next steps are > > Very cool :). I will engage you after the holidays. > > Happy Holidays! > > > > On Mon, Dec 23, 2024 at 4:30 AM Milan Katwal <milankatwal2...@gmail.com> > wrote: > >> Thank you for your detailed response and for clarifying the situation >> regarding the name in your documentation. >> >> First of all, I sincerely apologize for addressing the public mailing >> list while reporting this. It was a mistake on my part, and I will ensure >> it does not happen again in future communications. I appreciate your >> understanding and the clarification that this is not a security issue. >> >> Regarding the two options you provided, I would be happy to proceed with >> option 1 and transfer the name to you but now I don't have control over >> that project because someone has reported that project and the project got >> removed from the pypi registry and now I am unable to claim it also. As >> for your invitation to collaborate on PEP, I am honored and grateful for >> the opportunity. Please let me know how I can get involved and what the >> next steps are >> >> On Fri, Dec 20, 2024 at 11:41 PM Jarek Potiuk <ja...@potiuk.com> wrote: >> >>> Hello Milan, >>> >>> First of all NEVER mix private and public mailing lists when you are >>> reporting a potential security vulnerability - which you did by addressing >>> dev@airflow.apache.org in this message. >>> Luckily this is not a security vulnerability - so your mistake has no >>> bad consequences, so I continue keeping dev@airflow.apache.org on the >>> cc: >>> >>> This is just a proposed name that appeared in our docs in main that we >>> have not released it, and it's not a security issue - because - we have not >>> released it yet (and your reservation made it impossible for now to release >>> it). There are two options now: >>> >>> 1) Can you please transfer it to me (potiuk is my id in PyPI) and we >>> will continue developing the provider. That would be best. >>> 2) We will change the name >>> >>> I think it's best if you do 1). >>> >>> And since it's not the first time you send (undoubtedly automated) >>> similar email to secur...@apache.org which is (undoubtedly) result of >>> automated scanning of github repositories, I think it's a nice service for >>> use - it's the second time a name was accidentally added to unreleased >>> documentation in an Apache Project, you took that over, and then you >>> transfer it to rightful owner. This is a really nice "service" you provide. >>> Thanks for that. >>> >>> But on a more serious note - as you know - we cannot follow your >>> recommendation (not until https://peps.python.org/pep-0752/ gets >>> implemented and approved). Other than that your recommendation is cool - >>> but completely not practical, so it's not worth too much. >>> >>> But I know the author of the PEP (Ofek) is looking for people from the >>> community to join the PEP as co-authors and explain why it is needed. And >>> you seem one of the best people who can explain why, how you automated it >>> and what should be done to fix it - would you like to join us? I actually >>> offered Ofek that I will join the PEP, but having someone like you who is >>> not the user who needs it but someone who "actively exploited the weakness >>> of PyPI" might be a super-valuable addition to have you as co-author. Can >>> you please let me know if you would like to be added there. >>> >>> This way - if we manage to get the PEP through approval and get >>> implemented in PyPI - we will be able to solve the problem systemically. >>> >>> J. >>> >>> >>> On Fri, Dec 20, 2024 at 3:32 PM Milan Katwal <milankatwal2...@gmail.com> >>> wrote: >>> >>>> I have reported this vulnerability to hackerone ibb #2871958 they said >>>> to contact to the team so I am submitting again here >>>> Description >>>> I discovered and claimed the PyPI package >>>> apache-airflow-providers-edge, a package associated with the Apache Airflow >>>> project. This vulnerability enables supply chain attacks by allowing >>>> malicious actors to publish unauthorized packages under a trusted >>>> namespace. >>>> Steps for Reproduction >>>> >>>> 1. visit >>>> >>>> https://github.com/apache/airflow/blob/1b67b4386c91ddcb7dc80fcce4d0fe0b701efc78/docs/apache-airflow-providers-edge/index.rst?plain=1#L49 >>>> 2. click on the link >>>> 3. you will be redirected to my pypi package >>>> >>>> Exact Report >>>> https://hackerone.com/reports/925585 >>>> References >>>> https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610 >>>> Recommendation >>>> Reserve all namespaces under apache-airflow for official maintainers >>>> POC >>>> { >>>> *Image F3803461*: Screenshot_2024-12-01_at_9.45.31_AM.png 515.27 KiB >>>> Zoom in >>>> <https://hackerone.com/bugs?subject=user&report_id=2871958&view=all&substates%5B%5D=editing&substates%5B%5D=new&substates%5B%5D=triaged&substates%5B%5D=needs-more-info&substates%5B%5D=resolved&substates%5B%5D=informative&substates%5B%5D=not-applicable&substates%5B%5D=duplicate&substates%5B%5D=spam&substates%5B%5D=retesting&substates%5B%5D=pending-program-review&reported_to_team=&text_query=&program_states%5B%5D=2&program_states%5B%5D=3&program_states%5B%5D=4&program_states%5B%5D=5&sort_type=latest_activity&sort_direction=descending&limit=25&page=1#activity-31628130> >>>> Zoom >>>> out >>>> <https://hackerone.com/bugs?subject=user&report_id=2871958&view=all&substates%5B%5D=editing&substates%5B%5D=new&substates%5B%5D=triaged&substates%5B%5D=needs-more-info&substates%5B%5D=resolved&substates%5B%5D=informative&substates%5B%5D=not-applicable&substates%5B%5D=duplicate&substates%5B%5D=spam&substates%5B%5D=retesting&substates%5B%5D=pending-program-review&reported_to_team=&text_query=&program_states%5B%5D=2&program_states%5B%5D=3&program_states%5B%5D=4&program_states%5B%5D=5&sort_type=latest_activity&sort_direction=descending&limit=25&page=1#activity-31628130> >>>> Copy >>>> <https://hackerone.com/bugs?subject=user&report_id=2871958&view=all&substates%5B%5D=editing&substates%5B%5D=new&substates%5B%5D=triaged&substates%5B%5D=needs-more-info&substates%5B%5D=resolved&substates%5B%5D=informative&substates%5B%5D=not-applicable&substates%5B%5D=duplicate&substates%5B%5D=spam&substates%5B%5D=retesting&substates%5B%5D=pending-program-review&reported_to_team=&text_query=&program_states%5B%5D=2&program_states%5B%5D=3&program_states%5B%5D=4&program_states%5B%5D=5&sort_type=latest_activity&sort_direction=descending&limit=25&page=1#activity-31628130> >>>> Download >>>> <https://hackerone-us-west-2-production-attachments.s3.us-west-2.amazonaws.com/0ohbn4sk5o4mmbbhzkwmndgnyiad?response-content-disposition=attachment%3B%20filename%3D%22Screenshot_2024-12-01_at_9.45.31_AM.png%22%3B%20filename%2A%3DUTF-8%27%27Screenshot_2024-12-01_at_9.45.31_AM.png&response-content-type=image%2Fpng&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAQGK6FURQVM4XPYGL%2F20241220%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20241220T142518Z&X-Amz-Expires=3600&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEMX%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMiJHMEUCIQCPAOacNvn1yRbkMZOTMmwbe0jOZY7qSnAR1omlFA7bsQIgbmRtqA69zEJ%2FHNl3kuK7B%2B7P8aVFmGW6DJyIELRpd78quwUIjv%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARADGgwwMTM2MTkyNzQ4NDkiDFBHKLduc7IFPukkkyqPBe7xJVb6%2BsfK5F%2BBPPbGKMl8tUY%2Fx864KXUVSkuzJIE2lKgPJC0D25jv3zU2Z7PoWf9JKniGhvzeIko4JkXNAePTZqKQ6cTbi5E1oVeCQXnYvv2cD8vdym6rK2Iwz9p0uaS6UYPCTBDFeFWVe0bVZlEKJmZLc0OMbMani33DCHgZn5hLstAVYle3th0PcDX%2Bs65F5nhDMmHu7w8r7rG9VLzg40HSmMzL5H3h8zVteVs5KZNNOh3ZD1vlejsKsF7m9%2BBOgU8lJ8TSYH7IaeMiSrPTal%2Fvs680nTM5THfZdpru1V7KFNyA9MTrGVkaP1IYB77I1rZupClPu79Rr8RK67FoEtR2hcJxvSPBe0SwqEGf7ubLoylXnc358%2FLcY9pDvKCEMUdXqdsd2iuTNRenUfMJbSCFTffaDG55pAEhiF34oTZyJFeUu%2BatMdjtruzZ8kGNWX4EZHdqWu9ggoskjfE2AUgrdXE5YY%2Bdq8OxHa0c4ePLH45u3RTI74c3tvAkrPARkOnOlndupJq3Wqa%2By4s9uxQ2np5bF96dFNIBnLTrcduoffncXxMGnqmdgfB6QOzSH7yrRE234e8Mkk10TFa9ShURZqx%2B1WsvL%2BsCJCb%2BG7NdAnoGGE0Q1ZtKCuk4B72P4VKJJ2DVlU%2F%2BiKKwdCpO%2BaB9swx8pnPC0SgGwaAzOzO%2BTZHoKF8859mxvGf4SgyWEuGB%2BoHIYwsv6BIdVhSnsK0SbV2FpYL29uPOEkxIteYIYM7GGW%2Fw3edQzrmv%2FZ%2FUa10fIT2n7LHNSYgRPQYGA1EoMfa3UiAPPezI5zoERCNuayBqg39uLL%2BicaJ%2FHzDwNSktWttGOthi%2FRoZUgdgtZIzRIdj55ZlR9az%2FhgwocmVuwY6sQGxjOShaB4joBw%2FXf%2F2l%2BCC79kY7d5bwoe3vMBbNhnNUo1Qlq%2Bmpx7hKshc5a0w4OMPfwnlwiywI2yrQJ81D7B%2BZtk2hcB6%2B6IHoK1r8Lz74TTDar7CkS6HqGQhlMS%2F%2F%2BI%2FH%2BAUq9CNKEDwm0wLnOdsaqe59dzT0vjgbY3YkHtHA5UII%2Bc1yrR8GtMTsxlUg3gn%2FfX3BWR6J2Vouwx5VZduX2CcboeuAvloyx8HCBCAApY%3D&X-Amz-SignedHeaders=host&X-Amz-Signature=8acf4bdf1f6a85db73c393e0c40b2074c9ba6be7d90546a0aec99ce28caedc7b> >>>> } { >>>> *Image F3803462*: Screenshot_2024-12-01_at_9.46.17_AM.png 390.67 KiB >>>> Zoom in >>>> <https://hackerone.com/bugs?subject=user&report_id=2871958&view=all&substates%5B%5D=editing&substates%5B%5D=new&substates%5B%5D=triaged&substates%5B%5D=needs-more-info&substates%5B%5D=resolved&substates%5B%5D=informative&substates%5B%5D=not-applicable&substates%5B%5D=duplicate&substates%5B%5D=spam&substates%5B%5D=retesting&substates%5B%5D=pending-program-review&reported_to_team=&text_query=&program_states%5B%5D=2&program_states%5B%5D=3&program_states%5B%5D=4&program_states%5B%5D=5&sort_type=latest_activity&sort_direction=descending&limit=25&page=1#activity-31628130> >>>> Zoom >>>> out >>>> <https://hackerone.com/bugs?subject=user&report_id=2871958&view=all&substates%5B%5D=editing&substates%5B%5D=new&substates%5B%5D=triaged&substates%5B%5D=needs-more-info&substates%5B%5D=resolved&substates%5B%5D=informative&substates%5B%5D=not-applicable&substates%5B%5D=duplicate&substates%5B%5D=spam&substates%5B%5D=retesting&substates%5B%5D=pending-program-review&reported_to_team=&text_query=&program_states%5B%5D=2&program_states%5B%5D=3&program_states%5B%5D=4&program_states%5B%5D=5&sort_type=latest_activity&sort_direction=descending&limit=25&page=1#activity-31628130> >>>> Copy >>>> <https://hackerone.com/bugs?subject=user&report_id=2871958&view=all&substates%5B%5D=editing&substates%5B%5D=new&substates%5B%5D=triaged&substates%5B%5D=needs-more-info&substates%5B%5D=resolved&substates%5B%5D=informative&substates%5B%5D=not-applicable&substates%5B%5D=duplicate&substates%5B%5D=spam&substates%5B%5D=retesting&substates%5B%5D=pending-program-review&reported_to_team=&text_query=&program_states%5B%5D=2&program_states%5B%5D=3&program_states%5B%5D=4&program_states%5B%5D=5&sort_type=latest_activity&sort_direction=descending&limit=25&page=1#activity-31628130> >>>> Download >>>> <https://hackerone-us-west-2-production-attachments.s3.us-west-2.amazonaws.com/ys6e1cpi9b3rk12zex3u3uff6bg8?response-content-disposition=attachment%3B%20filename%3D%22Screenshot_2024-12-01_at_9.46.17_AM.png%22%3B%20filename%2A%3DUTF-8%27%27Screenshot_2024-12-01_at_9.46.17_AM.png&response-content-type=image%2Fpng&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAQGK6FURQVM4XPYGL%2F20241220%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20241220T142518Z&X-Amz-Expires=3600&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEMX%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMiJHMEUCIQCPAOacNvn1yRbkMZOTMmwbe0jOZY7qSnAR1omlFA7bsQIgbmRtqA69zEJ%2FHNl3kuK7B%2B7P8aVFmGW6DJyIELRpd78quwUIjv%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARADGgwwMTM2MTkyNzQ4NDkiDFBHKLduc7IFPukkkyqPBe7xJVb6%2BsfK5F%2BBPPbGKMl8tUY%2Fx864KXUVSkuzJIE2lKgPJC0D25jv3zU2Z7PoWf9JKniGhvzeIko4JkXNAePTZqKQ6cTbi5E1oVeCQXnYvv2cD8vdym6rK2Iwz9p0uaS6UYPCTBDFeFWVe0bVZlEKJmZLc0OMbMani33DCHgZn5hLstAVYle3th0PcDX%2Bs65F5nhDMmHu7w8r7rG9VLzg40HSmMzL5H3h8zVteVs5KZNNOh3ZD1vlejsKsF7m9%2BBOgU8lJ8TSYH7IaeMiSrPTal%2Fvs680nTM5THfZdpru1V7KFNyA9MTrGVkaP1IYB77I1rZupClPu79Rr8RK67FoEtR2hcJxvSPBe0SwqEGf7ubLoylXnc358%2FLcY9pDvKCEMUdXqdsd2iuTNRenUfMJbSCFTffaDG55pAEhiF34oTZyJFeUu%2BatMdjtruzZ8kGNWX4EZHdqWu9ggoskjfE2AUgrdXE5YY%2Bdq8OxHa0c4ePLH45u3RTI74c3tvAkrPARkOnOlndupJq3Wqa%2By4s9uxQ2np5bF96dFNIBnLTrcduoffncXxMGnqmdgfB6QOzSH7yrRE234e8Mkk10TFa9ShURZqx%2B1WsvL%2BsCJCb%2BG7NdAnoGGE0Q1ZtKCuk4B72P4VKJJ2DVlU%2F%2BiKKwdCpO%2BaB9swx8pnPC0SgGwaAzOzO%2BTZHoKF8859mxvGf4SgyWEuGB%2BoHIYwsv6BIdVhSnsK0SbV2FpYL29uPOEkxIteYIYM7GGW%2Fw3edQzrmv%2FZ%2FUa10fIT2n7LHNSYgRPQYGA1EoMfa3UiAPPezI5zoERCNuayBqg39uLL%2BicaJ%2FHzDwNSktWttGOthi%2FRoZUgdgtZIzRIdj55ZlR9az%2FhgwocmVuwY6sQGxjOShaB4joBw%2FXf%2F2l%2BCC79kY7d5bwoe3vMBbNhnNUo1Qlq%2Bmpx7hKshc5a0w4OMPfwnlwiywI2yrQJ81D7B%2BZtk2hcB6%2B6IHoK1r8Lz74TTDar7CkS6HqGQhlMS%2F%2F%2BI%2FH%2BAUq9CNKEDwm0wLnOdsaqe59dzT0vjgbY3YkHtHA5UII%2Bc1yrR8GtMTsxlUg3gn%2FfX3BWR6J2Vouwx5VZduX2CcboeuAvloyx8HCBCAApY%3D&X-Amz-SignedHeaders=host&X-Amz-Signature=41e5bb97909011f2f74ea7d79c3919e4f70190c5996ec7b9d43c77d71c07698f> >>>> } >>>> Impact >>>> Perform supply chain attacks by uploading malicious packages under a >>>> trusted Apache namespace Distribute malware or steal sensitive data from >>>> users Remote code execution code injection >>>> >>>
