Mauro, First of all, thanks for proposing contribution of valuable effort on Apache Storm! Really appreciated.
I don't know about C#/.net but I have been working on the change of multilang, so adding my voice here. 1. Major concern from my side for code donation is sustainability. Not sure but according to my experience, most of Storm PMCs and contributors don't look using Windows at their dev. environment. It doesn't strictly mean they don't have experience with C#, but relatively higher chance. I'm sorry to the Azure team, but I recently found a forgotten major pull request on storm-eventhubs, which doesn't look like no one could review and maintain. It might become real pain if we receive the code which we can't/don't maintain, hence I'd rather not vote +1 unless at least two PMCs know C# well, understand the code completely, and willing to volunteer to maintain. 2. As you may know, all of multilang adapters in Storm repo are actually close to "example" of implementations. They're just implementations of the protocol, and don't provide any language specific optimization as well as language-standard code style. Most of Python users in Storm community would rather use StreamParse, and it is not uncommon to see StreamParse question in user group (whereas they have their own Github repo and issue as well). I would like to see adapter (and more) projects really looking attractive for other languages as well. 3. How it affect releasing Storm? We don't publish them as package in its language package environment. If NuGet is one of them, we need to add the sequence of release phase for NuGet while releasing Storm, which was not happened yet, and will become another pain. Moreover, if it's the case, I don't feel needs for having strict coupled between Storm and .net adapter package. For user side it's not a "battery-included" and there's no difference between maintaining inside/outside. You could freely use user/dev list to announce new .net adapter release and such announcements are happening from other projects as well. 4. It's completely a thought on my own, but I feel more needs of having more language-native way of supporting language instead of keep improving multilang. (Not meant to discontinue.) We will introduce streams API in Storm 2.0.0, a higher-level API like Trident but typed and record-to-record processing. We haven't supported Trident in multilang, but I'd like to see support streams API in non-JVM language, not only defining protocol, but also have it as first-class support (users should be able to construct their own topology with only using the language. there's thrift but not that convenient.). IMHO, according to Spark's "Lesson learned" (Databricks had a poll and published it), I think it's really clear that Python should be first (and only, R might be another good to have). Thanks for reading a wall of text. Regardless of whether we could include .net adapter into Storm core or not, thanks again for crafting .net adapter and proposing for donation. Jungtaek Lim (HeartSaVioR) 2018년 2월 2일 (금) 오전 2:03, Mauro Giusti <[email protected]>님이 작성: > Hello Storm devs - > > We are working on a project that uses Storm and C# / .net core components. > > As part of that, we would love to contribute to the Storm project with a > multilang protocol sample that uses our C# component/adapter. > The adapter will be a NuGet package, and we intend to publish the source > code of this component as well. > > With that in mind, I have two questions: > > * Would you prefer the code for the .net adapter (not the sample: that > will be on the Storm repo, just the adapter) to be hosted inside the Storm > repo or in a separate GitHub repo? We see pros and cons: we might need to > introduce .net core compilation in the Storm repo if we host the adapter > there, the code will be a bit more scattered and harder to test if we host > the adapter outside. > * Can you help us review / answer design questions about our adapter > and the multilang protocol? Is there a specific set of people that is more > knowledgeable about this and/or wants to help in this specific area? > > Thank you, > Mauro Giusti > Azure Team, Microsoft. >
