GitHub user parshimers added a comment to the discussion: Two efforts to convert a Python program to a Texera workflow
I think this is an important topic. Learning how to solve conflicts in the direction of the project that naturally will arise, in an asynchronous and professional manner in a public forum, is hard. Since I am new to the project, and I don't want to force a viewpoint, I will describe some things that happened in AsterixDB that kind of have this flavor, and perhaps the community can use these examples to inform the eventual decision. - Initially in AsterixDB, we used RTrees for spatial storage. These came after BTrees. As it turns out, there is a method of using BTrees for spatial indexing. You simply use a space-filling curve to translate the 2d point into some index on a 1d line, and build a BTree on that. There was also another way to do this that I can't remember. Someone implemented all three of these designs, and benchmarked them. It turned out that RTrees were the most performant all along. Therefore a separate implementation of 2 other indexes was completely thrown out, and the only thing that was merged back into master from it was a minor optimization to not do per-page latching on immutable index data during search. The only real product of the effort was a research publication. - I was helping a new contributor to work on a new task. They had already contributed a relatively large change that nobody else wanted to touch due to how time-consuming and tedious it was to do. One issue that at that point, had been an issue for a very long time in AsterixDB, was the CSV parser. It was written by hand, and was very difficult to follow and understand. It often gave errors that didn't make a lot of sense. Since many of the initial uses of AsterixDB involved CSV this was a particularly acute issue. I was helping him to basically implement an idea I had, which was to simply use some other CSV parser from a library, rather than trying to fix the existing one. A similar thing had been done to great effect for parsing JSON, instead of using the ADM parser which was likewise handwritten. We got a few weeks into the task, when some other contributors wanted to discuss simply fixing the other CSV parser. They wanted to use it relatively immediately. They also had many other committers who were quite experienced and ready to work on it basically full-time, whereas the contributor I was helping was a student and hence only had some number of hours per week to dedicate to the task. Even though we had made some progress, worked openly, and had issues filed, I couldn't convince enough people to go with my approach and work together. The contributor basically had to backtrack, which was a very uncomfortable and disappointing thing. He did manage to make a great contribution after that, that is actively used to this day, so it ended up OK. It was a bitter pill, however. GitHub link: https://github.com/apache/texera/discussions/8521#discussioncomment-18585940 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
