Ilan: So I have never had any disagreements with your analysis of what does not work. I have never had any competing designs or approaches. I am not against different designs.
When I assert this design works and scales, it's mainly to point out that design is never the problem I've seen here. I've gone through a couple times to discover all the problems - make everything work and work stable and fast. When you say, who cares about all these other things, about scale and performance, what about fundamental issues that prevent basic good behavior like leadership and leadership sync up, I have no disagreement there. If you are using the system today, those are the things I would want addressed. The problems that I see that have me caring little about other designs are that those fundamental blocks have been in about the same state since the system went out. In 2013 and 2014, when I had moved to Cloudera and like two companies used cloud, I understood why things were in that state. My charge was HDFS, and when I could, I would scour the email lists for anyone even trying the system. Almost no one else worked on it. I made quick hack improvements or additions when I could. >From then on, use took off. The number of developers doing things on cloud took off. A thousand features and code and improvements and changes came in. Many companies started making and saving millions of dollars on the system. And those fundamental issues remained the same. Most of the core cloud code itself remained the same. Things got improved here and there. LIR came in, and then was redone to be even better. But then look at the issues you are facing that it solves easily with tiny effort but still has not solved. Look at the code that fuels the basic issues you are concerned with and how much if it is almost exactly what it always was. Look at the mountain of development and effort that has been put on top of it. That is why new designs did not excite me in 2018, 2019 or today. How do they solve the fact that the current designs could have worked 6 years ago. 4 years ago. 3 years ago. There are some great memes out there lining me up with Trump. It's all broken and dilapidated. And only I can fix it. Mexico will pay for it. I'm a playful person with a child's mind. I'll lean right into that kind of fun. But partially why I enjoy that humor so much is because I enjoy irony, absurdity, and sarcasm. That's my language. The opposite of reality is a fun place. But the basic fundamental problem I have seen and don't see how new designs or anything that's been suggested to me will fix is that I'm not the only one that can fix it. Fixing the most basic problems that everyone that has used cloud has struggled with, invested huge outside effort or restrictions working around, gone out of their way to build around - is not genius work. In many cases it's not even major work. And yet those fundamental problems and that fundamental code remains little changed. For ages. And even still. As arguments and bike shedding and tower building have soared along on top. And so what is wrong? And even if I fixed it all. Fixed everything else too. What would change about the fundamental issue that has caused this situation? Am I the only one that can fix it? Two, three years ago it's a funny dig. Today, things are in the same state. And that is why I will offer help in whatever you or anyone else someday attempts to do better or other. But I don't see things just waiting for some fix to turn these basic issues around and so I'm not looking to lead that effort or design new approaches. I've got some implementation effort to add for anyone that does want to do that though. - Mark
