Hi Chris, Thanks for offering to join. I agree that maintaining this in the upstream project would be much better than keeping a separate fork.
My current rough plan is to proceed in three steps: 1. First, adjust only the JVM options and the default configuration parameters. We will run a series of tests under a hard 512 MiB combined memory limit for the ConfigNode and DataNode, in order to determine the best configuration for this target and characterize the read and write workloads it can support. 2. Next, try to run the DataNode and ConfigNode in a single JVM process. Since the ConfigNode module already depends on the DataNode module, this may be relatively straightforward. For example, the ConfigNode main entry point could also invoke the DataNode startup logic. At this stage, we would still keep the existing RPC communication path between the ConfigNode and DataNode to minimize code changes. 3. Finally, abstract the communication layer behind an interface, with one RPC-based implementation and one in-VM function-call implementation. To keep the code simple and maintainable, the interface may continue to use many of the existing Thrift request and response types. Therefore, the in-VM implementation may still depend on the generated Thrift classes for parameter definitions, while bypassing the network call itself. The first two steps are mainly configuration changes and should require relatively little code modification. I plan to submit PRs for them, and it would be great if you could help review them or share parameter recommendations based on your experience. For the third step, I would be very happy to work on it together. Could you share the current status of your fork, the repository link, and the approach you have taken so far? We can study the design together and decide how to move it upstream. Best regards, Yuan
