Re: Apache Druid Project Structure
Hi Gian, Sounds great. I've started going through code and will create a PR once I've enough understanding of project structure and internal workflow. Thank you :) On 2021/08/18 16:38:13, Gian Merlino wrote: > Hey Jeet, > > It sounds useful, maybe something in the main README.md or in the docs at > https://druid.apache.org/docs/latest/development/overview.html. If you are > volunteering to contribute it then that sounds awesome. Otherwise, whoever > is reading this… know that we all think it's a good idea > > On Wed, Aug 18, 2021 at 12:03 AM Jeet Patel wrote: > > > Hi Gian, > > > > This was very helpful information. > > Do you think it's a good idea to create a readme explaining at a > > high-level the project structure. As you explained it, this might be a very > > helpful information to new comers who are looking to contribute to the > > project and make them feel more confident knowing the project layout. > > > > Thank you, > > Jeet > > > > On 2021/08/17 17:12:33, Gian Merlino wrote: > > > Hey Jeet, > > > > > > I think it is a case of "it seemed like a good idea at the time". Some > > > things about the current layout do work well: one is that there is > > actually > > > a lot of common query engine code between anything that handles queries. > > > That's historical, broker, peon, and indexer. That common query engine > > code > > > today is mostly in "core" and "processing". Another is that Druid SQL is > > > architected as a layer that sits atop the native query system, and it's > > all > > > cleanly separated into its own "sql" module. Outside of the query engine > > > code, there is a bunch of historical, broker, and coordinator specific > > > stuff in the "server" module that could be broken out into 3 separate > > > modules, but I suppose the appropriate cost/benefit hasn't been there for > > > someone to actually do that. > > > > > > On Mon, Aug 16, 2021 at 7:07 AM Jeet Patel wrote: > > > > > > > Hello, > > > > > > > > A question about how druid directory structure came into existence. > > Druid > > > > has processes like historical, coordinator, overlord, broker, etc. > > > > > > > > We see that the current project root level directories are like > > > > > > > > druid > > > > |- indexing-service > > > > |- services > > > > |- sql > > > > |- core > > > > ... > > > > ... > > > > > > > > Can someone explain why this directory structure was formed instead of > > > > having something like following and place the code/modules related to > > the > > > > processes in their respective folders? > > > > > > > > druid > > > > |- historical > > > > |- broker > > > > |- coordinator > > > > |- extensions > > > > ... > > > > ... > > > > > > > > It would be great to know the background of this topic. > > > > > > > > Thank you :) > > > > > > > > - > > > > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org > > > > For additional commands, e-mail: dev-h...@druid.apache.org > > > > > > > > > > > > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org > > For additional commands, e-mail: dev-h...@druid.apache.org > > > > > - To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org For additional commands, e-mail: dev-h...@druid.apache.org
Re: Apache Druid Project Structure
Hey Jeet, It sounds useful, maybe something in the main README.md or in the docs at https://druid.apache.org/docs/latest/development/overview.html. If you are volunteering to contribute it then that sounds awesome. Otherwise, whoever is reading this… know that we all think it's a good idea On Wed, Aug 18, 2021 at 12:03 AM Jeet Patel wrote: > Hi Gian, > > This was very helpful information. > Do you think it's a good idea to create a readme explaining at a > high-level the project structure. As you explained it, this might be a very > helpful information to new comers who are looking to contribute to the > project and make them feel more confident knowing the project layout. > > Thank you, > Jeet > > On 2021/08/17 17:12:33, Gian Merlino wrote: > > Hey Jeet, > > > > I think it is a case of "it seemed like a good idea at the time". Some > > things about the current layout do work well: one is that there is > actually > > a lot of common query engine code between anything that handles queries. > > That's historical, broker, peon, and indexer. That common query engine > code > > today is mostly in "core" and "processing". Another is that Druid SQL is > > architected as a layer that sits atop the native query system, and it's > all > > cleanly separated into its own "sql" module. Outside of the query engine > > code, there is a bunch of historical, broker, and coordinator specific > > stuff in the "server" module that could be broken out into 3 separate > > modules, but I suppose the appropriate cost/benefit hasn't been there for > > someone to actually do that. > > > > On Mon, Aug 16, 2021 at 7:07 AM Jeet Patel wrote: > > > > > Hello, > > > > > > A question about how druid directory structure came into existence. > Druid > > > has processes like historical, coordinator, overlord, broker, etc. > > > > > > We see that the current project root level directories are like > > > > > > druid > > > |- indexing-service > > > |- services > > > |- sql > > > |- core > > > ... > > > ... > > > > > > Can someone explain why this directory structure was formed instead of > > > having something like following and place the code/modules related to > the > > > processes in their respective folders? > > > > > > druid > > > |- historical > > > |- broker > > > |- coordinator > > > |- extensions > > > ... > > > ... > > > > > > It would be great to know the background of this topic. > > > > > > Thank you :) > > > > > > - > > > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org > > > For additional commands, e-mail: dev-h...@druid.apache.org > > > > > > > > > > - > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org > For additional commands, e-mail: dev-h...@druid.apache.org > >
Re: Apache Druid Project Structure
Hi Gian, This was very helpful information. Do you think it's a good idea to create a readme explaining at a high-level the project structure. As you explained it, this might be a very helpful information to new comers who are looking to contribute to the project and make them feel more confident knowing the project layout. Thank you, Jeet On 2021/08/17 17:12:33, Gian Merlino wrote: > Hey Jeet, > > I think it is a case of "it seemed like a good idea at the time". Some > things about the current layout do work well: one is that there is actually > a lot of common query engine code between anything that handles queries. > That's historical, broker, peon, and indexer. That common query engine code > today is mostly in "core" and "processing". Another is that Druid SQL is > architected as a layer that sits atop the native query system, and it's all > cleanly separated into its own "sql" module. Outside of the query engine > code, there is a bunch of historical, broker, and coordinator specific > stuff in the "server" module that could be broken out into 3 separate > modules, but I suppose the appropriate cost/benefit hasn't been there for > someone to actually do that. > > On Mon, Aug 16, 2021 at 7:07 AM Jeet Patel wrote: > > > Hello, > > > > A question about how druid directory structure came into existence. Druid > > has processes like historical, coordinator, overlord, broker, etc. > > > > We see that the current project root level directories are like > > > > druid > > |- indexing-service > > |- services > > |- sql > > |- core > > ... > > ... > > > > Can someone explain why this directory structure was formed instead of > > having something like following and place the code/modules related to the > > processes in their respective folders? > > > > druid > > |- historical > > |- broker > > |- coordinator > > |- extensions > > ... > > ... > > > > It would be great to know the background of this topic. > > > > Thank you :) > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org > > For additional commands, e-mail: dev-h...@druid.apache.org > > > > > - To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org For additional commands, e-mail: dev-h...@druid.apache.org
Re: Apache Druid Project Structure
Hey Jeet, I think it is a case of "it seemed like a good idea at the time". Some things about the current layout do work well: one is that there is actually a lot of common query engine code between anything that handles queries. That's historical, broker, peon, and indexer. That common query engine code today is mostly in "core" and "processing". Another is that Druid SQL is architected as a layer that sits atop the native query system, and it's all cleanly separated into its own "sql" module. Outside of the query engine code, there is a bunch of historical, broker, and coordinator specific stuff in the "server" module that could be broken out into 3 separate modules, but I suppose the appropriate cost/benefit hasn't been there for someone to actually do that. On Mon, Aug 16, 2021 at 7:07 AM Jeet Patel wrote: > Hello, > > A question about how druid directory structure came into existence. Druid > has processes like historical, coordinator, overlord, broker, etc. > > We see that the current project root level directories are like > > druid > |- indexing-service > |- services > |- sql > |- core > ... > ... > > Can someone explain why this directory structure was formed instead of > having something like following and place the code/modules related to the > processes in their respective folders? > > druid > |- historical > |- broker > |- coordinator > |- extensions > ... > ... > > It would be great to know the background of this topic. > > Thank you :) > > - > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org > For additional commands, e-mail: dev-h...@druid.apache.org > >
Apache Druid Project Structure
Hello, A question about how druid directory structure came into existence. Druid has processes like historical, coordinator, overlord, broker, etc. We see that the current project root level directories are like druid |- indexing-service |- services |- sql |- core ... ... Can someone explain why this directory structure was formed instead of having something like following and place the code/modules related to the processes in their respective folders? druid |- historical |- broker |- coordinator |- extensions ... ... It would be great to know the background of this topic. Thank you :) - To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org For additional commands, e-mail: dev-h...@druid.apache.org
