+1 non-connectors to top level +1 to renaming external to connectors As for storm-kaka, if we are already touching the external modules, all the modules should be a submodule of a parent module called storm-kafka. I don’t think we should have 3 parent modules as we currently have (storm-kafka, storm-kafka-client, storm-kafka-monitor)
The structure should be something along the lines (I don’t mean the exact names; we should find better ones. storm-kafka and storm-kafka-client are not very self explanatory in my opinion) + storm-kafka + monitoring + new-client + old-client If we have to create new modules or submodules (e.g. under utils) so be it. The code should be in a module that is named after what its doing. Hugo > On Mar 24, 2017, at 11:15 AM, Priyank Shah <[email protected]> wrote: > > +1 to moving non-conncectors to top level. I think we should keep > stom-kafka-monitor under external or connectors(after renaming). > > Jungtaek, just to clarify on what you said regarding storm core referencing > storm-kafka-monitor. Like you said its just calling the script from ui jvm. > There is no dependency in terms of class files needed to run the script from > ui. The script itself adds a –cp argument and all it needs is > storm-kafka-monitor jar in classpath. As far as packaging the script is > concerned we can do what Satish suggested. i.e. move it to > storm-kafka-monitor in source and while packaging put it under bin. > Reiterating to make sure I am not mis-understanding anything. > > On 3/24/17, 9:14 AM, "Harsha Chintalapani" <[email protected]> wrote: > > +1 on moving non-connectors to top-level like sql and storm-perf. > Regarding storm-kafka-monitor we can move this into "util" folder or keep > in the external. > -Harsha > > On Fri, Mar 24, 2017 at 2:23 AM Satish Duggana <[email protected]> > wrote: > >> storm-kafka-monitor is not a connector by itself but it is related to kafka >> connectors. So, any utility related to that connector should be part of >> that connector module(can be a submodule) instead of a top level module. >> core/ui uses this utility referring directly in a hacky way, which we may >> want to fix later. storm-kafka-monitor script exists in bin directory which >> can be moved to storm-kafka-monitor module and the same script can be >> packaged as part of storm/bin directory while packaging the distribution. >> >> Thanks, >> ~Satish. >> >> On Fri, Mar 24, 2017 at 1:07 PM, Jungtaek Lim <[email protected]> wrote: >> >>> storm-kafka-monitor is referred by storm-core, though it's referenced via >>> executing command. Yes it's a bit odd to place it as top directory, but >>> it's not a connector for that reason too. Neither is ideal for me, so >>> ironically, either is fine. >>> >>> - Jungtaek Lim (HeartSaVioR) >>> >>> 2017년 3월 24일 (금) 오후 4:19, Satish Duggana <[email protected]>님이 >> 작성: >>> >>>> +1 except for storm-kafka-monitor module as this utility is more about >>>> querying topic/partition offsets of kafka spouts in a topology. Do not >> we >>>> want to push this module into connectors/kafka as a submodule along >> with >>>> other submodules including old/new kafka spout modules? >>>> >>>> Thanks, >>>> Satish. >>>> >>>> >>>> On Fri, Mar 24, 2017 at 12:10 PM, Arun Iyer <[email protected]> >>> wrote: >>>> >>>>> +1 >>>>> >>>>> Makes sense to move the non-connectors to top level and keep only the >>>>> connectors under “connectors” folder. >>>>> >>>>> >>>>> On 3/24/17, 12:00 PM, "Jungtaek Lim" <[email protected]> wrote: >>>>> >>>>>> (Sent this yesterday but can't find this from storm-dev mbox... >>> sending >>>> it >>>>>> again) >>>>>> >>>>>> Hi dev, >>>>>> >>>>>> I'd like to start discussion regarding moving non-connectors modules >>> out >>>>> of >>>>>> external, maybe top directory. >>>>>> >>>>>> "external" directory has non-connectors (SQL, Flux, >>> storm-kafka-monitor, >>>>>> storm-submit-tools), and except Flux, others should be placed to the >>>>> binary >>>>>> dist. since Storm itself (not from user topology) needs to refer >> them. >>>>>> >>>>>> They're actually tied to the core of Storm, so I feel that it would >> be >>>>>> better to treat them (including Flux) as non-external, maybe same >>> level >>>> as >>>>>> storm-core. >>>>>> (I'm not sure what "external" actually means for Storm project btw.) >>>>>> >>>>>> In addition, after doing that I'd like to change the directory name >>>>>> "external" to "connector" or so, so that the name could be >>>> self-describing >>>>>> and we can only place connectors to that directory. >>>>>> (I know it would be painful for already opened pull requests, so no >>>> strong >>>>>> opinion regarding this.) >>>>>> >>>>>> Looking forward to your opinion! >>>>>> >>>>>> Thanks, >>>>>> Jungtaek Lim (HeartSaVioR) >>>>> >>>>> >>>> >>> >> > >
