Another possibility is to keep the ‘external' module, and create sub modules
under it. The legacy structure would remain intact, while making things more
modular. An idea would be:
+ external
+ connectors
+ tools
+ monitoring
+ etc
Hugo
> On Mar 24, 2017, at 12:34 PM, P. Taylor Goetz <[email protected]> wrote:
>
> For the background on why “external” was selected, you have to go back to a
> lengthy discussion in Feb. 2014.
>
> Here’s the start of the thread:
>
> http://mail-archives.apache.org/mod_mbox/storm-dev/201402.mbox/%[email protected]%3e
>
> <http://mail-archives.apache.org/mod_mbox/storm-dev/201402.mbox/%[email protected]%3E>
>
> It continues into March:
>
> http://mail-archives.apache.org/mod_mbox/storm-dev/201403.mbox/%3ccadimvzum1d3om30zayqq4xxe1vjbn7fumqcsgu+524oqgec...@mail.gmail.com%3e
>
> I’m -1 on renaming “external”. That’s the name chosen by the community and it
> has been the norm for 3 years. Changing it would likely confuse users.
>
> One of the ideas behind “external” was that it would contain components that
> were not essential to running storm. That line has recently blurred with some
> non-connector code sneaking in, so I’m okay with moving non-connector code
> out of external. Another point in that thread was a desire to avoid
> cluttering up the root directory, so we need to be careful about what the
> destination for those components is.
>
> -Taylor
>
>> On Mar 24, 2017, at 3:11 PM, Hugo Da Cruz Louro <[email protected]>
>> wrote:
>>
>> +1 non-connectors to top level
>> +1 to renaming external to connectors
>>
>> As for storm-kaka, if we are already touching the external modules, all the
>> modules should be a submodule of a parent module called storm-kafka. I don’t
>> think we should have 3 parent modules as we currently have (storm-kafka,
>> storm-kafka-client, storm-kafka-monitor)
>>
>> The structure should be something along the lines (I don’t mean the exact
>> names; we should find better ones. storm-kafka and storm-kafka-client are
>> not very self explanatory in my opinion)
>>
>> + storm-kafka
>> + monitoring
>> + new-client
>> + old-client
>>
>> If we have to create new modules or submodules (e.g. under utils) so be it.
>> The code should be in a module that is named after what its doing.
>>
>> Hugo
>>
>>> On Mar 24, 2017, at 11:15 AM, Priyank Shah <[email protected]> wrote:
>>>
>>> +1 to moving non-conncectors to top level. I think we should keep
>>> stom-kafka-monitor under external or connectors(after renaming).
>>>
>>> Jungtaek, just to clarify on what you said regarding storm core referencing
>>> storm-kafka-monitor. Like you said its just calling the script from ui jvm.
>>> There is no dependency in terms of class files needed to run the script
>>> from ui. The script itself adds a –cp argument and all it needs is
>>> storm-kafka-monitor jar in classpath. As far as packaging the script is
>>> concerned we can do what Satish suggested. i.e. move it to
>>> storm-kafka-monitor in source and while packaging put it under bin.
>>> Reiterating to make sure I am not mis-understanding anything.
>>>
>>> On 3/24/17, 9:14 AM, "Harsha Chintalapani" <[email protected]> wrote:
>>>
>>> +1 on moving non-connectors to top-level like sql and storm-perf.
>>> Regarding storm-kafka-monitor we can move this into "util" folder or keep
>>> in the external.
>>> -Harsha
>>>
>>> On Fri, Mar 24, 2017 at 2:23 AM Satish Duggana <[email protected]>
>>> wrote:
>>>
>>>> storm-kafka-monitor is not a connector by itself but it is related to kafka
>>>> connectors. So, any utility related to that connector should be part of
>>>> that connector module(can be a submodule) instead of a top level module.
>>>> core/ui uses this utility referring directly in a hacky way, which we may
>>>> want to fix later. storm-kafka-monitor script exists in bin directory which
>>>> can be moved to storm-kafka-monitor module and the same script can be
>>>> packaged as part of storm/bin directory while packaging the distribution.
>>>>
>>>> Thanks,
>>>> ~Satish.
>>>>
>>>> On Fri, Mar 24, 2017 at 1:07 PM, Jungtaek Lim <[email protected]> wrote:
>>>>
>>>>> storm-kafka-monitor is referred by storm-core, though it's referenced via
>>>>> executing command. Yes it's a bit odd to place it as top directory, but
>>>>> it's not a connector for that reason too. Neither is ideal for me, so
>>>>> ironically, either is fine.
>>>>>
>>>>> - Jungtaek Lim (HeartSaVioR)
>>>>>
>>>>> 2017년 3월 24일 (금) 오후 4:19, Satish Duggana <[email protected]>님이
>>>> 작성:
>>>>>
>>>>>> +1 except for storm-kafka-monitor module as this utility is more about
>>>>>> querying topic/partition offsets of kafka spouts in a topology. Do not
>>>> we
>>>>>> want to push this module into connectors/kafka as a submodule along
>>>> with
>>>>>> other submodules including old/new kafka spout modules?
>>>>>>
>>>>>> Thanks,
>>>>>> Satish.
>>>>>>
>>>>>>
>>>>>> On Fri, Mar 24, 2017 at 12:10 PM, Arun Iyer <[email protected]>
>>>>> wrote:
>>>>>>
>>>>>>> +1
>>>>>>>
>>>>>>> Makes sense to move the non-connectors to top level and keep only the
>>>>>>> connectors under “connectors” folder.
>>>>>>>
>>>>>>>
>>>>>>> On 3/24/17, 12:00 PM, "Jungtaek Lim" <[email protected]> wrote:
>>>>>>>
>>>>>>>> (Sent this yesterday but can't find this from storm-dev mbox...
>>>>> sending
>>>>>> it
>>>>>>>> again)
>>>>>>>>
>>>>>>>> Hi dev,
>>>>>>>>
>>>>>>>> I'd like to start discussion regarding moving non-connectors modules
>>>>> out
>>>>>>> of
>>>>>>>> external, maybe top directory.
>>>>>>>>
>>>>>>>> "external" directory has non-connectors (SQL, Flux,
>>>>> storm-kafka-monitor,
>>>>>>>> storm-submit-tools), and except Flux, others should be placed to the
>>>>>>> binary
>>>>>>>> dist. since Storm itself (not from user topology) needs to refer
>>>> them.
>>>>>>>>
>>>>>>>> They're actually tied to the core of Storm, so I feel that it would
>>>> be
>>>>>>>> better to treat them (including Flux) as non-external, maybe same
>>>>> level
>>>>>> as
>>>>>>>> storm-core.
>>>>>>>> (I'm not sure what "external" actually means for Storm project btw.)
>>>>>>>>
>>>>>>>> In addition, after doing that I'd like to change the directory name
>>>>>>>> "external" to "connector" or so, so that the name could be
>>>>>> self-describing
>>>>>>>> and we can only place connectors to that directory.
>>>>>>>> (I know it would be painful for already opened pull requests, so no
>>>>>> strong
>>>>>>>> opinion regarding this.)
>>>>>>>>
>>>>>>>> Looking forward to your opinion!
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Jungtaek Lim (HeartSaVioR)
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
