Xin, Do you already have patches? Or just waiting for this before doing some works?
2017년 3월 31일 (금) 오후 3:10, Xin Wang <[email protected]>님이 작성: > +1 on keeping the current external/storm-* in place and move flux, sql, > storm-submit-tools into top-level. > > BTW, I am waiting for the discuss result before submitting several SQL > related PRs. :) > > - Xin > > 2017-03-31 10:41 GMT+08:00 Jungtaek Lim <[email protected]>: > > > Yeah sure I'm OK to just apply it for master branch. > > Are you okay for moving them to root directory without renaming? Or do > you > > want to rename or suggest other base directory? > > > > 2017년 3월 31일 (금) 오전 11:36, P. Taylor Goetz <[email protected]>님이 작성: > > > > > I'm hesitant to change the layout of the 1.x release. People do some > > crazy > > > things when it comes to operations that we can't predict. I'm okay with > > > doing this on the master/2.0 branch, but I'm hesitant on the 1.x > branch. > > > > > > -Taylor > > > > > > > On Mar 30, 2017, at 9:01 PM, Jungtaek Lim <[email protected]> wrote: > > > > > > > > Once we just released Storm 1.1.0, I guess we can continue discussion > > > here. > > > > > > > > I checked the PR list, and there're no open PRs for among Flux, Storm > > > SQL, > > > > and Storm submit tool. So it's good to go. > > > > > > > > I think we have consensus to move non-connectors modules to out of > > > > external. There're also some interest about renaming "external" to > > > > "connectors", but given that "external" is chosen by community and > has > > > been > > > > the norm for years, so I agree it would be better not to rename it. > We > > > can > > > > do it later when there's another chance of doing it. > > > > > > > > We didn't decide where to move, but most of us seems to be OK to move > > to > > > > the top directory. > > > > Taylor, any further opinion regarding this? > > > > Suppose we're moving them to the top directory (or whatever the same > > base > > > > directory), what would be good names for them? Flux doesn't have > prefix > > > > 'storm' so a bit different, but if we're OK we skip renaming it. > > > > > > > > I'll do the work when Taylor is OK for changing this. > > > > > > > > - Jungtaek Lim (HeartSaVioR) > > > > > > > > 2017년 3월 27일 (월) 오전 11:16, Jungtaek Lim <[email protected]>님이 작성: > > > > > > > >> 1. apply versions > > > >> > > > >> First plan was applying this only for master, but realized that > > > >> contributors should make two patches for every PRs when we apply > this > > to > > > >> only master. > > > >> > > > >> So in order to make less inconvenience, it would be better to apply > > this > > > >> for both 1.x and master. It also affects opened pull requests so we > > > would > > > >> like to check that relevant PRs are open, and apply it later than > > > reviewing > > > >> them. > > > >> > > > >> I agree with Harsha. No need to make change for current release > vote. > > > >> > > > >> 2. naming issue for "external" > > > >> > > > >> "external" makes me feel that it's related to "external" component, > > say, > > > >> outside of Storm. That's why I suggest moving non-connectors to out > of > > > >> "external". IMHO "connector" is still more intuitive and > > > self-describing, > > > >> but I understand that renaming the directory structure would be > > painful, > > > >> and "storm-kafka-monitor" is an example of what it's not a > "connector" > > > but > > > >> an "external". So I'm OK to keep it as "external". > > > >> > > > >> 3. where to move non-connectors > > > >> > > > >> Except Flux they're directly supported by Storm. I mean "storm.py" > is > > > >> aware of them and supports them, so for me they are eligible to move > > to > > > the > > > >> top directory. I'm open to other suggestion as well. > > > >> > > > >> - Jungtaek Lim (HeartSaVioR) > > > >> > > > >> 2017년 3월 26일 (일) 오후 12:14, Harsha Chintalapani <[email protected]>님이 > > 작성: > > > >> > > > >> We should this in next release of 1.x or 2.0. I am +1 on continue > with > > > >> current release. > > > >> -Harsha > > > >>> On Fri, Mar 24, 2017 at 8:53 PM P. Taylor Goetz <[email protected] > > > > > wrote: > > > >>> > > > >>> The question remains if we want to do this in the 1.1.0 release, or > > > >> later. > > > >>> > > > >>> If it's the 1.1.0 release we need to make the changes and cut > another > > > RC. > > > >>> I'm fine with that, but want to make sure we have consensus before > > > going > > > >>> down that road. > > > >>> > > > >>> -Taylor > > > >>> > > > >>>> On Mar 24, 2017, at 10:57 PM, Harsha Chintalapani < > [email protected]> > > > >>> wrote: > > > >>>> > > > >>>> Agree on change like this would be confusing to the users. Lets > keep > > > >> the > > > >>>> original plan of moving non-connectors modules of external instead > > of > > > >>>> introducing new changes > > > >>>> that are not in scope of this discussion. > > > >>>> My +1 still stands on keeping the current external/storm-* in > place > > > and > > > >>>> move just sql and storm-perf into top-level. We can have > discussion > > > for > > > >>>> storm 2.0 if we want to do > > > >>>> more changes. > > > >>>> > > > >>>> -Harsha > > > >>>> > > > >>>>> On Fri, Mar 24, 2017 at 4:31 PM P. Taylor Goetz < > [email protected] > > > > > > >>> wrote: > > > >>>>> > > > >>>>> If we decide to change the structure of the distribution like > > this, I > > > >>>>> think we should do it in masrwe/2.0. If we want this for 1.1.0 we > > > need > > > >>> to > > > >>>>> cut a new release candidate. > > > >>>>> > > > >>>>> Changing the structure of the distribution file structure can be > > > >>>>> disruptive for users. Even the change to no longer include > > connector > > > >>>>> binaries, as we've learned, will be a headache for some users. > > > >>>>> > > > >>>>> IMHO, from an ops perspective, changes like this should be > handled > > > >> like > > > >>>>> API changes. > > > >>>>> > > > >>>>> -Taylor > > > >>>>> > > > >>>>>> On Mar 24, 2017, at 4:07 PM, Hugo Da Cruz Louro < > > > >>> [email protected]> > > > >>>>> wrote: > > > >>>>>> > > > >>>>>> Another possibility is to keep the ‘external' module, and create > > sub > > > >>>>> modules under it. The legacy structure would remain intact, while > > > >> making > > > >>>>> things more modular. An idea would be: > > > >>>>>> > > > >>>>>> + external > > > >>>>>> + connectors > > > >>>>>> + tools > > > >>>>>> + monitoring > > > >>>>>> + etc > > > >>>>>> > > > >>>>>> Hugo > > > >>>>>> > > > >>>>>>> On Mar 24, 2017, at 12:34 PM, P. Taylor Goetz < > [email protected] > > > > > > >>>>> wrote: > > > >>>>>>> > > > >>>>>>> For the background on why “external” was selected, you have to > go > > > >> back > > > >>>>> to a lengthy discussion in Feb. 2014. > > > >>>>>>> > > > >>>>>>> Here’s the start of the thread: > > > >>>>>>> > > > >>>>>>> > > > >>>>> > > > >>> > > > >> > > > http://mail-archives.apache.org/mod_mbox/storm-dev/201402. > > mbox/%[email protected]%3e > > > >>>>> < > > > >>>>> > > > >>> > > > >> > > > http://mail-archives.apache.org/mod_mbox/storm-dev/201402. > > mbox/%[email protected]%3E > > > >>>>>> > > > >>>>>>> > > > >>>>>>> It continues into March: > > > >>>>>>> > > > >>>>>>> > > > >>>>> > > > >>> > > > >> > > > http://mail-archives.apache.org/mod_mbox/storm-dev/201403.mbox/% > > 3ccadimvzum1d3om30zayqq4xxe1vjbn7fumqcsgu+524oqgec...@mail.gmail.com%3e > > > >>>>>>> > > > >>>>>>> I’m -1 on renaming “external”. That’s the name chosen by the > > > >> community > > > >>>>> and it has been the norm for 3 years. Changing it would likely > > > confuse > > > >>>>> users. > > > >>>>>>> > > > >>>>>>> One of the ideas behind “external” was that it would contain > > > >>> components > > > >>>>> that were not essential to running storm. That line has recently > > > >> blurred > > > >>>>> with some non-connector code sneaking in, so I’m okay with moving > > > >>>>> non-connector code out of external. Another point in that thread > > was > > > a > > > >>>>> desire to avoid cluttering up the root directory, so we need to > be > > > >>> careful > > > >>>>> about what the destination for those components is. > > > >>>>>>> > > > >>>>>>> -Taylor > > > >>>>>>> > > > >>>>>>>> On Mar 24, 2017, at 3:11 PM, Hugo Da Cruz Louro < > > > >>>>> [email protected]> wrote: > > > >>>>>>>> > > > >>>>>>>> +1 non-connectors to top level > > > >>>>>>>> +1 to renaming external to connectors > > > >>>>>>>> > > > >>>>>>>> As for storm-kaka, if we are already touching the external > > > modules, > > > >>>>> all the modules should be a submodule of a parent module called > > > >>>>> storm-kafka. I don’t think we should have 3 parent modules as we > > > >>> currently > > > >>>>> have (storm-kafka, storm-kafka-client, storm-kafka-monitor) > > > >>>>>>>> > > > >>>>>>>> The structure should be something along the lines (I don’t > mean > > > the > > > >>>>> exact names; we should find better ones. storm-kafka and > > > >>>>> storm-kafka-client are not very self explanatory in my opinion) > > > >>>>>>>> > > > >>>>>>>> + storm-kafka > > > >>>>>>>> + monitoring > > > >>>>>>>> + new-client > > > >>>>>>>> + old-client > > > >>>>>>>> > > > >>>>>>>> If we have to create new modules or submodules (e.g. under > > utils) > > > >> so > > > >>>>> be it. The code should be in a module that is named after what > its > > > >>> doing. > > > >>>>>>>> > > > >>>>>>>> Hugo > > > >>>>>>>> > > > >>>>>>>>> On Mar 24, 2017, at 11:15 AM, Priyank Shah < > > > [email protected] > > > >>> > > > >>>>> wrote: > > > >>>>>>>>> > > > >>>>>>>>> +1 to moving non-conncectors to top level. I think we should > > keep > > > >>>>> stom-kafka-monitor under external or connectors(after renaming). > > > >>>>>>>>> > > > >>>>>>>>> Jungtaek, just to clarify on what you said regarding storm > core > > > >>>>> referencing storm-kafka-monitor. Like you said its just calling > the > > > >>> script > > > >>>>> from ui jvm. There is no dependency in terms of class files > needed > > to > > > >>> run > > > >>>>> the script from ui. The script itself adds a –cp argument and all > > it > > > >>> needs > > > >>>>> is storm-kafka-monitor jar in classpath. As far as packaging the > > > >> script > > > >>> is > > > >>>>> concerned we can do what Satish suggested. i.e. move it to > > > >>>>> storm-kafka-monitor in source and while packaging put it under > bin. > > > >>>>> Reiterating to make sure I am not mis-understanding anything. > > > >>>>>>>>> > > > >>>>>>>>> On 3/24/17, 9:14 AM, "Harsha Chintalapani" <[email protected]> > > > >> wrote: > > > >>>>>>>>> > > > >>>>>>>>> +1 on moving non-connectors to top-level like sql and > > storm-perf. > > > >>>>>>>>> Regarding storm-kafka-monitor we can move this into "util" > > folder > > > >> or > > > >>>>> keep > > > >>>>>>>>> in the external. > > > >>>>>>>>> -Harsha > > > >>>>>>>>> > > > >>>>>>>>> On Fri, Mar 24, 2017 at 2:23 AM Satish Duggana < > > > >>>>> [email protected]> > > > >>>>>>>>> wrote: > > > >>>>>>>>> > > > >>>>>>>>>> storm-kafka-monitor is not a connector by itself but it is > > > >> related > > > >>>>> to kafka > > > >>>>>>>>>> connectors. So, any utility related to that connector should > > be > > > >>> part > > > >>>>> of > > > >>>>>>>>>> that connector module(can be a submodule) instead of a top > > level > > > >>>>> module. > > > >>>>>>>>>> core/ui uses this utility referring directly in a hacky way, > > > >> which > > > >>>>> we may > > > >>>>>>>>>> want to fix later. storm-kafka-monitor script exists in bin > > > >>>>> directory which > > > >>>>>>>>>> can be moved to storm-kafka-monitor module and the same > script > > > >> can > > > >>> be > > > >>>>>>>>>> packaged as part of storm/bin directory while packaging the > > > >>>>> distribution. > > > >>>>>>>>>> > > > >>>>>>>>>> Thanks, > > > >>>>>>>>>> ~Satish. > > > >>>>>>>>>> > > > >>>>>>>>>>> On Fri, Mar 24, 2017 at 1:07 PM, Jungtaek Lim < > > > >> [email protected]> > > > >>>>> wrote: > > > >>>>>>>>>>> > > > >>>>>>>>>>> storm-kafka-monitor is referred by storm-core, though it's > > > >>>>> referenced via > > > >>>>>>>>>>> executing command. Yes it's a bit odd to place it as top > > > >>> directory, > > > >>>>> but > > > >>>>>>>>>>> it's not a connector for that reason too. Neither is ideal > > for > > > >> me, > > > >>>>> so > > > >>>>>>>>>>> ironically, either is fine. > > > >>>>>>>>>>> > > > >>>>>>>>>>> - Jungtaek Lim (HeartSaVioR) > > > >>>>>>>>>>> > > > >>>>>>>>>>> 2017년 3월 24일 (금) 오후 4:19, Satish Duggana < > > > >>> [email protected] > > > >>>>>> 님이 > > > >>>>>>>>>> 작성: > > > >>>>>>>>>>> > > > >>>>>>>>>>>> +1 except for storm-kafka-monitor module as this utility > is > > > >> more > > > >>>>> about > > > >>>>>>>>>>>> querying topic/partition offsets of kafka spouts in a > > > topology. > > > >>> Do > > > >>>>> not > > > >>>>>>>>>> we > > > >>>>>>>>>>>> want to push this module into connectors/kafka as a > > submodule > > > >>> along > > > >>>>>>>>>> with > > > >>>>>>>>>>>> other submodules including old/new kafka spout modules? > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Thanks, > > > >>>>>>>>>>>> Satish. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> On Fri, Mar 24, 2017 at 12:10 PM, Arun Iyer < > > > >>> [email protected] > > > >>>>>> > > > >>>>>>>>>>> wrote: > > > >>>>>>>>>>>> > > > >>>>>>>>>>>>> +1 > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Makes sense to move the non-connectors to top level and > > keep > > > >>> only > > > >>>>> the > > > >>>>>>>>>>>>> connectors under “connectors” folder. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>>> On 3/24/17, 12:00 PM, "Jungtaek Lim" <[email protected] > > > > > >>> wrote: > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> (Sent this yesterday but can't find this from storm-dev > > > >> mbox... > > > >>>>>>>>>>> sending > > > >>>>>>>>>>>> it > > > >>>>>>>>>>>>>> again) > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Hi dev, > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> I'd like to start discussion regarding moving > > non-connectors > > > >>>>> modules > > > >>>>>>>>>>> out > > > >>>>>>>>>>>>> of > > > >>>>>>>>>>>>>> external, maybe top directory. > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> "external" directory has non-connectors (SQL, Flux, > > > >>>>>>>>>>> storm-kafka-monitor, > > > >>>>>>>>>>>>>> storm-submit-tools), and except Flux, others should be > > > placed > > > >>> to > > > >>>>> the > > > >>>>>>>>>>>>> binary > > > >>>>>>>>>>>>>> dist. since Storm itself (not from user topology) needs > to > > > >>> refer > > > >>>>>>>>>> them. > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> They're actually tied to the core of Storm, so I feel > that > > > it > > > >>>>> would > > > >>>>>>>>>> be > > > >>>>>>>>>>>>>> better to treat them (including Flux) as non-external, > > maybe > > > >>> same > > > >>>>>>>>>>> level > > > >>>>>>>>>>>> as > > > >>>>>>>>>>>>>> storm-core. > > > >>>>>>>>>>>>>> (I'm not sure what "external" actually means for Storm > > > >> project > > > >>>>> btw.) > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> In addition, after doing that I'd like to change the > > > >> directory > > > >>>>> name > > > >>>>>>>>>>>>>> "external" to "connector" or so, so that the name could > be > > > >>>>>>>>>>>> self-describing > > > >>>>>>>>>>>>>> and we can only place connectors to that directory. > > > >>>>>>>>>>>>>> (I know it would be painful for already opened pull > > > requests, > > > >>> so > > > >>>>> no > > > >>>>>>>>>>>> strong > > > >>>>>>>>>>>>>> opinion regarding this.) > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Looking forward to your opinion! > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Thanks, > > > >>>>>>>>>>>>>> Jungtaek Lim (HeartSaVioR) > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>> > > > >>>>>>> > > > >>>>>> > > > >>>>> > > > >>> > > > >> > > > >> > > > > > >
