Offline discussions on this topic with Gwen Shapira and Qian Xu happened last week. ( forgot to update the thread)
The resource returned from the rest API is a config object, hence the v1/config/link/ and v1/config/job will be kept. For the command line, it was suggested to use JOB and LINK as the main entity as follows. NOTE: the output of the following commands is still a MConfig. object show job --config name edit job --config name Regards Veena On Tue, Feb 10, 2015 at 8:13 AM, Veena Basavaraj <[email protected]> wrote: > Gwen, > > PING? > > The pending question seems to be the following. > > -Should the command line/rest API use config as the entity or should the > config editing be a sub-child of job/link objects.? > > Thoughts? > > > Regards > Veena > > > > > Best, > *./Vee* > > On Sat, Feb 7, 2015 at 6:50 AM, Veena Basavaraj <[email protected]> > wrote: > >> Answers inline, thanks for the feedback, good points. I have tried to >> answer them. Let me know if I was unclear. >> >> >> >> >> Best, >> *./Vee* >> >> On Fri, Feb 6, 2015 at 4:57 PM, Gwen Shapira <[email protected]> >> wrote: >> >>> Hi, >>> >>> Reviewed the design in the wiki ( >>> https://cwiki.apache.org/confluence/display/SQOOP/Sqoop+Config+as+Top+Level+Entity >>> ). >>> Thanks for writing such detailed plan. I think its a good idea allow >>> direct editing of configs and the scope of changes look right. >>> >>> Few questions: >>> >>> 1. In requirements, you mention editing inputs per submission and >>> referred to SQOOP-2025. However, SQOOP-2025 discusses storing history, and >>> I'm not sure it makes sense for history to be editable (well, except for >>> soviet history...). Did you mean to allow editing history? Or are you >>> referring to something else? >>> >> >> When the design wiki was first written the SQOOP-2025 was in discussion. >> In future I hope we store config per submission and not overwrite it, at >> that point we should allow reading by submission ID. Editing on submission >> history was not intended, I have corrected the wiki into 2 separate bullet >> points. >> >> - Read the Config Inputs by Type/SubType and By Job /Submission ( >> since SQOOP-2025 <https://issues.apache.org/jira/browse/SQOOP-2025> we >> may be able to have configs by submissionId) >> - Update the Config Inputs by Type/SubType for the latest/last >> submission in the job. We should not allow editing previous submissions >> and >> it should be read only >> >> >>> 2. CLI commands: >>> "show config foo --type JOB --subType from --id 1" >>> >>> I see few possible issues here: >>> 1. I think users don't see config names, so they won't be able to know >>> about "foo" >>> >> Config objects per type are lists. So ability to edit per list is easier >> since they dont have to go through filling all other unrelated configs in >> the list. Users do see the names when they list the configs per connector. >> Am I missing something? >> >> >> 2. We don't want to use IDs, in CLI (thats an issue across the board, so >>> we may leave it here and fix somewhere else) >>> >> >> We have not yet added support for names in any commands so far, there is >> ticket for it, at that point it makes sense to support names, id exists so >> far for every other command and hence I think it is consistent. >> >> >> >>> 3. Having type and subtype seems a bit confusing. Actually, since we >>> don't allow creating configs directly, users may not view them as "first >>> class". >>> In their perspective, they created jobs and links and now they get to >>> view and edit parts of those jobs and links. >>> >> >> This is the point I am trying to fix as well, when creating a job and >> link, instead of having to provide the same set of config inputs every >> time, it is convenient to give a config name/ id, especially when creating >> REST calls, it is highly erratic and verbose and difficult to fill all the >> config inputs in a JSON structure ( for the POST), when we could just give >> a config name or id. so the intention of this ticket is the user should >> start seeing config as a first class citizen. I will make this statement >> explicit in the design wiki if it is not already. Speaking from the >> experience of using these apis in HUE app it is unnecessarily hard. >> >>> >>> How about just adding config-type as a filter to the existing job and >>> link commands? >>> >>> "show job --name my_job --config-type from" >>> and >>> "alter job --name my_job --config-type from" >>> >>> This seems to also match the REST API better. Although user facing >>> commands don't have to match REST API. >>> Perhaps others want to chime in here? >>> >> Answered above and explained it again below. >> >>> >>> 3. In REST API, why are we using subType and not type? Is type already >>> used somewhere? >>> >> >> I spent a few iterations to understand what can be the best way here. >> As I said above, in future as more connectors are added, we can see >> config objects with more inputs, and if we keep extending the config/input >> annotation as per https://issues.apache.org/jira/browse/SQOOP-1643, it >> might be useful to have configs and inputs as first class citizens and >> having both rest apis/ command line support to edit / read them >> individually and not having associated with a JOB/ LINK. >> >> Currently there is a top level ENUM ( MConfigType ), we have 2 values for >> it LINK/ JOB. This is what I refer to as type in the command line. In case >> of rest API. I used it a a resource name, >> v1/config/LINK?name=?&id=?&subType=, >> but if this is confusing we can also use it as >> >> v1/config?type=LINk&name=?&id=?&subType= >> >> >> Second, >> >> If you see the changes proposed to the MConfigType, it stores the >> subTypes as part of the Type. >> >> At one point, I thought why not have direction as a parameter for type, >> JOB, but direction is not relevant to all configurables. i,e if for the >> driver configs, "direction" has no meaning. Similarly for the type "LINK" >> there is no concept of direction. >> >> Hence I went with the subType, where subType is a second level hierarchy >> for distinguishing the types of configs that are supported in sqoop >> >> Alternatives are possible, but we have to bear in mind that config/ >> config inputs cannot be associated with jobs and links. they are associated >> with connectors. >> >> The config input values are associated with jobs and links rather. >> >> >>> 4. Repository changes: Yes! I suspect we need those anyway. >>> >> Ok I assume the current signature is fine with you then >> >> >> >>> >>> Gwen >>> >>> >>> >>> >> >> >
