Here is the jira for this: APEXMALHAR-1979 <https://issues.apache.org/jira/browse/APEXMALHAR-1979>.
If there are no further replies on this topic, I'll update the Jira with summary of discussion and create a pull request soon. Also, If its fine I would skip the nested POJO and Jar output support of this utility for now. I'll add the support for it in next iteration if any use case comes in. Thanks, Chinmay. On Fri, Jan 22, 2016 at 7:16 PM, Chinmay Kolhatkar <[email protected]> wrote: > OK. We'll do with JSONObject. > > ~ Chinmay > On 22 Jan 2016 19:05, "Bhupesh Chawda" <[email protected]> wrote: > >> Yes. So in the JDBC output case, a json object could be readily >> constructed >> by the operator to get a POJO class rather than playing around with JSON >> syntax. >> >> On Fri, Jan 22, 2016 at 6:54 PM, Chinmay Kolhatkar < >> [email protected]> >> wrote: >> >> > @bhupesh, would it help if JSONObject is taken as input to this utility >> > rather than json string? >> > >> > Thanks, >> > Chinmay >> > On 22 Jan 2016 18:45, "Chinmay Kolhatkar" <[email protected]> >> wrote: >> > >> > > Got it. I think approach 2 is still possible with json based schema. >> > > So basically, the operator can create json schema as mention in first >> > mail >> > > based on column name and type. Then pass that json schema to this >> utility >> > > to get class to be used directly. >> > > >> > > Thanks, >> > > Chinmay. >> > > On 22 Jan 2016 18:38, "Priyanka Gugale" <[email protected]> >> > wrote: >> > > >> > >> Say I need to emit pojo from JDBCInputOperator. I will use on fly >> POJO >> > >> generator for this purpose instead of providing predefined class. >> > >> Now to generate POJO on fly there are two options: >> > >> 1. Get json as config from user as you specified. >> > >> 2. Based on column name and type generate pojo fields and hence pojo >> > class >> > >> without much of user configuration. (I am only considering one level >> > here, >> > >> not the nested pojo) >> > >> >> > >> Option 1 is must have to provide user a way to control how we >> generate >> > >> pojo. But option 2 is useful when user simply want to put column >> values >> > in >> > >> some object. >> > >> >> > >> -Priyanka >> > >> >> > >> On Fri, Jan 22, 2016 at 6:30 PM, Chinmay Kolhatkar < >> > >> [email protected]> >> > >> wrote: >> > >> >> > >> > Hi Priyanka, >> > >> > >> > >> > Can you give an example? >> > >> > >> > >> > Thanks, >> > >> > Chinmay. >> > >> > >> > >> > ~ Chinmay >> > >> > On 22 Jan 2016 18:14, "Priyanka Gugale" <[email protected]> >> > >> wrote: >> > >> > >> > >> > > We do need configuration in some form like json, but I was >> thinking >> > we >> > >> > can >> > >> > > have default wherever possible. This is an option and not >> > replacement >> > >> to >> > >> > > json config. >> > >> > > >> > >> > > -Priyanka >> > >> > > >> > >> > > On Fri, Jan 22, 2016 at 4:34 PM, Chinmay Kolhatkar < >> > >> > > [email protected]> >> > >> > > wrote: >> > >> > > >> > >> > > > @priyanka >> > >> > > > >> > >> > > > Advantage with JSON schema is that one can represent nested >> POJO >> > >> > > definition >> > >> > > > in that. If you're referring to FieldInfo, I'm not sure if >> that is >> > >> > > possible >> > >> > > > there. >> > >> > > > >> > >> > > > Even if we chose not to have nested POJO now, I think JSON >> schema >> > >> input >> > >> > > to >> > >> > > > this utility gives more chance to expand easily later. >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > > > On Fri, Jan 22, 2016 at 3:30 PM, Chinmay Kolhatkar < >> > >> > > > [email protected]> >> > >> > > > wrote: >> > >> > > > >> > >> > > > > Sure. We can make this utility to provide byte[] for compiled >> > >> class. >> > >> > > > > Though the utility will give a OutputStream object.. One can >> get >> > >> > byte[] >> > >> > > > > out of that. >> > >> > > > > >> > >> > > > > On Fri, Jan 22, 2016 at 2:56 PM, Timothy Farkas < >> > >> [email protected] >> > >> > > >> > >> > > > > wrote: >> > >> > > > > >> > >> > > > >> Suggestion: +1 for automatic generation of schema. The user >> > >> should >> > >> > be >> > >> > > > able >> > >> > > > >> to override the a manually specified schema though. >> > >> > > > >> >> > >> > > > >> Question: If the mechanism used to generate the class can >> > >> produce a >> > >> > > > byte[] >> > >> > > > >> array, then you can send the byte[] array for the class to >> > >> > downstream >> > >> > > > >> operators and load the class from the byte[] array in each >> > >> operator. >> > >> > > An >> > >> > > > >> example of how to do this is here: >> > >> > > > >> >> > >> > > > >> >> > >> > > > >> >> > >> > > > >> > >> > > >> > >> > >> > >> >> > >> http://stackoverflow.com/questions/1781091/java-how-to-load-class-stored-as-byte-into-the-jvm >> > >> > > > >> >> > >> > > > >> Thanks, >> > >> > > > >> Tim >> > >> > > > >> >> > >> > > > >> On Fri, Jan 22, 2016 at 12:59 AM, Priyanka Gugale < >> > >> > > > >> [email protected]> >> > >> > > > >> wrote: >> > >> > > > >> >> > >> > > > >> > Hi, >> > >> > > > >> > >> > >> > > > >> > Suggestion: >> > >> > > > >> > This came up with one of the discussion with Pramod, will >> it >> > >> be a >> > >> > > good >> > >> > > > >> > idea, for database input operators to generate pojo based >> on >> > >> the >> > >> > > > >> selected >> > >> > > > >> > column field names and types? No need to accept json input >> > from >> > >> > > user. >> > >> > > > >> > >> > >> > > > >> > Question: >> > >> > > > >> > How can we share this POJO among multiple operators when >> > >> > application >> > >> > > > is >> > >> > > > >> > already launched and class is generated on the fly? >> > >> > > > >> > >> > >> > > > >> > -Priyanka >> > >> > > > >> > >> > >> > > > >> > On Thu, Jan 21, 2016 at 11:16 PM, Chinmay Kolhatkar < >> > >> > > > >> > [email protected] >> > >> > > > >> > > wrote: >> > >> > > > >> > >> > >> > > > >> > > Hi All, >> > >> > > > >> > > >> > >> > > > >> > > We're planning to add a utility in malhar-library for >> > >> > generating a >> > >> > > > >> POJO >> > >> > > > >> > > class on the fly from given JSON schema. >> > >> > > > >> > > >> > >> > > > >> > > Use case is where the application is provided with >> schema >> > and >> > >> > that >> > >> > > > >> needs >> > >> > > > >> > to >> > >> > > > >> > > be used in one or more operators either as a tuple over >> the >> > >> > stream >> > >> > > > OR >> > >> > > > >> for >> > >> > > > >> > > processing. >> > >> > > > >> > > >> > >> > > > >> > > General Design: >> > >> > > > >> > > 1. Utility will be provided with fqcn of the class and >> > schema >> > >> > > > >> definition >> > >> > > > >> > > provided as json. >> > >> > > > >> > > 2. The schema definition will look like following: >> > >> > > > >> > > { >> > >> > > > >> > > "fqcn":"<qualified class name>", >> > >> > > > >> > > "fields": [ >> > >> > > > >> > > { >> > >> > > > >> > > "name":"field1", >> > >> > > > >> > > "type":"long" >> > >> > > > >> > > }, >> > >> > > > >> > > { >> > >> > > > >> > > "name":"field2", >> > >> > > > >> > > "type": "string" >> > >> > > > >> > > } >> > >> > > > >> > > ] >> > >> > > > >> > > } >> > >> > > > >> > > 3. Supported types identified in "type" JSON field are: >> > >> > > > >> > > boolean, char, byte, short, int, float, long, >> double >> > >> > > > >> > > 4. The output of this utility will be a generated .class >> > >> file in >> > >> > > the >> > >> > > > >> form >> > >> > > > >> > > of FSDataOutputStream. >> > >> > > > >> > > 5. Xbean asm5 library will be used for this. >> > >> > > > >> > > 6. Following methods will be added to the generated >> class: >> > >> > > > >> > > a. Getter/Setter methods for given fields. >> > >> > > > >> > > b. simple toString - Generate string equivalent for >> all >> > >> the >> > >> > > > fields >> > >> > > > >> > > c. hashCode method - calculate the overall hashCode >> > using >> > >> > > > >> individual >> > >> > > > >> > > field hashcodes, similar to how String.hashCode >> generates >> > >> > hashCode >> > >> > > > of >> > >> > > > >> > > string using chars in it. >> > >> > > > >> > > d. equals method - Similar to how String.equals >> method >> > >> has >> > >> > > done >> > >> > > > it >> > >> > > > >> > > using individual char. >> > >> > > > >> > > >> > >> > > > >> > > >> > >> > > > >> > > Questions I have about the functionality: >> > >> > > > >> > > 1. Should the utility also support nested pojo >> definition >> > via >> > >> > > schema >> > >> > > > >> > json? >> > >> > > > >> > > In such case the field definition can look like >> > >> following: >> > >> > > > >> > > { "name":"nestedField", "type":"<fqcn of nested >> > class>", >> > >> > > > >> > > "fields":[....] } >> > >> > > > >> > > The nested class will be provided >> > >> > > > >> > > >> > >> > > > >> > > 2. Should the utility also support giving out the >> > >> > JarOutputStream >> > >> > > > >> which >> > >> > > > >> > > will have a generated jar? >> > >> > > > >> > > >> > >> > > > >> > > Please let me know your thoughts. >> > >> > > > >> > > >> > >> > > > >> > > Thanks, >> > >> > > > >> > > Chinmay. >> > >> > > > >> > > >> > >> > > > >> > >> > >> > > > >> >> > >> > > > > >> > >> > > > > >> > >> > > > >> > >> > > >> > >> > >> > >> >> > > >> > >> >
