Hi All, I've update the jira with teh summary of this discussion. Jira is APEXMALHAR-1979.
Also, the pull request is created for this: https://github.com/apache/incubator-apex-malhar/pull/181 Please review the pull request. Thanks, Chinmay. On Wed, Jan 27, 2016 at 5:14 PM, Chinmay Kolhatkar <[email protected]> wrote: > Here is the jira for this: APEXMALHAR-1979 > <https://issues.apache.org/jira/browse/APEXMALHAR-1979>. > > If there are no further replies on this topic, I'll update the Jira with > summary of discussion and create a pull request soon. > > Also, If its fine I would skip the nested POJO and Jar output support of > this utility for now. I'll add the support for it in next iteration if any > use case comes in. > > Thanks, > Chinmay. > > > > On Fri, Jan 22, 2016 at 7:16 PM, Chinmay Kolhatkar < > [email protected]> wrote: > >> OK. We'll do with JSONObject. >> >> ~ Chinmay >> On 22 Jan 2016 19:05, "Bhupesh Chawda" <[email protected]> wrote: >> >>> Yes. So in the JDBC output case, a json object could be readily >>> constructed >>> by the operator to get a POJO class rather than playing around with JSON >>> syntax. >>> >>> On Fri, Jan 22, 2016 at 6:54 PM, Chinmay Kolhatkar < >>> [email protected]> >>> wrote: >>> >>> > @bhupesh, would it help if JSONObject is taken as input to this utility >>> > rather than json string? >>> > >>> > Thanks, >>> > Chinmay >>> > On 22 Jan 2016 18:45, "Chinmay Kolhatkar" <[email protected]> >>> wrote: >>> > >>> > > Got it. I think approach 2 is still possible with json based schema. >>> > > So basically, the operator can create json schema as mention in first >>> > mail >>> > > based on column name and type. Then pass that json schema to this >>> utility >>> > > to get class to be used directly. >>> > > >>> > > Thanks, >>> > > Chinmay. >>> > > On 22 Jan 2016 18:38, "Priyanka Gugale" <[email protected]> >>> > wrote: >>> > > >>> > >> Say I need to emit pojo from JDBCInputOperator. I will use on fly >>> POJO >>> > >> generator for this purpose instead of providing predefined class. >>> > >> Now to generate POJO on fly there are two options: >>> > >> 1. Get json as config from user as you specified. >>> > >> 2. Based on column name and type generate pojo fields and hence pojo >>> > class >>> > >> without much of user configuration. (I am only considering one level >>> > here, >>> > >> not the nested pojo) >>> > >> >>> > >> Option 1 is must have to provide user a way to control how we >>> generate >>> > >> pojo. But option 2 is useful when user simply want to put column >>> values >>> > in >>> > >> some object. >>> > >> >>> > >> -Priyanka >>> > >> >>> > >> On Fri, Jan 22, 2016 at 6:30 PM, Chinmay Kolhatkar < >>> > >> [email protected]> >>> > >> wrote: >>> > >> >>> > >> > Hi Priyanka, >>> > >> > >>> > >> > Can you give an example? >>> > >> > >>> > >> > Thanks, >>> > >> > Chinmay. >>> > >> > >>> > >> > ~ Chinmay >>> > >> > On 22 Jan 2016 18:14, "Priyanka Gugale" <[email protected] >>> > >>> > >> wrote: >>> > >> > >>> > >> > > We do need configuration in some form like json, but I was >>> thinking >>> > we >>> > >> > can >>> > >> > > have default wherever possible. This is an option and not >>> > replacement >>> > >> to >>> > >> > > json config. >>> > >> > > >>> > >> > > -Priyanka >>> > >> > > >>> > >> > > On Fri, Jan 22, 2016 at 4:34 PM, Chinmay Kolhatkar < >>> > >> > > [email protected]> >>> > >> > > wrote: >>> > >> > > >>> > >> > > > @priyanka >>> > >> > > > >>> > >> > > > Advantage with JSON schema is that one can represent nested >>> POJO >>> > >> > > definition >>> > >> > > > in that. If you're referring to FieldInfo, I'm not sure if >>> that is >>> > >> > > possible >>> > >> > > > there. >>> > >> > > > >>> > >> > > > Even if we chose not to have nested POJO now, I think JSON >>> schema >>> > >> input >>> > >> > > to >>> > >> > > > this utility gives more chance to expand easily later. >>> > >> > > > >>> > >> > > > >>> > >> > > > >>> > >> > > > On Fri, Jan 22, 2016 at 3:30 PM, Chinmay Kolhatkar < >>> > >> > > > [email protected]> >>> > >> > > > wrote: >>> > >> > > > >>> > >> > > > > Sure. We can make this utility to provide byte[] for >>> compiled >>> > >> class. >>> > >> > > > > Though the utility will give a OutputStream object.. One >>> can get >>> > >> > byte[] >>> > >> > > > > out of that. >>> > >> > > > > >>> > >> > > > > On Fri, Jan 22, 2016 at 2:56 PM, Timothy Farkas < >>> > >> [email protected] >>> > >> > > >>> > >> > > > > wrote: >>> > >> > > > > >>> > >> > > > >> Suggestion: +1 for automatic generation of schema. The user >>> > >> should >>> > >> > be >>> > >> > > > able >>> > >> > > > >> to override the a manually specified schema though. >>> > >> > > > >> >>> > >> > > > >> Question: If the mechanism used to generate the class can >>> > >> produce a >>> > >> > > > byte[] >>> > >> > > > >> array, then you can send the byte[] array for the class to >>> > >> > downstream >>> > >> > > > >> operators and load the class from the byte[] array in each >>> > >> operator. >>> > >> > > An >>> > >> > > > >> example of how to do this is here: >>> > >> > > > >> >>> > >> > > > >> >>> > >> > > > >> >>> > >> > > > >>> > >> > > >>> > >> > >>> > >> >>> > >>> http://stackoverflow.com/questions/1781091/java-how-to-load-class-stored-as-byte-into-the-jvm >>> > >> > > > >> >>> > >> > > > >> Thanks, >>> > >> > > > >> Tim >>> > >> > > > >> >>> > >> > > > >> On Fri, Jan 22, 2016 at 12:59 AM, Priyanka Gugale < >>> > >> > > > >> [email protected]> >>> > >> > > > >> wrote: >>> > >> > > > >> >>> > >> > > > >> > Hi, >>> > >> > > > >> > >>> > >> > > > >> > Suggestion: >>> > >> > > > >> > This came up with one of the discussion with Pramod, >>> will it >>> > >> be a >>> > >> > > good >>> > >> > > > >> > idea, for database input operators to generate pojo >>> based on >>> > >> the >>> > >> > > > >> selected >>> > >> > > > >> > column field names and types? No need to accept json >>> input >>> > from >>> > >> > > user. >>> > >> > > > >> > >>> > >> > > > >> > Question: >>> > >> > > > >> > How can we share this POJO among multiple operators when >>> > >> > application >>> > >> > > > is >>> > >> > > > >> > already launched and class is generated on the fly? >>> > >> > > > >> > >>> > >> > > > >> > -Priyanka >>> > >> > > > >> > >>> > >> > > > >> > On Thu, Jan 21, 2016 at 11:16 PM, Chinmay Kolhatkar < >>> > >> > > > >> > [email protected] >>> > >> > > > >> > > wrote: >>> > >> > > > >> > >>> > >> > > > >> > > Hi All, >>> > >> > > > >> > > >>> > >> > > > >> > > We're planning to add a utility in malhar-library for >>> > >> > generating a >>> > >> > > > >> POJO >>> > >> > > > >> > > class on the fly from given JSON schema. >>> > >> > > > >> > > >>> > >> > > > >> > > Use case is where the application is provided with >>> schema >>> > and >>> > >> > that >>> > >> > > > >> needs >>> > >> > > > >> > to >>> > >> > > > >> > > be used in one or more operators either as a tuple >>> over the >>> > >> > stream >>> > >> > > > OR >>> > >> > > > >> for >>> > >> > > > >> > > processing. >>> > >> > > > >> > > >>> > >> > > > >> > > General Design: >>> > >> > > > >> > > 1. Utility will be provided with fqcn of the class and >>> > schema >>> > >> > > > >> definition >>> > >> > > > >> > > provided as json. >>> > >> > > > >> > > 2. The schema definition will look like following: >>> > >> > > > >> > > { >>> > >> > > > >> > > "fqcn":"<qualified class name>", >>> > >> > > > >> > > "fields": [ >>> > >> > > > >> > > { >>> > >> > > > >> > > "name":"field1", >>> > >> > > > >> > > "type":"long" >>> > >> > > > >> > > }, >>> > >> > > > >> > > { >>> > >> > > > >> > > "name":"field2", >>> > >> > > > >> > > "type": "string" >>> > >> > > > >> > > } >>> > >> > > > >> > > ] >>> > >> > > > >> > > } >>> > >> > > > >> > > 3. Supported types identified in "type" JSON field are: >>> > >> > > > >> > > boolean, char, byte, short, int, float, long, >>> double >>> > >> > > > >> > > 4. The output of this utility will be a generated >>> .class >>> > >> file in >>> > >> > > the >>> > >> > > > >> form >>> > >> > > > >> > > of FSDataOutputStream. >>> > >> > > > >> > > 5. Xbean asm5 library will be used for this. >>> > >> > > > >> > > 6. Following methods will be added to the generated >>> class: >>> > >> > > > >> > > a. Getter/Setter methods for given fields. >>> > >> > > > >> > > b. simple toString - Generate string equivalent >>> for all >>> > >> the >>> > >> > > > fields >>> > >> > > > >> > > c. hashCode method - calculate the overall hashCode >>> > using >>> > >> > > > >> individual >>> > >> > > > >> > > field hashcodes, similar to how String.hashCode >>> generates >>> > >> > hashCode >>> > >> > > > of >>> > >> > > > >> > > string using chars in it. >>> > >> > > > >> > > d. equals method - Similar to how String.equals >>> method >>> > >> has >>> > >> > > done >>> > >> > > > it >>> > >> > > > >> > > using individual char. >>> > >> > > > >> > > >>> > >> > > > >> > > >>> > >> > > > >> > > Questions I have about the functionality: >>> > >> > > > >> > > 1. Should the utility also support nested pojo >>> definition >>> > via >>> > >> > > schema >>> > >> > > > >> > json? >>> > >> > > > >> > > In such case the field definition can look like >>> > >> following: >>> > >> > > > >> > > { "name":"nestedField", "type":"<fqcn of nested >>> > class>", >>> > >> > > > >> > > "fields":[....] } >>> > >> > > > >> > > The nested class will be provided >>> > >> > > > >> > > >>> > >> > > > >> > > 2. Should the utility also support giving out the >>> > >> > JarOutputStream >>> > >> > > > >> which >>> > >> > > > >> > > will have a generated jar? >>> > >> > > > >> > > >>> > >> > > > >> > > Please let me know your thoughts. >>> > >> > > > >> > > >>> > >> > > > >> > > Thanks, >>> > >> > > > >> > > Chinmay. >>> > >> > > > >> > > >>> > >> > > > >> > >>> > >> > > > >> >>> > >> > > > > >>> > >> > > > > >>> > >> > > > >>> > >> > > >>> > >> > >>> > >> >>> > > >>> > >>> >> >
