@Bertty 3pm CET time right? :p On Tue, Jan 4, 2022 at 1:09 PM kamalesh palanisamy <[email protected]> wrote:
> That’s perfect. I am looking forward to it. > > On Tue, Jan 4, 2022 at 3:55 AM Bertty Contreras <[email protected]> > wrote: > > > Hi Kamalesh, > > > > I sent the invitation on Friday at 3pm :D. > > > > If you have any questions let me know. > > > > Best regards, > > Bertty > > > > On Tue, Jan 4, 2022 at 1:00 AM kamalesh palanisamy < > [email protected]> > > wrote: > > > >> Okay that sounds perfect. Thank you! > >> > >> On Mon, Jan 3, 2022 at 4:52 AM Bertty Contreras <[email protected]> > >> wrote: > >> > >>> So nice, I will organize my schedule for it and i will come back to you > >>> with options. Mean while I will collecting all the designs and other > >>> elements that are done and could help you with the implementation of > the > >>> new feature ;), > >>> > >>> Best regards, > >>> Bertty > >>> > >>> On Mon 3. Jan 2022 at 04:41, kamalesh palanisamy < > [email protected]> > >>> wrote: > >>> > >>>> Hi, > >>>> Thank you for the explanation. Yes, I feel it would be better if we > >>>> could discuss it so that everything is clear. I am free from > >>>> Wednesday-Saturday anytime after 3 PM Germany time. You can select > which > >>>> every day suits your schedule best during this time. > >>>> > >>>> Thanks, > >>>> Kamalesh P > >>>> > >>>> > >>>> On Sun, Jan 2, 2022 at 6:28 PM Bertty Contreras <[email protected]> > >>>> wrote: > >>>> > >>>>> The main is idea of wayang is to provide a layer that pick the best > >>>>> combination of platform to process a query, you can see the details > on the > >>>>> paper rheemix[1] > >>>>> > >>>>> Then providing a SQL-API will allow to transform a query into > >>>>> different operators of wayang that will allow optimization with > platform > >>>>> that only have SQL like postgres with platforms that don’t SQL > lenguaje > >>>>> like giraph. > >>>>> > >>>>> The idea to use calcite, is coming from the intermediate > >>>>> representation that calcite generates that will allows us to create > the > >>>>> wayang plan with an “udf” that are translateble again to SQL or > >>>>> translatable to a executable code that can be executed by flink, as > an > >>>>> example. > >>>>> > >>>>> Imagen the query that it said something like: > >>>>> > >>>>> Select A.a,A.b,A.c from A join A.a = X.a …. > >>>>> > >>>>> Then X(10TB) is on HDFS and A(100MB) is on postgres, then the plan to > >>>>> execute will something like: > >>>>> > >>>>> Select A.a from A(1MB), this file is small then you can do broadcast > >>>>> and filter using flink. > >>>>> > >>>>> Then the join results are just 2 records, the wayang will perform the > >>>>> query on postgres using the 2 record as condition. > >>>>> > >>>>> But also could occurs that the join answer is 1TB, in that case, the > >>>>> data of postgres will be move to HDFS and the all the rest of the > process > >>>>> will be on using flink. > >>>>> > >>>>> Currently the optimizer is taking the decision of what platform will > >>>>> be used depending on the amount of data to process and data > movement. Then > >>>>> the SQL-API will provide an way of “freedom” the decisions because > we will > >>>>> have all the intermediate representation to performs changes. > >>>>> > >>>>> After we have the SQL-API we will be adding platforms that just > >>>>> support and SQL ;), as you said. > >>>>> > >>>>> The idea of using the intermediate representation it maybe sound > weird > >>>>> to you, but we can have a meeting to explain you better, then you can > >>>>> understand better the full concept and also give us your feedback, > let me > >>>>> if hyou are available and when and I will freedom my schedule for it > ;). > >>>>> I’m in Germany just to you figure if we have some timezone > differences ;). > >>>>> > >>>>> Best regards, > >>>>> Bertty > >>>>> > >>>>> [1] > >>>>> https://wayang.apache.org/assets/pdf/paper/journal_vldb.pdf > >>>>> > >>>>> > >>>>> On Sun 2. Jan 2022 at 17:43, kamalesh palanisamy < > >>>>> [email protected]> wrote: > >>>>> > >>>>>> Hi Bertty, > >>>>>> Thank you for the information! I would love to work on adding the > SQL > >>>>>> API for Wayang. Basically, now I need to add a new platform for the > >>>>>> wayang-platforms that supports SQL through apache calcite? Am I > right? > >>>>>> Please do correct me if I am wrong. > >>>>>> > >>>>>> Thanks, > >>>>>> Kamalesh P > >>>>>> > >>>>>> > >>>>>> On Sun, Jan 2, 2022 at 3:36 AM Bertty Contreras < > [email protected]> > >>>>>> wrote: > >>>>>> > >>>>>>> Hi Kamalesh, > >>>>>>> > >>>>>>> Currently, Apache Wayang(Incubating) has the issues listed in Jira > >>>>>>> [1]. One feature that the community didn't have time to work on is > the SQL > >>>>>>> API for Apache Wayang(Incubating) [2]; the main idea is to use > Apache > >>>>>>> Calcite [3] as the parser of the SQL and then do something like > Spark > >>>>>>> adapter of calcite [4]. If you want to contribute to this feature, > it will > >>>>>>> be so awesome :D. > >>>>>>> > >>>>>>> If you found another issue interesting, let me know, or even if you > >>>>>>> have some idea of a feature will be so awesome too :D > >>>>>>> > >>>>>>> Best regards, > >>>>>>> Bertty > >>>>>>> > >>>>>>> [1] https://issues.apache.org/jira/projects/WAYANG > >>>>>>> [2] > >>>>>>> > https://issues.apache.org/jira/projects/WAYANG/issues/WAYANG-25?filter=allopenissues > >>>>>>> [3] https://calcite.apache.org > >>>>>>> [4] https://github.com/apache/calcite/tree/master/spark > >>>>>>> > >>>>>>> On Sun, Jan 2, 2022 at 6:50 AM kamalesh palanisamy < > >>>>>>> [email protected]> wrote: > >>>>>>> > >>>>>>>> Hi, > >>>>>>>> My name is Kamalesh and I am currently looking to contribute to > the > >>>>>>>> project, but I couldn't find any proper issues. Can you help me > >>>>>>>> with any > >>>>>>>> features you would like me to contribute to?. Thanks! > >>>>>>>> Thanks, > >>>>>>>> Kamalesh P > >>>>>>>> > >>>>>>> -- > >> Thanks, > >> Kamalesh P > >> > > -- > Thanks, > Kamalesh P >
