Re: Re: [DISCUSS] Support JSON functions in Flink SQL
y supports the parser part and the runtime > part. > >> > > > > >> We only need to integrate it in Flink and add good > coverage > >> > > tests. > >> > > > > >> 1) However, I think we should also design the corresponding > >> JSON > >> > > > > Functions > >> > > > > >> API for Table API which is very important. > >> > > > > >> I don't have a clear idea about how to support all the > JSON > >> > > > Function > >> > > > > >> syntax in Table API. And this may need more discussions. > >> > > > > >> 2) IMO, it deserves a FLIP (especially for the Table API > part). > >> > You > >> > > > can > >> > > > > >> follow the FLIP process [1] to start a FLIP proposal. > >> > > > > >> 3) I think we only need to implement it in blink planner as > we > >> are > >> > > > going > >> > > > > >> to > >> > > > > >> deprecate old planner. > >> > > > > >>So could you update the implementation section in the doc > >> > because > >> > > > the > >> > > > > >> implementation in blink planner should be different. > >> > > > > >> 4) It would be better to have an implementation plan to > >> priority > >> > the > >> > > > > >> sub-tasks. > >> > > > > >> From my point of view, JSON_VALUE is the most important > and > >> > > > > JSON_TABLE > >> > > > > >> gets the least priority. > >> > > > > >> > >> > > > > >> I also left some comments in the google doc. > >> > > > > >> > >> > > > > >> Hi @JingsongLee , > >> > > > > >> > >> > > > > >> I think we don't need to wait for FLIP-51. As we don't have a > >> > clear > >> > > > > >> progress of FLIP-51. > >> > > > > >> And as far as I know, it will add a few of PlannerExpressions > >> > which > >> > > > can > >> > > > > be > >> > > > > >> refactored easily during FLIP-51. > >> > > > > >> > >> > > > > >> > >> > > > > >> Cheers, > >> > > > > >> Jark > >> > > > > >> > >> > > > > >> [1]: > >> > > > > >> > >> > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals > >> > > > > >> > >> > > > > >> > >> > > > > >> > >> > > > > >> On Thu, 5 Sep 2019 at 19:29, vino yang < > yanghua1...@gmail.com> > >> > > wrote: > >> > > > > >> > >> > > > > >> > +1 to have JSON functions in Flink SQL > >> > > > > >> > > >> > > > > >> > JingsongLee 于2019年9月5日周四 > >> > > > 下午4:46写道： > >> > > > > >> > > >> > > > > >> > > +1 > >> > > > > >> > > Nice document. I think it is easier to do after > expression > >> > > > > >> reworking[1]. > >> > > > > >> > > By the way, which planner do you want to start? > >> > > > > >> > > > >> > > > > >> > > [1] > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design > >> > > > > >> > > > >> > > > > >> > > Best, > >> > > > > >> > > Jingsong Lee > >> > > > > >> > > > >> > > > > >> > > > >> > > > > >
Re: Re: [DISCUSS] Support JSON functions in Flink SQL
t). >> > You >> > > > can >> > > > > >> follow the FLIP process [1] to start a FLIP proposal. >> > > > > >> 3) I think we only need to implement it in blink planner as we >> are >> > > > going >> > > > > >> to >> > > > > >> deprecate old planner. >> > > > > >>So could you update the implementation section in the doc >> > because >> > > > the >> > > > > >> implementation in blink planner should be different. >> > > > > >> 4) It would be better to have an implementation plan to >> priority >> > the >> > > > > >> sub-tasks. >> > > > > >> From my point of view, JSON_VALUE is the most important and >> > > > > JSON_TABLE >> > > > > >> gets the least priority. >> > > > > >> >> > > > > >> I also left some comments in the google doc. >> > > > > >> >> > > > > >> Hi @JingsongLee , >> > > > > >> >> > > > > >> I think we don't need to wait for FLIP-51. As we don't have a >> > clear >> > > > > >> progress of FLIP-51. >> > > > > >> And as far as I know, it will add a few of PlannerExpressions >> > which >> > > > can >> > > > > be >> > > > > >> refactored easily during FLIP-51. >> > > > > >> >> > > > > >> >> > > > > >> Cheers, >> > > > > >> Jark >> > > > > >> >> > > > > >> [1]: >> > > > > >> >> > > > > >> >> > > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> On Thu, 5 Sep 2019 at 19:29, vino yang >> > > wrote: >> > > > > >> >> > > > > >> > +1 to have JSON functions in Flink SQL >> > > > > >> > >> > > > > >> > JingsongLee 于2019年9月5日周四 >> > > > 下午4:46写道： >> > > > > >> > >> > > > > >> > > +1 >> > > > > >> > > Nice document. I think it is easier to do after expression >> > > > > >> reworking[1]. >> > > > > >> > > By the way, which planner do you want to start? >> > > > > >> > > >> > > > > >> > > [1] >> > > > > >> > > >> > > > > >> > >> > > > > >> >> > > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design >> > > > > >> > > >> > > > > >> > > Best, >> > > > > >> > > Jingsong Lee >> > > > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > -- >> > > > > >> > > From:TANG Wen-hui >> > > > > >> > > Send Time:2019年9月5日(星期四) 14:36 >> > > > > >> > > To:dev >> > > > > >> > > Subject:Re: Re: [DISCUSS] Support JSON functions in Flink >> SQL >> > > > > >> > > >> > > > > >> > > +1 >> > > > > >> > > I have done similar work before. >> > > > > >> > > Looking forward to discussing this feature. >> > > > > >> > > >> > > > > >> > > Best >> > > > > >> > > wenhui >> > > > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > > >> > > winifred.wenhui.t...@gmail.com >> > > > > >> > > >> > > > > >> > > From: Kurt Young >> > > > > >> > > Date: 2019-09-05 14:00 >&
Re: Re: [DISCUSS] Support JSON functions in Flink SQL
> > the > > > > > >> sub-tasks. > > > > > >> From my point of view, JSON_VALUE is the most important and > > > > > JSON_TABLE > > > > > >> gets the least priority. > > > > > >> > > > > > >> I also left some comments in the google doc. > > > > > >> > > > > > >> Hi @JingsongLee , > > > > > >> > > > > > >> I think we don't need to wait for FLIP-51. As we don't have a > > clear > > > > > >> progress of FLIP-51. > > > > > >> And as far as I know, it will add a few of PlannerExpressions > > which > > > > can > > > > > be > > > > > >> refactored easily during FLIP-51. > > > > > >> > > > > > >> > > > > > >> Cheers, > > > > > >> Jark > > > > > >> > > > > > >> [1]: > > > > > >> > > > > > >> > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals > > > > > >> > > > > > >> > > > > > >> > > > > > >> On Thu, 5 Sep 2019 at 19:29, vino yang > > > wrote: > > > > > >> > > > > > >> > +1 to have JSON functions in Flink SQL > > > > > >> > > > > > > >> > JingsongLee 于2019年9月5日周四 > > > > 下午4:46写道： > > > > > >> > > > > > > >> > > +1 > > > > > >> > > Nice document. I think it is easier to do after expression > > > > > >> reworking[1]. > > > > > >> > > By the way, which planner do you want to start? > > > > > >> > > > > > > > >> > > [1] > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design > > > > > >> > > > > > > > >> > > Best, > > > > > >> > > Jingsong Lee > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > -- > > > > > >> > > From:TANG Wen-hui > > > > > >> > > Send Time:2019年9月5日(星期四) 14:36 > > > > > >> > > To:dev > > > > > >> > > Subject:Re: Re: [DISCUSS] Support JSON functions in Flink > SQL > > > > > >> > > > > > > > >> > > +1 > > > > > >> > > I have done similar work before. > > > > > >> > > Looking forward to discussing this feature. > > > > > >> > > > > > > > >> > > Best > > > > > >> > > wenhui > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > winifred.wenhui.t...@gmail.com > > > > > >> > > > > > > > >> > > From: Kurt Young > > > > > >> > > Date: 2019-09-05 14:00 > > > > > >> > > To: dev > > > > > >> > > CC: Anyang Hu > > > > > >> > > Subject: Re: [DISCUSS] Support JSON functions in Flink SQL > > > > > >> > > +1 to add JSON support to Flink. We also see lots of > > > requirements > > > > > for > > > > > >> > JSON > > > > > >> > > related functions in our internal platform. Since these are > > > > already > > > > > >> SQL > > > > > >> > > standard, I think it's a good time to add them to Flink. > > > > > >> > > > > > > > >> > > Best, > > > > > >> > > Kurt > > > > > >> > > > > > > > >> > > > > > > > >> > > On Thu, Sep 5, 2019 at 10:37 AM Qi Luo > > > > wrote: > > > >
Re: Re: [DISCUSS] Support JSON functions in Flink SQL
Hi Jark, Thank you very much，I will improve this document as soon as possible. My confluence username is ForwardXu. Best, Forward Jark Wu 于2019年12月2日周一 上午11:56写道： > Hi Forward, > > Sorry for the late reply. > As I said before, it would be better to include the JSON functions API for > Table API. > For example, what the "json_exist", "json_value", "json_query", etc... > functions in Table API looks like. > And it would be better to follow FLIP template [1] which include "Public > Interface" (brief list of public interface), and "Proposed changes" > (detailed proposal). > > Once the design doc looks good, you can update it to FLIP and start a vote. > > Btw, what's your username in confluence? > > > Best, > Jark > > > [1]: https://cwiki.apache.org/confluence/display/FLINK/FLIP+Template > > > On Mon, 2 Dec 2019 at 11:44, Forward Xu wrote: > > > Hi Jingsong Lee : > > > > Thank you very much, I need to apply for FLIP permission. Do I need to > > create a FLIP for this? > > > > Best, > > Forward > > > > Jingsong Li 于2019年12月2日周一 上午11:40写道： > > > > > Hi Forward: > > > > > > Document looks good to me. > > > I think you can just start doing this. > > > They all work very independently, so I don't think there's any obvious > > > blocking. > > > > > > Best, > > > Jingsong Lee > > > > > > On Sat, Nov 30, 2019 at 10:59 AM Forward Xu > > > wrote: > > > > > > > Hi everyone, It's been a long time since I started this discussion. > Do > > > you > > > > have anything to add and improve? > > > > Best, > > > > Forward > > > > > > > > Forward Xu 于2019年9月22日周日 下午6:30写道： > > > > > > > > > Hi Jack, > > > > > Thank you very much for your reply, google doc I have updated, and > > some > > > > of > > > > > your questions I replied. > > > > > In addition, I want to apply for Flip permissions for this > purpose. > > > > > > > > > > Best, > > > > > Forward > > > > > > > > > > Jark Wu 于2019年9月20日周五 下午9:53写道： > > > > > > > > > >> Hi Forward, > > > > >> > > > > >> Sorry for the late reply. I have went through the design doc and I > > > think > > > > >> it > > > > >> is very nice. > > > > >> > > > > >> Here are my thoughts and suggestions: > > > > >> > > > > >> 0) I think support JSON functions in SQL is not complicated. > Because > > > > >> Calcite already supports the parser part and the runtime part. > > > > >> We only need to integrate it in Flink and add good coverage > > tests. > > > > >> 1) However, I think we should also design the corresponding JSON > > > > Functions > > > > >> API for Table API which is very important. > > > > >> I don't have a clear idea about how to support all the JSON > > > Function > > > > >> syntax in Table API. And this may need more discussions. > > > > >> 2) IMO, it deserves a FLIP (especially for the Table API part). > You > > > can > > > > >> follow the FLIP process [1] to start a FLIP proposal. > > > > >> 3) I think we only need to implement it in blink planner as we are > > > going > > > > >> to > > > > >> deprecate old planner. > > > > >>So could you update the implementation section in the doc > because > > > the > > > > >> implementation in blink planner should be different. > > > > >> 4) It would be better to have an implementation plan to priority > the > > > > >> sub-tasks. > > > > >> From my point of view, JSON_VALUE is the most important and > > > > JSON_TABLE > > > > >> gets the least priority. > > > > >> > > > > >> I also left some comments in the google doc. > > > > >> > > > > >> Hi @JingsongLee , > > > > >> > > > > >> I think we don't need to wait for FLIP-51. As we don't have a > clear > > > > >> progress of FLIP-51. > > > > >> And as far as I know, it will add a few of PlannerExpressions > which > > >
Re: Re: [DISCUSS] Support JSON functions in Flink SQL
Hi Forward, Sorry for the late reply. As I said before, it would be better to include the JSON functions API for Table API. For example, what the "json_exist", "json_value", "json_query", etc... functions in Table API looks like. And it would be better to follow FLIP template [1] which include "Public Interface" (brief list of public interface), and "Proposed changes" (detailed proposal). Once the design doc looks good, you can update it to FLIP and start a vote. Btw, what's your username in confluence? Best, Jark [1]: https://cwiki.apache.org/confluence/display/FLINK/FLIP+Template On Mon, 2 Dec 2019 at 11:44, Forward Xu wrote: > Hi Jingsong Lee : > > Thank you very much, I need to apply for FLIP permission. Do I need to > create a FLIP for this? > > Best, > Forward > > Jingsong Li 于2019年12月2日周一 上午11:40写道： > > > Hi Forward: > > > > Document looks good to me. > > I think you can just start doing this. > > They all work very independently, so I don't think there's any obvious > > blocking. > > > > Best, > > Jingsong Lee > > > > On Sat, Nov 30, 2019 at 10:59 AM Forward Xu > > wrote: > > > > > Hi everyone, It's been a long time since I started this discussion. Do > > you > > > have anything to add and improve? > > > Best, > > > Forward > > > > > > Forward Xu 于2019年9月22日周日 下午6:30写道： > > > > > > > Hi Jack, > > > > Thank you very much for your reply, google doc I have updated, and > some > > > of > > > > your questions I replied. > > > > In addition, I want to apply for Flip permissions for this purpose. > > > > > > > > Best, > > > > Forward > > > > > > > > Jark Wu 于2019年9月20日周五 下午9:53写道： > > > > > > > >> Hi Forward, > > > >> > > > >> Sorry for the late reply. I have went through the design doc and I > > think > > > >> it > > > >> is very nice. > > > >> > > > >> Here are my thoughts and suggestions: > > > >> > > > >> 0) I think support JSON functions in SQL is not complicated. Because > > > >> Calcite already supports the parser part and the runtime part. > > > >> We only need to integrate it in Flink and add good coverage > tests. > > > >> 1) However, I think we should also design the corresponding JSON > > > Functions > > > >> API for Table API which is very important. > > > >> I don't have a clear idea about how to support all the JSON > > Function > > > >> syntax in Table API. And this may need more discussions. > > > >> 2) IMO, it deserves a FLIP (especially for the Table API part). You > > can > > > >> follow the FLIP process [1] to start a FLIP proposal. > > > >> 3) I think we only need to implement it in blink planner as we are > > going > > > >> to > > > >> deprecate old planner. > > > >>So could you update the implementation section in the doc because > > the > > > >> implementation in blink planner should be different. > > > >> 4) It would be better to have an implementation plan to priority the > > > >> sub-tasks. > > > >> From my point of view, JSON_VALUE is the most important and > > > JSON_TABLE > > > >> gets the least priority. > > > >> > > > >> I also left some comments in the google doc. > > > >> > > > >> Hi @JingsongLee , > > > >> > > > >> I think we don't need to wait for FLIP-51. As we don't have a clear > > > >> progress of FLIP-51. > > > >> And as far as I know, it will add a few of PlannerExpressions which > > can > > > be > > > >> refactored easily during FLIP-51. > > > >> > > > >> > > > >> Cheers, > > > >> Jark > > > >> > > > >> [1]: > > > >> > > > >> > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals > > > >> > > > >> > > > >> > > > >> On Thu, 5 Sep 2019 at 19:29, vino yang > wrote: > > > >> > > > >> > +1 to have JSON functions in Flink SQL > > > >> > > > > >> > JingsongLee 于2019年9月5日周四 > > 下午4:46写道： > >
Re: Re: [DISCUSS] Support JSON functions in Flink SQL
Hi Jingsong Lee : Thank you very much, I need to apply for FLIP permission. Do I need to create a FLIP for this? Best, Forward Jingsong Li 于2019年12月2日周一 上午11:40写道： > Hi Forward: > > Document looks good to me. > I think you can just start doing this. > They all work very independently, so I don't think there's any obvious > blocking. > > Best, > Jingsong Lee > > On Sat, Nov 30, 2019 at 10:59 AM Forward Xu > wrote: > > > Hi everyone, It's been a long time since I started this discussion. Do > you > > have anything to add and improve? > > Best, > > Forward > > > > Forward Xu 于2019年9月22日周日 下午6:30写道： > > > > > Hi Jack, > > > Thank you very much for your reply, google doc I have updated, and some > > of > > > your questions I replied. > > > In addition, I want to apply for Flip permissions for this purpose. > > > > > > Best, > > > Forward > > > > > > Jark Wu 于2019年9月20日周五 下午9:53写道： > > > > > >> Hi Forward, > > >> > > >> Sorry for the late reply. I have went through the design doc and I > think > > >> it > > >> is very nice. > > >> > > >> Here are my thoughts and suggestions: > > >> > > >> 0) I think support JSON functions in SQL is not complicated. Because > > >> Calcite already supports the parser part and the runtime part. > > >> We only need to integrate it in Flink and add good coverage tests. > > >> 1) However, I think we should also design the corresponding JSON > > Functions > > >> API for Table API which is very important. > > >> I don't have a clear idea about how to support all the JSON > Function > > >> syntax in Table API. And this may need more discussions. > > >> 2) IMO, it deserves a FLIP (especially for the Table API part). You > can > > >> follow the FLIP process [1] to start a FLIP proposal. > > >> 3) I think we only need to implement it in blink planner as we are > going > > >> to > > >> deprecate old planner. > > >>So could you update the implementation section in the doc because > the > > >> implementation in blink planner should be different. > > >> 4) It would be better to have an implementation plan to priority the > > >> sub-tasks. > > >> From my point of view, JSON_VALUE is the most important and > > JSON_TABLE > > >> gets the least priority. > > >> > > >> I also left some comments in the google doc. > > >> > > >> Hi @JingsongLee , > > >> > > >> I think we don't need to wait for FLIP-51. As we don't have a clear > > >> progress of FLIP-51. > > >> And as far as I know, it will add a few of PlannerExpressions which > can > > be > > >> refactored easily during FLIP-51. > > >> > > >> > > >> Cheers, > > >> Jark > > >> > > >> [1]: > > >> > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals > > >> > > >> > > >> > > >> On Thu, 5 Sep 2019 at 19:29, vino yang wrote: > > >> > > >> > +1 to have JSON functions in Flink SQL > > >> > > > >> > JingsongLee 于2019年9月5日周四 > 下午4:46写道： > > >> > > > >> > > +1 > > >> > > Nice document. I think it is easier to do after expression > > >> reworking[1]. > > >> > > By the way, which planner do you want to start? > > >> > > > > >> > > [1] > > >> > > > > >> > > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design > > >> > > > > >> > > Best, > > >> > > Jingsong Lee > > >> > > > > >> > > > > >> > > -- > > >> > > From:TANG Wen-hui > > >> > > Send Time:2019年9月5日(星期四) 14:36 > > >> > > To:dev > > >> > > Subject:Re: Re: [DISCUSS] Support JSON functions in Flink SQL > > >> > > > > >> > > +1 > > >> > > I have done similar work before. > > >> > > Looking forward to discussing this feature. > > >> > > > > >>
Re: Re: [DISCUSS] Support JSON functions in Flink SQL
Hi Forward: Document looks good to me. I think you can just start doing this. They all work very independently, so I don't think there's any obvious blocking. Best, Jingsong Lee On Sat, Nov 30, 2019 at 10:59 AM Forward Xu wrote: > Hi everyone, It's been a long time since I started this discussion. Do you > have anything to add and improve? > Best, > Forward > > Forward Xu 于2019年9月22日周日 下午6:30写道： > > > Hi Jack, > > Thank you very much for your reply, google doc I have updated, and some > of > > your questions I replied. > > In addition, I want to apply for Flip permissions for this purpose. > > > > Best, > > Forward > > > > Jark Wu 于2019年9月20日周五 下午9:53写道： > > > >> Hi Forward, > >> > >> Sorry for the late reply. I have went through the design doc and I think > >> it > >> is very nice. > >> > >> Here are my thoughts and suggestions: > >> > >> 0) I think support JSON functions in SQL is not complicated. Because > >> Calcite already supports the parser part and the runtime part. > >> We only need to integrate it in Flink and add good coverage tests. > >> 1) However, I think we should also design the corresponding JSON > Functions > >> API for Table API which is very important. > >> I don't have a clear idea about how to support all the JSON Function > >> syntax in Table API. And this may need more discussions. > >> 2) IMO, it deserves a FLIP (especially for the Table API part). You can > >> follow the FLIP process [1] to start a FLIP proposal. > >> 3) I think we only need to implement it in blink planner as we are going > >> to > >> deprecate old planner. > >>So could you update the implementation section in the doc because the > >> implementation in blink planner should be different. > >> 4) It would be better to have an implementation plan to priority the > >> sub-tasks. > >> From my point of view, JSON_VALUE is the most important and > JSON_TABLE > >> gets the least priority. > >> > >> I also left some comments in the google doc. > >> > >> Hi @JingsongLee , > >> > >> I think we don't need to wait for FLIP-51. As we don't have a clear > >> progress of FLIP-51. > >> And as far as I know, it will add a few of PlannerExpressions which can > be > >> refactored easily during FLIP-51. > >> > >> > >> Cheers, > >> Jark > >> > >> [1]: > >> > >> > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals > >> > >> > >> > >> On Thu, 5 Sep 2019 at 19:29, vino yang wrote: > >> > >> > +1 to have JSON functions in Flink SQL > >> > > >> > JingsongLee 于2019年9月5日周四 下午4:46写道： > >> > > >> > > +1 > >> > > Nice document. I think it is easier to do after expression > >> reworking[1]. > >> > > By the way, which planner do you want to start? > >> > > > >> > > [1] > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design > >> > > > >> > > Best, > >> > > Jingsong Lee > >> > > > >> > > > >> > > -- > >> > > From:TANG Wen-hui > >> > > Send Time:2019年9月5日(星期四) 14:36 > >> > > To:dev > >> > > Subject:Re: Re: [DISCUSS] Support JSON functions in Flink SQL > >> > > > >> > > +1 > >> > > I have done similar work before. > >> > > Looking forward to discussing this feature. > >> > > > >> > > Best > >> > > wenhui > >> > > > >> > > > >> > > > >> > > winifred.wenhui.t...@gmail.com > >> > > > >> > > From: Kurt Young > >> > > Date: 2019-09-05 14:00 > >> > > To: dev > >> > > CC: Anyang Hu > >> > > Subject: Re: [DISCUSS] Support JSON functions in Flink SQL > >> > > +1 to add JSON support to Flink. We also see lots of requirements > for > >> > JSON > >> > > related functions in our internal platform. Since these are already > >> SQL > >> > > standard, I think it's a good time to add them to Flink.
Re: Re: [DISCUSS] Support JSON functions in Flink SQL
Hi everyone, It's been a long time since I started this discussion. Do you have anything to add and improve? Best, Forward Forward Xu 于2019年9月22日周日 下午6:30写道： > Hi Jack, > Thank you very much for your reply, google doc I have updated, and some of > your questions I replied. > In addition, I want to apply for Flip permissions for this purpose. > > Best, > Forward > > Jark Wu 于2019年9月20日周五 下午9:53写道： > >> Hi Forward, >> >> Sorry for the late reply. I have went through the design doc and I think >> it >> is very nice. >> >> Here are my thoughts and suggestions: >> >> 0) I think support JSON functions in SQL is not complicated. Because >> Calcite already supports the parser part and the runtime part. >> We only need to integrate it in Flink and add good coverage tests. >> 1) However, I think we should also design the corresponding JSON Functions >> API for Table API which is very important. >> I don't have a clear idea about how to support all the JSON Function >> syntax in Table API. And this may need more discussions. >> 2) IMO, it deserves a FLIP (especially for the Table API part). You can >> follow the FLIP process [1] to start a FLIP proposal. >> 3) I think we only need to implement it in blink planner as we are going >> to >> deprecate old planner. >>So could you update the implementation section in the doc because the >> implementation in blink planner should be different. >> 4) It would be better to have an implementation plan to priority the >> sub-tasks. >> From my point of view, JSON_VALUE is the most important and JSON_TABLE >> gets the least priority. >> >> I also left some comments in the google doc. >> >> Hi @JingsongLee , >> >> I think we don't need to wait for FLIP-51. As we don't have a clear >> progress of FLIP-51. >> And as far as I know, it will add a few of PlannerExpressions which can be >> refactored easily during FLIP-51. >> >> >> Cheers, >> Jark >> >> [1]: >> >> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals >> >> >> >> On Thu, 5 Sep 2019 at 19:29, vino yang wrote: >> >> > +1 to have JSON functions in Flink SQL >> > >> > JingsongLee 于2019年9月5日周四 下午4:46写道： >> > >> > > +1 >> > > Nice document. I think it is easier to do after expression >> reworking[1]. >> > > By the way, which planner do you want to start? >> > > >> > > [1] >> > > >> > >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design >> > > >> > > Best, >> > > Jingsong Lee >> > > >> > > >> > > ---------------------- >> > > From:TANG Wen-hui >> > > Send Time:2019年9月5日(星期四) 14:36 >> > > To:dev >> > > Subject:Re: Re: [DISCUSS] Support JSON functions in Flink SQL >> > > >> > > +1 >> > > I have done similar work before. >> > > Looking forward to discussing this feature. >> > > >> > > Best >> > > wenhui >> > > >> > > >> > > >> > > winifred.wenhui.t...@gmail.com >> > > >> > > From: Kurt Young >> > > Date: 2019-09-05 14:00 >> > > To: dev >> > > CC: Anyang Hu >> > > Subject: Re: [DISCUSS] Support JSON functions in Flink SQL >> > > +1 to add JSON support to Flink. We also see lots of requirements for >> > JSON >> > > related functions in our internal platform. Since these are already >> SQL >> > > standard, I think it's a good time to add them to Flink. >> > > >> > > Best, >> > > Kurt >> > > >> > > >> > > On Thu, Sep 5, 2019 at 10:37 AM Qi Luo wrote: >> > > >> > > > We also see strong demands from our SQL users for JSON/Date related >> > > > functions. >> > > > >> > > > Also +Anyang Hu >> > > > >> > > > On Wed, Sep 4, 2019 at 9:51 PM Jark Wu wrote: >> > > > >> > > > > Hi Forward, >> > > > > >> > > > > Thanks for bringing this discussion and preparing the nice design. >> > > > > I think it's nice to have the JSON functions in the next release. >> > > > > We have received some requirements
Re: Re: [DISCUSS] Support JSON functions in Flink SQL
Hi Jack, Thank you very much for your reply, google doc I have updated, and some of your questions I replied. In addition, I want to apply for Flip permissions for this purpose. Best, Forward Jark Wu 于2019年9月20日周五 下午9:53写道： > Hi Forward, > > Sorry for the late reply. I have went through the design doc and I think it > is very nice. > > Here are my thoughts and suggestions: > > 0) I think support JSON functions in SQL is not complicated. Because > Calcite already supports the parser part and the runtime part. > We only need to integrate it in Flink and add good coverage tests. > 1) However, I think we should also design the corresponding JSON Functions > API for Table API which is very important. > I don't have a clear idea about how to support all the JSON Function > syntax in Table API. And this may need more discussions. > 2) IMO, it deserves a FLIP (especially for the Table API part). You can > follow the FLIP process [1] to start a FLIP proposal. > 3) I think we only need to implement it in blink planner as we are going to > deprecate old planner. >So could you update the implementation section in the doc because the > implementation in blink planner should be different. > 4) It would be better to have an implementation plan to priority the > sub-tasks. > From my point of view, JSON_VALUE is the most important and JSON_TABLE > gets the least priority. > > I also left some comments in the google doc. > > Hi @JingsongLee , > > I think we don't need to wait for FLIP-51. As we don't have a clear > progress of FLIP-51. > And as far as I know, it will add a few of PlannerExpressions which can be > refactored easily during FLIP-51. > > > Cheers, > Jark > > [1]: > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals > > > > On Thu, 5 Sep 2019 at 19:29, vino yang wrote: > > > +1 to have JSON functions in Flink SQL > > > > JingsongLee 于2019年9月5日周四 下午4:46写道： > > > > > +1 > > > Nice document. I think it is easier to do after expression > reworking[1]. > > > By the way, which planner do you want to start? > > > > > > [1] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design > > > > > > Best, > > > Jingsong Lee > > > > > > > > > -- > > > From:TANG Wen-hui > > > Send Time:2019年9月5日(星期四) 14:36 > > > To:dev > > > Subject:Re: Re: [DISCUSS] Support JSON functions in Flink SQL > > > > > > +1 > > > I have done similar work before. > > > Looking forward to discussing this feature. > > > > > > Best > > > wenhui > > > > > > > > > > > > winifred.wenhui.t...@gmail.com > > > > > > From: Kurt Young > > > Date: 2019-09-05 14:00 > > > To: dev > > > CC: Anyang Hu > > > Subject: Re: [DISCUSS] Support JSON functions in Flink SQL > > > +1 to add JSON support to Flink. We also see lots of requirements for > > JSON > > > related functions in our internal platform. Since these are already SQL > > > standard, I think it's a good time to add them to Flink. > > > > > > Best, > > > Kurt > > > > > > > > > On Thu, Sep 5, 2019 at 10:37 AM Qi Luo wrote: > > > > > > > We also see strong demands from our SQL users for JSON/Date related > > > > functions. > > > > > > > > Also +Anyang Hu > > > > > > > > On Wed, Sep 4, 2019 at 9:51 PM Jark Wu wrote: > > > > > > > > > Hi Forward, > > > > > > > > > > Thanks for bringing this discussion and preparing the nice design. > > > > > I think it's nice to have the JSON functions in the next release. > > > > > We have received some requirements for this feature. > > > > > > > > > > I can help to shepherd this JSON functions effort and will leave > > > comments > > > > > in the design doc in the next days. > > > > > > > > > > Hi Danny, > > > > > > > > > > The new introduced JSON functions are from SQL:2016, not from > MySQL. > > > > > So there no JSON type is needed. According to the SQL:2016, the > > > > > representation of JSON data can be "character string" which is also > > > > > the current implementation in Calcite[1]. > > > > > > > >
Re: Re: [DISCUSS] Support JSON functions in Flink SQL
+1 to have JSON functions in Flink SQL JingsongLee 于2019年9月5日周四 下午4:46写道： > +1 > Nice document. I think it is easier to do after expression reworking[1]. > By the way, which planner do you want to start? > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design > > Best, > Jingsong Lee > > > -- > From:TANG Wen-hui > Send Time:2019年9月5日(星期四) 14:36 > To:dev > Subject:Re: Re: [DISCUSS] Support JSON functions in Flink SQL > > +1 > I have done similar work before. > Looking forward to discussing this feature. > > Best > wenhui > > > > winifred.wenhui.t...@gmail.com > > From: Kurt Young > Date: 2019-09-05 14:00 > To: dev > CC: Anyang Hu > Subject: Re: [DISCUSS] Support JSON functions in Flink SQL > +1 to add JSON support to Flink. We also see lots of requirements for JSON > related functions in our internal platform. Since these are already SQL > standard, I think it's a good time to add them to Flink. > > Best, > Kurt > > > On Thu, Sep 5, 2019 at 10:37 AM Qi Luo wrote: > > > We also see strong demands from our SQL users for JSON/Date related > > functions. > > > > Also +Anyang Hu > > > > On Wed, Sep 4, 2019 at 9:51 PM Jark Wu wrote: > > > > > Hi Forward, > > > > > > Thanks for bringing this discussion and preparing the nice design. > > > I think it's nice to have the JSON functions in the next release. > > > We have received some requirements for this feature. > > > > > > I can help to shepherd this JSON functions effort and will leave > comments > > > in the design doc in the next days. > > > > > > Hi Danny, > > > > > > The new introduced JSON functions are from SQL:2016, not from MySQL. > > > So there no JSON type is needed. According to the SQL:2016, the > > > representation of JSON data can be "character string" which is also > > > the current implementation in Calcite[1]. > > > > > > Best, > > > Jark > > > > > > > > > [1]: https://calcite.apache.org/docs/reference.html#json-functions > > > > > > > > > On Wed, 4 Sep 2019 at 21:22, Xu Forward > wrote: > > > > > > > hi Danny Chan ,Thank you very much for your reply, your help can help > > me > > > > further improve this discussion. > > > > Best > > > > forward > > > > > > > > Danny Chan 于2019年9月4日周三 下午8:50写道： > > > > > > > > > Thanks Xu Forward for bring up this topic, I think the JSON > functions > > > are > > > > > very useful especially for those MySQL users. > > > > > > > > > > I saw that you have done some work within the Apache Calcite, > that’s > > a > > > > > good start, but this is one concern from me, Flink doesn’t support > > JSON > > > > > type internal, so how to represent a JSON object in Flink maybe a > key > > > > point > > > > > we need to resolve. In Calcite, we use ANY type to represent as the > > > JSON, > > > > > but I don’t think it is the right way to go, maybe we can have a > > > > discussion > > > > > here. > > > > > > > > > > Best, > > > > > Danny Chan > > > > > 在 2019年9月4日 +0800 PM8:34，Xu Forward ，写道： > > > > > > Hi everybody, > > > > > > > > > > > > I'd like to kick off a discussion on Support JSON functions in > > Flink > > > > SQL. > > > > > > > > > > > > The entire plan is divided into two steps: > > > > > > 1. Implement Support SQL 2016-2017 JSON functions in Flink > SQL[1]. > > > > > > 2. Implement non-Support SQL 2016-2017 JSON functions in Flink > SQL, > > > > such > > > > > as > > > > > > JSON_TYPE in Mysql, JSON_LENGTH, etc. Very useful JSON functions. > > > > > > > > > > > > Would love to hear your thoughts. > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1JfaFYIFOAY8P2pFhOYNCQ9RTzwF4l85_bnTvImOLKMk/edit#heading=h.76mb88ca6yjp > > > > > > > > > > > > Best, > > > > > > ForwardXu > > > > > > > > > > > > > > >
Re: Re: [DISCUSS] Support JSON functions in Flink SQL
+1 Nice document. I think it is easier to do after expression reworking[1]. By the way, which planner do you want to start? [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design Best, Jingsong Lee -- From:TANG Wen-hui Send Time:2019年9月5日(星期四) 14:36 To:dev Subject:Re: Re: [DISCUSS] Support JSON functions in Flink SQL +1 I have done similar work before. Looking forward to discussing this feature. Best wenhui winifred.wenhui.t...@gmail.com From: Kurt Young Date: 2019-09-05 14:00 To: dev CC: Anyang Hu Subject: Re: [DISCUSS] Support JSON functions in Flink SQL +1 to add JSON support to Flink. We also see lots of requirements for JSON related functions in our internal platform. Since these are already SQL standard, I think it's a good time to add them to Flink. Best, Kurt On Thu, Sep 5, 2019 at 10:37 AM Qi Luo wrote: > We also see strong demands from our SQL users for JSON/Date related > functions. > > Also +Anyang Hu > > On Wed, Sep 4, 2019 at 9:51 PM Jark Wu wrote: > > > Hi Forward, > > > > Thanks for bringing this discussion and preparing the nice design. > > I think it's nice to have the JSON functions in the next release. > > We have received some requirements for this feature. > > > > I can help to shepherd this JSON functions effort and will leave comments > > in the design doc in the next days. > > > > Hi Danny, > > > > The new introduced JSON functions are from SQL:2016, not from MySQL. > > So there no JSON type is needed. According to the SQL:2016, the > > representation of JSON data can be "character string" which is also > > the current implementation in Calcite[1]. > > > > Best, > > Jark > > > > > > [1]: https://calcite.apache.org/docs/reference.html#json-functions > > > > > > On Wed, 4 Sep 2019 at 21:22, Xu Forward wrote: > > > > > hi Danny Chan ,Thank you very much for your reply, your help can help > me > > > further improve this discussion. > > > Best > > > forward > > > > > > Danny Chan 于2019年9月4日周三 下午8:50写道： > > > > > > > Thanks Xu Forward for bring up this topic, I think the JSON functions > > are > > > > very useful especially for those MySQL users. > > > > > > > > I saw that you have done some work within the Apache Calcite, that’s > a > > > > good start, but this is one concern from me, Flink doesn’t support > JSON > > > > type internal, so how to represent a JSON object in Flink maybe a key > > > point > > > > we need to resolve. In Calcite, we use ANY type to represent as the > > JSON, > > > > but I don’t think it is the right way to go, maybe we can have a > > > discussion > > > > here. > > > > > > > > Best, > > > > Danny Chan > > > > 在 2019年9月4日 +0800 PM8:34，Xu Forward ，写道： > > > > > Hi everybody, > > > > > > > > > > I'd like to kick off a discussion on Support JSON functions in > Flink > > > SQL. > > > > > > > > > > The entire plan is divided into two steps: > > > > > 1. Implement Support SQL 2016-2017 JSON functions in Flink SQL[1]. > > > > > 2. Implement non-Support SQL 2016-2017 JSON functions in Flink SQL, > > > such > > > > as > > > > > JSON_TYPE in Mysql, JSON_LENGTH, etc. Very useful JSON functions. > > > > > > > > > > Would love to hear your thoughts. > > > > > > > > > > [1] > > > > > > > > > > > > > > > https://docs.google.com/document/d/1JfaFYIFOAY8P2pFhOYNCQ9RTzwF4l85_bnTvImOLKMk/edit#heading=h.76mb88ca6yjp > > > > > > > > > > Best, > > > > > ForwardXu > > > > > > > > > >
Re: Re: [DISCUSS] Support JSON functions in Flink SQL
+1 I have done similar work before. Looking forward to discussing this feature. Best wenhui winifred.wenhui.t...@gmail.com From: Kurt Young Date: 2019-09-05 14:00 To: dev CC: Anyang Hu Subject: Re: [DISCUSS] Support JSON functions in Flink SQL +1 to add JSON support to Flink. We also see lots of requirements for JSON related functions in our internal platform. Since these are already SQL standard, I think it's a good time to add them to Flink. Best, Kurt On Thu, Sep 5, 2019 at 10:37 AM Qi Luo wrote: > We also see strong demands from our SQL users for JSON/Date related > functions. > > Also +Anyang Hu > > On Wed, Sep 4, 2019 at 9:51 PM Jark Wu wrote: > > > Hi Forward, > > > > Thanks for bringing this discussion and preparing the nice design. > > I think it's nice to have the JSON functions in the next release. > > We have received some requirements for this feature. > > > > I can help to shepherd this JSON functions effort and will leave comments > > in the design doc in the next days. > > > > Hi Danny, > > > > The new introduced JSON functions are from SQL:2016, not from MySQL. > > So there no JSON type is needed. According to the SQL:2016, the > > representation of JSON data can be "character string" which is also > > the current implementation in Calcite[1]. > > > > Best, > > Jark > > > > > > [1]: https://calcite.apache.org/docs/reference.html#json-functions > > > > > > On Wed, 4 Sep 2019 at 21:22, Xu Forward wrote: > > > > > hi Danny Chan ,Thank you very much for your reply, your help can help > me > > > further improve this discussion. > > > Best > > > forward > > > > > > Danny Chan 于2019年9月4日周三 下午8:50写道： > > > > > > > Thanks Xu Forward for bring up this topic, I think the JSON functions > > are > > > > very useful especially for those MySQL users. > > > > > > > > I saw that you have done some work within the Apache Calcite, that’s > a > > > > good start, but this is one concern from me, Flink doesn’t support > JSON > > > > type internal, so how to represent a JSON object in Flink maybe a key > > > point > > > > we need to resolve. In Calcite, we use ANY type to represent as the > > JSON, > > > > but I don’t think it is the right way to go, maybe we can have a > > > discussion > > > > here. > > > > > > > > Best, > > > > Danny Chan > > > > 在 2019年9月4日 +0800 PM8:34，Xu Forward ，写道： > > > > > Hi everybody, > > > > > > > > > > I'd like to kick off a discussion on Support JSON functions in > Flink > > > SQL. > > > > > > > > > > The entire plan is divided into two steps: > > > > > 1. Implement Support SQL 2016-2017 JSON functions in Flink SQL[1]. > > > > > 2. Implement non-Support SQL 2016-2017 JSON functions in Flink SQL, > > > such > > > > as > > > > > JSON_TYPE in Mysql, JSON_LENGTH, etc. Very useful JSON functions. > > > > > > > > > > Would love to hear your thoughts. > > > > > > > > > > [1] > > > > > > > > > > > > > > > https://docs.google.com/document/d/1JfaFYIFOAY8P2pFhOYNCQ9RTzwF4l85_bnTvImOLKMk/edit#heading=h.76mb88ca6yjp > > > > > > > > > > Best, > > > > > ForwardXu > > > > > > > > > >
