Subscribe
- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
Subscribe
- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
Re: [ANNOUNCE] New committer: Xiong Duan
Congratulations ! Best, Jin Chunwei Lei 于2021年10月25日周一 下午4:58写道： > Congratulations, Xiong! > > > Best, > Chunwei > > > On Sun, Oct 24, 2021 at 6:34 AM Haisheng Yuan wrote: > > > Congrats, Xiong! > > > > On 2021/10/23 21:23:59, Francis Chuang wrote: > > > Congratulations! > > > > > > On 24/10/2021 12:03 am, Stamatis Zampetakis wrote: > > > > Apache Calcite's Project Management Committee (PMC) has invited Xiong > > Duan > > > > to > > > > become a committer, and we are pleased to announce that they have > > accepted. > > > > > > > > Xiong has pushed a lot of high quality patches, fixing and improving > > code > > > > around > > > > aggregations and sub-queries, in a rather short period of time. > Apart > > from > > > > code > > > > contributions, Xiong has been regularly reviewing PRs in GitHub and > > helping > > > > out > > > > others in various JIRA issues. > > > > > > > > Xiong, welcome, thank you for your contributions, and we look forward > > to > > > > your > > > > further interactions with the community! If you wish, please feel > free > > to > > > > tell > > > > us more about yourself and what you are working on. > > > > > > > > Stamatis (on behalf of the Apache Calcite PMC) > > > > > > > > > >
Re: [ANNOUNCE] New committer: Zhaohui Xu
Congratulations ! Jin Michael Mior 于2021年10月15日周五 上午12:56写道： > Welcome Zhaohui! > -- > Michael Mior > mm...@apache.org > > Le mer. 6 oct. 2021 à 16:48, Stamatis Zampetakis a > écrit : > > > > Apache Calcite's Project Management Committee (PMC) has invited Zhaohui > Xu > > to > > become a committer, and we are pleased to announce that they have > accepted. > > > > Numbers speak for themselves and Zhaohui has over 30 commits already in > > master > > and more than 20 open pull requests waiting to get in. Great record so > far > > including > > (but not limited to) improvements and fixes in the view based > > rewriting modules, > > JSON serialization, metadata, and field trimming. > > > > Zhaohui, welcome, thank you for your contributions, and we look forward > to > > your > > further interactions with the community! If you wish, please feel free to > > tell > > us more about yourself and what you are working on. > > > > Stamatis (on behalf of the Apache Calcite PMC) >
Re: [VOTE] FLIP-184: Refine ShuffleMaster lifecycle management for pluggable shuffle service framework
+1 (non-binding) Best, Jin Guowei Ma 于2021年7月19日周一 上午9:41写道： > +1(binding) > > Best, > Guowei > > > On Fri, Jul 16, 2021 at 5:36 PM Yingjie Cao > wrote: > > > Hi all, > > > > I'd like to start a vote on FLIP-184 [1] which was > > discussed in [2] [3]. The vote will be open for at least 72 hours > > until 7.21 unless there is an objection. > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-184%3A+Refine+ShuffleMaster+lifecycle+management+for+pluggable+shuffle+service+framework > > [2] > > > > > https://lists.apache.org/thread.html/radbbabfcfb6bec305ddf7aeefb983232f96b18ba013f0ae2ee500288%40%3Cdev.flink.apache.org%3E > > [3] > > > > > https://lists.apache.org/thread.html/r93e3a72506f3e7ffd3c1ab860b5d1a21f8a47b059f2f2fdd05ca1d46%40%3Cdev.flink.apache.org%3E > > >
Re: [ANNOUNCE] New Apache Flink Committer - Yuan Mei
Congratulations Yuan~! Roc Marshal 于2021年7月8日周四 上午11:28写道： > Congratulations, Yuan! > > > > > > > > > > > > > > > > At 2021-07-08 01:21:40, "Yu Li" wrote: > >Hi all, > > > >On behalf of the PMC, I’m very happy to announce Yuan Mei as a new Flink > >committer. > > > >Yuan has been an active contributor for more than two years, with code > >contributions on multiple components including kafka connectors, > >checkpointing, state backends, etc. Besides, she has been actively > involved > >in community activities such as helping manage releases, discussing > >questions on dev@list, supporting users and giving talks at conferences. > > > >Please join me in congratulating Yuan for becoming a Flink committer! > > > >Cheers, > >Yu >
Re: [ANNOUNCE] New Apache Flink Committer - Yang Wang
Congratulations Yang Wang! Austin Cawley-Edwards 于2021年7月7日周三 下午9:29写道： > Congrats Yang! > > On Wed, Jul 7, 2021 at 7:26 AM Jingsong Li wrote: > > > Congratulations, Yang! > > > > Best, > > Jingsong > > > > On Wed, Jul 7, 2021 at 6:43 PM Arvid Heise wrote: > > > > > Congratulations! > > > > > > On Wed, Jul 7, 2021 at 12:17 PM godfrey he > wrote: > > > > > > > Congratulations, Yang! > > > > > > > > Best, > > > > Godfrey > > > > > > > > Lijie Wang 于2021年7月7日周三 下午5:59写道： > > > > > > > > > Congratulations Yang! > > > > > > > > > > Till Rohrmann 于2021年7月7日周三 下午5:29写道： > > > > > > > > > > > Congratulations, Yang! > > > > > > > > > > > > Cheers, > > > > > > Till > > > > > > > > > > > > On Wed, Jul 7, 2021 at 9:41 AM Roman Khachatryan < > ro...@apache.org > > > > > > > > wrote: > > > > > > > > > > > > > Congrats! > > > > > > > > > > > > > > Regards, > > > > > > > Roman > > > > > > > > > > > > > > > > > > > > > On Wed, Jul 7, 2021 at 8:28 AM Qingsheng Ren < > renqs...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > > > > Congratulations Yang! > > > > > > > > > > > > > > > > -- > > > > > > > > Best Regards, > > > > > > > > > > > > > > > > Qingsheng Ren > > > > > > > > Email: renqs...@gmail.com > > > > > > > > On Jul 7, 2021, 2:26 PM +0800, Rui Li >, > > > > > wrote: > > > > > > > > > Congratulations Yang ~ > > > > > > > > > > > > > > > > > > On Wed, Jul 7, 2021 at 1:01 PM Benchao Li < > > > libenc...@apache.org> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Congratulations! > > > > > > > > > > > > > > > > > > > > Peter Huang 于2021年7月7日周三 > > > > 下午12:54写道： > > > > > > > > > > > > > > > > > > > > > Congratulations, Yang. > > > > > > > > > > > > > > > > > > > > > > Best Regards > > > > > > > > > > > Peter Huang > > > > > > > > > > > > > > > > > > > > > > On Tue, Jul 6, 2021 at 9:48 PM Dian Fu < > > > > dian0511...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Congratulations, Yang, > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > Dian > > > > > > > > > > > > > > > > > > > > > > > > > 2021年7月7日 上午10:46，Jary Zhen > 写道： > > > > > > > > > > > > > > > > > > > > > > > > > > Congratulations, Yang Wang. > > > > > > > > > > > > > > > > > > > > > > > > > > Best > > > > > > > > > > > > > Jary > > > > > > > > > > > > > > > > > > > > > > > > > > Yun Gao > 于2021年7月7日周三 > > > > > > 上午10:38写道： > > > > > > > > > > > > > > > > > > > > > > > > > > > Congratulations Yang! > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > Yun > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > Sender:Jark Wu > > > > > > > > > > > > > > Date:2021/07/07 10:20:27 > > > > > > > > > > > > > > Recipient:dev > > > > > > > > > > > > > > Cc:Yang Wang; < > > > > > > > wangyang0...@apache.org> > > > > > > > > > > > > > > Theme:Re: [ANNOUNCE] New Apache Flink Committer - > > > Yang > > > > > Wang > > > > > > > > > > > > > > > > > > > > > > > > > > > > Congratulations Yang Wang! > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > Jark > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, 7 Jul 2021 at 10:09, Xintong Song < > > > > > > > tonysong...@gmail.com> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On behalf of the PMC, I'm very happy to > announce > > > Yang > > > > > > Wang > > > > > > > as a new > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > committer. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yang has been a very active contributor for > more > > > than > > > > > two > > > > > > > years, > > > > > > > > > > > mainly > > > > > > > > > > > > > > > focusing on Flink's deployment components. > He's a > > > > main > > > > > > > contributor > > > > > > > > > > > and > > > > > > > > > > > > > > > maintainer of Flink's native Kubernetes > > deployment > > > > and > > > > > > > native > > > > > > > > > > > > Kubernetes > > > > > > > > > > > > > > > HA. He's also very active on the mailing lists, > > > > > > > participating in > > > > > > > > > > > > > > > discussions and helping with user questions. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please join me in congratulating Yang Wang for > > > > > becoming a > > > > > > > Flink > > > > > > > > > > > > > > committer! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > Best, > > > > > >
Re: [ANNOUNCE] New PMC member: Guowei Ma
Congratulations, Guowei~ ! Best, Jin Xintong Song 于2021年7月7日周三 上午9:37写道： > Congratulations, Guowei~! > > Thank you~ > > Xintong Song > > > > On Wed, Jul 7, 2021 at 9:31 AM Qingsheng Ren wrote: > > > Congratulations Guowei! > > > > -- > > Best Regards, > > > > Qingsheng Ren > > Email: renqs...@gmail.com > > 2021年7月7日 +0800 09:30 Leonard Xu ，写道： > > > Congratulations! Guowei Ma > > > > > > Best, > > > Leonard > > > > > > > ÔÚ 2021Äê7ÔÂ6ÈÕ£¬21:56£¬Kurt Young дµÀ£º > > > > > > > > Hi all! > > > > > > > > I'm very happy to announce that Guowei Ma has joined the Flink PMC! > > > > > > > > Congratulations and welcome Guowei! > > > > > > > > Best, > > > > Kurt > > > > > >
Re: [ANNOUNCE] New committer: Vladimir Ozerov
Congratulations ~ Best, Jin guangyuan wang 于2021年6月24日周四 上午9:50写道： > Congratulations! > > Francis Chuang 于2021年6月24日周四 上午6:39写道： > > > Congrats, Vladimir! > > > > Francis > > > > On 24/06/2021 7:48 am, Haisheng Yuan wrote: > > > Congratulations and thanks for your contributions, Vladimir! > > > > > > Regards, > > > Haisheng > > > > > > On 2021/06/23 21:34:40, Stamatis Zampetakis wrote: > > >> Apache Calcite's Project Management Committee (PMC) has invited > Vladimir > > >> Ozerov to > > >> become a committer, and we are pleased to announce that he has > accepted. > > >> > > >> Vladimir is among the few people who know very well the internal > > workings > > >> of the > > >> Calcite optimizer. He started and participated in many discussions > about > > >> the core engine and contributed ideas and code for making it better. > > >> Moreover, Vladimir has blogged and talked about Calcite in various > > >> conferences and meetups giving publicity and showcasing the > > capabilities of > > >> the project. > > >> > > >> Vladimir, welcome, thank you for your contributions, and we look > > forward to > > >> your > > >> further interactions with the community! If you wish, please feel free > > to > > >> tell > > >> us more about yourself and what you are working on. > > >> > > >> Stamatis (on behalf of the Apache Calcite PMC) > > >> > > >
Re: The partition tracker should support remote shuffle properly
Hi, Till ~ Thanks for your comments. Yes, I think we are on the same page – we are discussing how should JMPartitionTracker manage partitions properly ~ > If the ShuffleMaster should lose a result partition, then a reading task should fail with a PartitionException which will invalidate the partition on the JobMasterPartitionTracker so that it is reproduced. 1. True, the reproduction of a lost upstream partition could be triggered by the shuffle-read failure of downstream. But it tends to be unacceptable in the production environment for batch processing jobs. Say there are 100 upstream partitions lost due to failure of an external shuffle worker, if there's no notification from ShuffleMaster to JMPartitionTracker, 100 PartitionExceptions on downstream will happen and upstream partitions will be reproduced one by one sequentially. The time overhead will be unacceptable. From this point of view, I tend to think ShuffleMaster should have the ability to unregister partitions by locationID on JMPartitionTracker. > I think that deciding whether the partition is stored externally or not can be answered by using the ShuffleDescriptor.storesLocalResourcesOn method. 2. Yes, I agree it works (though it's not easy to access ShuffleDescriptor by tmID from JMPartitionTracker at this moment). But if we agree that my first point is valid, JMPartitionTracker should maintain the index from locationID to partition. Then it will be straightforward to check whether a partition is accommodated on remote by comparing its tmID and locationID; Best, Jin Till Rohrmann 于2021年6月1日周二 下午4:59写道： > Hi Jin, > > thanks for starting this discussion and the initiative to implement a > remote shuffle service. It has always been the idea of the ShuffleService > abstraction to make this possible and we probably have overlooked some > details. > > What I have understood from your description, you would like to introduce > the locationID which points to the location where the result partition is > stored (potentially external). Using the locationID and the tmID it is > possible to say whether the partition is stored externally or not. > > I think that deciding whether the partition is stored externally or not (or > more precisely whether the partition occupies resources on the TM) can be > answered by using the ShuffleDescriptor.storesLocalResourcesOn method. If > it returns some ResourceID then we have to tell the TM about the release. > If not, then we only tell the shuffle master about the partition release. > How the data can be accessed on the external system is then encapsulated by > the ShuffleMaster and the ShuffleDescriptor. The logic for releasing the > partitions on the TMs and the ShuffleMaster should already be implemented > in the JobMasterPartitionTrackerImpl. > > I think what we need to change is that we don't stop the tracking of > completed partitions when a TM on which the producers run disconnects and > if we store the result partition externally. This is required to make > partitions survive in case of TM failures. What this also requires is to > distinguish between finished and in-progress partitions. > > What indeed is currently not implemented is the channel from the > ShuffleMaster to the JobMasterPartitionTrackerImpl. This is, however, not a > big problem atm. If the ShuffleMaster should lose a result partition, then > a reading task should fail with a PartitionException which will invalidate > the partition on the JMPartitionTracker so that it is reproduced. Listening > to the ShuffleMaster would be an optimization to learn more quickly about > this fact and to avoid a restart cycle. > > Did I understand you correctly, Jin, and do my comments make sense? > > Cheers, > Till > > On Wed, May 26, 2021 at 5:52 AM XING JIN wrote: > > > Hi devs ~ > > Recently our team designed and started to build Flink remote shuffle > > service based on 'pluggable shuffle service framework'[1] for batch > > processing jobs. We found some potential enhancements could be made on > > 'pluggable shuffle service' and created an umbrella JIRA[2]. I raise this > > DISCUSSION and want to hear broader feedback / comments on one ticket [3] > > -- "The partition tracker should support remote shuffle properly". > > > > In current Flink, data partition is bound with the ResourceID of TM in > > Execution#startTrackingPartitions and JM partition tracker will stop > > tracking corresponding partitions when a TM > > disconnects(JobMaster#disconnectTaskManager), i.e. the lifecycle of > shuffle > > data is bound with computing resource (TM). It works fine for internal > > shuffle service, but doesn't for remote shuffle service. Note that > shuffle > > data is accommodated on remote, the lifecycle of a comp
The partition tracker should support remote shuffle properly
Hi devs ~ Recently our team designed and started to build Flink remote shuffle service based on 'pluggable shuffle service framework'[1] for batch processing jobs. We found some potential enhancements could be made on 'pluggable shuffle service' and created an umbrella JIRA[2]. I raise this DISCUSSION and want to hear broader feedback / comments on one ticket [3] -- "The partition tracker should support remote shuffle properly". In current Flink, data partition is bound with the ResourceID of TM in Execution#startTrackingPartitions and JM partition tracker will stop tracking corresponding partitions when a TM disconnects(JobMaster#disconnectTaskManager), i.e. the lifecycle of shuffle data is bound with computing resource (TM). It works fine for internal shuffle service, but doesn't for remote shuffle service. Note that shuffle data is accommodated on remote, the lifecycle of a completed partition is capable to be decoupled with TM, i.e. TM is totally fine to be released when no computing task is on it and further shuffle reading requests could be directed to remote shuffle cluster. In addition, when a TM is lost, its completed data partitions on remote shuffle cluster could avoid reproducing. The issue mentioned above is because Flink JobMasterPartitionTracker mixed up partition's locationID (where the partition is located) and tmID (which TM the partition is produced from). In TM internal shuffle, partition's locationID is the same with tmID, but it is not in remote shuffle; JobMasterPartitionTracker as an independent component should be able to differentiate locationID and tmID of a partition, thus to handle the lifecycle of a partition properly; We propose that JobMasterPartitionTracker indexes partitions with both locationID and tmID. The process of registration and unregistration will be like below: A. Partition Registration - Execution#registerProducedPartitions registers partition to ShuffleMaster and gets a ShuffleDescriptor. Current ShuffleDescriptor#storesLocalResourcesOn returns the location of the producing TM ONLY IF the partition occupies local resources there. We propose to change this method a proper name and always return the locationID of the partition. It might be as below: ResourceID getLocationID(); - Execution#registerProducePartitions then registers partition to JMPartitionTracker with tmID (ResourceID of TaskManager from TaskManagerLocation) and the locationID (acquired in above step). JobMasterPartitionTracker will indexes a partition with both tmID and locationID; B. Invokes from JM and ShuffleMaster JobMasterPartitionTracker listens invokes from both JM and ShuffleMaster. - When JMPartitionTracker hears from JobMaster#disconnectTaskManager that a TM disconnects, it will check whether the disconnected tmID equals a certain locationID of a partition. If so, tracking of the corresponding partition will be stopped. - When JobMasterPartitionTracker hears from ShuffleMaster that a data location gets lost, it will unregister corresponding partitions by locationID; C. Partition Unregistration When unregister a partition, JobMasterPartitionTracker removes the corresponding indexes to tmID and locationID firstly, and then release the partition according to shuffle service types: - If the locationID equals to the tmID, it indicates the partition is accommodated by TM internal shuffle service, JMPartitionTracker will invoke TaskExecutorGateway for the release; - If the locationID doesn't equal to tmID, it indicates the partition is accommodated by external shuffle service, JMPartitionTracker will invoke ShuffleMaster for the release; With the above change, JobMasterPartitionTracker can adapt with customized shuffle service properly for partition's lifecycle. Looking forward to inputs on this ~~ Best, Jin [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-31%3A+Pluggable+Shuffle+Service [2] https://issues.apache.org/jira/browse/FLINK-22672 [3] https://issues.apache.org/jira/browse/FLINK-22676
Re: [ANNOUNCE] Ruben Quesada Lopez joins Calcite PMC
Congrats, Ruben! 953396112 <953396...@qq.com> 于2020年8月12日周三 上午7:47写道： > Congratulations,Ruben! > > > xzh > --原始邮件-- > 发件人: > "dev" > < > zabe...@gmail.com; > 发送时间:2020年8月12日(星期三) 凌晨5:53 > 收件人:"dev" > 主题:[ANNOUNCE] Ruben Quesada Lopez joins Calcite PMC > > > > I'm pleased to announce that Ruben has accepted an invitation to > join the Calcite PMC. Ruben has been a consistent and helpful > figure in the Calcite community for which we are very grateful. We > look forward to the continued contributions and support. > > Please join me in congratulating Ruben! > > - Stamatis (on behalf of the Calcite PMC)
Re: [ANNOUNCE] Apache Calcite 1.23.0 released
Thanks a lot for driving this, Haisheng! Best, Jin Stamatis Zampetakis 于2020年5月28日周四 上午7:14写道： > Many thanks again Haisheng, the release has your signature on it and it is > in many places :) > > On Wed, May 27, 2020 at 9:34 PM Haisheng Yuan wrote: > > > The Apache Calcite team is pleased to announce the release of Apache > > Calcite 1.23.0. > > > > Calcite is a dynamic data management framework. Its cost-based > > optimizer converts queries, represented in relational algebra, into > > executable plans. Calcite supports many front-end languages and > > back-end data engines, and includes an SQL parser and, as a > > sub-project, the Avatica JDBC driver. > > > > This release comes two months after 1.22.0. It includes more than > > 100 resolved issues, comprising of a few new features as well as > > general improvements and bug-fixes. It includes support for top > > down trait request and trait enforcement without abstract converter, > > ClickHouse dialect, SESSION and HOP Table functions, and many > > more bug fixes and improvements. > > > > You can start using it in Maven by simply updating your dependency to: > > > > > > org.apache.calcite > > calcite-core > > 1.23.0 > > > > > > If you'd like to download the source release, you can find it here: > > > > https://calcite.apache.org/downloads/ > > > > You can read more about the release (including release notes) here: > > > > https://calcite.apache.org/news/2020/05/23/release-1.23.0/ > > > > We welcome your help and feedback. For more information on how to > > report problems, and to get involved, visit the project website at: > > > > https://calcite.apache.org/ > > > > Thanks to everyone involved! > > >
Re: [VOTE] Release apache-calcite-1.23.0 (release candidate 1)
Thanks, Haisheng ~ Local Calcite build with tests enabled on Linux: *OK* Calcite-based system test suite: *OK* Vote: +1 (non-binding) Francis Chuang 于2020年5月18日周一 下午2:00写道： > Thanks for making this release available for voting, Haisheng! > > Verified GPG signature - OK > Verified SHA512 - OK > Ran tests per HOWTO (./gradlew check) - OK > Quickly skimmed release notes - OK > Spotted checked a few JARs in the Maven repository - OK > > Environment (OpenJDK:latest docker container): > Gradle 6.3 (via gradlew) > Debian GNU/Linux 8 > openjdk version "1.8.0_111" > OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-2~bpo8+1-b14) > OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode) > > My vote is: +1 (binding) > > Francis > > On 16/05/2020 2:02 pm, Haisheng Yuan wrote: > > Hi all, > > > > I have created a build for Apache Calcite 1.23.0, release > > candidate 1. > > > > Thanks to everyone who has contributed to this release. > > > > You can read the release notes here: > > > https://github.com/apache/calcite/blob/calcite-1.23.0-rc1/site/_docs/history.md > > > > The commit to be voted upon: > > > https://gitbox.apache.org/repos/asf?p=calcite.git;a=commit;h=b708fdc46d4c5fd4c5a6c7a398823318a7b4dce3 > > > > Its hash is b708fdc46d4c5fd4c5a6c7a398823318a7b4dce3 > > > > Tag: > > https://github.com/apache/calcite/tree/calcite-1.23.0-rc1 > > > > The artifacts to be voted on are located here: > > https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-1.23.0-rc1 > > (revision 39622) > > > > The hashes of the artifacts are as follows: > > > 961c4f13199e199c669a6168ba655a9492bdd80d644da375a684b732c0b628b8a2ffacea5da97c82e8702a8e3bf7a1f58784baa49509fb3c48ef593259e11f46 > > *apache-calcite-1.23.0-src.tar.gz > > > > A staged Maven repository is available for review at: > > > https://repository.apache.org/content/repositories/orgapachecalcite-1089/org/apache/calcite/ > > > > Release artifacts are signed with the following key: > > https://dist.apache.org/repos/dist/release/calcite/KEYS > > > > N.B. > > To create the jars and test Apache Calcite: "./gradlew build". > > > > If you do not have a Java environment available, you can run the tests > > using docker. To do so, install docker and docker-compose, then run > > "docker-compose run test" from the root of the directory. > > > > Please vote on releasing this package as Apache Calcite 1.23.0. > > > > The vote is open for the next 72 hours and passes if a majority of at > > least three +1 PMC votes are cast. > > > > [ ] +1 Release this package as Apache Calcite 1.23.0 > > [ ] 0 I don't feel strongly about it, but I'm okay with the release > > [ ] -1 Do not release this package because... > > > > > > Here is my vote: > > > > +1 (binding) > > > > Thanks, > > Haisheng Yuan > > >
Re: [ANNOUNCE] New committer: Jin Xing
Hi, Julian ~ For introduction of my company [1]. It's under Alibaba Group and doing financial business. Regarding "powered by" page[2], it will be great if we could add an entry and logo on it ;D Thanks a lot ! Jin [1] https://en.wikipedia.org/wiki/Ant_Financial <https://en.wikipedia.org/wiki/Ant_Financial> [2] https://calcite.apache.org/docs/powered_by.html <https://calcite.apache.org/docs/powered_by.html#alibaba-maxcompute> XING JIN 于2020年5月1日周五 上午10:54写道： > Thanks a lot, Julian ~ > I'm not from MaxCompute team, but from big data platform in Alibaba Ant > Financial Group. > Actually we cooperate a lot with MaxCompute, it's our sister team. > > Jin > > Julian Hyde 于2020年5月1日周五 上午1:48写道： > >> Welcome Jin! Thanks for your contributions so far, looking forward to >> more! >> >> Are you on the MaxCompute project? It’s already on our “powered by” >> page[1], so I think people are familiar with it. >> >> Julian >> >> [1] https://calcite.apache.org/docs/powered_by.html#alibaba-maxcompute < >> https://calcite.apache.org/docs/powered_by.html#alibaba-maxcompute> >> >> >> > On Apr 29, 2020, at 5:06 AM, XING JIN wrote: >> > >> > Thanks a lot ~ >> > Calcite is a great project and it's great honor for me to work with you >> > guys. I really appreciate the help from community. >> > I'm working in Alibaba. My team builds big data system to optimize batch >> > and streaming jobs. We use Calcite to process Sql queries and >> accommodate >> > to different physical engines. >> > I'm very excited to become Calcite committer and looking forward to make >> > more contributions. >> > >> > Best regards, >> > Jin >> > >> > >> > Zoltan Haindrich 于2020年4月29日周三 下午1:58写道： >> > >> >> Congrats! >> >> >> >> On 4/29/20 7:32 AM, Enrico Olivelli wrote: >> >>> Congratulations! >> >>> >> >>> Enrico >> >>> >> >>> Il Mer 29 Apr 2020, 04:51 Feng Zhu ha >> scritto: >> >>> >> >>>> Congrations! >> >>>> >> >>>> best, >> >>>> Feng >> >>>> >> >>>> Chunwei Lei 于2020年4月29日周三 上午10:16写道： >> >>>> >> >>>>> Congrats, Jin! >> >>>>> >> >>>>> >> >>>>> Best, >> >>>>> Chunwei >> >>>>> >> >>>>> >> >>>>> On Wed, Apr 29, 2020 at 10:07 AM Forward Xu > > >> >>>>> wrote: >> >>>>> >> >>>>>> Congrats >> >>>>>> >> >>>>>> >> >>>>>> best, >> >>>>>> >> >>>>>> Forward >> >>>>>> >> >>>>>> 953396112 <953396...@qq.com> 于2020年4月29日周三 上午8:21写道： >> >>>>>> >> >>>>>>> Congrats, Jin Xing! >> >>>>>>> >> >>>>>>> >> >>>>>>> ---Original--- >> >>>>>>> From: "Stamatis Zampetakis"> >>>>>>> Date: Wed, Apr 29, 2020 05:47 AM >> >>>>>>> To: "dev"> >>>>>>> Subject: [ANNOUNCE] New committer: Jin Xing >> >>>>>>> >> >>>>>>> >> >>>>>>> Apache Calcite's Project Management Committee (PMC) has invited >> Jin >> >>>>> Xing >> >>>>>> to >> >>>>>>> become a committer, and we are pleased to announce that he has >> >>>>> accepted. >> >>>>>>> >> >>>>>>> Jin has contributed a lot of code in the project and many >> >>>>>>> recent improvements in >> >>>>>>> materialized view matching have his signature on them. Apart from >> >>>> code >> >>>>>>> contributions, Jin provides valuable help to the community by >> doing >> >>>>>> reviews >> >>>>>>> and >> >>>>>>> answering questions in the devlist. >> >>>>>>> >> >>>>>>> Jin, welcome, thank you for your contributions, and we look >> forward >> >>>> to >> >>>>>> your >> >>>>>>> further interactions with the community! If you wish, please feel >> >>>> free >> >>>>> to >> >>>>>>> tell >> >>>>>>> us more about yourself and what you are working on. >> >>>>>>> >> >>>>>>> Stamatis (on behalf of the Apache Calcite PMC) >> >>>>>> >> >>>>> >> >>>> >> >>> >> >> >> >>
Re: [ANNOUNCE] New committer: Jin Xing
Thanks a lot, Julian ~ I'm not from MaxCompute team, but from big data platform in Alibaba Ant Financial Group. Actually we cooperate a lot with MaxCompute, it's our sister team. Jin Julian Hyde 于2020年5月1日周五 上午1:48写道： > Welcome Jin! Thanks for your contributions so far, looking forward to more! > > Are you on the MaxCompute project? It’s already on our “powered by” > page[1], so I think people are familiar with it. > > Julian > > [1] https://calcite.apache.org/docs/powered_by.html#alibaba-maxcompute < > https://calcite.apache.org/docs/powered_by.html#alibaba-maxcompute> > > > > On Apr 29, 2020, at 5:06 AM, XING JIN wrote: > > > > Thanks a lot ~ > > Calcite is a great project and it's great honor for me to work with you > > guys. I really appreciate the help from community. > > I'm working in Alibaba. My team builds big data system to optimize batch > > and streaming jobs. We use Calcite to process Sql queries and accommodate > > to different physical engines. > > I'm very excited to become Calcite committer and looking forward to make > > more contributions. > > > > Best regards, > > Jin > > > > > > Zoltan Haindrich 于2020年4月29日周三 下午1:58写道： > > > >> Congrats! > >> > >> On 4/29/20 7:32 AM, Enrico Olivelli wrote: > >>> Congratulations! > >>> > >>> Enrico > >>> > >>> Il Mer 29 Apr 2020, 04:51 Feng Zhu ha scritto: > >>> > >>>> Congrations! > >>>> > >>>> best, > >>>> Feng > >>>> > >>>> Chunwei Lei 于2020年4月29日周三 上午10:16写道： > >>>> > >>>>> Congrats, Jin! > >>>>> > >>>>> > >>>>> Best, > >>>>> Chunwei > >>>>> > >>>>> > >>>>> On Wed, Apr 29, 2020 at 10:07 AM Forward Xu > >>>>> wrote: > >>>>> > >>>>>> Congrats > >>>>>> > >>>>>> > >>>>>> best, > >>>>>> > >>>>>> Forward > >>>>>> > >>>>>> 953396112 <953396...@qq.com> 于2020年4月29日周三 上午8:21写道： > >>>>>> > >>>>>>> Congrats, Jin Xing! > >>>>>>> > >>>>>>> > >>>>>>> ---Original--- > >>>>>>> From: "Stamatis Zampetakis" >>>>>>> Date: Wed, Apr 29, 2020 05:47 AM > >>>>>>> To: "dev" >>>>>>> Subject: [ANNOUNCE] New committer: Jin Xing > >>>>>>> > >>>>>>> > >>>>>>> Apache Calcite's Project Management Committee (PMC) has invited Jin > >>>>> Xing > >>>>>> to > >>>>>>> become a committer, and we are pleased to announce that he has > >>>>> accepted. > >>>>>>> > >>>>>>> Jin has contributed a lot of code in the project and many > >>>>>>> recent improvements in > >>>>>>> materialized view matching have his signature on them. Apart from > >>>> code > >>>>>>> contributions, Jin provides valuable help to the community by doing > >>>>>> reviews > >>>>>>> and > >>>>>>> answering questions in the devlist. > >>>>>>> > >>>>>>> Jin, welcome, thank you for your contributions, and we look forward > >>>> to > >>>>>> your > >>>>>>> further interactions with the community! If you wish, please feel > >>>> free > >>>>> to > >>>>>>> tell > >>>>>>> us more about yourself and what you are working on. > >>>>>>> > >>>>>>> Stamatis (on behalf of the Apache Calcite PMC) > >>>>>> > >>>>> > >>>> > >>> > >> > >
Re: [ANNOUNCE] New committer: Jin Xing
Thanks a lot ~ Calcite is a great project and it's great honor for me to work with you guys. I really appreciate the help from community. I'm working in Alibaba. My team builds big data system to optimize batch and streaming jobs. We use Calcite to process Sql queries and accommodate to different physical engines. I'm very excited to become Calcite committer and looking forward to make more contributions. Best regards, Jin Zoltan Haindrich 于2020年4月29日周三 下午1:58写道： > Congrats! > > On 4/29/20 7:32 AM, Enrico Olivelli wrote: > > Congratulations! > > > > Enrico > > > > Il Mer 29 Apr 2020, 04:51 Feng Zhu ha scritto: > > > >> Congrations! > >> > >> best, > >> Feng > >> > >> Chunwei Lei 于2020年4月29日周三 上午10:16写道： > >> > >>> Congrats, Jin! > >>> > >>> > >>> Best, > >>> Chunwei > >>> > >>> > >>> On Wed, Apr 29, 2020 at 10:07 AM Forward Xu > >>> wrote: > >>> > Congrats > > > best, > > Forward > > 953396112 <953396...@qq.com> 于2020年4月29日周三 上午8:21写道： > > > Congrats, Jin Xing! > > > > > > ---Original--- > > From: "Stamatis Zampetakis" > Date: Wed, Apr 29, 2020 05:47 AM > > To: "dev" > Subject: [ANNOUNCE] New committer: Jin Xing > > > > > > Apache Calcite's Project Management Committee (PMC) has invited Jin > >>> Xing > to > > become a committer, and we are pleased to announce that he has > >>> accepted. > > > > Jin has contributed a lot of code in the project and many > > recent improvements in > > materialized view matching have his signature on them. Apart from > >> code > > contributions, Jin provides valuable help to the community by doing > reviews > > and > > answering questions in the devlist. > > > > Jin, welcome, thank you for your contributions, and we look forward > >> to > your > > further interactions with the community! If you wish, please feel > >> free > >>> to > > tell > > us more about yourself and what you are working on. > > > > Stamatis (on behalf of the Apache Calcite PMC) > > >>> > >> > > >
Re: How to give type in case of type mismatch while doing union
Hi, Anjali ~ Are you doing the UNION by Sql ? If so, can you give the Sql content ? Are you doing the UNION on RelNodes ?, If so, you need to do type CAST. Jin Anjali Shrishrimal 于2020年4月27日周一 下午4:25写道： > Hi, > > While doing union of 2 RelNodes with different types, I am getting NPE. (I > am using calcite 1.21.0) > java.lang.NullPointerException: at index 0 > at > com.google.common.collect.ObjectArrays.checkElementNotNull(ObjectArrays.java:225) > at > com.google.common.collect.ObjectArrays.checkElementsNotNull(ObjectArrays.java:215) > at > com.google.common.collect.ObjectArrays.checkElementsNotNull(ObjectArrays.java:209) > at > com.google.common.collect.ImmutableList.construct(ImmutableList.java:346) > at > com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:258) > at > org.apache.calcite.rel.type.RelDataTypeFactoryImpl.canonize(RelDataTypeFactoryImpl.java:373) > at > org.apache.calcite.rel.type.RelDataTypeFactoryImpl.createStructType(RelDataTypeFactoryImpl.java:155) > at > org.apache.calcite.rel.type.RelDataTypeFactoryImpl.createStructType(RelDataTypeFactoryImpl.java:146) > at > org.apache.calcite.rel.type.RelDataTypeFactory$Builder.build(RelDataTypeFactory.java:569) > at > org.apache.calcite.rel.type.RelDataTypeFactoryImpl.leastRestrictiveStructuredType(RelDataTypeFactoryImpl.java:257) > at > org.apache.calcite.sql.type.SqlTypeFactoryImpl.leastRestrictiveSqlType(SqlTypeFactoryImpl.java:285) > at > org.apache.calcite.sql.type.SqlTypeFactoryImpl.leastRestrictive(SqlTypeFactoryImpl.java:156) > at > org.apache.calcite.rel.core.SetOp.deriveRowType(SetOp.java:107) > > If the column types (family types) are different, currently the derived > type is null. Is there any way to control that? > Where can I define the type in case of mismatch ? > > > Thank you, > Anjali Shrishrimal >
Re: [ANNOUNCE] New committer: Vineet Garg
Congrats, Vineet! Best, Jin Fan Liya 于2020年4月26日周日 上午11:45写道： > Congratulations, Vineet! > > Best, > Liya Fan > > On Sun, Apr 26, 2020 at 10:30 AM Aman Sinha wrote: > > > Congratulations Vineet ! > > > > -Aman > > > > On Sat, Apr 25, 2020 at 7:15 PM Feng Zhu wrote: > > > > > Congratulations, well deserved! > > > > > > best, > > > Feng > > > > > > Chunwei Lei 于2020年4月26日周日 上午10:12写道： > > > > > > > Congrats, Vineet! > > > > > > > > > > > > Best, > > > > Chunwei > > > > > > > > > > > > On Sun, Apr 26, 2020 at 8:24 AM Haisheng Yuan > > wrote: > > > > > > > > > Congrats, Vineet! > > > > > > > > > > On 2020/04/25 22:18:35, Forward Xu wrote: > > > > > > Congratulations > > > > > > > > > > > > best, > > > > > > Forward > > > > > > > > > > > > Francis Chuang 于2020年4月26日周日 > 上午6:04写道： > > > > > > > > > > > > > Congrats, Vineet! > > > > > > > > > > > > > > On 26/04/2020 7:52 am, Stamatis Zampetakis wrote: > > > > > > > > Apache Calcite's Project Management Committee (PMC) has > invited > > > > > Vineet > > > > > > > > Garg to become a committer, and we are pleased to announce > that > > > he > > > > > > > > has accepted. > > > > > > > > > > > > > > > > With the first code contribution in Calcite back in 2017, > > Vineet > > > is > > > > > > > > definitely > > > > > > > > not new to the project. Since then he has contributed many > > > patches, > > > > > > > > fixing and improving various modules of Calcite, notably > things > > > > > around > > > > > > > > subqueries. > > > > > > > > > > > > > > > > Vineet, welcome, thank you for your contributions, and we > look > > > > > forward > > > > > > > > your further interactions with the community! If you wish, > > please > > > > > feel > > > > > > > > free to tell us more about yourself and what you are working > > on. > > > > > > > > > > > > > > > > Stamatis (on behalf of the Apache Calcite PMC) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: Understanding annotations of SqlGroupingFunction
Filed a JIRA: https://issues.apache.org/jira/browse/CALCITE-3950
XING JIN 于2020年4月22日周三 下午2:51写道：
> Hi Vineet ~
> +1 on your analysis.
> Checking below case in agg.iq. We can see that the behavior of GROUPING
> function in Calcite is the same as Hive.
>
> # GROUPING in SELECT clause of CUBE query
> select deptno, job, count(*) as c, grouping(deptno) as d,
> grouping(job) j, grouping(deptno, job) as x
> from "scott".emp
> group by cube(deptno, job);
> ++---++---+---+---+
> | DEPTNO | JOB | C | D | J | X |
> ++---++---+---+---+
> | 10 | CLERK | 1 | 0 | 0 | 0 |
> | 10 | MANAGER | 1 | 0 | 0 | 0 |
> | 10 | PRESIDENT | 1 | 0 | 0 | 0 |
> | 10 | | 3 | 0 | 1 | 1 |
> | 20 | ANALYST | 2 | 0 | 0 | 0 |
> | 20 | CLERK | 2 | 0 | 0 | 0 |
> | 20 | MANAGER | 1 | 0 | 0 | 0 |
> | 20 | | 5 | 0 | 1 | 1 |
> | 30 | CLERK | 1 | 0 | 0 | 0 |
> | 30 | MANAGER | 1 | 0 | 0 | 0 |
> | 30 | SALESMAN | 4 | 0 | 0 | 0 |
> | 30 | | 6 | 0 | 1 | 1 |
> || ANALYST | 2 | 1 | 0 | 2 |
> || CLERK | 4 | 1 | 0 | 2 |
> || MANAGER | 3 | 1 | 0 | 2 |
> || PRESIDENT | 1 | 1 | 0 | 2 |
> || SALESMAN | 4 | 1 | 0 | 2 |
> || | 14 | 1 | 1 | 3 |
> ++---++---+---+---+
>
> IMHO, we might rectify the doc of SqlGroupingFunction as below:
>
> * 0 if both deptno and gender are being grouped,
> * 1 if only deptno is being grouped,
> * 2 if only gender is being groped,
> * 3 if neither deptno nor gender are being grouped.
>
>
> - Jin
>
>
> Vineet G 于2020年4月22日周三 上午5:18写道：
>
>> I expect that the user behavior for the GROUPING in both hive and calcite
>> is same. It’s just the documentation which is a bit confusing.
>> e.g. comment line on grouping : if both deptno and gender are being
>> grouped
>>
>> should really mean that the row which represents the grand total i.e
>> without group by expression. will return 3.
>>
>> FYI Hive had its behavior fixed with
>> https://issues.apache.org/jira/browse/HIVE-16102 <
>> https://issues.apache.org/jira/browse/HIVE-16102> and currently uses
>> GROUPING ID function.
>>
>> Vineet Garg
>>
>>
>> > On Apr 21, 2020, at 10:42 AM, Julian Hyde
>> wrote:
>> >
>> > Suppose we have one row that represents the total for department 10,
>> and another that represents the grand total of all departments. Which row
>> would we say that department is “grouped” (in Calcite’s parlance) or
>> “aggregated” in (Hive’s parlance)?
>> >
>> > I find the terms confusing. It’s possible that Calcite has them
>> “wrong”.
>> >
>> > I would simply run a query like
>> >
>> > SELECT deptno, job, GROUPING(deptno, job), COUNT(*)
>> > FROM emp
>> > GROUP BY CUBE (deptno, job)
>> >
>> > and see whether Hive and Calcite return the same result.
>> >
>> >> On Apr 20, 2020, at 6:58 PM, ZZY wrote:
>> >>
>> >> Hi, Hyde:
>> >> It's confused me that some annotations in
>> >> Calcite(org.apache.calcite.sql.fun.SqlGroupingFunction.java) :
>> >> /**
>> >> * The {@code GROUPING} function.
>> >> *
>> >> * Accepts 1 or more arguments.
>> >> * Example: {@code GROUPING(deptno, gender)} returns
>> >> * 3 if both deptno and gender are being grouped,
>> >> * 2 if only deptno is being grouped,
>> >> * 1 if only gender is being groped,
>> >> * 0 if neither deptno nor gender are being grouped.
>> >> *
>> >> * This function is defined in the SQL standard.
>> >> * {@code GROUPING_ID} is a non-standard synonym.
>> >> *
>> >> * Some examples are in {@code agg.iq}.
>> >> */
>> >>
>> >> The annotations above seems conflicts with other implementations like
>> Hive(
>> >>
>> https://cwiki.apache.org/confluence/display/Hive/Enhanced+Aggregation%2C+Cube%2C+Grouping+and+Rollup?spm=ata.13261165.0.0.528c6dfcXalQFy#EnhancedAggregation,Cube,GroupingandRollup-Groupingfunction
>> >> )
>> >>
>> >> Notice that: "The grouping function indicates whether an expression in
>> a
>> >> GROUP BY clause is aggregated or not for a given row. The value 0
>> >> represents a column that is part of the grouping set, while the value 1
>> >> represents a column that is not part of the grouping set. "
>> >>
>> >>
>> >> It is clearly that 0 and 1 bit have different interpretation between
>> >> annotations in Calcite and in Hive. And I did not figure out why...
>> >>
>> >> Any feedback can give me on this would be highly appreciated.
>> >>
>> >> Best regards!
>>
>>
Re: Understanding annotations of SqlGroupingFunction
Hi Vineet ~
+1 on your analysis.
Checking below case in agg.iq. We can see that the behavior of GROUPING
function in Calcite is the same as Hive.
# GROUPING in SELECT clause of CUBE query
select deptno, job, count(*) as c, grouping(deptno) as d,
grouping(job) j, grouping(deptno, job) as x
from "scott".emp
group by cube(deptno, job);
++---++---+---+---+
| DEPTNO | JOB | C | D | J | X |
++---++---+---+---+
| 10 | CLERK | 1 | 0 | 0 | 0 |
| 10 | MANAGER | 1 | 0 | 0 | 0 |
| 10 | PRESIDENT | 1 | 0 | 0 | 0 |
| 10 | | 3 | 0 | 1 | 1 |
| 20 | ANALYST | 2 | 0 | 0 | 0 |
| 20 | CLERK | 2 | 0 | 0 | 0 |
| 20 | MANAGER | 1 | 0 | 0 | 0 |
| 20 | | 5 | 0 | 1 | 1 |
| 30 | CLERK | 1 | 0 | 0 | 0 |
| 30 | MANAGER | 1 | 0 | 0 | 0 |
| 30 | SALESMAN | 4 | 0 | 0 | 0 |
| 30 | | 6 | 0 | 1 | 1 |
|| ANALYST | 2 | 1 | 0 | 2 |
|| CLERK | 4 | 1 | 0 | 2 |
|| MANAGER | 3 | 1 | 0 | 2 |
|| PRESIDENT | 1 | 1 | 0 | 2 |
|| SALESMAN | 4 | 1 | 0 | 2 |
|| | 14 | 1 | 1 | 3 |
++---++---+---+---+
IMHO, we might rectify the doc of SqlGroupingFunction as below:
* 0 if both deptno and gender are being grouped,
* 1 if only deptno is being grouped,
* 2 if only gender is being groped,
* 3 if neither deptno nor gender are being grouped.
- Jin
Vineet G 于2020年4月22日周三 上午5:18写道：
> I expect that the user behavior for the GROUPING in both hive and calcite
> is same. It’s just the documentation which is a bit confusing.
> e.g. comment line on grouping : if both deptno and gender are being
> grouped
>
> should really mean that the row which represents the grand total i.e
> without group by expression. will return 3.
>
> FYI Hive had its behavior fixed with
> https://issues.apache.org/jira/browse/HIVE-16102 <
> https://issues.apache.org/jira/browse/HIVE-16102> and currently uses
> GROUPING ID function.
>
> Vineet Garg
>
>
> > On Apr 21, 2020, at 10:42 AM, Julian Hyde
> wrote:
> >
> > Suppose we have one row that represents the total for department 10, and
> another that represents the grand total of all departments. Which row would
> we say that department is “grouped” (in Calcite’s parlance) or “aggregated”
> in (Hive’s parlance)?
> >
> > I find the terms confusing. It’s possible that Calcite has them “wrong”.
> >
> > I would simply run a query like
> >
> > SELECT deptno, job, GROUPING(deptno, job), COUNT(*)
> > FROM emp
> > GROUP BY CUBE (deptno, job)
> >
> > and see whether Hive and Calcite return the same result.
> >
> >> On Apr 20, 2020, at 6:58 PM, ZZY wrote:
> >>
> >> Hi, Hyde:
> >> It's confused me that some annotations in
> >> Calcite(org.apache.calcite.sql.fun.SqlGroupingFunction.java) :
> >> /**
> >> * The {@code GROUPING} function.
> >> *
> >> * Accepts 1 or more arguments.
> >> * Example: {@code GROUPING(deptno, gender)} returns
> >> * 3 if both deptno and gender are being grouped,
> >> * 2 if only deptno is being grouped,
> >> * 1 if only gender is being groped,
> >> * 0 if neither deptno nor gender are being grouped.
> >> *
> >> * This function is defined in the SQL standard.
> >> * {@code GROUPING_ID} is a non-standard synonym.
> >> *
> >> * Some examples are in {@code agg.iq}.
> >> */
> >>
> >> The annotations above seems conflicts with other implementations like
> Hive(
> >>
> https://cwiki.apache.org/confluence/display/Hive/Enhanced+Aggregation%2C+Cube%2C+Grouping+and+Rollup?spm=ata.13261165.0.0.528c6dfcXalQFy#EnhancedAggregation,Cube,GroupingandRollup-Groupingfunction
> >> )
> >>
> >> Notice that: "The grouping function indicates whether an expression in a
> >> GROUP BY clause is aggregated or not for a given row. The value 0
> >> represents a column that is part of the grouping set, while the value 1
> >> represents a column that is not part of the grouping set. "
> >>
> >>
> >> It is clearly that 0 and 1 bit have different interpretation between
> >> annotations in Calcite and in Hive. And I did not figure out why...
> >>
> >> Any feedback can give me on this would be highly appreciated.
> >>
> >> Best regards!
>
>
Re: The Travis CI build failure
I have force pushed several times, but it keeps failing.
Chunwei Lei 于2020年4月20日周一 上午10:49写道：
> I am a little confused too. Sometimes the traivs ci build of master fails
> when I merge
> the pr which passes all tests.
>
>
>
> Best,
> Chunwei
>
>
> On Mon, Apr 20, 2020 at 10:33 AM XING JIN wrote:
>
> > In PR https://github.com/apache/calcite/pull/1926 and
> > https://github.com/apache/calcite/pull/1925, The Travis CI build keeps
> > failing with below messages. The exception has nothing to do with the
> > change in PR. There seems to be something wrong with the building
> > environment.
> >
> > ...
> >
> >
> /home/travis/build/apache/calcite/spark/src/main/java/org/apache/calcite/adapter/spark/HttpServer.java:42:
> > error: cannot access java.lang
> > class HttpServer {
> > ^
> > zip END header not found
> >
> >
> /home/travis/build/apache/calcite/spark/src/main/java/org/apache/calcite/adapter/spark/HttpServer.java:43:
> > error: cannot find symbol
> > private static String localIpAddress;
> > ^
> > symbol: class String
> > location: class HttpServer
> >
> >
> /home/travis/build/apache/calcite/spark/src/main/java/org/apache/calcite/adapter/spark/HttpServer.java:45:
> > error: cannot find symbol
> > private final File resourceBase;
> > ^
> > symbol: class File
> > location: class HttpServer
> > ...
> >
> > Jin
> >
>
The Travis CI build failure
In PR https://github.com/apache/calcite/pull/1926 and
https://github.com/apache/calcite/pull/1925, The Travis CI build keeps
failing with below messages. The exception has nothing to do with the
change in PR. There seems to be something wrong with the building
environment.
...
/home/travis/build/apache/calcite/spark/src/main/java/org/apache/calcite/adapter/spark/HttpServer.java:42:
error: cannot access java.lang
class HttpServer {
^
zip END header not found
/home/travis/build/apache/calcite/spark/src/main/java/org/apache/calcite/adapter/spark/HttpServer.java:43:
error: cannot find symbol
private static String localIpAddress;
^
symbol: class String
location: class HttpServer
/home/travis/build/apache/calcite/spark/src/main/java/org/apache/calcite/adapter/spark/HttpServer.java:45:
error: cannot find symbol
private final File resourceBase;
^
symbol: class File
location: class HttpServer
...
Jin
Re: How to get the used columns (categorized by tables) from a SQL
You can use SqlValidator.getFieldOrigins for a SqlNode, or use
RelMetadataQuery.getColumnOrigins for a RelNode.
- Jin
Julian Hyde 于2020年4月17日周五 上午7:29写道：
> At the RelNode level, you can use RelMetadataQuery.getColumnOrigins.
>
> But I’m sure that there’s something available at the SqlNode level, after
> validation. PreparedExplain.getFieldOrigins() is what the JDBC driver uses
> to provide ResultSetMetadata.getColumnName(int) etc.
>
> Julian
>
>
>
> > On Apr 16, 2020, at 4:17 PM, Feng Zhu wrote:
> >
> > You can customize a RelVisitor to visit the RelNode, and collect the
> > columns from TableScan's rowtype.
> >
> > 王驰 于2020年4月16日周四 下午11:23写道：
> >
> >> Hi folks,
> >>
> >>
> >> We're using CalCite in our project for SQL parsing and validating. We
> have
> >> a question: is there any way to get all used columns from a SQL?
> >>
> >>
> >> Consider the following case:
> >>
> >> we have two tables `user` and `user_region`. And the SQL is like
> >>
> >>
> >> ```
> >>
> >> SELECT
> >>
> >>id, name, age, country, province, city
> >>
> >> FROM
> >>
> >>user
> >>
> >>INNER JOIN user_region ON user.id = user_region.user_id
> >>
> >> WHERE age > 18;
> >>
> >> ```
> >>
> >>
> >> The result will be a Map with two keys:
> >>
> >>
> >> - 'user' --> List('id', 'name', 'age')
> >> - 'user_region' --> List('country', 'province', 'city', 'user_id')
> >>
> >>
> >> ==
> >>
> >> I've tried with SqlValidator (along with SqlValidatorScope,
> >> SqlValidatorNamespace) but found no easy ways to do this.
> >> I also tried to replay the validate process in our code but it seems
> >> impossible since most classes used are package private.
> >>
> >>
> >> Could you please give us some suggestions? Thanks!
> >>
> >>
>
>
Re: Cannot parse query that contains colons for casting in postgres sql
Hi, Rick ~
Babel parser supports casting by double colon [1]
[1] https://issues.apache.org/jira/browse/CALCITE-2843
-Jin
Rick Liao 于2020年1月9日周四 上午6:05写道：
> Hello all,
>
> I'm trying to parse a postgres sql query that contains :: for casting. The
> parser fails when it reaches the colon. I'm using version 1.21.0. This is
> what I'm running (with a simple query to showcase the error):
>
> SqlParser.Config parserConfig =
>
> SqlParser.configBuilder().setConformance(SqlConformanceEnum.LENIENT).build();
> SqlParser parser1 = SqlParser.create("SELECT 'test'::text t from
> countries", parserConfig);
> SqlNode parsedQuery1 = parser1.parseQuery();
>
> The error:
>
> org.apache.calcite.sql.parser.SqlParseException: Encountered ":" at line 1,
> column 14.
>
> Was expecting one of:
>
>
>
> "AS" ...
>
> "EXCEPT" ...
>
> "FETCH" ...
>
> "FROM" ...
>
> "INTERSECT" ...
>
> "LIMIT" ...
>
> "OFFSET" ...
>
> "ORDER" ...
>
> "MINUS" ...
>
> "UESCAPE" ...
>
> "UNION" ...
>
> ...
>
> "," ...
>
> ...
>
> ...
>
> ...
>
> ...
>
> ...
>
> "." ...
>
> "NOT" ...
>
> "IN" ...
>
> "<" ...
>
> "<=" ...
>
> ">" ...
>
> ">=" ...
>
> "=" ...
>
> "<>" ...
>
> "!=" ...
>
> "BETWEEN" ...
>
> "LIKE" ...
>
> "SIMILAR" ...
>
> "+" ...
>
> "-" ...
>
> "*" ...
>
> "/" ...
>
> "%" ...
>
> "||" ...
>
> "AND" ...
>
> "OR" ...
>
> "IS" ...
>
> "MEMBER" ...
>
> "SUBMULTISET" ...
>
> "CONTAINS" ...
>
> "OVERLAPS" ...
>
> "EQUALS" ...
>
> "PRECEDES" ...
>
> "SUCCEEDS" ...
>
> "MULTISET" ...
>
> "[" ...
>
>
>
> at
>
> org.apache.calcite.sql.parser.impl.SqlParserImpl.convertException(SqlParserImpl.java:343)
>
> at
>
> org.apache.calcite.sql.parser.impl.SqlParserImpl.normalizeException(SqlParserImpl.java:142)
>
> at
> org.apache.calcite.sql.parser.SqlParser.handleException(SqlParser.java:147)
>
> at org.apache.calcite.sql.parser.SqlParser.parseQuery(SqlParser.java:162)
>
> Perhaps, I have just not found the right conformance for postgres sql.
>
> Thanks for your help!
>
> Rick
>
Re: [DISCUSS] CALCITE-3661, CALCITE-3665, MaterializationTest vs HR schema statistics
Hi, Vladimir ~ In ReflectiveSchema, Statistics of FieldTable is given as UNKNOWN[1][2]. When reading a table's row count, if no statistics given, a default value of 100 will be returned [3] -- this is relatively a bigger value compared with the fields defined in HRFKUKSchema. When a materialized view gets matched, view-sql is executed and the values are wrapped in an ArrayTable and accurate row count is given [4]. So I'm not sure when a materialized view containing JOIN gets matched but cannot help reduce cost of the plan. HRFKUKSchema is only used in MaterializationTest. There's no existing test checking content of the query result. Most of them checks whether same results are returned no matter if materialized view is used or not. If we add rows to existing emps table, how can tests be invalidated ? Best, Jin [1] https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/adapter/java/ReflectiveSchema.java#L369 [2] https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/schema/Statistics.java#L40 [3] https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/RelOptTableImpl.java#L239 [4] https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/adapter/clone/ArrayTable.java#L82 Vladimir Sitnikov 于2020年1月4日周六 上午4:21写道： > Hi, > > It looks like MaterializationTest heavily relies on inaccurate statistics > for hr.emps and hr.depts tables. > > I was trying to improve statistic estimation for better join planning (see > https://github.com/apache/calcite/pull/1712 ), > and it looks like better estimates open the eyes of the optimizer, and now > it realizes it does not really need to use materialized view > for 4-row long table. > > In other words, the cost of the table access is more-or-less the same as > the cost of the materialized view access. > > It looks like the way to go here is to add hr_with_extra_rows scheme so it > contains the same emps and depts tables, but it should > have bigger tables. > Adding rows to the existing emps table is not an option because it would > invalidate lots of tests. > > Does anybody have better ideas? > > Vladimir >
Re: Re: [ANNOUNCE] New Calcite PMC chair: Stamatis Zampetakis
Congratulations Stamatis! -Jin Chunwei Lei 于2019年12月19日周四 上午10:33写道： > Congratulations Stamatis! > > > Best, > Chunwei > > > On Thu, Dec 19, 2019 at 9:36 AM Danny Chan wrote: > > > Congratulations Stamatis! > > > > Best, > > Danny Chan > > 在 2019年12月19日 +0800 AM7:37，dev@calcite.apache.org，写道： > > > > > > Congratulations Stamatis! > > >
Re: [DISCUSS] Move CalciteAssert and similar "test framework" classes to its own testlib module
Hi, Vladimir
> I suggest we do not publish testlib artifacts to Nexus (who needs
> Calcite-related assertions and things like that?)
+1 to have the module be published.
Some of my users write tests based on functionalities provided from Calcite
test framework.
Best,
Jin
Michael Mior 于2019年12月18日周三 上午7:47写道：
> Sounds good to me. Although I'd still like to see the module be
> published. I'm currently using it my notebooks project
> (https://github.com/michaelmior/calcite-notebooks) because some of the
> test code removes boilerplate when writing examples.
> --
> Michael Mior
> mm...@apache.org
>
> Le mar. 17 déc. 2019 à 15:49, Vladimir Sitnikov
> a écrit :
> >
> > Hi,
> >
> > Currently
> > org.apache.calcite.test.CalciteAssert, org.apache.calcite.util.TestUtil,
> > and probably other classes are hard to find.
> > I suggest we create a testlib module (or something like that), that would
> > contain "test framework" code.
> >
> > That would make test framework easier to discover (one could open the
> full
> > set of packages in the testlib),
> > and it would make "autocomplete" better (especially for adapters).
> >
> > I suggest we do not publish testlib artifacts to Nexus (who needs
> > Calcite-related assertions and things like that?)
> >
> > Currently testImplementation(project(":core", "testClasses")) is used in
> > lots of places, and it causes "core"
> > test clases appear in autocomplete menu for adapters.
> > For example, typing "new Sql..." in GeodeAssertions suggests
> > SqlAdvisorJdbcTest which is valid, yet it is weird.
> >
> > What do you think?
> >
> > Just in case you did not know: file movement in Git keeps history (even
> in
> > case there are slight modifications).
> > However, the key idea here is to keep files in their current packages,
> but
> > move them into teslib module.
> >
> > Vladimir
>
Re: Problems to validate ddl commands
Hi Anthony ~ Calcite supports to validate DDL; I cannot open your link, would you please give some details for your problem ? Best, Jin Anthony Louis 于2019年12月17日周二 上午4:08写道： > When I try to validate a sql statement that contains "CREATE TABLE" it > throws an error that is described in this gist > https://gist.github.com/anthonylouisbsb/5f7d57886836cfac1742366722afcf41. > I > would like to know if is possible to validate a query using ddl statements? >
Re: [DISCUSS] Tests vs multiline strings
Before coming to Calcite, I works quite some time on Scala. The code style shares some similarities with Kotlin -- It's much simple and easier to read when you write a test. But we should think twice whether to bring in another language. To answer Haisheng's question: Because default lex config is Lex.ORACLE. Best, Jin Chunwei Lei 于2019年12月16日周一 上午10:34写道： > I agree with Julian. > > Changing to Kotlin needs lots of error, but gets a little gain. Besides, It > costs much more > time to write a test if developers are not familiar with Kotlin. I > prefer to use Java as it is now. > > > Best, > Chunwei > > > On Mon, Dec 16, 2019 at 9:02 AM Julian Hyde wrote: > > > I don't think we should do this. > > > > Multi-line strings are a bit unwieldy, but they're not a major > > problem. Transitioning our tests to a different language (Kotlin) is a > > drastic solution. It requires developers to understand a new language, > > and it loses all history from the source files. > > > > Julian > > > > On Sun, Dec 15, 2019 at 4:37 AM Vladimir Sitnikov > > wrote: > > > > > > I've filed two PRs to evaluate the impact of the replacement. > > > > > > $ to _: https://github.com/apache/calcite/pull/1659 > > > 203.3sec, 5510 completed, 3 failed, 91 skipped, Gradle Test Run > > > :core:test > > > > > > $ to #: https://github.com/apache/calcite/pull/1660 > > > 196.7sec, 5510 completed, 53 failed, 91 skipped, Gradle Test Run > > > :core:test > > > > > > There are test failures, however, both of them are almost ready. > > > > > > Both $ and _ are valid "Java identifier start" characters, so variable > > name > > > like _3 is valid. > > > If we use # instead of $, then code generator needs to be updated as > well > > > as it sometimes uses Java variables like $3, and #3 would become > invalid. > > > > > > Any thoughts? > > > > > > Vladimir > > >
Re: CI passed error tests
Thanks a lot, Stamatis~ I will run slow tests from my local. Best, Jin 在 2019年12月14日星期六，Stamatis Zampetakis 写道： > Hi Jin, > > If the test run fine locally at your machine (e.g., ./gradlew testSlow) > then they should be fine in the CI. > Labels are only available for committers yes! > > Best, > Stamatis > > On Fri, Dec 13, 2019 at 4:52 PM XING JIN wrote: > > > Thanks a lot Ruben ~ > > I'm working on "[CALCITE-3478] Restructure tests for materialized views" > ( > > https://github.com/apache/calcite/pull/1560) , which contains "slow > > tests". > > But I'm not sure how to add the label to PR. Is it only available for > > committers ? > > BTW, It's great if someone can help review > > https://github.com/apache/calcite/pull/1560 . I've rebased the PR to > > resolve conflicts for several times. > > > > Best, > > Jin > > > > Ruben Q L 于2019年12月13日周五 下午4:00写道： > > > > > Hi Jin, > > > > > > this is the expected behavior: slow tests are not executed by default > in > > PR > > > CI. > > > If you want slow tests to be executed in your PR, you need to > explicitly > > > add the label "slow-tests-needed", as specified by [1]. > > > You can see a PR example with this label here [2]. > > > > > > Best regards, > > > Ruben > > > > > > [1] https://issues.apache.org/jira/browse/CALCITE-3141 > > > [2] https://github.com/apache/calcite/pull/1651 > > > > > > > > > Le ven. 13 déc. 2019 à 08:41, XING JIN a > > écrit : > > > > > > > Hi guys, > > > > I made a PR and run continuous integration tests. [1] > > > > A error test contained in the PR and tagged with @slowTest. > > > > The tests should be failed but CI passed by mistake. > > > > I doubt our current CI is not running with 'testSlow' configuration. > > > Isn't > > > > it ? > > > > I'm not sure if I should create a JIRA. > > > > > > > > Best, > > > > Jin > > > > > > > > [1] https://github.com/apache/calcite/pull/1653 > > > > > > > > > >
Re: CI passed error tests
Thanks a lot Ruben ~ I'm working on "[CALCITE-3478] Restructure tests for materialized views" ( https://github.com/apache/calcite/pull/1560) , which contains "slow tests". But I'm not sure how to add the label to PR. Is it only available for committers ? BTW, It's great if someone can help review https://github.com/apache/calcite/pull/1560 . I've rebased the PR to resolve conflicts for several times. Best, Jin Ruben Q L 于2019年12月13日周五 下午4:00写道： > Hi Jin, > > this is the expected behavior: slow tests are not executed by default in PR > CI. > If you want slow tests to be executed in your PR, you need to explicitly > add the label "slow-tests-needed", as specified by [1]. > You can see a PR example with this label here [2]. > > Best regards, > Ruben > > [1] https://issues.apache.org/jira/browse/CALCITE-3141 > [2] https://github.com/apache/calcite/pull/1651 > > > Le ven. 13 déc. 2019 à 08:41, XING JIN a écrit : > > > Hi guys, > > I made a PR and run continuous integration tests. [1] > > A error test contained in the PR and tagged with @slowTest. > > The tests should be failed but CI passed by mistake. > > I doubt our current CI is not running with 'testSlow' configuration. > Isn't > > it ? > > I'm not sure if I should create a JIRA. > > > > Best, > > Jin > > > > [1] https://github.com/apache/calcite/pull/1653 > > >
CI passed error tests
Hi guys, I made a PR and run continuous integration tests. [1] A error test contained in the PR and tagged with @slowTest. The tests should be failed but CI passed by mistake. I doubt our current CI is not running with 'testSlow' configuration. Isn't it ? I'm not sure if I should create a JIRA. Best, Jin [1] https://github.com/apache/calcite/pull/1653
Re: lex parameter does not get taken into account for views
Filed a JIRA: https://issues.apache.org/jira/browse/CALCITE-3549
Danny Chan 于2019年12月2日周一 上午9:06写道：
> Dear Erin ~
>
> You are right, Calcite now always hard code the parser config for JDBC
> query[1], that means, Lex config for view expanding is not supported yet,
> can you log a issue and probably fire a patch to support that?
>
> [1]
> https://github.com/apache/calcite/blob/ab136b5f76a4cb951e847fcba6b414c5e80dbbe6/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L385
>
> Best,
> Danny Chan
> 在 2019年12月2日 +0800 AM4:19，Erin Drummond ，写道：
> > Hello,
> >
> > I am currently attempting to test out the views feature of Calcite.
> >
> > I am using a connection string like so:
> > jdbc:calcite:lex=JAVA;model=inline:
> >
> > I have created a JdbcSchema that is attached to an actual database - lets
> > call it "t1". I can successfully run the query "select * from t1.table
> > limit 10"
> >
> > I then created another schema, defined like so:
> > {
> > "name": "viewtest",
> > "tables": [
> > { "name": "test_view", "type": "view", "sql": "select * from
> > t1.table limit 10" }
> > ]
> > }
> >
> > The idea is to set up connections to a bunch of physical data sources and
> > then have a "views" schema that just contains views over them.
> >
> > However, when attempting to run the query "select * from
> > viewtest.test_view", I get the following exception:
> >
> > StatementCallback; uncategorized SQLException for SQL [select * from
> > viewtest.test_view]; SQL state [null]; error code [0];
> > Error while executing SQL \"select * from viewtest.test_view\": From line
> > 1, column 15 to line 1, column 32: Object 'T1' not found; did you mean
> 't1'?
> >
> > I can clearly see that lex=JAVA is not being used for the view and the
> > default lex=ORACLE is being used. If I change the view SQL to 'select *
> > from "t1".table limit 10' it works - but I don't want to have to quote
> the
> > identifiers oracle style, I would like the lex=JAVA that I specified on
> the
> > connection string to flow down to views as well.
> >
> > What am I missing here?
> >
> > Cheers,
> > Erin
>
Re: [ANNOUNCE] Haisheng Yuan joins Calcite PMC
Congratulations Haisheng ~ You well deserved ! Kevin Risden 于2019年11月12日周二 上午3:13写道： > Congrats and welcome! > > Kevin Risden > > > On Mon, Nov 11, 2019 at 2:10 PM Chunhui Shi wrote: > > > Congratulations! > > > > On Mon, Nov 11, 2019 at 10:09 AM Jinfeng Ni wrote: > > > > > Congratulations! > > > > > > > > > On Tue, Nov 12, 2019 at 1:23 AM Rui Wang wrote: > > > > > > > > Congrats HaiSheng! > > > > > > > > > > > > -Rui > > > > > > > > On Mon, Nov 11, 2019 at 8:05 AM Stamatis Zampetakis < > zabe...@gmail.com > > > > > > > wrote: > > > > > > > > > Congrats Haisheng! > > > > > > > > > > Reviews, code contributions, design discussions, helping users, and > > > many > > > > > more things for improving the project. > > > > > > > > > > Personally, I also learn a lot from our interactions. > > > > > > > > > > All these are much appreciated; keep it up!! > > > > > > > > > > Best, > > > > > Stamatis > > > > > > > > > > On Mon, Nov 11, 2019, 4:17 PM Michael Mior > wrote: > > > > > > > > > > > Welcome and congratulations HaiSheng! > > > > > > -- > > > > > > Michael Mior > > > > > > mm...@apache.org > > > > > > > > > > > > Le dim. 10 nov. 2019 à 22:45, Francis Chuang > > > > > > a écrit : > > > > > > > > > > > > > > I'm pleased to announce that Haisheng has accepted an > invitation > > to > > > > > > > join the Calcite PMC. Haisheng has been a consistent and > helpful > > > > > > > figure in the Calcite community for which we are very grateful. > > We > > > > > > > look forward to the continued contributions and support. > > > > > > > > > > > > > > Please join me in congratulating Haisheng! > > > > > > > > > > > > > > - Francis (on behalf of the Calcite PMC) > > > > > > > > > > > > > > > > >
Re: Question about Interpreter and Corelations
You may modify your sql and operator Collect should not reference correlate
variables.
XING JIN 于2019年11月9日周六 下午12:32写道：
> Currently RelDecorrelator doesn't support a decorrelateRel(Collect),
> that's why decorrelate failed
>
> Zoltan Farkas 于2019年11月9日周六 上午5:46写道：
>
>> Done: https://issues.apache.org/jira/browse/CALCITE-3488 <
>> https://issues.apache.org/jira/browse/CALCITE-3488>
>> let me know if you need any more info.
>>
>> thank you
>>
>> —Z
>>
>> > On Nov 8, 2019, at 2:17 PM, Julian Hyde wrote:
>> >
>> > As I said, I think you should log a bug.
>> >
>> > On Fri, Nov 8, 2019 at 10:23 AM Haisheng Yuan
>> wrote:
>> >>
>> >> I am afraid this query can't be easily decorrelated.
>> >>
>> >> - Haisheng
>> >>
>> >> --
>> >> 发件人：Zoltan Farkas
>> >> 日 期：2019年11月08日 22:46:53
>> >> 收件人：
>> >> 主 题：Re: Question about Interpreter and Corelations
>> >>
>> >> Thanks Julian,
>> >>
>> >> Any idea how could I de-corelate the query?
>> >> RelDecorrelator.decorrelateQuery does not seem to alter the plan.
>> >>
>> >> Will log a jira for the interpreter issue.
>> >>
>> >> thank you
>> >>
>> >> —Z
>> >>
>> >>
>> >>
>> >>> On Nov 7, 2019, at 5:08 PM, Julian Hyde
>> wrote:
>> >>>
>> >>> I don’t recall what the interpreter is currently capable of, but you
>> should log a bug.
>> >>>
>> >>> I wonder whether if you could get the query to work in the
>> interpreter if you decorrelated the query first.
>> >>>
>> >>> Julian
>> >>>
>> >>>> On Nov 7, 2019, at 11:16, Zoltan Farkas
>> wrote:
>> >>>>
>> >>>> for a test query with the following plan:
>> >>>>
>> >>>> LogicalProject(name=[$1], friends=[$4])
>> >>>> LogicalCorrelate(correlation=[$cor0], joinType=[inner],
>> requiredColumns=[{0}])
>> >>>> LogicalTableScan(table=[[characters]])
>> >>>> Collect(field=[EXPR$0])
>> >>>>LogicalProject(name=[$2])
>> >>>> LogicalJoin(condition=[=($0, $1)], joinType=[inner])
>> >>>>LogicalProject(characterId2=[$1])
>> >>>> LogicalFilter(condition=[=($0, $cor0.characterId)])
>> >>>> LogicalTableScan(table=[[friendships]])
>> >>>>LogicalProject(characterId=[$0], name=[$1])
>> >>>> LogicalTableScan(table=[[characters]])
>> >>>>
>> >>>> I get :
>> >>>>
>> >>>> java.lang.nsupportedOperationException
>> >>>> at
>> o.a.c.i.JaninoRexCompiler.lambda$compile$0(JaninoRexCompiler.java:94)[calcite-core-1.21.0.jar:1.21.0]
>> >>>> at
>> o.a.c.a.e.RexToLixTranslator.translate0(RexToLixTranslator.java:714)[^]
>> >>>> at ^.translate(^:199)[^]
>> >>>> at ^.translate0(^:684)[^]
>> >>>> at ^.translate(^:199)[^]
>> >>>> at ^.translate(^:194)[^]
>> >>>> ...
>> >>>>
>> >>>> when trying to use the Interpreter.
>> >>>>
>> >>>> Is this a current limitation?
>> >>>>
>> >>>> the query is :
>> >>>>
>> >>>> select name,
>> >>>>ARRAY(select c2.name from friendships f, characters c2
>> >>>> where f.characterId1 = c.characterId and
>> f.characterId2 = c2.characterId) as friends
>> >>>> from characters c
>> >>>>
>> >>>> let me know
>> >>>>
>> >>>> thank you
>> >>>>
>> >>>> —Z
>>
>>
Re: Question about Interpreter and Corelations
Currently RelDecorrelator doesn't support a decorrelateRel(Collect), that's
why decorrelate failed
Zoltan Farkas 于2019年11月9日周六 上午5:46写道：
> Done: https://issues.apache.org/jira/browse/CALCITE-3488 <
> https://issues.apache.org/jira/browse/CALCITE-3488>
> let me know if you need any more info.
>
> thank you
>
> —Z
>
> > On Nov 8, 2019, at 2:17 PM, Julian Hyde wrote:
> >
> > As I said, I think you should log a bug.
> >
> > On Fri, Nov 8, 2019 at 10:23 AM Haisheng Yuan
> wrote:
> >>
> >> I am afraid this query can't be easily decorrelated.
> >>
> >> - Haisheng
> >>
> >> --
> >> 发件人：Zoltan Farkas
> >> 日 期：2019年11月08日 22:46:53
> >> 收件人：
> >> 主 题：Re: Question about Interpreter and Corelations
> >>
> >> Thanks Julian,
> >>
> >> Any idea how could I de-corelate the query?
> >> RelDecorrelator.decorrelateQuery does not seem to alter the plan.
> >>
> >> Will log a jira for the interpreter issue.
> >>
> >> thank you
> >>
> >> —Z
> >>
> >>
> >>
> >>> On Nov 7, 2019, at 5:08 PM, Julian Hyde
> wrote:
> >>>
> >>> I don’t recall what the interpreter is currently capable of, but you
> should log a bug.
> >>>
> >>> I wonder whether if you could get the query to work in the interpreter
> if you decorrelated the query first.
> >>>
> >>> Julian
> >>>
> On Nov 7, 2019, at 11:16, Zoltan Farkas
> wrote:
>
> for a test query with the following plan:
>
> LogicalProject(name=[$1], friends=[$4])
> LogicalCorrelate(correlation=[$cor0], joinType=[inner],
> requiredColumns=[{0}])
> LogicalTableScan(table=[[characters]])
> Collect(field=[EXPR$0])
> LogicalProject(name=[$2])
> LogicalJoin(condition=[=($0, $1)], joinType=[inner])
> LogicalProject(characterId2=[$1])
> LogicalFilter(condition=[=($0, $cor0.characterId)])
> LogicalTableScan(table=[[friendships]])
> LogicalProject(characterId=[$0], name=[$1])
> LogicalTableScan(table=[[characters]])
>
> I get :
>
> java.lang.nsupportedOperationException
> at
> o.a.c.i.JaninoRexCompiler.lambda$compile$0(JaninoRexCompiler.java:94)[calcite-core-1.21.0.jar:1.21.0]
> at
> o.a.c.a.e.RexToLixTranslator.translate0(RexToLixTranslator.java:714)[^]
> at ^.translate(^:199)[^]
> at ^.translate0(^:684)[^]
> at ^.translate(^:199)[^]
> at ^.translate(^:194)[^]
> ...
>
> when trying to use the Interpreter.
>
> Is this a current limitation?
>
> the query is :
>
> select name,
> ARRAY(select c2.name from friendships f, characters c2
> where f.characterId1 = c.characterId and
> f.characterId2 = c2.characterId) as friends
> from characters c
>
> let me know
>
> thank you
>
> —Z
>
>
Re: [DISCUSS] Proposal to add API to force rules matching specific rels
Hi Vladimir, I think the way PlannerTests#GoodSingleRule and EnumerableXXXRule work may help you ~ They work by a top-down fashion, but when matching parent, they convert the children explicitly. You may try below steps: 1. Construct a rule LogicalParentRule to match the LogicalParent without distribution/physical requirement for the LogicalChild; 2. In this rule, you call 'planner.changeTraits' on the LogicalChild to build a new child with physical convention. Note that at this moment only an empty RelSubset is created and no PhysicalChild exists. 3. Then set the RelNode to be the new input of LogicalParent; By above steps, you can build a parent-child relationship between LogicalParent and PhysicalChild, and at last the PhysicalParentRule will be fired based on this relationship. I have a commit to illustrate my idea, check VolcanoPlannerTest#testDEV in below link, hope it may help you ~ https://github.com/jinxing64/calcite/tree/demo Also I'm +1 with Seliverstov that to get all parents of a set, which against the current check in RelSubset#getParentRels Best, Jin Vladimir Ozerov 于2019年11月5日周二 下午6:41写道： > Hi Xiening, > > I read the thread about on-demand trait requests. It seems pretty similar > to what I am trying to achieve, as it facilitates the bottom-up propagation > of physical traits. In fact, both your and my strategy propagate traits > bottom-up, but I do this through rules, which also fire bottom-up, while in > your case only the traits are propagated bottom-up, while rules continue > working in a top-down fashion. > > However, I am thinking of how I would potentially implement my optimizer > with your approach, and it feels like with on-demand traits resulting > implementation of metadata queries may become very complex to that point > that it will look like another set of rules, parallel to the already > existing ruleset. For example, consider that I have a couple of distributed > tables in an OLTP application. These tables have a number of indexes, and I > would like to join them. First, I have a number of choices on how to join > tables with respect to distribution. Then, I have a number of choices on > which access method to use. Because sometimes it is beneficial to pick > index scans instead of table scans even without index conditions, for > example, to preserve a comfortable collation. So when my logical scan > receives such metadata request, it typically cannot return all possible > combinations, because there are too many of them. Instead, some heuristical > or cost-based logic will be used to calculate a couple of most prospective > ones. But it seems that we will have to duplicate the same logic in the > corresponding rule, aren't we? > > I would love to read your design because this is a really interesting > topic, and it is of great importance for the distributed engines developed > on top of Calcite since proper use of distribution and collation is the key > success factor for efficient query optimization. > > Regards, > Vladimir. > > пт, 1 нояб. 2019 г. в 00:40, Xiening Dai : > > > Actually we solved this problem in our setup using a mechanism called > > “Pull-Up Traits”, which explores the possible trait set of children’s > input > > to decide parent’s physical properties. In order to determine child input > > trait, you would have to look at child’s children, and all the way to the > > leaves nodes or a barrier. A barrier is a rel node which cannot derive > any > > traits regardless the input. A good example would be a user define > function > > which would throw off any distribution or collation. Then we realize just > > pulling up is not enough, sometimes we would need to look at parent’s > > requirement as well. So we try to solve this in a unified framework, > which > > we call “On Demand Trait” and implement it as part of the framework so > > anyone can be benefited. I hope Haisheng can share a design doc once we > > have more concrete ideas. > > > > > > > On Oct 31, 2019, at 11:37 AM, Jinfeng Ni wrote: > > > > > > Hi Vladimir, > > > > > > The SubsetTransformer interface and the iterating over the RelNodes > > > within a RelSubset in Drill is exactly implemented to do the trait > > > propagation. We also had to rely on AbstractConverter to fire > > > necessary rule to avoid the CanNotPlan issue. At some point, Calcite > > > community chooses to remove AbstractConverter and Drill had to add it > > > back, which is probably one of the main reasons for us to continue > > > using a Calcite fork. I still remember we constantly had to deal with > > > the dilemma between "CanNotPlan" and long planing time due to explored > > > search space. > > > > > > Glad to see more people are joining the effort to solve this long > > > overdue issue, something missing in Calcite's core optimizer framework > > > "since before Calcite was Calcite" (Jacques's words). > > > > > > Jinfeng > > > > > > > > > On Thu, Oct 31, 2019 at 3:38 AM Vladimir Ozerov > > wrote: > > >> > > >> Hi
Re: Re: [DISCUSS] On-demand traitset request
Hi Haisheng, Thanks a lot for sharing this great proposal ~ For short I understand your idea as below: 1. Derive the distributions/collations that children COULD/MIGHT offer 2. Decide the best distributions/collations by first point and computing logic of operator, say gropuings in Aggregate; It comes to me that another important part is that children operator should also provide the corresponding COST for the POSSIBLE distribution/collation. The COST is not for the final plan, but a hypothesis. Take below example SELECT DISTINCT c, b FROM ( SELECT R.c c, S.b b FROM R, S WHERE R.a=S.a and R.b=S.b and R.c=S.c) t; Suppose R is ordered by (c, b, a), and S is ordered by (b, c, a). Aggregate +--- InnerJoin |--- TableScan on R +--- TableScan on S InnerJoin should deliver that its possible collations and corresponding costs at the same time. - If ordered by (c, b, a) my cost is ... - If ordered by (b, c, a) my cost is ... - If ordered by (a, b, c) my cost is ... By which Aggregate decide the 'best' required collation. By this way we can better limit the searching space and also target the relatively optimized (if not best) plan. Also when you say "I didn't say adding to RelNode, but a new API/interface for physical operator only.", I'm not so clear; Currently the physical operators in Calcite like EnumerableHashJoin, EnumerableMergeJoin, when created, their physical behavior(like real collations) are determined. So I belive you intend to add new API at upper layer, but there's no physical optimizing phase in Calcite at this moment. Where do you want to add the new API, can you specify ? Thanks, Jin Jinfeng Ni 于2019年11月6日周三 上午1:56写道： > @Haisheng, @Xiening, > > Thanks for pointing that previous email out. Overall, I agree that > the physical trait enforcement should be done in the engine, not in > the rule. For the rule, it should only specify the request, and the > corresponding transformation, and let the engine to explore the search > space. It will be great if we can revamp the Volcano optimizer > framework, to do that way. > > In terms of search space, it's always a tradeoff between the space > searched and the optimality of the plan found. I think it's fine for > the engine to explore a potential big search space, as long as it has > effective "bound-and-prune" strategy. In the original Volcano paper, > there is a way to prune the search space based on the best plan found > so far, using the parameter "limit". When an implementable plan is > found, a "real" cost is obtained, which could be used to prune > un-necessary search space. That's actually the advantage of Volcano's > "top-down" approach. However, seems to me that Calcite's Volcano did > not apply that approach effectively, because of the existence of > AbstractConverter. > > > On Sun, Nov 3, 2019 at 10:12 PM Haisheng Yuan > wrote: > > > > Hi Jinfeng, > > > > I think you might have missed the email about proposed API for physical > operators I sent out previously in [1]. > > > > We don't need request all the permutation, which is also impossible in > practice, the search space is going to explode. > > > > In the example in email [1], I already talked about your concen on > passing down parent request into children may lead to less optimal plan. > Besically join operator can send 2 collation optimization requests, one is > to pass request down, the other one is ignore the parent's request. > > > > Using AbstractConverter to enforce properties is inapporpriate, which > handles all the optimization work to physical operator providers, meaning > there is almost no physical level optimization mechanism in Calcite. SQL > Server and Greenplum's optimizer, which are Cascades framework based, > implemented the property enforcement in the core optimizer engine, not > through AbstractConverter and rules, physical operators just need to > implement those methods (or similar) I mentioned in email [1]. My goal is > completely abolishing AbstractConverter. > > > > [1] > http://mail-archives.apache.org/mod_mbox/calcite-dev/201910.mbox/%3cd75b20f4-542a-4a73-897e-66ab426494c1.h.y...@alibaba-inc.com%3e > > > > - Haisheng > > > > -- > > 发件人：Jinfeng Ni > > 日 期：2019年11月01日 14:10:30 > > 收件人： > > 主 题：Re: [DISCUSS] On-demand traitset request > > > > Hi Xiening, > > > > "Let say if R and S doesn’t have sorting properties at all. In your > > case, we would end up adding enforcers for LHS and RHS to get > > collation (a, b, c). Then we would need another enforcer to get > > collation (b, c). This is a sub optimal plan as we could have use (b, > > c, a) for join." > > > > In such case, for step 2 when MergeJoin request a permutation match of > > (a, b,c) on both it's input, it is not necessary to end up with > > collation (a, b, c) only. Since it request "permutation", MJ could ask > > all possible satisfying collations, which include (b, c, a). In other > > words, the steps I
Re: [ANNOUNCE] Danny Chan joins Calcite PMC
Congratulations ! Danny ~ OpenInx 于2019年10月31日周四 上午9:33写道： > Congrats, Danny! Well deserve. > > On Thu, Oct 31, 2019 at 9:22 AM Leonard Xu wrote: > > > Congratulation！ Danny > > > > > > > On 2019年10月31日, at 上午9:06, Andrei Sereda wrote: > > > > > > Congrats, Danny! > > > > > > On Wed, Oct 30, 2019 at 8:08 PM Julian Hyde wrote: > > > > > >> Congratulations, Danny! Thank you for all of your hard work on code, > > >> reviewing others' work, answering questions, and generally making our > > >> community welcoming to all! > > >> > > >> Julian > > >> > > >> On Wed, Oct 30, 2019 at 4:32 PM Michael Mior > wrote: > > >>> > > >>> Welcome on board Danny! Glad to have you. > > >>> -- > > >>> Michael Mior > > >>> mm...@apache.org > > >>> > > >>> Le mer. 30 oct. 2019 à 17:22, Francis Chuang > > >>> a écrit : > > > > I'm pleased to announce that Danny has accepted an invitation to > > join the Calcite PMC. Danny has been a consistent and helpful > > figure in the Calcite community for which we are very grateful. We > > look forward to the continued contributions and support. > > > > Please join me in congratulating Danny! > > > > - Francis (on behalf of the Calcite PMC) > > >> > > > > >
Re: [DISCUSS] State of the project 2019
Thanks a lot for your work Francis and +1 for Stamatis to be the next PMC chair ~ ~ Calcite is a great project and it offers as a strong SQL engine to help users to build data systems. My group uses Calcite to accelerate SQLs from multiple data engines by heuristic based, cost based and materialization based optimization. As a developer, I really benefit a lot from Calcite and the help from this great community. Calcite is not well known ? Yes, it might be less heard by the end users (who focus on data analysis), but most of data engine developers (AFAIK) know that Calcite is always a good choice when they want to build a strong SQL engine for their data systems. This is the charm of Calcite. Finally, I agree with Muhammad Gelbana to strenghten documentation for Calcite. Best, Jin Alessandro Solimando 于2019年10月24日周四 下午4:41写道： > Hello, > I have the impression too that Calcite is lesser known than other Apache > projects, but this is not that surprising to me given the nature of the > project itself, which requires in-depth knowledge of a rather complex > topic, which makes it is less prone to a casual use and plug adoption. > But IMO this is a point of strength and not a weakness, and it's what makes > Calcite so fascinating! This been said, spreading the word never hurts. > > I also agree that the community is much more active now than in the past > two years, the effort has always been high but now there are more people > involved, which makes it easier. > > On my side I mostly follow the "high-level" discussions more than the > individual tickets. Such discussions are constructive, pleasant to read and > instructive. > > A big plus also for the high-quality research articles directly stemming > from the efforts of this community. > > Last but not least, Stamatis would make a great PMC chair! > > I take the chance to thank you all for the great deal of effort in keeping > the project alive at such high standards. > > Best regards, > Alessandro > > > > On Wed, 23 Oct 2019 at 17:00, Ruben Q L wrote: > > > Hello everyone, > > > > First of all, thanks for your work, Francis. > > > > While I am relatively new in this community I must admit that I totally > > agree with Danny Chan when he says that we had some *"interesting > > discussions on the mailing list and we did have some valuable > > conclusion(like the Join expression rework, the trait sets propagation, > the > > metadata etc.)"*. I think we must continue in this direction and keep on > > our good work together. > > > > Apart from that, I agree with Muhammad Gelbana too: ideally we could have > > more technical documentation of the main modules; but I am also aware > that > > it can be very time consuming to generate such documentation, and also to > > maintain it, and we could easily have out-of-date material on our hands. > > So, I am not sure how we could proceed. > > Despite being a great project, I also have the impression that Calcite is > > not so well-known, and we need to "spread the word" to reach a bigger > > number of users. > > > > I cannot compare with previous years, but I what I have seen so far is a > > healthy community, with many people collaborating and willing to help > > beginner (and advanced) users. > > > > Finally, I think Stamatis is a great choice for PMC chair :) > > > > Best, > > Ruben Quesada Lopez > > > > > > > > Le mer. 23 oct. 2019 à 15:09, Michael Mior a écrit : > > > > > Thanks for the summary Francis and +1 for Stamatis as the next PMC > chair > > :) > > > > > > -- > > > Michael Mior > > > mm...@apache.org > > > > > > Le mer. 23 oct. 2019 à 02:36, Stamatis Zampetakis > a > > > écrit : > > > > > > > > It's been a great year so far. Thanks for your great work Francis, > you > > > did > > > > an excellent job! > > > > > > > > I also like a lot the way that the mailing list works and the fact > that > > > we > > > > are very active. Apart from helping each other it also centralizes a > > lot > > > of > > > > information, where people may find an answer to their problem quite > > > easily. > > > > When I first started to learn about Calcite, the list and the > archives > > > were > > > > indispensable. > > > > > > > > It is also very positive that we don't encounter very often > regressions > > > > (and most of those encountered were minor), which shows that the > > quality > > > > standards of the project are really high and reviewers are really > > paying > > > > attention on what is going in. I believe a lot in the motto of > quality > > > over > > > > quantity. > > > > > > > > Although we are managing PRs a bit better than the previous years, we > > > have > > > > to improve more. I had in my mind that if people assigned to > themselves > > > the > > > > review of the JIRA/PR the feeling of responsibility would make things > > > > advance faster. I tried applying this to myself but it didn't really > > work > > > > much better since there is certainly a lack of time. Other than lack > of > > > > time
Re: [DISCUSSION] Extension of Metadata Query
+1 on Danny's comment. If we use MedataFactory to customize and use RelMetadataQuery for convenience, that will make user confused. Danny Chan 于2019年10月18日周五 下午12:33写道： > That is the point, we should supply a way to extend the RelMetadataQuery > conveniently for Calcite, because in most of the RelOptRules, user would > use the code like: > > RelOptRuleCall.getMetadataQuery > > To get a RMQ instead of using AbstractRelNode.metadata() to fetch a > MedataFactory. > > We should at lest unity the metadata query entrance/interfaces, or it > would confuse a lot. > > Best, > Danny Chan > 在 2019年10月18日 +0800 AM12:23，Seliverstov Igor ，写道： > > At least in our project (Apache Ignite) we use > AbstractRelNode.metadata(). > > > > But it so because there is no way to put our metadata type into > > RelMetadataQuery without changes in Calcite. > > > > Regards, > > Igor > > > > чт, 17 окт. 2019 г., 19:16 Xiening Dai : > > > > > MetadataFactory is still useful. It provides a way to access Metadata > > > directly. If someone creates a new type of Metadata class, it can be > > > accessed through AbstractRelNode.metadata(). This way you don’t need to > > > update RelMetadataQuery interface to include the getter for this new > meta. > > > Although I don’t see this pattern being used often, but I do think it > is > > > still useful and shouldn’t be removed. > > > > > > > > > For your second point, I think you would still need a way to keep > > > RelMetadataQuery object during a rule call. If you choose to create new > > > instance, you will have to pass it around while applying the rule. That > > > actually complicates things a lot. > > > > > > > > > > On Oct 17, 2019, at 12:49 AM, XING JIN > wrote: > > > > > > > > 1. RelMetadataQuery covers the functionality of MetadataFactory, why > > > should > > > > we keep/maintain both of them ? shall we just deprecate > MetadataFactory. > > > I > > > > see MetadataFactory is rarely used in current code. Also I > > > > think MetadataFactory is not good place to offering customized > metadata, > > > > which will make user confused for the difference between > RelMetadataQuery > > > > and MetadataFactory. > > > > > > > > > Customized RelMetadataQuery with code generated meta handler for > > > > customized metadata, also can provide convenient way to get metadata. > > > > It makes sense for me. > > > > > > > > 2. If the natural lifespan of a RelMetadataQuery is a RelOptCall, > shall > > > we > > > > deprecate RelOptCluster#getMetadataQuery ? If a user wants to get the > > > > metadata but without a RelOptCall, he/she will need to create a new > > > > instance of RelMetadataQuery. > > > > > > > > Xiening Dai 于2019年10月17日周四 上午2:27写道： > > > > > > > > > I have seen both patterns in current code base. In most places, for > > > > > example SubQueryRemoveRule, AggregateUnionTrasposeRule > > > > > SortJoinTransposeRule, etc., RelOptCluster.getMetadataQuery() is > used. > > > And > > > > > there are a few other places where new RelMetadataQuery instance is > > > > > created, which Haisheng attempts to fix. > > > > > > > > > > Currently RelOptCluster.invalidateMetadataQuery() is called at the > end > > > of > > > > > RelOptRuleCall.transformTo(). So the lifespan of RelMetadataQuery > is > > > > > guaranteed to be within a RelOptCall. I think Haisheng’s fix is > safe. > > > > > > > > > > > > > > > > On Oct 16, 2019, at 1:53 AM, Danny Chan > wrote: > > > > > > > > > > > > This is the reason I was struggling for the discussion. > > > > > > > > > > > > Best, > > > > > > Danny Chan > > > > > > 在 2019年10月16日 +0800 AM11:23，dev@calcite.apache.org，写道： > > > > > > > > > > > > > > RelMetadataQuery > > > > > > > > > > > > > > > > >
Re: [DISCUSSION] Extension of Metadata Query
BTW, I think one JIRA number discussed in the thread would be https://issues.apache.org/jira/browse/CALCITE-2855 not CALCITE-2885 Best, Jin XING JIN 于2019年10月17日周四 下午3:49写道： > 1. RelMetadataQuery covers the functionality of MetadataFactory, why > should we keep/maintain both of them ? shall we just > deprecate MetadataFactory. I see MetadataFactory is rarely used in current > code. Also I think MetadataFactory is not good place to offering customized > metadata, which will make user confused for the difference > between RelMetadataQuery and MetadataFactory. > > > Customized RelMetadataQuery with code generated meta handler for > customized metadata, also can provide convenient way to get metadata. > It makes sense for me. > > 2. If the natural lifespan of a RelMetadataQuery is a RelOptCall, shall we > deprecate RelOptCluster#getMetadataQuery ? If a user wants to get the > metadata but without a RelOptCall, he/she will need to create a new > instance of RelMetadataQuery. > > Xiening Dai 于2019年10月17日周四 上午2:27写道： > >> I have seen both patterns in current code base. In most places, for >> example SubQueryRemoveRule, AggregateUnionTrasposeRule >> SortJoinTransposeRule, etc., RelOptCluster.getMetadataQuery() is used. And >> there are a few other places where new RelMetadataQuery instance is >> created, which Haisheng attempts to fix. >> >> Currently RelOptCluster.invalidateMetadataQuery() is called at the end of >> RelOptRuleCall.transformTo(). So the lifespan of RelMetadataQuery is >> guaranteed to be within a RelOptCall. I think Haisheng’s fix is safe. >> >> >> > On Oct 16, 2019, at 1:53 AM, Danny Chan wrote: >> > >> > This is the reason I was struggling for the discussion. >> > >> > Best, >> > Danny Chan >> > 在 2019年10月16日 +0800 AM11:23，dev@calcite.apache.org，写道： >> >> >> >> RelMetadataQuery >> >>
Re: [DISCUSSION] Extension of Metadata Query
1. RelMetadataQuery covers the functionality of MetadataFactory, why should we keep/maintain both of them ? shall we just deprecate MetadataFactory. I see MetadataFactory is rarely used in current code. Also I think MetadataFactory is not good place to offering customized metadata, which will make user confused for the difference between RelMetadataQuery and MetadataFactory. > Customized RelMetadataQuery with code generated meta handler for customized metadata, also can provide convenient way to get metadata. It makes sense for me. 2. If the natural lifespan of a RelMetadataQuery is a RelOptCall, shall we deprecate RelOptCluster#getMetadataQuery ? If a user wants to get the metadata but without a RelOptCall, he/she will need to create a new instance of RelMetadataQuery. Xiening Dai 于2019年10月17日周四 上午2:27写道： > I have seen both patterns in current code base. In most places, for > example SubQueryRemoveRule, AggregateUnionTrasposeRule > SortJoinTransposeRule, etc., RelOptCluster.getMetadataQuery() is used. And > there are a few other places where new RelMetadataQuery instance is > created, which Haisheng attempts to fix. > > Currently RelOptCluster.invalidateMetadataQuery() is called at the end of > RelOptRuleCall.transformTo(). So the lifespan of RelMetadataQuery is > guaranteed to be within a RelOptCall. I think Haisheng’s fix is safe. > > > > On Oct 16, 2019, at 1:53 AM, Danny Chan wrote: > > > > This is the reason I was struggling for the discussion. > > > > Best, > > Danny Chan > > 在 2019年10月16日 +0800 AM11:23，dev@calcite.apache.org，写道： > >> > >> RelMetadataQuery > >
Re: [QUESTION] Pushing up evaluations from LogicalProjects
Hi Rommel ~ I'm very happy you tried the patch ~ Well the stacktrace you provided seems outdated ~ I updated the the code several hours ago. The current commit id is 05d0e2d9fbadec060e54c622824b3725df36aab0 Could you please try the case again and send me the exception and related information ? Or you can just reply on https://github.com/apache/calcite/pull/1500 or https://issues.apache.org/jira/browse/CALCITE-3405 I will try to fix ASAP Best, Jin Rommel Quintanilla 于2019年10月15日周二 下午7:22写道： > Hi Jin, your contribution is awesome! thanks. > > I tested it and works like a charm in most cases. Definitely clearer than > the approach that I was trying. However, on the application I'm working on, > I found a weird issue that I wasn't able to reproduce as a unit test on the > calcite-core project. Btw, I'm using your changes after Haisheng's revision. > > So, it is as follows, I have this simple query: select l_extendedprice-1 > from lineitem > where the type of l_extendedprice is DOUBLE. Before applying > ProjectTableScanRule the logical plan is: > > LogicalProject(EXPR$0=[-($5, 1)]) > LogicalTableScan(table=[[main, lineitem]]) > > So far all is fine, but when the ProjectTableScanRule was applied I got: > > java.lang.AssertionError: type mismatch: > ref: > DOUBLE > input: > BIGINT > at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31) > at org.apache.calcite.plan.RelOptUtil.eq(RelOptUtil.java:2000) > at > org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:125) > at > org.apache.calcite.rex.RexChecker.visitInputRef(RexChecker.java:57) > at org.apache.calcite.rex.RexInputRef.accept(RexInputRef.java:112) > at org.apache.calcite.rex.RexChecker.visitCall(RexChecker.java:140) > at org.apache.calcite.rex.RexChecker.visitCall(RexChecker.java:57) > at org.apache.calcite.rex.RexCall.accept(RexCall.java:191) > at org.apache.calcite.rel.core.Project.isValid(Project.java:192) > at org.apache.calcite.rel.core.Project.(Project.java:83) > at > org.apache.calcite.rel.logical.LogicalProject.(LogicalProject.java:62) > at > org.apache.calcite.rel.logical.LogicalProject.create(LogicalProject.java:112) > at > org.apache.calcite.rel.logical.LogicalProject.create(LogicalProject.java:100) > at > org.apache.calcite.rel.core.RelFactories$ProjectFactoryImpl.createProject(RelFactories.java:158) > at > org.apache.calcite.tools.RelBuilder.project(RelBuilder.java:1414) > at > org.apache.calcite.tools.RelBuilder.project(RelBuilder.java:1252) > at > org.apache.calcite.tools.RelBuilder.project(RelBuilder.java:1241) > at > com.blazingdb.calcite.rules.ProjectTableScanRule.apply(ProjectTableScanRule.java:147) > at > com.blazingdb.calcite.rules.ProjectTableScanRule$1.onMatch(ProjectTableScanRule.java:65) > at > org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:319) > at > org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:560) > at > org.apache.calcite.plan.hep.HepPlanner.depthFirstApply(HepPlanner.java:374) > at > org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:436) > at > org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:256) > at > org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:127) > at > org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:215) > at > org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:202) > at > com.blazingdb.calcite.application.RelationalAlgebraGenerator.getOptimizedRelationalAlgebra(RelationalAlgebraGenerator.java:202) > at > com.blazingdb.calcite.catalog.BlazingRulesTest.generateSQLTest(BlazingRulesTest.java:274) > > Any idea why could this be happening? > > Thank you in advance. > > On 2019/10/12 09:12:20, XING JIN wrote: > > Hi Stamatis, > > In current code, BindableTableScan is only created by rules of > > ProjectTableScanRule and FilterTableScanRule. I think it's better to keep > > as it is? > > I made a PR for CALCITE-3405 -- > https://github.com/apache/calcite/pull/1500 > > > > The idea of the PR is quite straightforward: > > 1. Analyze the parent Project -- collect all the needed fields; > > 2. Column pruning by pushing down the needed fields to BindableTableScan; > > 3. Adjust RexInputRefs in parent Project > > > > @Haisheng @Stamatis It would be great if you can give a review when you > > have time ~~~ Thanks a lot ! > > > > Best, > > Jin > > > > > > Stamati
Re: Re: Re: [QUESTION] Pushing up evaluations from LogicalProjects
I mean, testPushNonSimpleMappingProject [1] runs with RelFieldTrimmer enabled, which is done by [2] [1] https://github.com/apache/calcite/pull/1500/files#diff-3c834a32d46b821b5241e132f2ae6bfaR324 [2] https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L249 XING JIN 于2019年10月14日周一 下午4:03写道： > Yes, that's how testPushNonSimpleMappingProject [1] works > > [1] > https://github.com/apache/calcite/pull/1500/files#diff-3c834a32d46b821b5241e132f2ae6bfaR324 > > Danny Chan 于2019年10月14日周一 下午3:36写道： > >> > But why the final BindableTableScan is not pruned ? >> >> The RelFieldTrimmer default is turned off, you should open it explicitly. >> >> Best, >> Danny Chan >> 在 2019年10月14日 +0800 AM11:51，dev@calcite.apache.org，写道： >> > >> > BINDABLE_TABLE_SCAN_RULE >> >
Re: Re: Re: [QUESTION] Pushing up evaluations from LogicalProjects
Yes, that's how testPushNonSimpleMappingProject [1] works [1] https://github.com/apache/calcite/pull/1500/files#diff-3c834a32d46b821b5241e132f2ae6bfaR324 Danny Chan 于2019年10月14日周一 下午3:36写道： > > But why the final BindableTableScan is not pruned ? > > The RelFieldTrimmer default is turned off, you should open it explicitly. > > Best, > Danny Chan > 在 2019年10月14日 +0800 AM11:51，dev@calcite.apache.org，写道： > > > > BINDABLE_TABLE_SCAN_RULE >
Re: Re: Re: [QUESTION] Pushing up evaluations from LogicalProjects
Also regarding that Projects merging is common in Calcite optimization rules, we should always remember to avoid merging for cases like the one given by Rommel. I think that would be hard. Best, Jin XING JIN 于2019年10月14日周一 上午11:51写道： > Hi, Stamatis, Danny~ > > Thanks for explain ~ > > > "The consumer in the case of P1 is the project which only needs $0, $2, > $5, > $6 so the trimmer could slim down the scan by projecting only these > fields." > > I think RelFieldTrimmer is already doing this by [1]. > But why the final BindableTableScan is not pruned ? I think the answer is > that RelFieldTrimmer merges projects when trimming fields for Project: > [2][3] > > > "RelFieldTrimmer can be used to transform the plan P1 to > plan P2 and then ProjectTableScanRule can be used to introduce the > BindableTableScan." > > If we wanna go by this approach, shall we avoid the project merging as > mentioned in RelFieldTrimmer [2][3] ? > > Well, as an independent functionality for column pruning, I still suggest > ProjectTableScanRule have the the ability to handle such complex scenarios. > > How do you think? > > Best, > Jin > > > > [1] > https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/RelFieldTrimmer.java#L1054 > [2] > https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/RelFieldTrimmer.java#L416 > > [3] > https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L1327 > > Danny Chan 于2019年10月14日周一 上午9:42写道： > >> +1, RelFieldTrimmer is the role to trim the unused fields. >> >> Best, >> Danny Chan >> 在 2019年10月13日 +0800 AM6:25，dev@calcite.apache.org，写道： >> > >> > RelFieldTrimmer >> >
Re: Re: Re: [QUESTION] Pushing up evaluations from LogicalProjects
Hi, Stamatis, Danny~ Thanks for explain ~ > "The consumer in the case of P1 is the project which only needs $0, $2, $5, $6 so the trimmer could slim down the scan by projecting only these fields." I think RelFieldTrimmer is already doing this by [1]. But why the final BindableTableScan is not pruned ? I think the answer is that RelFieldTrimmer merges projects when trimming fields for Project: [2][3] > "RelFieldTrimmer can be used to transform the plan P1 to plan P2 and then ProjectTableScanRule can be used to introduce the BindableTableScan." If we wanna go by this approach, shall we avoid the project merging as mentioned in RelFieldTrimmer [2][3] ? Well, as an independent functionality for column pruning, I still suggest ProjectTableScanRule have the the ability to handle such complex scenarios. How do you think? Best, Jin [1] https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/RelFieldTrimmer.java#L1054 [2] https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/RelFieldTrimmer.java#L416 [3] https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L1327 Danny Chan 于2019年10月14日周一 上午9:42写道： > +1, RelFieldTrimmer is the role to trim the unused fields. > > Best, > Danny Chan > 在 2019年10月13日 +0800 AM6:25，dev@calcite.apache.org，写道： > > > > RelFieldTrimmer >
Re: Re: [QUESTION] Pushing up evaluations from LogicalProjects
Sure we can ~ If we use BindableTableScanRule to derive a BindableTableScan from ProjectableFilterableTable, that would happen during a stage of optimization run by RelOptPlanner. But RelFieldTrimmer works right during conversion of Sql to Rel. Wang Yanlin <1989yanlinw...@163.com> 于2019年10月12日周六 下午5:56写道： > Can we just use Bindables.BINDABLE_TABLE_SCAN_RULE to translate the > table scan to BindableTableScan ? > > > > -- > > Best, > Wang Yanlin > > > > At 2019-10-12 17:12:20, "XING JIN" wrote: > >Hi Stamatis, > >In current code, BindableTableScan is only created by rules of > >ProjectTableScanRule and FilterTableScanRule. I think it's better to keep > >as it is? > >I made a PR for CALCITE-3405 -- > https://github.com/apache/calcite/pull/1500 > > > >The idea of the PR is quite straightforward: > >1. Analyze the parent Project -- collect all the needed fields; > >2. Column pruning by pushing down the needed fields to BindableTableScan; > >3. Adjust RexInputRefs in parent Project > > > >@Haisheng @Stamatis It would be great if you can give a review when you > >have time ~~~ Thanks a lot ! > > > >Best, > >Jin > > > > > >Stamatis Zampetakis 于2019年10月12日周六 下午3:00写道： > > > >> Hi Rommel, > >> > >> I was hoping that this could be done at least by RelFieldTrimmer [1]. > Are > >> you using it already? > >> > >> Best, > >> Stamatis > >> > >> [1] > >> > >> > https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/RelFieldTrimmer.java > >> > >> On Sat, Oct 12, 2019 at 6:06 AM XING JIN > wrote: > >> > >> > Filed a JIRA: > >> > https://issues.apache.org/jira/browse/CALCITE-3405 > >> > > >> > Haisheng Yuan 于2019年10月12日周六 上午4:34写道： > >> > > >> > > Yes, definitely. > >> > > > >> > > You can go through the project expression with InputFinder to find > all > >> > the > >> > > used columns, create a logical project with those columns, and > remap > >> the > >> > > top project with new column indexes. > >> > > > >> > > On the other hand, instead of creating a new intermidiate pogical > >> > project, > >> > > we can also update ProjectTableScanRule to accept LogicalProject > that > >> is > >> > > not a simple mapping, and do the same task I mentioned above. > >> > > > >> > > - Haisheng > >> > > > >> > > -- > >> > > 发件人：Rommel Quintanilla > >> > > 日 期：2019年10月12日 03:15:31 > >> > > 收件人： > >> > > 主 题：[QUESTION] Pushing up evaluations from LogicalProjects > >> > > > >> > > Hi, maybe you can help me. > >> > > I have this portion from a larger logical plan: > >> > > .. > >> > > LogicalProject(l_orderkey=[$0], l_suppkey=[$2], *=[*($5, > -(1, > >> > > $6))]) > >> > > LogicalTableScan(table=[[main, lineitem]]) > >> > > .. > >> > > > >> > > Because the LogicalProject above contains an evaluation, the > >> > > ProjectTableScanRule can't convert it to a BindableTableScan. > >> > > > >> > > I wonder if somehow the evaluation could be pushed up more or less > like > >> > > this: > >> > > .. > >> > > LogicalProject(l_orderkey=[$0], l_suppkey=[$1], *=[*($2, -(1, > $3))]) > >> > > LogicalProject(l_orderkey=[$0], l_suppkey=[$2], > l_extendedprice=[$5], > >> > > l_discount=[$6]]) > >> > > LogicalTableScan(table=[[main, lineitem]]) > >> > > .. > >> > > > >> > > Regards. > >> > > > >> > > >> >
Re: [QUESTION] Pushing up evaluations from LogicalProjects
Hi Stamatis, In current code, BindableTableScan is only created by rules of ProjectTableScanRule and FilterTableScanRule. I think it's better to keep as it is? I made a PR for CALCITE-3405 -- https://github.com/apache/calcite/pull/1500 The idea of the PR is quite straightforward: 1. Analyze the parent Project -- collect all the needed fields; 2. Column pruning by pushing down the needed fields to BindableTableScan; 3. Adjust RexInputRefs in parent Project @Haisheng @Stamatis It would be great if you can give a review when you have time ~~~ Thanks a lot ! Best, Jin Stamatis Zampetakis 于2019年10月12日周六 下午3:00写道： > Hi Rommel, > > I was hoping that this could be done at least by RelFieldTrimmer [1]. Are > you using it already? > > Best, > Stamatis > > [1] > > https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql2rel/RelFieldTrimmer.java > > On Sat, Oct 12, 2019 at 6:06 AM XING JIN wrote: > > > Filed a JIRA: > > https://issues.apache.org/jira/browse/CALCITE-3405 > > > > Haisheng Yuan 于2019年10月12日周六 上午4:34写道： > > > > > Yes, definitely. > > > > > > You can go through the project expression with InputFinder to find all > > the > > > used columns, create a logical project with those columns, and remap > the > > > top project with new column indexes. > > > > > > On the other hand, instead of creating a new intermidiate pogical > > project, > > > we can also update ProjectTableScanRule to accept LogicalProject that > is > > > not a simple mapping, and do the same task I mentioned above. > > > > > > - Haisheng > > > > > > -- > > > 发件人：Rommel Quintanilla > > > 日 期：2019年10月12日 03:15:31 > > > 收件人： > > > 主 题：[QUESTION] Pushing up evaluations from LogicalProjects > > > > > > Hi, maybe you can help me. > > > I have this portion from a larger logical plan: > > > .. > > > LogicalProject(l_orderkey=[$0], l_suppkey=[$2], *=[*($5, -(1, > > > $6))]) > > > LogicalTableScan(table=[[main, lineitem]]) > > > .. > > > > > > Because the LogicalProject above contains an evaluation, the > > > ProjectTableScanRule can't convert it to a BindableTableScan. > > > > > > I wonder if somehow the evaluation could be pushed up more or less like > > > this: > > > .. > > > LogicalProject(l_orderkey=[$0], l_suppkey=[$1], *=[*($2, -(1, $3))]) > > > LogicalProject(l_orderkey=[$0], l_suppkey=[$2], l_extendedprice=[$5], > > > l_discount=[$6]]) > > > LogicalTableScan(table=[[main, lineitem]]) > > > .. > > > > > > Regards. > > > > > >
Re: [QUESTION] Pushing up evaluations from LogicalProjects
Filed a JIRA: https://issues.apache.org/jira/browse/CALCITE-3405 Haisheng Yuan 于2019年10月12日周六 上午4:34写道： > Yes, definitely. > > You can go through the project expression with InputFinder to find all the > used columns, create a logical project with those columns, and remap the > top project with new column indexes. > > On the other hand, instead of creating a new intermidiate pogical project, > we can also update ProjectTableScanRule to accept LogicalProject that is > not a simple mapping, and do the same task I mentioned above. > > - Haisheng > > -- > 发件人：Rommel Quintanilla > 日 期：2019年10月12日 03:15:31 > 收件人： > 主 题：[QUESTION] Pushing up evaluations from LogicalProjects > > Hi, maybe you can help me. > I have this portion from a larger logical plan: > .. > LogicalProject(l_orderkey=[$0], l_suppkey=[$2], *=[*($5, -(1, > $6))]) > LogicalTableScan(table=[[main, lineitem]]) > .. > > Because the LogicalProject above contains an evaluation, the > ProjectTableScanRule can't convert it to a BindableTableScan. > > I wonder if somehow the evaluation could be pushed up more or less like > this: > .. > LogicalProject(l_orderkey=[$0], l_suppkey=[$1], *=[*($2, -(1, $3))]) > LogicalProject(l_orderkey=[$0], l_suppkey=[$2], l_extendedprice=[$5], > l_discount=[$6]]) > LogicalTableScan(table=[[main, lineitem]]) > .. > > Regards. >
Re: Adding RelOptMaterializations to a planner
Hi Shubham, In my understanding, same RelOptPlanner is the way to go, as one VolcanoPlanner#registerMaterializations calls RelOptMaterializations#useMaterializedViews and picks the best algebra expression tree. I'd suggest to create all the materializations into a VolcanoPlanner and then "findBestExp". Please note that there are two kinds of strategies for materialized view optimization: 1. Substitution based materialized view optimization 2. Rewriting using plan structural information For the first one, only RelOptMaterializations#useMaterializedViews is enough, you can even don't rely on VolcanoPlanner, i.e. call RelOptMaterializations#useMaterializedViews explicitly and pick the best one by yourself. But for the second one, you need to to rely on VolcanoPlanner and register corresponding rules from AbstractMaterializedViewRule. The second one tend to be smarter but only supports SPJA pattern. In short, when you enable config of "materializationEnabled" for connection property, both of the two strategies above are enabled. In addition, I'd suggest to do canonicalization before materialized view optimization, which helps a lot for materialized view matching. I'm also doing some work for materialized view optimization. It would be great to have more discussion on this :) https://issues.apache.org/jira/browse/CALCITE-3334 https://docs.google.com/document/d/1JpwGNFE3hw3yXb7W3-95-jXKClZC5UFPKbuhgYDuEu4/edit#heading=h.bmvjxz1h5evc Best, Jin
Re: How to get columnName as `COUNT(*)` , not `EXPR$0`
It's my pleasure, you are welcome ~ Juan Pan 于2019年9月29日周日 下午2:24写道： > Hi XING, > I appreciate your kindness. :-D Your detailed and prompt replies really > helped me a lot. > I will review the java doc you mentioned. > > > Best wishes, > Trista > > > Juan Pan > > > panj...@apache.org > Juan Pan(Trista), Apache ShardingSphere > > > On 09/29/2019 13:58，XING JIN wrote： > You can check the below doc of SqlValidatorUtil#getAlias for explanation: > > /** > * Derives an alias for a node, and invents a mangled identifier if it > * cannot. > * > * Examples: > * > * > * Alias: "1 + 2 as foo" yields "foo" > * Identifier: "foo.bar.baz" yields "baz" > * Anything else yields "expr$ordinal" > * > * > * @return An alias, if one can be derived; or a synthetic alias > * "expr$ordinal" if ordinal 0; otherwise null > */ > public static String getAlias(SqlNode node, int ordinal) > > But from my experience, you'd better not rely on above logic heavily. If > you really care about the output name, just give it an alias explicitly. > > Juan Pan 于2019年9月29日周日 下午1:27写道： > > That means Calcite can only return real columnName or columnLabel from > simple column or alias. And any aggregate function, or calculate expression > without alias, parsing expression, i.e, `EXPR$0` will be returned? > > > Juan Pan > > > panj...@apache.org > Juan Pan(Trista), Apache ShardingSphere > > > On 09/29/2019 13:16，XING JIN wrote： > If no column name given explicitly, e.g. by alias or simple identifier, > Calcite will derive one but not from the aggregate function. > > Juan Pan 于2019年9月29日周日 下午1:12写道： > > Thank for your reply. It is a indirect way to get columnName. > > > Calcite can not return the real columnName from SQL, is it right? > > > Juan Pan > > > panj...@apache.org > Juan Pan(Trista), Apache ShardingSphere > > > On 09/29/2019 12:21，XING JIN wrote： > You can try to give an alias for the selected column. > > Juan Pan 于2019年9月29日周日 上午11:39写道： > > > > Hi everyone, > > > I executed SQL `select count(*) from tb1` through Calcite and > resultSet.getMetaData().getColumnName(i) in my project. But the result is > `EXPR$0` not `COUNT(*)`. > > > Is there any way to get real columnName? > > > Thanks for your attention. > > > Regard, > Trista > > > > > Juan Pan > > > panj...@apache.org > Juan Pan(Trista), Apache ShardingSphere > > > > >
Re: How to get columnName as `COUNT(*)` , not `EXPR$0`
You can check the below doc of SqlValidatorUtil#getAlias for explanation: /** * Derives an alias for a node, and invents a mangled identifier if it * cannot. * * Examples: * * * Alias: "1 + 2 as foo" yields "foo" * Identifier: "foo.bar.baz" yields "baz" * Anything else yields "expr$ordinal" * * * @return An alias, if one can be derived; or a synthetic alias * "expr$ordinal" if ordinal 0; otherwise null */ public static String getAlias(SqlNode node, int ordinal) But from my experience, you'd better not rely on above logic heavily. If you really care about the output name, just give it an alias explicitly. Juan Pan 于2019年9月29日周日 下午1:27写道： > That means Calcite can only return real columnName or columnLabel from > simple column or alias. And any aggregate function, or calculate expression > without alias, parsing expression, i.e, `EXPR$0` will be returned? > > > Juan Pan > > > panj...@apache.org > Juan Pan(Trista), Apache ShardingSphere > > > On 09/29/2019 13:16，XING JIN wrote： > If no column name given explicitly, e.g. by alias or simple identifier, > Calcite will derive one but not from the aggregate function. > > Juan Pan 于2019年9月29日周日 下午1:12写道： > > Thank for your reply. It is a indirect way to get columnName. > > > Calcite can not return the real columnName from SQL, is it right? > > > Juan Pan > > > panj...@apache.org > Juan Pan(Trista), Apache ShardingSphere > > > On 09/29/2019 12:21，XING JIN wrote： > You can try to give an alias for the selected column. > > Juan Pan 于2019年9月29日周日 上午11:39写道： > > > > Hi everyone, > > > I executed SQL `select count(*) from tb1` through Calcite and > resultSet.getMetaData().getColumnName(i) in my project. But the result is > `EXPR$0` not `COUNT(*)`. > > > Is there any way to get real columnName? > > > Thanks for your attention. > > > Regard, > Trista > > > > > Juan Pan > > > panj...@apache.org > Juan Pan(Trista), Apache ShardingSphere > > > >
Re: How to get columnName as `COUNT(*)` , not `EXPR$0`
If no column name given explicitly, e.g. by alias or simple identifier, Calcite will derive one but not from the aggregate function. Juan Pan 于2019年9月29日周日 下午1:12写道： > Thank for your reply. It is a indirect way to get columnName. > > > Calcite can not return the real columnName from SQL, is it right? > > > Juan Pan > > > panj...@apache.org > Juan Pan(Trista), Apache ShardingSphere > > > On 09/29/2019 12:21，XING JIN wrote： > You can try to give an alias for the selected column. > > Juan Pan 于2019年9月29日周日 上午11:39写道： > > > > Hi everyone, > > > I executed SQL `select count(*) from tb1` through Calcite and > resultSet.getMetaData().getColumnName(i) in my project. But the result is > `EXPR$0` not `COUNT(*)`. > > > Is there any way to get real columnName? > > > Thanks for your attention. > > > Regard, > Trista > > > > > Juan Pan > > > panj...@apache.org > Juan Pan(Trista), Apache ShardingSphere > > >
Re: How to get columnName as `COUNT(*)` , not `EXPR$0`
You can try to give an alias for the selected column. Juan Pan 于2019年9月29日周日 上午11:39写道： > > > Hi everyone, > > > I executed SQL `select count(*) from tb1` through Calcite and > resultSet.getMetaData().getColumnName(i) in my project. But the result is > `EXPR$0` not `COUNT(*)`. > > > Is there any way to get real columnName? > > > Thanks for your attention. > > > Regard, > Trista > > > > > Juan Pan > > > panj...@apache.org > Juan Pan(Trista), Apache ShardingSphere > >
Re: Trivial query simplification
"v = 1 and v is null"
cannot be simplified to "v = 1" not matter v is nullable or not nullable
If you really mean that "v is not null", I made below test case in
RelOptRulesTest.java for illustration:
// mgr is nullable
@Test public void testDEV() throws Exception {
HepProgram program = new HepProgramBuilder()
.addRuleInstance(ReduceExpressionsRule.FILTER_INSTANCE)
.build();
final String sql = "select deptno"
+ " from emp"
+ " where mgr = 10 and mgr is not null";
checkPlanning(new HepPlanner(program), sql);
}
The plan is
LogicalProject(DEPTNO=[$7])
LogicalFilter(condition=[=($3, 10)])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
Enrico ~ you may try ReduceExpressionsRule.FILTER_INSTANCE
Feng Zhu 于2019年9月24日周二 下午5:50写道：
> Hi, Enrico,
> I'm a little confused about your expectations. Could you clarify it?
> Moreover, is it right for the below simplification (do you mean v is not
> null)?
> (v=1 and v is null) -> v=1
> (do you mean v is not null?)
>
> Best regards
>
> Enrico Olivelli 于2019年9月24日周二 下午5:41写道：
>
> > Hi,
> > I have a query like
> > SELECT * FROM MYTABLE WHERE v = 1 and v is null
> >
> > I am expecting Calcite to simplify it to
> > SELECT * FROM MYTABLE WHERE v = 1
> >
> > but this does not happen.
> >
> > Is any rule I should enable in order to make it happen ?
> >
> > This is the configuration of my Volcano planner:
> >
> > final FrameworkConfig config = Frameworks.newConfigBuilder()
> > .parserConfig()
> > .defaultSchema(...)
> > .traitDefs()
> > .programs(Programs.ofRules(Programs.RULE_SET))
> > .build();
> >
> > Best regards
> > Enrico
> >
>
Re: Re: [ANNOUNCE] New committer: Julian Feinauer
Congrats, Julian ! You are well deserved ~ Haisheng Yuan 于2019年9月18日周三 上午10:38写道： > Congrats, Julian! > > - Haisheng > > -- > 发件人：Chunwei Lei > 日 期：2019年09月18日 10:30:31 > 收件人： > 主 题：Re: [ANNOUNCE] New committer: Julian Feinauer > > Congratulations, Julian! > > > > Best, > Chunwei > > > On Wed, Sep 18, 2019 at 9:24 AM Danny Chan wrote: > > > Congratulations, Muhammad ! Welcome to join us ! Thanks for your huge > > contribution for the Match Recognize. > > > > Best, > > Danny Chan > > 在 2019年9月18日 +0800 AM5:55，Francis Chuang ，写道： > > > Apache Calcite's Project Management Committee (PMC) has invited Julian > > > Feinauer to become a committer, and we are pleased to announce that he > > > has accepted. > > > > > > Julian is an active contributor to the Calcite code base and has been > > > active on the mailing list answering questions, participating in > > > discussions and voting for releases. > > > > > > Julian, welcome, thank you for your contributions, and we look forward > > > your further interactions with the community! If you wish, please feel > > > free to tell us more about yourself and what you are working on. > > > > > > Francis (on behalf of the Apache Calcite PMC) > > > >
Re: [ANNOUNCE] New committer: Muhammad Gelbana
Congrats, Muhammad ! 王炎林 <1989yanlinw...@163.com> 于2019年9月18日周三 上午10:38写道： > Congratulations, Muhammad! > > > Best, > Yanlin > > > > > > > At 2019-09-18 05:58:53, "Francis Chuang" wrote: > >Apache Calcite's Project Management Committee (PMC) has invited Muhammad > >Gelbana to become a committer, and we are pleased to announce that he > >has accepted. > > > >Muhammad is an active contributor and has contributed numerous patches > >to Calcite. He has also been extremely active on the mailing list, > >helping out new users and participating in design discussions. > > > >Muhammad, welcome, thank you for your contributions, and we look forward > >your further interactions with the community! If you wish, please feel > >free to tell us more about yourself and what you are working on. > > > >Francis (on behalf of the Apache Calcite PMC) >
Re: Is it possible to retrieve materialized view query rewrite from Calcite Plan
I think it will work. I once did similar things and was playing around with
VolcanoPlanner and RelOptMaterializations.
Shubham Kumar 于2019年9月16日周一 下午8:32写道：
> Hey everyone,
>
> Thank you for the help.
>
> @Stamatis, @Danny : Thanks for pointing out that the reltosql api should
> be used after volcano planner execution which considers materializations
> and lattices, I understand it now.
>
> @Xing: Yeah, it works perfectly fine, I am able to see simple substitutions
> in action done by Calcite Planner!
>
> Although I guess "*org.apache.calcite.tes*t" API can't be accessed in some
> other project. To extract out the materialized query, I was playing around
> with Prepare and optimize of Calcite Prepare class[1]. Let me know if I am
> moving in the correct direction.
>
> Since, now I am able to get the rewrite query, I wanted to share the exact
> use case which I am trying to implement and get your insights:
>
> I have explicitly defined materialized views which are stored in data
> source and defining it in *root.schema.materializations *of model file
> (keeping the "*view*" field empty as its already exists). Now, I will be
> completely responsible for maintaining this materialized view table in my
> datasource as various datasources don't have built in Materialized View
> like Mysql etc. Finally, I aim the rewrite query that I retrieve from
> Calcite to execute it using Spark SQL (data of tables will be stored in
> HDFS). So basically query rewrite of Calcite will be a layer before Spark
> SQL and it uses the optimized query using materialized views for making the
> actual Spark plan and executing it in a distributed manner.
>
> I would love to have your insights on this on whether this is will be
> feasible and if it would improve query performance.
>
> Problems that I am facing currently is that while using "
> *org.apache.calcite.adapter.jdbc.JdbcSchema$Factory*" as the factory method
> I get the following exception:
>
>
> *Cannot define materialization; parent schema 'test_calcite' is not a
> SemiMutableSchema*
>
> Currently I am using CloneSchema$Factory for testing things out but I guess
> it maintains in memory tables which eventually I won't be able to use in
> case of large tables. I was wondering if I would have to write my own
> Factory classes and implementations for my use case or is there something
> already present.
>
> [1]
>
>
> https://github.com/apache/calcite/blob/73023148e7f37d494f6caf92b01b090f6dde13cd/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L320
>
> Thanks
> Shubham
>
>
> On Mon, Sep 16, 2019 at 11:32 AM XING JIN wrote:
>
> > Hi, Shubham Kumar
> > If I understand correctly, you want get the optimized(by materialized
> view)
> > SQL String. I wrote a simple test as below, please check if it's helpful
> > for you.
> >
> > @Test public void testDEV() {
> > final String m = "select \"deptno\", \"empid\", \"name\""
> > + "from \"emps\" where \"deptno\" = 10";
> > final String q = "select \"empid\" + 1 as x, \"name\""
> > + "from \"emps\" where \"deptno\" = 10";
> >
> > CalciteAssert.that()
> > .withMaterializations(HR_FKUK_MODEL, "m0", m)
> > .query(q)
> > .withHook(Hook.SUB, (Consumer) r ->
> > {
> > RelToSqlConverter converter =
> > new RelToSqlConverter(CalciteSqlDialect.DEFAULT);
> > final SqlNode sqlNode = converter.visitChild(0, r).asStatement();
> >
> >
> System.out.println(sqlNode.toSqlString(CalciteSqlDialect.DEFAULT).getSql());
> > })
> > .enableMaterializations(true)
> > .explainContains("hr, m0");
> > }
> >
> > The output is as below:
> >
> > SELECT "empid" + 1 AS "X", "name"
> > FROM "hr"."m0"
> >
> >
> > Danny Chan 于2019年9月16日周一 下午1:44写道：
> >
> > > Hi, Shubham Kumar ~
> > >
> > > > However, I wanted the optimized rewrite SQL query if possible, not
> just
> > > the
> > > > plan. So far, I tried to use the rel2sql api but when I print
> > > > RelNode.toString(), it gives me a plan which involves scanning the
> raw
> > > > tables and hence the SQL generated by rel2sql is not the one which
> > > utilized
> > > > the materialized view. Any pointers to get the rewrite query.
> > >
> > > Did you try the RelNode.toString() after the volcano planner promotion,
> > it
> > > is not expected to involve scanning the raw tables.
> > >
> > > Best,
> > > Danny Chan
> > > 在 2019年9月11日 +0800 PM9:42，dev@calcite.apache.org，写道：
> > > >
> > > > However, I wanted the optimized rewrite SQL query if possible, not
> just
> > > the
> > > > plan. So far, I tried to use the rel2sql api but when I print
> > > > RelNode.toString(), it gives me a plan which involves scanning the
> raw
> > > > tables and hence the SQL generated by rel2sql is not the one which
> > > utilized
> > > > the materialized view. Any pointers to get the rewrite query.
> > >
> >
>
>
> --
> Thanks & Regards
>
> Shubham Kumar
>
Re: [DISCUSS] Refinement for Substitution-Based MV Matching
Thanks a lot Danny and Haisheng for comments on the doc ~ And more comments are welcome ~ Best, Jin XING JIN 于2019年9月16日周一 下午9:17写道： > Hi Contributors, > > I'm writing this email and hope to start a discussion about > https://issues.apache.org/jira/browse/CALCITE-3334 . > > The approach of substitution-based materialized view matching is effective > for its simplicity and extensibility. But now and then we confront > materialization matching failures especially for SQLs with multiple nested > levels. We keep thinking that how many more rules still need to be added > and can we enumerate all the common matching patterns. > > The existing rules in SubstitutionVisitor & > MaterializedViewSubstitutionVisitor are created by heuristic and > experience. They are quite straightforward. But from my understanding the > design of rule set is not systematic enough. Especially compensating > Project or Filter are not given enough good care. > > I raised a doc -- > https://docs.google.com/document/d/1JpwGNFE3hw3yXb7W3-95-jXKClZC5UFPKbuhgYDuEu4 > to > propose refinement for substitution-based materialized view matching from > two aspects: > 1. Canonicalize before materialization matching; > 2. Separate matching rules into two categories and enumerate matching > patterns which need to be covered by rules. > > It's great if you can share some comments on this. > > Best, > Jin > >
[DISCUSS] Refinement for Substitution-Based MV Matching
Hi Contributors, I'm writing this email and hope to start a discussion about https://issues.apache.org/jira/browse/CALCITE-3334 . The approach of substitution-based materialized view matching is effective for its simplicity and extensibility. But now and then we confront materialization matching failures especially for SQLs with multiple nested levels. We keep thinking that how many more rules still need to be added and can we enumerate all the common matching patterns. The existing rules in SubstitutionVisitor & MaterializedViewSubstitutionVisitor are created by heuristic and experience. They are quite straightforward. But from my understanding the design of rule set is not systematic enough. Especially compensating Project or Filter are not given enough good care. I raised a doc -- https://docs.google.com/document/d/1JpwGNFE3hw3yXb7W3-95-jXKClZC5UFPKbuhgYDuEu4 to propose refinement for substitution-based materialized view matching from two aspects: 1. Canonicalize before materialization matching; 2. Separate matching rules into two categories and enumerate matching patterns which need to be covered by rules. It's great if you can share some comments on this. Best, Jin
Re: Is it possible to retrieve materialized view query rewrite from Calcite Plan
Hi, Shubham Kumar
If I understand correctly, you want get the optimized(by materialized view)
SQL String. I wrote a simple test as below, please check if it's helpful
for you.
@Test public void testDEV() {
final String m = "select \"deptno\", \"empid\", \"name\""
+ "from \"emps\" where \"deptno\" = 10";
final String q = "select \"empid\" + 1 as x, \"name\""
+ "from \"emps\" where \"deptno\" = 10";
CalciteAssert.that()
.withMaterializations(HR_FKUK_MODEL, "m0", m)
.query(q)
.withHook(Hook.SUB, (Consumer) r ->
{
RelToSqlConverter converter =
new RelToSqlConverter(CalciteSqlDialect.DEFAULT);
final SqlNode sqlNode = converter.visitChild(0, r).asStatement();
System.out.println(sqlNode.toSqlString(CalciteSqlDialect.DEFAULT).getSql());
})
.enableMaterializations(true)
.explainContains("hr, m0");
}
The output is as below:
SELECT "empid" + 1 AS "X", "name"
FROM "hr"."m0"
Danny Chan 于2019年9月16日周一 下午1:44写道：
> Hi, Shubham Kumar ~
>
> > However, I wanted the optimized rewrite SQL query if possible, not just
> the
> > plan. So far, I tried to use the rel2sql api but when I print
> > RelNode.toString(), it gives me a plan which involves scanning the raw
> > tables and hence the SQL generated by rel2sql is not the one which
> utilized
> > the materialized view. Any pointers to get the rewrite query.
>
> Did you try the RelNode.toString() after the volcano planner promotion, it
> is not expected to involve scanning the raw tables.
>
> Best,
> Danny Chan
> 在 2019年9月11日 +0800 PM9:42，dev@calcite.apache.org，写道：
> >
> > However, I wanted the optimized rewrite SQL query if possible, not just
> the
> > plan. So far, I tried to use the rel2sql api but when I print
> > RelNode.toString(), it gives me a plan which involves scanning the raw
> > tables and hence the SQL generated by rel2sql is not the one which
> utilized
> > the materialized view. Any pointers to get the rewrite query.
>
Re: Match Converter Rule based on Child Nodes
Hmm,, I get it So maybe you can try HepPlanner with match order of BOTTOM_UP; Or if VolcanoPlanner is necessary, how about call the optimization multiple times, e.g. wrap it with a loop Jin rahul patwari 于2019年8月23日周五 上午10:59写道： > Hi Jin, > > We wanted to transform LogicalJoin to different Join Types in the external > system depending on the LogicalJoin Child Properties(which are in external > convention). > As LogicalJoin can be a child of LogicalJoin, because of VolcanoPlanner's > top-down approach, the child LogicalJoin is not converted first and we end > up in "Not enough rules to find the Cheapest Plan". > > Regards, > Rahul > > On Fri, Aug 23, 2019 at 8:17 AM XING JIN wrote: > > > If I understand correctly, you can try below steps: > > 1. Create a rule of Converter to convert child nodes to append external > > Convention; > > 2. Create a rule of RelOptRule to convert the parent node -- check the > > Convention of child node when matching > > > > Is it applicable ? > > > > rahul patwari 于2019年8月22日周四 下午11:18写道： > > > > > Hi, > > > > > > The properties of the child nodes correspond to the external > Convention. > > > So, the child nodes have to be converted before the parent can be > > > transformed. > > > If the property doesn't match (or) if the property cannot be > > > determined(child node not converted), the rule cannot be applied. So, > we > > > end up in "Not enough Rules to find the Cheapest Plan". > > > Is there a way to convert the Child nodes before the parent? > > > Can VolcanoPlanner be configured to work in bottom-up fashion? > > > > > > Regards, > > > Rahul > > > > > > On Thu, Aug 22, 2019 at 3:33 PM XING JIN > > wrote: > > > > > > > I guess in Rahul's case, the child node of join should be converted > > > first, > > > > thus physical properties will be appended. > > > > But RelNode doesn't have a reference to parent, thus an independent > > > > RelOptRule is needed to convert LogicalJoin. > > > > Just as Michael said, you need to override the method of > > > RelOptRule#matches > > > > and check the properties of child node. > > > > > > > > Best, > > > > Jin > > > > > > > > Stamatis Zampetakis 于2019年8月21日周三 下午5:50写道： > > > > > > > > > The Volcano planner works in a top-down fashion. It starts by > > > > transforming > > > > > the root and move towards the leafs of the plan. Due to this when > > > > > transforming a logical join its inputs are still in the logical > > > > convention > > > > > so in principle they should not have any physical properties. > > > > > > > > > > Normally when the physical join algorithm is chosen the respective > > rule > > > > > should be responsible for demanding the inputs of the join to have > > > > certain > > > > > properties. > > > > > > > > > > Long story short, I think in your use case you should not make the > > rule > > > > > match based on the properties of the child nodes but it should > match > > > > > unconditionally and if necessary demand some properties in its > > inputs. > > > > If I > > > > > remember well the EnumerableMergeJoinRule follows this approach so > > you > > > > can > > > > > have a look there. > > > > > > > > > > Best, > > > > > Stamatis > > > > > > > > > > On Tue, Aug 20, 2019, 5:07 PM Michael Mior > wrote: > > > > > > > > > > > If you just want to control whether the rule gets applied, you > can > > > > > > override RelOptRule#matches which canreturns a boolean indicating > > > > > > whether the rule should be applied. > > > > > > -- > > > > > > Michael Mior > > > > > > mm...@apache.org > > > > > > > > > > > > Le ven. 9 août 2019 à 08:48, rahul patwari > > > > > > a écrit : > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > We want to create a ConverterRule which converts the default > > > calling > > > > > > > Convention to external storage-specific calling convention > > > depending > > > > on > > > > > > the > > > > > > > Children nodes, like RelOptRule. > > > > > > > > > > > > > > For example, depending on the properties of the child nodes, we > > > want > > > > to > > > > > > > convert LogicalJoin to external system's specific Join > > > > implementation. > > > > > > > > > > > > > > Currently, ConverterRule > > > > > > > < > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/calcite/blob/5212d6c47e36995943f4d955a1714bf03eb08e7e/core/src/main/java/org/apache/calcite/rel/convert/ConverterRule.java#L75 > > > > > > > > > > > > > > cannot take Children and Child Policy is > > > > > > RelOptRuleOperandChildPolicy.ANY. > > > > > > > > > > > > > > What is the preferred way to achieve this task? > > > > > > > > > > > > > > Thanks, > > > > > > > Rahul > > > > > > > > > > > > > > > > > > > > >
Re: Mapping arithmetic '+' operator with Date and Timestamp operands to target dialect functions
I guess in your approach, you need to maintain a customized `SqlImplementor` ? How about add a RelOptRule and detect pattern of "date + interval '1' DAY" and use a RexShuttle to convert to "target_func(date, interval '1' DAY)" Thus only a RelOptRule needs to be maintained. Sachin Rohaj 于2019年8月23日周五 下午11:32写道： > Hello, > > In one of my use case i am working on DATE and TIMESTAMP based operands. We > have > select date + interval '1' DAY from table; > > I need to map this to target dialect. > select target_func(date, interval '1' DAY) from table; > > *Approach*: I am planning to intercept the '+' operator in the > 'SqlImplementor' class using 'call.type.getSqlTypeName()' and delegate that > responsibility to get the 'target_func' from the specific dialect. > For exampe: 'dialect.getTargetFunction(SqlTypeName)'. > > Thanks, > Sachin >
Re: Match Converter Rule based on Child Nodes
If I understand correctly, you can try below steps: 1. Create a rule of Converter to convert child nodes to append external Convention; 2. Create a rule of RelOptRule to convert the parent node -- check the Convention of child node when matching Is it applicable ? rahul patwari 于2019年8月22日周四 下午11:18写道： > Hi, > > The properties of the child nodes correspond to the external Convention. > So, the child nodes have to be converted before the parent can be > transformed. > If the property doesn't match (or) if the property cannot be > determined(child node not converted), the rule cannot be applied. So, we > end up in "Not enough Rules to find the Cheapest Plan". > Is there a way to convert the Child nodes before the parent? > Can VolcanoPlanner be configured to work in bottom-up fashion? > > Regards, > Rahul > > On Thu, Aug 22, 2019 at 3:33 PM XING JIN wrote: > > > I guess in Rahul's case, the child node of join should be converted > first, > > thus physical properties will be appended. > > But RelNode doesn't have a reference to parent, thus an independent > > RelOptRule is needed to convert LogicalJoin. > > Just as Michael said, you need to override the method of > RelOptRule#matches > > and check the properties of child node. > > > > Best, > > Jin > > > > Stamatis Zampetakis 于2019年8月21日周三 下午5:50写道： > > > > > The Volcano planner works in a top-down fashion. It starts by > > transforming > > > the root and move towards the leafs of the plan. Due to this when > > > transforming a logical join its inputs are still in the logical > > convention > > > so in principle they should not have any physical properties. > > > > > > Normally when the physical join algorithm is chosen the respective rule > > > should be responsible for demanding the inputs of the join to have > > certain > > > properties. > > > > > > Long story short, I think in your use case you should not make the rule > > > match based on the properties of the child nodes but it should match > > > unconditionally and if necessary demand some properties in its inputs. > > If I > > > remember well the EnumerableMergeJoinRule follows this approach so you > > can > > > have a look there. > > > > > > Best, > > > Stamatis > > > > > > On Tue, Aug 20, 2019, 5:07 PM Michael Mior wrote: > > > > > > > If you just want to control whether the rule gets applied, you can > > > > override RelOptRule#matches which canreturns a boolean indicating > > > > whether the rule should be applied. > > > > -- > > > > Michael Mior > > > > mm...@apache.org > > > > > > > > Le ven. 9 août 2019 à 08:48, rahul patwari > > > > a écrit : > > > > > > > > > > Hi, > > > > > > > > > > We want to create a ConverterRule which converts the default > calling > > > > > Convention to external storage-specific calling convention > depending > > on > > > > the > > > > > Children nodes, like RelOptRule. > > > > > > > > > > For example, depending on the properties of the child nodes, we > want > > to > > > > > convert LogicalJoin to external system's specific Join > > implementation. > > > > > > > > > > Currently, ConverterRule > > > > > < > > > > > > > > > > https://github.com/apache/calcite/blob/5212d6c47e36995943f4d955a1714bf03eb08e7e/core/src/main/java/org/apache/calcite/rel/convert/ConverterRule.java#L75 > > > > > > > > > > cannot take Children and Child Policy is > > > > RelOptRuleOperandChildPolicy.ANY. > > > > > > > > > > What is the preferred way to achieve this task? > > > > > > > > > > Thanks, > > > > > Rahul > > > > > > > > > >
Re: Match Converter Rule based on Child Nodes
I guess in Rahul's case, the child node of join should be converted first, thus physical properties will be appended. But RelNode doesn't have a reference to parent, thus an independent RelOptRule is needed to convert LogicalJoin. Just as Michael said, you need to override the method of RelOptRule#matches and check the properties of child node. Best, Jin Stamatis Zampetakis 于2019年8月21日周三 下午5:50写道： > The Volcano planner works in a top-down fashion. It starts by transforming > the root and move towards the leafs of the plan. Due to this when > transforming a logical join its inputs are still in the logical convention > so in principle they should not have any physical properties. > > Normally when the physical join algorithm is chosen the respective rule > should be responsible for demanding the inputs of the join to have certain > properties. > > Long story short, I think in your use case you should not make the rule > match based on the properties of the child nodes but it should match > unconditionally and if necessary demand some properties in its inputs. If I > remember well the EnumerableMergeJoinRule follows this approach so you can > have a look there. > > Best, > Stamatis > > On Tue, Aug 20, 2019, 5:07 PM Michael Mior wrote: > > > If you just want to control whether the rule gets applied, you can > > override RelOptRule#matches which canreturns a boolean indicating > > whether the rule should be applied. > > -- > > Michael Mior > > mm...@apache.org > > > > Le ven. 9 août 2019 à 08:48, rahul patwari > > a écrit : > > > > > > Hi, > > > > > > We want to create a ConverterRule which converts the default calling > > > Convention to external storage-specific calling convention depending on > > the > > > Children nodes, like RelOptRule. > > > > > > For example, depending on the properties of the child nodes, we want to > > > convert LogicalJoin to external system's specific Join implementation. > > > > > > Currently, ConverterRule > > > < > > > https://github.com/apache/calcite/blob/5212d6c47e36995943f4d955a1714bf03eb08e7e/core/src/main/java/org/apache/calcite/rel/convert/ConverterRule.java#L75 > > > > > > cannot take Children and Child Policy is > > RelOptRuleOperandChildPolicy.ANY. > > > > > > What is the preferred way to achieve this task? > > > > > > Thanks, > > > Rahul > > >
Re: Assigning jira issue
Thanks a lot ~ Francis Francis Chuang 于2019年8月9日周五 上午11:13写道： > Hi Xing Jin, > > I've added you to the contributor role. > > Francis > > On 9/08/2019 12:52 pm, XING JIN wrote: > > Hi Francis, > > I have the same request with Sahith. > > Could you please add me to the contributor list as well ? > > My user name is -- jin xing > > Thanks a lot ! > > > > Francis Chuang 于2019年8月9日周五 上午5:58写道： > > > >> Hey Sahith, > >> > >> I've added you to the contributor role and assigned the issue to you. > >> > >> Francis > >> > >> On 8/08/2019 11:54 pm, sahith.re...@gmail.com wrote: > >>> Hello, > >>> > >>> I created issue CALCITE-3237, > >> https://issues.apache.org/jira/browse/CALCITE-3237, and I submitted a > PR > >> for it as well. Can this issue be assigned to me? My jira username is > >> snallapa. Thank you! > >>> > >>> Thanks, > >>> > >>> Sahith > >>> > >> > > >
Re: Assigning jira issue
Hi Francis, I have the same request with Sahith. Could you please add me to the contributor list as well ? My user name is -- jin xing Thanks a lot ! Francis Chuang 于2019年8月9日周五 上午5:58写道： > Hey Sahith, > > I've added you to the contributor role and assigned the issue to you. > > Francis > > On 8/08/2019 11:54 pm, sahith.re...@gmail.com wrote: > > Hello, > > > > I created issue CALCITE-3237, > https://issues.apache.org/jira/browse/CALCITE-3237, and I submitted a PR > for it as well. Can this issue be assigned to me? My jira username is > snallapa. Thank you! > > > > Thanks, > > > > Sahith > > >
[jira] [Created] (HDFS-2075) Add Number of Reporting Nodes to namenode web UI
Add Number of Reporting Nodes to namenode web UI -- Key: HDFS-2075 URL: https://issues.apache.org/jira/browse/HDFS-2075 Project: Hadoop HDFS Issue Type: Improvement Components: name-node, tools Affects Versions: 0.20.2, 0.20.1 Reporter: Xing Jin Priority: Minor Fix For: 0.20.3 The namenode web UI misses some information when safemode is on (e.g., the number of reporting nodes). These information will help us understand the system status. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2075) Add Number of Reporting Nodes to namenode web UI
[ https://issues.apache.org/jira/browse/HDFS-2075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xing Jin updated HDFS-2075: --- Attachment: HDFS-2075.patch.txt patch for the issue Add Number of Reporting Nodes to namenode web UI -- Key: HDFS-2075 URL: https://issues.apache.org/jira/browse/HDFS-2075 Project: Hadoop HDFS Issue Type: Improvement Components: name-node, tools Affects Versions: 0.20.1, 0.20.2 Reporter: Xing Jin Priority: Minor Fix For: 0.20.3 Attachments: HDFS-2075.patch.txt Original Estimate: 1h Remaining Estimate: 1h The namenode web UI misses some information when safemode is on (e.g., the number of reporting nodes). These information will help us understand the system status. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-1659) parse_url_tuple: a UDTF version of parse_url
[ https://issues.apache.org/jira/browse/HIVE-1659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xing Jin updated HIVE-1659: --- Attachment: HIVE-1659.patch This patch contains implementation of parse_url_tuple function, as well as a test file and a test output. parse_url_tuple: a UDTF version of parse_url - Key: HIVE-1659 URL: https://issues.apache.org/jira/browse/HIVE-1659 Project: Hadoop Hive Issue Type: New Feature Affects Versions: 0.5.0 Reporter: Ning Zhang Attachments: HIVE-1659.patch The UDF parse_url take s a URL, parse it and extract QUERY/PATH etc from it. However it can only extract an atomic value from the URL. If we want to extract multiple piece of information, we need to call the function many times. It is desirable to parse the URL once and extract all needed information and return a tuple in a UDTF. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1659) parse_url_tuple: a UDTF version of parse_url
[ https://issues.apache.org/jira/browse/HIVE-1659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xing Jin updated HIVE-1659: --- Attachment: HIVE-1659.patch2 generate patch after 'svn up' and in root directory parse_url_tuple: a UDTF version of parse_url - Key: HIVE-1659 URL: https://issues.apache.org/jira/browse/HIVE-1659 Project: Hadoop Hive Issue Type: New Feature Affects Versions: 0.5.0 Reporter: Ning Zhang Attachments: HIVE-1659.patch, HIVE-1659.patch2 The UDF parse_url take s a URL, parse it and extract QUERY/PATH etc from it. However it can only extract an atomic value from the URL. If we want to extract multiple piece of information, we need to call the function many times. It is desirable to parse the URL once and extract all needed information and return a tuple in a UDTF. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1659) parse_url_tuple: a UDTF version of parse_url
[ https://issues.apache.org/jira/browse/HIVE-1659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xing Jin updated HIVE-1659: --- Attachment: HIVE-1659.patch3 update show_functions.q and 'svn up' again. Thanks. Do I need to run all unitests on my machine? parse_url_tuple: a UDTF version of parse_url - Key: HIVE-1659 URL: https://issues.apache.org/jira/browse/HIVE-1659 Project: Hadoop Hive Issue Type: New Feature Affects Versions: 0.5.0 Reporter: Ning Zhang Attachments: HIVE-1659.patch, HIVE-1659.patch2, HIVE-1659.patch3 The UDF parse_url take s a URL, parse it and extract QUERY/PATH etc from it. However it can only extract an atomic value from the URL. If we want to extract multiple piece of information, we need to call the function many times. It is desirable to parse the URL once and extract all needed information and return a tuple in a UDTF. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1659) parse_url_tuple: a UDTF version of parse_url
[ https://issues.apache.org/jira/browse/HIVE-1659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xing Jin updated HIVE-1659: --- Status: Patch Available (was: Open) Affects Version/s: 0.5.0 Add new function parse_url_tuple for HIVE. parse_url_tuple: a UDTF version of parse_url - Key: HIVE-1659 URL: https://issues.apache.org/jira/browse/HIVE-1659 Project: Hadoop Hive Issue Type: New Feature Affects Versions: 0.5.0 Reporter: Ning Zhang The UDF parse_url take s a URL, parse it and extract QUERY/PATH etc from it. However it can only extract an atomic value from the URL. If we want to extract multiple piece of information, we need to call the function many times. It is desirable to parse the URL once and extract all needed information and return a tuple in a UDTF. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AVRO-484) avro C++ HTTP rpc support
avro C++ HTTP rpc support - Key: AVRO-484 URL: https://issues.apache.org/jira/browse/AVRO-484 Project: Avro Issue Type: Improvement Components: c++ Reporter: Xing Jin Upon Scott Banachowski's request, this is to add HTTP transport support for the rpc usage of avro C++ version. The current implementation uses HTTP POST to send over a request body (of any content) and the server side responds with a standard HTTP response body (of any content). Current status: it only requires a user to provide a streambuf input and define a function that contains the business logic to generate the serialized content into a response streambuf. testhttpserver would be a good example there. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[Tinyos-help] energy cost
Spam detection software, running on the system mail.Millennium.Berkeley.EDU, has identified this incoming email as possible spam. The original message has been attached to this so you can view it (if it isn't spam) or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Hi everyone, Does anyone know how to calculate the energy cost to send a packet from one mote to another one? Does the TinyOS offer some function or call to get it ?? Thanks a lot. Xing Jin How low will we go? Check out Yahoo! Messengers low PC-to-Phone call rates. [...] Content analysis details: (5.0 points, 5.0 required) pts rule name description -- -- 0.5 FROM_ENDS_IN_NUMS From: ends in numbers 2.2 FORGED_YAHOO_RCVD 'From' yahoo.com does not match 'Received' headers 0.0 HTML_MESSAGE BODY: HTML included in message 0.1 HTML_50_60 BODY: Message is 50% to 60% HTML 0.4 DNS_FROM_RFC_ABUSE RBL: Envelope sender in abuse.rfc-ignorant.org 0.5 DNS_FROM_RFC_WHOIS RBL: Envelope sender in whois.rfc-ignorant.org 1.4 DNS_FROM_RFC_POST RBL: Envelope sender in postmaster.rfc-ignorant.org The original message was not completely plain text, and may be unsafe to open with some email clients; in particular, it may contain a virus, or confirm that your address can receive spam. If you wish to view it, it may be safer to save it to a file and open it with an editor. ---BeginMessage--- Hi everyone, Does anyone know how to calculate the energy cost to send a packet from one mote to another one?Does the TinyOS offer some function or call to get it?? Thanks a lot. Xing Jin How low will we go? Check out Yahoo! Messengers low PC-to-Phone call rates.---End Message--- ___ Tinyos-help mailing list Tinyos-help@Millennium.Berkeley.EDU https://mail.millennium.berkeley.edu/cgi-bin/mailman/listinfo/tinyos-help
[Tinyos-help] the range of the mica2 temperature readings
Hi everyone, Wouldsomeone tell me the range of the mica2 thermistor readings?Or the ceilus temperature reading range (format like :0~30C)?Thanks a lot. Xing Jin Yahoo! Music Unlimited - Access over 1 million songs. Try it free. ___ Tinyos-help mailing list Tinyos-help@Millennium.Berkeley.EDU https://mail.millennium.berkeley.edu/cgi-bin/mailman/listinfo/tinyos-help
[Tinyos-help] delay time and memory calculation
Hi everyone , If I wanna get the time and memory cost on some paragragh of the code(ex, a function call), what should I do ? For real expriment, not for simulation. Does tinyos offer some system call to get the real time value or memory cost value at some time point? Thanks a lot . Xing Jin __Do You Yahoo!?Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com ___ Tinyos-help mailing list Tinyos-help@Millennium.Berkeley.EDU https://mail.millennium.berkeley.edu/cgi-bin/mailman/listinfo/tinyos-help
[Tinyos-help] modification on TOSBase and show them on Xlisten
Hi , I am collecting data info using XSensorMDA500 on 3 different mote nodes and send them to a base station. And on base staion , I changed the TOSBase to calculate the average value of each three. Butthe averagecan not been successfully gotten and shownthrought Xlisten.Take temerature forexample,mythought is just pick thethermistor value ofeach packet and add three of them together and get 1/3 of the sum. Is this a correct way to get the average value of temperature?However, what I care more is how to display the average value continously on the pc? I set a index which range from 0~2 and make it return to 0 after it arrives at 2. Everytime ,when it is 2, call the UARTSend to send the info. But it can not get info to display through Xlisten. Can anyone give me some hint?I appreciate it very much. Xing Jin __Do You Yahoo!?Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com ___ Tinyos-help mailing list Tinyos-help@Millennium.Berkeley.EDU https://mail.millennium.berkeley.edu/cgi-bin/mailman/listinfo/tinyos-help
[Tinyos-help] print problem
Hi, everyone, I am trying to printsome value statement on PC to in order to make sure the program is on the correct way. For example, base station receives three values from other motes and get the average, then dosome other calculation. I wanttoprint out the average value on the pc whenI am doing the experiment.But I do not know how. I thought DBG display was just for simulation.(or I am wrong???) Is there any statement that can print out the value, or write the value into a file??? Can anyone tell me? Thanks a lot. XingJin Sneak preview the all-new Yahoo.com. It's not radically different. Just radically better. ___ Tinyos-help mailing list Tinyos-help@Millennium.Berkeley.EDU https://mail.millennium.berkeley.edu/cgi-bin/mailman/listinfo/tinyos-help
[Tinyos-help] Surge-Reliable Running problem!???
om1 -dpart=ATmega128 --wr_fuse_e=ff --erase --uplad if=build/mica2/main.srec.outFirmware Version: 2.1Atmel AVR ATmega128 is found.Uploading: flash Fuse Extended Byte set to 0xffcan anyone tell me what happened ? and How to slove this problem?Thanks a lot. Xing Jin Love cheap thrills? Enjoy PC-to-Phone calls to 30+ countries for just 2¢/min with Yahoo! Messenger with Voice.___ Tinyos-help mailing list Tinyos-help@Millennium.Berkeley.EDU https://mail.millennium.berkeley.edu/cgi-bin/mailman/listinfo/tinyos-help
[Tinyos-help] about surge_reliable test on xbow
Hi everyone, I am new to tinyos. I am doing a test on surge_reliable.I use Xbow to do this test.Does anyone tell me what the result looks like? I download the program on the moteusing "make mica2 install, 0 mib510,com1" and "export [EMAIL PROTECTED],java net.tinyos.tools.Listen" to listen.But no data is received. And after I take the mote off the programming broad, d0/d1/d2(the red ,green yellow leds), none of them is on.No ligth to show the mote is in work. I do not know which step is wrong. I did not change any line in the code. Please help me !!! I appreciate it. Thanks in advance. Xing Jin Blab-away for as little as 1¢/min. Make PC-to-Phone Calls using Yahoo! Messenger with Voice.___ Tinyos-help mailing list Tinyos-help@Millennium.Berkeley.EDU https://mail.millennium.berkeley.edu/cgi-bin/mailman/listinfo/tinyos-help
[Tinyos-help] how to slove the aggregation part ?
Hi , I am a beginner in this field. When I complied my code, it says that TestTempM.nc:82: warning: passing argument 1 of `Send.send' makes pointer from integer without a castTestTempM.nc:82: too many arguments to function `Send.send'TestTempM.nc: In function `ReceiveMsg.receive':TestTempM.nc:148: warning: passing argument 1 of `Send.send' makes pointer frominteger without a castTestTempM.nc:148: too many arguments to function `Send.send'But I do not know whichcausedthis. Does anyone have the same experience ? I will appreciate it very much if someone helps me. Thanks.___ Tinyos-help mailing list Tinyos-help@Millennium.Berkeley.EDU https://mail.millennium.berkeley.edu/cgi-bin/mailman/listinfo/tinyos-help
How to pop up the symbol dialog in application on Tungsten C?
Hi all, As Tungsten C use keyboard and only pop up symbol dialog when special symboles are needed. It seems it only works on Palm buildin controls such as Field, is there any function or event for poping up the dialog anytime when application needs it? Thanks, Xing Jin -- For information on using the Palm Developer Forums, or to unsubscribe, please see http://www.palmos.com/dev/support/forums/
Re: Does anyone notice this bug on Tungsten C?
I also reported it yesterday, thanks a lot, and I will check it on other TCs. cheers Xing Flex [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED] Yep - in the developer area. I've already reported 3 :) by the way my TC have no such bug - hardware issue. Chris Apers [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED] Is there a bug tracker for palmos related bugs ? Hi, I found an interesting bug whilst I am porting our application from Sony Clie to Tungsten C: There is no pen down event at screen Y = 310 while the Tungsten C screen size is 320 * 320. The bug also is found on all other applications installed in the device, I assume it should be a problem on the hardware(Touch Screen) rather than a OS bug. Any suggestion or comment is welcome. Cheers, Xing -- For information on using the Palm Developer Forums, or to unsubscribe, please see http://www.palmos.com/dev/support/forums/ -- For information on using the Palm Developer Forums, or to unsubscribe, please see http://www.palmos.com/dev/support/forums/
Does anyone notice this bug on Tungsten C?
Hi, I found an interesting bug whilst I am porting our application from Sony Clie to Tungsten C: There is no pen down event at screen Y = 310 while the Tungsten C screen size is 320 * 320. The bug also is found on all other applications installed in the device, I assume it should be a problem on the hardware(Touch Screen) rather than a OS bug. Any suggestion or comment is welcome. Cheers, Xing -- For information on using the Palm Developer Forums, or to unsubscribe, please see http://www.palmos.com/dev/support/forums/
Re: Where can I find a fit SDK for Tungsten 5-way navigator?
But you can not find any information about the SDK unless you register and login the Palm SG program. Ben Combee [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED] At 10:40 AM 7/10/2003, Xing Jin wrote: Many thanks! Finally I found it after I registered as Plugin project member. What's a strange and stupid things! Why is this strange and stupid? The 5-way navigator is a proprietary feature of devices from Palm Solutions Group, so it makes sense that the description of the APIs and keystrokes would be part of the Palm SG developer program, not the PalmSource generic Palm OS SDK. -- Ben Combee [EMAIL PROTECTED] CodeWarrior for Palm OS technical lead Palm OS programming help @ www.palmoswerks.com -- For information on using the Palm Developer Forums, or to unsubscribe, please see http://www.palmos.com/dev/support/forums/
Where can I find a fit SDK for Tungsten 5-way navigator?
Hi, I failed to find the event definition about the 5-way navigator supported on Tungsten C device, could anyone give me hand on this? Thanks a lot, Xing Jin -- For information on using the Palm Developer Forums, or to unsubscribe, please see http://www.palmos.com/dev/support/forums/
Re: Where can I find a fit SDK for Tungsten 5-way navigator?
Thanks for your reply and I actually suffered from finding the sdk on the web you point out today. I downloaded the newest PalmOS5 RS2 SDK but there is nothing about the 5-way navigator. Any other suggestion? cheers, Xing Aaron Ardiri [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED] I failed to find the event definition about the 5-way navigator supported on Tungsten C device, could anyone give me hand on this? seeding area :) www.palmos.com --- Aaron Ardiri[EMAIL PROTECTED] CEO - CTO +46 70 656 1143 Mobile Wizardry http://www.mobilewizardry.com/ -- For information on using the Palm Developer Forums, or to unsubscribe, please see http://www.palmos.com/dev/support/forums/
Re: Where can I find a fit SDK for Tungsten 5-way navigator?
Many thanks! Finally I found it after I registered as Plugin project member. What's a strange and stupid things! regards, Xing Aaron Ardiri [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED] Thanks for your reply and I actually suffered from finding the sdk on the web you point out today. I downloaded the newest PalmOS5 RS2 SDK but there is nothing about the 5-way navigator. it is a seperate download. you need to login to get it. this means you need to sign NDA and stuff - it is there. --- Aaron Ardiri[EMAIL PROTECTED] CEO - CTO +46 70 656 1143 Mobile Wizardry http://www.mobilewizardry.com/ -- For information on using the Palm Developer Forums, or to unsubscribe, please see http://www.palmos.com/dev/support/forums/
