Re: [DISCUSSION] Extension of Metadata Query

[Yuzhao Chen]

Thanks Julian for your detail reply,


1. I checked the Janino gened-code and the RelMetadataQuery/RelMetadataProvidor 
and almost can make sure MetadataQuery only use the 
RelMetadataProvidor#handlers method for multiple dispatch, the 
RelMetadataProvidor#apply is only used for MetadataFactory.
2. I agree that we should provide new RelMetadataProvider for extension, but 
the RelMetadataQuery has top level interfaces for metadata query, these top 
level interfaces should be extendable and sync with our metadata handlers.


Best,
Danny Chan
在 2019年6月5日 +0800 AM1:48，Julian Hyde ，写道：
> > 1. Why we have 2 methods in RelMetadataProvider?
>
> The metadata system is complicated. We need to allow multiple handlers
> for any given call. So, making a metadata call involves multiple
> dispatch [1] based on the metadata method being called, the type of
> RelNode, and the handlers that are registered. Also it needs to cache
> results, and detect cycles. And the dispatch needs to be efficient, so
> we generate janino code to do the dispatch, and re-generate when new
> handlers or sub-classes of RelNode are added.
>
> I forget details, the two methods are basically required to allow us
> to generate code to do the dispatch.
>
> > 2. We should make the RelMetadataQuery in RelOptCluster pluggable.
>
> I disagree. RelMetadataQuery is not an extension point. Its sole
> purpose is to to keep state (the cache and cycle-checking).
> RelMetadataProvider is the extension point. (By analogy, if you are
> un-parsing an AST, you let each AST sub-class handle unparsing itself,
> but the unparsed text goes to a simple StringBuilder. RelMetadataQuery
> is in the role of the StringBuilder. In a complex system, it is nice
> to keep some of the components simple, or at least keep them to
> prescribed roles.)
>
> Julian
>
> [1] https://en.wikipedia.org/wiki/Multiple_dispatch
>
> On Sun, Jun 2, 2019 at 11:19 PM Yuzhao Chen  wrote:
> >
> > Currently we provide answer to metadata query through RelMetadataProvider 
> > [1], there are some sub-classes of it:
> >
> > RelMetadataProvider
> > |
> > |- VolcanoRelMetadataProvider
> > |- ChainedRelMetadataProvider/DefaultRelMetadataProvider
> > |- HepRelMetadataProvider
> > |- CachingRelMetadataProvider
> > |- ReflectiveRelMetadataProvider
> > |- JaninoRelMetadataProvider
> >
> > The RelMetadataProvider has two methods: #apply and #handlers, the #apply 
> > method seems a programming interface and there is a demo code how we can 
> > use it:
> >
> > RelMetadataProvider provider;
> > LogicalFilter filter;
> > RexNode predicate;
> > Function function =
> > provider.apply(LogicalFilter.class, Selectivity.class};
> > Selectivity selectivity = function.apply(filter);
> > Double d = selectivity.selectivity(predicate);
> >
> > But let's see our RelOptCluster's member variables[2], there are 
> > MetadataFactory and RelMetadataQuery which all can be used to query the 
> > metadata, for MetadataFactory, there is a default impl named 
> > MetadataFactoryImpl which will invoke RelMetadataProvider#apply internally, 
> > for RelMetadataQuery, it will invoke RelMetadataProvider#handlers (finally 
> > composed and codeden by JaninoRelMetadataProvider).
> >
> > In our planning phrase, we can invoke RelOptRuleCall#getMetadataQuery to 
> > get the MQ and query the metadata.
> >
> > For extension of metadata handlers, we can set our customized 
> > RelMetadataProvider in RelOptCluster[3]. But for RelMetadataQuery, we have 
> > no way to extend it now, because the RelOptCluster always has a singleton 
> > instance [4] which is only the default implementation.
> >
> >
> > My question is as follows:
> >
> > 1. Why we have 2 methods in RelMetadataProvider, and why we need the 
> > MetadataFactory and RelMetadataProvider#apply ? It seems that it's function 
> > is already been overriden by RelMetadataQuery(The difference is that 
> > MetadataFactory use Reflection and RelMetadataQuery use gened bytes code).
> > 2. We should make the RelMetadataQuery in RelOptCluster pluggable.
> >
> >
> > [1] 
> > https://github.com/apache/calcite/blob/b0e83c469ff57257c1ea621ff943ca76f626a9b7/core/src/main/java/org/apache/calcite/rel/metadata/RelMetadataProvider.java#L38
> > [2] 
> > https://github.com/apache/calcite/blob/b0e83c469ff57257c1ea621ff943ca76f626a9b7/core/src/main/java/org/apache/calcite/plan/RelOptCluster.java#L49
> > [3] 
> > https://github.com/apache/calcite/blob/b0e83c469ff57257c1ea621ff943ca76f626a9b7/core/src/main/java/org/apache/calcite/plan/RelOptCluster.java#L135
> > [4] 
> > https://github.com/apache/calcite/blob/b0e83c469ff57257c1ea621ff943ca76f626a9b7/core/src/main/java/org/apache/calcite/plan/RelOptCluster.java#L151
> >
> >
> >
> > Best,
> > Danny Chan

Re: [DISCUSSION] Extension of Metadata Query

[Chunwei Lei]

Thanks for raising this, Danny.  Actually I have the same question too.

> RelMetadataQuery is not an extension point. Its sole purpose is to to
keep state (the cache and cycle-checking)
I think users may extend the RelMetadataQuery if wanting to query more
metadata, such as TopK values.


Best,
Chunwei


On Wed, Jun 5, 2019 at 6:44 AM Stamatis Zampetakis 
wrote:

> Thanks for bringing this up Danny.
>
> I guess the discussion came up due to CALCITE-2885 [1]. Looking back into
> it, I am not sure that the intention there is to make the RelMetadataQuery
> pluggable. We could possibly solve the performance problem without
> extending the RelMetadataQuery. We have to look again into this case.
>
> For more details regarding the existence of the two methods in
> RelMetadataProvider have a look in CALCITE-604 [2]. More general for the
> design of RelMetadataProvider you may find useful the description in [3].
>
> Best,
> Stamatis
>
> [1] https://issues.apache.org/jira/browse/CALCITE-2885
> [2] https://issues.apache.org/jira/browse/CALCITE-604
> [3]
>
> https://web.archive.org/web/20140624040836/www.hydromatic.net/wiki/RelationalExpressionMetadata/
>
> On Tue, Jun 4, 2019 at 7:48 PM Julian Hyde  wrote:
>
> > > 1. Why we have 2 methods in RelMetadataProvider?
> >
> > The metadata system is complicated. We need to allow multiple handlers
> > for any given call. So, making a metadata call involves multiple
> > dispatch [1] based on the metadata method being called, the type of
> > RelNode, and the handlers that are registered. Also it needs to cache
> > results, and detect cycles. And the dispatch needs to be efficient, so
> > we generate janino code to do the dispatch, and re-generate when new
> > handlers or sub-classes of RelNode are added.
> >
> > I forget details, the two methods are basically required to allow us
> > to generate code to do the dispatch.
> >
> > > 2. We should make the RelMetadataQuery in RelOptCluster pluggable.
> >
> > I disagree. RelMetadataQuery is not an extension point. Its sole
> > purpose is to to keep state (the cache and cycle-checking).
> > RelMetadataProvider is the extension point. (By analogy, if you are
> > un-parsing an AST, you let each AST sub-class handle unparsing itself,
> > but the unparsed text goes to a simple StringBuilder. RelMetadataQuery
> > is in the role of the StringBuilder. In a complex system, it is nice
> > to keep some of the components simple, or at least keep them to
> > prescribed roles.)
> >
> > Julian
> >
> > [1] https://en.wikipedia.org/wiki/Multiple_dispatch
> >
> > On Sun, Jun 2, 2019 at 11:19 PM Yuzhao Chen 
> wrote:
> > >
> > > Currently we provide answer to metadata query through
> > RelMetadataProvider [1], there are some sub-classes of it:
> > >
> > > RelMetadataProvider
> > > |
> > > |- VolcanoRelMetadataProvider
> > > |- ChainedRelMetadataProvider/DefaultRelMetadataProvider
> > > |- HepRelMetadataProvider
> > > |- CachingRelMetadataProvider
> > > |- ReflectiveRelMetadataProvider
> > > |- JaninoRelMetadataProvider
> > >
> > > The RelMetadataProvider has two methods: #apply and #handlers, the
> > #apply method seems a programming interface and there is a demo code how
> we
> > can use it:
> > >
> > > RelMetadataProvider provider;
> > > LogicalFilter filter;
> > > RexNode predicate;
> > > Function function =
> > > provider.apply(LogicalFilter.class, Selectivity.class};
> > > Selectivity selectivity = function.apply(filter);
> > > Double d = selectivity.selectivity(predicate);
> > >
> > > But let's see our RelOptCluster's member variables[2], there are
> > MetadataFactory and RelMetadataQuery which all can be used to query the
> > metadata, for MetadataFactory, there is a default impl named
> > MetadataFactoryImpl which will invoke RelMetadataProvider#apply
> internally,
> > for RelMetadataQuery, it will invoke RelMetadataProvider#handlers
> (finally
> > composed and codeden by JaninoRelMetadataProvider).
> > >
> > > In our planning phrase, we can invoke RelOptRuleCall#getMetadataQuery
> to
> > get the MQ and query the metadata.
> > >
> > > For extension of metadata handlers, we can set our customized
> > RelMetadataProvider in RelOptCluster[3]. But for RelMetadataQuery, we
> have
> > no way to extend it now, because the RelOptCluster always has a singleton
> > instance [4] which is only the default implementation.
> > >
> > >
> > > My question is as follows:
> > >
> > > 1. Why we have 2 methods in RelMetadataProvider, and why we need the
> > MetadataFactory and RelMetadataProvider#apply ? It seems that it's
> function
> > is already been overriden by RelMetadataQuery(The difference is that
> > MetadataFactory use Reflection and RelMetadataQuery use gened bytes
> code).
> > > 2. We should make the RelMetadataQuery in RelOptCluster pluggable.
> > >
> > >
> > > [1]
> >
> https://github.com/apache/calcite/blob/b0e83c469ff57257c1ea621ff943ca76f626a9b7/core/src/main/java/org/apache/calcite/rel/metadata/RelMetadataProvider.java#L38
> > > [2]
> >
> https:

Re: Correlate Join SemiJoin transformation

[Yuzhao Chen]

I think the complexity mostly comes from the value generator, which is the 
engineering foundation of current decorrelation.

Best,
Danny Chan
在 2019年6月5日 +0800 AM10:45，Chunwei Lei ，写道：
> Thanks for raising this, Haisheng.
>
> Do you mean that we should have better subquery unnesting?
>
>
>
>
> Best,
> Chunwei
>
>
> On Tue, Jun 4, 2019 at 8:27 AM Haisheng Yuan  wrote:
>
> > Hi,
> >
> > Since we have commited CALCITE-2969, and in the next release of 1.21, we
> > will deprecate EquiJoin and make enumerable hash join support non-equi join
> > condition, the optimization door is open to us.
> >
> > Currently Calcite often generates complicated and inefficient plan for
> > boolean context correlated subquery (putting value context subquery aside,
> > which is more complex during unnesting). e.g.
> > https://issues.apache.org/jira/browse/CALCITE-2857
> > https://issues.apache.org/jira/browse/CALCITE-2874
> >
> > There are many more cases if you want to try and find out. Some underlying
> > reasons are:
> > 1. SemJoin doesn't support non-equi condition
> > 2. Correlate comes with Aggregate in many cases
> >
> > Sometimes, SubqueryRemoveRule generates Correlate with Aggregate,
> > sometimes generates Join with Aggregate. Then we use SemiJoinRule to match
> > Join(Rel, Agg) pattern to see whether we can transform it to semijoin or
> > not. This is counter intuitive, because we already know it will be a
> > SemiJoin or AntiSemi when we generate Correlate for the subquery. In
> > current way, we may miss the chance to get back SemiJoin for it, and we
> > almost lose the opportunity to do IndexScan if there is index available.
> >
> > In addition, I see 2 rules JoinAddRedundantSemiJoinRule and
> > SemiJoinRemoveRule seem to provide the chance to do index scan. But through
> > the test cases related with the 2 rules, I don't see any chance for a
> > better plan with them and they don't seem to be able to generate plan with
> > index scan. I doubt whether they are used at all. Can someone shed some
> > light on this and give us some examples, if I get it wrong?
> >
> > The transformation flow for boolean context sub-query in my head is:
> > sub-query -> Correate(Semi/AntiSemi) without Aggregate (SubqueryRemoveRule)
> > Correlate(Semi/AntiSemi) -> NestedLoopJoin (correlate implementation rule)
> > Correlate(Semi/AntiSemi) -> IndexedNestedLoopJoin (index apply rule)
> > Correlate(Semi/AntiSemi) -> SemiJoin/AntiSemiJoin (Semi Correlate to Semi
> > Join rule)
> > SemiJoin -> InnerJoin(rel, Agg) (SemiJoin to Join rule)
> > InnerJoin(rel, Agg) -> InnerJoin(Agg, rel) (Join reorder rule)
> > SemiJoin/InnerJoin -> HashJoin/MergeJoin/NLJ (Join implementation rule)
> >
> > This is a long shot, and will involve tons of changes.
> > Any thoughts?
> >
> > - Haisheng Yuan

Re: Giving the Calcite logo some love

[Yuzhao Chen]

I would prefer 
https://github.com/zabetak/calcite/blob/calcite-logo/site/img/logo-alt2-v1.svg
if the hammer can be smaller :)

Best,
Danny Chan
在 2019年6月5日 +0800 AM8:32，Albert ，写道：
> I will vote "logo-alt1-v5.svg
> "
> , looks nice.
>
> On Wed, Jun 5, 2019 at 6:11 AM Stamatis Zampetakis 
> wrote:
>
> > I created a branch to gather all alternative logos [1].
> >
> > Among the two aforementioned proposals, I added a few more variants (check
> > logo-alt* under the img directory).
> > Have a look and let me know what you think. I'm open to any ideas and
> > suggestions.
> >
> > [1] https://github.com/zabetak/calcite/tree/calcite-logo/site/img
> >
> > On Tue, Jun 4, 2019 at 7:25 PM Julian Hyde  wrote:
> >
> > > I prefer both over the current logo. (And I made the current logo.)
> > >
> > > Let's keep the discussion going, and get to a new logo.
> > >
> > > On Tue, Jun 4, 2019 at 9:42 AM Ivan Grgurina 
> > wrote:
> > > >
> > > > I prefer the one Daniel sent. It looks cleaner, but maybe the "periodic
> > > table" logo can be made better by simplifying it, the shadow on the
> > letter
> > > C is... unusual.
> > > >
> > > > 
> > > > From: Stamatis Zampetakis 
> > > > Sent: Tuesday, June 4, 2019 6:32 PM
> > > > To: dev@calcite.apache.org; humbed...@apache.org
> > > > Subject: Re: Giving the Calcite logo some love
> > > >
> > > > Thanks for digging this out Daniel!
> > > >
> > > > At this point we have two candidates:
> > > > http://humbedooh.com/calcite-proposed.svg
> > > > https://svgshare.com/s/86r
> > > >
> > > > Do we like any of above more than our current logo (the way they are or
> > > > with slight modifications) ?
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Jun 3, 2019 at 3:15 AM Yuzhao Chen 
> > wrote:
> > > >
> > > > > Oh, I see a big hammer, thanks Daniel !
> > > > >
> > > > > Best,
> > > > > Danny Chan
> > > > > 在 2019年6月3日 +0800 AM6:21，Daniel Gruno ，写道：
> > > > > > Found it!
> > > > > >
> > > > > > http://humbedooh.com/calcite-proposed.svg
> > > > > >
> > > > > > Thanks, Wayback Machine!
> > > > > >
> > > > > > On 19/05/2019 10.01, Stamatis Zampetakis wrote:
> > > > > > > Hi all,
> > > > > > >
> > > > > > > We started this discussion about a year ago and many people were
> > > > > > > positive with the idea of having a new logo for Calcite.
> > > > > > > We had some nice proposals at the time and maybe now somebody
> > else
> > > has
> > > > > > > also new ideas/designs to contribute.
> > > > > > >
> > > > > > > @Daniel: Is there any chance that you still have the logos you
> > > proposed
> > > > > > > somewhere available? The old link [1] does not work anymore.
> > > > > > >
> > > > > > > Best,
> > > > > > > Stamatis
> > > > > > >
> > > > > > > [1]
> > > > >
> > > http://www.apache.org/logos/comdev-test/res/calcite/calcite-proposed.svg
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Aug 30, 2018 at 12:02 AM Stamatis Zampetakis <
> > > > > zabe...@gmail.com
> > > > > > > > wrote:
> > > > > > >
> > > > > > > Vladimir>Could you flip rhombus so it goes right-up?
> > > > > > >
> > > > > > > https://svgshare.com/s/86r
> > > > > > >
> > > > > > > I hope this is what you meant.
> > > > > > >
> > > > > > > Best,
> > > > > > > Stamatis
> > > > > > >
> > > > > > > Στις Τετ, 29 Αυγ 2018 στις 6:30 μ.μ., ο/η Michael Mior
> > > > > > > mailto:mm...@apache.org>> έγραψε:
> > > > > > >
> > > > > > > Just a note since we're on the topic that whatever logos we come
> > > > > > > up with
> > > > > > > should be sure to have TM clearly indicated.
> > > > > > >
> > > > > > > --
> > > > > > > Michael Mior
> > > > > > > mm...@apache.org 
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Le mer. 29 août 2018 à 03:13, Julian Hyde
> > > > > > > mailto:jhyde.apa...@gmail.com>> a
> > écrit :
> > > > > > >
> > > > > > > > Yes indeed!
> > > > > > > >
> > > > > > > > If someone feels inspired to produce a logo, here’s my
> > > > > > > suggestion of a
> > > > > > > > theme/image: a spider, specifically a Barn Spider (Araneus
> > > > > > > Cavaticus)[1].
> > > > > > > > It was the origin of the name “avatica”, connects and spins
> > > > > > > webs, and the
> > > > > > > > eponymous individual in Charlotte’s Web had rather exceptional
> > > > > > > > communication skills.
> > > > > > > >
> > > > > > > > Julian
> > > > > > > >
> > > > > > > > [1] https://en.m.wikipedia.org/wiki/Barn_spider
> > > > > > > >
> > > > > > > > > On Aug 28, 2018, at 9:49 PM, Francis Chuang
> > > > > > > mailto:francischu...@apache.org>>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > The designs I have seen so far look really good! Would it
> > > > > > > also make
> > > > > > > > sense to design a variant for Avatica as well? This is what
> > > > > > > the current
> > > > > > > > Avatica logo looks like:
> > > > > > > https://calcite.apache.org/avatica/img/logo.png
> > > > > >

Re: Correlate Join SemiJoin transformation

[Chunwei Lei]

Thanks for raising this, Haisheng.

Do you mean that we should have better subquery unnesting?




Best,
Chunwei


On Tue, Jun 4, 2019 at 8:27 AM Haisheng Yuan  wrote:

> Hi,
>
> Since we have commited CALCITE-2969, and in the next release of 1.21, we
> will deprecate EquiJoin and make enumerable hash join support non-equi join
> condition, the optimization door is open to us.
>
> Currently Calcite often generates complicated and inefficient plan for
> boolean context correlated subquery (putting value context subquery aside,
> which is more complex during unnesting). e.g.
> https://issues.apache.org/jira/browse/CALCITE-2857
> https://issues.apache.org/jira/browse/CALCITE-2874
>
> There are many more cases if you want to try and find out. Some underlying
> reasons are:
> 1. SemJoin doesn't support non-equi condition
> 2. Correlate comes with Aggregate in many cases
>
> Sometimes, SubqueryRemoveRule generates Correlate with Aggregate,
> sometimes generates Join with Aggregate. Then we use SemiJoinRule to match
> Join(Rel, Agg) pattern to see whether we can transform it to semijoin or
> not. This is counter intuitive, because we already know it will be a
> SemiJoin or AntiSemi when we generate Correlate for the subquery. In
> current way, we may miss the chance to get back SemiJoin for it, and we
> almost lose the opportunity to do IndexScan if there is index available.
>
> In addition, I see 2 rules JoinAddRedundantSemiJoinRule and
> SemiJoinRemoveRule seem to provide the chance to do index scan. But through
> the test cases related with the 2 rules, I don't see any chance for a
> better plan with them and they don't seem to be able to generate plan with
> index scan. I doubt whether they are used at all. Can someone shed some
> light on this and give us some examples, if I get it wrong?
>
> The transformation flow for boolean context sub-query in my head is:
> sub-query -> Correate(Semi/AntiSemi) without Aggregate (SubqueryRemoveRule)
> Correlate(Semi/AntiSemi) -> NestedLoopJoin (correlate implementation rule)
> Correlate(Semi/AntiSemi) -> IndexedNestedLoopJoin (index apply rule)
> Correlate(Semi/AntiSemi) -> SemiJoin/AntiSemiJoin (Semi Correlate to Semi
> Join rule)
> SemiJoin -> InnerJoin(rel, Agg) (SemiJoin to Join rule)
> InnerJoin(rel, Agg) ->  InnerJoin(Agg, rel) (Join reorder rule)
> SemiJoin/InnerJoin -> HashJoin/MergeJoin/NLJ (Join implementation rule)
>
> This is a long shot, and will involve tons of changes.
> Any thoughts?
>
> - Haisheng Yuan

Re: Execute multiple RelNodes from single RelNode

[Chunwei Lei]

Hi, Ivan.

Could you please explain more detail for your question? Maybe give an
example.


Best,
Chunwei


On Tue, Jun 4, 2019 at 8:59 PM Ivan Grgurina  wrote:

> Can I create multiple top-level RelNodes out of the single SqlNode in
> SqlToRelConverter class or single RelNode through RelOptRules and have
> Calcite push it to the databases?
>
>
> *Ivan Grgurina*
>
> Research Assistant (ZEMRIS)
> --
>
> 
> 
>
>
>

Re: [DISCUSS] Automated website builds

[Chunwei Lei]

+1. Thanks for your work, Francis.



Best,
Chunwei


On Wed, Jun 5, 2019 at 1:46 AM Josh Elser  wrote:

> +1
>
> On 6/4/19 8:43 AM, Michael Mior wrote:
> > I see no reason not to test this out. I'd say go for it! Thanks Francis
> :)
> > --
> > Michael Mior
> > mm...@apache.org
> >
> > Le lun. 3 juin 2019 à 21:04, Francis Chuang 
> a écrit :
> >>
> >> A few months ago, I raised the possibility to automating our website
> >> builds, so that we will not need to manually build the site on our
> >> machines and push it to the calcite-site repository.
> >>
> >> I've been quite busy and haven't been able to look into it, however I am
> >> planning to look into in the next few weeks.
> >>
> >> In generally, I think this is quite doable if we run the website build
> >> on the git-websites node, however I'd like to look at what's installed
> >> on those nodes and whether docker is available. It would also be a good
> >> opportunity to run some test builds and I also have a few proposal
> >> regarding the way websites are currently being built that I'd like to
> >> confirm and test first.
> >>
> >> If it's okay, I'd like to create a test-site branch on calcite,
> >> calcite-avatica and calcite-avatica-go as well as calcite-site,
> >> calcite-avatica-site and calcite-avatica-go-site jobs on Jenkins to test
> >> the waters.
> >>
> >> Please let me know if this is okay.
> >>
> >> Francis
>

Re: Giving the Calcite logo some love

[Chunwei Lei]

I will vote the one Daniel sent[1]. But I think t will be better if the
first letter can be upper case, namely Calcite.


[1] http://humbedooh.com/calcite-proposed.svg

Best,
Chunwei


On Wed, Jun 5, 2019 at 8:32 AM Albert  wrote:

> I will vote "logo-alt1-v5.svg
> <
> https://github.com/zabetak/calcite/blob/calcite-logo/site/img/logo-alt1-v5.svg
> >"
> , looks nice.
>
> On Wed, Jun 5, 2019 at 6:11 AM Stamatis Zampetakis 
> wrote:
>
> > I created a branch to gather all alternative logos [1].
> >
> > Among the two aforementioned proposals, I added a few more variants
> (check
> > logo-alt* under the img directory).
> > Have a look and let me know what you think. I'm open to any ideas and
> > suggestions.
> >
> > [1] https://github.com/zabetak/calcite/tree/calcite-logo/site/img
> >
> > On Tue, Jun 4, 2019 at 7:25 PM Julian Hyde  wrote:
> >
> > > I prefer both over the current logo. (And I made the current logo.)
> > >
> > > Let's keep the discussion going, and get to a new logo.
> > >
> > > On Tue, Jun 4, 2019 at 9:42 AM Ivan Grgurina 
> > wrote:
> > > >
> > > > I prefer the one Daniel sent. It looks cleaner, but maybe the
> "periodic
> > > table" logo can be made better by simplifying it, the shadow on the
> > letter
> > > C is... unusual.
> > > >
> > > > 
> > > > From: Stamatis Zampetakis 
> > > > Sent: Tuesday, June 4, 2019 6:32 PM
> > > > To: dev@calcite.apache.org; humbed...@apache.org
> > > > Subject: Re: Giving the Calcite logo some love
> > > >
> > > > Thanks for digging this out Daniel!
> > > >
> > > > At this point we have two candidates:
> > > > http://humbedooh.com/calcite-proposed.svg
> > > > https://svgshare.com/s/86r
> > > >
> > > > Do we like any of above more than our current logo (the way they are
> or
> > > > with slight modifications) ?
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Jun 3, 2019 at 3:15 AM Yuzhao Chen 
> > wrote:
> > > >
> > > > > Oh, I see a big hammer, thanks Daniel !
> > > > >
> > > > > Best,
> > > > > Danny Chan
> > > > > 在 2019年6月3日 +0800 AM6:21，Daniel Gruno ，写道：
> > > > > > Found it!
> > > > > >
> > > > > > http://humbedooh.com/calcite-proposed.svg
> > > > > >
> > > > > > Thanks, Wayback Machine!
> > > > > >
> > > > > > On 19/05/2019 10.01, Stamatis Zampetakis wrote:
> > > > > > > Hi all,
> > > > > > >
> > > > > > > We started this discussion about a year ago and many people
> were
> > > > > > > positive with the idea of having a new logo for Calcite.
> > > > > > > We had some nice proposals at the time and maybe now somebody
> > else
> > > has
> > > > > > > also new ideas/designs to contribute.
> > > > > > >
> > > > > > > @Daniel: Is there any chance that you still have the logos you
> > > proposed
> > > > > > > somewhere available? The old link [1] does not work anymore.
> > > > > > >
> > > > > > > Best,
> > > > > > > Stamatis
> > > > > > >
> > > > > > > [1]
> > > > >
> > >
> http://www.apache.org/logos/comdev-test/res/calcite/calcite-proposed.svg
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Aug 30, 2018 at 12:02 AM Stamatis Zampetakis <
> > > > > zabe...@gmail.com
> > > > > > > > wrote:
> > > > > > >
> > > > > > > Vladimir>Could you flip rhombus so it goes right-up?
> > > > > > >
> > > > > > > https://svgshare.com/s/86r
> > > > > > >
> > > > > > > I hope this is what you meant.
> > > > > > >
> > > > > > > Best,
> > > > > > > Stamatis
> > > > > > >
> > > > > > > Στις Τετ, 29 Αυγ 2018 στις 6:30 μ.μ., ο/η Michael Mior
> > > > > > > mailto:mm...@apache.org>> έγραψε:
> > > > > > >
> > > > > > > Just a note since we're on the topic that whatever logos we
> come
> > > > > > > up with
> > > > > > > should be sure to have TM clearly indicated.
> > > > > > >
> > > > > > > --
> > > > > > > Michael Mior
> > > > > > > mm...@apache.org 
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Le mer. 29 août 2018 à 03:13, Julian Hyde
> > > > > > > mailto:jhyde.apa...@gmail.com>> a
> > écrit :
> > > > > > >
> > > > > > > > Yes indeed!
> > > > > > > >
> > > > > > > > If someone feels inspired to produce a logo, here’s my
> > > > > > > suggestion of a
> > > > > > > > theme/image: a spider, specifically a Barn Spider (Araneus
> > > > > > > Cavaticus)[1].
> > > > > > > > It was the origin of the name “avatica”, connects and spins
> > > > > > > webs, and the
> > > > > > > > eponymous individual in Charlotte’s Web had rather
> exceptional
> > > > > > > > communication skills.
> > > > > > > >
> > > > > > > > Julian
> > > > > > > >
> > > > > > > > [1] https://en.m.wikipedia.org/wiki/Barn_spider
> > > > > > > >
> > > > > > > > > On Aug 28, 2018, at 9:49 PM, Francis Chuang
> > > > > > > mailto:francischu...@apache.org>>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > The designs I have seen so far look really good! Would it
> > > > > > > also make
> > > > > > > > sense to design a variant for Avatica as well? This is what
> > > > > > > the current
> > > > > > > > Avatica logo looks like:
> >

Re: Linq expressions to RexNode

[Hongze Zhang]

Although requirement of such functionality is not usual but there is actually a 
method 
"org.apache.calcite.prepare.CalcitePrepareImpl.EmptyScalarTranslator.toRex(Expression
 expression)" which intended to do translation from linq4j expressions to rex 
nodes. But since the method is currently supposed to be serving Calcite 
internal calls only I don't think everything is there.

Hongze

> On Jun 5, 2019, at 05:15, Haisheng Yuan  wrote:
> 
> Hi Khai,
> 
> I don't think there is such kind of utility function in Calcite.
> 
> - Haisheng 
> Yuan--
> 发件人：Khai Tran
> 日　期：2019年06月05日 01:28:21
> 收件人：dev@calcite.apache.org (dev@calcite.apache.org)
> 主　题：Linq expressions to RexNode
> 
> Just a bit strange, but just wonder if in calcite code base, is there any 
> util function to convert linq expression back to RexNode?
> 

Re: Giving the Calcite logo some love

[Albert]

I will vote "logo-alt1-v5.svg
"
, looks nice.

On Wed, Jun 5, 2019 at 6:11 AM Stamatis Zampetakis 
wrote:

> I created a branch to gather all alternative logos [1].
>
> Among the two aforementioned proposals, I added a few more variants (check
> logo-alt* under the img directory).
> Have a look and let me know what you think. I'm open to any ideas and
> suggestions.
>
> [1] https://github.com/zabetak/calcite/tree/calcite-logo/site/img
>
> On Tue, Jun 4, 2019 at 7:25 PM Julian Hyde  wrote:
>
> > I prefer both over the current logo. (And I made the current logo.)
> >
> > Let's keep the discussion going, and get to a new logo.
> >
> > On Tue, Jun 4, 2019 at 9:42 AM Ivan Grgurina 
> wrote:
> > >
> > > I prefer the one Daniel sent. It looks cleaner, but maybe the "periodic
> > table" logo can be made better by simplifying it, the shadow on the
> letter
> > C is... unusual.
> > >
> > > 
> > > From: Stamatis Zampetakis 
> > > Sent: Tuesday, June 4, 2019 6:32 PM
> > > To: dev@calcite.apache.org; humbed...@apache.org
> > > Subject: Re: Giving the Calcite logo some love
> > >
> > > Thanks for digging this out Daniel!
> > >
> > > At this point we have two candidates:
> > > http://humbedooh.com/calcite-proposed.svg
> > > https://svgshare.com/s/86r
> > >
> > > Do we like any of above more than our current logo (the way they are or
> > > with slight modifications) ?
> > >
> > >
> > >
> > >
> > > On Mon, Jun 3, 2019 at 3:15 AM Yuzhao Chen 
> wrote:
> > >
> > > > Oh, I see a big hammer, thanks Daniel !
> > > >
> > > > Best,
> > > > Danny Chan
> > > > 在 2019年6月3日 +0800 AM6:21，Daniel Gruno ，写道：
> > > > > Found it!
> > > > >
> > > > > http://humbedooh.com/calcite-proposed.svg
> > > > >
> > > > > Thanks, Wayback Machine!
> > > > >
> > > > > On 19/05/2019 10.01, Stamatis Zampetakis wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > We started this discussion about a year ago and many people were
> > > > > > positive with the idea of having a new logo for Calcite.
> > > > > > We had some nice proposals at the time and maybe now somebody
> else
> > has
> > > > > > also new ideas/designs to contribute.
> > > > > >
> > > > > > @Daniel: Is there any chance that you still have the logos you
> > proposed
> > > > > > somewhere available? The old link [1] does not work anymore.
> > > > > >
> > > > > > Best,
> > > > > > Stamatis
> > > > > >
> > > > > > [1]
> > > >
> > http://www.apache.org/logos/comdev-test/res/calcite/calcite-proposed.svg
> > > > > >
> > > > > >
> > > > > > On Thu, Aug 30, 2018 at 12:02 AM Stamatis Zampetakis <
> > > > zabe...@gmail.com
> > > > > > > wrote:
> > > > > >
> > > > > > Vladimir>Could you flip rhombus so it goes right-up?
> > > > > >
> > > > > > https://svgshare.com/s/86r
> > > > > >
> > > > > > I hope this is what you meant.
> > > > > >
> > > > > > Best,
> > > > > > Stamatis
> > > > > >
> > > > > > Στις Τετ, 29 Αυγ 2018 στις 6:30 μ.μ., ο/η Michael Mior
> > > > > > mailto:mm...@apache.org>> έγραψε:
> > > > > >
> > > > > > Just a note since we're on the topic that whatever logos we come
> > > > > > up with
> > > > > > should be sure to have TM clearly indicated.
> > > > > >
> > > > > > --
> > > > > > Michael Mior
> > > > > > mm...@apache.org 
> > > > > >
> > > > > >
> > > > > >
> > > > > > Le mer. 29 août 2018 à 03:13, Julian Hyde
> > > > > > mailto:jhyde.apa...@gmail.com>> a
> écrit :
> > > > > >
> > > > > > > Yes indeed!
> > > > > > >
> > > > > > > If someone feels inspired to produce a logo, here’s my
> > > > > > suggestion of a
> > > > > > > theme/image: a spider, specifically a Barn Spider (Araneus
> > > > > > Cavaticus)[1].
> > > > > > > It was the origin of the name “avatica”, connects and spins
> > > > > > webs, and the
> > > > > > > eponymous individual in Charlotte’s Web had rather exceptional
> > > > > > > communication skills.
> > > > > > >
> > > > > > > Julian
> > > > > > >
> > > > > > > [1] https://en.m.wikipedia.org/wiki/Barn_spider
> > > > > > >
> > > > > > > > On Aug 28, 2018, at 9:49 PM, Francis Chuang
> > > > > > mailto:francischu...@apache.org>>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > The designs I have seen so far look really good! Would it
> > > > > > also make
> > > > > > > sense to design a variant for Avatica as well? This is what
> > > > > > the current
> > > > > > > Avatica logo looks like:
> > > > > > https://calcite.apache.org/avatica/img/logo.png
> > > > > > > >
> > > > > > > > Francis
> > > > > > > >
> > > > > > > > > On 29/08/2018 7:08 AM, Vladimir Sitnikov wrote:
> > > > > > > > > Stamatis>How about something like the following:
> > > > > > > > >
> > > > > > > > > There's left-to-right vs right-to-left issue, however I
> > > > > > would claim that
> > > > > > > > > the direction of improvement is right+up.
> > > > > > > > > For instance: BTC price is good when plots go to the right
> > > > > > and go
> >

[jira] [Created] (CALCITE-3111) Allow custom implementations of Correlate in RelDecorrelator

[Juhwan Kim (JIRA)]

Juhwan Kim created CALCITE-3111:
---

 Summary: Allow custom implementations of Correlate in 
RelDecorrelator
 Key: CALCITE-3111
 URL: https://issues.apache.org/jira/browse/CALCITE-3111
 Project: Calcite
  Issue Type: Improvement
  Components: core
Reporter: Juhwan Kim
Assignee: Juhwan Kim


Currently, RelDecorrelator code only works for LogicalCorrelate. Decorrelating 
through Calcite would become much more flexible if it allows using custom 
implementations of Correlate. This would require refactoring all logical rels 
used in RelDecorrelator to the abstract ones(e.g LogicalCorrelate -> Correlate).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSSION] Extension of Metadata Query

[Stamatis Zampetakis]

Thanks for bringing this up Danny.

I guess the discussion came up due to CALCITE-2885 [1]. Looking back into
it, I am not sure that the intention there is to make the RelMetadataQuery
pluggable. We could possibly solve the performance problem without
extending the RelMetadataQuery. We have to look again into this case.

For more details regarding the existence of the two methods in
RelMetadataProvider have a look in CALCITE-604 [2]. More general for the
design of RelMetadataProvider you may find useful the description in [3].

Best,
Stamatis

[1] https://issues.apache.org/jira/browse/CALCITE-2885
[2] https://issues.apache.org/jira/browse/CALCITE-604
[3]
https://web.archive.org/web/20140624040836/www.hydromatic.net/wiki/RelationalExpressionMetadata/

On Tue, Jun 4, 2019 at 7:48 PM Julian Hyde  wrote:

> > 1. Why we have 2 methods in RelMetadataProvider?
>
> The metadata system is complicated. We need to allow multiple handlers
> for any given call. So, making a metadata call involves multiple
> dispatch [1] based on the metadata method being called, the type of
> RelNode, and the handlers that are registered. Also it needs to cache
> results, and detect cycles. And the dispatch needs to be efficient, so
> we generate janino code to do the dispatch, and re-generate when new
> handlers or sub-classes of RelNode are added.
>
> I forget details, the two methods are basically required to allow us
> to generate code to do the dispatch.
>
> > 2. We should make the RelMetadataQuery in RelOptCluster pluggable.
>
> I disagree. RelMetadataQuery is not an extension point. Its sole
> purpose is to to keep state (the cache and cycle-checking).
> RelMetadataProvider is the extension point. (By analogy, if you are
> un-parsing an AST, you let each AST sub-class handle unparsing itself,
> but the unparsed text goes to a simple StringBuilder. RelMetadataQuery
> is in the role of the StringBuilder. In a complex system, it is nice
> to keep some of the components simple, or at least keep them to
> prescribed roles.)
>
> Julian
>
> [1] https://en.wikipedia.org/wiki/Multiple_dispatch
>
> On Sun, Jun 2, 2019 at 11:19 PM Yuzhao Chen  wrote:
> >
> > Currently we provide answer to metadata query through
> RelMetadataProvider [1], there are some sub-classes of it:
> >
> > RelMetadataProvider
> > |
> > |- VolcanoRelMetadataProvider
> > |- ChainedRelMetadataProvider/DefaultRelMetadataProvider
> > |- HepRelMetadataProvider
> > |- CachingRelMetadataProvider
> > |- ReflectiveRelMetadataProvider
> > |- JaninoRelMetadataProvider
> >
> > The RelMetadataProvider has two methods: #apply and #handlers, the
> #apply method seems a programming interface and there is a demo code how we
> can use it:
> >
> > RelMetadataProvider provider;
> > LogicalFilter filter;
> > RexNode predicate;
> > Function function =
> > provider.apply(LogicalFilter.class, Selectivity.class};
> > Selectivity selectivity = function.apply(filter);
> > Double d = selectivity.selectivity(predicate);
> >
> > But let's see our RelOptCluster's member variables[2], there are
> MetadataFactory and RelMetadataQuery which all can be used to query the
> metadata, for MetadataFactory, there is a default impl named
> MetadataFactoryImpl which will invoke RelMetadataProvider#apply internally,
> for RelMetadataQuery, it will invoke RelMetadataProvider#handlers (finally
> composed and codeden by JaninoRelMetadataProvider).
> >
> > In our planning phrase, we can invoke RelOptRuleCall#getMetadataQuery to
> get the MQ and query the metadata.
> >
> > For extension of metadata handlers, we can set our customized
> RelMetadataProvider in RelOptCluster[3]. But for RelMetadataQuery, we have
> no way to extend it now, because the RelOptCluster always has a singleton
> instance [4] which is only the default implementation.
> >
> >
> > My question is as follows:
> >
> > 1. Why we have 2 methods in RelMetadataProvider, and why we need the
> MetadataFactory and RelMetadataProvider#apply ? It seems that it's function
> is already been overriden by RelMetadataQuery(The difference is that
> MetadataFactory use Reflection and RelMetadataQuery use gened bytes code).
> > 2. We should make the RelMetadataQuery in RelOptCluster pluggable.
> >
> >
> > [1]
> https://github.com/apache/calcite/blob/b0e83c469ff57257c1ea621ff943ca76f626a9b7/core/src/main/java/org/apache/calcite/rel/metadata/RelMetadataProvider.java#L38
> > [2]
> https://github.com/apache/calcite/blob/b0e83c469ff57257c1ea621ff943ca76f626a9b7/core/src/main/java/org/apache/calcite/plan/RelOptCluster.java#L49
> > [3]
> https://github.com/apache/calcite/blob/b0e83c469ff57257c1ea621ff943ca76f626a9b7/core/src/main/java/org/apache/calcite/plan/RelOptCluster.java#L135
> > [4]
> https://github.com/apache/calcite/blob/b0e83c469ff57257c1ea621ff943ca76f626a9b7/core/src/main/java/org/apache/calcite/plan/RelOptCluster.java#L151
> >
> >
> >
> > Best,
> > Danny Chan
>

Re: Giving the Calcite logo some love

[Stamatis Zampetakis]

I created a branch to gather all alternative logos [1].

Among the two aforementioned proposals, I added a few more variants (check
logo-alt* under the img directory).
Have a look and let me know what you think. I'm open to any ideas and
suggestions.

[1] https://github.com/zabetak/calcite/tree/calcite-logo/site/img

On Tue, Jun 4, 2019 at 7:25 PM Julian Hyde  wrote:

> I prefer both over the current logo. (And I made the current logo.)
>
> Let's keep the discussion going, and get to a new logo.
>
> On Tue, Jun 4, 2019 at 9:42 AM Ivan Grgurina  wrote:
> >
> > I prefer the one Daniel sent. It looks cleaner, but maybe the "periodic
> table" logo can be made better by simplifying it, the shadow on the letter
> C is... unusual.
> >
> > 
> > From: Stamatis Zampetakis 
> > Sent: Tuesday, June 4, 2019 6:32 PM
> > To: dev@calcite.apache.org; humbed...@apache.org
> > Subject: Re: Giving the Calcite logo some love
> >
> > Thanks for digging this out Daniel!
> >
> > At this point we have two candidates:
> > http://humbedooh.com/calcite-proposed.svg
> > https://svgshare.com/s/86r
> >
> > Do we like any of above more than our current logo (the way they are or
> > with slight modifications) ?
> >
> >
> >
> >
> > On Mon, Jun 3, 2019 at 3:15 AM Yuzhao Chen  wrote:
> >
> > > Oh, I see a big hammer, thanks Daniel !
> > >
> > > Best,
> > > Danny Chan
> > > 在 2019年6月3日 +0800 AM6:21，Daniel Gruno ，写道：
> > > > Found it!
> > > >
> > > > http://humbedooh.com/calcite-proposed.svg
> > > >
> > > > Thanks, Wayback Machine!
> > > >
> > > > On 19/05/2019 10.01, Stamatis Zampetakis wrote:
> > > > > Hi all,
> > > > >
> > > > > We started this discussion about a year ago and many people were
> > > > > positive with the idea of having a new logo for Calcite.
> > > > > We had some nice proposals at the time and maybe now somebody else
> has
> > > > > also new ideas/designs to contribute.
> > > > >
> > > > > @Daniel: Is there any chance that you still have the logos you
> proposed
> > > > > somewhere available? The old link [1] does not work anymore.
> > > > >
> > > > > Best,
> > > > > Stamatis
> > > > >
> > > > > [1]
> > >
> http://www.apache.org/logos/comdev-test/res/calcite/calcite-proposed.svg
> > > > >
> > > > >
> > > > > On Thu, Aug 30, 2018 at 12:02 AM Stamatis Zampetakis <
> > > zabe...@gmail.com
> > > > > > wrote:
> > > > >
> > > > > Vladimir>Could you flip rhombus so it goes right-up?
> > > > >
> > > > > https://svgshare.com/s/86r
> > > > >
> > > > > I hope this is what you meant.
> > > > >
> > > > > Best,
> > > > > Stamatis
> > > > >
> > > > > Στις Τετ, 29 Αυγ 2018 στις 6:30 μ.μ., ο/η Michael Mior
> > > > > mailto:mm...@apache.org>> έγραψε:
> > > > >
> > > > > Just a note since we're on the topic that whatever logos we come
> > > > > up with
> > > > > should be sure to have TM clearly indicated.
> > > > >
> > > > > --
> > > > > Michael Mior
> > > > > mm...@apache.org 
> > > > >
> > > > >
> > > > >
> > > > > Le mer. 29 août 2018 à 03:13, Julian Hyde
> > > > > mailto:jhyde.apa...@gmail.com>> a écrit :
> > > > >
> > > > > > Yes indeed!
> > > > > >
> > > > > > If someone feels inspired to produce a logo, here’s my
> > > > > suggestion of a
> > > > > > theme/image: a spider, specifically a Barn Spider (Araneus
> > > > > Cavaticus)[1].
> > > > > > It was the origin of the name “avatica”, connects and spins
> > > > > webs, and the
> > > > > > eponymous individual in Charlotte’s Web had rather exceptional
> > > > > > communication skills.
> > > > > >
> > > > > > Julian
> > > > > >
> > > > > > [1] https://en.m.wikipedia.org/wiki/Barn_spider
> > > > > >
> > > > > > > On Aug 28, 2018, at 9:49 PM, Francis Chuang
> > > > > mailto:francischu...@apache.org>>
> > > > > > wrote:
> > > > > > >
> > > > > > > The designs I have seen so far look really good! Would it
> > > > > also make
> > > > > > sense to design a variant for Avatica as well? This is what
> > > > > the current
> > > > > > Avatica logo looks like:
> > > > > https://calcite.apache.org/avatica/img/logo.png
> > > > > > >
> > > > > > > Francis
> > > > > > >
> > > > > > > > On 29/08/2018 7:08 AM, Vladimir Sitnikov wrote:
> > > > > > > > Stamatis>How about something like the following:
> > > > > > > >
> > > > > > > > There's left-to-right vs right-to-left issue, however I
> > > > > would claim that
> > > > > > > > the direction of improvement is right+up.
> > > > > > > > For instance: BTC price is good when plots go to the right
> > > > > and go
> > > > > > upward.
> > > > > > > >
> > > > > > > > https://svgur.com/s/83y is slanted backward.
> > > > > > > > That creates perception of "Calcite holding back the
> > > > > progress" or
> > > > > > "Apache
> > > > > > > > pushing C away" or something like that.
> > > > > > > > Could you flip rhombus so it goes right-up?
> > > > > > > >
> > > > > > > >
> > > > > > > > Vladimir
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>

Re: Linq expressions to RexNode

[Haisheng Yuan]

Hi Khai,

I don't think there is such kind of utility function in Calcite.

- Haisheng 
Yuan--
发件人：Khai Tran
日　期：2019年06月05日 01:28:21
收件人：dev@calcite.apache.org (dev@calcite.apache.org)
主　题：Linq expressions to RexNode

Just a bit strange, but just wonder if in calcite code base, is there any util 
function to convert linq expression back to RexNode?

[jira] [Created] (CALCITE-3110) Enable parallel execution of parameterized test

[Haisheng Yuan (JIRA)]

Haisheng Yuan created CALCITE-3110:
--

 Summary: Enable parallel execution of parameterized test
 Key: CALCITE-3110
 URL: https://issues.apache.org/jira/browse/CALCITE-3110
 Project: Calcite
  Issue Type: Improvement
  Components: core
Reporter: Haisheng Yuan


The patch of CALCITE-2812 introduced a parameterized test, but parameterized 
JUnit test hangs when running in parallel mode due to [1]. So we annotated the 
test case as not thread safe avoid parallel execution.

Enable parallel execution of EnumerableRepeatUnionHierarchyTest when issue [1] 
is fixed.

See discussion at [2].

[1] https://issues.apache.org/jira/browse/SUREFIRE-1430
[2] 
https://lists.apache.org/thread.html/d9205b98873d60102ed53e46ffde2fa63632eeca948ee1c9ab810827@%3Cdev.calcite.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Re: Re: Re: Re: Re: [DISCUSS] Parallel parameterized unit tests

[Haisheng Yuan]

Looks like we already used https://github.com/stephenc/jcip-annotations in 
pom.xml, I will keep pom.xml untouched.

- Haisheng 
Yuan--
发件人：Julian Hyde
日　期：2019年06月05日 02:39:43
收件人：dev@calcite.apache.org
主　题：Re: Re: Re: Re: Re: [DISCUSS] Parallel parameterized unit tests

Be sure to use a version of jcip-annotations that has an acceptable license.

https://github.com/stephenc/jcip-annotations seems suitable; the
original, https://search.maven.org/artifact/net.jcip/jcip-annotations/1.0/jar,
does not.

On Tue, Jun 4, 2019 at 11:36 AM Haisheng Yuan  wrote:
>
> Yes, I will do it.
>
> - Haisheng Yuan
>
> --
> 发件人：Julian Hyde
> 日　期：2019年06月05日 02:23:24
> 收件人：dev@calcite.apache.org
> 抄　送：Stamatis Zampetakis
> 主　题：Re: Re: Re: Re: [DISCUSS] Parallel parameterized unit tests
>
> I think we have quorum. Can someone please commit this?
>
> Do we need to log a JIRA case to remind us to remove this workaround in 
> future?
>
> On Tue, Jun 4, 2019 at 11:10 AM Ruben Q L  wrote:
> >
> > I also confirm: @net.jcip.annotations.NotThreadSafe works for me too. I
> > agree, this seems the cleanest solution.
> >
> > Le mar. 4 juin 2019 à 19:54, Haisheng Yuan  a
> > écrit :
> >
> > > I tried the annotaion @net.jcip.annotations.NotThreadSafe that Stamatis
> > > suggested, it works for me. This might be the least change for us with
> > > little impact. So what is the plan? Can we incorporate this change before
> > > 1.20 release? Currently I have to explicitly disable/annotate the
> > > parameterized test to be able to pass the whole test suite.
> > >
> > > - Haisheng
> > > Yuan--
> > > 发件人：Stamatis Zampetakis
> > > 日 期：2019年06月04日 07:13:00
> > > 收件人：
> > > 主 题：Re: Re: Re: [DISCUSS] Parallel parameterized unit tests
> > >
> > > It seems that we can disable parallel execution selectively for certain
> > > test classes by annotating them with @net.jcip.annotations.NotThreadSafe.
> > > Maybe this can do the trick for parameterized tests while we wait for the
> > > permanent fix.
> > >
> > >
> > >
> > > On Mon, Jun 3, 2019 at 10:57 PM Haisheng Yuan 
> > > wrote:
> > >
> > > > I am inclining to change threadCount to 4, then see how it works in CI,
> > > at
> > > > least it solves the problem on my laptop.
> > > >
> > > > If the problem still shows up in CI, or it solves the problem but makes
> > > > the CI test much slower, I would rather tear up the parameterized test
> > > case.
> > > >
> > > > - Haisheng
> > > > Yuan--
> > > > 发件人：Haisheng Yuan
> > > > 日 期：2019年06月04日 02:28:40
> > > > 收件人：Julian Hyde; dev
> > > > 主 题：Re: Re: [DISCUSS] Parallel parameterized unit tests
> > > >
> > > > > Am I correct that this problem only shows up on machines with 1 CPU?
> > > >
> > > > No, my laptop has 2 CPUs, it still shows up.
> > > >
> > > > Thanks ~
> > > > Haisheng Yuan
> > > > --
> > > > 发件人：Julian Hyde
> > > > 日 期：2019年06月04日 01:56:56
> > > > 收件人：dev
> > > > 主 题：Re: [DISCUSS] Parallel parameterized unit tests
> > > >
> > > > Am I correct that this problem only shows up on machines with 1 CPU?
> > > >
> > > > Such machines exist in virtualization environments (e.g. Travis) but the
> > > > machines we use for everyday development have multiple CPUs. So, could 
> > > > we
> > > > add a parameter so that we can turn off parallel tests only in CI?
> > > >
> > > > My goal is to avoid a drop in speed and test coverage while we wait for 
> > > > a
> > > > workaround.
> > > >
> > > > Julian
> > > >
> > > >
> > > > > On Jun 3, 2019, at 2:29 AM, Ruben Q L  wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > thanks for opening this discussion, Stamatis.
> > > > >
> > > > > I think there might be a third way. We could try to check if tuning
> > > > > surefire configuration solves the problem.
> > > > > Right now on calcite's pom.xml file we have:
> > > > >
> > > > > both
> > > > > 1
> > > > > true
> > > > >
> > > > > Thanks to Haisheng's tests, we know that changing threadCount from 1
> > > > > to 4 seems to be a workaround, at least on some environments.
> > > > > Moreover, I have noticed that parallel "both" value is deprecated [1],
> > > > > so maybe we should try some other values and see if that solves the
> > > > > problem without disabling parallelization.
> > > > >
> > > > > [1]
> > > >
> > > http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#parallel
> > > > >
> > > > >
> > > > >
> > > > > Le lun. 3 juin 2019 à 11:00, Stamatis Zampetakis  a
> > > > > écrit :
> > > > >
> > > > >> Hi all,
> > > > >>
> > > > >> The past few days we had observed various problems on Jenkins (and
> > > > >> elsewhere) which made our test suite hang.
> > > > >>
> > > > >> Haisheng and Ruben found out that the problem was the parallel
>

Re: Re: Re: Re: Re: [DISCUSS] Parallel parameterized unit tests

[Julian Hyde]

Be sure to use a version of jcip-annotations that has an acceptable license.

https://github.com/stephenc/jcip-annotations seems suitable; the
original, https://search.maven.org/artifact/net.jcip/jcip-annotations/1.0/jar,
does not.

On Tue, Jun 4, 2019 at 11:36 AM Haisheng Yuan  wrote:
>
> Yes, I will do it.
>
> - Haisheng Yuan
>
> --
> 发件人：Julian Hyde
> 日　期：2019年06月05日 02:23:24
> 收件人：dev@calcite.apache.org
> 抄　送：Stamatis Zampetakis
> 主　题：Re: Re: Re: Re: [DISCUSS] Parallel parameterized unit tests
>
> I think we have quorum. Can someone please commit this?
>
> Do we need to log a JIRA case to remind us to remove this workaround in 
> future?
>
> On Tue, Jun 4, 2019 at 11:10 AM Ruben Q L  wrote:
> >
> > I also confirm: @net.jcip.annotations.NotThreadSafe works for me too. I
> > agree, this seems the cleanest solution.
> >
> > Le mar. 4 juin 2019 à 19:54, Haisheng Yuan  a
> > écrit :
> >
> > > I tried the annotaion @net.jcip.annotations.NotThreadSafe that Stamatis
> > > suggested, it works for me. This might be the least change for us with
> > > little impact. So what is the plan? Can we incorporate this change before
> > > 1.20 release? Currently I have to explicitly disable/annotate the
> > > parameterized test to be able to pass the whole test suite.
> > >
> > > - Haisheng
> > > Yuan--
> > > 发件人：Stamatis Zampetakis
> > > 日 期：2019年06月04日 07:13:00
> > > 收件人：
> > > 主 题：Re: Re: Re: [DISCUSS] Parallel parameterized unit tests
> > >
> > > It seems that we can disable parallel execution selectively for certain
> > > test classes by annotating them with @net.jcip.annotations.NotThreadSafe.
> > > Maybe this can do the trick for parameterized tests while we wait for the
> > > permanent fix.
> > >
> > >
> > >
> > > On Mon, Jun 3, 2019 at 10:57 PM Haisheng Yuan 
> > > wrote:
> > >
> > > > I am inclining to change threadCount to 4, then see how it works in CI,
> > > at
> > > > least it solves the problem on my laptop.
> > > >
> > > > If the problem still shows up in CI, or it solves the problem but makes
> > > > the CI test much slower, I would rather tear up the parameterized test
> > > case.
> > > >
> > > > - Haisheng
> > > > Yuan--
> > > > 发件人：Haisheng Yuan
> > > > 日 期：2019年06月04日 02:28:40
> > > > 收件人：Julian Hyde; dev
> > > > 主 题：Re: Re: [DISCUSS] Parallel parameterized unit tests
> > > >
> > > > > Am I correct that this problem only shows up on machines with 1 CPU?
> > > >
> > > > No, my laptop has 2 CPUs, it still shows up.
> > > >
> > > > Thanks ~
> > > > Haisheng Yuan
> > > > --
> > > > 发件人：Julian Hyde
> > > > 日 期：2019年06月04日 01:56:56
> > > > 收件人：dev
> > > > 主 题：Re: [DISCUSS] Parallel parameterized unit tests
> > > >
> > > > Am I correct that this problem only shows up on machines with 1 CPU?
> > > >
> > > > Such machines exist in virtualization environments (e.g. Travis) but the
> > > > machines we use for everyday development have multiple CPUs. So, could 
> > > > we
> > > > add a parameter so that we can turn off parallel tests only in CI?
> > > >
> > > > My goal is to avoid a drop in speed and test coverage while we wait for 
> > > > a
> > > > workaround.
> > > >
> > > > Julian
> > > >
> > > >
> > > > > On Jun 3, 2019, at 2:29 AM, Ruben Q L  wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > thanks for opening this discussion, Stamatis.
> > > > >
> > > > > I think there might be a third way. We could try to check if tuning
> > > > > surefire configuration solves the problem.
> > > > > Right now on calcite's pom.xml file we have:
> > > > >
> > > > > both
> > > > > 1
> > > > > true
> > > > >
> > > > > Thanks to Haisheng's tests, we know that changing threadCount from 1
> > > > > to 4 seems to be a workaround, at least on some environments.
> > > > > Moreover, I have noticed that parallel "both" value is deprecated [1],
> > > > > so maybe we should try some other values and see if that solves the
> > > > > problem without disabling parallelization.
> > > > >
> > > > > [1]
> > > >
> > > http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#parallel
> > > > >
> > > > >
> > > > >
> > > > > Le lun. 3 juin 2019 à 11:00, Stamatis Zampetakis  a
> > > > > écrit :
> > > > >
> > > > >> Hi all,
> > > > >>
> > > > >> The past few days we had observed various problems on Jenkins (and
> > > > >> elsewhere) which made our test suite hang.
> > > > >>
> > > > >> Haisheng and Ruben found out that the problem was the parallel
> > > > execution of
> > > > >> parameterized tests [1]. I see two ways to unblock this situation and
> > > > avoid
> > > > >> such problems in the future:
> > > > >>
> > > > >> * disable parallel execution of unit tests;
> > > > >> * forbid the use of @RunWith(Parameterized.class) annotation;
> > > > >>
> > > > >> at least till [1] is 

Re: Re: Re: Re: Re: [DISCUSS] Parallel parameterized unit tests

[Haisheng Yuan]

Yes, I will do it.

- Haisheng 
Yuan--
发件人：Julian Hyde
日　期：2019年06月05日 02:23:24
收件人：dev@calcite.apache.org
抄　送：Stamatis Zampetakis
主　题：Re: Re: Re: Re: [DISCUSS] Parallel parameterized unit tests

I think we have quorum. Can someone please commit this?

Do we need to log a JIRA case to remind us to remove this workaround in future?

On Tue, Jun 4, 2019 at 11:10 AM Ruben Q L  wrote:
>
> I also confirm: @net.jcip.annotations.NotThreadSafe works for me too. I
> agree, this seems the cleanest solution.
>
> Le mar. 4 juin 2019 à 19:54, Haisheng Yuan  a
> écrit :
>
> > I tried the annotaion @net.jcip.annotations.NotThreadSafe that Stamatis
> > suggested, it works for me. This might be the least change for us with
> > little impact. So what is the plan? Can we incorporate this change before
> > 1.20 release? Currently I have to explicitly disable/annotate the
> > parameterized test to be able to pass the whole test suite.
> >
> > - Haisheng
> > Yuan--
> > 发件人：Stamatis Zampetakis
> > 日 期：2019年06月04日 07:13:00
> > 收件人：
> > 主 题：Re: Re: Re: [DISCUSS] Parallel parameterized unit tests
> >
> > It seems that we can disable parallel execution selectively for certain
> > test classes by annotating them with @net.jcip.annotations.NotThreadSafe.
> > Maybe this can do the trick for parameterized tests while we wait for the
> > permanent fix.
> >
> >
> >
> > On Mon, Jun 3, 2019 at 10:57 PM Haisheng Yuan 
> > wrote:
> >
> > > I am inclining to change threadCount to 4, then see how it works in CI,
> > at
> > > least it solves the problem on my laptop.
> > >
> > > If the problem still shows up in CI, or it solves the problem but makes
> > > the CI test much slower, I would rather tear up the parameterized test
> > case.
> > >
> > > - Haisheng
> > > Yuan--
> > > 发件人：Haisheng Yuan
> > > 日 期：2019年06月04日 02:28:40
> > > 收件人：Julian Hyde; dev
> > > 主 题：Re: Re: [DISCUSS] Parallel parameterized unit tests
> > >
> > > > Am I correct that this problem only shows up on machines with 1 CPU?
> > >
> > > No, my laptop has 2 CPUs, it still shows up.
> > >
> > > Thanks ~
> > > Haisheng Yuan
> > > --
> > > 发件人：Julian Hyde
> > > 日 期：2019年06月04日 01:56:56
> > > 收件人：dev
> > > 主 题：Re: [DISCUSS] Parallel parameterized unit tests
> > >
> > > Am I correct that this problem only shows up on machines with 1 CPU?
> > >
> > > Such machines exist in virtualization environments (e.g. Travis) but the
> > > machines we use for everyday development have multiple CPUs. So, could we
> > > add a parameter so that we can turn off parallel tests only in CI?
> > >
> > > My goal is to avoid a drop in speed and test coverage while we wait for a
> > > workaround.
> > >
> > > Julian
> > >
> > >
> > > > On Jun 3, 2019, at 2:29 AM, Ruben Q L  wrote:
> > > >
> > > > Hi all,
> > > >
> > > > thanks for opening this discussion, Stamatis.
> > > >
> > > > I think there might be a third way. We could try to check if tuning
> > > > surefire configuration solves the problem.
> > > > Right now on calcite's pom.xml file we have:
> > > >
> > > > both
> > > > 1
> > > > true
> > > >
> > > > Thanks to Haisheng's tests, we know that changing threadCount from 1
> > > > to 4 seems to be a workaround, at least on some environments.
> > > > Moreover, I have noticed that parallel "both" value is deprecated [1],
> > > > so maybe we should try some other values and see if that solves the
> > > > problem without disabling parallelization.
> > > >
> > > > [1]
> > >
> > http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#parallel
> > > >
> > > >
> > > >
> > > > Le lun. 3 juin 2019 à 11:00, Stamatis Zampetakis  a
> > > > écrit :
> > > >
> > > >> Hi all,
> > > >>
> > > >> The past few days we had observed various problems on Jenkins (and
> > > >> elsewhere) which made our test suite hang.
> > > >>
> > > >> Haisheng and Ruben found out that the problem was the parallel
> > > execution of
> > > >> parameterized tests [1]. I see two ways to unblock this situation and
> > > avoid
> > > >> such problems in the future:
> > > >>
> > > >> * disable parallel execution of unit tests;
> > > >> * forbid the use of @RunWith(Parameterized.class) annotation;
> > > >>
> > > >> at least till [1] is resolved.
> > > >>
> > > >> What do you think?
> > > >>
> > > >> [1] https://issues.apache.org/jira/browse/SUREFIRE-1430
> > > >>
> > >
> > >
> >
> >

Re: Re: Re: Re: [DISCUSS] Parallel parameterized unit tests

[Julian Hyde]

I think we have quorum. Can someone please commit this?

Do we need to log a JIRA case to remind us to remove this workaround in future?

On Tue, Jun 4, 2019 at 11:10 AM Ruben Q L  wrote:
>
> I also confirm: @net.jcip.annotations.NotThreadSafe works for me too. I
> agree, this seems the cleanest solution.
>
> Le mar. 4 juin 2019 à 19:54, Haisheng Yuan  a
> écrit :
>
> > I tried the annotaion @net.jcip.annotations.NotThreadSafe that Stamatis
> > suggested, it works for me. This might be the least change for us with
> > little impact. So what is the plan? Can we incorporate this change before
> > 1.20 release? Currently I have to explicitly disable/annotate the
> > parameterized test to be able to pass the whole test suite.
> >
> > - Haisheng
> > Yuan--
> > 发件人：Stamatis Zampetakis
> > 日 期：2019年06月04日 07:13:00
> > 收件人：
> > 主 题：Re: Re: Re: [DISCUSS] Parallel parameterized unit tests
> >
> > It seems that we can disable parallel execution selectively for certain
> > test classes by annotating them with @net.jcip.annotations.NotThreadSafe.
> > Maybe this can do the trick for parameterized tests while we wait for the
> > permanent fix.
> >
> >
> >
> > On Mon, Jun 3, 2019 at 10:57 PM Haisheng Yuan 
> > wrote:
> >
> > > I am inclining to change threadCount to 4, then see how it works in CI,
> > at
> > > least it solves the problem on my laptop.
> > >
> > > If the problem still shows up in CI, or it solves the problem but makes
> > > the CI test much slower, I would rather tear up the parameterized test
> > case.
> > >
> > > - Haisheng
> > > Yuan--
> > > 发件人：Haisheng Yuan
> > > 日 期：2019年06月04日 02:28:40
> > > 收件人：Julian Hyde; dev
> > > 主 题：Re: Re: [DISCUSS] Parallel parameterized unit tests
> > >
> > > > Am I correct that this problem only shows up on machines with 1 CPU?
> > >
> > > No, my laptop has 2 CPUs, it still shows up.
> > >
> > > Thanks ~
> > > Haisheng Yuan
> > > --
> > > 发件人：Julian Hyde
> > > 日 期：2019年06月04日 01:56:56
> > > 收件人：dev
> > > 主 题：Re: [DISCUSS] Parallel parameterized unit tests
> > >
> > > Am I correct that this problem only shows up on machines with 1 CPU?
> > >
> > > Such machines exist in virtualization environments (e.g. Travis) but the
> > > machines we use for everyday development have multiple CPUs. So, could we
> > > add a parameter so that we can turn off parallel tests only in CI?
> > >
> > > My goal is to avoid a drop in speed and test coverage while we wait for a
> > > workaround.
> > >
> > > Julian
> > >
> > >
> > > > On Jun 3, 2019, at 2:29 AM, Ruben Q L  wrote:
> > > >
> > > > Hi all,
> > > >
> > > > thanks for opening this discussion, Stamatis.
> > > >
> > > > I think there might be a third way. We could try to check if tuning
> > > > surefire configuration solves the problem.
> > > > Right now on calcite's pom.xml file we have:
> > > >
> > > > both
> > > > 1
> > > > true
> > > >
> > > > Thanks to Haisheng's tests, we know that changing threadCount from 1
> > > > to 4 seems to be a workaround, at least on some environments.
> > > > Moreover, I have noticed that parallel "both" value is deprecated [1],
> > > > so maybe we should try some other values and see if that solves the
> > > > problem without disabling parallelization.
> > > >
> > > > [1]
> > >
> > http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#parallel
> > > >
> > > >
> > > >
> > > > Le lun. 3 juin 2019 à 11:00, Stamatis Zampetakis  a
> > > > écrit :
> > > >
> > > >> Hi all,
> > > >>
> > > >> The past few days we had observed various problems on Jenkins (and
> > > >> elsewhere) which made our test suite hang.
> > > >>
> > > >> Haisheng and Ruben found out that the problem was the parallel
> > > execution of
> > > >> parameterized tests [1]. I see two ways to unblock this situation and
> > > avoid
> > > >> such problems in the future:
> > > >>
> > > >> * disable parallel execution of unit tests;
> > > >> * forbid the use of @RunWith(Parameterized.class) annotation;
> > > >>
> > > >> at least till [1] is resolved.
> > > >>
> > > >> What do you think?
> > > >>
> > > >> [1] https://issues.apache.org/jira/browse/SUREFIRE-1430
> > > >>
> > >
> > >
> >
> >

Re: Re: Re: Re: [DISCUSS] Parallel parameterized unit tests

[Ruben Q L]

I also confirm: @net.jcip.annotations.NotThreadSafe works for me too. I
agree, this seems the cleanest solution.

Le mar. 4 juin 2019 à 19:54, Haisheng Yuan  a
écrit :

> I tried the annotaion @net.jcip.annotations.NotThreadSafe that Stamatis
> suggested, it works for me. This might be the least change for us with
> little impact. So what is the plan? Can we incorporate this change before
> 1.20 release? Currently I have to explicitly disable/annotate the
> parameterized test to be able to pass the whole test suite.
>
> - Haisheng
> Yuan--
> 发件人：Stamatis Zampetakis
> 日 期：2019年06月04日 07:13:00
> 收件人：
> 主 题：Re: Re: Re: [DISCUSS] Parallel parameterized unit tests
>
> It seems that we can disable parallel execution selectively for certain
> test classes by annotating them with @net.jcip.annotations.NotThreadSafe.
> Maybe this can do the trick for parameterized tests while we wait for the
> permanent fix.
>
>
>
> On Mon, Jun 3, 2019 at 10:57 PM Haisheng Yuan 
> wrote:
>
> > I am inclining to change threadCount to 4, then see how it works in CI,
> at
> > least it solves the problem on my laptop.
> >
> > If the problem still shows up in CI, or it solves the problem but makes
> > the CI test much slower, I would rather tear up the parameterized test
> case.
> >
> > - Haisheng
> > Yuan--
> > 发件人：Haisheng Yuan
> > 日 期：2019年06月04日 02:28:40
> > 收件人：Julian Hyde; dev
> > 主 题：Re: Re: [DISCUSS] Parallel parameterized unit tests
> >
> > > Am I correct that this problem only shows up on machines with 1 CPU?
> >
> > No, my laptop has 2 CPUs, it still shows up.
> >
> > Thanks ~
> > Haisheng Yuan
> > --
> > 发件人：Julian Hyde
> > 日 期：2019年06月04日 01:56:56
> > 收件人：dev
> > 主 题：Re: [DISCUSS] Parallel parameterized unit tests
> >
> > Am I correct that this problem only shows up on machines with 1 CPU?
> >
> > Such machines exist in virtualization environments (e.g. Travis) but the
> > machines we use for everyday development have multiple CPUs. So, could we
> > add a parameter so that we can turn off parallel tests only in CI?
> >
> > My goal is to avoid a drop in speed and test coverage while we wait for a
> > workaround.
> >
> > Julian
> >
> >
> > > On Jun 3, 2019, at 2:29 AM, Ruben Q L  wrote:
> > >
> > > Hi all,
> > >
> > > thanks for opening this discussion, Stamatis.
> > >
> > > I think there might be a third way. We could try to check if tuning
> > > surefire configuration solves the problem.
> > > Right now on calcite's pom.xml file we have:
> > >
> > > both
> > > 1
> > > true
> > >
> > > Thanks to Haisheng's tests, we know that changing threadCount from 1
> > > to 4 seems to be a workaround, at least on some environments.
> > > Moreover, I have noticed that parallel "both" value is deprecated [1],
> > > so maybe we should try some other values and see if that solves the
> > > problem without disabling parallelization.
> > >
> > > [1]
> >
> http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#parallel
> > >
> > >
> > >
> > > Le lun. 3 juin 2019 à 11:00, Stamatis Zampetakis  a
> > > écrit :
> > >
> > >> Hi all,
> > >>
> > >> The past few days we had observed various problems on Jenkins (and
> > >> elsewhere) which made our test suite hang.
> > >>
> > >> Haisheng and Ruben found out that the problem was the parallel
> > execution of
> > >> parameterized tests [1]. I see two ways to unblock this situation and
> > avoid
> > >> such problems in the future:
> > >>
> > >> * disable parallel execution of unit tests;
> > >> * forbid the use of @RunWith(Parameterized.class) annotation;
> > >>
> > >> at least till [1] is resolved.
> > >>
> > >> What do you think?
> > >>
> > >> [1] https://issues.apache.org/jira/browse/SUREFIRE-1430
> > >>
> >
> >
>
>

Re: Re: Re: Re: [DISCUSS] Parallel parameterized unit tests

[Haisheng Yuan]

I tried the annotaion @net.jcip.annotations.NotThreadSafe that Stamatis 
suggested, it works for me. This might be the least change for us with little 
impact. So what is the plan? Can we incorporate this change before 1.20 
release? Currently I have to explicitly disable/annotate the parameterized test 
to be able to pass the whole test suite.

- Haisheng 
Yuan--
发件人：Stamatis Zampetakis
日　期：2019年06月04日 07:13:00
收件人：
主　题：Re: Re: Re: [DISCUSS] Parallel parameterized unit tests

It seems that we can disable parallel execution selectively for certain
test classes by annotating them with @net.jcip.annotations.NotThreadSafe.
Maybe this can do the trick for parameterized tests while we wait for the
permanent fix.



On Mon, Jun 3, 2019 at 10:57 PM Haisheng Yuan 
wrote:

> I am inclining to change threadCount to 4, then see how it works in CI, at
> least it solves the problem on my laptop.
>
> If the problem still shows up in CI, or it solves the problem but makes
> the CI test much slower, I would rather tear up the parameterized test case.
>
> - Haisheng
> Yuan--
> 发件人：Haisheng Yuan
> 日 期：2019年06月04日 02:28:40
> 收件人：Julian Hyde; dev
> 主 题：Re: Re: [DISCUSS] Parallel parameterized unit tests
>
> > Am I correct that this problem only shows up on machines with 1 CPU?
>
> No, my laptop has 2 CPUs, it still shows up.
>
> Thanks ~
> Haisheng Yuan
> --
> 发件人：Julian Hyde
> 日 期：2019年06月04日 01:56:56
> 收件人：dev
> 主 题：Re: [DISCUSS] Parallel parameterized unit tests
>
> Am I correct that this problem only shows up on machines with 1 CPU?
>
> Such machines exist in virtualization environments (e.g. Travis) but the
> machines we use for everyday development have multiple CPUs. So, could we
> add a parameter so that we can turn off parallel tests only in CI?
>
> My goal is to avoid a drop in speed and test coverage while we wait for a
> workaround.
>
> Julian
>
>
> > On Jun 3, 2019, at 2:29 AM, Ruben Q L  wrote:
> >
> > Hi all,
> >
> > thanks for opening this discussion, Stamatis.
> >
> > I think there might be a third way. We could try to check if tuning
> > surefire configuration solves the problem.
> > Right now on calcite's pom.xml file we have:
> >
> > both
> > 1
> > true
> >
> > Thanks to Haisheng's tests, we know that changing threadCount from 1
> > to 4 seems to be a workaround, at least on some environments.
> > Moreover, I have noticed that parallel "both" value is deprecated [1],
> > so maybe we should try some other values and see if that solves the
> > problem without disabling parallelization.
> >
> > [1]
> http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#parallel
> >
> >
> >
> > Le lun. 3 juin 2019 à 11:00, Stamatis Zampetakis  a
> > écrit :
> >
> >> Hi all,
> >>
> >> The past few days we had observed various problems on Jenkins (and
> >> elsewhere) which made our test suite hang.
> >>
> >> Haisheng and Ruben found out that the problem was the parallel
> execution of
> >> parameterized tests [1]. I see two ways to unblock this situation and
> avoid
> >> such problems in the future:
> >>
> >> * disable parallel execution of unit tests;
> >> * forbid the use of @RunWith(Parameterized.class) annotation;
> >>
> >> at least till [1] is resolved.
> >>
> >> What do you think?
> >>
> >> [1] https://issues.apache.org/jira/browse/SUREFIRE-1430
> >>
>
>

Re: [DISCUSSION] Extension of Metadata Query

[Julian Hyde]

> 1. Why we have 2 methods in RelMetadataProvider?

The metadata system is complicated. We need to allow multiple handlers
for any given call. So, making a metadata call involves multiple
dispatch [1] based on the metadata method being called, the type of
RelNode, and the handlers that are registered. Also it needs to cache
results, and detect cycles. And the dispatch needs to be efficient, so
we generate janino code to do the dispatch, and re-generate when new
handlers or sub-classes of RelNode are added.

I forget details, the two methods are basically required to allow us
to generate code to do the dispatch.

> 2. We should make the RelMetadataQuery in RelOptCluster pluggable.

I disagree. RelMetadataQuery is not an extension point. Its sole
purpose is to to keep state (the cache and cycle-checking).
RelMetadataProvider is the extension point. (By analogy, if you are
un-parsing an AST, you let each AST sub-class handle unparsing itself,
but the unparsed text goes to a simple StringBuilder. RelMetadataQuery
is in the role of the StringBuilder. In a complex system, it is nice
to keep some of the components simple, or at least keep them to
prescribed roles.)

Julian

[1] https://en.wikipedia.org/wiki/Multiple_dispatch

On Sun, Jun 2, 2019 at 11:19 PM Yuzhao Chen  wrote:
>
> Currently we provide answer to metadata query through RelMetadataProvider 
> [1], there are some sub-classes of it:
>
> RelMetadataProvider
> |
> |- VolcanoRelMetadataProvider
> |- ChainedRelMetadataProvider/DefaultRelMetadataProvider
> |- HepRelMetadataProvider
> |- CachingRelMetadataProvider
> |- ReflectiveRelMetadataProvider
> |- JaninoRelMetadataProvider
>
> The RelMetadataProvider has two methods: #apply and #handlers, the #apply 
> method seems a programming interface and there is a demo code how we can use 
> it:
>
> RelMetadataProvider provider;
> LogicalFilter filter;
> RexNode predicate;
> Function function =
> provider.apply(LogicalFilter.class, Selectivity.class};
> Selectivity selectivity = function.apply(filter);
> Double d = selectivity.selectivity(predicate);
>
> But let's see our RelOptCluster's member variables[2], there are 
> MetadataFactory and RelMetadataQuery which all can be used to query the 
> metadata, for MetadataFactory, there is a default impl named 
> MetadataFactoryImpl which will invoke RelMetadataProvider#apply internally, 
> for RelMetadataQuery, it will invoke RelMetadataProvider#handlers (finally 
> composed and codeden by JaninoRelMetadataProvider).
>
> In our planning phrase, we can invoke RelOptRuleCall#getMetadataQuery to get 
> the MQ and query the metadata.
>
> For extension of metadata handlers, we can set our customized 
> RelMetadataProvider in RelOptCluster[3]. But for RelMetadataQuery, we have no 
> way to extend it now, because the RelOptCluster always has a singleton 
> instance [4] which is only the default implementation.
>
>
> My question is as follows:
>
> 1. Why we have 2 methods in RelMetadataProvider, and why we need the 
> MetadataFactory and RelMetadataProvider#apply ? It seems that it's function 
> is already been overriden by RelMetadataQuery(The difference is that 
> MetadataFactory use Reflection and RelMetadataQuery use gened bytes code).
> 2. We should make the RelMetadataQuery in RelOptCluster pluggable.
>
>
> [1] 
> https://github.com/apache/calcite/blob/b0e83c469ff57257c1ea621ff943ca76f626a9b7/core/src/main/java/org/apache/calcite/rel/metadata/RelMetadataProvider.java#L38
> [2] 
> https://github.com/apache/calcite/blob/b0e83c469ff57257c1ea621ff943ca76f626a9b7/core/src/main/java/org/apache/calcite/plan/RelOptCluster.java#L49
> [3] 
> https://github.com/apache/calcite/blob/b0e83c469ff57257c1ea621ff943ca76f626a9b7/core/src/main/java/org/apache/calcite/plan/RelOptCluster.java#L135
> [4] 
> https://github.com/apache/calcite/blob/b0e83c469ff57257c1ea621ff943ca76f626a9b7/core/src/main/java/org/apache/calcite/plan/RelOptCluster.java#L151
>
>
>
> Best,
> Danny Chan

Re: [DISCUSS] Automated website builds

[Josh Elser]

+1

On 6/4/19 8:43 AM, Michael Mior wrote:

I see no reason not to test this out. I'd say go for it! Thanks Francis :)
--
Michael Mior
mm...@apache.org

Le lun. 3 juin 2019 à 21:04, Francis Chuang  a écrit :


A few months ago, I raised the possibility to automating our website
builds, so that we will not need to manually build the site on our
machines and push it to the calcite-site repository.

I've been quite busy and haven't been able to look into it, however I am
planning to look into in the next few weeks.

In generally, I think this is quite doable if we run the website build
on the git-websites node, however I'd like to look at what's installed
on those nodes and whether docker is available. It would also be a good
opportunity to run some test builds and I also have a few proposal
regarding the way websites are currently being built that I'd like to
confirm and test first.

If it's okay, I'd like to create a test-site branch on calcite,
calcite-avatica and calcite-avatica-go as well as calcite-site,
calcite-avatica-site and calcite-avatica-go-site jobs on Jenkins to test
the waters.

Please let me know if this is okay.

Francis

Re: Pluggable JDBC types

[Muhammad Gelbana]

The only difference I need to achieve while handling both types, is the
returned column type name (ResultSet.getMetaData().getColumnTypeName(int
index)).
The returned value is VARCHAR even if the column type is a user defined
type with the alias TEXT.

While getting the column type name using a real PostgreSQL connection for a
TEXT column, is TEXT, not VARCHAR.

Thanks,
Gelbana


On Tue, Jun 4, 2019 at 6:23 PM Stamatis Zampetakis 
wrote:

> I am not sure what problem exactly we are trying to solve here (sorry for
> that).
> From what I understood so far the requirement is to introduce a new
> built-in SQL type (i.e., TEXT).
> However, I am still trying to understand why do we need this.
> Are we going to treat TEXT and VARCHAR differently?
>
> On Tue, Jun 4, 2019 at 5:18 PM Muhammad Gelbana 
> wrote:
>
> > Thanks Lai, I beleive your analysis is correct.
> >
> > Which brings up another question:
> > Is it ok if we add support for what I'm trying to do here ? I can gladly
> > work on that but I need to know if it will be accepted.
> >
> > Thanks,
> > Gelbana
> >
> >
> > On Tue, Jun 4, 2019 at 8:38 AM Lai Zhou  wrote:
> >
> > > @Muhammad Gelbana，I think you just register an alias-name 'TEXT' for
> the
> > > SqlType  'VARCHAR'.
> > > The parser did the right thing here, see
> > >
> > >
> >
> https://github.com/apache/calcite/blob/9721283bd0ce46a337f51a3691585cca8003e399/core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java#L1566
> > > When the parser encountered a 'text' SqlIdentifier, it would get the
> type
> > > from the rootSchema, the type was SqlTypeName.VARCHAR here , that you
> > > registered before.
> > > If you really need a new sqlType named 'text' rather than an
> alias-name,
> > I
> > > guess you need to introduce a new kind of SqlTypeName .
> > >
> > >
> > >
> > >
> > > Muhammad Gelbana  于2019年6月3日周一 下午6:54写道：
> > >
> > > > Is that different from what I mentioned in my Jira comment ? Here it
> is
> > > > again:
> > > >
> > > > Connection connection = DriverManager.getConnection("jdbc:calcite:",
> > > info);
> > > >
> > > >
> > >
> >
> connection.unwrap(CalciteConnection.class).getRootSchema().unwrap(CalciteSchema.class).add("
> > > > *TEXT*", new RelProtoDataType() {
> > > >
> > > > @Override
> > > > public RelDataType apply(RelDataTypeFactory factory) {
> > > > return
> > > >
> factory.createTypeWithNullability(factory.createJavaType(String.class),
> > > > false);
> > > > // return
> > > >
> > > >
> > >
> >
> factory.createTypeWithNullability(factory.createSqlType(SqlTypeName.VARCHAR),
> > > > false); // Has the same effect
> > > > }
> > > > });
> > > >
> > > > This still returns a column type name of VARCHAR, not *TEXT*.
> > > >
> > > > I tried providing the type through the model as the UdtTest does but
> > it's
> > > > giving me the same output.
> > > >
> > > > Thanks,
> > > > Gelbana
> > > >
> > > >
> > > > On Mon, Jun 3, 2019 at 9:59 AM Julian Hyde  wrote:
> > > >
> > > > > User-defined types are probably the way to go.
> > > > >
> > > > > > On Jun 2, 2019, at 8:28 PM, Muhammad Gelbana <
> m.gelb...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > That was my first attempt and it worked, but Julian pointed out
> > that
> > > I
> > > > > can
> > > > > > support a type without modifying the parser (which I prefer) but
> I
> > > > > couldn't
> > > > > > get it to return the column type name as I wish.
> > > > > >
> > > > > > Thanks,
> > > > > > Gelbana
> > > > > >
> > > > > >
> > > > > > On Mon, Jun 3, 2019 at 3:13 AM Yuzhao Chen  >
> > > > wrote:
> > > > > >
> > > > > >> You don’t need to, just define a new type name in parser[1] and
> > > > > translate
> > > > > >> it to VARCHAR is okey.
> > > > > >>
> > > > > >> [1]
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/calcite/blob/b0e83c469ff57257c1ea621ff943ca76f626a9b7/server/src/main/codegen/config.fmpp#L375
> > > > > >>
> > > > > >> Best,
> > > > > >> Danny Chan
> > > > > >> 在 2019年6月3日 +0800 AM6:09，Muhammad Gelbana  > >，写道：
> > > > > >>> That I understand now. But how can I support casting to TEXT
> and
> > > > having
> > > > > >> the
> > > > > >>> returned column type name as TEXT (ie. Not VARCHAR) ?
> > > > > >>>
> > > > > >>> Thanks,
> > > > > >>> Gelbana
> > > > > >>>
> > > > > >>>
> > > > > >>> On Sun, Jun 2, 2019 at 7:41 PM Julian Hyde 
> > > wrote:
> > > > > >>>
> > > > >  The parser should only parse, not validate. This is a very
> > > important
> > > > >  organizing principle for the parser.
> > > > > 
> > > > >  If I write “x :: text” or “x :: foo” it is up to the type
> system
> > > > >  (implemented in the validator and elsewhere) to figure out
> > whether
> > > > > >> “text”
> > > > >  or “foo” are valid types.
> > > > > 
> > > > >  Logically, “x :: foo” is the same as “CAST(x AS foo)”. The
> > parser
> > > > > >> should
> > > > >  produce the same SqlCall in both cases. Then

Linq expressions to RexNode

[Khai Tran]

Just a bit strange, but just wonder if in calcite code base, is there any util 
function to convert linq expression back to RexNode?

Re: Giving the Calcite logo some love

[Julian Hyde]

I prefer both over the current logo. (And I made the current logo.)

Let's keep the discussion going, and get to a new logo.

On Tue, Jun 4, 2019 at 9:42 AM Ivan Grgurina  wrote:
>
> I prefer the one Daniel sent. It looks cleaner, but maybe the "periodic 
> table" logo can be made better by simplifying it, the shadow on the letter C 
> is... unusual.
>
> 
> From: Stamatis Zampetakis 
> Sent: Tuesday, June 4, 2019 6:32 PM
> To: dev@calcite.apache.org; humbed...@apache.org
> Subject: Re: Giving the Calcite logo some love
>
> Thanks for digging this out Daniel!
>
> At this point we have two candidates:
> http://humbedooh.com/calcite-proposed.svg
> https://svgshare.com/s/86r
>
> Do we like any of above more than our current logo (the way they are or
> with slight modifications) ?
>
>
>
>
> On Mon, Jun 3, 2019 at 3:15 AM Yuzhao Chen  wrote:
>
> > Oh, I see a big hammer, thanks Daniel !
> >
> > Best,
> > Danny Chan
> > 在 2019年6月3日 +0800 AM6:21，Daniel Gruno ，写道：
> > > Found it!
> > >
> > > http://humbedooh.com/calcite-proposed.svg
> > >
> > > Thanks, Wayback Machine!
> > >
> > > On 19/05/2019 10.01, Stamatis Zampetakis wrote:
> > > > Hi all,
> > > >
> > > > We started this discussion about a year ago and many people were
> > > > positive with the idea of having a new logo for Calcite.
> > > > We had some nice proposals at the time and maybe now somebody else has
> > > > also new ideas/designs to contribute.
> > > >
> > > > @Daniel: Is there any chance that you still have the logos you proposed
> > > > somewhere available? The old link [1] does not work anymore.
> > > >
> > > > Best,
> > > > Stamatis
> > > >
> > > > [1]
> > http://www.apache.org/logos/comdev-test/res/calcite/calcite-proposed.svg
> > > >
> > > >
> > > > On Thu, Aug 30, 2018 at 12:02 AM Stamatis Zampetakis <
> > zabe...@gmail.com
> > > > > wrote:
> > > >
> > > > Vladimir>Could you flip rhombus so it goes right-up?
> > > >
> > > > https://svgshare.com/s/86r
> > > >
> > > > I hope this is what you meant.
> > > >
> > > > Best,
> > > > Stamatis
> > > >
> > > > Στις Τετ, 29 Αυγ 2018 στις 6:30 μ.μ., ο/η Michael Mior
> > > > mailto:mm...@apache.org>> έγραψε:
> > > >
> > > > Just a note since we're on the topic that whatever logos we come
> > > > up with
> > > > should be sure to have TM clearly indicated.
> > > >
> > > > --
> > > > Michael Mior
> > > > mm...@apache.org 
> > > >
> > > >
> > > >
> > > > Le mer. 29 août 2018 à 03:13, Julian Hyde
> > > > mailto:jhyde.apa...@gmail.com>> a écrit :
> > > >
> > > > > Yes indeed!
> > > > >
> > > > > If someone feels inspired to produce a logo, here’s my
> > > > suggestion of a
> > > > > theme/image: a spider, specifically a Barn Spider (Araneus
> > > > Cavaticus)[1].
> > > > > It was the origin of the name “avatica”, connects and spins
> > > > webs, and the
> > > > > eponymous individual in Charlotte’s Web had rather exceptional
> > > > > communication skills.
> > > > >
> > > > > Julian
> > > > >
> > > > > [1] https://en.m.wikipedia.org/wiki/Barn_spider
> > > > >
> > > > > > On Aug 28, 2018, at 9:49 PM, Francis Chuang
> > > > mailto:francischu...@apache.org>>
> > > > > wrote:
> > > > > >
> > > > > > The designs I have seen so far look really good! Would it
> > > > also make
> > > > > sense to design a variant for Avatica as well? This is what
> > > > the current
> > > > > Avatica logo looks like:
> > > > https://calcite.apache.org/avatica/img/logo.png
> > > > > >
> > > > > > Francis
> > > > > >
> > > > > > > On 29/08/2018 7:08 AM, Vladimir Sitnikov wrote:
> > > > > > > Stamatis>How about something like the following:
> > > > > > >
> > > > > > > There's left-to-right vs right-to-left issue, however I
> > > > would claim that
> > > > > > > the direction of improvement is right+up.
> > > > > > > For instance: BTC price is good when plots go to the right
> > > > and go
> > > > > upward.
> > > > > > >
> > > > > > > https://svgur.com/s/83y is slanted backward.
> > > > > > > That creates perception of "Calcite holding back the
> > > > progress" or
> > > > > "Apache
> > > > > > > pushing C away" or something like that.
> > > > > > > Could you flip rhombus so it goes right-up?
> > > > > > >
> > > > > > >
> > > > > > > Vladimir
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >

Re: Giving the Calcite logo some love

[Ivan Grgurina]

I prefer the one Daniel sent. It looks cleaner, but maybe the "periodic table" 
logo can be made better by simplifying it, the shadow on the letter C is... 
unusual.


From: Stamatis Zampetakis 
Sent: Tuesday, June 4, 2019 6:32 PM
To: dev@calcite.apache.org; humbed...@apache.org
Subject: Re: Giving the Calcite logo some love

Thanks for digging this out Daniel!

At this point we have two candidates:
http://humbedooh.com/calcite-proposed.svg
https://svgshare.com/s/86r

Do we like any of above more than our current logo (the way they are or
with slight modifications) ?




On Mon, Jun 3, 2019 at 3:15 AM Yuzhao Chen  wrote:

> Oh, I see a big hammer, thanks Daniel !
>
> Best,
> Danny Chan
> 在 2019年6月3日 +0800 AM6:21，Daniel Gruno ，写道：
> > Found it!
> >
> > http://humbedooh.com/calcite-proposed.svg
> >
> > Thanks, Wayback Machine!
> >
> > On 19/05/2019 10.01, Stamatis Zampetakis wrote:
> > > Hi all,
> > >
> > > We started this discussion about a year ago and many people were
> > > positive with the idea of having a new logo for Calcite.
> > > We had some nice proposals at the time and maybe now somebody else has
> > > also new ideas/designs to contribute.
> > >
> > > @Daniel: Is there any chance that you still have the logos you proposed
> > > somewhere available? The old link [1] does not work anymore.
> > >
> > > Best,
> > > Stamatis
> > >
> > > [1]
> http://www.apache.org/logos/comdev-test/res/calcite/calcite-proposed.svg
> > >
> > >
> > > On Thu, Aug 30, 2018 at 12:02 AM Stamatis Zampetakis <
> zabe...@gmail.com
> > > > wrote:
> > >
> > > Vladimir>Could you flip rhombus so it goes right-up?
> > >
> > > https://svgshare.com/s/86r
> > >
> > > I hope this is what you meant.
> > >
> > > Best,
> > > Stamatis
> > >
> > > Στις Τετ, 29 Αυγ 2018 στις 6:30 μ.μ., ο/η Michael Mior
> > > mailto:mm...@apache.org>> έγραψε:
> > >
> > > Just a note since we're on the topic that whatever logos we come
> > > up with
> > > should be sure to have TM clearly indicated.
> > >
> > > --
> > > Michael Mior
> > > mm...@apache.org 
> > >
> > >
> > >
> > > Le mer. 29 août 2018 à 03:13, Julian Hyde
> > > mailto:jhyde.apa...@gmail.com>> a écrit :
> > >
> > > > Yes indeed!
> > > >
> > > > If someone feels inspired to produce a logo, here’s my
> > > suggestion of a
> > > > theme/image: a spider, specifically a Barn Spider (Araneus
> > > Cavaticus)[1].
> > > > It was the origin of the name “avatica”, connects and spins
> > > webs, and the
> > > > eponymous individual in Charlotte’s Web had rather exceptional
> > > > communication skills.
> > > >
> > > > Julian
> > > >
> > > > [1] https://en.m.wikipedia.org/wiki/Barn_spider
> > > >
> > > > > On Aug 28, 2018, at 9:49 PM, Francis Chuang
> > > mailto:francischu...@apache.org>>
> > > > wrote:
> > > > >
> > > > > The designs I have seen so far look really good! Would it
> > > also make
> > > > sense to design a variant for Avatica as well? This is what
> > > the current
> > > > Avatica logo looks like:
> > > https://calcite.apache.org/avatica/img/logo.png
> > > > >
> > > > > Francis
> > > > >
> > > > > > On 29/08/2018 7:08 AM, Vladimir Sitnikov wrote:
> > > > > > Stamatis>How about something like the following:
> > > > > >
> > > > > > There's left-to-right vs right-to-left issue, however I
> > > would claim that
> > > > > > the direction of improvement is right+up.
> > > > > > For instance: BTC price is good when plots go to the right
> > > and go
> > > > upward.
> > > > > >
> > > > > > https://svgur.com/s/83y is slanted backward.
> > > > > > That creates perception of "Calcite holding back the
> > > progress" or
> > > > "Apache
> > > > > > pushing C away" or something like that.
> > > > > > Could you flip rhombus so it goes right-up?
> > > > > >
> > > > > >
> > > > > > Vladimir
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Giving the Calcite logo some love

[Stamatis Zampetakis]

Thanks for digging this out Daniel!

At this point we have two candidates:
http://humbedooh.com/calcite-proposed.svg
https://svgshare.com/s/86r

Do we like any of above more than our current logo (the way they are or
with slight modifications) ?




On Mon, Jun 3, 2019 at 3:15 AM Yuzhao Chen  wrote:

> Oh, I see a big hammer, thanks Daniel !
>
> Best,
> Danny Chan
> 在 2019年6月3日 +0800 AM6:21，Daniel Gruno ，写道：
> > Found it!
> >
> > http://humbedooh.com/calcite-proposed.svg
> >
> > Thanks, Wayback Machine!
> >
> > On 19/05/2019 10.01, Stamatis Zampetakis wrote:
> > > Hi all,
> > >
> > > We started this discussion about a year ago and many people were
> > > positive with the idea of having a new logo for Calcite.
> > > We had some nice proposals at the time and maybe now somebody else has
> > > also new ideas/designs to contribute.
> > >
> > > @Daniel: Is there any chance that you still have the logos you proposed
> > > somewhere available? The old link [1] does not work anymore.
> > >
> > > Best,
> > > Stamatis
> > >
> > > [1]
> http://www.apache.org/logos/comdev-test/res/calcite/calcite-proposed.svg
> > >
> > >
> > > On Thu, Aug 30, 2018 at 12:02 AM Stamatis Zampetakis <
> zabe...@gmail.com
> > > > wrote:
> > >
> > > Vladimir>Could you flip rhombus so it goes right-up?
> > >
> > > https://svgshare.com/s/86r
> > >
> > > I hope this is what you meant.
> > >
> > > Best,
> > > Stamatis
> > >
> > > Στις Τετ, 29 Αυγ 2018 στις 6:30 μ.μ., ο/η Michael Mior
> > > mailto:mm...@apache.org>> έγραψε:
> > >
> > > Just a note since we're on the topic that whatever logos we come
> > > up with
> > > should be sure to have TM clearly indicated.
> > >
> > > --
> > > Michael Mior
> > > mm...@apache.org 
> > >
> > >
> > >
> > > Le mer. 29 août 2018 à 03:13, Julian Hyde
> > > mailto:jhyde.apa...@gmail.com>> a écrit :
> > >
> > > > Yes indeed!
> > > >
> > > > If someone feels inspired to produce a logo, here’s my
> > > suggestion of a
> > > > theme/image: a spider, specifically a Barn Spider (Araneus
> > > Cavaticus)[1].
> > > > It was the origin of the name “avatica”, connects and spins
> > > webs, and the
> > > > eponymous individual in Charlotte’s Web had rather exceptional
> > > > communication skills.
> > > >
> > > > Julian
> > > >
> > > > [1] https://en.m.wikipedia.org/wiki/Barn_spider
> > > >
> > > > > On Aug 28, 2018, at 9:49 PM, Francis Chuang
> > > mailto:francischu...@apache.org>>
> > > > wrote:
> > > > >
> > > > > The designs I have seen so far look really good! Would it
> > > also make
> > > > sense to design a variant for Avatica as well? This is what
> > > the current
> > > > Avatica logo looks like:
> > > https://calcite.apache.org/avatica/img/logo.png
> > > > >
> > > > > Francis
> > > > >
> > > > > > On 29/08/2018 7:08 AM, Vladimir Sitnikov wrote:
> > > > > > Stamatis>How about something like the following:
> > > > > >
> > > > > > There's left-to-right vs right-to-left issue, however I
> > > would claim that
> > > > > > the direction of improvement is right+up.
> > > > > > For instance: BTC price is good when plots go to the right
> > > and go
> > > > upward.
> > > > > >
> > > > > > https://svgur.com/s/83y is slanted backward.
> > > > > > That creates perception of "Calcite holding back the
> > > progress" or
> > > > "Apache
> > > > > > pushing C away" or something like that.
> > > > > > Could you flip rhombus so it goes right-up?
> > > > > >
> > > > > >
> > > > > > Vladimir
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Pluggable JDBC types

[Stamatis Zampetakis]

I am not sure what problem exactly we are trying to solve here (sorry for
that).
>From what I understood so far the requirement is to introduce a new
built-in SQL type (i.e., TEXT).
However, I am still trying to understand why do we need this.
Are we going to treat TEXT and VARCHAR differently?

On Tue, Jun 4, 2019 at 5:18 PM Muhammad Gelbana  wrote:

> Thanks Lai, I beleive your analysis is correct.
>
> Which brings up another question:
> Is it ok if we add support for what I'm trying to do here ? I can gladly
> work on that but I need to know if it will be accepted.
>
> Thanks,
> Gelbana
>
>
> On Tue, Jun 4, 2019 at 8:38 AM Lai Zhou  wrote:
>
> > @Muhammad Gelbana，I think you just register an alias-name 'TEXT' for the
> > SqlType  'VARCHAR'.
> > The parser did the right thing here, see
> >
> >
> https://github.com/apache/calcite/blob/9721283bd0ce46a337f51a3691585cca8003e399/core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java#L1566
> > When the parser encountered a 'text' SqlIdentifier, it would get the type
> > from the rootSchema, the type was SqlTypeName.VARCHAR here , that you
> > registered before.
> > If you really need a new sqlType named 'text' rather than an alias-name,
> I
> > guess you need to introduce a new kind of SqlTypeName .
> >
> >
> >
> >
> > Muhammad Gelbana  于2019年6月3日周一 下午6:54写道：
> >
> > > Is that different from what I mentioned in my Jira comment ? Here it is
> > > again:
> > >
> > > Connection connection = DriverManager.getConnection("jdbc:calcite:",
> > info);
> > >
> > >
> >
> connection.unwrap(CalciteConnection.class).getRootSchema().unwrap(CalciteSchema.class).add("
> > > *TEXT*", new RelProtoDataType() {
> > >
> > > @Override
> > > public RelDataType apply(RelDataTypeFactory factory) {
> > > return
> > > factory.createTypeWithNullability(factory.createJavaType(String.class),
> > > false);
> > > // return
> > >
> > >
> >
> factory.createTypeWithNullability(factory.createSqlType(SqlTypeName.VARCHAR),
> > > false); // Has the same effect
> > > }
> > > });
> > >
> > > This still returns a column type name of VARCHAR, not *TEXT*.
> > >
> > > I tried providing the type through the model as the UdtTest does but
> it's
> > > giving me the same output.
> > >
> > > Thanks,
> > > Gelbana
> > >
> > >
> > > On Mon, Jun 3, 2019 at 9:59 AM Julian Hyde  wrote:
> > >
> > > > User-defined types are probably the way to go.
> > > >
> > > > > On Jun 2, 2019, at 8:28 PM, Muhammad Gelbana 
> > > > wrote:
> > > > >
> > > > > That was my first attempt and it worked, but Julian pointed out
> that
> > I
> > > > can
> > > > > support a type without modifying the parser (which I prefer) but I
> > > > couldn't
> > > > > get it to return the column type name as I wish.
> > > > >
> > > > > Thanks,
> > > > > Gelbana
> > > > >
> > > > >
> > > > > On Mon, Jun 3, 2019 at 3:13 AM Yuzhao Chen 
> > > wrote:
> > > > >
> > > > >> You don’t need to, just define a new type name in parser[1] and
> > > > translate
> > > > >> it to VARCHAR is okey.
> > > > >>
> > > > >> [1]
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/calcite/blob/b0e83c469ff57257c1ea621ff943ca76f626a9b7/server/src/main/codegen/config.fmpp#L375
> > > > >>
> > > > >> Best,
> > > > >> Danny Chan
> > > > >> 在 2019年6月3日 +0800 AM6:09，Muhammad Gelbana  >，写道：
> > > > >>> That I understand now. But how can I support casting to TEXT and
> > > having
> > > > >> the
> > > > >>> returned column type name as TEXT (ie. Not VARCHAR) ?
> > > > >>>
> > > > >>> Thanks,
> > > > >>> Gelbana
> > > > >>>
> > > > >>>
> > > > >>> On Sun, Jun 2, 2019 at 7:41 PM Julian Hyde 
> > wrote:
> > > > >>>
> > > >  The parser should only parse, not validate. This is a very
> > important
> > > >  organizing principle for the parser.
> > > > 
> > > >  If I write “x :: text” or “x :: foo” it is up to the type system
> > > >  (implemented in the validator and elsewhere) to figure out
> whether
> > > > >> “text”
> > > >  or “foo” are valid types.
> > > > 
> > > >  Logically, “x :: foo” is the same as “CAST(x AS foo)”. The
> parser
> > > > >> should
> > > >  produce the same SqlCall in both cases. Then the parser’s job is
> > > done.
> > > > 
> > > >  Julian
> > > > 
> > > > 
> > > > > On Jun 2, 2019, at 6:42 AM, Muhammad Gelbana <
> > m.gelb...@gmail.com>
> > > >  wrote:
> > > > >
> > > > > I'm trying to support the PostgreSQL TEXT type[1]. It's
> > basically a
> > > >  VARCHAR.
> > > > >
> > > > > As Julian mentioned in his comment on Jira, I don't need to
> > define
> > > a
> > > > > keyword to achieve what I need so I tried exploring that and
> here
> > > is
> > > >  what I
> > > > > observed so far:
> > > > >
> > > > > 1. If I define a new keyword in the parser, I face no trouble
> > > > >> whatsoever
> > > > > except for the numerous wiring I need to do for
> > RexToLixTranslator,

Re: Pluggable JDBC types

[Muhammad Gelbana]

Thanks Lai, I beleive your analysis is correct.

Which brings up another question:
Is it ok if we add support for what I'm trying to do here ? I can gladly
work on that but I need to know if it will be accepted.

Thanks,
Gelbana


On Tue, Jun 4, 2019 at 8:38 AM Lai Zhou  wrote:

> @Muhammad Gelbana，I think you just register an alias-name 'TEXT' for the
> SqlType  'VARCHAR'.
> The parser did the right thing here, see
>
> https://github.com/apache/calcite/blob/9721283bd0ce46a337f51a3691585cca8003e399/core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java#L1566
> When the parser encountered a 'text' SqlIdentifier, it would get the type
> from the rootSchema, the type was SqlTypeName.VARCHAR here , that you
> registered before.
> If you really need a new sqlType named 'text' rather than an alias-name, I
> guess you need to introduce a new kind of SqlTypeName .
>
>
>
>
> Muhammad Gelbana  于2019年6月3日周一 下午6:54写道：
>
> > Is that different from what I mentioned in my Jira comment ? Here it is
> > again:
> >
> > Connection connection = DriverManager.getConnection("jdbc:calcite:",
> info);
> >
> >
> connection.unwrap(CalciteConnection.class).getRootSchema().unwrap(CalciteSchema.class).add("
> > *TEXT*", new RelProtoDataType() {
> >
> > @Override
> > public RelDataType apply(RelDataTypeFactory factory) {
> > return
> > factory.createTypeWithNullability(factory.createJavaType(String.class),
> > false);
> > // return
> >
> >
> factory.createTypeWithNullability(factory.createSqlType(SqlTypeName.VARCHAR),
> > false); // Has the same effect
> > }
> > });
> >
> > This still returns a column type name of VARCHAR, not *TEXT*.
> >
> > I tried providing the type through the model as the UdtTest does but it's
> > giving me the same output.
> >
> > Thanks,
> > Gelbana
> >
> >
> > On Mon, Jun 3, 2019 at 9:59 AM Julian Hyde  wrote:
> >
> > > User-defined types are probably the way to go.
> > >
> > > > On Jun 2, 2019, at 8:28 PM, Muhammad Gelbana 
> > > wrote:
> > > >
> > > > That was my first attempt and it worked, but Julian pointed out that
> I
> > > can
> > > > support a type without modifying the parser (which I prefer) but I
> > > couldn't
> > > > get it to return the column type name as I wish.
> > > >
> > > > Thanks,
> > > > Gelbana
> > > >
> > > >
> > > > On Mon, Jun 3, 2019 at 3:13 AM Yuzhao Chen 
> > wrote:
> > > >
> > > >> You don’t need to, just define a new type name in parser[1] and
> > > translate
> > > >> it to VARCHAR is okey.
> > > >>
> > > >> [1]
> > > >>
> > >
> >
> https://github.com/apache/calcite/blob/b0e83c469ff57257c1ea621ff943ca76f626a9b7/server/src/main/codegen/config.fmpp#L375
> > > >>
> > > >> Best,
> > > >> Danny Chan
> > > >> 在 2019年6月3日 +0800 AM6:09，Muhammad Gelbana ，写道：
> > > >>> That I understand now. But how can I support casting to TEXT and
> > having
> > > >> the
> > > >>> returned column type name as TEXT (ie. Not VARCHAR) ?
> > > >>>
> > > >>> Thanks,
> > > >>> Gelbana
> > > >>>
> > > >>>
> > > >>> On Sun, Jun 2, 2019 at 7:41 PM Julian Hyde 
> wrote:
> > > >>>
> > >  The parser should only parse, not validate. This is a very
> important
> > >  organizing principle for the parser.
> > > 
> > >  If I write “x :: text” or “x :: foo” it is up to the type system
> > >  (implemented in the validator and elsewhere) to figure out whether
> > > >> “text”
> > >  or “foo” are valid types.
> > > 
> > >  Logically, “x :: foo” is the same as “CAST(x AS foo)”. The parser
> > > >> should
> > >  produce the same SqlCall in both cases. Then the parser’s job is
> > done.
> > > 
> > >  Julian
> > > 
> > > 
> > > > On Jun 2, 2019, at 6:42 AM, Muhammad Gelbana <
> m.gelb...@gmail.com>
> > >  wrote:
> > > >
> > > > I'm trying to support the PostgreSQL TEXT type[1]. It's
> basically a
> > >  VARCHAR.
> > > >
> > > > As Julian mentioned in his comment on Jira, I don't need to
> define
> > a
> > > > keyword to achieve what I need so I tried exploring that and here
> > is
> > >  what I
> > > > observed so far:
> > > >
> > > > 1. If I define a new keyword in the parser, I face no trouble
> > > >> whatsoever
> > > > except for the numerous wiring I need to do for
> RexToLixTranslator,
> > > > JavaTypeFactoryImpl, SqlTypeAssignmentRules and SqlTypeName. I
> > won't
> > > >> be
> > > > suprised if I'm missing anything but doing what I did at first
> > > >> managed to
> > > > get my queries through.
> > > >
> > > > 2. If I define the type by plugging it in through the root
> schema,
> > I
> > > >> face
> > > > two problems: a) The field cannot be declared as nullable because
> > the
> > >  query
> > > > I'm using for testing gets data from (VALUES()) which doesn't
> > produce
> > >  null
> > > > values, so an exception is thrown. b) The returned column type
> name
> > > >> is
> > > > VARC

Execute multiple RelNodes from single RelNode

[Ivan Grgurina]

Can I create multiple top-level RelNodes out of the single SqlNode in 
SqlToRelConverter class or single RelNode through RelOptRules and have Calcite 
push it to the databases?


Ivan Grgurina
Research Assistant (ZEMRIS)

[cid:884cd68d-1828-46af-a016-73c1c9994211]
 [cid:798dc92a-3770-40ee-8dc9-8a2db7e82d98] 

Re: [DISCUSS] Automated website builds

[Michael Mior]

I see no reason not to test this out. I'd say go for it! Thanks Francis :)
--
Michael Mior
mm...@apache.org

Le lun. 3 juin 2019 à 21:04, Francis Chuang  a écrit :
>
> A few months ago, I raised the possibility to automating our website
> builds, so that we will not need to manually build the site on our
> machines and push it to the calcite-site repository.
>
> I've been quite busy and haven't been able to look into it, however I am
> planning to look into in the next few weeks.
>
> In generally, I think this is quite doable if we run the website build
> on the git-websites node, however I'd like to look at what's installed
> on those nodes and whether docker is available. It would also be a good
> opportunity to run some test builds and I also have a few proposal
> regarding the way websites are currently being built that I'd like to
> confirm and test first.
>
> If it's okay, I'd like to create a test-site branch on calcite,
> calcite-avatica and calcite-avatica-go as well as calcite-site,
> calcite-avatica-site and calcite-avatica-go-site jobs on Jenkins to test
> the waters.
>
> Please let me know if this is okay.
>
> Francis

[jira] [Created] (CALCITE-3109) RepeatUnion improvements

[Ruben Quesada Lopez (JIRA)]

Ruben Quesada Lopez created CALCITE-3109:


 Summary: RepeatUnion improvements
 Key: CALCITE-3109
 URL: https://issues.apache.org/jira/browse/CALCITE-3109
 Project: Calcite
  Issue Type: Task
Reporter: Ruben Quesada Lopez
Assignee: Ruben Quesada Lopez


After the implementation of CALCITE-2812, there are some pending details to be 
addressed in order to make the RepeatUnion more consistent:
- Replace {{String tableName}} with a {{Table}} (perhaps wrapped in a 
{{RelOptTable}}).
- Rename {{maxRep}} as {{iterationLimit}}. Change this field to allow all 
integer values, and treat all negative values as "no limit".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Re: Extracting all columns used in a query

[Ivan Grgurina]

Hi Adam,

I developed a solution to your problem (and mine 🙂 ) as part of my master 
thesis. Code is available at [1].

If you have any questions regarding the code, feel free to ask me.

[1] 
https://github.com/igrgurina/multicloud_rewriter/blob/master/core/src/main/java/cloud/sec/core/adapter/jdbc/MultiCloudDataManager.java
[https://avatars0.githubusercontent.com/u/2978480?s=400&v=4]
igrgurina/multicloud_rewriter
Contribute to igrgurina/multicloud_rewriter development by creating an account 
on GitHub.
github.com




Ivan Grgurina
Research Assistant (ZEMRIS)

[cid:f99e49a4-e831-4f9a-b440-6322e4462e75]
 [cid:6299c77a-e109-43ef-b370-4abc0d34a3f1] 



From: Haisheng Yuan 
Sent: Tuesday, June 4, 2019 11:45 AM
To: Stamatis Zampetakis; Apache Calcite dev list
Subject: Re: Re: Extracting all columns used in a query

Hi Adam,

Calcite defintely can do this. But can you first clarify what do you mean by 
all the columns in a query?
e.g.
R(r1, r2,r3), S(s1,s2,s3)
SELECT r1+s1 FROM R,S WHERE r3=s3;

What do you want to extract from this query? r1,r3 for R and s1, s3 for S?
And why do you want do that?

- Haisheng 
Yuan--
发件人：Stamatis Zampetakis
日　期：2019年06月04日 17:26:18
收件人：
主　题：Re: Extracting all columns used in a query

Hey Adam,

I am not sure exactly what information you need, and at which level
(SqlNode/RelNode), but maybe you can exploit what is present in RelRoot
[1].
Follow the calls to the constructor to see which APIs can provide you what
you need (check for instance, SqlToRelConverter.convertQuery [2]).

Best,
Stamatis

[1]
https://github.com/apache/calcite/blob/7f33215ffaf9c0b8f4bef082913c910c77bf4427/core/src/main/java/org/apache/calcite/rel/RelRoot.java#L89
[2]
https://github.com/apache/calcite/blob/7f33215ffaf9c0b8f4bef082913c910c77bf4427/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L553

On Tue, Jun 4, 2019 at 12:39 AM Muhammad Gelbana 
wrote:

> I don't konw if there is an API for that but visiting the  parsed/validated
> SqlNode tree can do what you asked for.
>
> Thanks,
> Gelbana
>
>
> On Tue, Jun 4, 2019 at 12:12 AM Adam Rivelli  wrote:
>
> > Hi all,
> >
> > I'm trying to extract all of the (fully qualified) columns used by a
> query
> > - similar to the information provided by
> > RelMetadataQuery.getTableReferences()
> > <
> >
> https://calcite.apache.org/apidocs/org/apache/calcite/rel/metadata/RelMetadataQuery.html#getTableReferences-org.apache.calcite.rel.RelNode-
> > >,
> > but for column references. Is this possible to do using Calcite?
> >
> > I've been looking through the API docs and experimenting with the API,
> but
> > I haven't found a straightforward way of doing this. Any help or
> > information is appreciated.
> >
> > Adam
> >
>

Re: Re: Extracting all columns used in a query

[Haisheng Yuan]

Hi Adam,

Calcite defintely can do this. But can you first clarify what do you mean by 
all the columns in a query?
e.g.
R(r1, r2,r3), S(s1,s2,s3)
SELECT r1+s1 FROM R,S WHERE r3=s3;

What do you want to extract from this query? r1,r3 for R and s1, s3 for S?
And why do you want do that?

- Haisheng 
Yuan--
发件人：Stamatis Zampetakis
日　期：2019年06月04日 17:26:18
收件人：
主　题：Re: Extracting all columns used in a query

Hey Adam,

I am not sure exactly what information you need, and at which level
(SqlNode/RelNode), but maybe you can exploit what is present in RelRoot
[1].
Follow the calls to the constructor to see which APIs can provide you what
you need (check for instance, SqlToRelConverter.convertQuery [2]).

Best,
Stamatis

[1]
https://github.com/apache/calcite/blob/7f33215ffaf9c0b8f4bef082913c910c77bf4427/core/src/main/java/org/apache/calcite/rel/RelRoot.java#L89
[2]
https://github.com/apache/calcite/blob/7f33215ffaf9c0b8f4bef082913c910c77bf4427/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L553

On Tue, Jun 4, 2019 at 12:39 AM Muhammad Gelbana 
wrote:

> I don't konw if there is an API for that but visiting the  parsed/validated
> SqlNode tree can do what you asked for.
>
> Thanks,
> Gelbana
>
>
> On Tue, Jun 4, 2019 at 12:12 AM Adam Rivelli  wrote:
>
> > Hi all,
> >
> > I'm trying to extract all of the (fully qualified) columns used by a
> query
> > - similar to the information provided by
> > RelMetadataQuery.getTableReferences()
> > <
> >
> https://calcite.apache.org/apidocs/org/apache/calcite/rel/metadata/RelMetadataQuery.html#getTableReferences-org.apache.calcite.rel.RelNode-
> > >,
> > but for column references. Is this possible to do using Calcite?
> >
> > I've been looking through the API docs and experimenting with the API,
> but
> > I haven't found a straightforward way of doing this. Any help or
> > information is appreciated.
> >
> > Adam
> >
>

Re: Extracting all columns used in a query

[Stamatis Zampetakis]

Hey Adam,

I am not sure exactly what information you need, and at which level
(SqlNode/RelNode), but maybe you can exploit what is present in RelRoot
[1].
Follow the calls to the constructor to see which APIs can provide you what
you need (check for instance, SqlToRelConverter.convertQuery [2]).

Best,
Stamatis

[1]
https://github.com/apache/calcite/blob/7f33215ffaf9c0b8f4bef082913c910c77bf4427/core/src/main/java/org/apache/calcite/rel/RelRoot.java#L89
[2]
https://github.com/apache/calcite/blob/7f33215ffaf9c0b8f4bef082913c910c77bf4427/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L553

On Tue, Jun 4, 2019 at 12:39 AM Muhammad Gelbana 
wrote:

> I don't konw if there is an API for that but visiting the  parsed/validated
> SqlNode tree can do what you asked for.
>
> Thanks,
> Gelbana
>
>
> On Tue, Jun 4, 2019 at 12:12 AM Adam Rivelli  wrote:
>
> > Hi all,
> >
> > I'm trying to extract all of the (fully qualified) columns used by a
> query
> > - similar to the information provided by
> > RelMetadataQuery.getTableReferences()
> > <
> >
> https://calcite.apache.org/apidocs/org/apache/calcite/rel/metadata/RelMetadataQuery.html#getTableReferences-org.apache.calcite.rel.RelNode-
> > >,
> > but for column references. Is this possible to do using Calcite?
> >
> > I've been looking through the API docs and experimenting with the API,
> but
> > I haven't found a straightforward way of doing this. Any help or
> > information is appreciated.
> >
> > Adam
> >
>