Jin>Some of my users write tests based on functionalities provided from
Calcite
Jin>test framework
Frankly speaking, it was somewhat surprising for me, and it does bring
extra demand for having a proper calcite-test-framework.
The naming is hard, however, I would suggest
Stamatis> do we have another option so that the build does not fail on
Windows?
It boils down to a question: should the build fail if the source files
contain unexpected line endings?
Suppose someone uses Windows, and they create test_sql_plan.txt file with
wrong line endings (==LF).
Should the
Vladimir Sitnikov created CALCITE-3623:
--
Summary: Replace Spotless with Autostyle
Key: CALCITE-3623
URL: https://issues.apache.org/jira/browse/CALCITE-3623
Project: Calcite
Issue Type
Stamatis>To begin with we could try to hold a single meetup per month and
see later
Stamatis>on how it goes
It might be nice to try, however, it did not survive long the last time :(
Stamatis>The ranges should be rather large so that it is easier to find
Stamatis>some overlapping among us
An
AFAIK it does wait for the Nexus to release, however, I have no idea if
that means the files are immediately available.
Vladimir
Francis>My assumption is that they should be equivalent, but I would love
to get
Francis>some confirmation first as releasing a broken 1.17.0 would cause
more work.
For instance,
Zhenghua>the implementation was similar to PostgreSQL's
PostgreSQL DB stores timestamps similar to "UNIX timestamp" (it uses int8),
and it does that for both "with" and "without" time zone.
In other words, PostgreSQL cannot have "OffsetDateTime" semantics :(
Vladimir
Kevin>and every response was against the change
This ^^ is false.
Haisheng Yuan: "Unless we change $ to other symbol, I am inclined to keep
using Java"
https://lists.apache.org/thread.html/e3cbf20d22f7a4cc870626c0bc41b99430c2cc37318052d083166653%40%3Cdev.calcite.apache.org%3E
Rui Wang: "Can we
Julian>All of those changes were done by Vladimir. Not one of these changes
Julian>adds a single feature, or fixes a single bug, in "Calcite the
Julian>product", the library we deliver to our end-users.
Julian, I see you are frustrated that one of your use-cases is broken.
It is really sad it
Hi,
Currently
org.apache.calcite.test.CalciteAssert, org.apache.calcite.util.TestUtil,
and probably other classes are hard to find.
I suggest we create a testlib module (or something like that), that would
contain "test framework" code.
That would make test framework easier to discover (one
Vladimir>Quidem, CalciteAssert
Michael>If you want to propose removing either of these, we could have a
Michael>discussion about it, but you're talking about code which is already
Michael>heavily used throughout Calcite.
The point of "we assume contributors are good at Java, thus we must keep
the
Michael>However, we know our
Michael>current contributors are reasonably fluent in Java. I'm not sure
about
Michael>Kotlin.
1) New contributions are not fluent in Quidem at all
2) New contributions are not fluent in CalciteAssert at all
3) As I said, if someone does not want to get familiar with
https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-avatica-1.16.0-rc1/apache-calcite-avatica-1.16.0-src.tar.gz
does not build if Windows with default settings is used.
The error messages for "./gradlew build" are as follows:
-// dependency on it during compilation\n
Kevin>Focusing on the technical side of things, I agree that introducing a
new
Kevin>language is of little benefit currently
Kevin, what is your opinion on removing Quidem language?
Focusing on the technical side, it is a standalone language.
The language is not Java, it has limited tooling, it
Julian>actions over the last few weeks have left me angry, depressed, and
burned out with the project
That is sad. Please accept my apologies for hurting your feelings.
Julian>Of course, a lot of the things he is removing are things that I
personally have created, so I naturally take this more
Feng>Introducing another language
It is the same language that is used for the build scripts, so it is not a
new language.
Danny> in the code evolving and the Scala has many
Danny> tricky problems especially the version compatibility
Kotlin has strong backward compatibility.
I've filed two PRs to evaluate the impact of the replacement.
$ to _: https://github.com/apache/calcite/pull/1659
203.3sec, 5510 completed, 3 failed, 91 skipped, Gradle Test Run
:core:test
$ to #: https://github.com/apache/calcite/pull/1660
196.7sec, 5510 completed, 53 failed, 91 skipped,
Hi,
Calcite tests sometimes require multiline strings.
For instance: input SQL, output plan.
TL;DR: let's use Kotlin to improve test code readability. What do you think?
Unfortunately, Java <14 lacks multiline strings, so the current tests are
not that pretty :(
They are hard to read, hard to
Julian>For all other purpose, just use git.
"use git" contradicts with
http://www.apache.org/legal/release-policy.html#publication
legal/release-policy> Projects MUST direct outsiders towards official
releases rather than raw source repositories, nightly builds, snapshots,
release candidates, or
Julian>The git repo, at a particular commit, has an objective contents
That is true for binary blobs.
However, text files are converted on checkout as per core.autocrlf and
core.eol settings.
Julian>I suspect that when you use ‘git checkout’ with particular options
I suspect you are not very
Michael>The source code should be what unprocessed,
Michael>but since Windows
I do not get what you mean by "unprocessed".
Even Maven build had lots of exclude-include patterns, so it did "process"
the sources.
Then, the current source release contains LICENSE file that contains some
processing
>I think a source distribution should contain the raw, unprocessed source
files.
What do you mean by "raw"?
If I check out the same repository on Windows and macOS, I would get
**different** file contents for *.java files.
Windows machine would check out files as CRLF, and macOS would checkout
Hi,
https://github.com/apache/calcite/pull/1625 fixes concurrency issues with
Checkstyle and improves "code style" error messages.
if no-one objects within three days, I'll assume lazy consensus and commit
it.
You can find more info here:
Thank everybody for your participation.
The vote results are:
+1 Vladimir
-0 Michael
-1 Julian (it can not count as a veto because there's no technical
justification)
Unfortunately, the vote is undecided.
Vladimir
Stamatis>I mean that there are a few people who are a bit skeptical with
the change
Stamatis>so it seems that more convincing elements are needed.
There's no way to convince people who do not want to analyze Checkstyle /
build configuration themselves.
Everybody was happy with Maven, and there
Julian>I’m curious why the instructions include ‘-Prelease’.
The instructions in the vote mail provide a way to reproduce the archive.
In other words, the build would produce exactly the same file names and
contents (with the same checksums)
Julian>People should be able to unpack and build the
Kevin>fails on the zip when extracted.
zip has CRLF line endings, and tar.gz has LF line endings, so you should
select
the archive based on your needs and environment setup.
The archives are not identical on purpose.
Vladimir
>has one blank line after copyright
Beam seems to have mixed style as well:
blank line before "package":
https://github.com/apache/beam/blob/166c6de33f2491c4c9bd27511cc71e33f8d2a894/buildSrc/src/main/groovy/org/apache/beam/gradle/GrpcVendoring.groovy#L18
no blank before "package":
Hi,
I've noticed that package-info.java files have different copyright
"formatting" than all the other Java files.
The difference is as follows: package-info.java has a blank line after the
copyright, and all the other files do not have that blank line.
Sample blank like:
Francis>There are also links to Avatica docs in
Francis>the side bar and it would be a bit strange to have them always
point to
Francis>the master version of Avatica.
gradle.properties references the Avatica version, so we could print the
appropriate links.
Michael>that need to be made that are
>There are changes
>such as documentation for new features, etc that should not be published
>immediately
Could we publish "master" documentation under /master or /nightly folders?
Anyway, it makes sense to publish multiple versions of the documentation.
So why don't we publish master
>As I was trying to update the website recently I noticed that there is a
>mixture of all kind of commits
The same goes for me.
The rebase-cherrypick dance seems to be more complicated than it should.
Vladimir
Stamatis> I still think we can solve this here by discussing a bit more
What do you mean by that?
Stamatis> Actualy, I think that it was stuck every single time that there
was an
error.
No idea. It does not stuck every time, and I have seen a lot of
checkstyle-triggered build failures.
Enrico>AFAIK in all of the apache projects I am contributing to as
committer every
Enrico>committer can sign the artifacts and be release manager
There existed META effort (which was hosted at
https://checker.apache.org/#META-files), however, it looks like
checker.apache.org was dropped recently
Francis>I noticed that the tag was being pushed to the calcite repository
Francis>and not the avatica repository
Indeed. The current script uses the same "folder" for both dist.apache.org
and Git repository name.
So a new indirection is needed behind the lines of
>I was referring to your own comment on the PR which indicated
>two missing checks
Exactly. You don't seem to evaluate the importance (and use frequency) of
the dropped checks.
In any case, you don't want to express your opinion as +1 or -1. That is
fine.
The vote is open till 2019-12-09
Michael> -0 because I haven't found checkstyle violations
Michael> and I don't like the idea of losing checks which are currently in
Michael> place
Would you please re-evaluate? I seems like you did not know which checks
are there and which are not.
To my best knowledge, the only missing check
Michael>wee could always pull the relevan code into Calcite
Do you realize Checkstyle does not support source-code based plugins?
It requires jar files. It would be non-trivial to build a custom checkstyle
plugin in the beginning of Calcite build and use it later in the same build.
Danny> it normalize the file end pretty well(avoid all kinds of file end)
I just realized there's JIRA which answers it:
https://issues.apache.org/jira/browse/CALCITE-490
One can't avoid 'file end'. The file has to end with something, and the use
of '// End' does not avoid file end.
'// End
Project-specific rules are missing, thus general rules apply:
https://www.apache.org/foundation/voting.html#votes-on-code-modification ,
https://www.apache.org/foundation/voting.html
It says:
> In this scenario, a negative vote constitutes a veto
Michael>I didn't interpret Jullian's -1 as a veto
> https://www.apache.org/foundation/voting.html#Veto
> A code-modification proposal may be stopped dead in its tracks by a -1
vote by a qualified voter.
> This constitutes a veto, and it cannot be overruled nor overridden by
anyone.
> Vetos stand
Julian>-1
I see your pain, however, please remember:
ASF> https://www.apache.org/foundation/voting.html#Veto
ASF> A veto without a justification is invalid and has no weight
The PR is a clear improvement, and we can always make it better (e.g. by
committing more code).
We even can roll it back
Hi,
I suggest we drop HydromaticFileSetCheck from the Checkstyle configuration.
It was a great journey (thanks Julian for HydromaticFileSetCheck), however,
lots of tools have improved since then, so we could upgrade to the better
tooling now.
[ ] +1, drop HydromaticFileSetCheck, just do it
[ ]
Vladimir Sitnikov created CALCITE-3559:
--
Summary: Drop HydromaticFileSetCheck from Checkstyle configuration
Key: CALCITE-3559
URL: https://issues.apache.org/jira/browse/CALCITE-3559
Project
Andrei>Perhaps split it into two parts:
That makes sense.
parameterized and classrules seem to be not that trivial to migrate.
Then I would revert several changes in
https://github.com/apache/calcite/pull/1621 and it would become a
committable PR.
Vladimir
Stamatis> If we go for the bulk modifications approach why not migrating
completely to JUnit5?
Migration to JUnit5 seems to be ok.
I've filed a draft that updates the APIs:
https://github.com/apache/calcite/pull/1621
It does not seem to touch too many places, and in most of the cases, it is
a
I find Java Flight Recorder and async-profiler to be good enough.
Both tools are free to use, they both can be used to profile production
systems, and they both produce actionable results.
Stamatis, Andrei, have you considered JFR / async-profiler?
Even though I do not object asking for a
Vladimir Sitnikov created CALCITE-3540:
--
Summary: CalciteAssert.pooled() does not seem to work
Key: CALCITE-3540
URL: https://issues.apache.org/jira/browse/CALCITE-3540
Project: Calcite
>IntelliJ IDEA 2017.3.5 (Ultimate Edition)
Could you please upgrade to 2019.3?
Vladimir
Juan> tried to `Go to File > Open… and open up
Which version of IDEA you are using?
Vladimir
>IntelliJ frequently fails to correctly load a project
It never happens to me.
I've been working recently on Gradle itself, Apache JMeter, Calcite
Avatica, Calcite, and it works.
Julian, is the issue reproducible?
Can you provide the exact steps?
Are there errors in IDEA logs? (
>I started with *core.autocrlf true*
Please ensure you are using core.autocrlf true, checkout the repository
from scratch, and retry.
It does work on my machine. It does work in GitHub Actions / Windows, it
does work at AppVeyor.
C:\temp>git config --get core.autocrlf
true
C:\temp>git config
Are you using core.autocrlf true? (which is a recommended configuration for
Windows by the way:
https://help.github.com/en/github/using-git/configuring-git-to-handle-line-endings
)
>the file
core\build\generated\sources\version\org\apache\calcite\jdbc\CalciteDriverVersion.java
which seems to be
>I also notice this output at the end, is this a known issue?
>Deprecated Gradle features were used in this build, making it incompatible
with Gradle 7.0.
It is.
>Also is there an easy way to specify number of threads for build and test
run?
Please find the documentation here:
>Here is the stacktrace:
java.lang.NoClassDefFoundError: org/gradle/api/internal/plugin
s/DefaultConvention
at org.jetbrains.plugins.gradle.tooling.builder.ProjectExtensio
nsDataBuilderImpl.buildAll(ProjectExtensionsDataBuilderImpl.groovy:50)
What is IDEA version?
Vladimir
>I have tried the build with new gradle facilities and it is much faster !
Thank you.
I hope you would like that "wrong import order", "whitespace at the end of
the file", "wrong copyright", etc issues
are now reported in a diff-style, and they are auto-fixable.
>Such as RelOptRulesTest, I run
Thank you, everybody, for the review.
I've merged the change, so the build is Gradle-based now. I hope I did not
break too many workflows with that change.
Just in case, the new build steps are on the website (
https://calcite.apache.org/docs/howto.html#running-tests ).
$ ./gradlew assemble #
Francis, thanks for raising this up.
+1 for disabling AppVeyor
I think we should need to keep Travis around for a while to evaluate how
GitHub Actions work.
Frankly speaking, I find Travis logs more responsive and a bit easier to
see at the moment (e.g. they are colored, etc, etc).
>integration
Vladimir Sitnikov created CALCITE-3516:
--
Summary: Bump net.java.dev.jna:jna to 5.5.0
Key: CALCITE-3516
URL: https://issues.apache.org/jira/browse/CALCITE-3516
Project: Calcite
Issue
Julian>We won’t ask you to merge or rebase again for a while
I suggest we merge Redis adapter **after** Gradle migration.
It would imply the need to convert pom.xml into Gradle build script.
Vladimir
Update: the build script is ready, the documentation is updated, CI passes
(including a new CI check with "build Avatica from master => test with
Calcite".
The only missing bit is removal of pom.xml files.
Note: both Maven and Gradle scripts work at the same time from the same
commit, however,
The error is caused by
private static final boolean USE_CANONICAL_HOSTNAME = true;
in
https://github.com/apache/calcite-avatica/blob/73fac75befaaf3caa3871656d0d0b7adbd91f7cf/core/src/main/java/org/apache/calcite/avatica/remote/AvaticaCommonsHttpClientSpnegoImpl.java#L58
In other words,
The Gradle build scripts are ready (
https://github.com/apache/calcite/pull/1571 )
There might be minor issues like "a resource file missing in one of the
jars",
however, I expect there will be no more major changes.
Unfortunately, there are changes that prevent building with both Maven and
>Currently we have two ways of running/skipping a slow test
You wanted to say "at least two", did you?
There are properties like calcite.test.druid which target a single test
class.
>Does this make sense considering the ongoing work of moving to Gradle and
JUnit5?
The current Gradle-JUnit5
I typically use http://www.webgraphviz.com/
Try pasting digraph { ... } part of the exception there.
Vladimir
Xiening,
The bug is "obvious" when you open a Graphviz diagram.
Just wondering: have you tried?
The issue is EnumerableTableFunctionScan consumes Subset#0.NONE.[0] as an
input, which is never implementable.
The fix is trivial: EnumerableTableFunctionScanRule must convert inputs to
As you might know, Avatica is already migrated to Gradle.
On top of that, Calcite is almost there as well:
https://github.com/apache/calcite/pull/1571
The missing bit is probably the notion of tests "fast/slow/etc/etc".
Unfortunately, there are tests that assert on folder layout like
>localhost but the principal being looked up is FQDN.
Where that FQDN is taken from?
Vladimir
The list you provide is outdated.
>these plugins weren't added to the Gradle build at
>some point.
Please feel free to execute "./gradlew tasks" and check what is available
To my best knowledge, the build script is good enough.
The missing items are
> * "Unused dependency"
> * "Used but
Hi,
I've added GitHub Actions for CI, and it looks like AvaticaSpnegoTest fails
again.
The failure is the same for Maven and Gradle, so I expect it is not caused
by Gradle migration
Maven: https://github.com/vlsi/calcite-avatica/runs/296688961#step:4:31296
Gradle:
As a follow-up, I would like to thank everybody who helped to polish the
change: Stamatis, Francis, et al.
Vladimir
Ok, so I committed Maven -> Gradle change.
Even though I expect it to be ok, there might be glitches here and there.
It would be great if someone could verify if the generated pom.xml
files are good.
Note: Avatica pom.xml files are generated at build time, so they will be
different from those
>I think it's also possible to get CALCITE-3158 (replace Maven with Gradle)
I think this is more or less ready to go.
The commits should probably be squashed, and I can do that if no-one else
plans to review the change.
There's "remove protobuf-generated files" change.
Is it truly needed to
> mvn clean test
[ERROR] The goal you specified requires a project to execute but there is
no POM in this directory
Vladimir, please push missing files
Vladimir
Francis>files updated to switch over to using a Gradle docker container to
run
Francis>the build/release steps as well.
Docker does unify the environment, however, I'm not sure it is good to make
Docker the only option for the release.
Gradle enables single-command "build+test+publish to
Roman Elizarov raises valid points re 'interrupts are too hard (or even
impossible) to get right':
https://twitter.com/relizarov/status/1184460504238100480
Vladimir
Statamis,
"cooperative to interrupt" sounds a nice idea, however, I have been bitten
multiple times by improper interrupt handling (not really with Calcite, but
with other projects).
In other words, it is good when everybody supports that.
However, the other libraries might receive unexpected
I guess the missing bit is the bot that automatically creates JIRA for
Dependabot issues.
What if we create one?
Vladimir
>Definitely would simplify some of
>the RM steps for a release.
I've recently added GitHub Actions config to test Apache JMeter for Windows
and macOS.
So far I'm impressed.
Pros:
* It starts quite fast. Appveyor might take 2-3hours to even start the
build.
Actions start very fast, and it does
>One diff, for example,
>is that spotless imports static first but checkstyle prefers static last,
>which I haven't found a way to config.
I agree Spotless documentation is a bit vague, however Spotless-Gradle
orders statics for me.
Here's an example:
// import statics first
importOrder("static
Stamatis>Out of curiosity does anybody know if popular DBMS (Postgres)
support "hoisting"?
PostgreSQL does support it, and here's a reproducible case when that
feature makes the query 300 times slower:
https://gist.github.com/vlsi/df08cbef370b2e86a5c1
Vladimir
Rui>Import order and length of a single line of code is too long were what I
faced.
I guess both of them could be configured in IDE.
Even though it might look boring/complicated, configuring import order in
IDE pays off quickly.
Rui>In Apache Beam, we are using gradle and have adopted
Rui>when developing in Calcite repo?
Hi, could you please clarify which issues do you run into the most?
Rui>auto-formatter to fix checkstyle error
That would really be awesome.
Unfortunately, my experience with Checkstyle project 5 years ago was that
core developers did not value "automatic
Thanks everybody for helpful comments.
At this point I'm confident the change is good, so I'm inclined to just
commit it and cleanup glitches later if any.
I'm inclined to remove pom.xml files from Git control as well.
PS. There was Gradle-produced release candidate. You can find it under the
Stamatis, thanks for your work on this.
Stamatis>The checksum hash that was communicated in the vote email was wrong
Stamatis>given
Stamatis>that the correct one was send along with the artifacts and people
used this
Stamatis>for the checks I assume there is no problem.
I'm inclined that we
+1
There are OsAdapterTest.testPs and testPsDistinct failures, however they
are well known.
Vladimir
Frankly speaking I would refrain from adding `table WITH ...`
It is not clear if you allow to add hints at subquery level. I think we
should allow that.
For instance, https://github.com/ossc-db/pg_hint_plan allows top-level
hints only, and that is really hard to use when SQL is build dynamically.
It is a "well known"
https://issues.apache.org/jira/browse/CALCITE-2816 PsTableFunction
fails in Russian locale
In other words, "float number parsing is locale-sensitive".
Vladimir
Hi,
Can you please try if
https://github.com/apache/calcite-avatica/pull/85/files solves the problem
for you?
The issue is known as CALCITE-2704.
Vladimir
Ah, good old drill-fmpp-plugin
For instance: we migrate to Gradle, and the problem is solved automatically.
Just a datapoint: Apache JMeter has recently switched to Gradle.
The PR for Avatica (including a release candidate waiting for review) is
ready:
On the past weekend I've updated release plugins, so most of my notes are
resolved.
TL;DR:
1) ./gradlew prepareVote -Prc=1 -Pasf # build and stage RC1
2) ./gradlew prepareVote -Prc=2 -Pasf # build and stage RC2
2) ./gradlew publishDist -Prc=2 -Pasf # release RC2 as a release
-Prc is mandatory
Michael>Sounds like a great idea to me. I assume the Javadoc there is for
the
current snapshot?
Javadocs are built during the release candidate preparation.
Here's Git history:
https://github.com/apache/jmeter-site-preview/commits/gh-pages
I have just created JMeter RC2, and you can see the diff
Here's "site preview" repository for Apache JMeter.
I think it does simplify the review at virtually zero cost.
Any thoughts?
--- cut ---
You can read the New and Noteworthy section with some screenshots to
illustrate improvements and full list of changes at:
NOTE: This is NOT a vote for releasing Apache Calcite Avatica.
The purpose is mail is to let you know the state of Gradle-based build.
If would be nice if you could review Apache Calcite Avatica 1.16.0-rc1
artifacts.
Note: I have not removed Maven-related files from Git, however the
intention is
Here's an update: Gradle-based build seems to be in a quite a good shape.
Just in case: it builds source artifacts (in zip and tgz formats), it
assembles licenses, it runs tests.
The missing bit is /docker/pom.xml -- I have no idea what it does.
Any feedback?
Vladimir
Julian>It is difficult to build LICENSE and NOTICE files automatically
So what?
I claim that automatic license compilation produces better results than
currently present license files in Apache Calcite source code.
I claim that automatic license compilation produces better results than
currently
Francis> I've been burnt in the past where generated documentation is for
master
Francis> and not the latest release, causing some confusion.
On contrary, it is better when the documentation has version selector, so
one could pick an appropriate version.
Then "snapshot" documentation is good for
Francis>javadocs takes around 20 minutes to build
I did not thought it takes so much time.
"tag" trigger for javadocs is clever, and I just thought we might want to
be able to update the wording on the site javadoc without releasing Calcite.
That is why I suggested to build javadoc for all site
Francis>- For Javadocs (Calcite and Avatica) trigger and build when a final
version is tagged.
I think javadocs could be built on each push to site branch.
Vladimir
Julian>I don’t think we should drop Java 8 or 9 support yet.
I support we keep Java 8, and we drop Java 9.
Vladimir
401 - 500 of 780 matches
Mail list logo
