2018-01-24 0:08 GMT+01:00 Lukasz Cwik <[email protected]>:
>
>
> On Tue, Jan 23, 2018 at 2:51 PM, Romain Manni-Bucau <[email protected]
> > wrote:
>
>>
>>
>> 2018-01-23 23:44 GMT+01:00 Kenneth Knowles <[email protected]>:
>>
>>> Romain - really good point about docs getting out of date.
>>>
>>> > On 01/23/2018 09:29 AM, Romain Manni-Bucau wrote:
>>>> >> Yep,
>>>> >>
>>>> >> a compromise can be to ensure all custom tasks have a description
>>>> maybe:
>>>>
>>>
>>> This is a great idea.
>>>
>>> Let's do both! I think the thing that the comments in the gradle code
>>> cannot capture are the ways that you might combine them, like the way you
>>> override properties, etc.
>>>
>>> On Tue, Jan 23, 2018 at 9:13 AM, Ismaël Mejía <[email protected]> wrote:
>>>
>>>> I used to test kafka
>>>> with a different version of the client quickly by doing this with
>>>> maven:
>>>>
>>>> mvn verify -Prelease -Dkafka.clients.version=0.10.0.1 -pl
>>>> 'sdks/java/io/kafka'
>>>>
>>>> but I don't have any idea of how to do this on Gradle.
>>>>
>>>
>>> Did you figure this out. Luke - can you suggest something?
>>>
>>>
> Some ideas:
> * We can define a static set of properties like we did in maven in the
> gradle.properties which allow users to override them.
>
Isn't it only for the daemon start and therefore now that usable when you
run it to validate you didn't break the compat with previous versions?
> * We can just dynamically set versions of libraries by adding a function
> to build_rules.gradle which checks to see if there is a property defined
> for that library and use it. Many ways to map names from just the library
> name, or library name + group automatically. This way we don't have to
> maintain a large list of properties and each library can be overridden from
> the command line.
>
+1, kind of maven -D becoming gradle -P so quite natural IMHO
> * If we defined everything in the root build.gradle then you can edit the
> single line related to the property version.
> * Add a special property which is a list of overrides you want for library
> dependencies/versions and have it applied in build_rules.gradle.
>
Can be needed as well for dependencies "linked" but more as a nice to have
rather as a blocker IMHO
>
>
>> Of course everybody has a different workflow, but I am pretty sure
>>>> there are some common tasks that we can document for people (like me)
>>>> that are new to gradle.
>>>
>>>
>>> Yea, actually, it never even occurred to me that you would use the
>>> command line to test against other Kafka versions :-).
>>>
>>> I have private gists with lots of maven invocations for doing things
>>> like running ValidatesRunner or example ITs on just a particular runner. It
>>> always requires multiple maven commands that are both multiple lines. I was
>>> halfway for a while but I am now all gradle. I will start building the same
>>> thing.
>>>
>>>
>>>> - How to skip the Java/Python/Go build depending on your priorities.
>>>>
>>>
>>> IMO they should NOT run by default. Everything should be
>>> dependency-driven. When I ask to run Java SDK tests, or Java examples IT on
>>> the Flink Runner, it is *incorrect* for any Python or Go builds to run.
>>>
>>
> This is already the case. Only if you do './gradlew build' will everything
> get built. Unless your specifically saying build only Go code which is
> vague because we have some cross language dependencies where Go requires
> java to build. Best to stick with gradles buildNeeded and buildDependents.
>
>
>>
>>>
>>>> - How to run an individual IntegrationTest or ValidatesRunner test.
>>>>
>>>
>>> Yes, #1 use case
>>>
>>>
>>>> - How to skip findbugs,checkstyle,javadoc gneeration, etc. To have an
>>>> ultra quick build.
>>>>
>>>
>>> Again, I think they should be independent tasks. I should be able to run
>>> *any* of them without running *any* of the others. It is incorrect for
>>> anything to cause any other thing to run if it does not directly require
>>> its outputs.
>>>
>>> There can be an aggregated "verify" command but I will actually very
>>> rarely run that until I am done with a large chunk of work.
>>>
>>> As for what the aggregated "verify" command does, people kept arguing
>>> about what to make default. As long as we have a correct build for
>>> individual checks (aka not running extra things) then I am happy for the
>>> default to be long and slow, but we should still build profiles for both.
>>>
>>
>> Hmm, did you read it right Kenn? I think the idea was to skip all
>> validation/sanity checks tasks at once (gradle xxxx -Pfast) instead of
>> doing it manually (gradle -x findbugs -x checkstyle etc...)
>>
>
> Add to build_rules.gradle to not apply these plugins if the 'fast' profile
> is enabled.
>
+1, that was what I had in mind
Side note on that topic: shouldnt build_rules be split in versions and
shared_plugins scripts?
>
>
>>
>>
>>>
>>> Kenn
>>>
>>>
>>>
>>>> >>
>>>> >> diff --git a/examples/java/build.gradle b/examples/java/build.gradle
>>>> >> index 0fc0b17df..001bd8b38 100644
>>>> >> --- a/examples/java/build.gradle
>>>> >> +++ b/examples/java/build.gradle
>>>> >> @@ -130,7 +130,7 @@ def preCommitAdditionalFlags = [
>>>> >> dataflowStreamingRunner: [ "--streaming=true" ],
>>>> >> ]
>>>> >> for (String runner : preCommitRunners) {
>>>> >> - tasks.create(name: runner + "PreCommit", type: Test) {
>>>> >> + tasks.create(name: runner + "PreCommit", type: Test, description:
>>>> "Run tests
>>>> >> for runner ${runner.replace('Runner', '')}") {
>>>> >> def preCommitBeamTestPipelineOptions = [
>>>> >> "--project=apache-beam-testing",
>>>> >> "--tempRoot=gs://temp-storage-for-end-to-end-tests",
>>>> >>
>>>> >>
>>>> >>
>>>> >> Romain Manni-Bucau
>>>> >> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>> >> <https://rmannibucau.metawerx.net/> | Old Blog
>>>> >> <http://rmannibucau.wordpress.com> | Github <
>>>> https://github.com/rmannibucau> |
>>>> >> LinkedIn <https://www.linkedin.com/in/rmannibucau>
>>>> >>
>>>> >> 2018-01-23 8:45 GMT+01:00 Jean-Baptiste Onofré <[email protected]
>>>> >> <mailto:[email protected]>>:
>>>> >>
>>>> >> Hi Romain,
>>>> >>
>>>> >> I think we are pretty close: agree to add some explicit tasks &
>>>> projects names.
>>>> >>
>>>> >> We can add additional tasks like skipAudit, for instance.
>>>> >>
>>>> >> As reminder, gradle tasks provides the list of tasks and gradle
>>>> projects
>>>> >> provides the list of projects/modules.
>>>> >>
>>>> >> Regards
>>>> >> JB
>>>> >>
>>>> >> On 01/23/2018 08:12 AM, Romain Manni-Bucau wrote:
>>>> >> > Hmm, I have to admit docs dont have my favor cause they are
>>>> easily outdated and
>>>> >> > hard to search but you hit a good point. Starting by renaming
>>>> properly the tasks
>>>> >> > and maybe writing what is done in build files - since it is
>>>> code and even "api
>>>> >> > for dev", it requires as much comments than the main api - can
>>>> be better to start.
>>>> >> >
>>>> >> > Also a big switch flag to bypass checkstyle/findbugs/... can
>>>> be good while in
>>>> >> > dev since these phases cost a looot for nothing while you
>>>> validates your code in
>>>> >> > runners modules for instance.
>>>> >> >
>>>> >> > Le 23 janv. 2018 07:15, "Kenneth Knowles" <[email protected]
>>>> <mailto:[email protected]>
>>>> >> > <mailto:[email protected] <mailto:[email protected]>>> a écrit :
>>>> >> >
>>>> >> > On Mon, Jan 22, 2018 at 10:03 PM, Romain Manni-Bucau <
>>>> [email protected] <mailto:[email protected]>
>>>> >> > <mailto:[email protected] <mailto:
>>>> [email protected]>>> wrote:
>>>> >> >
>>>> >> > @Kenneth: why not dropping the doc for a script with
>>>> comments in the
>>>> >> > project? A "RUNME.sh" ;).
>>>> >> >
>>>> >> >
>>>> >> > That's cool, too, but also any shell one liner can be a
>>>> gradle one
>>>> >> liner or
>>>> >> > mvn two/three liner :-). it is just trading one command
>>>> that you cannot
>>>> >> > guess easily for a different one that you still can't
>>>> guess easily.
>>>> >> >
>>>> >> > For example, are the SparkRunner ValidatesRunner tests in
>>>> the
>>>> >> SparkRunner or
>>>> >> > the core SDK or a third module that integrates the two?
>>>> And why would you
>>>> >> > know that the example ITs are called
>>>> "sparkRunnerPreCommit"? It
>>>> >> doesn't even
>>>> >> > make sense really to have "precommit" or "postcommit"
>>>> except as aliases to
>>>> >> > make it easy to repro Jenkins' behavior - they have no
>>>> other intrinsic
>>>> >> meaning.
>>>> >> >
>>>> >> > So I was proposing a mapping from "full sentence +
>>>> description" to one
>>>> >> liner
>>>> >> > to help people navigate the targets that we set up. Some
>>>> web page or doc
>>>> >> > that people can just quickly scan to find out to do common
>>>> things, easier
>>>> >> > than groovy or XML.
>>>> >> >
>>>> >> > Kenn
>>>> >> >
>>>> >> >
>>>> >>
>>>> >> --
>>>> >> Jean-Baptiste Onofré
>>>> >> [email protected] <mailto:[email protected]>
>>>> >> http://blog.nanthrax.net
>>>> >> Talend - http://www.talend.com
>>>> >>
>>>> >>
>>>> >
>>>> > --
>>>> > Jean-Baptiste Onofré
>>>> > [email protected]
>>>> > http://blog.nanthrax.net
>>>> > Talend - http://www.talend.com
>>>>
>>>
>>>
>>
>
