jmckenzie-dev commented on code in PR #2554: URL: https://github.com/apache/cassandra/pull/2554#discussion_r1286292443
########## .build/config/jobs.cfg: ########## @@ -0,0 +1,183 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# +# Contains definitions of all ci jobs in the system. Composed into pipelines; see pipelines.cfg +# + +# Job config format is key-value, colon delimited. +# Order does not matter, and subsequent jobs are delimited by either subsequent "job:N" tags or EOF +# All parameters should be processed in sequential order with child parameters executing after parent +# +# job:<name> +# description:<description> +# env:{key=val,key=val,key=val} +# string:Comma-delimited list of key value pairs to export in runtime env for test +# NOTE: env parameters should *all* be evaluated from the root of the hierarchy. Child env params that +# override the parent are acceptable but the totality of env should be a composite of each env: +# from root on down in sequential order. +# cmd:<to_run> +# string:Command used to run the tests resolved by testlist +# testlist:{command} +# string:Command run on system to generate comma-delimited list of all test units to run in target +# Note:This is expected to run in a terminal shell context +# The following parameters indicate the max allowable of a given resource +# memory:N +# integer:in GB +# cpu:N +# integer:logical cpu count +# storage:N +# integer:GB allowable used +# +# A job can inherit the configuration from another job via the following syntax: +# job:base_job +# ... params ... +# job:child +# parent:base_job +# override:new_val +# Any parameter defined in the child job is expected to override the parameters of the parent +# +# Other parameters of interest: +# Test timeouts per suite are in build.xml + +#----------------------------------------------------------------------------- +# Base reference profile; add new parameters here to ensure all jobs have a reasonable default +#----------------------------------------------------------------------------- +job:base + memory:6 + cpu:4 + storage:100 + env:ANT_TEST_OPTS="-Dno-build-test=true -Dtmp.dir=${TMP_DIR}" + +#----------------------------------------------------------------------------- +# JVM based single node tests +#----------------------------------------------------------------------------- +job:unit + parent:base + testlist:find test/unit -name '*Test.java' | sed "s;^test/unit/;;g" | sort + cmd:ant test ${ANT_TEST_OPTS} + +job:cdc + parent:unit + description:Runs test suites with CDC enabled + +job:compression + parent:unit + description:Runs test suites with SSTable compression enabled + +job:test-oa + parent:unit + description:Tests oa sstable format + +job:test-system-keyspace-directory + parent:unit + description:Execute unit tests with a system keyspaces directory configured + +job:test-tries + parent:unit + description:Execute unit tests with trie memtables + +#----------------------------------------------------------------------------- +# JVM based multi-node tests +#----------------------------------------------------------------------------- +job:jvm-dtest + parent:base + description:Execute in-jvm dtests excepting upgrade tests + testlist:find test/distributed -name '*Test.java' | sed "s;^test/distributed/;;g" | grep -v "upgrade" sort + cmd:ant test ${ANT_TEST_OPTS} + +job:jvm-dtest-upgrade + parent:jvm-dtest + description:Execute upgrade in-jvm dtests + testlist:find test/distributed -name '*Test.java' | sed "s;^test/distributed/;;g" | grep "upgrade" sort + +#----------------------------------------------------------------------------- +# Python based tests +#----------------------------------------------------------------------------- +job:dtest + parent:base + description:Python based multi-node distributed tests with default configuration + memory:16 + cpu:4 + storage:100 + env:PYTHONIOENCODING="utf-8",PYTHONUNBUFFERED=true,CASS_DRIVER_NO_EXTENSIONS=true,CASS_DRIVER_NO_CYTHON=true,CCM_MAX_HEAP_SIZE="1024M",CCM_HEAP_NEWSIZE="512M",CCM_CONFIG_DIR=${DIST_DIR}/.ccm,NUM_TOKENS="16",CASSANDRA_SKIP_SYNC=true,TMPDIR="$(mktemp -d /tmp/run-python-dtest.XXXXXX)" + env:DTEST_ARGS="--use-vnodes --num-tokens=${NUM_TOKENS} --skip-resource-intensive-tests" + +job:dtest-novnode + parent:dtest + description:Python based multi-node distributed tests without vnodes + +job:dtest-offheap + parent:dtest + description:Python based multi-node distributed tests with off-heap memtables + env:DTEST_ARGS="--skip-resource-intensive-tests --keep-failed-test-dir" + env:DTEST_ARGS="--use-vnodes --num-tokens=${NUM_TOKENS} --use-off-heap-memtables --skip-resource-intensive-tests" Review Comment: Similar to above, this is spelled out in the documentation for the file: ``` # env:{key=val,key=val,key=val} # string:Comma-delimited list of key value pairs to export in runtime env for test # NOTE: env parameters should *all* be evaluated from the root of the hierarchy. Child env params that # override the parent are acceptable but the totality of env should be a composite of each env: # from root on down in sequential order. ``` Now, diverging from .yaml semantics here: probably not a great idea. That said, the current paradigm (ability to inherit params from base) is somewhat predicated on the additive capabilities here (i.e. > 1 env: key / value can exist, append only). My preference is to keep the ability to have divergent env in a base job config vs. one declared below, but if you'd prefer the locality (up to and including completely removing the ability to derive from a base config), I can see the merits of that argument. Would make for a much uglier, longer, repetitive configuration file, down which road leads things like thousands of lines of text in our .circle configs. So not my _preference_, but also not a hill I'm looking to die on. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
